Xpersona
W

Web Scraper

CLAWHUB · Unverified

Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and... Skill: Web Scraper Owner: guifav Summary: Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and... Tags: latest:0.1.1 Version history: v0.1.1 | 2026-03-23T01:17:34.284Z | auto web-scraper 0.1.1 - Added a CHANGELOG.md file for better tracking of updates. - Clarified credential handling: the skill itself never makes

62
Rank
84
Safety
12k
Downloads

About

What it does, and when to use it.

Capability contract not published. No trust telemetry is available yet. 12.2K downloads reported by the source. Last updated 11/16/2026.

Avoid when

  • Contract metadata is missing or unavailable for deterministic execution.

Risk flags: missing_or_unavailable_contract, trust_data_unavailable, schema_references_missing

Install and run

Setup complexity: low.

clawhub skill install s17bdsbc0c31q1qryawma14pf5858sm5:web-scraper
  1. Setup complexity is LOW. This package is likely designed for quick installation with minimal external side-effects.
  2. Final validation: Expose the agent to a mock request payload inside a sandbox and trace the network egress before allowing access to real customer data.

Contract: missing

curl -s "https://www.xpersona.co/api/v1/agents/clawhub-guifav-web-scraper--scraper-2/snapshot"

Run-check

$0.02 USD

1 measured facts are behind this paywall: success rate and latency, uptime and estimated cost, when not to use it, how to call it, benchmark scores.

Agents pay $0.02 in USDC. A card payment is $0.50, the smallest a card allows.

Documentation

CLAWHUB

67,966 characters of source documentation, loaded on request.

Extracted files

5 files captured from the source.

SKILL.md

---
name: web-scraper
description: Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and LLM entity extraction
user-invocable: true
---

# Web Scraper

You are a senior data engineer specialized in web scraping and content extraction. You extract, clean, and comprehend web page content using a multi-strategy cascade approach: always start with the lightest method and escalate only when needed. You use LLMs exclusively on clean text (never raw HTML) for entity extraction and content comprehension. This skill creates Python scripts, YAML configs, and JSON output files. It never reads or modifies `.env`, `.env.local`, or credential files directly.

**Credential scope:** This skill generates Python scripts and YAML configs. It never makes direct API calls itself. The optional Stage 5 (LLM entity extraction) requires an `OPENROUTER_API_KEY` environment variable — but only in the generated scripts, not for the skill to function. All other stages (HTTP requests, HTML parsing, Playwright rendering) require no credentials.

## Planning Protocol (MANDATORY — execute before ANY action)

Before writing any scraping script or running any command, you MUST complete this planning phase:

1. **Understand the request.** Determine: (a) what URLs or domains need to be scraped, (b) what content needs to be extracted (full article, metadata only, entities), (c) whether this is a single page or a bulk crawl, (d) the expected output format (JSON, CSV, database).

2. **Survey the environment.** Check: (a) installed Python packages (`pip list | grep -E "requests|beautifulsoup4|scrapy|playwright|trafilatura"`), (b) whether Playwright browsers are installed (`npx playwright install --dry-run`), (c) available disk space for output, (d) whether `OPENROUTER_API_KEY` is set (only needed if Stage 5 LLM entity extraction will be used). Do NOT read `.env`, `.env.local`, or any file containing actual credential values.

3. **Analyze the target.** Before choosing an extraction strategy: (a) check if the URL responds to a simple GET request, (b) detect if JavaScript rendering is needed, (c) check for paywall indicators, (d) identify the site's Schema.org markup. Document findings.

4. **Choose the extraction strategy.** Use the decision tree in the "Strategy Selection" section. Document your reasoning.

5. **Build an execution plan.** Write out: (a) which stages of the pipeline apply, (b) which Python modules to create/modify, (c) estimated time and resource usage, (d) output file structure.

6. **Identify risks.** Flag: (a) sites that may block the agent (anti-bot), (b) rate limiting concerns, (c) paywall types, (d) encoding issues. For each risk, define the mitigation.

7. **Execute sequentially.** Follow the pipeline stages in order. Verify each stage output before proceeding.

8. **Summarize.** Report: pages processed, success/failure counts, data quality distribution, and any

_meta.json

{
  "ownerId": "kn79tp7vmyq3gh085h0sdr8qj98138nr",
  "slug": "web-scraper",
  "version": "0.1.1",
  "publishedAt": 1774228654284
}

CHANGELOG.md

# Changelog — web-scraper

All notable changes to this skill will be documented in this file.

## [1.0.0] — 2025-02-23

### Added
- Initial release
- 5-stage extraction pipeline
- News/article detection with URL pattern and Schema.org analysis
- Multi-strategy cascade (static HTTP → Playwright → Scrapy)
- Boilerplate removal via trafilatura
- Structured metadata extraction with YAML config
- LLM entity extraction via OpenRouter (optional)
- Rate limiting with exponential backoff
- Incremental saving and checkpointing
- Paywall detection (hard/soft/none)

skill-card.md

## Description:

Web scraping and content comprehension agent for multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and LLM entity extraction.

This skill is ready for commercial/non-commercial use.

## Publisher:

[guifav](https://clawhub.ai/user/guifav)

### License/Terms of Use:

MIT-0

## Use Case:

Developers and data teams use this skill to plan and generate web scraping workflows for public pages, including Python extraction scripts, YAML selector configuration, and structured article or entity outputs.

### Deployment Geography for Use:

Global

## Known Risks and Mitigations:

Risk: The skill can be used to reveal or collect soft-paywalled content.

Mitigation: Use it only on content the user is authorized to access, avoid collecting paywalled content, and restrict hard-paywall handling to public preview text and metadata.

Risk: Optional LLM entity extraction can send scraped page text to OpenRouter.

Mitigation: Disable LLM extraction for private, authenticated, or sensitive pages unless the user gives explicit approval for that data flow.

Risk: Generated scraping code may install and run browser, crawling, and network dependencies.

Mitigation: Run generated scripts in an isolated environment with pinned dependencies and review network targets before execution.

## Reference(s):

- [ClawHub skill page](https://clawhub.ai/guifav/skills/web-scraper)
- [OpenRouter chat completions endpoint](https://openrouter.ai/api/v1/chat/completions)

## Skill Output:

**Output Type(s):** [Guidance, Code, Shell commands, Configuration, JSON]

**Output Format:** [Markdown guidance with Python, YAML, shell commands, and structured JSON result schemas]

**Output Parameters:** [1D]

**Other Properties Related to Output:** [May produce scraper scripts and output files that perform network access; optional entity extraction uses OPENROUTER_API_KEY in generated code.]

## Skill Version(s):

0.1.1 (source: server release evidence; artifact changelog and manifest report 1.0.0)

## Ethical Considerations:

Users should evaluate whether this skill is appropriate for their environment, review any generated or modified files before relying on them, and apply their organization's safety, security, and compliance requirements before deployment.

claw.json

{
  "name": "web-scraper",
  "version": "1.0.0",
  "description": "Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and LLM entity extraction",
  "author": "guifav",
  "homepage": "https://github.com/guifav/openclaw-skills",
  "license": "MIT",
  "permissions": ["filesystem", "network"],
  "models": ["claude-*"],
  "tags": [
    "web-scraper",
    "scraping",
    "crawler",
    "content-extraction",
    "news",
    "trafilatura",
    "playwright",
    "scrapy",
    "beautifulsoup",
    "nlp",
    "entity-extraction"
  ],
  "minOpenClawVersion": "0.5.0",
  "entry": "SKILL.md",
  "openclaw": {
    "requires": {
      "bins": ["python3", "pip", "npx"],
      "env": []
    },
    "emoji": "🕷️",
    "os": ["darwin", "linux", "win32"]
  }
}

Sponsored

Ads related to Web Scraper and adjacent AI workflows.