freemcplab / Annuaire / Playground / web-scraper

● Communauté yfe404 ⚡ Instantané

web-scraper

Name: web-scraper (Claude Skill)
Author: yfe404

Phased reconnaissance scraper that prefers APIs > sitemaps > HTML, picks the right Apify template (Cheerio vs Playwright), and only escalates anti-detection when signals warrant.

Intelligent web scraping via Claude Code. Runs a 5-phase reconnaissance (Phases 0–5) that tries the cheapest extraction path first: public APIs, sitemap feeds, structured data. Only when those fail does it consider browser automation, and only when protection signals appear does it layer on stealth. Built around TypeScript-first Apify Actor development, with templates (Cheerio for static, Playwright for JS-heavy).

▶ Voir la démo 📦 Installer 💡 Cas d'usage

Pourquoi l'utiliser

Fonctionnalités clés

Phased reconnaissance — APIs > sitemaps > static HTML > browser
Template picker: Cheerio for static, Playwright for JS-heavy
Apify Actor TypeScript-first workflow
Selective stealth: only when protection signals detected
Session recording + replay for debugging

Démo en direct

Aperçu en pratique

web-scraper-skill.replay ▶ prêt

0/0

Installation

Dépôt de l’auteur · Documentation officielle des Skills Claude

Cas d'usage

Usages concrets : web-scraper

Scrape a static listing site into a structured dataset

👤 Data engineers pulling public data (directories, price lists, public records) ⏱ ~45 min intermediate

Quand l'utiliser : You need a dataset from a public site that doesn't have an API.

Déroulement

Let the skill do recon

Use web-scraper. Target: https://example.com/listings. I want name + URL + category. Recon first — tell me the cheapest extraction path.✓ Copié

→ Skill reports: 'sitemap.xml available, use Cheerio'
Scaffold the Apify Actor

Scaffold a TypeScript Apify Cheerio actor for that extraction.✓ Copié

→ Actor tree + main.ts ready to run
Run and iterate

Run locally on 10 pages; tighten selectors if needed.✓ Copié

→ Clean JSON output

Résultat : An Apify Actor you can deploy for scheduled scrapes.

Pièges

Jumping to Playwright when Cheerio would do — Trust the recon — headful browsers 10x the cost unnecessarily

Combiner avec : apify · filesystem

Discover and use a site's undocumented JSON API instead of HTML

👤 Scraper devs who want reliability ⏱ ~30 min intermediate

Quand l'utiliser : The page is a SPA and HTML is gross, but the XHR calls are clean JSON.

Déroulement

Run API-discovery phase

Use web-scraper phase 1 — API discovery on https://example.com/app. Enumerate XHR/fetch endpoints.✓ Copié

→ List of endpoints with observed payloads
Build the JSON-based actor

Generate an actor that hits those endpoints directly with auth as needed.✓ Copié

→ Lightweight fetch-based actor

Résultat : A far more stable scrape than HTML parsing.

Pièges

Private/session-auth APIs that break when token rotates — Plan token-refresh logic or fall back to browser flow

Combinaisons

Associez-le à d'autres MCPs pour un effet X10

web-scraper-skill + apify

Deploy the scaffolded actor to Apify for scheduled runs

Deploy this actor to my Apify account and schedule it daily.✓ Copié

web-scraper-skill + filesystem

Keep the actor code in-repo alongside the consuming app

Scaffold into scrapers/ and commit with the main project.✓ Copié

Outils

Ce que ce MCP expose

Outil	Entrées	Quand appeler
recon	url	Always first
scaffold_actor	template (cheerio\|playwright), target	After recon picks template
record_session	url	Debugging dynamic sites
run_local	actor path, limit	Iteration phase

Coût et limites

Coût d'exécution

Quota d'API: Apify has its own compute + proxy quotas
Tokens par appel: Moderate — scaffold and iteration loops
Monétaire: Free skill; Apify costs separate
Astuce: Prefer Cheerio — one of the cheapest run profiles on Apify

Sécurité

Permissions, secrets, portée

Portées minimales : Apify API token with actor:read + actor:write

Stockage des identifiants : APIFY_TOKEN in env

Sortie de données : Whatever sites you target + Apify platform

Respect robots.txt and each site's Terms of Service. Skill is not an anti-ToS tool — use it on data you're allowed to collect.
Public, non-authenticated data only unless you have explicit permission.

Dépannage

Erreurs courantes et correctifs

Cheerio returns empty selectors

Content is JS-rendered — rerun recon, expect Playwright template

Playwright times out

Bump navigation timeout; consider waiting for specific selector instead of networkidle

403 / bot-block page

Stop and reconsider. This is the legit signal to re-check ToS, not a cue to escalate stealth.

Alternatives

web-scraper vs autres

Alternative	Quand l'utiliser	Compromis
Direct Apify console	You already know which template you need	No recon phase
firecrawl	You just need markdown of a page, not structured extraction	No actor scaffolding

Plus

Ressources

📖 Lire le README officiel sur GitHub

🐙 Voir les issues ouvertes

🔍 Parcourir les 400+ serveurs MCP et Skills