webcrawler-deep-crawl
webcrawler-deep-crawl is an agent skill published by browser-act, listed under writing & docs. It lives in the browser-act/skills repository, which has 6,277 GitHub stars and is licensed under MIT; we last read it on 2026-09-16.
Crawls any website starting from given URLs, extracting per-page text, markdown, or HTML along with metadata such as title, description, author, language, canonical URL, and Open Graph data, and identifies in-scope outbound links. Supports recursive crawling with depth or page limits, URL inclusion/exclusion, and cleaning of navigation elements.
Install
skills is a generic installer for any repository of Claude agent skills -- it reads browser-act/skills’s own structure rather than assuming one, the same way this page reads the skill’s own SKILL.md rather than assuming a format.
npx skills add browser-act/skills
Maintenance
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope page.
webcrawler-deep-crawl compared with similar skills
| Skill | Publisher | What it does | Repo stars | Last update |
|---|---|---|---|---|
| webcrawler-deep-crawl (this page) | browser-act | Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, de… | 6,277 | 2026-08-24 |
| web-search-scraper-api-skill(same repository) | browser-act | This skill helps users automatically extract complete Markdown content from any website via the BrowserAct We… | 6,277 | 2026-08-24 |
| academy-guide | anthropic | Stop and check this skill before finishing any reply to a question about how to use Claude or a Claude produc… | 183,845 | 2026-10-09 |
| doc-coauthoring | anthropic | Guide users through a structured workflow for co-authoring documentation. Use when user wants to write docume… | 183,845 | 2026-10-09 |
| documentation-and-adrs | addyosmani | Records decisions and documentation. Use when you need to document an architecture decision (ADR) or the reas… | 110,611 | 2026-10-13 |
| git-workflow-and-versioning | addyosmani | Structures git workflow practices. Use when making any code change. Use when committing, branching, resolving… | 110,611 | 2026-10-13 |
| source-driven-development | addyosmani | Grounds every implementation decision in official documentation. Use when you want to verify an approach agai… | 110,611 | 2026-10-13 |
Skills from the same repository first, then others for this job by repository stars. Figures are from each repository when we last read it.
Common questions about webcrawler-deep-crawl
- What does webcrawler-deep-crawl do?
- Crawls any website starting from given URLs, extracting per-page text, markdown, or HTML along with metadata such as title, description, author, language, canonical URL, and Open Graph data, and identifies in-scope outbound links. Supports recursive crawling with depth or page limits, URL inclusion/exclusion, and cleaning of navigation elements.
- How do I install webcrawler-deep-crawl?
- Run npx skills add browser-act/skills. The skills installer reads the repository’s own structure, so it works for any repository of agent skills.
- Who publishes webcrawler-deep-crawl?
- browser-act.
- Is webcrawler-deep-crawl maintained, and what licence does it use?
- Its repository has 6,277 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-08-24. The repository declares the MIT licence.
More agent skills for writing & docs
- technical-blog-writingby 101-skills
- documentation-and-adrsby addyosmani
- git-workflow-and-versioningby addyosmani
- source-driven-developmentby addyosmani
- vitepressby antfu
- academy-guideby anthropic