TheCrawler
TheCrawler is an MCP server, listed under browser & scraping. It reaches playwright and thecrawler. It needs THECRAWLER_API_KEY to authenticate. Its repository has 2 GitHub stars and is licensed under AGPL-3.0; we last read it on 2026-10-22.
TheCrawler is an open-source web scraper that uses LLMs for structured data extraction. It supports PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, and analytics tracker detection. The tool provides structured error handling and adaptive switching between Cheerio and Playwright. Available via CLI, npm, REST API, and MCP server.
Install
MCP servers are packaged differently from each other -- some run via npx, some as a Docker image, some as a plain script you clone and run. TheCrawler’s own README did not state one plainly enough for this page to quote it rather than guess, so it will not guess. The repository below states it.
What it reaches, and what it needs
Maintenance
Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.
TheCrawler compared with similar servers
| Server | Reaches | Needs | Stars | Last update | Licence |
|---|---|---|---|---|---|
| TheCrawler (this page) | playwright, thecrawler | THECRAWLER_API_KEY | 2 | 2026-07-02 | AGPL-3.0 |
| fetcher-mcp | playwright(1 shared) | None listed | 1,088 | 2026-01-14 | MIT |
| MCP-Server-Playwright | playwright(1 shared) | None listed | 297 | 2025-06-05 | MIT |
| cloakbrowser-mcp | playwright(1 shared) | None listed | 172 | 2026-10-24 | MIT |
| RedBook-Search-Comment-MCP | playwright(1 shared) | None listed | 44 | 2025-04-23 | — |
| aginxbrowser | playwright, puppeteer(1 shared) | None listed | 44 | 2026-10-25 | Apache-2.0 |
Ranked by services in common with TheCrawler, then by stars. Figures are from each repository when we last read it.
Common questions about TheCrawler
- What does TheCrawler do?
- TheCrawler is an open-source web scraper that uses LLMs for structured data extraction. It supports PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, and analytics tracker detection. The tool provides structured error handling and adaptive switching between Cheerio and Playwright. Available via CLI, npm, REST API, and MCP server.
- How do I install TheCrawler?
- Its README did not state an install command plainly enough for us to quote one, so we do not guess. The repository has the instructions: https://github.com/manchittlab/TheCrawler
- What does TheCrawler connect to, and what does it need?
- It reaches playwright and thecrawler. The credentials we found in its source are THECRAWLER_API_KEY.
- Is TheCrawler official and is it maintained?
- We found nothing naming it as a vendor’s official server. Its repository has 2 GitHub stars and its state in our directory is listed. Its last commit we recorded was on 2026-07-02.
- What licence does TheCrawler use?
- The repository declares the AGPL-3.0 licence.
More MCP servers for browser & scraping
- mcp-chrome12,501 stars
- webclaw2,386 stars
- fetcher-mcp1,088 stars
- puppeteer-mcp-server486 stars
- moling339 stars
- MCP-Server-Playwright297 stars