TheCrawler

io.github.manchittlab/thecrawler

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Connect this server

Add this to your MCP client config — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor).Suits: Claude Desktop · Cursor.

{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler@0.5.8"
      ]
    }
  }
}

A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.

Category: Search & Web
Version: 0.5.8
Registry: active
Published: 7/2/2026

Browse all Search & Web →

DeepSeek Web Search

Matthewyin

MCP server for DeepSeek web Search & Fetch: scored web search + page-to-Markdown fetch

Search & WebAdded 2 weeks ago

io.github.MetaLift-AI/metalift

MetaLift-AI

Metalift MCP for AI agents: web search (2 credits), scrape, crawl, and map tools

Search & WebAdded 4 months ago

Browserbase

mindstone

Browserbase MCP server: cloud browser sessions, agents, contexts, fetch, search, functions

Search & WebAdded 2 months ago

acrawl

Mingye-Lu

Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.

Search & WebAdded 1 week ago

io.github.mrfentmen/books-mcp

mrfentmen

Search books and fetch volume details from the Google Books API. No key required.

Search & WebAdded 1 month ago

io.github.mrfentmen/ucsc-genome-mcp

mrfentmen

UCSC Genome Browser: list genomes, fetch DNA ranges and search genes by name

Search & WebAdded 1 week ago

Add the badge to your site

Show visitors that TheCrawler is listed here — it’s also the link we check for a free listing.

HTML — for your website

<a href="https://serverdex.io/server/io-github-manchittlab-thecrawler?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="TheCrawler — listed on Serverdex" width="123" height="40" loading="lazy"></a>

Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.

Markdown — for your README

[![TheCrawler — listed on Serverdex](https://serverdex.io/brand/badge.svg)](https://serverdex.io/server/io-github-manchittlab-thecrawler?utm_source=badge)

Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.