AI Crawlability Check
● Activeio.github.Milan344s/ai-crawlability-check
Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.
Connect this server
One click to install, or copy the config below. Suits: Cursor · HTTP clients.
Or add it manually — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor):
{
"mcpServers": {
"ai-crawlability-check": {
"url": "https://mcp.milan-novotny.com/mcp"
}
}
}claude mcp add --transport http ai-crawlability-check https://mcp.milan-novotny.com/mcpcode --add-mcp '{"name":"ai-crawlability-check","type":"http","url":"https://mcp.milan-novotny.com/mcp"}'Syntax from each client's docs (Claude Code: claude mcp add; Codex: codex mcp add; VS Code: code --add-mcp). Placeholders in angle brackets are values you fill in. Codex takes hosted servers in config.toml ([mcp_servers.<name>] url = …) rather than on the command line.
A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.
- Category
- Search & Web
- Version
- 2.0.0
- Registry
- active
- Published
- 11/19/2026
Is it safe? Is it maintained?
Serverdex check: Nov 22, 2026Looks healthy: AI Crawlability Check is maintained, open source, hosted endpoint answering without a key.
- Maintained
- Active
- Last commit 3 days ago (Nov 19, 2026), 0 open issues.
- Official registry
- Active
- Published as io.github.Milan344s/ai-crawlability-check in the official MCP registry, version 2.0.0.
- Source
- Open source, MIT
- Public repository under the MIT licence, 1 stars, 0 forks.
- Runs
- Hosted by the publisher
- Your client talks to mcp.milan-novotny.com: prompts and tool arguments leave your machine for that host.
- Credentials
- None needed to connect
- The hosted endpoint completed the Serverdex initialize + tools/list handshake with no key (checked Nov 22, 2026); individual tools may still ask for one.
- Owner
- Unclaimed
- Nobody has claimed this page yet; the data above comes from the registry, GitHub and the live check.
Facts only: last commit and licence from GitHub, the entry's status from the official registry, credentials from the live check. Not a code audit — run the security scan on the endpoint yourself and see the status history.
Live check
checked Nov 22, 2026, 07:08 UTCStatus history & uptime →The endpoint answered the MCP handshake as "AI Crawlability Check" and listed 5 tools. Answering since Nov 21, 2026.
Context cost ≈ 1.1k tokens of tool definitions a client loads per conversation (estimate: characters ÷ 4) across 5 tools.
Tools (5)
- check_ai_crawler_access1 param
Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time bots versus training bots. Also checks whether the server answers an automated request, noindex and nosnippet directives, declared sitemaps, Content-Signal lines and llms.txt. Use for "are we blocking GPTBot or ClaudeBot?", "can ChatGPT read this page?" or an AI crawler audit of robots.txt.
Show parametersHide parameters
urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
check_ai_indexability1 paramChecks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverage, response time, HTML size, structured data validity, dates and language. Use for "why is this page not showing up in ChatGPT or Perplexity?", "is this page indexable?" or to check a JavaScript-rendered page. It does not query any index.
Show parametersHide parameters
urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
check_ai_citability1 paramFetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, freshness and llms.txt, plus a short crawler-access summary. Use for "will AI cite this page?", "is this article LLM-friendly?" or a GEO check. Tuned for articles and guides.
Show parametersHide parameters
urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
check_seo_basics2 paramsFetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to enable the keyword checks. Does not cover rankings or backlinks.
Show parametersHide parameters
urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.keywordstringTarget keyword or phrase. Optional; without it the keyword checks are skipped.
get_review_rubric1 paramReturns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.
Show parametersHide parameters
typestringrequiredWhich rubric to return.
Checked by Serverdex with a plain initialize + tools/list call to mcp.milan-novotny.com/mcp, no credentials, about once a day. A failed check is never a removal.
On GitHub
read Nov 21, 2026- Stars
- 1
- Forks
- 0
- Last push
- Nov 19, 2026
- License
- MIT
JavaScriptai-crawlerscloudflare-workersgeollms-txtmcpmcp-servermodel-context-protocolremote-mcp-serverrobots-txtseo
From the README
AI Crawlability Check AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable. It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing. • Endpoint: https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication) • Documentation: https://www.milan-novotny.com/tools/ai-crawlability-mcp/ • Privacy and terms: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/ • MCP Registry name: io.github.Milan344s/ai-crawlability-check Connect Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL. Claude Code Cursor and other clients with a JSON config Then ask in plain language: • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?" • "Check whether this page can be indexed by AI search engines, and list the fixes." • "Run all the crawlability checks on our five most important URLs and give me one table." Tools All five tools are read-only. Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure. Example result checkaicrawler_access on a page whose robots.txt blocks most AI crawlers (shortened):…
Read the full README on GitHub →Remotes
- streamable-http
https://mcp.milan-novotny.com/mcp
Alternatives to AI Crawlability Check
All AI Crawlability Check alternatives →Best Search & Web servers →The most-starred Search & Web servers in the index, with their live-check result.
| Server | Stars | Last push | Live check |
|---|---|---|---|
| Browser Use | 121,623 | Nov 2026 | Working |
| Scrapling MCP Server | 104,822 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,028 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,014 | Nov 2026 | Local / not checked |
| com.puter MCP Server | 44,182 | Nov 2026 | Needs sign-in |
AI Crawlability Check: questions people ask
- Is AI Crawlability Check a remote or a local MCP server?
- According to its entry in the official MCP registry, AI Crawlability Check ships as a hosted endpoint (streamable-http) that your client connects to by URL.
- Does AI Crawlability Check need an API key?
- Its hosted endpoint completed the Serverdex live check — an MCP initialize + tools/list call — with no credentials on Nov 22, 2026. Individual tools may still ask for a key when called.
- Is AI Crawlability Check safe to use?
- Looks healthy: AI Crawlability Check is maintained, open source, hosted endpoint answering without a key. In detail — maintained: Active; official registry: Active; source: Open source, MIT; runs: Hosted by the publisher; credentials: None needed to connect. Serverdex check of Nov 22, 2026. This is a reading of the repository, the registry entry and the live check, not a code audit; run the Serverdex security scan on the endpoint before trusting it with real data.
- Is AI Crawlability Check still maintained?
- Active: the repository's last commit 3 days ago. Last push Nov 19, 2026, 1 GitHub stars, MIT licence. Read from GitHub on Nov 21, 2026.
- What tools does AI Crawlability Check expose?
- 5 tools as listed by the server itself during the live check, including check_ai_crawler_access, check_ai_indexability, check_ai_citability, check_seo_basics, get_review_rubric. The full list with parameters is in the Live check section above.
- How do I add AI Crawlability Check to Claude, Cursor or VS Code?
- Copy the config in "Connect this server" into your client's MCP config file (claude_desktop_config.json, ~/.cursor/mcp.json or .vscode/mcp.json), or use the one-click Add to Cursor / Add to VS Code buttons and the one-line commands for Claude Code and Codex; then restart the client.
More Search & Web servers
Browse all Search & Web →acrawl
Mingye-Lu
Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.
Unrendered: AI Crawler Checker
missiongrowth
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
Agent Ready
mlava
Scan any URL for AI agent readability — Vercel Spec, llmstxt.org, and agent-protocol manifests.
PyScrappy
mldsveda
Web-scraping toolkit with 22 tools for structured web data as JSON for AI agents.
mobileproxy MCP Server
mobileproxy
Control dedicated 4G mobile and residential proxies on mobileproxy.space from AI agents.
Web & Company Intelligence
moha-tah
Tech stack, company profile, sitemap, URL status and EU fuel price tools, pay per result.
Add the badge to your site
Show visitors that AI Crawlability Check is listed here — it’s also the link we check for a free listing.
HTML — for your website
<a href="https://serverdex.io/server/io-github-milan344s-ai-crawlability-check?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="AI Crawlability Check — listed on Serverdex" width="123" height="40" loading="lazy"></a>Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.
Markdown — for your README
[](https://serverdex.io/server/io-github-milan344s-ai-crawlability-check?utm_source=badge)Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.
Install buttons — for your README
[](https://serverdex.io/install/cursor/io-github-milan344s-ai-crawlability-check) [](https://serverdex.io/install/vscode/io-github-milan344s-ai-crawlability-check) [](https://serverdex.io/install/claude-code/io-github-milan344s-ai-crawlability-check)One click opens Cursor or VS Code with this server's config prefilled; the Claude Code button lands on the one-line command. The links point at Serverdex, so when your config changes the buttons keep working.
Install buttons — HTML for your docs
<a href="https://serverdex.io/install/cursor/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-cursor.svg" alt="Add to Cursor — via Serverdex" width="140" height="28"></a>
<a href="https://serverdex.io/install/vscode/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-vscode.svg" alt="Add to VS Code — via Serverdex" width="147" height="28"></a>
<a href="https://serverdex.io/install/claude-code/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-claude-code.svg" alt="Open in Claude Code — via Serverdex" width="180" height="28"></a>Same buttons as plain links for a docs page or website.
Prefer a live badge with the score, stars and live check? Score badge (links to a page that explains the number and back to this listing).