AI Crawlability Check

● Active

io.github.Milan344s/ai-crawlability-check

Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.

Connect this server

One click to install, or copy the config below. Suits: Cursor · HTTP clients.

Add to Cursor Add to VS CodeClaude Desktop / Claude Code: paste the config below.

Or add it manually — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor):

{
  "mcpServers": {
    "ai-crawlability-check": {
      "url": "https://mcp.milan-novotny.com/mcp"
    }
  }
}
One line, from the terminal
Claude Code
claude mcp add --transport http ai-crawlability-check https://mcp.milan-novotny.com/mcp
VS Code
code --add-mcp '{"name":"ai-crawlability-check","type":"http","url":"https://mcp.milan-novotny.com/mcp"}'

Syntax from each client's docs (Claude Code: claude mcp add; Codex: codex mcp add; VS Code: code --add-mcp). Placeholders in angle brackets are values you fill in. Codex takes hosted servers in config.toml ([mcp_servers.<name>] url = …) rather than on the command line.

A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.

Category
Search & Web
Version
2.0.0
Registry
active
Published
11/19/2026
Listed in:Official MCP registry ↗how this is checked

Is it safe? Is it maintained?

Serverdex check: Nov 22, 2026

Looks healthy: AI Crawlability Check is maintained, open source, hosted endpoint answering without a key.

Maintained
Active
Last commit 3 days ago (Nov 19, 2026), 0 open issues.
Official registry
Active
Published as io.github.Milan344s/ai-crawlability-check in the official MCP registry, version 2.0.0.
Source
Open source, MIT
Public repository under the MIT licence, 1 stars, 0 forks.
Runs
Hosted by the publisher
Your client talks to mcp.milan-novotny.com: prompts and tool arguments leave your machine for that host.
Credentials
None needed to connect
The hosted endpoint completed the Serverdex initialize + tools/list handshake with no key (checked Nov 22, 2026); individual tools may still ask for one.
Owner
Unclaimed
Nobody has claimed this page yet; the data above comes from the registry, GitHub and the live check.

Facts only: last commit and licence from GitHub, the entry's status from the official registry, credentials from the live check. Not a code audit — run the security scan on the endpoint yourself and see the status history.

Live check

checked Nov 22, 2026, 07:08 UTCStatus history & uptime →
● Working69 msprotocol 2025-06-18v2.0.0tools

The endpoint answered the MCP handshake as "AI Crawlability Check" and listed 5 tools. Answering since Nov 21, 2026.

Context cost ≈ 1.1k tokens of tool definitions a client loads per conversation (estimate: characters ÷ 4) across 5 tools.

Tools (5)

  • Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time bots versus training bots. Also checks whether the server answers an automated request, noindex and nosnippet directives, declared sitemaps, Content-Signal lines and llms.txt. Use for "are we blocking GPTBot or ClaudeBot?", "can ChatGPT read this page?" or an AI crawler audit of robots.txt.

    Show parameters
    • urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
  • check_ai_indexability1 param

    Checks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverage, response time, HTML size, structured data validity, dates and language. Use for "why is this page not showing up in ChatGPT or Perplexity?", "is this page indexable?" or to check a JavaScript-rendered page. It does not query any index.

    Show parameters
    • urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
  • check_ai_citability1 param

    Fetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, freshness and llms.txt, plus a short crawler-access summary. Use for "will AI cite this page?", "is this article LLM-friendly?" or a GEO check. Tuned for articles and guides.

    Show parameters
    • urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
  • check_seo_basics2 params

    Fetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to enable the keyword checks. Does not cover rankings or backlinks.

    Show parameters
    • urlstringrequiredPublic http(s) URL of the page to check, for example https://example.com/pricing.
    • keywordstringTarget keyword or phrase. Optional; without it the keyword checks are skipped.
  • get_review_rubric1 param

    Returns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.

    Show parameters
    • typestringrequiredWhich rubric to return.

Checked by Serverdex with a plain initialize + tools/list call to mcp.milan-novotny.com/mcp, no credentials, about once a day. A failed check is never a removal.

On GitHub

read Nov 21, 2026
Stars
1
Forks
0
Last push
Nov 19, 2026
License
MIT

JavaScriptai-crawlerscloudflare-workersgeollms-txtmcpmcp-servermodel-context-protocolremote-mcp-serverrobots-txtseo

From the README

AI Crawlability Check AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable. It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing. • Endpoint: https://mcp.milan-novotny.com/mcp (Streamable HTTP, no authentication) • Documentation: https://www.milan-novotny.com/tools/ai-crawlability-mcp/ • Privacy and terms: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/ • MCP Registry name: io.github.Milan344s/ai-crawlability-check Connect Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL. Claude Code Cursor and other clients with a JSON config Then ask in plain language: • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?" • "Check whether this page can be indexed by AI search engines, and list the fixes." • "Run all the crawlability checks on our five most important URLs and give me one table." Tools All five tools are read-only. Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure. Example result checkaicrawler_access on a page whose robots.txt blocks most AI crawlers (shortened):…

Read the full README on GitHub →

Remotes

  • streamable-httphttps://mcp.milan-novotny.com/mcp

The most-starred Search & Web servers in the index, with their live-check result.

ServerStarsLast pushLive check
Browser Use121,623Nov 2026Working
Scrapling MCP Server104,822Nov 2026Local / not checked
Chrome DevTools MCP58,028Nov 2026Local / not checked
Chrome DevTools MCP58,014Nov 2026Local / not checked
com.puter MCP Server44,182Nov 2026Needs sign-in

AI Crawlability Check: questions people ask

Is AI Crawlability Check a remote or a local MCP server?
According to its entry in the official MCP registry, AI Crawlability Check ships as a hosted endpoint (streamable-http) that your client connects to by URL.
Does AI Crawlability Check need an API key?
Its hosted endpoint completed the Serverdex live check — an MCP initialize + tools/list call — with no credentials on Nov 22, 2026. Individual tools may still ask for a key when called.
Is AI Crawlability Check safe to use?
Looks healthy: AI Crawlability Check is maintained, open source, hosted endpoint answering without a key. In detail — maintained: Active; official registry: Active; source: Open source, MIT; runs: Hosted by the publisher; credentials: None needed to connect. Serverdex check of Nov 22, 2026. This is a reading of the repository, the registry entry and the live check, not a code audit; run the Serverdex security scan on the endpoint before trusting it with real data.
Is AI Crawlability Check still maintained?
Active: the repository's last commit 3 days ago. Last push Nov 19, 2026, 1 GitHub stars, MIT licence. Read from GitHub on Nov 21, 2026.
What tools does AI Crawlability Check expose?
5 tools as listed by the server itself during the live check, including check_ai_crawler_access, check_ai_indexability, check_ai_citability, check_seo_basics, get_review_rubric. The full list with parameters is in the Live check section above.
How do I add AI Crawlability Check to Claude, Cursor or VS Code?
Copy the config in "Connect this server" into your client's MCP config file (claude_desktop_config.json, ~/.cursor/mcp.json or .vscode/mcp.json), or use the one-click Add to Cursor / Add to VS Code buttons and the one-line commands for Claude Code and Codex; then restart the client.
Browse all Search & Web →

Add the badge to your site

Show visitors that AI Crawlability Check is listed here — it’s also the link we check for a free listing.

AI Crawlability Check — listed on Serverdex

HTML — for your website

<a href="https://serverdex.io/server/io-github-milan344s-ai-crawlability-check?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="AI Crawlability Check — listed on Serverdex" width="123" height="40" loading="lazy"></a>

Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.

Markdown — for your README

[![AI Crawlability Check — listed on Serverdex](https://serverdex.io/brand/badge.svg)](https://serverdex.io/server/io-github-milan344s-ai-crawlability-check?utm_source=badge)

Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.

Add to CursorAdd to VS CodeOpen in Claude Code

Install buttons — for your README

[![Add to Cursor](https://serverdex.io/brand/install-cursor.svg)](https://serverdex.io/install/cursor/io-github-milan344s-ai-crawlability-check) [![Add to VS Code](https://serverdex.io/brand/install-vscode.svg)](https://serverdex.io/install/vscode/io-github-milan344s-ai-crawlability-check) [![Open in Claude Code](https://serverdex.io/brand/install-claude-code.svg)](https://serverdex.io/install/claude-code/io-github-milan344s-ai-crawlability-check)

One click opens Cursor or VS Code with this server's config prefilled; the Claude Code button lands on the one-line command. The links point at Serverdex, so when your config changes the buttons keep working.

Install buttons — HTML for your docs

<a href="https://serverdex.io/install/cursor/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-cursor.svg" alt="Add to Cursor — via Serverdex" width="140" height="28"></a>
<a href="https://serverdex.io/install/vscode/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-vscode.svg" alt="Add to VS Code — via Serverdex" width="147" height="28"></a>
<a href="https://serverdex.io/install/claude-code/io-github-milan344s-ai-crawlability-check"><img src="https://serverdex.io/brand/install-claude-code.svg" alt="Open in Claude Code — via Serverdex" width="180" height="28"></a>

Same buttons as plain links for a docs page or website.

Prefer a live badge with the score, stars and live check? Score badge (links to a page that explains the number and back to this listing).