MCP tool reference · #453 of 563 by publishers
check_ai_crawler_access
Exposed by 8 servers from 8 publishers, read from the live servers by the Serverdex check; most of them sit in Search & Web; 87.5% of them take a url parameter; the most-starred is AI Crawlability Check (1 stars).
check_ai_crawler_access(url: string, …)Parameters
share = % of the 8 servers whose schema has it · required = % of those that mark it required| Parameter | Type | Share | Required | Usual description |
|---|---|---|---|---|
| url | string | 87.5% | 100% | Any URL on the site to check, e.g. https://example.com |
| website | string | 12.5% | 100% | The public website to check, like example.com or https://www.example.com. One site per call. |
Source: the input schema each server returns for check_ai_crawler_access, aggregated over the 8 index-tier servers; type and description are the values most servers use. Rebuilt every 6 hours.
Servers with check_ai_crawler_access
8 of 8 · 8 publishers · 8 answering live · most-starred first, 3 per publisher| # | Server | Publisher | Category | ★ | Live | How it describes check_ai_crawler_access |
|---|---|---|---|---|---|---|
| 1 | AI Crawlability Checkmilan344s · Search & Web | milan344s | Search & Web | 1 | yes | Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time bots versus training bots. Also checks whether the server answers an automated request, noindex and nosnippet directive |
| 2 | Tsoden AI Visibility Auditnotformal · Search & Web | notformal | Search & Web | — | yes | Check which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others) a site's robots.txt allows or blocks, and whether its /llms.txt exists and is valid. Returns a shareable result page on https://serp.tsoden.ai. |
| 3 | llms.txt 檢查工具panda2143 · AI & Agents | panda2143 | AI & Agents | — | yes | 讀取網站 robots.txt,回報 GPTBot、ChatGPT-User、ClaudeBot、Claude-User、PerplexityBot 等 13 種 AI 爬蟲是被允許還是封鎖。 |
| 4 | Preferlypreferly-ai · Search & Web | preferly-ai | Search & Web | — | yes | Use this when the user wants to know whether AI crawlers and AI tools can access a specific website. It reads the site's robots.txt and reports, for 10+ AI crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended), whether each is allowed or blocked and which rule decides it. It also reports whether the homepage loads for an automated visitor or shows a |
| 5 | GetBotRank Free AI Search Toolslazycipher · Search & Web | lazycipher | Search & Web | — | yes | Request a page as GPTBot, ClaudeBot, PerplexityBot and other AI crawlers and compare with a normal browser to find robots.txt or CDN blocks. |
| 6 | whereamimentionedchatzisathanasios · Other | chatzisathanasios | Other | — | yes | Check whether a site's robots.txt allows or blocks each known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about thirty others), with the exact rule and line that decided it. |
| 7 | Rankwise0xrome · Education & Reference | 0xrome | Education & Reference | — | yes | Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) are allowed or blocked by a website's robots.txt. Returns each crawler's status. |
| 8 | Stormapboriskendev · Developer Tools | boriskendev | Developer Tools | — | yes | Check whether AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot) may access a public URL according to its robots.txt, and score the page's machine readability (status, title, meta description, canonical, JSON-LD, server-rendered text). Returns a 0-100 score and a list of findings. |
Stars from GitHub as stored in the Serverdex index (re-read every 7 days per server); live = answered the last daily check of its hosted endpoint.