Crawlsight
● Activeio.github.CloudCorpRecords/crawlsight
Check a site's AI visibility: crawler access, llms.txt generate and validate, robots.txt lint.
Connect this server
One click to install, or copy the config below. Suits: Claude Desktop · Cursor.
Or add it manually — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor):
{
"mcpServers": {
"crawlsight": {
"command": "npx",
"args": [
"-y",
"@reneturcios/crawlsight@0.1.0",
"mcp"
]
}
}
}claude mcp add crawlsight -- npx -y @reneturcios/crawlsight@0.1.0 mcpcodex mcp add crawlsight -- npx -y @reneturcios/crawlsight@0.1.0 mcpcode --add-mcp '{"name":"crawlsight","type":"stdio","command":"npx","args":["-y","@reneturcios/crawlsight@0.1.0","mcp"]}'Syntax from each client's docs (Claude Code: claude mcp add; Codex: codex mcp add; VS Code: code --add-mcp). Placeholders in angle brackets are values you fill in.
A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.
- Category
- Search & Web
- Version
- 0.1.0
- Registry
- active
- Published
- 11/17/2026
Is it safe? Is it maintained?
Serverdex check: Nov 19, 2026Looks healthy: Crawlsight is maintained, open source.
- Maintained
- Active
- Last commit 4 days ago (Nov 17, 2026), 0 open issues.
- Official registry
- Active
- Published as io.github.CloudCorpRecords/crawlsight in the official MCP registry, version 0.1.0.
- Source
- Open source, MIT
- Public repository under the MIT licence, 0 stars, 0 forks.
- Runs
- On your machine
- A local npm/Docker package started by your client; it can reach whatever your user account can.
- Credentials
- None declared
- The registry entry declares no required environment variables.
- Owner
- Unclaimed
- Nobody has claimed this page yet; the data above comes from the registry, GitHub and the live check.
Facts only: last commit and licence from GitHub, the entry's status from the official registry, credentials from the live check. Not a code audit — run the security scan on the endpoint yourself.
On GitHub
read Nov 18, 2026- Stars
- 0
- Forks
- 0
- Last push
- Nov 17, 2026
- License
- MIT
TypeScript
From the README
crawlsight How visible is a website to AI search? crawlsight is a small forwarding client for the hosted AI Visibility service: the tool list is served from your own machine using the captured definitions, and each call is sent upstream, with the service's answer returned exactly as it arrived. All four captured tools are exposed. You can run a full visibility check on a site, generate a fresh llms.txt for it from its real pages, validate the llms.txt it already ships, and lint its robots.txt for the kinds of mistakes that quietly block AI crawlers. Every input is a public URL or pasted public text, and every tool only reads and analyses public endpoints. The tools • check_site — Check a site's AI-search visibility: which AI crawlers (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and more) its robots.txt allows, whether the homepage ships readable text without JavaScript, and whether an llms.txt exists. Takes a required url. • generatellmstxt — Generate an llms.txt file from a site's real pages and sitemap. Takes a required url and an optional credit boolean (default true) that adds a one-line attribution back to the tool. • validatellmstxt — Validate an existing llms.txt against the spec. Takes a required url; the site's /llms.txt is fetched and validated. • lintrobotstxt — Lint a robots.txt for AI-crawler-specific mistakes. Pass url to fetch it live, or robots_txt to lint pasted text directly.…
Read the full README on GitHub →Alternatives to Crawlsight
All Crawlsight alternatives →Best Search & Web servers →The most-starred Search & Web servers in the index, with their live-check result.
| Server | Stars | Last push | Live check |
|---|---|---|---|
| Browser Use | 121,623 | Nov 2026 | Working |
| Scrapling MCP Server | 104,822 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,028 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,014 | Nov 2026 | Local / not checked |
| com.puter MCP Server | 44,182 | Nov 2026 | Needs sign-in |
Crawlsight: questions people ask
- Is Crawlsight a remote or a local MCP server?
- According to its entry in the official MCP registry, Crawlsight ships as a local package (npm, Docker) that your client starts on your machine.
- Is Crawlsight safe to use?
- Looks healthy: Crawlsight is maintained, open source. In detail — maintained: Active; official registry: Active; source: Open source, MIT; runs: On your machine; credentials: None declared. Serverdex check of Nov 19, 2026. This is a reading of the repository, the registry entry and the live check, not a code audit; run the Serverdex security scan on the endpoint before trusting it with real data.
- Is Crawlsight still maintained?
- Active: the repository's last commit 4 days ago. Last push Nov 17, 2026, 0 GitHub stars, MIT licence. Read from GitHub on Nov 18, 2026.
- How do I add Crawlsight to Claude, Cursor or VS Code?
- Copy the config in "Connect this server" into your client's MCP config file (claude_desktop_config.json, ~/.cursor/mcp.json or .vscode/mcp.json), or use the one-click Add to Cursor / Add to VS Code buttons and the one-line commands for Claude Code and Codex; then restart the client.
More from CloudCorpRecords
All 4 servers by CloudCorpRecords →- Muse MarketplaceOpen marketplace and registry for shareable Muse assets: avatars, personas, voices, and bundles.E-commerce & Retail
- OpensourceitAI-welcoming open source community: agents submit and discuss projects. No AI ban.Other
- ShelfSignalThe open store index for AI agents: merchants list real stock, agents find local food and stores.E-commerce & Retail
- ProdForgeConnects MCP servers, packages, security checks, and release gates in one repo.Developer Tools
More Search & Web servers
Browse all Search & Web →CloudGrid
cloudgrid-io
Plug apps, sites and HTML pages into CloudGrid and get a live URL, with databases and AI built in.
Cluefinch MCP
cluefinch
Local-first web research for AI agents with SearXNG search and SSRF-safe page retrieval.
c2n Web Components
code2nguyen
Exact APIs, examples, theme tokens and markup validation for the c2n (c2-*) Lit web components.
IPBulucu Network Tools
codeonthetable
IP, DNS, TCP port, partial DNSBL and IPv4 subnet tools using ipbulucu.org.
Riveter
riveterhq
MCP server for Riveter's enrichment, scraping, and monitoring API
Google Search Console MCP Server
conorbronsdon
Google Search Console for AI agents: search analytics, sitemaps, URL inspection.
Add the badge to your site
Show visitors that Crawlsight is listed here — it’s also the link we check for a free listing.
HTML — for your website
<a href="https://serverdex.io/server/io-github-cloudcorprecords-crawlsight?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="Crawlsight — listed on Serverdex" width="123" height="40" loading="lazy"></a>Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.
Markdown — for your README
[](https://serverdex.io/server/io-github-cloudcorprecords-crawlsight?utm_source=badge)Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.
Install buttons — for your README
[](https://serverdex.io/install/cursor/io-github-cloudcorprecords-crawlsight) [](https://serverdex.io/install/vscode/io-github-cloudcorprecords-crawlsight) [](https://serverdex.io/install/claude-code/io-github-cloudcorprecords-crawlsight)One click opens Cursor or VS Code with this server's config prefilled; the Claude Code button lands on the one-line command. The links point at Serverdex, so when your config changes the buttons keep working.
Install buttons — HTML for your docs
<a href="https://serverdex.io/install/cursor/io-github-cloudcorprecords-crawlsight"><img src="https://serverdex.io/brand/install-cursor.svg" alt="Add to Cursor — via Serverdex" width="140" height="28"></a>
<a href="https://serverdex.io/install/vscode/io-github-cloudcorprecords-crawlsight"><img src="https://serverdex.io/brand/install-vscode.svg" alt="Add to VS Code — via Serverdex" width="147" height="28"></a>
<a href="https://serverdex.io/install/claude-code/io-github-cloudcorprecords-crawlsight"><img src="https://serverdex.io/brand/install-claude-code.svg" alt="Open in Claude Code — via Serverdex" width="180" height="28"></a>Same buttons as plain links for a docs page or website.
Prefer a live badge with the score, stars and live check? Score badge (links to a page that explains the number and back to this listing).