Agent Web Census
● Activeorg.datacensus/agents
Open data on MCP servers (with live health) and which AI crawlers top websites block.
Connect this server
One click to install, or copy the config below. Suits: Cursor · HTTP clients.
Or add it manually — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor):
{
"mcpServers": {
"agents": {
"url": "https://datacensus.org/mcp"
}
}
}claude mcp add --transport http agents https://datacensus.org/mcpcode --add-mcp '{"name":"agents","type":"http","url":"https://datacensus.org/mcp"}'Syntax from each client's docs (Claude Code: claude mcp add; Codex: codex mcp add; VS Code: code --add-mcp). Placeholders in angle brackets are values you fill in. Codex takes hosted servers in config.toml ([mcp_servers.<name>] url = …) rather than on the command line.
A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.
- Category
- Search & Web
- Version
- 0.1.0
- Registry
- active
- Published
- 11/19/2026
Is it safe? Is it maintained?
Serverdex check: Nov 22, 2026Looks healthy: Agent Web Census is maintained, open source, hosted endpoint answering without a key.
- Maintained
- Active
- Last commit today (Nov 20, 2026), 0 open issues.
- Official registry
- Active
- Published as org.datacensus/agents in the official MCP registry, version 0.1.0.
- Source
- Open source, MIT
- Public repository under the MIT licence, 0 stars, 0 forks.
- Runs
- Hosted by the publisher
- Your client talks to datacensus.org: prompts and tool arguments leave your machine for that host.
- Credentials
- None needed to connect
- The hosted endpoint completed the Serverdex initialize + tools/list handshake with no key (checked Nov 22, 2026); individual tools may still ask for one.
- Owner
- Unclaimed
- Nobody has claimed this page yet; the data above comes from the registry, GitHub and the live check.
Facts only: last commit and licence from GitHub, the entry's status from the official registry, credentials from the live check. Not a code audit — run the security scan on the endpoint yourself and see the status history.
Live check
checked Nov 22, 2026, 07:08 UTCStatus history & uptime →The endpoint answered the MCP handshake as "datacensus" and listed 4 tools. Answering since Nov 21, 2026.
Context cost ≈ 694 tokens of tool definitions a client loads per conversation (estimate: characters ÷ 4) across 4 tools.
Tools (4)
find_mcp_servers5 paramsSearch the official MCP Registry (~65,000 servers) by keyword. Matches name, title and description; ranks servers whose remote endpoint passed a live initialize check first, then by number of tools. Returns name, title, description, remote/package types, health (ok = works without auth, auth = alive but needs auth, fail = unreachable), tool count and a registry link.
Show parametersHide parameters
querystringrequiredKeywords, e.g. 'postgres database' or 'github issues'.categorystringOptional category filter, e.g. 'finance & crypto', 'databases' (substring match).transportstringremote = hosted endpoint, package = installable (npm/pypi/...), any (default).working_onlybooleanOnly servers whose remote endpoint responded OK or requires auth (default true). Package-only servers are kept unless transport is 'remote'.limitintegerMax results (1-25).
mcp_ecosystem_statsHeadline numbers of the MCP ecosystem: servers in the official registry, new servers in the last 7/30 days, remote vs package share, top package registries and categories, plus the weekly health check of remote servers (share working, requiring auth, failing, median tools and latency).
site_ai_crawler_policy1 paramWhich AI crawlers does a website block in robots.txt? Looks up a domain in our scan of the top 10,000 websites (Tranco list) and returns, per AI bot (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, ...), whether it is blocked, partly restricted or explicitly allowed, plus llms.txt presence.
Show parametersHide parameters
domainstringrequiredDomain or URL, e.g. 'nytimes.com' or 'https://www.nytimes.com/section'.
ai_crawler_stats1 paramAggregate AI-crawler blocking statistics across the top 10,000 websites: percentage of sites with a reachable robots.txt that fully block, partly restrict or explicitly allow each AI bot. Give a bot name for one bot, or omit for the overall table.
Show parametersHide parameters
botstringOptional bot name, e.g. 'GPTBot', 'ClaudeBot', 'Google-Extended' (case-insensitive).
Checked by Serverdex with a plain initialize + tools/list call to datacensus.org/mcp, no credentials, about once a day. A failed check is never a removal.
On GitHub
read Nov 21, 2026- Stars
- 0
- Forks
- 0
- Last push
- Nov 20, 2026
- License
- MIT
Python
From the README
Agent Web Census Live, daily-updated, measured statistics on the AI agent ecosystem — no estimates, no made-up numbers. • MCP servers: full daily crawl of the official MCP Registry → totals, growth, deployment (remote vs. package), registries, categories, publishers. • AI crawlers & the web: weekly scan of robots.txt and llms.txt on the top 10,000 websites (Tranco) → who blocks GPTBot, ClaudeBot, Google-Extended, CCBot… Site: GitHub Pages (built by .github/workflows/update.yml). Data: data//latest.json, data//history.csv — CC BY 4.0, cite as “Agent Web Census”. Transparency This project is built and operated by an AI agent (Claude, by Anthropic) with a human sponsor. The scanner identifies itself as AgentWebCensusBot and fetches only robots.txt and llms.txt (max 3 requests per domain per week). Run locally
Read the full README on GitHub →Remotes
- streamable-http
https://datacensus.org/mcp
Alternatives to Agent Web Census
All Agent Web Census alternatives →Best Search & Web servers →The most-starred Search & Web servers in the index, with their live-check result.
| Server | Stars | Last push | Live check |
|---|---|---|---|
| Browser Use | 121,623 | Nov 2026 | Working |
| Scrapling MCP Server | 104,822 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,028 | Nov 2026 | Local / not checked |
| Chrome DevTools MCP | 58,014 | Nov 2026 | Local / not checked |
| com.puter MCP Server | 44,182 | Nov 2026 | Needs sign-in |
Agent Web Census: questions people ask
- Is Agent Web Census a remote or a local MCP server?
- According to its entry in the official MCP registry, Agent Web Census ships as a hosted endpoint (streamable-http) that your client connects to by URL.
- Does Agent Web Census need an API key?
- Its hosted endpoint completed the Serverdex live check — an MCP initialize + tools/list call — with no credentials on Nov 22, 2026. Individual tools may still ask for a key when called.
- Is Agent Web Census safe to use?
- Looks healthy: Agent Web Census is maintained, open source, hosted endpoint answering without a key. In detail — maintained: Active; official registry: Active; source: Open source, MIT; runs: Hosted by the publisher; credentials: None needed to connect. Serverdex check of Nov 22, 2026. This is a reading of the repository, the registry entry and the live check, not a code audit; run the Serverdex security scan on the endpoint before trusting it with real data.
- Is Agent Web Census still maintained?
- Active: the repository's last commit today. Last push Nov 20, 2026, 0 GitHub stars, MIT licence. Read from GitHub on Nov 21, 2026.
- What tools does Agent Web Census expose?
- 4 tools as listed by the server itself during the live check, including find_mcp_servers, mcp_ecosystem_stats, site_ai_crawler_policy, ai_crawler_stats. The full list with parameters is in the Live check section above.
- How do I add Agent Web Census to Claude, Cursor or VS Code?
- Copy the config in "Connect this server" into your client's MCP config file (claude_desktop_config.json, ~/.cursor/mcp.json or .vscode/mcp.json), or use the one-click Add to Cursor / Add to VS Code buttons and the one-line commands for Claude Code and Codex; then restart the client.
More Search & Web servers
Browse all Search & Web →OpenSERP
openserpapi
Search Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia with original ranks, plus page extraction.
Domain Liveness
modelcontextprotocol
Is this business's website still there? Checks apex and www, and DNS separately from HTTP.
Portproof
portproofas
Mobile 4G/5G and residential proxies for AI agents: build, test and pay per GB on shared pools.
Ghostlight
sylin-org
Visible local browser automation in signed-in Chromium, with optional policy and audit.
ship.page
bitgate
Deploy HTML from any agent: POST markup, get a live URL. Static hosting API with MCP tools.
twocents
twocents-page
Human feedback for AI agents: share HTML, get a live review link, read anchored notes as markdown.
Add the badge to your site
Show visitors that Agent Web Census is listed here — it’s also the link we check for a free listing.
HTML — for your website
<a href="https://serverdex.io/server/org-datacensus-agents?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="Agent Web Census — listed on Serverdex" width="123" height="40" loading="lazy"></a>Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.
Markdown — for your README
[](https://serverdex.io/server/org-datacensus-agents?utm_source=badge)Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.
Install buttons — for your README
[](https://serverdex.io/install/cursor/org-datacensus-agents) [](https://serverdex.io/install/vscode/org-datacensus-agents) [](https://serverdex.io/install/claude-code/org-datacensus-agents)One click opens Cursor or VS Code with this server's config prefilled; the Claude Code button lands on the one-line command. The links point at Serverdex, so when your config changes the buttons keep working.
Install buttons — HTML for your docs
<a href="https://serverdex.io/install/cursor/org-datacensus-agents"><img src="https://serverdex.io/brand/install-cursor.svg" alt="Add to Cursor — via Serverdex" width="140" height="28"></a>
<a href="https://serverdex.io/install/vscode/org-datacensus-agents"><img src="https://serverdex.io/brand/install-vscode.svg" alt="Add to VS Code — via Serverdex" width="147" height="28"></a>
<a href="https://serverdex.io/install/claude-code/org-datacensus-agents"><img src="https://serverdex.io/brand/install-claude-code.svg" alt="Open in Claude Code — via Serverdex" width="180" height="28"></a>Same buttons as plain links for a docs page or website.
Prefer a live badge with the score, stars and live check? Score badge (links to a page that explains the number and back to this listing).