io.github.diegoperezg7/evalsmith-mcp

io.github.diegoperezg7/evalsmith-mcp

Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

View repository →

Connect this server

Add this to your MCP client config — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor).Suits: Claude Desktop · Cursor.

{
  "mcpServers": {
    "evalsmith-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "evalsmith-mcp@0.1.0"
      ],
      "env": {
        "EVALSMITH_JUDGE_BASE_URL": "http://localhost:11434/v1",
        "EVALSMITH_JUDGE_MODEL": "llama3.1:8b",
        "EVALSMITH_JUDGE_API_KEY": "<your EVALSMITH_JUDGE_API_KEY>",
        "LOG_LEVEL": "info"
      }
    }
  }
}
Environment variables
  • EVALSMITH_JUDGE_BASE_URLOpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost)
  • EVALSMITH_JUDGE_MODELModel used by the optional LLM judge
  • EVALSMITH_JUDGE_API_KEYsecretAPI key for the judge endpoint, if it needs one
  • LOG_LEVELpino log level; logs go to stderr

A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.

Category
Developer Tools
Version
0.1.0
Registry
active
Published
10/28/2026

More Developer Tools servers

Browse all Developer Tools →

mcp-guard

diegoperezg7

Security scanner for MCP servers: tool poisoning, shadowing, permissions, secrets, transport.

Developer ToolsAdded 1 week ago

mcp-mexico

DiegoTepichin

Official Mexican public data for AI agents: Banxico rates, INEGI inflation, UMA and ISR tables.

Developer ToolsAdded 2 weeks ago

namegen

diegouqz

Region-typical random names for AI agents. Remote MCP + HTTP API, paid in USDC via x402.

Developer ToolsAdded 1 week ago

MCPAmpel - MCP Security Scanner

MCPAmpel

Scan installed MCP servers for security vulnerabilities with 16 detection engines.

Developer ToolsAdded 7 months ago

Wan27 MCP — AI Image and Video Generation

DieselNiu

Generate AI images and videos, estimate credits and track tasks. Paid Wan27 subscription required.

Developer ToolsAdded 1 month ago

DiffCI

DiffCI

Fail-closed, change-aware verification for AI coding agents with affected-test guidance.

Developer ToolsAdded 3 days ago

Add the badge to your site

Show visitors that io.github.diegoperezg7/evalsmith-mcp is listed here — it’s also the link we check for a free listing.

io.github.diegoperezg7/evalsmith-mcp — listed on Serverdex

HTML — for your website

<a href="https://serverdex.io/server/io-github-diegoperezg7-evalsmith-mcp?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="io.github.diegoperezg7/evalsmith-mcp — listed on Serverdex" width="123" height="40" loading="lazy"></a>

Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.

Markdown — for your README

[![io.github.diegoperezg7/evalsmith-mcp — listed on Serverdex](https://serverdex.io/brand/badge.svg)](https://serverdex.io/server/io-github-diegoperezg7-evalsmith-mcp?utm_source=badge)

Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.