MCP Server Test Harness: Conformance + Test Cases for CI

Test any MCP server before you ship it: stdio or Streamable HTTP, against MCP 2026-07-28 (stateless, server/discover) and the 2025-11-25 initialize handshake. One Python file, standard library only, Python 3.9+. Works with servers written in any language.

mcptest check runs these without any test cases:

./tools/demo.sh                                              # see it on a good and a deliberately broken server
python3 tools/mcptest.py check -- python3 path/to/server.py  # your stdio server (era auto-detected)
python3 tools/mcptest.py check --era both -- node dist/index.js
python3 tools/mcptest.py check --url http://127.0.0.1:8000/mcp

Exit codes: 0 all passed, 1 a check or case failed, 2 the server could not be started/reached or the suite is invalid. --strict also fails on warnings.

python3 tools/mcptest.py init -- python3 server.py > mcp.tests.yaml   # one smoke case per tool, from tools/list
python3 tools/mcptest.py run mcp.tests.yaml --junit report.xml

server:
  command: [python3, server.py]     # or  url: http://127.0.0.1:8000/mcp  (+ headers: {Authorization: "Bearer ..."})
  env: {API_BASE: "http://localhost:9000"}
  cwd: .                            # optional, relative to this file
era: both                           # auto | modern | legacy | both
cases:
  - name: search returns results
    tool: search
    args: {query: "mcp", limit: 3}
    expect:
      json: {count: 3}              # subset match on structuredContent (or JSON text)
      text_contains: [mcp]
      regex: '"title": ".+"'
      max_ms: 1500
  - name: empty query is a tool error
    tool: search
    args: {query: ""}
    expect: {is_error: true}
  - name: wrong argument is refused
    tool: search
    args: {q: "x"}
    expect: {protocol_error: -32602}

Before each call, mcptest validates the case's args against the tool's inputSchema (so a broken test case is reported as such), and if the tool declares an outputSchema it validates structuredContent against it. JSON Schema support is a practical subset: type, properties, required, additionalProperties, items, enum, const, min/max, lengths, pattern, item counts, anyOf/oneOf/allOf (no $ref).

.github/workflows/mcp-tests.yml runs the suite on every push and PR and uploads the JUnit report. --json out.json writes a machine-readable result too.

This folder is a Claude Code plugin with an mcp-test skill: claude --plugin-dir ./mcp-server-test-harness, then ask "test my MCP server".

• tools/mcptest.py the harness (+ miniyaml.py, a small YAML reader)
• examples/good_server.py a well-behaved demo server; examples/buggy_server.py has 8 common bugs, all caught; examples/http_server.py serves the demo over HTTP (--sse for event-stream replies)
• examples/demo.tests.yaml a 5-case suite run on both eras
• tests/ 19 tests: schema validator, both eras over stdio, HTTP JSON + SSE, header checks, every bug in the buggy server, suites, JUnit, init

• It tests the protocol surface and your cases; it is not a full schema-level certification of the MCP spec, and it is not affiliated with or endorsed by the MCP project or Anthropic.
• No OAuth flow: pass a ready token with headers: in the suite.
• Resources, prompts, subscriptions/listen, tasks and multi round-trip (input_required) results are not exercised.
• JSON Schema checks are a subset (no $ref, format, or conditional keywords).
• The unknown-tool and parse-error checks follow the spec's recommended codes; some SDKs differ, which shows as a WARN or FAIL you can judge yourself.