mcptest docs GitHub

Troubleshooting

Entries here are organized by symptom. If mcptest just printed an error, copy the exact message into your browser's find-in-page (Cmd-F or Ctrl-F) and the match should land on the right entry. CLI error messages also print a link in the form https://mcptest.sh/docs/troubleshooting#<anchor> so you can jump straight from a failing terminal to the right fix.

Every entry follows the same shape: the verbatim symptom, the likely cause, a numbered fix, a short note on why the failure happens, and cross-links to related docs or flags.

If you cannot find your symptom, run mcptest doctor first. It checks the common environmental issues (binary on PATH, config readable, transport reachable, auth tokens present) and prints a one-line summary per check.

1. Connection and transport

<a id="server-unreachable-exit-code-3"></a> <a id="server-unreachable"></a>

Server unreachable

Symptom

error: server unreachable: connection attempt failed
exit code: 1

Likely cause

The server process or HTTP endpoint named in your test file is not running, or the address you gave mcptest does not match where the server actually listens.

Fix

  1. Run mcptest doctor --url <url> to probe the connection layer by layer, or mcptest doctor to print the resolved target and command line.
  2. For a stdio server, run the command yourself in the same shell and confirm it exits cleanly with no missing-binary error.
  3. For an HTTP server, run curl -v <url>/health (or the equivalent endpoint your server exposes) and confirm it returns 200.
  4. If the server lives in another repo, rebuild it. A stale binary that crashes on startup looks identical to "unreachable" from mcptest's side.

Why it happens

A transport setup failure surfaces as exit code 1 with a diagnostic message; v1.0 does not use a dedicated transport exit code. mcptest gives up before the handshake starts so your test suite does not hang.

Related

<a id="connection-refused-on-localhost"></a>

Connection refused on localhost:PORT

Symptom

error: connection refused: tcp://127.0.0.1:8765

Likely cause

Nothing is listening on that port. Either the server failed to bind, it bound to a different interface (0.0.0.0 vs 127.0.0.1 vs ::1), or it is still starting up.

Fix

  1. Confirm the port is open: lsof -iTCP:8765 -sTCP:LISTEN (macOS, Linux) or Get-NetTCPConnection -LocalPort 8765 (Windows PowerShell).
  2. If empty, start the server and watch its log for a "listening on" line.
  3. If the server listens on ::1 but you wrote 127.0.0.1, change one of them. IPv4 and IPv6 loopback are not the same socket.
  4. If the server is slow to start, pass --wait-for-ready so mcptest polls until the server accepts initialize before the first request. For a URL server you can also set a wait_for_ready: probe in the server: block.

Why it happens

ECONNREFUSED means the OS handed mcptest a "no listener" answer immediately. This is different from a timeout, where something is listening but not responding. Look at the kernel's view of the socket, not the server's logs.

Related

<a id="tls-handshake-failed"></a>

TLS handshake failed

Symptom

error: TLS handshake failed: invalid peer certificate: UnknownIssuer

Likely cause

The server's certificate is self-signed, expired, or issued by a CA your machine does not trust. mcptest uses rustls and the OS trust store; it does not silently fall back to plaintext.

Fix

  1. For local development with a self-signed cert, pass --insecure-skip-verify and the warning that prints. Never use that flag against a production server.
  2. For a real but uncommon CA, add the CA cert to your OS trust store and restart your shell so mcptest picks up the new roots.
  3. For an expired cert, renew it on the server side. mcptest will not bypass an expired cert even with verbose flags.
  4. To confirm what the server is presenting, run openssl s_client -connect host:443 -showcerts and inspect the chain.

Why it happens

rustls is strict on purpose. Most "TLS handshake failed" reports trace back to a chain the client cannot complete, not a protocol mismatch.

Related

<a id="protocol-version-mismatch"></a>

Protocol version mismatch: server returned X, client expects Y

Symptom

error: protocol version mismatch: server returned 2024-11-05, client expects 2025-03-26

Likely cause

mcptest pins a specific MCP protocol revision per release. Your server speaks a different revision and did not negotiate down.

Fix

  1. Check mcptest --version. The output includes the MCP revision it targets.
  2. Upgrade the server to the matching revision, or pin mcptest to a release that matches your server. The release notes list the supported revision.

Why it happens

MCP is still evolving. mcptest refuses to guess: if the initialize response returns a version it does not understand, it fails fast rather than try to parse a payload with the wrong shape.

Related

<a id="server-initialized-but-tools-list-empty"></a>

Server initialized but tools/list returned empty

Symptom

warning: server initialized successfully but tools/list returned 0 tools

Likely cause

The server is up and the handshake worked, but the server is not registering any tools. Either the tool registry is empty, the server is waiting on asynchronous initialization, or the server reports tools under a different capability.

Fix

  1. Run mcptest tools --url <url> to print the raw tools/list response.
  2. Check the server's capabilities block in the initialize response. If tools is absent or {"listChanged": true} without a current list, the server is telling you it will send tools later.
  3. If your server reads tools from a config file, confirm the file is on the path the server sees.
  4. If your tests expect prompts or resources instead, switch to prompts/list or resources/list assertions.

Why it happens

An empty tools list is a valid response. mcptest warns but does not error so your assertions can decide whether empty is expected.

Related

2. Auto-discovery

<a id="no-servers-discovered"></a>

No servers discovered; mcptest doctor shows zero

Symptom

$ mcptest doctor
no MCP servers found

Likely cause

Auto-discovery looks in a fixed set of locations (Claude Desktop config, Cursor config, your project's mcptest.yaml). None of those exist on your machine, or they exist but are empty.

Fix

  1. Run mcptest doctor --verbose to see every path mcptest checked.
  2. If you use Claude Desktop, open it once so it creates its config file, then add a server entry from the UI.
  3. To point mcptest at a config explicitly, pass --config path/to/file.yaml.
  4. To register a server inline for a single test run, use --server name=command args.

Why it happens

Auto-discovery is convenience, not magic. mcptest will not crawl your home directory; it checks the well-known paths and stops.

Related

<a id="server-name-conflict"></a>

Server name conflict: ambiguous reference to <name>

Symptom

error: ambiguous reference to server "github": found in claude-desktop and project config

Likely cause

Two discovered configs both define a server with the same name and your test references it by that name. mcptest will not guess which one you meant.

Fix

  1. Run mcptest list-servers to see every source and the name it uses.
  2. In your test file, qualify the server: server: claude-desktop:github.
  3. Or rename one of them in the source config so the names no longer collide.

Why it happens

Discovered configs are merged for convenience, but the merge is not authoritative. A silent override would lead to tests running against the wrong server for weeks before someone noticed.

Related

<a id="claude-desktop-config-parse-failed"></a>

Claude Desktop config file exists but parse failed

Symptom

warning: ~/Library/Application Support/Claude/claude_desktop_config.json: parse failed at line 12
discovery falling back to project config only

Likely cause

The Claude Desktop config has malformed JSON. A common cause is hand-editing the file and leaving a trailing comma or unquoted key.

Fix

  1. Open the file in a JSON-aware editor and look at the line the warning reports.
  2. Validate with jq . < ~/Library/Application\ Support/Claude/claude_desktop_config.json.
  3. Restore from a known good backup if Claude Desktop wrote a bad file (rare but reported).
  4. Restart Claude Desktop after fixing so it does not overwrite your edit.

Why it happens

mcptest does not silently skip broken configs; it warns so you know your discovery surface is smaller than you expected.

Related

<a id="wrong-path-on-linux"></a>

Wrong path: ~/Library/Application Support not found (Linux)

Symptom

debug: skipping path ~/Library/Application Support/Claude (not present)

Likely cause

That path is macOS-only. On Linux, Claude Desktop (when available) reads from ~/.config/Claude/. The debug line is informational, not an error, but if you see it on Linux it means mcptest is checking macOS paths that will never exist.

Fix

  1. Confirm your OS. mcptest's discovery logic uses dirs::config_dir() so it should pick the right base on each platform.
  2. If you see macOS paths on Linux, your build is misdetecting the platform. Run mcptest --version and confirm the target triple.
  3. On Linux, drop your config at ~/.config/Claude/claude_desktop_config.json or use a project-local mcptest.yaml.

Why it happens

mcptest probes each known host's config location. Paths that do not exist on your platform are skipped silently at info level, debug level if you asked.

Related

3. Auth

<a id="oauth-flow-opened-browser-but-hung"></a>

OAuth flow opened browser but hung

Symptom

mcptest prints "opening browser for OAuth authorization" and then waits forever. The browser tab loads the provider but no token arrives.

Likely cause

The OAuth callback cannot reach mcptest's local listener. Usually the listener port is firewalled, the redirect URI registered with the provider does not match what mcptest sent, or you closed the tab before approving.

Fix

  1. Cancel with Ctrl-C and rerun with --debug. The log prints the listener port and the exact redirect URI it sent.
  2. Compare the redirect URI with the one registered in your OAuth application. They must match byte for byte, including the trailing slash.
  3. If a firewall blocks 127.0.0.1:<port>, allow it for mcptest.
  4. On a headless box where no browser can open, run mcptest login --no-browser and complete the flow by pasting the printed URL into a browser elsewhere.

Why it happens

OAuth's authorization code flow needs the provider to redirect back to a URL mcptest can serve. Anything between the browser and the local port (firewall, proxy, VPN) can drop that redirect.

Related

<a id="401-after-mcptest-login"></a>

401 from server even after mcptest login

Symptom

error: HTTP 401 Unauthorized from https://api.example.com/mcp
hint: token missing or rejected

Likely cause

You authenticated but the token is not being sent, the token is for a different audience, or the server expects a different header. mcptest stores tokens per-server, keyed on the server name, so a token logged in for prod-api will not be sent to staging-api.

Fix

  1. Re-run mcptest login <server> (or mcptest login --url <url>) for the exact server name your test references; tokens are cached per server.
  2. If you authenticated against a different server name, log in again with the correct one.
  3. For a static token, confirm the env var named by bearer_token_env holds a valid token and is exported in the shell that runs mcptest.
  4. If your provider issues audience-scoped tokens, confirm the aud claim matches the server.

Why it happens

mcptest will not paper over a 401. If the token store has a token but the server rejects it, the most likely cause is a wrong audience or an expired token, not a bug in mcptest's token cache.

Related

<a id="token-expired-mid-suite"></a>

Token expired; tests fail mid-suite

Symptom

Tests 1 through 5 pass. Test 6 fails with HTTP 401. Tests 7 through 12 also fail with HTTP 401.

Likely cause

The access token expired during the run. mcptest does not auto-refresh in-flight; if a token expires mid-suite, the remaining requests fail.

Fix

  1. Re-run mcptest login <server> before the suite to mint a fresh token.
  2. If your provider supports refresh tokens, enable them in your OAuth app and log in again. mcptest stores the refresh token and refreshes on a 401, but only on the first 401 per request (see OAuth refresh).
  3. For a static bearer token, rotate the value in the env var named by bearer_token_env before a long run.

Why it happens

mcptest refreshes once per request on a 401 rather than proactively rotating tokens mid-suite, which would mean some tests run under one identity and some under another.

Related

<a id="env-var-not-set"></a>

Env var ${GITHUB_TOKEN} referenced but not set

Symptom

error: undefined environment variable: GITHUB_TOKEN
referenced from: examples/github.yaml:14

Likely cause

Your test file interpolates ${GITHUB_TOKEN} but the variable is not in your shell environment or .env file. mcptest fails closed; it will not substitute an empty string.

Fix

  1. Set it in your shell: export GITHUB_TOKEN=... and rerun.
  2. Or add it to a .env file at the project root. mcptest reads .env automatically via dotenvy.
  3. If the variable is optional, use ${GITHUB_TOKEN:-} to default to empty, or ${GITHUB_TOKEN:-fallback} for a literal fallback.
  4. To see the resolved config without running the tests, use mcptest run --print-config.

Why it happens

Silent substitution of empty strings has burned too many people. mcptest errors on undefined refs by default so a missing token does not show up as a confusing 401 three steps later.

Related

4. Test failures

<a id="schema-validation-failed-but-response-looks-right"></a>

Schema validation failed but the response looks right

Symptom

test "list users" FAILED
  schema validation: expected string, got integer at $.users[0].id

The response in your terminal clearly shows "id": "abc-123".

Likely cause

You are looking at the wrong test run, the wrong response (mcptest may have captured a paginated second page), or your schema is stricter than the actual data on a different record.

Fix

  1. Rerun with --debug and inspect the raw response in the wire log, or capture the full run with --reporter json --output run.json and read the recorded response there.
  2. Confirm which array element failed. The path $.users[0] is the first item; if your schema runs on every item, the failing one may be index 47.
  3. Loosen the schema if the field is genuinely polymorphic, or fix the data on the server side.

Why it happens

Schema errors quote the JSON Pointer of the failing node, not the response as a whole. Skimming the response top-to-bottom can mislead you when only one nested field is wrong.

Related

mcptest run prints "no config" or validates the wrong file

Symptom

mcptest run: no config at /path/to/mcptest.yaml

or, after pointing --config at the file mcptest init wrote:

mcptest run: config validation failed for mcptest.yml
  at root: Additional properties are not allowed ('parallel', 'reporter', 'timeout_secs' were unexpected)

Likely cause

mcptest run with no --config looks for mcptest.yaml (with the .yaml spelling) in the current directory. mcptest init scaffolds two files: mcptest.yml, which holds project defaults (reporter, parallelism, timeout), and tests/example.yaml, the actual suite. mcptest.yml is not a suite, so validating or running it fails the schema check, and the .yml name does not match the .yaml default.

Fix

  1. Point run at the suite: mcptest run --config tests/example.yaml.
  2. Or rename your suite to mcptest.yaml so the bare mcptest run finds it.
  3. Do not pass mcptest.yml (the project-defaults file) to run or validate; it is read for defaults, not as a test suite.

Why it happens

The defaults file and the suite are separate concerns: defaults rarely change, suites change often. Keeping them in different files (and different extensions) avoids reloading defaults on every edit, at the cost of this one-time naming surprise.

Related

<a id="snapshot-mismatch-on-every-run"></a>

Snapshot mismatch on every run (non-deterministic output)

Symptom

test "search docs" FAILED
  snapshot mismatch: response differs from .mcptest/snapshots/search_docs.json

The diff shows fields like timestamp, request_id, or floating-point scores changing every run.

Likely cause

Your assertion is comparing the whole response but the response includes non-deterministic fields. The snapshot matcher records the full value and diffs it byte for byte on later runs, so any volatile field fails it.

Fix

  1. Stop snapshotting the whole response. Use a contains matcher with just the stable fields: against an object it passes when those fields match and ignores the rest, so volatile fields like timestamp and request_id never enter the comparison.
  2. For a value you cannot pin exactly, switch to a looser deterministic matcher such as regex for a formatted string.
  3. Once the response is genuinely deterministic, regenerate the snapshot with mcptest run --update-snapshots.

Why it happens

Snapshot tests are powerful and brittle. The fix is to assert only the fields that should be stable, not to scrub the response in the server.

Related

<a id="tests-time-out-at-60s"></a>

Tests time out at 60s but the server replied

Symptom

test "long query" FAILED
  timeout: no response within 60s

The server log shows the response was sent at 12 seconds.

Likely cause

mcptest received the response but is still waiting on something else: an async notification, a follow-up tool call, or the server forgot to close the JSON-RPC envelope. The timeout fires because the matcher is incomplete.

Fix

  1. Run with --debug and look for received frame log lines after the 12-second mark. If they are absent, the response never reached mcptest.
  2. If frames are present but the matcher is waiting on a missing field, the matcher is overspecified. Compare your expected shape with the captured actual response.
  3. If your test expects a notifications/cancelled or progress update that the server does not send, remove that expectation.
  4. Adjust the timeout for slow tests: timeout_ms: 120000 per test or --timeout 120 globally (the flag takes seconds).

Why it happens

The 60-second default is a safety net. A test that genuinely needs longer is fine, but a test that times out despite a fast response is almost always a matcher bug.

Related

<a id="bail-exited-on-test-3"></a>

--bail exited on test 3 but tests 4 through 10 were not actually broken

Symptom

$ mcptest run --bail
... test 3 FAILED
suite halted (--bail)

You expected tests 4 through 10 to also run.

Likely cause

--bail halts on the first failure on purpose. If you want to see every failure in one run, do not pass --bail.

Fix

  1. Drop the --bail flag for a full run (--bail is sugar for --maxfail 1).
  2. To stop after a specific number of failures instead, pass --maxfail 5.
  3. To run only a subset, filter by name substring with --filter login.

Why it happens

--bail is for CI where one failure invalidates the rest, or for fast iteration where you only care about the next failing test. It is not a default.

Related

5. CI

<a id="github-actions-cache-restore-failed"></a>

GitHub Actions: cache restore failed

Symptom

Warning: Failed to restore: getCacheEntry failed: Cache not found for input keys: ...

Likely cause

The cache key has changed (a new mcptest version, a new lockfile hash, a new OS image) so there is no existing cache to restore. This is a warning, not a failure; the job continues and saves a new cache for next time.

Fix

  1. Read the warning, not the failure. If the job finishes, you do not need to act.
  2. If you do want a stable cache hit, pin the cache key on values that change less often (commit SHA of mcptest.yaml, for example).
  3. To purge old caches manually, use the GitHub Actions cache UI under repository Settings.

Why it happens

actions/cache reports cache misses as warnings so jobs do not fail when the cache is empty (for example, on the first run after a key change).

Related

<a id="gitlab-code-quality-not-visible"></a>

GitLab CI: Code Quality report not visible in MR

Symptom

mcptest writes a Code Quality JSON report. The job succeeds. The merge request does not show any quality findings.

Likely cause

The report path in artifacts.reports.codequality does not match where mcptest wrote the file, or the file is empty (no findings is rendered as no widget).

Fix

  1. Confirm the artifact path: artifacts: { reports: { codequality: mcptest-quality.json } }. The path is relative to the job's working directory.
  2. Check the artifact list at the top of the job log. If the file is missing, the report path is wrong.
  3. Open the JSON. An empty array is valid but shows nothing in the UI.
  4. GitLab only renders Code Quality on MR pages, not on commit pages. Open the MR view, not the pipeline view.

Why it happens

GitLab's report integration requires both the right artifact key and a non-empty payload. Either missing means the widget does not render.

Related

<a id="junit-rejected-by-test-reporter"></a>

JUnit XML rejected by dorny/test-reporter

Symptom

Error: Cannot find any valid test result file matching pattern

or

Error: Unsupported XML schema: <testsuites> element missing

Likely cause

The reporter expects standard JUnit XML. mcptest emits JUnit by default but the path in your workflow may not match, or you ran the reporter before the mcptest step finished writing.

Fix

  1. Confirm mcptest writes junit.xml: mcptest run --reporter junit --output junit.xml.
  2. In the workflow, set path: junit.xml on the reporter step.
  3. Make sure the reporter runs if: always() so it executes even when mcptest fails the build.
  4. Open the XML in a text editor. It must start with <?xml ...?> and a <testsuites> root. If it does not, you may be reading the wrong file.

Why it happens

The JUnit schema is not strictly versioned; reporters disagree on what counts as valid. mcptest emits the most-compatible shape (testsuites with nested testsuite and testcase). If your reporter still complains, file a bug with the offending XML.

Related

<a id="works-locally-fails-in-ci"></a>

mcptest run works locally but fails in CI

Symptom

Tests pass on your laptop. The same mcptest run command in CI fails with mismatched assertions or missing servers.

Likely cause

Either the CI environment lacks something you have locally (an env var, a secret, a cached file under .mcptest/), or the CI runner is a different OS or arch and the server behaves differently.

Fix

  1. Run mcptest doctor in CI as the first step. The output will list what is missing.
  2. Confirm every env var the test file references is set in CI as a secret or workflow variable.
  3. Make sure your CI workflow checks out a fresh clone. Stale .mcptest/ directories from a previous run can hide problems.
  4. If the failure is OS-specific, see the next entry.

Why it happens

"Works on my machine" is almost always a difference in environment, not in mcptest. The doctor command exists to surface those differences quickly.

Related

<a id="different-results-linux-vs-macos"></a>

Different test results between Linux and macOS runners

Symptom

The same test passes on ubuntu-latest and fails on macos-latest, or vice versa.

Likely cause

The server under test depends on a platform-specific binary, the test references a path that exists on one OS and not the other, or filesystem case-sensitivity differs (Linux is case-sensitive by default, macOS is not).

Fix

  1. Look at the failure on each runner. If it is a "binary not found" error, install the binary in your CI setup for that OS.
  2. If the failure is a path mismatch (/tmp vs /var/folders/...), use ${TMPDIR} rather than hardcoding /tmp.
  3. For case-sensitivity, rename files so the on-disk case matches the reference exactly.
  4. If only one OS is in scope for your project, drop the other runner from your matrix.

Why it happens

mcptest itself is identical across platforms (single static binary, no native deps). Differences come from the server, the shell, or the test data.

Related

6. Performance

<a id="first-run-30-seconds"></a>

First run took 30 seconds; expected the cache to help

Symptom

$ mcptest run
... 30.4s

The docs mention a cache. Where is it.

Likely cause

The cache helps on the second run. The first run has no cache to hit; mcptest must do every transport setup, every schema compile, every handshake from scratch.

Fix

  1. Run the suite a second time. Subsequent runs should be substantially faster if the suite is cacheable.
  2. Confirm the cache directory exists: ls .mcptest/cache.
  3. If the second run is also slow, see the next entry.

Why it happens

Cold caches are cold. mcptest does not pre-warm anything; the first run pays the cost so later runs do not.

Related

<a id="cache-hit-but-test-still-ran"></a>

Cache hit but the test still ran (looking for the cache indicator)

Symptom

The summary line shows tests running, not a "cached" indicator.

Likely cause

The pretty reporter shows cache status in a column that may be hidden in narrow terminals, or your test has conditions that bypass the cache (a fresh server start, a new auth token, or a cache: never directive on the test).

Fix

  1. Run with --reporter json --output run.json and inspect the recorded cache status per test; the structured output is the reliable signal when the terminal column is hard to read.
  2. If a test you expected to cache did not, check for cache: never in the YAML or in inherited defaults.
  3. Confirm the run is not bypassing the cache wholesale: --no-cache, --no-cache-read, and --no-cache-write all skip cache reads.
  4. Cache invalidation also fires when the schema URL changes; if you edited the schema, expect a miss.

Why it happens

The cache is keyed on inputs that affect the response shape. Anything that changes those inputs invalidates the entry. Some of those triggers are not obvious until you inspect the JSON report.

Related

<a id="memory-grows-during-long-run"></a>

Memory grows during a long run

Symptom

A 30-minute suite uses 200 MB at start and 1.5 GB by the end. RSS grows roughly linearly.

Likely cause

mcptest holds captured responses in memory by default so the final report can include them. A long suite that captures big responses (large tools/list results, big resource reads) accumulates that data.

Fix

  1. Use --reporter json --output run.ndjson. The JSON reporter writes one object per line as tests finish, so results are flushed incrementally rather than held until the end.
  2. Split a very long suite into chunks with --shard INDEX/TOTAL (or narrow it with --filter) and combine the reports afterward. Each shard is a separate, shorter-lived process.
  3. Drop --retry on a long run if it is set; retained retry attempts add to the in-memory footprint.

Why it happens

The default optimizes for "I can re-inspect any failure after the run." That is great for small suites and heavier for very long ones. Sharding keeps each worker process short-lived so memory is reclaimed between chunks.

Related

7. Agent tests

<a id="agent-no-credential"></a>

Agent test runs against the stub even though I set the API key

Symptom:

INFO mcptest::cli: no credential, falling back to stub provider
     env_var=ANTHROPIC_API_KEY agent=weather query model=claude-sonnet-4-5

Three usual causes:

  1. Env var typo. ANTHROPIC_KEY instead of ANTHROPIC_API_KEY, GOOGLE_KEY instead of GEMINI_API_KEY (or GOOGLE_API_KEY), etc. The CLI logs the exact name it looked up; copy that.

  2. Set in a different shell. export ANTHROPIC_API_KEY=... only lives in the shell that ran the export. If you run mcptest from a separate terminal, the var is not there. Either export in both shells or load from a .env file:

    echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env
mcptest run --env-file .env --record
  3. Empty string. ANTHROPIC_API_KEY="" counts as missing. present_env treats blank as unset on purpose; check with env | grep ANTHROPIC.

<a id="agent-cassette-stale"></a>

"cassette stale on field <field>" when running an agent test

cassette for agent `weather query` model `gpt-5` is stale on field
`model` (cassette=`gpt-4o`, yaml=`gpt-5`); rerun with --record

You changed the model id, prompt, or system prompt in YAML, but the cassette on disk was recorded under a different value. The loader refuses to replay the wrong recording. Two fixes:

<a id="agent-wrong-tool"></a>

Real model picks the wrong tool, stub passes

Most common failure when you first switch from --record against a stub to a real provider:

mcptest::weather query [gpt-5]  FAIL
        tool_calls[0].name: expected get_weather, got search

The stub returns whatever you scripted; a real model picks based on the tool descriptions plus the prompt. Three options:

  1. Tighten the system prompt so the model has less room to misroute. Be explicit: "Call get_weather when asked about weather. Do not call any other tool."
  2. Improve the tool description. The model reads the description field returned by tools/list. A clearer description routes models better than a stricter prompt.
  3. Drop the model from the matrix if it consistently misroutes. That is what the matrix is for: surfacing model fit.

<a id="agent-unknown-namespace"></a>

"model called tool X but no server is configured"

Multi-server runs namespace every tool as <server>__<tool>. If the stub emits the bare tool name in a multi-server context, the driver cannot route it. Fix the stub script to emit the namespaced name:

// wrong (single-server convention):
StubReply::ToolUse { name: "create_issue".into(), args: ... }

// right (multi-server):
StubReply::ToolUse { name: "issues__create_issue".into(), args: ... }

For real models this error means the model invented a tool name the catalog does not include. Usually a prompt issue; tighten it.

<a id="agent-budget-trip"></a>

"agent run failed: spend cap exceeded"

The per-test or per-suite USD-cent cap tripped before the loop finished. Either:

Note the per-suite cap multiplies across a matrix run; if you have four models, each priced at one cent per call, a full record pass costs four cents.

<a id="agent-jury-split"></a>

llm-jury verdict says split decision

expect[0] `final_response` failed: llm-jury failed: jury: 1/3 passing
(quorum 0.50, alpha 0.412)

1/3 passing plus a low Krippendorff alpha (under 0.6) means the jurors disagreed. Two paths:

<a id="agent-cassette-missing-key"></a>

Cassette file exists but mcptest run re-records every time

You probably have --record set in your CI environment. The flag forces a live dispatch regardless of cassette state. Drop the flag for the replay run; only set it when you intend to rewrite the recording.

<a id="proxy-cannot-reach-server"></a>

"connect to <server> failed: error trying to connect" behind a corporate network

Symptom

mcptest run: connect to `prod-api` failed: error trying to connect

while every other tool on the machine can reach the same URL.

Cause

Your shell has HTTPS_PROXY exported but mcptest was launched from a context that did not inherit it (a launchd service, a systemd unit, an IDE that scrubs the env, or a Docker container).

Fix

Confirm what mcptest sees:

mcptest doctor
# proxy: default (env: HTTP_PROXY / HTTPS_PROXY / NO_PROXY)

If the summary says default (env: ...) but the request still fails, the env var is not actually reaching the process. Pass the proxy explicitly:

mcptest --https-proxy http://proxy.corp:8443 --noproxy localhost run

For the full proxy reference see url-targets.md.

<a id="proxy-no-proxy-conflict"></a>

"--no-proxy cannot be combined with --proxy"

Symptom

error: --no-proxy cannot be combined with --proxy / --http-proxy / --https-proxy

Cause

You set both an explicit proxy URL and --no-proxy. mcptest does not infer which one wins.

Fix

Decide which run you want: --no-proxy to skip every proxy (including env vars), or --proxy <URL> to route through one. Drop the other flag.

8. Installation

<a id="brew-bottle-not-found"></a>

brew install fails with bottle not found

Symptom

Error: mcptest: no bottle available for your OS or architecture

Likely cause

Homebrew publishes prebuilt bottles for the common architectures. If you are on an unsupported combination (older macOS, Linux distro Homebrew does not prebuild), Homebrew falls back to building from source, which fails if the formula does not allow it.

Fix

  1. Run brew update first; the bottle for your platform may have been published since you last updated.
  2. To build from source, pass --build-from-source. This requires a Rust toolchain.
  3. Alternative: install with cargo install mcptest or download the static binary from the GitHub Releases page.

Why it happens

Bottles are per-platform. If yours is missing, either it was not built or the build was uploaded to a tap you have not added.

Related

<a id="cargo-install-fails-on-fresh-machine"></a>

cargo install mcptest fails on a fresh machine

Symptom

error: linker `cc` not found

or

error: failed to run custom build command for `openssl-sys`

Likely cause

Your machine has Rust installed but is missing a C toolchain or system libraries that some transitive dependency needs.

Fix

  1. On Ubuntu/Debian: sudo apt install build-essential pkg-config libssl-dev.
  2. On Fedora/RHEL: sudo dnf install gcc pkgconfig openssl-devel.
  3. On macOS: xcode-select --install.
  4. On Windows: install the Visual Studio Build Tools with the "Desktop development with C++" workload.
  5. After installing the toolchain, rerun cargo install mcptest.

Why it happens

mcptest itself uses rustls and avoids OpenSSL, but some optional dependencies in transitive crates may still want a C compiler. A clean Rust install is not enough on minimal images.

Related

<a id="docker-image-works-locally-fails-in-ci"></a>

Docker image works locally but not in CI

Symptom

docker run mcptest/mcptest:latest mcptest run works on your laptop. In CI, the same image errors with "permission denied" or "exec format error."

Likely cause

Fix

  1. For permissions: mount with --user $(id -u):$(id -g) so the in-container user matches your host.
  2. For architecture: pull the right tag explicitly, for example mcptest/mcptest:1.2.0-arm64, or use Docker Buildx in your CI to pick the matching variant.
  3. Confirm the image platform: docker inspect --format '{{.Architecture}}' mcptest/mcptest:latest.

Why it happens

Docker manifests are usually multi-arch, but cached layers, mirror configurations, and CI runner mismatches can still serve the wrong image.

Related

<a id="windows-not-found-in-path"></a>

Windows: mcptest not found in PATH after install

Symptom

PS> mcptest
mcptest : The term 'mcptest' is not recognized ...

Likely cause

The installer placed the binary in a directory that is not on your PATH, or the current shell was started before the install finished and has not picked up the change.

Fix

  1. Open a new PowerShell window. Path changes do not apply to existing shells.
  2. Confirm the binary exists. For cargo install, it lands in %USERPROFILE%\.cargo\bin\mcptest.exe. For the MSI installer, it lands in C:\Program Files\mcptest\.
  3. Add the directory to PATH via System Properties > Environment Variables if it is missing.
  4. Confirm with where.exe mcptest.

Why it happens

Windows does not refresh PATH for running processes. New environment variables only apply to processes started after the change.

Related

How to add a troubleshooting entry

This page is hand-curated. Anchors are stable URLs, not generated from headings, so we can rename a heading later without breaking external links. When you add an entry:

  1. Copy the template below and place it in the right section. If the section list does not cover the symptom, add the section in the order it appears in a typical user's workflow (connection, then discovery, then auth, and so on).
  2. Pick a kebab-case anchor that captures the symptom in a few words. Once merged, the anchor is a contract: do not rename it. If a better anchor comes along, add the new one as a second <a id="..."> and leave the old one as well.
  3. Quote the symptom verbatim. Strip ANSI color codes but keep the rest. A user copy-pasting their error should land on this entry.
  4. Keep the fix short and runnable. No paragraphs of theory. If the background matters, it goes in "Why it happens."
  5. Cross-link rather than duplicate. If mcptest doctor deserves its own page, link to it.
  6. No em-dashes. Use commas, periods, or parentheticals.

Template:

<a id="kebab-case-anchor"></a>

### Verbatim symptom or error message

**Symptom**

The exact text the user sees.

**Likely cause**

One or two lines.

**Fix**

1. Step one.
2. Step two.

**Why it happens**

Two or three sentences of background.

**Related**

- Link to other docs.
- Relevant CLI flags.