Frequently asked questions

Short, factual answers to the questions most people ask before, during, and right after their first run. Each answer leads with the most important fact so an AI agent or search engine can lift it cleanly. For depth, follow the links.

About mcptest

What is mcptest?

mcptest is an open-source Rust CLI for testing Model Context Protocol (MCP) servers. It runs declarative YAML tests against any MCP server (stdio, streamable HTTP, or legacy SSE transport), validates spec compliance, enforces performance budgets, and ships nine reporters (pretty, JSON, JUnit, Markdown, HTML, SARIF, GitLab Code Quality, NDJSON, TAP). Apache-2.0 licensed, single-binary install, published by Soap Bucket LLC. See what is mcptest.

Who builds mcptest?

Soap Bucket LLC (the legal entity behind soapbucket.com) builds and maintains mcptest. The project is Apache-2.0 licensed and accepts community contributions through GitHub.

What does mcptest cost?

The mcptest CLI is free under Apache-2.0.

Is mcptest production-ready?

Yes. v1.0 is the stable release. CI runs cargo fmt, clippy, doc, build, and test on every push across Ubuntu, macOS, and Windows. Release artifacts are signed and verifiable with the published SHA256SUMS sidecar.

How does mcptest compare to other MCP tools?

mcptest is the CI-grade test runner for MCP servers. It is not a debugger (use your editor's MCP tooling), not a fuzzer (use a property-testing library), and not a load tester. It is a declarative spec runner with deterministic record/replay and machine-readable reporters. See concepts for the framing.

Install

How do I install mcptest?

On macOS or Linux, run brew install soapbucket/tap/mcptest or curl -sSL https://download.mcptest.sh/install.sh | sh. On Windows, run the install script under WSL or download a signed release archive from GitHub Releases. See the install guide.

Which platforms does mcptest support?

Linux (x86_64, aarch64), macOS (Intel and Apple Silicon), Windows under WSL or Git Bash. The release pipeline cross-compiles for all five targets. Native Windows PowerShell support is on the roadmap.

Can I install mcptest via Docker?

Yes. The image is soapbucket/mcptest:latest, built on gcr.io/distroless/static. Run with docker run --rm -v "$PWD:/work" -w /work soapbucket/mcptest:latest run. See Docker and package runners.

Can I run mcptest in GitHub Actions?

Yes. Install the CLI in a workflow step with curl -fsSL https://download.mcptest.sh/install.sh | sh, then call mcptest run. A composite action is staged as an example under examples/mcptest-action/ but is not yet published, so use the install step for now. See the CI integration guide.

How do I verify the install checksum?

Every release tag publishes a SHA256SUMS file alongside the archives. On Linux, run sha256sum -c mcptest-<version>-<target>.tar.gz.sha256. On macOS, run shasum -a 256 -c instead. The install script does this verification automatically.

Transports and authentication

What MCP transports does mcptest support?

Three: stdio (subprocess), streamable HTTP, and legacy SSE. The same YAML test format works against all three. See concepts.

Does mcptest work with OAuth-protected servers?

Yes. mcptest supports OAuth 2.1 with PKCE and Dynamic Client Registration (RFC 7591). Tokens are discovered from environment variables, refreshed automatically when they expire mid-run, and redacted from every reporter. See authentication.

Can I test a server behind a bearer token?

Yes. Set auth: { bearer_token_env: MCPTEST_TOKEN } on the server entry in your YAML and export the token in the shell before running. mcptest sends Authorization: Bearer ${MCPTEST_TOKEN} and redacts the value from logs and reporter output.

How do I add custom headers to my MCP server requests?

Custom request headers (auth.headers) are reserved for a v1.1 release; the v1 schema accepts the key but the loader logs a warning rather than applying it. For v1, authenticate with bearer_token_env or oauth. See authentication.

Writing tests

What does an mcptest test file look like?

A single YAML file with a servers block (named MCP servers under test) and a tools block (one entry per test). Each test names the server, the tool, the args, and an expect block with one or more matchers. See getting started for a five-minute walkthrough and the YAML reference for the full grammar.

What matchers does mcptest support?

Each assertion targets a value by JSON path and applies one matcher: exact, contains, subset, regex, schema (JSON Schema), and snapshot cover most suites. The string matchers are contains-all, contains-any, icontains, starts-with, and levenshtein; the structural ones are is-json and is-valid-tools-call; llm-judge and llm-jury grade with a model; and not inverts any other matcher. Latency and token limits are separate per-test fields (max_duration_ms, max_response_tokens), not matchers. See the YAML reference for the complete list.

Yes. The imports block at the top of a YAML file merges other YAML files in. Use it to factor common server definitions, variables, or shared test snippets into reusable libraries.

How do I record and replay cassettes?

Run mcptest run --record once to capture; cassettes land under <suite_dir>/cassettes/. Every plain mcptest run after that replays the matching cassette offline, no flag needed. Cassettes are deterministic JSON, normalized so a recording from today and one from next week diff cleanly. See cassettes.

How do I use variables and secrets in my tests?

Variables live in the top-level variables block or in a .env file the loader picks up automatically. Reference them with ${name} in any string field. Secrets live in environment variables, never in the YAML. Every reporter redacts known secret fields by default.

Can I run mcptest against multiple servers in one file?

Yes. List every server under the servers block and reference them by name on each test's server: field. See multi-server suites.

What is the expected-failures baseline?

A JSON file that records which tests are currently known to fail. The runner fails only when a green test turns red, or when a test in the baseline starts passing (so you remember to remove it from the baseline). Adopt mcptest on a server with flaky tests without blocking the PR queue.

Compliance and scorecards

What is the compliance corpus?

A curated set of test scenarios mcptest can run against any MCP server to grade conformance with the protocol spec. Categories include PROTO (handshake), SCHEMA (JSON Schema validation), SEQ (sequencing), TOOL (tool-call semantics), RES (resources), and EDGE (edge cases). See the compliance baseline.

How does mcptest grade an MCP server?

mcptest emits a compliance score on the A+ through F scale, weighted by category severity. A+ is full pass on every scenario, F is below the floor on at least one critical category. See the compliance grade for the rubric.

What is the official conformance bridge?

mcptest can replay the official Model Context Protocol conformance suite through the same matcher engine. See official conformance.

CI integration

Which CI providers does mcptest support?

Every CI platform. mcptest is a single static binary with stable exit codes (0 pass, 1 test failure, 2 config error or bad arguments, 5 cost cap exceeded, 6 coverage below threshold, 7 no tests selected) and machine-readable reporters (JUnit, JSON, SARIF, GitLab Code Quality). First-class integrations: GitHub Actions, GitLab CI, CircleCI. See CI integration and the exit-code table.

What is the SARIF reporter for?

Emitting findings to GitHub Code Scanning, GitLab Security Dashboard, and any other tool that consumes SARIF 2.1.0. See the SARIF reporter.

How do I cache mcptest runs in CI?

mcptest writes a content-addressed cache under ~/.cache/mcptest (or $XDG_CACHE_HOME/mcptest). Cache that directory in your CI provider's cache step keyed on the YAML hash. See cache eligibility.

Reporters and output

Which reporters does mcptest ship?

Nine: pretty, JSON, JUnit, Markdown, HTML, SARIF, GitLab Code Quality, NDJSON, and TAP. Pick one with --reporter and pair it with --output <path> to choose the sink. See the CLI reference.

Can I post-process a run?

Yes. Run with --reporter json --output run.json to capture the canonical run envelope, then mcptest report --format <format> run.json to re-render in any reporter style. See the CLI reference.

Troubleshooting

`mcptest run` failed because my server is unreachable. Which exit code?

A transport failure (the server process or HTTP endpoint named in your YAML cannot be reached) surfaces as exit code 1 with a diagnostic message; mcptest does not use a dedicated transport exit code in v1.0. Run mcptest doctor --url <url> to probe the connection layer by layer, or mcptest doctor to print the resolved target and command line, then verify the server starts independently. See troubleshooting.

How do I see the raw JSON-RPC frames mcptest sends?

Run with --verbose or --debug. Verbose prints variable resolution and discovered files. Debug prints every JSON-RPC frame mcptest sends and every frame it receives, with headers redacted. See the CLI reference.

My server passes locally but fails in CI. What is different?

Common causes, in order: a missing environment variable (mcptest doctor lists every variable the suite references), a different network boundary (CI cannot reach localhost or a VPN-only host), a stale cassette, or a clock skew if the server signs requests. See troubleshooting.

How do I file a bug?

Open an issue at github.com/soapbucket/mcptest/issues with the output of mcptest --version, mcptest doctor, and the YAML that reproduces the issue. Redact secrets first.