mcptest docs GitHub

Generate tests with an LLM

You do not have to write the first mcptest.yml by hand. Hand a capable model your server's tool list and a grounded prompt, and it returns a starter suite you then refine.

The prompt lives at examples/authoring-prompt.md. It is a plain-text prompt grounded in the v1 YAML schema: the top-level keys, the tools and agents shapes, the full matcher list, and a worked example that validates. Copy it into Claude, GPT, Gemini, or whichever model you already use.

The workflow

  1. Get your server's tool list. Run mcptest discover --url <host> for an HTTP server, call tools/list against a stdio server, or just paste the tool names and argument shapes you already know.
  2. Paste the prompt from examples/authoring-prompt.md, then append your server's start command or URL and that tool list.
  3. Save the model's output as mcptest.yml.
  4. Validate it: mcptest validate --config mcptest.yml. Paste any errors back to the model and ask for a fix. The schema is the source of truth, so a clean validate means the file is well formed.
  5. Run it: mcptest run --config mcptest.yml. Tighten the assertions the model guessed at, and delete the tests that do not earn their place.

What to expect from the model

The prompt asks for deterministic assertions (exact, contains, schema, regex) on the happy path, plus one error path per tool. Treat the result as a first draft. A model is good at the mechanical shape and the obvious cases; you still own the assertions that encode what your server is actually supposed to do.

Keep the published JSON Schema in your editor while you edit. The directive at the top of every generated file, # yaml-language-server: $schema=https://mcptest.sh/schema/v1.json, gives you inline validation as you type.

See also