SARIF reporter

mcptest emits SARIF, the Static Analysis Results Interchange Format (v2.1.0), so the same run that proves your MCP server works can also surface findings in any tool that already speaks SARIF: GitHub code scanning, GitLab Ultimate, Azure DevOps Advanced Security, Sonar, Semgrep, and the VS Code SARIF Viewer.

Run this example. Run the reference server suite, then render the saved run as SARIF:

mcptest run --config examples/reference-server/tests/smoke.yml --reporter json --output run.json
mcptest report run.json --format sarif --output mcptest.sarif

Note. CI is currently disabled for this repo during heavy development. The snippets below are written to be drop-in when CI is re-enabled. The reporter itself is exercised by cargo test --workspace, including the SARIF schema validation tests under crates/mcptest-core/tests/sarif_validation.rs.

When to pick SARIF

Use SARIF when the consumer cares about findings, not test counts. A typical SARIF consumer renders each result inline on a pull request as a code annotation pointing at the YAML line that authored the test, and groups results by rule across runs. That is a better fit for the compliance suite than JUnit, which models results as test cases.

If your CI consumer wants test totals (X passed, Y failed), keep JUnit. If it wants a security-style finding stream, use SARIF.

What the reporter emits

Every failing test becomes one result entry:

The run object carries tool.driver.{name, version, informationUri, rules}. Every distinct rule_id seen in the run lands in the rules array with a shortDescription, fullDescription, helpUri (https://mcptest.sh/compliance/<RULE-ID>), and defaultConfiguration.level matching the RFC 2119 severity. Run-level timing lives under runs[0].properties.timing (total_duration_ms, cache_hits, started_at, ended_at).

Passing tests are not emitted as result entries because SARIF models a defect stream. Use the JSON reporter or JUnit reporter when you need the full pass list.

GitHub Actions

name: mcptest

on:
  pull_request:
  push:
    branches: [main]

jobs:
  mcptest:
    runs-on: ubuntu-latest
    permissions:
      security-events: write   # required for upload-sarif
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Install mcptest
        run: curl -sSL https://download.mcptest.sh/install.sh | sh

      - name: Run mcptest with SARIF output
        run: |
          mcptest run \
            --config tests/mcp.yaml \
            --reporter sarif \
            --output mcptest.sarif
        continue-on-error: true   # let the SARIF upload run even on failure

      - name: Upload SARIF to GitHub code scanning
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: mcptest.sarif
          category: mcptest

GitHub displays each finding on the "Files changed" tab of the pull request, with the ruleId (for example PROTO-001) and a click-through to https://mcptest.sh/compliance/PROTO-001. The Security tab aggregates findings by rule so a regression spike on EDGE-* rules surfaces at a glance.

Local rendering from a saved JSON run

If you already capture the JSON run envelope, re-render it without a second run:

mcptest report run.json --format sarif --output mcptest.sarif

The redaction policy is re-applied at the dispatch site, so secrets sealed by the JSON reporter stay sealed when SARIF emits.

Validation

crates/mcptest-core/tests/sarif_validation.rs validates every fixture render (sample, sample_full, sample_all_pass) against a vendored SARIF v2.1.0 schema subset under crates/mcptest-core/tests/fixtures/sarif-v2.1.0.schema.json. Extend the fixture when the reporter starts emitting a new field; the official upstream schema is the source of truth, and the link is pinned in the fixture header.