@spboucher/pangram-experiment-mcpv1.0.0

pangram-experiment-mcp

An MCP server for one kind of experiment: revise a text step by step, each step for an editorial objective fixed in advance, and record what the Pangram AI-text detector reports for every version.

The detector score is an observed outcome. The server is built so that it cannot quietly become a target:

• the editorial plan is frozen and hashed before any score exists, and an objective that mentions detection is refused;
• each step takes exactly one revised text, which is recorded and evaluated whatever the result;
• every score comes from a real evaluation of exactly the stored text, and a text is scored once;
• standalone evaluations are closed while an experiment is running;
• the experiment stops if a revision loses factual content.

The server calls no language model. The agent writes the revisions. This is a measuring instrument: it is not meant for, and does not help with, making a text pass a detector.

What leaves your machine

Each evaluation sends the full text to Pangram, a third-party service, with your API key, and is billed to that key. Do not submit confidential text. Nothing is sent anywhere else: no telemetry, no other host. Requests carry a generic User-Agent (pangram-experiment/1.0). The key is read from the environment; it is never written to a file, a log or a tool answer, and it is redacted from error bodies.

Requirements

Node.js 20 or later, and a Pangram API key.

Set up

skills download @spboucher/pangram-experiment-mcp --extract ./pangram-experiment-mcp
cd pangram-experiment-mcp && npm ci --omit=dev

Claude Code:

claude mcp add --transport stdio pangram-experiment \
  --env PANGRAM_API_KEY=<your key> \
  --env PANGRAM_EXPERIMENT_ROOT=/absolute/path/to/project \
  -- node /absolute/path/to/pangram-experiment-mcp/server.mjs

Codex (~/.codex/config.toml):

[mcp_servers.pangram-experiment]
command = "node"
args = ["/absolute/path/to/pangram-experiment-mcp/server.mjs"]
env = { PANGRAM_API_KEY = "<your key>", PANGRAM_EXPERIMENT_ROOT = "/absolute/path/to/project" }

Cursor (.cursor/mcp.json):

{ "mcpServers": { "pangram-experiment": { "type": "stdio", "command": "node", "args": ["/absolute/path/to/pangram-experiment-mcp/server.mjs"], "env": { "PANGRAM_API_KEY": "<your key>", "PANGRAM_EXPERIMENT_ROOT": "/absolute/path/to/project" } } } }

These snippets follow the sibling packages; this server was tested with the official MCP client over stdio, not registered in each of these clients.

Environment

Tools

Order of work: experiment_init → experiment_baseline → (experiment_next → write the revision → experiment_submit) for each step → experiment_report.

What is stored

<root>/.pangram-experiments/<id>/
  experiment.json    the frozen design, with its hash
  history.jsonl      append-only: init, step, score, score_failed, stopped
  texts/NN.txt       the original (00) and each revision
  raw/NN.json        Pangram's raw answer for that text
  report/            the report files

Files are created with mode 0600. A step record is written before Pangram is called, so a failure leaves a step without a score, never the reverse. Editing the design or a stored text is detected by hash and stops the experiment.

Measures

For each revision the server computes, against the original and the previous version: word-set overlap (Jaccard), cosine of word counts, length ratio, and which numbers and capitalised names of the original are missing. These are lexical proxies, not a measure of meaning. Next to them it stores the agent's own assessment: six scores from 1 to 5 and a verdict on each key fact declared at the start.

The experiment stops when a key fact is judged lost, when more numbers are missing than the design allows (default: none), or when fewer than the required share of names remain (default: 80 %).

The Pangram API as used here

Verified on 2026-11-10 with real calls: POST /task with {text, model} returns a task_id (and a notice when the requested model is served by a newer one); GET /task/{id} returns stage: STAGE_INFERENCE, then STAGE_SUCCESS with fraction_ai, fraction_ai_assisted, fraction_human, labels and per-segment windows. The text echoed back can differ from the submitted one in white space; any other difference makes the server discard the result.

Not verified: failure stages, error bodies, rate limits, billing, length limits, other models, and whether the same text always gets the same result. These are handled generically: a non-success answer is kept as it came, and no score is ever produced locally.

Tests

npm test                                   # 21 tests, no network
PANGRAM_API_KEY=... npm run test:live      # 1 test, 3 real evaluations of an invented text

The offline tests replace Pangram with a fake built from two recorded answers (test/fixtures/). They cover a full run, every refusal, the frozen plan, the stop rule, the budget, retries, failed and mismatched evaluations, the report files, path confinement, and the absence of the key from every file and answer. The protocol tests drive the server over stdio with the official client against a local HTTP stand-in.

Limits

• One text per experiment, one run, objectives applied in sequence: the result describes that trajectory and supports no general or causal claim.
• A detector score is not evidence of who or what wrote a text.
• The server cannot see how a revision was written. It guarantees one submission per step, not that the agent ignored earlier scores.
• The anchor check sees digits and capitalised names only. Facts in words or in lower case are protected only as declared key facts.
• The quality scores are the revising agent's own.
• The record is hash-checked but unsigned: someone with write access to the directory can rebuild it.
• Tested on macOS with Node 26 only; engines allows Node 20 and later.

Licence

MIT.