Benchmarks

Reproducible measurements of the compression pipeline, using the plugin's own modules end to end (token counting, planning, chunk summarization with its retry path, consolidation, content-addressed caching).

Running

npm run bench:static   # pure structural/determinism checks — no model needed
npm run bench          # full run — needs LM Studio running with a model loaded

Full-run results land in bench/results/<date>-<model>.{json,md} and are committed to the repo as the public record.

Metrics

Recall (compacted) — seeded facts answered correctly by the model from the compacted view alone (temperature 0, substring grading with multiple accepted forms). The core fidelity number.
Recall (consolidated) — the same quiz after a forced L1 consolidation; the delta vs compacted recall measures consolidation drift.
Reduction — token size of the compacted view vs the full transcript.
Chunk latency — wall time per chunk summarization call.
Structure — compacted views are well-formed (no mid-tail system messages, no orphan tool results, a user message present).
Cache — recomputing hashes over the same input reuses every cached chunk with zero further model calls.

Fixtures

Deterministic (seeded PRNG), generated in fixtures.ts: a coding-agent chat with seeded constraints, a research chat with sourced findings, a contradictory-updates chat (only the LATEST value counts as correct — stale recall scores as a miss by design), and a tool-spam chat with oversized results.

Caveats

Results are comparable only within one model; every result file is stamped with the model id and settings.
Local model output varies between quant builds and runtime versions even at temperature 0 — treat small recall deltas as noise, structural/cache columns as hard pass/fail.
bench/ is excluded from the plugin build (tsconfig.json includes src only) and is not part of the published artifact's runtime.

Benchmarks

Running

npm run bench:static   # pure structural/determinism checks — no model needed
npm run bench          # full run — needs LM Studio running with a model loaded

Full-run results land in bench/results/<date>-<model>.{json,md} and are committed to the repo as the public record.

Metrics

Recall (compacted) — seeded facts answered correctly by the model from the compacted view alone (temperature 0, substring grading with multiple accepted forms). The core fidelity number.
Recall (consolidated) — the same quiz after a forced L1 consolidation; the delta vs compacted recall measures consolidation drift.
Reduction — token size of the compacted view vs the full transcript.
Chunk latency — wall time per chunk summarization call.
Structure — compacted views are well-formed (no mid-tail system messages, no orphan tool results, a user message present).
Cache — recomputing hashes over the same input reuses every cached chunk with zero further model calls.

Fixtures

Caveats

Results are comparable only within one model; every result file is stamped with the model id and settings.
Local model output varies between quant builds and runtime versions even at temperature 0 — treat small recall deltas as noise, structural/cache columns as hard pass/fail.
bench/ is excluded from the plugin build (tsconfig.json includes src only) and is not part of the published artifact's runtime.

context-compressor

context-compressor

context-compressor

context-compressor

Benchmarks

Running

Metrics

Fixtures

Caveats

Benchmarks

Running

Metrics

Fixtures

Caveats