vibeLM live benchmark — aggregate results (in progress)

Run started 2026-07-11 ~12:45, live in the real LM Studio desktop app against the real vibeLM plugin with real tools attached. No mocks, no headless scripts.

Pre-run anomalies

• benchmark/live-results/README.md (named as the fixture-list source of truth in mega-prompts.md) does not exist. Proceeded with mega-prompts.md as authoritative, as instructed.
• config.json's preferredModel was qwen/qwen3-4b, which the user's hard rule forbids. Left untouched at user's instruction (model selection was done manually via the LM Studio GUI dropdown each run, never via this config field).
• Model context length is not held constant across models: some auto-load at 4096 (nemotron), others at ~200K-262K (the qwen/glm models). This means "same budget, fair comparison" (the stated intent of STANDARD_LOAD_CONFIG in models.mjs) isn't actually happening in this GUI-driven run, since load parameters were left on auto per user instruction ("do not touch model settings"). Worth knowing when reading pass/fail below — nemotron's failure mode is partly a context-budget artifact that the higher-context models don't share.
• mega-10 (ssh into a real dev box) is blocked: no ssh target/credentials available. Not attempted.

Real plugin bugs found (both root-caused, not fixed — fixes are out of scope for a live benchmark run)

Bug 1 — stale plan bleeds into brand-new chats

src/toolsProvider.ts:790-801. On a history-fingerprint mismatch (i.e. this really is a new/different conversation), the bootstrap logic still unconditionally re-asserts the last persisted plan from runtime-state.json. That behavior is only correct for the "process restarted mid-conversation" case; it can't distinguish that from "this is an actually new chat." First hit on the very first live turn of the whole benchmark: a fresh chat asking for a weather CLI resurrected an old, unrelated "build a REST API backend" plan and started writing src/types.ts for a Task API. Worked around per-model-run by deleting runtime-state.json between models (user-approved). Backup of the first stale file: live-results/_stale-runtime-state-backup.json.

Bug 2 — no forcing function for new instructions once a plan reads "done"

src/toolsProvider.ts:1986-2012 (the amend tool). Its safety gate only blocks premature completion when plan.steps still has pending entries. Nothing turns a new user instruction, arriving after the existing plan is 100% "done," into a new plan step. Result: once turn 1 of a scenario completes and its plan is fully marked done, the model can (and, reproducibly, does) just reply in plain text to any follow-up ask and stop — no tool call, no amend call, nothing blocking it. Reproduced identically across two unrelated model families (qwen3.6-35b-a3b and glm-4.7-flash) on the exact same mega-01 turn 2 ("cache it for an hour") — both models ignored the instruction and re-stated turn 1's already-done work instead. session-log.jsonl confirms amend was never even called in either case.

Bug 3 — file tools silently mishandle a literal ~ in the path argument

src/toolsProvider.ts:540-547 (sandboxPath). All file tools (write_file, read_file, append_file, rename_file, search_files, delete_file) share this helper, which does nothing but path.resolve(workspace, requestedPath) plus a containment check. Node's resolve never expands ~ — a model-supplied path like ~/Desktop/sandbox/weather-cli/index.js gets joined verbatim onto the workspace root, producing a literal ~-named subdirectory (e.g. /Users/.../sandbox/~/Desktop/sandbox/weather-cli/index.js) instead of writing to the intended location. The containment check still passes (it's nested under the workspace), so the tool reports "action":"written" success while silently writing to the wrong place. bash_terminal is unaffected (real shell expands ~ natively). Confirmed cross-model: found stray package.json/index.js at the same garbled path left behind by two different models (qwen3.6-27b and gemma-4-e4b GGUF) in separate sessions — this has been silently corrupting file locations throughout the benchmark whenever a model uses a ~-prefixed path instead of a workspace-relative one. Suggested minimal fix: reject or expand (os.homedir()) any whose first segment is literally , before calling . Not fixed yet — flagging for a follow-up branch alongside Bug 1/Bug 2.

Anomaly — UI abort race

Sending a new turn before the previous turn's trailing tool-call sequence had fully finished triggered a real LM Studio host-level error ("Failed to send message — prediction loop handler did not abort in time"). Not vibeLM-specific (references lmstudio/default-prediction-loop-handler), but worth knowing: always wait for full turn completion before sending the next one.

mega-01 results by model (weather-cli, 9-turn scenario — driven only as far as each model's failure was conclusively reproduced, per instructions not to grind out turns that only re-confirm an already-diagnosed failure)

Model	Context (auto)	T1 (build)	T2 (add caching)	Verdict	Root cause
nvidia/nemotron-3-nano-omni	4096	PASS (after 1 retry; 2 transient tool-call-gen errors self-recovered)	FAIL	FAIL	Context budget exhausted — model lost track of completed work, stopped calling tools, only produced text status reports
qwen/qwen3.6-35b-a3b	262144	PASS (clean, fast, self-tested via bash_terminal)	FAIL (confirmed twice — re-asked explicitly, still ignored)	FAIL	Bug 2 (no forcing function for new instructions once plan is "done")
qwen/qwen3-coder-30b	262144	FAIL — real files written, real npm install, but used OpenWeatherMap with a hardcoded placeholder API key (YOUR_API_KEY); doesn't run without manual signup + source edit	not reached	FAIL	Didn't satisfy "free api" / "just works" intent; also wandered early on set_workspace before self-correcting
zai-org/glm-4.7-flash	~202752	PASS (clean, wttr.in, verified working)	FAIL	FAIL	Bug 2
qwen/qwen3.6-27b	262144	PASS — best self-correction seen: caught and fixed its own wrong file path unprompted, real self-testing incl. error handling	FAIL	FAIL	Bug 2 fix deployed but didn't help here — this model never called create_plan with real tracked steps in T1 (ad-hoc tool calls only), so there was no "plan reads done" state for the fix to re-open. The fix (PR #39) only closes the gap for models that actually use plan/step tooling; models that skip it entirely need a broader follow-up fix.
google/gemma-4-26b-a4b-qat	n/a	BLOCKED	not reached	BLOCKED	Real MLX/LM Studio framework limitation: mlx-vlm batched vision path does not support KV cache quantization yet. Not retried with different load settings (forbidden).

Full detail per model: nemotron-3-nano-omni-mega-01.md, qwen3.6-35b-a3b-mega-01.md, qwen3-coder-30b-mega-01.md, glm-4.7-flash-mega-01.md, qwen3.6-27b-mega-01.md, gemma-4-26b-a4b-mega-01.md, gpt-oss-20b-mega-01.md, glm-4.6v-flash-mega-01.md, gemma-4-e4b-mlx-mega-01.md, gemma-4-e4b-gguf-mega-01.md, qwen3.5-9b-mega-01.md, granite-4-h-tiny-mega-01.md.

PR #39 status: both fixes built/tested/deployed locally mid-run (build.sh reinstall). Verified against qwen3.6-27b — deployed correctly (no crash, no regression) but exposed a real coverage gap: the fix only engages for models that use create_plan/update_plan_step. Not yet verified against a model that uses real plan tooling for T2 (glm-4.7-flash and qwen3.6-35b-a3b were both tested pre-fix).

mega-01 across all 12 local models: COMPLETE

Every model in the authoritative local list has now been attempted (or conclusively blocked) on mega-01. Final tally: 0 full passes (every model that produced output failed by T2 at the latest; several failed T1 outright), 2 BLOCKED (real MLX/LM Studio framework limitation, unrelated to vibeLM), 1 reproducible empty-completion model failure (granite-4-h-tiny). 3 real vibeLM plugin bugs found and root-caused (Bug 1, Bug 2 partially fixed via PR #39, Bug 3 newly found and not yet fixed).

Emerging pattern

Of the 10 models that actually produced output, every single one that reached T2 failed T2 ("cache it for an hour" / equivalent follow-up instruction after apparent completion), and several failed even earlier at T1 (wrong/misplaced files, no free API used, empty completion, or — in glm-4.6v-flash's case — total tool-use hallucination with zero real tool calls). T2 failure breaks into two mechanisms: (a) Bug 2 proper — plan reads "done", new instruction gets swallowed (qwen3.6-35b-a3b, glm-4.7-flash, both pre-fix) — and (b) models that never use plan/step tooling at all, where PR #39's fix structurally cannot apply (qwen3.6-27b, gpt-oss-20b, both post-fix). A broader fix — detecting "does this new message ask for something not reflected in the last assistant turn," independent of plan state — looks necessary, not optional.

A second cross-cutting pattern: models frequently over-claim success. gpt-oss-20b narrated file edits that were never executed; gemma-4-e4b (MLX) claimed "Core Logic Written" for a file that didn't exist at the claimed path; gemma-4-e4b (GGUF) marked a plan step "done" based on a write_file call that silently wrote to a garbled ~-prefixed path (Bug 3). In every case, only checking real disk state (not the model's chat response) caught this — reinforcing that self-report verification is not optional for this kind of benchmark.

Not yet run

• mega-02 through mega-09, all models.
• mega-10: blocked (no ssh target).

Time/scope reality check

Each live turn takes 20s-3min of real generation time depending on model size/verbosity; a 9-turn scenario is realistically 15-30 minutes of real, continuously-driven GUI time per model when things go well, more when a model needs a retry. The full authoritative matrix (9 in-scope scenarios × 12 local models) is many hours of continuous live driving, not something completable in one sitting. Given two real, cross-model plugin bugs have already been found and root-caused (arguably the most valuable finding a benchmark like this can produce), pausing here to report rather than mechanically re-confirming Bug 2 on 8 more models is the efficient call — happy to keep grinding through the rest of the matrix on your signal.