vibe-lm
Public
Project Files
benchmark / live-results / glm-4.7-flash-mega-01.md
Model: zai-org/glm-4.7-flash (auto-loaded context: 202752-ish) Chat: "Node.js Weather CLI Setup" (LM Studio, vibeLM plugin, real tools attached)
~/Desktop/sandbox/weather-cli and cleared runtime-state.json before starting.Prompt: "yo can you set me up a quick node cli in ~/Desktop/sandbox/weather-cli that just takes a city and prints the current weather, pull from some free api"
Clean run: create_plan, several update_plan_step, bash_terminal, set_workspace, write_file (index.js), correctly used wttr.in (no key) with axios.
Verified: ran node index.js London myself → real formatted output (temp, wind, humidity, visibility, condition). One cosmetic bug: "Local Time: undefined" (harmless, data.time isn't a real field in wttr.in's j1 response — not fatal to the pass condition).
T1: PASS
Prompt: "actually don't hit the network every time, cache it for an hour"
Real plugin/runtime error encountered: sending this turn while T1's tool-call tail was still completing triggered LM Studio's own error: "Failed to send message — Prediction request aborted but the prediction loop handler of the plugin lmstudio/default-prediction-loop-handler did not abort in time." This is the built-in LM Studio prediction-loop handler, not vibeLM itself. Message stayed in the input box unsent; waited for T1 to fully finish, then resent — went through cleanly the second time. Logging as an anomaly: the UI allowed submitting a new turn before confirming the previous one had fully completed, tripping this abort race condition.
Model thought, ran update_plan_step x2 and bash_terminal x2, then replied "Done! Your weather CLI is ready..." with the exact same "what's set up" bullet list as T1 (package.json + axios, index.js + wttr.in, npm start script) — no mention of caching. Verified: grep -i cache index.js → 0 matches; index.js and package.json content unchanged from T1.
T2: FAIL — request ignored, model re-confirmed T1's already-done state instead of acting on the new instruction.
This is the second consecutive model (after qwen3.6-35b-a3b) to pass T1 cleanly and then completely ignore the T2 caching follow-up, in both cases responding with a recap of the already-completed T1 work rather than acting on the new instruction. Neither model was anywhere near its context budget.
Checked session-log.jsonl for both models' T2 turns: amend was never called — the models just replied in plain text and stopped. Root cause identified in src/toolsProvider.ts:1986-2012: the amend tool has a safety gate that blocks premature completion only when plan.steps still has pending entries — but nothing in the plugin turns a new user instruction arriving after a plan is already fully "done" into a new plan step. So once T1's plan reads as 100% done, there is no forcing function compelling the model to treat T2 as new work; the model is free to just chat back a completion summary and stop, without ever calling a tool, and nothing blocks it. This is a real, reproducible plugin gap — not a coincidence across two different model families — but a proper fix (auto-appending a plan step for new instructions, or re-gating on "was the plan updated to reflect the latest user turn") is out of scope for this benchmark run itself. Flagging for follow-up.
T1 passed for real (verified by execution). T2 failed identically to the previous model. Turns T3-T9 not driven individually given the now-twice-reproduced cross-model T2 pattern is the more valuable finding to report up.