vibe-lm
Public
Project Files
benchmark / live-results / qwen3.6-27b-mega-01.md
Model: qwen/qwen3.6-27b (auto-loaded context: 262144)
Chat: "Unnamed Chat" (LM Studio, vibeLM plugin, real tools attached)
Note: this run happened after PR #39's two bug fixes were built and deployed locally (./build.sh) mid-session.
~/Desktop/sandbox/weather-cli and cleared runtime-state.json before starting../build.sh mid-session (to deploy the PR #39 fixes) broke the live WebSocket connection for the chat that was already open when the reinstall happened. Every subsequent tool call in that chat failed with WebSocket closed by the client (create_plan, set_workspace, bash_terminal, get_config — 8 consecutive real tool-execution failures, not generation failures). Fixed by toggling the vibe-lm plugin off/on in the integrations panel and starting a fresh chat. Retried T1 cleanly from there. This is a caveat for future benchmark runs: never reinstall the plugin while a chat is actively mid-turn.Prompt: "yo can you set me up a quick node cli in ~/Desktop/sandbox/weather-cli that just takes a city and prints the current weather, pull from some free api"
Very thorough run: set_workspace, 4x bash_terminal (checked node version, confirmed v22 native fetch), chose Open-Meteo (free, no key). First write_file landed at the wrong path (~/Desktop/sandbox/index.js instead of weather-cli/index.js) — model caught its own mistake unprompted ("Hmm, the workspace is set to ~/Desktop/sandbox, not weather-cli... that's wrong"), used get_config to confirm the real workspace, delete_file'd the stray file, and rewrote everything at the correct path. Then updated package.json with a proper bin entry, and actually ran the CLI itself via bash_terminal before replying, including testing error handling for a missing city.
Verified independently: find shows exactly weather-cli/index.js and weather-cli/package.json, no stray file anywhere. Ran node index.js London myself → real output (📍 London, United Kingdom / ☀️ 28.9°C / 💨 Wind: 13 km/h / 🕐 timestamp). Works cleanly.
T1: PASS — best self-correction behavior seen in the whole benchmark so far (caught and fixed its own path mistake without being told).
Prompt: "actually don't hit the network every time, cache it for an hour"
Checked runtime-state.json after this turn: plan.steps was [] (empty) — this model never called create_plan with real tracked steps during T1, it just executed tool calls ad-hoc. Only the auto-seeded goal-only plan existed. The PR #39 fix for stale-plan-completion only fires when plan.steps.length > 0 (deliberately, to avoid misfiring on the goal-only auto-plan) — so it correctly did not engage here, because there was no tracked "done" state to reopen.
Model's actual behavior: called bash_terminal twice to re-verify the existing CLI still works ("Looks like this was already wrapped up in the previous session... Let me just verify the files are still intact" → "All good — your weather CLI is still running fine. London's at 29°C"), then offered an unrelated menu (--fahrenheit flag, 5-day forecast, install globally) — never touched caching. Verified: grep -i cache index.js → 0 matches, file unchanged from T1.
T2: FAIL — same end result as the original Bug 2 repro (qwen3.6-35b-a3b, glm-4.7-flash), but a different root cause: this model doesn't use the plan/step-tracking tools at all, so there's no "plan reads done" state for the new fix to catch. This suggests the PR #39 fix only closes the gap for models that actually engage create_plan/update_plan_step — models that skip plan tooling entirely need a broader fix (e.g., a general "does this new message ask for anything not yet reflected in the last response" check, independent of plan state). Flagging as a known limitation of the current fix, not a regression.
T1 passed cleanly with the best self-correction quality observed in this benchmark. T2 failed for a related but mechanically distinct reason from the original bug repro — worth a follow-up fix, but out of scope to chase further in this run.