vibe-lm
Public
Project Files
benchmark / live-results / gemma-4-e4b-mlx-mega-01.md
Model: google/gemma-4-e4b (MLX, auto-loaded context: 131072) Chat: "Unnamed Chat" (LM Studio, vibeLM plugin, real tools attached)
~/Desktop/sandbox/weather-cli and cleared runtime-state.json before starting.gemma-4-26b-a4b-qat variant, which was blocked by an mlx-vlm KV-cache-quantization error) — the smaller E4B model has no such incompatibility.computer-use screenshot capture returned nil for ~2 minutes mid-session (during the previous model's run, glm-4.6v-flash) and recovered on its own; also left a stray keystroke ("5") in the chat input box that was caught and cleared before it could be sent. Noted for the record, no impact on this model's actual run.Prompt: "yo can you set me up a quick node cli in ~/Desktop/sandbox/weather-cli that just takes a city and prints the current weather, pull from some free api"
Real tool-use sequence (verified via session-log.jsonl, session 881013cd-1a59-42fe-b096-996b3bc9323a):
Net effect: index.js and node_modules/ were written to ~/Desktop/sandbox/ (the workspace root), not ~/Desktop/sandbox/weather-cli/. Only weather-cli/package.json (from the npm init -y in the combined mkdir && cd && npm init command) ended up in the right place. Verified independently: find ~/Desktop/sandbox/weather-cli shows only package.json; find ~/Desktop/sandbox -maxdepth 2 -name index.js finds it at the workspace root instead.
On top of the misplaced files, the generated index.js hardcodes const API_KEY = "YOUR_SECRET_API_KEY"; against OpenWeatherMap and refuses to run without the user manually signing up and pasting in a real key — the model's own closing summary explicitly listed "Get an API key" and "Insert Key" as required next steps for the user. This is the same "doesn't actually satisfy 'free API' " failure mode seen with qwen3-coder-30b and glm-4.6v-flash.
T1: FAIL — two independent, compounding failures: (1) the model's own multi-step tool sequence didn't account for bash_terminal calls being stateless (fresh shell per call) and used a bare relative path for write_file, so real work landed in the wrong directory; (2) even where it landed, the CLI requires a manual paid/signup API key rather than a genuinely free no-key API.
Notable because the model's own chat summary ("✅ Project Setup Steps Completed... Core Logic Written: index.js contains the complete CLI application structure") was not true — verified against real disk state, index.js didn't exist at the path it claimed. Not a vibeLM plugin bug (the model's own path handling in write_file's path argument was the bare string "index.js", and the model chose to run npm install in a follow-up call without re-establishing cd — both are model behavior, not documented false tool-success
results by the plugin), but a clear self-report/reality mismatch worth flagging as a recurring pattern across this benchmark.
web_search × 2 — looked for a free weather API.bash_terminal: mkdir -p ~/Desktop/sandbox/weather-cli && cd ~/Desktop/sandbox/weather-cli && npm init -y — succeeded.bash_terminal: npm install axios — succeeded (142 packages installed) but run as a separate tool call with no cd, so it executed from the workspace root, not the weather-cli subdirectory the previous step created.write_file with path: "index.js" (a bare relative path, not "weather-cli/index.js") — also resolved against the workspace root.