vibe-lm
Public
Project Files
benchmark / live-results / qwen3-coder-30b-mega-01.md
Model: qwen/qwen3-coder-30b (auto-loaded context: 262144) Chat: "Unnamed Chat" (LM Studio, vibeLM plugin, real tools attached)
~/Desktop/sandbox/weather-cli and cleared runtime-state.json before starting (same reasons as prior model runs).Prompt: "yo can you set me up a quick node cli in ~/Desktop/sandbox/weather-cli that just takes a city and prints the current weather, pull from some free api"
Model wandered early: called set_workspace on the (not-yet-existing) weather-cli path, misread the resulting error as a "permission issue," and tried moving the workspace to the user's home directory, then to Desktop, before correctly landing back on ~/Desktop/sandbox/weather-cli. Self-corrected without user intervention. Then: bash_terminal (mkdir/npm init), bash_terminal (npm install axios — real network install, ~40s), list_files, write_file (index.js), write_file (README.md), read_file + write_file (package.json bin entry), write_file (install.sh) + bash_terminal (chmod), write_file (a demo/test script) + bash_terminal, write_file (SUMMARY.md), list_files.
Verified on disk: real files exist (index.js, README.md, SUMMARY.md, install.sh, package.json, full node_modules/ from a real npm install axios). But index.js uses OpenWeatherMap with a hardcoded placeholder: const API_KEY = 'YOUR_API_KEY'; — the model's own code comments instruct the user to go sign up for a key and hand-edit the source before it will work.
Ran it myself: node index.js London → ❌ Invalid API key. Please set a valid API key from OpenWeatherMap. Does not work out of the box.
T1: FAIL — heavy real tool use and a well-organized project (README, install script, tests, summary), but the core deliverable doesn't satisfy "just takes a city and prints the current weather" — it requires manual signup + source edit before it does anything. wttr.in (or any no-key API) was available and used successfully by other models in this same benchmark; this model chose a keyed API without flagging that as a blocker to the user or asking for a key.
Not pursuing T2-T9 since the foundation itself doesn't meet the T1 pass condition (a working end-to-end tool) — further turns would be building on a non-functional base rather than producing new benchmark signal.