{
  "date": "2026-08-26",
  "modelId": "qwen/qwen3.8-27b",
  "driver": "installed",
  "contextLength": 16384,
  "results": [
    {
      "scenario": "S1",
      "title": "inspect → read → edit → verify",
      "driver": "installed",
      "success": true,
      "checks": [
        {
          "name": "bug fixed in src/math.js",
          "ok": true
        },
        {
          "name": "node --test passes",
          "ok": true
        },
        {
          "name": "test file untouched",
          "ok": true
        },
        {
          "name": "at least one applied transaction",
          "ok": true
        },
        {
          "name": "inspected before editing",
          "ok": true
        },
        {
          "name": "ran node --test through workspace_command",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 12,
        "toolCallsByName": {
          "workspace_notes": 2,
          "workspace_inspect": 4,
          "workspace_tasks": 4,
          "workspace_edit": 1,
          "workspace_command": 1
        },
        "rounds": 9,
        "invalidToolRequests": 0,
        "envelopeViolations": 0,
        "wallMs": 26558,
        "contextOverflow": false,
        "promptTokens": 65008,
        "predictedTokens": 2074
      },
      "contextLength": 16384
    },
    {
      "scenario": "S3",
      "title": "plan mode: propose, /accept, execute",
      "driver": "installed",
      "success": true,
      "checks": [
        {
          "name": "no mutation before approval",
          "ok": true
        },
        {
          "name": "plan proposed in phase 1",
          "ok": true
        },
        {
          "name": "exactly one plan approved",
          "ok": true
        },
        {
          "name": "bug fixed in src/math.js",
          "ok": true
        },
        {
          "name": "node --test passes",
          "ok": true
        },
        {
          "name": "test file untouched",
          "ok": true
        },
        {
          "name": "plan proposals after /accept: 0",
          "ok": true
        },
        {
          "name": "plan final status: completed",
          "ok": true
        },
        {
          "name": "workspace_plan complete called: true",
          "ok": true
        },
        {
          "name": "APPROVAL_REQUIRED before approval: 0",
          "ok": true
        },
        {
          "name": "stray planned transactions: 0",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 10,
        "toolCallsByName": {
          "workspace_notes": 2,
          "workspace_inspect": 4,
          "workspace_plan": 2,
          "workspace_edit": 1,
          "workspace_command": 1
        },
        "rounds": 8,
        "invalidToolRequests": 0,
        "envelopeViolations": 0,
        "wallMs": 24473,
        "contextOverflow": false,
        "promptTokens": 60406,
        "predictedTokens": 1895
      },
      "contextLength": 16384
    },
    {
      "scenario": "S9",
      "title": "full project from an empty directory",
      "driver": "installed",
      "success": false,
      "checks": [
        {
          "name": "package.json parses with scripts.test running node --test",
          "ok": true
        },
        {
          "name": "at least one file under src/",
          "ok": true
        },
        {
          "name": "at least one test file under tests/",
          "ok": true
        },
        {
          "name": "node --test exits 0 with at least three ok lines",
          "ok": true
        },
        {
          "name": "score: 4/4",
          "ok": true
        },
        {
          "name": "files created: package.json, src/wordcount.js, tests/wordcount.test.js",
          "ok": true
        },
        {
          "name": "ok lines: 3",
          "ok": true
        },
        {
          "name": "node --test exit: 0",
          "ok": true
        },
        {
          "name": "package private: true",
          "ok": true
        },
        {
          "name": "applied transactions: 1",
          "ok": true
        },
        {
          "name": "model ran node --test: yes (1)",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 12,
        "toolCallsByName": {
          "workspace_notes": 1,
          "workspace_inspect": 1,
          "workspace_tasks": 6,
          "workspace_edit": 1,
          "workspace_command": 2,
          "workspace_vcs": 1
        },
        "rounds": 9,
        "invalidToolRequests": 1,
        "envelopeViolations": 0,
        "wallMs": 44030,
        "contextOverflow": true,
        "promptTokens": 73281,
        "predictedTokens": 3812
      },
      "contextLength": 16384,
      "error": "context overflow: prompt 17368 > 16384"
    }
  ],
  "note": "Config verification: the installed session's tool list proves each scenario's `pluginConfig` reached the plugin —\n8 tools with no `workspace_plan` in S1 and S9 (permissionMode 2, auto) and 9 tools including it in S3 (permissionMode 1, plan).\n\nS9 ended on a hard prompt overflow: once the accumulated prompt exceeds the window LM Studio's engine returns 400,\nwhich `contextOverflowPolicy: \"stopAtLimit\"` cannot prevent (it stops generation, it cannot shrink an already-oversized prompt).\nAll 11 checks still passed — wordcount was built and `node --test`-green on disk before the overflow.\nTwo in-process control runs at the same context length reproduced the same hard overflow (prompt 18337 and 16629 > 16384, 4/4 scored both times),\nso this is S9 sitting at the edge of a 16384-token window for this model, not something the installed/SDK boundary introduces;\nthe committed in-process S9 row that passed with a warning was a sample that happened to land under the ceiling.\nS1 and S3 match the in-process rows check for check, and S3 matches its tool-call profile exactly."
}

{
  "date": "2026-08-26",
  "modelId": "qwen/qwen3.8-27b",
  "driver": "installed",
  "contextLength": 16384,
  "results": [
    {
      "scenario": "S1",
      "title": "inspect → read → edit → verify",
      "driver": "installed",
      "success": true,
      "checks": [
        {
          "name": "bug fixed in src/math.js",
          "ok": true
        },
        {
          "name": "node --test passes",
          "ok": true
        },
        {
          "name": "test file untouched",
          "ok": true
        },
        {
          "name": "at least one applied transaction",
          "ok": true
        },
        {
          "name": "inspected before editing",
          "ok": true
        },
        {
          "name": "ran node --test through workspace_command",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 12,
        "toolCallsByName": {
          "workspace_notes": 2,
          "workspace_inspect": 4,
          "workspace_tasks": 4,
          "workspace_edit": 1,
          "workspace_command": 1
        },
        "rounds": 9,
        "invalidToolRequests": 0,
        "envelopeViolations": 0,
        "wallMs": 26558,
        "contextOverflow": false,
        "promptTokens": 65008,
        "predictedTokens": 2074
      },
      "contextLength": 16384
    },
    {
      "scenario": "S3",
      "title": "plan mode: propose, /accept, execute",
      "driver": "installed",
      "success": true,
      "checks": [
        {
          "name": "no mutation before approval",
          "ok": true
        },
        {
          "name": "plan proposed in phase 1",
          "ok": true
        },
        {
          "name": "exactly one plan approved",
          "ok": true
        },
        {
          "name": "bug fixed in src/math.js",
          "ok": true
        },
        {
          "name": "node --test passes",
          "ok": true
        },
        {
          "name": "test file untouched",
          "ok": true
        },
        {
          "name": "plan proposals after /accept: 0",
          "ok": true
        },
        {
          "name": "plan final status: completed",
          "ok": true
        },
        {
          "name": "workspace_plan complete called: true",
          "ok": true
        },
        {
          "name": "APPROVAL_REQUIRED before approval: 0",
          "ok": true
        },
        {
          "name": "stray planned transactions: 0",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 10,
        "toolCallsByName": {
          "workspace_notes": 2,
          "workspace_inspect": 4,
          "workspace_plan": 2,
          "workspace_edit": 1,
          "workspace_command": 1
        },
        "rounds": 8,
        "invalidToolRequests": 0,
        "envelopeViolations": 0,
        "wallMs": 24473,
        "contextOverflow": false,
        "promptTokens": 60406,
        "predictedTokens": 1895
      },
      "contextLength": 16384
    },
    {
      "scenario": "S9",
      "title": "full project from an empty directory",
      "driver": "installed",
      "success": false,
      "checks": [
        {
          "name": "package.json parses with scripts.test running node --test",
          "ok": true
        },
        {
          "name": "at least one file under src/",
          "ok": true
        },
        {
          "name": "at least one test file under tests/",
          "ok": true
        },
        {
          "name": "node --test exits 0 with at least three ok lines",
          "ok": true
        },
        {
          "name": "score: 4/4",
          "ok": true
        },
        {
          "name": "files created: package.json, src/wordcount.js, tests/wordcount.test.js",
          "ok": true
        },
        {
          "name": "ok lines: 3",
          "ok": true
        },
        {
          "name": "node --test exit: 0",
          "ok": true
        },
        {
          "name": "package private: true",
          "ok": true
        },
        {
          "name": "applied transactions: 1",
          "ok": true
        },
        {
          "name": "model ran node --test: yes (1)",
          "ok": true
        }
      ],
      "metrics": {
        "toolCalls": 12,
        "toolCallsByName": {
          "workspace_notes": 1,
          "workspace_inspect": 1,
          "workspace_tasks": 6,
          "workspace_edit": 1,
          "workspace_command": 2,
          "workspace_vcs": 1
        },
        "rounds": 9,
        "invalidToolRequests": 1,
        "envelopeViolations": 0,
        "wallMs": 44030,
        "contextOverflow": true,
        "promptTokens": 73281,
        "predictedTokens": 3812
      },
      "contextLength": 16384,
      "error": "context overflow: prompt 17368 > 16384"
    }
  ],
  "note": "Config verification: the installed session's tool list proves each scenario's `pluginConfig` reached the plugin —\n8 tools with no `workspace_plan` in S1 and S9 (permissionMode 2, auto) and 9 tools including it in S3 (permissionMode 1, plan).\n\nS9 ended on a hard prompt overflow: once the accumulated prompt exceeds the window LM Studio's engine returns 400,\nwhich `contextOverflowPolicy: \"stopAtLimit\"` cannot prevent (it stops generation, it cannot shrink an already-oversized prompt).\nAll 11 checks still passed — wordcount was built and `node --test`-green on disk before the overflow.\nTwo in-process control runs at the same context length reproduced the same hard overflow (prompt 18337 and 16629 > 16384, 4/4 scored both times),\nso this is S9 sitting at the edge of a 16384-token window for this model, not something the installed/SDK boundary introduces;\nthe committed in-process S9 row that passed with a warning was a sample that happened to land under the ceiling.\nS1 and S3 match the in-process rows check for check, and S3 matches its tool-call profile exactly."
}