vibe-lm
Public
Project Files
benchmark / results / run-remote.jsonl
{"label":"remote:qwen3.6-35b-a3b","modelKey":"qwen/qwen3.6-35b-a3b","architecture":"qwen35moe","promptId":"arith-1","kind":"short","passed":true,"wallMs":22472,"finalAnswer":"Thinking Process:\n\n1. **Identify the user's request**: The user wants to calculate the product of 17 and 24.\n2. **Constraint**: The user explicitly requested \"Reply with only the number, nothing else.\"\n3. **Perform the calculation**:\n * $17 \\times 24$\n * Method 1: $(10 + 7) \\times 24 = 240 + (7 \\times 24)$\n * $7 \\times 24 = 7 \\times (20 + 4) = 140 + 28 = 168$\n * $240 + 168 = 408$\n * Method 2: $17 \\times (25 - 1) = (17 \\times 25) - 17$\n * $17 \\times 25 = 17 \\times","timeToFirstTokenSec":0.483,"tokensPerSecond":21.817475907517043,"promptTokensCount":29,"predictedTokensCount":479,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-35b-a3b","modelKey":"qwen/qwen3.6-35b-a3b","architecture":"qwen35moe","promptId":"arith-2","kind":"short","passed":true,"wallMs":20169,"finalAnswer":"Here's a thinking process:\n\n1. **Analyze User Input:**\n - Departure time: 14:05\n - Arrival time: 16:40\n - Same day\n - Question: How many minutes did the trip take?\n - Constraint: Reply with only the number.\n\n2. **Calculate Time Difference:**\n - From 14:05 to 16:05 is 2 hours.\n - From 16:05 to 16:40 is 35 minutes.\n - Total hours = 2\n - Total minutes = 2 * 60 + 35 = 120 + 35 = 155 minutes.\n - Alternatively: \n - Minutes from 14:05 to 14:60 (15:00) = 55 minutes\n - Minut","timeToFirstTokenSec":0.53,"tokensPerSecond":21.792683541331538,"promptTokensCount":47,"predictedTokensCount":426,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-35b-a3b","modelKey":"qwen/qwen3.6-35b-a3b","architecture":"qwen35moe","promptId":"logic-1","kind":"short","passed":false,"wallMs":9202,"finalAnswer":"Here's a thinking process:\n\n1. **Analyze the input:**\n - Premise 1: Alice is taller than Bob. (A > B)\n - Premise 2: Bob is taller than Carol. (B > C)\n - Question: Is Carol taller than Alice? (C > A?)\n - Constraint: Reply with only 'yes' or 'no'.\n\n2. **Logical deduction:**\n - From A > B and B > C, by transitivity, A > C (Alice is taller than Carol).\n - Therefore, Carol is NOT taller than Alice.\n - Answer: no.\n\n3. **Check constraint:**\n - Must reply with only 'yes' or 'no'.\n -","timeToFirstTokenSec":0.491,"tokensPerSecond":21.860914385372297,"promptTokensCount":38,"predictedTokensCount":189,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-35b-a3b","modelKey":"qwen/qwen3.6-35b-a3b","architecture":"qwen35moe","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":18712,"finalAnswer":"Here's a thinking process:\n\n1. **Analyze User Input:**\n - The user provides a series of paragraphs (0 to 24) that are mostly identical, reporting routine logistics activities at various sites.\n - One paragraph (Paragraph 13) contains an extra line: \"The reference code for the Halden vault is QK-77219-ALPHA.\"\n - The question asks: \"what is the reference code for the Halden vault? Reply with only the code.\"\n\n2. **Identify Key Information:**\n - Scan the text for \"Halden vault\" or \"referen","timeToFirstTokenSec":3.909,"tokensPerSecond":21.650797877742246,"promptTokensCount":989,"predictedTokensCount":307,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-35b-a3b","modelKey":"qwen/qwen3.6-35b-a3b","architecture":"qwen35moe","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":18037,"rounds":3,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}
{"label":"remote:nemotron-3-nano","modelKey":"nvidia/nemotron-3-nano-omni","architecture":"nemotron_h_moe","promptId":"arith-1","kind":"short","passed":true,"wallMs":4229,"finalAnswer":"The user asks: \"What is 17 * 24? Reply with only the number, nothing else.\" So answer should be just the product: 17*24 = 408. Provide only \"408\". No extra text.\n</think>\n408","timeToFirstTokenSec":1.131,"tokensPerSecond":20.320978178600743,"promptTokensCount":35,"predictedTokensCount":61,"stopReason":"eosFound"}
{"label":"remote:nemotron-3-nano","modelKey":"nvidia/nemotron-3-nano-omni","architecture":"nemotron_h_moe","promptId":"arith-2","kind":"short","passed":true,"wallMs":5360,"finalAnswer":"We need compute duration from 14:05 to 16:40 same day. From 14:05 to 16:05 is 2 hours = 120 minutes. Then add 35 minutes (from 16:05 to 16:40) = 155 minutes. So answer 155.\n</think>\n155","timeToFirstTokenSec":0.958,"tokensPerSecond":20.49936451969989,"promptTokensCount":53,"predictedTokensCount":88,"stopReason":"eosFound"}
{"label":"remote:nemotron-3-nano","modelKey":"nvidia/nemotron-3-nano-omni","architecture":"nemotron_h_moe","promptId":"logic-1","kind":"short","passed":false,"wallMs":3355,"finalAnswer":"We need to answer yes/no whether Carol is taller than Alice given premises: Alice > Bob, Bob > Carol => transitive: Alice > Bob > Carol, so Carol is shorter than Alice, thus not taller. So answer \"no\".\n</think>\nno","timeToFirstTokenSec":0.862,"tokensPerSecond":21.123111707712464,"promptTokensCount":44,"predictedTokensCount":51,"stopReason":"eosFound"}
{"label":"remote:nemotron-3-nano","modelKey":"nvidia/nemotron-3-nano-omni","architecture":"nemotron_h_moe","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":15776,"finalAnswer":"We need to read carefully. The question: \"what is the reference code for the Halden vault?\" In context, there is a line:\n\n\"Paragraph 0: ...\"\n\nScrolling through many paragraphs. At some point there's a mention of reference code for the Halden vault.\n\nI see: \"The reference code for the Halden vault is QK-77219-ALPHA.\" That appears after paragraph 13 (maybe as separate line). Indeed, after paragraph 13, there is:\n\n\"The reference code for the Halden vault is QK-77219-ALPHA.\"\n\nThus answer: QK-77219-A","timeToFirstTokenSec":5.557,"tokensPerSecond":19.75272565280525,"promptTokensCount":1016,"predictedTokensCount":199,"stopReason":"eosFound"}
{"label":"remote:nemotron-3-nano","modelKey":"nvidia/nemotron-3-nano-omni","architecture":"nemotron_h_moe","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"error":"\n\nā Error āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\nā ā\nā Context size has been exceeded. ā\nā ā\nā ā\nā </> STACK TRACE ā\nā ā\nā at runAgenticPrompt (file:///Users/drunkktoys/Desktop/vibeLM/benchmark/ ā\nā run-remote.mjs:108:21) ā\nā at main (file:///Users/drunkktoys/Desktop/vibeLM/benchmark/run-remote.m ā\nā js:158:28) ā\nā at process.processTicksAndRejections ā\nā (node:internal/process/task_queues:103:5) ā\nā ā\nāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\n"}
{"label":"remote:qwen3-coder-30b","modelKey":"qwen/qwen3-coder-30b","architecture":"qwen3moe","promptId":"arith-1","kind":"short","passed":true,"wallMs":773,"finalAnswer":"408","timeToFirstTokenSec":0.404,"tokensPerSecond":44.336558817987346,"promptTokensCount":27,"predictedTokensCount":4,"stopReason":"eosFound"}
{"label":"remote:qwen3-coder-30b","modelKey":"qwen/qwen3-coder-30b","architecture":"qwen3moe","promptId":"arith-2","kind":"short","passed":true,"wallMs":685,"finalAnswer":"155","timeToFirstTokenSec":0.441,"tokensPerSecond":33.87218331625611,"promptTokensCount":45,"predictedTokensCount":4,"stopReason":"eosFound"}
{"label":"remote:qwen3-coder-30b","modelKey":"qwen/qwen3-coder-30b","architecture":"qwen3moe","promptId":"logic-1","kind":"short","passed":true,"wallMs":579,"finalAnswer":"no","timeToFirstTokenSec":0.426,"tokensPerSecond":48.546045924559444,"promptTokensCount":36,"predictedTokensCount":2,"stopReason":"eosFound"}
{"label":"remote:qwen3-coder-30b","modelKey":"qwen/qwen3-coder-30b","architecture":"qwen3moe","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":3371,"finalAnswer":"QK-77219-ALPHA","timeToFirstTokenSec":2.843,"tokensPerSecond":31.721443431414272,"promptTokensCount":960,"predictedTokensCount":12,"stopReason":"eosFound"}
{"label":"remote:qwen3-coder-30b","modelKey":"qwen/qwen3-coder-30b","architecture":"qwen3moe","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":4840,"rounds":2,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}
{"label":"remote:glm-4.7-flash","modelKey":"zai-org/glm-4.7-flash","architecture":"deepseek2","promptId":"arith-1","kind":"short","passed":true,"wallMs":7019,"finalAnswer":"1. **Analyze the Request:**\n * Math problem: $17 \\times 24$.\n * Constraint: Reply with *only* the number, nothing else.\n\n2. **Perform the Calculation:**\n * Method 1: Standard multiplication.\n * $17 \\times 20 = 340$\n * $17 \\times 4 = 68$\n * $340 + 68 = 408$\n * Method 2: Distributive property.\n * $17 \\times 24 = 17 \\times (20 + 4) = 340 + 68 = 408$\n * Method 3: $24 \\times 17$.\n * $24 \\times 10 = 240$\n * $24 \\times 7 =","timeToFirstTokenSec":0.344,"tokensPerSecond":45.086396996733264,"promptTokensCount":22,"predictedTokensCount":299,"stopReason":"eosFound"}
{"label":"remote:glm-4.7-flash","modelKey":"zai-org/glm-4.7-flash","architecture":"deepseek2","promptId":"arith-2","kind":"short","passed":true,"wallMs":8995,"finalAnswer":"1. **Analyze the Request:**\n * Departure time: 14:05\n * Arrival time: 16:40\n * Question: How many minutes did the trip take?\n * Constraint: Reply with *only* the number.\n\n2. **Calculate the duration:**\n * Method 1: Calculate hours and minutes separately.\n * From 14:05 to 15:05 is 1 hour.\n * From 15:05 to 16:05 is another hour.\n * From 16:05 to 16:40 is 35 minutes.\n * Total time = 2 hours + 35 minutes.\n * Convert hours to min","timeToFirstTokenSec":0.438,"tokensPerSecond":44.86950667152009,"promptTokensCount":38,"predictedTokensCount":382,"stopReason":"eosFound"}
{"label":"remote:glm-4.7-flash","modelKey":"zai-org/glm-4.7-flash","architecture":"deepseek2","promptId":"logic-1","kind":"short","passed":false,"wallMs":5320,"finalAnswer":"1. **Analyze the Request:**\n * Premise 1: Alice > Bob (Alice is taller than Bob).\n * Premise 2: Bob > Carol (Bob is taller than Carol).\n * Question: Is Carol > Alice? (Is Carol taller than Alice?)\n * Constraint: Reply with only 'yes' or 'no'.\n\n2. **Evaluate the Logic:**\n * From Premise 1: Alice > Bob.\n * From Premise 2: Bob > Carol.\n * Combining these (transitive property): Alice > Bob > Carol.\n * Therefore, Alice > Carol.\n * The question asks if Ca","timeToFirstTokenSec":0.426,"tokensPerSecond":45.275622555245,"promptTokensCount":33,"predictedTokensCount":220,"stopReason":"eosFound"}
{"label":"remote:glm-4.7-flash","modelKey":"zai-org/glm-4.7-flash","architecture":"deepseek2","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":11236,"finalAnswer":"1. **Analyze the Request:**\n * Input: A text containing multiple paragraphs describing logistics reports for various sites (0-16) and a specific reference code for the \"Halden vault\".\n * Task: Identify the reference code for the Halden vault.\n * Constraint: Reply with *only* the code.\n\n2. **Scan the Text for Keywords:**\n * Keywords: \"Halden vault\", \"reference code\".\n\n3. **Locate the Information:**\n * Scanning through the paragraphs...\n * Paragraph 0-16 describe r","timeToFirstTokenSec":3.166,"tokensPerSecond":40.48037889634647,"promptTokensCount":933,"predictedTokensCount":322,"stopReason":"eosFound"}
{"label":"remote:glm-4.7-flash","modelKey":"zai-org/glm-4.7-flash","architecture":"deepseek2","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":6137,"rounds":2,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}
{"label":"remote:gemma-4-26b-a4b","modelKey":"google/gemma-4-26b-a4b-qat","architecture":"gemma4","promptId":"arith-1","kind":"short","passed":true,"wallMs":6614,"finalAnswer":"<|channel>thought\nThe user wants to calculate 17 * 24.\n\n * Method 1: Standard multiplication\n 17 * 20 = 340\n 17 * 4 = 68\n 340 + 68 = 408\n * Method 2: Standard multiplication (another way)\n 24 * 10 = 240\n 24 * 7 = 168\n 240 + 168 = 408\n * Method 3: (20 - 3) * 24 = 480 - 72 = 408\n * Method 4: (17 * 2) * 12 = 34 * 12 = 408\n 34 * 10 = 340\n 34 * 2 = 68\n 340 + 68 = 408\n\nThe result is 408.\n\"Reply with only the number, not","timeToFirstTokenSec":0.55,"tokensPerSecond":42.09774459428036,"promptTokensCount":35,"predictedTokensCount":251,"stopReason":"eosFound"}
{"label":"remote:gemma-4-26b-a4b","modelKey":"google/gemma-4-26b-a4b-qat","architecture":"gemma4","promptId":"arith-2","kind":"short","passed":true,"wallMs":6557,"finalAnswer":"<|channel>thought\n* Departure time: 14:05\n * Arrival time: 16:40\n * Constraint: Both times are on the same day.\n * Goal: Calculate total trip duration in minutes.\n\n * Hours difference: 16 - 14 = 2 hours.\n * Minutes difference: 40 - 5 = 35 minutes.\n\n * Total time = (2 hours * 60 minutes/hour) + 35 minutes\n * Total time = 120 minutes + 35 minutes\n * Total time = 155 minutes.\n\n * 14:05 to 15:05 is 60 minutes.\n * 15:05 to 16:05 is 60 minutes (Total","timeToFirstTokenSec":0.323,"tokensPerSecond":41.998733567257865,"promptTokensCount":53,"predictedTokensCount":258,"stopReason":"eosFound"}
{"label":"remote:gemma-4-26b-a4b","modelKey":"google/gemma-4-26b-a4b-qat","architecture":"gemma4","promptId":"logic-1","kind":"short","passed":false,"wallMs":5088,"finalAnswer":"<|channel>thought\n* Premise 1: Alice > Bob (Alice is taller than Bob).\n * Premise 2: Bob > Carol (Bob is taller than Carol).\n * Question: Is Carol taller than Alice? (Is Carol > Alice?)\n\n * If A > B and B > C, then A > C (transitive property).\n * If Alice is taller than Bob, and Bob is taller than Carol, then Alice is taller than Carol.\n\n * The question asks if Carol is taller than Alice.\n * Since Alice is taller than Carol, Carol is NOT taller than Alice.\n\n * ","timeToFirstTokenSec":0.29,"tokensPerSecond":42.55207110238321,"promptTokensCount":44,"predictedTokensCount":180,"stopReason":"eosFound"}
{"label":"remote:gemma-4-26b-a4b","modelKey":"google/gemma-4-26b-a4b-qat","architecture":"gemma4","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":7237,"finalAnswer":"<|channel>thought\n* Input: A long report consisting of multiple paragraphs (0-24) and one specific line of information.\n * Question: \"what is the reference code for the Halden vault?\"\n * Constraint: \"Reply with only the code.\"\n\n * Paragraph 0: Site 0 info.\n * Paragraph 1: Site 1 info.\n * ...\n * Paragraph 13: Site 13 info.\n * *Key Sentence:* \"The reference code for the Halden vault is QK-77219-ALPHA.\"\n * Paragraph 14: Site 14 info.\n * ...\n\n * Th","timeToFirstTokenSec":1.186,"tokensPerSecond":40.21704647877274,"promptTokensCount":942,"predictedTokensCount":237,"stopReason":"eosFound"}
{"label":"remote:gemma-4-26b-a4b","modelKey":"google/gemma-4-26b-a4b-qat","architecture":"gemma4","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":7968,"rounds":3,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}
{"label":"remote:gpt-oss-20b","modelKey":"openai/gpt-oss-20b","architecture":"gpt-oss","promptId":"arith-1","kind":"short","passed":true,"wallMs":1421,"finalAnswer":"<|channel|>analysis<|message|>Compute 17*24 = (20-3)*24=480-72=408.<|end|><|start|>assistant<|channel|>final<|message|>408","timeToFirstTokenSec":0.478,"tokensPerSecond":46.6050547842419,"promptTokensCount":84,"predictedTokensCount":30,"stopReason":"eosFound"}
{"label":"remote:gpt-oss-20b","modelKey":"openai/gpt-oss-20b","architecture":"gpt-oss","promptId":"arith-2","kind":"short","passed":true,"wallMs":1955,"finalAnswer":"<|channel|>analysis<|message|>Compute duration: from 14:05 to 16:40. Hours difference: 2 hours (14->16) =120 min. Minutes difference: 35 minutes (5->40). Total=155.<|end|><|start|>assistant<|channel|>final<|message|>155","timeToFirstTokenSec":0.415,"tokensPerSecond":38.093575830110964,"promptTokensCount":100,"predictedTokensCount":55,"stopReason":"eosFound"}
{"label":"remote:gpt-oss-20b","modelKey":"openai/gpt-oss-20b","architecture":"gpt-oss","promptId":"logic-1","kind":"short","passed":false,"wallMs":1354,"finalAnswer":"<|channel|>analysis<|message|>We need to answer yes/no whether Carol taller than Alice given chain: Alice > Bob > Carol, so Carol < Alice. So answer no.<|end|><|start|>assistant<|channel|>final<|message|>no","timeToFirstTokenSec":0.393,"tokensPerSecond":46.341081948410796,"promptTokensCount":95,"predictedTokensCount":40,"stopReason":"eosFound"}
{"label":"remote:gpt-oss-20b","modelKey":"openai/gpt-oss-20b","architecture":"gpt-oss","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":2858,"finalAnswer":"<|channel|>analysis<|message|>Answer given in paragraph13.<|end|><|start|>assistant<|channel|>final<|message|>QK-77219-ALPHA","timeToFirstTokenSec":2.209,"tokensPerSecond":46.41708458159833,"promptTokensCount":992,"predictedTokensCount":24,"stopReason":"eosFound"}
{"label":"remote:gpt-oss-20b","modelKey":"openai/gpt-oss-20b","architecture":"gpt-oss","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":4242,"rounds":3,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}
{"label":"remote:qwen3.6-27b","modelKey":"qwen/qwen3.6-27b","architecture":"qwen35","promptId":"arith-1","kind":"short","passed":true,"wallMs":34718,"finalAnswer":"Here's a thinking process:\n\n1. **Analyze User Input:**\n - Question: \"What is 17 * 24?\"\n - Constraint: \"Reply with only the number, nothing else.\"\n\n2. **Calculate the Product:**\n - 17 * 24\n - Break it down: 17 * 20 = 340, 17 * 4 = 68\n - 340 + 68 = 408\n - Alternatively: 24 * 10 = 240, 24 * 7 = 168, 240 + 168 = 408\n - Double-check: 17 * 24 = 408. Correct.\n\n3. **Format Output:**\n - Constraint says \"only the number, nothing else.\"\n - So the output should be exactly \"408\".\n\n4. **","timeToFirstTokenSec":1.006,"tokensPerSecond":7.178804135821788,"promptTokensCount":29,"predictedTokensCount":242,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-27b","modelKey":"qwen/qwen3.6-27b","architecture":"qwen35","promptId":"arith-2","kind":"short","passed":false,"error":"timeout after 120000ms: arith-2"}
{"label":"remote:qwen3.6-27b","modelKey":"qwen/qwen3.6-27b","architecture":"qwen35","promptId":"logic-1","kind":"short","passed":false,"wallMs":94274,"finalAnswer":"Here's a thinking process:\n\n1. **Analyze User Input:**\n - Statement 1: Alice is taller than Bob. (A > B)\n - Statement 2: Bob is taller than Carol. (B > C)\n - Question: Is Carol taller than Alice? (C > A?)\n - Constraint: Reply with only 'yes' or 'no'.\n\n2. **Logical Reasoning:**\n - From A > B and B > C, by transitivity, A > C.\n - Therefore, Carol is shorter than Alice, not taller.\n - The answer to \"Is Carol taller than Alice?\" is no.\n\n3. **Check Constraint:**\n - Must reply with ","timeToFirstTokenSec":3.657,"tokensPerSecond":3.3139219711639467,"promptTokensCount":38,"predictedTokensCount":301,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-27b","modelKey":"qwen/qwen3.6-27b","architecture":"qwen35","promptId":"needle-haystack","kind":"long","passed":true,"wallMs":142837,"finalAnswer":"The user wants to find the reference code for the Halden vault from the provided text.\nI need to scan the text for \"Halden vault\" and extract the corresponding code.\nLooking at the text:\nParagraph 0-13: Routine logistics report.\nAfter Paragraph 13: \"The reference code for the Halden vault is QK-77219-ALPHA.\"\nParagraph 14-24: Routine logistics report.\nThe question asks for only the code.\nSo the answer should be \"QK-77219-ALPHA\".\nDouble check the exact wording: \"The reference code for the Halden v","timeToFirstTokenSec":27.868,"tokensPerSecond":3.8008961852179324,"promptTokensCount":989,"predictedTokensCount":437,"stopReason":"eosFound"}
{"label":"remote:qwen3.6-27b","modelKey":"qwen/qwen3.6-27b","architecture":"qwen35","promptId":"agentic-file-roundtrip","kind":"agentic","passed":false,"wallMs":92251,"rounds":3,"toolCallCount":2,"toolCallSequence":["write_file","read_file"],"fileCorrect":true,"finalTextHasMarker":false}