AI Lab Plugin for LM Studio

A local AI engineering lab for systematic prompt engineering and model evaluation. Version prompts, build eval datasets, log model outputs, and compare results — all from a single chat session, all persisted locally.

Not a vibe-based workflow. Every prompt change is versioned and every model claim is backed by logged results.

What It Does

Prompt versioning — save named prompts with automatic version increments; never overwrite, always compare
Template variables — use {{variable}} syntax; fill and preview before running
Eval datasets — build named collections of test cases with inputs and expected outputs
Model result logging — record what each model output for each case, with a score and scoring method
Model comparison — leaderboard + side-by-side case-by-case comparison across models and prompt versions
Eval reports — structured narrative synthesis of what works, what fails, and what to test next

Installation

Load the built plugin folder in LM Studio.

Configuration

Field	Default	Description
Data Path	`~/ai-lab-data/`	SQLite database location

Tools

Prompt Management

Tool	Kind	What it does
`save_prompt`	Store-write	Save a new version of a prompt — auto-increments version, never overwrites
`get_prompt`	Store-read	Get a prompt by name (latest or specific version)
`list_prompts`	Store-read	List all prompts with latest version and description
`list_prompt_versions`	Store-read	List all versions of a specific prompt
`diff_prompts`	Compute	Line-by-line diff between two prompt versions
`run_prompt_template`	Compute	Fill `{{variable}}` placeholders and return the rendered prompt

Eval Datasets

Tool	Kind	What it does
`create_eval_dataset`	Store-write	Create a named dataset for organizing test cases
`add_eval_case`	Store-write	Add a test case with input and expected output
`list_eval_datasets`	Store-read	List all datasets with case counts
`get_eval_dataset`	Store-read	Get a dataset with all its cases and IDs

Model Evaluation

Tool	Kind	What it does
`log_model_result`	Store-write	Log a model's output for a case with optional score
`compare_models`	Store-read	Leaderboard + case-by-case comparison across models
`generate_eval_report`	Scaffold	Return a report payload for LLM to write a narrative eval summary

Prompt Versioning

Every save_prompt call creates a new version. Version 1 is always the baseline.

Use diff_prompts(name, 1, 3) to see what changed between baseline and current. Use compare_models with both prompt versions to see if the change actually helped.

Template Variables

Use {{variable_name}} in prompt content:

run_prompt_template(name, variables={"role": "reviewer", "user_input": "...", "language": "English"}) returns the filled prompt.

Eval Workflow

Scoring

Score	Meaning
`1.0`	Perfect — exactly matches expected or fully correct
`0.8`	Good — minor issues or slight deviation
`0.5`	Partial — got the right idea but incomplete or partially wrong
`0.2`	Poor — attempted but fundamentally wrong
`0.0`	Failure — completely wrong or refused
`null`	Not yet scored

Score labels (set scoreLabel to one of):

exact_match — exact string comparison
rubric — scored against a defined rubric
human — human-judged
llm_judge — scored by another LLM

Example Sessions

Saving a prompt:

"Save this classification prompt"

save_prompt(name="intent_classifier", content="Classify the intent of: {{input}}\nOptions: question, request, complaint", description="Basic intent classification")

Iterating on a prompt:

"Add chain-of-thought to the intent classifier"

save_prompt(name="intent_classifier", content="Let's think step by step...", description="Added CoT reasoning") → version 2 saved

diff_prompts(name="intent_classifier", versionA=1, versionB=2) → see exactly what changed

Setting up an eval:

"Create an eval for intent classification"

create_eval_dataset(name="intent_classification", description="Tests accurate intent labeling across 3 categories") → add_eval_case × 10

Comparing models:

"Which model is better at intent classification?"

compare_models(datasetName="intent_classification") → leaderboard by avg score, case-by-case breakdown

Writing a report:

"Write up our eval findings"

generate_eval_report(datasetName="intent_classification") → structured report: leaderboard, failure patterns, recommendation

Data

All data is local. Nothing is sent to any external service.

ai-lab