Installation

Load the built plugin folder in LM Studio.

Configuration

Tools

Prompt Management

Eval Datasets

Model Evaluation

Prompt Versioning

Every save_prompt call creates a new version. Version 1 is always the baseline.

Use diff_prompts(name, 1, 3) to see what changed between baseline and current. Use compare_models with both prompt versions to see if the change actually helped.

Template Variables

Use {{variable_name}} in prompt content:

run_prompt_template(name, variables={"role": "reviewer", "user_input": "...", "language": "English"}) returns the filled prompt.

Eval Workflow

Scoring

Score labels (set scoreLabel to one of):

• exact_match — exact string comparison
• rubric — scored against a defined rubric
• human — human-judged
• llm_judge — scored by another LLM

Example Sessions

Saving a prompt:

"Save this classification prompt"

save_prompt(name="intent_classifier", content="Classify the intent of: {{input}}\nOptions: question, request, complaint", description="Basic intent classification")

Iterating on a prompt:

"Add chain-of-thought to the intent classifier"

save_prompt(name="intent_classifier", content="Let's think step by step...", description="Added CoT reasoning") → version 2 saved

diff_prompts(name="intent_classifier", versionA=1, versionB=2) → see exactly what changed

Setting up an eval:

"Create an eval for intent classification"

create_eval_dataset(name="intent_classification", description="Tests accurate intent labeling across 3 categories") → add_eval_case × 10

Comparing models:

"Which model is better at intent classification?"

compare_models(datasetName="intent_classification") → leaderboard by avg score, case-by-case breakdown

Writing a report:

"Write up our eval findings"

generate_eval_report(datasetName="intent_classification") → structured report: leaderboard, failure patterns, recommendation

Data

All data is local. Nothing is sent to any external service.