Project Files
docs / initial-docs / SETUP.md
A new user has just downloaded the user-docs plugin and wants to configure it to their needs. The agent guides them through the setup process step by step ā not with a manual dump, but as an interactive conversation, one decision per step. Along the way it live-demonstrates:
find_doc)read_config)find_doc registers a relevant image: inspect with and display via ."This is your new user-docs plugin! I'll help you set it up.
Shall I guide you through the setup step by step? Or do you have a specific question right away?"
Two paths:
find_doc + screenshots as demo.What the agent does:
Agent narrative (example):
"Before we start with your agent model: user-docs requires a running LM Studio Server so the plugin can communicate."
ā Shows screenshot of the Developer panel
Explanation:
"Here's how to activate it:"
- Click Developer on the left in LM Studio.
- Enable the switch under Local Server (green toggle).
- Load at least one model from the right side ā that will be your agent model.
ā Wait for confirmation before starting Step 2.
What the agent does:
read_config() internally to check current configuration status.Agent narrative (example):
"Great, let's start with the most important setting: your agent model. This is the model that accompanies you through chats and executes all plugin commands."
ā Shows
agent-model_default.jpegoragent-model.jpeg(depending on current configuration)
Explanation:
"There are two good options:"
Default:
qwen/qwen3.6-35b-a3bā Compact and fast, works well for clearly instructed tasks. This is the default setting.More powerful:
qwen/qwen3.6-27bā Significantly more capable and flexible, handles tasks even without specific instructions. But: noticeably slower at response generation.
Additional info (if user asks):
"user-docs is primarily optimized for the LM Studio Server, but it's fully OpenAI-compatible. This means:"
- You can also use other local APIs ā e.g., oLMX or Unsloth Studio on your network.
- Cloud APIs work too: OpenAI (
https://api.openai.com/v1), Claude (https://api.anthropic.com/v1) and any others that support the OpenAI format.
ā Wait for user confirmation ("Looks good" / "I want to change"). ā If a change is desired: Agent briefly explains how (Plugin Settings ā Agent Model) and waits again before Step 3 begins.
"First: What do you want to treat as a local source?"
- Local Markdown/TXT notes?
- PDF manuals?
- Search through LM Studio conversations?
- Add external docs from GitHub/URL?
ā Wait for answer. This determines which directories need configuration and what can be demonstrated live in the showcase.
"Second: What do you mainly want to use
user-docsfor?"
- Is managing your documents your primary use case with this plugin?
- Or are you primarily using it to access different OpenAI-compatible backends from within LM Studio?
ā Wait for answer. This determines whether the config switch Auto-read user-docs Guide (autoReadUserDocsGuide) should be set to on or off (handled at the end of Step 4).
What the agent does:
read_config() internally to check current Notes Directory and Content Directories status.Agent narrative (example):
"Now for the heart of it: Where are your documents? user-docs supports three types of directories:"
ā Shows default or configured screenshot (depending on state)
(or)
Explanation:
"There are three types of content directories:"
1. Notes Directory (read-write access)
This is where the agent stores notes, instructions, and guides. The first document should beUSER-DOCS.mdā the plugin's core operational guide. It lives by default at:We copy this file into your Notes Directory so it can not only be read there but also developed as a "living document".
2. Content Directories (read-only)
This is where you place your own documentation ā PDFs, TXT files, and Markdown documents. Even large, complex docs with tables or images are fully searchable and indexed. You can search for arbitrary keywords here, but also look up specific filenames directly. Found documents can be read in full; images from MDs and chats can be extracted entirely ā for PDFs page by page.3. LM Studio Conversations as Content Directory
When you add~/.lmstudio/conversations, all your previous conversations are included in the index. Practical if you want to continue working with information or images from past chats without having to manually copy them out first.
ā Wait for user confirmation ("Looks good" / "I want to add/change directories").
ā If changes desired: Agent briefly explains how (Plugin Settings ā Content Directories) and waits again before the autoReadUserDocsGuide decision comes up.
Auto-read user-docs Guide (autoReadUserDocsGuide) ā depending on Step 3 answer:
"Now one more important switch, based on what you said in Step 3 about your main use case:"
ā Shows screenshot (same as above: content-directories_default.jpeg or content-directories.jpeg)
(or)
Explanation:
If the user named PKM / document management as their primary purpose (Default: ON):
"Since you primarily use
user-docsfor managing your documents, I recommend leaving the switchAuto-read user-docs Guideset to ON. That's the default."
- The file
USER-DOCS.mdis then automatically loaded at the start of every session and provided as context to the agent.- Starting a new session takes slightly longer because of this, but the agent comes fully operational right away ā it knows all tools, workflows, and rules without you having to explain them.
If the user wants to use the plugin "as-is" (OpenAI-compatible backends):
"Since you're primarily using
user-docsas an OpenAI-compatible wrapper for different backends, I recommend setting the switchAuto-read user-docs Guideto ."
ā Wait for user confirmation ("Looks good" / "I want to change"). ā If changes desired: Agent briefly explains how (Plugin Settings ā Auto-read user-docs Guide) and waits again before Step 5 begins.
What the agent does:
read_config() internally to check current status of embedding model and Qwen3-VL settings.Agent narrative (example):
"Almost done! Now we need two small models ā one for text search, one for image analysis:"
ā Shows screenshot of embedding/vision settings
Explanation:
Embedding model (
ggml-org/bge-m3-Q8_0-GGUF)
This model powers semantic search across your documents. It's small and fast, but needs to be available through the LM Studio Server (load it once in Local Server).Vision model (
qwen/qwen3-vl-8b)
For image analysis ā so the agent can read screenshots or detect objects in images. This one also works best via the LM Studio API.
Important notes:
"Both models are small and resource-efficient. It's recommended to run them on your LM Studio Server."
- Models other than those specified above are not recommended ā unless they're variants (finetunes, quantizations) of the listed standard models.
- If no LM Studio Server is available at all:
bge-m3-Q8_0-GGUFcan also come from an alternative/embeddingsendpoint, and Qwen3-VL-8B can be specified as a local model path (GGUF or MLX). The model then runs via a built-in Python server ā this is not recommended.
ā Wait for user confirmation ("Looks good" / "I want to change"). ā If changes desired: Agent briefly explains how and waits again before Step 6 begins.
What the agent does:
read_config() internally to check current prompt settings.annotate_image.Agent narrative (example):
"Last but not least, two fine-tuning options ā they're optional but good to understand:"
ā Shows screenshot of vision prompt settings and marks the input field with a live demo:
(and/or)
Explanation:
1. Vision Prompt (
analyse_image)
This prompt is sent to the vision model when the agent wants an image description ā e.g., as a "second opinion".
- Default is empty. Reason:
analyse_imageautomatically returns generation metadata (for Draw Things and ComfyUI-generated images), which are valuable for the agent but cannot be "seen" by it directly. An automatic visual description on top of that would often be redundant or only make sense if your chosen agent model struggles with visual tasks.- When the vision prompt is empty,
analyse_imageprovides a visual description ONLY when the agent explicitly asks for one ā e.g., to get another visual assessment.
Live demo recipe for the agent:
ā Wait for user confirmation ("Understood" / "I want to change"). ā If changes desired: Agent briefly explains how and waits again before Step 7 begins.
What the agent does:
baseUrl configuration.Agent narrative (example):
"One final important point: Where does your LM Studio Server run? That affects how you configure baseUrl."
ā Shows matching scenario diagram
"Everything local ā no networking, no ports to open. The simplest path."
"Here, agent model inference runs on the Mac Studio while embedding and vision stay local. Important: The LM Studio Server on your Mac Studio must be reachable from your network."
ā Wait for user confirmation ("Looks good" / "I have a different setup"). ā If their own configuration differs: Agent asks specifically about the differences and adapts the scenario accordingly.
"Setup complete! Would you like to run your first search now, just to see if everything works?"
ā On approval: Run a live search with
find_doc("user-docs setup configuration")and show the result + relevant screenshot. This is the final Aha-moment of the showcase.
review_imageshow_imageread_config() internally to check current configuration status.baseUrl is set correctly and if the local server needs activation.notesDirectory is set correctly and NOT EMPTY. The default: "current": "" must be filled properly. Recommended: ~/Documents/User-docs."current": values, especially if "isModified": true and compares them with "default":.read_config() parameters.
USER-DOCS.md on demand at any time ā that's always possible, regardless of how you set the switch.2. Object Detection Prompt (annotate_image)
This prompt tells Qwen3-VL what objects or regions to detect and mark in an image with bounding boxes.
annotate_image (just like analyse_image without a vision prompt) would return nothing at all ā no detections drawn. The built-in default ensures the tool always produces some output out of the box.~/.lmstudio/extensions/plugins/ceveyne/user-docs/docs/initial-docs/USER-DOCS.md
{
"tool": "annotate_image",
"targets": ["p15"],
"task": "Detect the black text input box that contains the vision prompt text \"Analyze this image based strictly on what is directly visible. Do not infer, assume, or complete information that is not present.\" Only mark that specific textarea/input field ā nothing else.",
"color": "blue",
"lineWeight": 7,
"frameAdjust": 1
}
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā MacBook Pro M5 ā
ā ā
ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā
ā ā LM Studio App ā ā
ā ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā
ā ā ā Plugin: user-docs ā ā ā
ā ā ā ā ā ā
ā ā ā vision-capability-primer: qwen/qwen3-vl-4b ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā
ā ā ā ā ā
ā ā ā OpenAI-compat. API ā ā
ā ā ā¼ ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā
ā ā ā LM Studio Server (local) ā ā ā
ā ā ā baseUrl: http://127.0.0.1:1234/v1 ā ā ā
ā ā ā Agent Model: qwen/qwen3.6-27b ā ā ā
ā ā ā Embedding Model: ggml-org/bge-m3-Q8_0-GGUF ā ā ā
ā ā ā Vision Model: qwen/qwen3-vl-8b ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā
ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā
ā ā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā MacBook Neo ā
ā ā
ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā ā LM Studio App ā ā ā Mac Studio M3 Ultra ā
ā ā ā ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā
ā ā ā Plugin: user-docs ā ā ā ā LM Studio Server ā
ā ā ā ā ā ā ā ā
ā ā ā vision-capability-primer: qwen/qwen3-vl-4b ā ā ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā Agent Model: ā
ā ā ā ā ā ā qwen/qwen3.6-27b ā
ā ā ā OpenAI-compat. API āāāāāāāāāāāāāāāāāāā¶ļøā http://<studio-ip>:1234/v1 ā
ā ā ā¼ ā ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā
ā ā ā LM Studio Server (local) ā ā ā ā ā
ā ā ā baseUrl: http://127.0.0.1:1234/v1 ā ā ā ā ā
ā ā ā Agent Model: qwen/qwen3.6-27b ā ā ā ā ā
ā ā ā Embedding Model: ggml-org/bge-m3-Q8_0-GGUF ā ā ā ā ā
ā ā ā Vision Model: qwen/qwen3-vl-8b ā ā ā ā ā
ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā
ā ā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā