big-rag-saved
Public
Forked from mindstudio/big-rag
Forked from mindstudio/big-rag
Project Files
README.md
A powerful RAG (Retrieval-Augmented Generation) plugin for LM Studio that can index and search through gigabytes or even terabytes (not tested) of document data. Hosted here: ari99/lm_studio_big_rag_plugin on GitHub.
.java, .py, .rs) via the Additional plain-text extensions setting| Category | Extensions |
|---|---|
| Documents | .pdf, .epub, .txt, .text |
| Markdown | .md, .mdx, .markdown, .mkd, .mkdn, .mdown |
| Web | .htm, .html, .xhtml |
| Images (OCR) | .bmp, .jpeg, .jpg, .png |
PDF, EPUB, HTML, and images use dedicated parsers. Plain-text and Markdown files use the text parser.
Any other plain-text extension can be indexed by listing it in Additional plain-text extensions in the chat Integrations sidebar (or via BIG_RAG_ADDITIONAL_EXTENSIONS for CLI indexing).
Common examples:
Format rules:
.java and java both work)# are comments; inline # after an extension is also stripped*, ?) are not allowedRejected automatically: binaries and formats that already have dedicated parsers or are unsafe to read as text — e.g. .exe, .zip, .jar, .docx, .pdf, .png. Rejections are logged as [BigRAG] Rejected additional extension … in developer logs.
After changing extensions: trigger a reindex (empty vector store + chat message, or Manual Reindex Trigger ON) so new file types are picked up. Pair with Exclude filename patterns when indexing source trees:
CLI / headless indexing:
Then choose one of the following:
| Goal | Command | Plugin id for REST |
|---|---|---|
| Dev (hot reload, chat UI) | npm run dev | Use installed copy for REST (see below) |
| Local install (REST + chat) | lms dev --install -y | mindstudio/big-rag |
| Publish to Hub | lms login then lms push -y | mindstudio/big-rag |
After code changes: npm run build then re-run npm run dev, lms dev --install -y, or lms push -y.
Hub page: lmstudio.ai/mindstudio/big-rag
Default chat workflow — no lms server start; the server runs inside the LM Studio app (http://localhost:1234 when enabled).
Automated tests (no UI):
Headless indexing (optional; LM Studio app must be open for embeddings):
Paths are set in package.json index script or via BIG_RAG_DOCS_DIR / BIG_RAG_DB_DIR env vars.
All plugin fields appear in the chat Integrations sidebar when Big RAG is enabled (expand the plugin row). There is no separate global settings screen for document paths.
The plugin provides the following configuration options:
/docs/a, /docs/b. Missing paths are skipped with a warning.*.png, node_modules/**, target/**. Applied after the extension gate; does not remove chunks already in the vector store.CLI equivalents: BIG_RAG_EXCLUDE_PATTERNS and BIG_RAG_ADDITIONAL_EXTENSIONS (semicolon-separated for env vars).
Big RAG registers a prompt preprocessor (automatic RAG in chat) and a tools provider (big_rag_search, big_rag_index_status) for /api/v1/chat.
REST tool calls have no chat session, so they do not read the sidebar directly. Paths come from (in order):
If you delete the JSON file, send a chat message again (with both paths set) or set the env vars before calling tools via curl.
/api/v1/chat)Load your API token (example: repo-root .env):
Index status (one tool — reliable):
Search (one tool — use limit 3 in the prompt to avoid context overflow):
Preprocessor-only RAG (no explicit tools — model answers using injected context; needs sidebar config + chat sync or env vars):
| Tool | Description |
|---|---|
big_rag_search | Embed a query and return matching passages (JSON with scores and file names) |
big_rag_index_status | Return chunk count, unique file count, and configured directory paths (needs vector store path; documents path is optional metadata) |
/v1/chat/completionsLM Studio does not support plugin integrations on the OpenAI-compatible chat endpoint. That is a platform limitation (not something this plugin can enable).
/v1/chat/completions)integrations field): Chat CompletionsPOST /api/v1/chat with an integrations block (examples above)Source code / repo RAG
Point Documents Directory at a project root, add extensions, and exclude build artifacts:
Reindex after changing extensions. Check developer logs for [Scanner] Additional plain-text extensions: … on startup.
Technical documentation
Larger Chunk Size (1024) and default retrieval settings work well for manuals and API docs.
maxConcurrentFiles if needed)maxConcurrentFiles on systems with limited resourcestool_format_generation_errorallowed_tools with a single entry).Authorization: Bearer $LM_API_TOKEN if API auth is enabled.BIG_RAG_ADDITIONAL_EXTENSIONS[BigRAG] Rejected additional extension (binary/built-in types are blocked)node_modules/**, etc.).big-rag-embedding.json or run a full reindex after changing the model.maxConcurrentFilesmaxConcurrentFiles to 1 or 2See Manual testing at the top of this README. Summary:
npm test — unit tests (extensions, parsers, retrieval helpers)npm run dev + LM Studio chat — E2E UI (dev plugin)npm run build && lms dev --install -y + curl to /api/v1/chat — E2E REST (installed plugin id mindstudio/big-rag; config synced via chat message or env vars)This plugin is based on the LM Studio plugin SDK. For more information:
ISC
cd big-rag-plugin && npm run dev (leave running; good for UI iteration)cd big-rag-plugin && npm run build && lms dev --install -ymixedbread-ai/mxbai-embed-large-v1 (Hub / lms get) and text-embedding-mxbai-embed-large-v1 (as shown in lms ls). Use one spelling consistently for indexing and retrieval so it matches .big-rag-embedding.json; switching spelling without reindexing can trigger a mismatch warning. Default: nomic-ai/nomic-embed-text-v1.5-GGUF..big-rag-embedding.json: Written under the vector store directory when the index has at least one chunk; records the model id and vector length used to build the index. If the configured model no longer matches this file, retrieval is blocked until you reindex or revert the setting. If the index has zero chunks, this file is removed so metadata cannot drift (including after manual shard deletion)..txtConfigure the plugin (one place — the chat sidebar):
/Users/user/Documents/MyLibrary)/Users/user/.lmstudio/big-rag-db)Settings → Integrations (gear menu) only controls tool-call confirmation — not document paths.
Initial indexing:
[BigRAG] lines)Query your documents:
http://localhost:1234)."mindstudio/big-rag" (owner/name from manifest.json). The dev id (dev/mindstudio/big-rag from npm run dev) works in chat UI but not in REST.npm run build && lms dev --install -y (local) or lms push -y (Hub).Authorization: Bearer $LM_API_TOKEN (docs).~/.lmstudio/big-rag-tools-config.json when the prompt preprocessor runs — both Documents Directory and Vector Store Directory must be set in chat for that sync to write. Send at least one chat message with Big RAG enabled after configuring paths.BIG_RAG_DOCS_DIR, BIG_RAG_DB_DIR, optional BIG_RAG_EMBEDDING_MODEL, BIG_RAG_RETRIEVAL_LIMIT, BIG_RAG_RETRIEVAL_AFFINITY_THRESHOLD.tool_format_generation_error because smaller models (e.g. Llama 3.1 8B) emit multiple tool calls in one generation block. Run separate curl requests instead.big_rag_search with limit=10 returns full passage text and can exceed the model context window (e.g. 14848 tokens). Ask for limit 3 or increase context length in LM Studio.~/.lmstudio/big-rag-tools-config.json, or set BIG_RAG_DOCS_DIR / BIG_RAG_DB_DIR.File Scanner (src/ingestion/fileScanner.ts):
Document Parsers (src/parsers/):
htmlParser.ts: Extracts text from HTML/HTM filespdfParser.ts: Extracts text from PDF filesepubParser.ts: Extracts text from EPUB filestextParser.ts: Reads plain text & Markdown files with optional Markdown strippingimageParser.ts: OCR for image filesdocumentParser.ts: Routes to appropriate parserVector Store (src/vectorstore/vectorStore.ts):
Index Manager (src/ingestion/indexManager.ts):
Prompt Preprocessor (src/promptPreprocessor.ts):
Retrieval module (src/rag/retrieval.ts):
Tools Provider (src/toolsProvider.ts):
big_rag_search and big_rag_index_status for REST API / agent integrationsretrievalAffinityThreshold~/.lmstudio/big-rag-tools-config.json.BIG_RAG_DOCS_DIR and BIG_RAG_DB_DIR on the LM Studio process."mindstudio/big-rag" and the plugin is installed (lms dev --install -y).big_rag_index_status only requires the vector store path; big_rag_search needs a usable index (indexing still needs the documents path).successfailedBIG_RAG_FAILURE_REPORT_PATH=/absolute/path/report.json when running npm run index (or via LM Studio env settings) to emit a JSON report containing all failure reasons and counts after indexing completes. This is useful when triaging stubborn PDFs such as blueprints or large scanned books.BIG_RAG_EMBEDDING_MODEL: Optional. When set for headless indexing (npm run index:cli / dist/cliIndex.js), overrides the default embedding model id (same default as the plugin’s Embedding Model setting). Empty/unset uses the built-in default from config.ts..java
.cs
.py
.rs
.go
.ts
.tsx
.js
.jsx
.c
.cpp
.h
.sql
.yaml
.toml
node_modules/**
target/**
bin/**
dist/**
.git/**
BIG_RAG_ADDITIONAL_EXTENSIONS=".java;.cs;.py" \
BIG_RAG_DOCS_DIR=/path/to/repo \
BIG_RAG_DB_DIR=/path/to/vectorstore \
npm run index
cd big-rag-plugin
npm install
npm run build
cd big-rag-plugin && npm test
cd big-rag-plugin && npm run index
export $(grep -v '^#' .env | xargs) # sets LM_API_TOKEN
curl -s http://127.0.0.1:1234/api/v1/chat \
-H "Authorization: Bearer $LM_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-id",
"input": "Call big_rag_index_status and report totalChunks.",
"integrations": [{
"type": "plugin",
"id": "mindstudio/big-rag",
"allowed_tools": ["big_rag_index_status"]
}],
"temperature": 0
}' | python3 -m json.tool
curl -s http://127.0.0.1:1234/api/v1/chat \
-H "Authorization: Bearer $LM_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-id",
"input": "Use big_rag_search to find content about rifle cleaning. Use limit 3.",
"integrations": [{
"type": "plugin",
"id": "mindstudio/big-rag",
"allowed_tools": ["big_rag_search"]
}],
"temperature": 0
}' | python3 -m json.tool
curl -s http://127.0.0.1:1234/api/v1/chat \
-H "Authorization: Bearer $LM_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-id",
"input": "What does the documentation say about rifling?",
"integrations": [{
"type": "plugin",
"id": "mindstudio/big-rag"
}],
"temperature": 0
}' | python3 -m json.tool
Additional plain-text extensions:
.java
.py
.ts
.tsx
Exclude filename patterns:
node_modules/**
target/**
dist/**
.git/**
big-rag-plugin/
├── src/
│ ├── config.ts # Plugin configuration schema
│ ├── index.ts # Main entry point
│ ├── promptPreprocessor.ts # RAG integration
│ ├── toolsProvider.ts # REST tools (search + index status)
│ ├── rag/
│ │ └── retrieval.ts # Shared search / vector store cache
│ ├── ingestion/
│ │ ├── fileScanner.ts # Directory scanning
│ │ └── indexManager.ts # Indexing orchestration
│ ├── parsers/
│ │ ├── documentParser.ts # Parser router
│ │ ├── htmlParser.ts # HTML parsing
│ │ ├── pdfParser.ts # PDF parsing
│ │ ├── epubParser.ts # EPUB parsing
│ │ ├── textParser.ts # Text parsing
│ │ └── imageParser.ts # OCR parsing
│ ├── vectorstore/
│ │ └── vectorStore.ts # Vectra sharded index integration
│ └── utils/
│ ├── additionalExtensions.ts # User-defined plain-text extension parsing
│ ├── coerceEmbedding.ts # Normalize embedding API vectors
│ ├── effectivePluginConfig.ts # Chat ↔ REST tools config sync
│ ├── embeddingIndexManifest.ts # Index embedding metadata on disk
│ ├── fileHash.ts # File hashing
│ └── textChunker.ts # Text chunking (by words)
├── manifest.json # Plugin manifest
├── package.json # Dependencies
├── tsconfig.json # TypeScript config
└── README.md # This file