A powerful RAG (Retrieval-Augmented Generation) plugin for LM Studio that can index and search through gigabytes or even terabytes (not tested) of document data. This is a custom update of ari99/lm_studio_big_rag_plugin. Hosted here: Tedyang2003/big-rag on GitHub.
Features
Massive Scale: Designed to handle large document collections (GB to TB scale)
Deep Directory Scanning: Recursively scans all subdirectories
Resilient PDF Parsing: Three-stage fallback pipeline per PDF — LM Studio's built-in document parser, then pdf-parse, then MuPDF-rendered page images run through Tesseract OCR — so scanned/blueprint-style PDFs still get indexed
OCR Support: Optional OCR for image files and image-based PDFs using Tesseract
Configurable File Exclusion: Skip files by glob pattern (e.g. *.png, archive/**) without touching your document tree
Customizable Prompt Template: Control how retrieved passages and the user query are assembled into the final prompt via {{rag_context}} / {{user_query}} macros
Pre-Indexing Sanity Checks: Verifies directory access, disk space, and free memory before a large indexing run and estimates its size/time
Resilient Indexing: An indexing lock prevents overlapping runs, and a per-file failure registry skips files that previously failed so incremental reindexes don't keep retrying broken documents
Headless CLI Indexing: Index a document set from the command line (npm run index:cli) without needing an LM Studio chat session
Vector Search: Uses Vectra with sharded indexes for efficient vector storage and retrieval (avoids single-file size limits)
Incremental Indexing: Automatically detects and skips already-indexed files
Concurrent Processing: Configurable concurrency for optimal performance
Persistent Storage: Vector embeddings are stored locally and persist across sessions
Supported File Types
Documents: PDF, EPUB, TXT, TEXT
Markdown: MD, MDX, Markdown, MDown, MKD, MKDN
Web Content: HTM, HTML, XHTML
Images (with OCR): BMP, JPEG, JPG, PNG
Archives: RAR (planned - currently not implemented)
Installation
Navigate to the plugin directory:
cd big-rag-plugin
Install dependencies:
npm install
Build the plugin:
npm run build
Run in development mode:
npm run dev
Configuration
The plugin provides the following configuration options in LM Studio:
Required Settings
Documents Directory: Root directory containing your documents (read access required)
Vector Store Directory: Where the vector database will be stored (read/write access required)
Embedding model
Embedding Model (plugin setting): String passed to LM Studio’s embedding load API. Both common forms can work for the same weights—for example mixedbread-ai/mxbai-embed-large-v1 (Hub / lms get) and text-embedding-mxbai-embed-large-v1 (as shown in lms ls). Use one spelling consistently for indexing and retrieval so it matches .big-rag-embedding.json; switching spelling without reindexing can trigger a mismatch warning. Default: nomic-ai/nomic-embed-text-v1.5-GGUF.
After changing the embedding model, run a full reindex (toggle Manual Reindex Trigger with Skip Previously Indexed Files off, or clear the vector store and let first-run indexing rebuild). Vectors from different models are not comparable in the same index.
.big-rag-embedding.json: Written under the vector store directory when the index has at least one chunk; records the model id and vector length used to build the index. If the configured model no longer matches this file, retrieval is blocked until you reindex or revert the setting. If the index has zero chunks, this file is removed so metadata cannot drift (including after manual shard deletion).
Indexes built with older plugin versions may have chunks but no manifest; retrieval still works, and a full reindex will create the manifest.
Retrieval Settings
Retrieval Limit (1-20, default: 5): Maximum number of chunks to return
Chunk Size (128-2048 tokens, default: 512): Size of text chunks for embedding
Chunk Overlap (0-512 tokens, default: 100): Overlap between consecutive chunks
Performance Settings
Max Concurrent Files (1-10, default: 1): Number of files to process simultaneously
Parser Delay (ms) (0-5000, default: 500): Wait time before parsing each document, inserted to help avoid WebSocket throttling against LM Studio
Enable OCR (default: true): Enable OCR for image files and image-based PDFs using LM Studio's built-in document parser
File Filtering
Exclude filename patterns (optional): One glob pattern per line, matched against each file's path relative to the Documents Directory (forward slashes). Lines starting with # are comments. Example: *.png excludes PNGs anywhere; archive/** excludes that subtree. This only prevents new files from being parsed/embedded — it does not remove chunks already in the vector store, so reindex or clear the store to drop previously indexed matches.
Reindexing Controls
Manual Reindex Trigger (toggle): Turn this ON and submit any chat message to force indexing to run on every chat session where the plugin is enabled. Flip it OFF once you’re done to stop the automatic reindex loop.
Skip Previously Indexed Files (default: true): If enabled while "Manual Reindex Trigger" is enabled, each manual run touches just the documents that are new or have changed since the last index (files that previously failed to parse are also skipped); if disabled, every chat rebuilds the entire index from scratch. Combine "Skip Previously Indexed Files" and "Manual Reindex Trigger" to choose between incremental updates or repeated full refreshes.
Automatic First-Run: If the vector store is empty, the plugin automatically indexes the configured documents the first time any chat message is processed—no manual input is required.
Indexing Lock: Only one indexing run (automatic or manual) can be active at a time; if you trigger a manual reindex while one is already running, the plugin reports it and skips the new request instead of running two jobs concurrently.
Prompt Template
Prompt Template (plugin setting): Customize how the retrieved passages and user query are assembled into the final prompt sent to the model. Must contain the {{rag_context}} and {{user_query}} macros — if either is missing, the plugin logs a warning and inserts it automatically so retrieval still works. Default is a simple "use these citations if relevant" instruction followed by the user's query.
Usage
Configure the Plugin:
Open LM Studio settings
Navigate to the Big RAG plugin configuration
Set your documents directory (e.g., /Users/user/Documents/MyLibrary)
Set your vector store directory (e.g., /Users/user/.lmstudio/big-rag-db)
Initial Indexing:
The first time you send a message, the plugin will automatically scan and index your documents
This process may take a while depending on the size of your document collection
Progress will be shown in the LM Studio interface
Query Your Documents:
Simply chat with your LM Studio model as usual
The plugin will automatically search your indexed documents for relevant content
Retrieved passages will be injected into the context for the model to use
Architecture
Components
File Scanner (src/ingestion/fileScanner.ts):
Recursively scans directories
Filters for supported file types
Applies exclude filename patterns before a file is ever parsed
Collects file metadata
Document Parsers (src/parsers/):
htmlParser.ts: Extracts text from HTML/HTM files
pdfParser.ts: Extracts text from PDF files via a three-stage fallback: LM Studio's built-in parseDocument API, then pdf-parse, then MuPDF-rendered page images OCR'd with Tesseract (capped at 50 pages); each stage records a specific failure reason if it produces too little text
epubParser.ts: Extracts text from EPUB files
textParser.ts: Reads plain text & Markdown files with optional Markdown stripping
imageParser.ts: OCR for image files
documentParser.ts: Routes to appropriate parser
Vector Store (src/vectorstore/vectorStore.ts):
Uses Vectra with sharded indexes (one shard in memory at a time; avoids V8 string size limits)
Supports incremental updates
Efficient similarity search
Index Manager (src/ingestion/indexManager.ts):
Orchestrates the indexing pipeline
Manages concurrent processing
Skips files that previously failed to parse (via the failed-file registry) during incremental runs
Handles progress reporting and per-run failure reason summaries/reports
Shared indexing pipeline reused by the plugin's automatic/manual triggers and by the standalone CLI
The CLI (npm run index:cli) indexes a documents directory into a vector store outside of LM Studio chat, configured entirely via BIG_RAG_* environment variables
Supporting Utilities (src/utils/):
sanityChecks.ts: Validates directory access, disk space, and free memory before the first indexing run and estimates dataset size/time
indexingLock.ts: Ensures only one indexing job runs at a time
failedFileRegistry.ts: Persists per-file failure reasons so unchanged, previously-failed files are skipped on incremental reindexes
fileExcludePatterns.ts: Parses and matches glob-based exclude patterns (from plugin config or BIG_RAG_EXCLUDE_PATTERNS)
Prompt Preprocessor (src/promptPreprocessor.ts):
Intercepts user queries
Performs vector search
Injects relevant context using the configurable prompt template
Performance Considerations
Large Datasets
Disk Space: The vector store requires additional disk space (typically 10-20% of original document size)
Initial Indexing: Can take several hours for TB-scale collections
Memory Usage: Scales with concurrent processing (reduce maxConcurrentFiles if needed)
Optimization Tips
Start Small: Test with a subset of documents first
Disable OCR: Unless you have many image-based documents, keep OCR disabled
Adjust Concurrency: Lower maxConcurrentFiles on systems with limited resources
Chunk Size: Larger chunks (1024-2048) work better for technical documents
Threshold Tuning: Adjust retrievalAffinityThreshold based on result quality
Troubleshooting
No Results Found
Check that documents directory is correctly configured
Verify that indexing completed successfully
Try lowering the retrieval affinity threshold
Check LM Studio logs for errors
Embedding model mismatch
If you see a message that the index was built with a different embedding model than the one in settings, either change Embedding Model back to the value recorded in .big-rag-embedding.json or run a full reindex after changing the model.
Dimension mismatch means the model’s output size changed; reindex after switching models or quantizations.
Slow Indexing
Reduce maxConcurrentFiles
Disable OCR if not needed
Ensure vector store directory is on a fast drive (SSD recommended)
Out of Memory
Reduce maxConcurrentFiles to 1 or 2
Process documents in batches by organizing them into subdirectories
Increase system swap space
OCR Not Working
Tesseract.js downloads language data on first use
Ensure internet connectivity during first OCR operation
Check that image files are valid and readable
Headless CLI Indexing
The plugin's indexing pipeline can also run outside of LM Studio chat via src/cliIndex.ts, useful for scripted or scheduled indexing of large collections:
npm run buildnode dist/cliIndex.js /path/to/documents /path/to/vector/store# orBIG_RAG_DOCS_DIR=/path/to/documents BIG_RAG_DB_DIR=/path/to/vector/store node dist/cliIndex.js
Configured entirely via environment variables:
BIG_RAG_DOCS_DIR / BIG_RAG_DB_DIR: documents and vector store directories (or pass as positional args)
BIG_RAG_EMBEDDING_MODEL: overrides the default embedding model id (same default as the plugin's Embedding Model setting)
BIG_RAG_FORCE_REINDEX (default: false): when true, rebuilds every file instead of skipping unchanged ones
BIG_RAG_PARSE_DELAY_MS (default: 500)
BIG_RAG_EXCLUDE_PATTERNS: semicolon-separated glob patterns (same syntax as the plugin's exclude filename patterns field)
BIG_RAG_FAILURE_REPORT_PATH: absolute path to write a JSON failure report to after indexing
Failure Reason Reporting
The CLI logs cumulative success / failed counts after each processed document.
Set BIG_RAG_FAILURE_REPORT_PATH=/absolute/path/report.json when running npm run index (or via LM Studio env settings) to emit a JSON report containing all failure reasons and counts after indexing completes. This is useful when triaging stubborn PDFs such as blueprints or large scanned books.
BIG_RAG_EMBEDDING_MODEL: Optional. When set for headless indexing (npm run index:cli / dist/cliIndex.js), overrides the default embedding model id (same default as the plugin’s Embedding Model setting). Empty/unset uses the built-in default from config.ts.
Limitations
RAR Archives: Not yet implemented (files are skipped)
Password-Protected Files: Not supported
Very Large Files: Individual files >100MB may cause memory issues
Non-English OCR: Currently only English OCR is configured