big-rag
Public
Forked from mindstudio/big-rag
Forked from mindstudio/big-rag
Project Files
README.md
A powerful RAG (Retrieval-Augmented Generation) plugin for LM Studio that can index and search through gigabytes or even terabytes (not tested) of document data. Hosted here: ari99/lm_studio_big_rag_plugin on GitHub.
The plugin provides the following configuration options in LM Studio:
maxConcurrentFiles if needed)maxConcurrentFiles on systems with limited resources.big-rag-embedding.json or run a full reindex after changing the model.maxConcurrentFilesmaxConcurrentFiles to 1 or 2Automated parser smoke tests cover HTML, Markdown, and plain text ingestion:
For end-to-end validation:
This plugin is based on the LM Studio plugin SDK. For more information:
ISC
mixedbread-ai/mxbai-embed-large-v1 (Hub / lms get) and text-embedding-mxbai-embed-large-v1 (as shown in lms ls). Use one spelling consistently for indexing and retrieval so it matches .big-rag-embedding.json; switching spelling without reindexing can trigger a mismatch warning. Default: nomic-ai/nomic-embed-text-v1.5-GGUF..big-rag-embedding.json: Written under the vector store directory when the index has at least one chunk; records the model id and vector length used to build the index. If the configured model no longer matches this file, retrieval is blocked until you reindex or revert the setting. If the index has zero chunks, this file is removed so metadata cannot drift (including after manual shard deletion).Configure the Plugin:
/Users/user/Documents/MyLibrary)/Users/user/.lmstudio/big-rag-db)Initial Indexing:
Query Your Documents:
File Scanner (src/ingestion/fileScanner.ts):
Document Parsers (src/parsers/):
htmlParser.ts: Extracts text from HTML/HTM filespdfParser.ts: Extracts text from PDF filesepubParser.ts: Extracts text from EPUB filestextParser.ts: Reads plain text & Markdown files with optional Markdown strippingimageParser.ts: OCR for image filesdocumentParser.ts: Routes to appropriate parserVector Store (src/vectorstore/vectorStore.ts):
Index Manager (src/ingestion/indexManager.ts):
Prompt Preprocessor (src/promptPreprocessor.ts):
retrievalAffinityThresholdsuccessfailedBIG_RAG_FAILURE_REPORT_PATH=/absolute/path/report.json when running npm run index (or via LM Studio env settings) to emit a JSON report containing all failure reasons and counts after indexing completes. This is useful when triaging stubborn PDFs such as blueprints or large scanned books.BIG_RAG_EMBEDDING_MODEL: Optional. When set for headless indexing (npm run index:cli / dist/cliIndex.js), overrides the default embedding model id (same default as the plugin’s Embedding Model setting). Empty/unset uses the built-in default from config.ts.cd big-rag-plugin
npm install
npm run build
npm run dev
big-rag-plugin/
├── src/
│ ├── config.ts # Plugin configuration schema
│ ├── index.ts # Main entry point
│ ├── promptPreprocessor.ts # RAG integration
│ ├── ingestion/
│ │ ├── fileScanner.ts # Directory scanning
│ │ └── indexManager.ts # Indexing orchestration
│ ├── parsers/
│ │ ├── documentParser.ts # Parser router
│ │ ├── htmlParser.ts # HTML parsing
│ │ ├── pdfParser.ts # PDF parsing
│ │ ├── epubParser.ts # EPUB parsing
│ │ ├── textParser.ts # Text parsing
│ │ └── imageParser.ts # OCR parsing
│ ├── vectorstore/
│ │ └── vectorStore.ts # Vectra sharded index integration
│ └── utils/
│ ├── coerceEmbedding.ts # Normalize embedding API vectors
│ ├── embeddingIndexManifest.ts # Index embedding metadata on disk
│ ├── fileHash.ts # File hashing
│ └── textChunker.ts # Text chunking
├── manifest.json # Plugin manifest
├── package.json # Dependencies
├── tsconfig.json # TypeScript config
└── README.md # This file
npm run test