Improvements (v1.1.0)

• Chunker fixed: chunkSize and chunkOverlap are now correctly interpreted as tokens, not words. Chunks match the configured size.
• Sentence-boundary chunking: new "Chunking Strategy" setting for higher-quality semantically coherent chunks (slower, better retrieval).
• Parallel chunk embedding: chunks are embedded in batches of 8, significantly reducing indexing time for large documents.
• VectorStore stats cache: getStats() and hasFile() now use cached or in-memory lookups after load, reducing redundant disk reads per chat turn.
• Min chunk length filter: short/noise chunks are discarded before embedding (configurable, default: 20 tokens).
• New file formats: .docx, .odt, .csv, .tsv, .json, .jsonl, .yaml, .yml, .rst, and .log are now indexed.
• Config change detection: changing documentsDirectory, vectorStoreDirectory, or embeddingModel in settings resets the in-memory store and sanity-check cache.
• Max query length: very long user messages are truncated before embedding to match embedding model input limits (configurable, default: 512 tokens).

Supported File Types

• Documents: PDF, EPUB, DOCX, ODT, TXT, TEXT, RST, LOG
• Structured text: CSV, TSV, JSON, JSONL, YAML, YML
• Markdown: MD, Markdown, MDown, MDX, MKD, MKDN
• Web content: HTM, HTML, XHTML
• Images with OCR: BMP, JPEG, JPG, PNG
• Archives: RAR is detected but not implemented yet

Installation

cd big-rag-plugin
npm install
npm run build
npm run dev

npm run build expects LM Studio's generated .lmstudio/entry.ts to exist. If you are working from a raw source checkout without that file, npx tsc still validates TypeScript compilation.

Configuration

Required Settings

• Documents Directory: root directory containing files to index.
• Vector Store Directory: directory where the sharded vector store and metadata are written.

Embedding

• Embedding Model: LM Studio embedding model id. Use one spelling consistently for indexing and retrieval. If you change this value, run a full reindex because vectors from different models are not comparable.

Retrieval Settings

• Retrieval Limit: maximum number of passages returned for each query. Default: 5.
• Retrieval Affinity Threshold: minimum similarity score. Default: 0.5.
• Max Query Length: estimated-token limit for the user query before embedding. Longer queries are truncated. Default: 512.

Chunking Settings

• Chunk Size: target chunk size in estimated tokens. Default: 512.
• Chunk Overlap: overlap between consecutive chunks in estimated tokens. Default: 100.
• Chunking Strategy: character uses fixed character windows and is fastest. sentence groups complete sentences for better semantic coherence. Default: character.
• Min Chunk Length: chunks shorter than this estimated-token count are discarded before embedding. Set to 0 to keep all chunks. Default: 20.

Indexing Settings

• Max Concurrent Files: number of files processed at once. Default: 1.
• Parser Delay (ms): delay before parsing each file to reduce local service pressure. Default: 500.
• Enable OCR: enables OCR for images and image-based PDFs. Default: true.
• Exclude Filename Patterns: one glob per line, matched against paths relative to Documents Directory.

Reindexing Controls

• Manual Reindex Trigger: turn on and send a chat message to run indexing.
• Skip Previously Indexed Files: when manual reindex is on, skip unchanged files if enabled; rebuild all files if disabled.
• Automatic First Run: if the vector store is empty, the first chat message triggers indexing automatically.

CLI Indexing

After compiling with npx tsc, run:

node dist/cliIndex.js /path/to/documents /path/to/vector-store

Useful environment variables:

• BIG_RAG_EMBEDDING_MODEL
• BIG_RAG_CHUNK_SIZE
• BIG_RAG_CHUNK_OVERLAP
• BIG_RAG_CHUNKING_STRATEGY (character or sentence)
• BIG_RAG_MIN_CHUNK_LENGTH
• BIG_RAG_MAX_CONCURRENT
• BIG_RAG_ENABLE_OCR
• BIG_RAG_PARSE_DELAY_MS
• BIG_RAG_EXCLUDE_PATTERNS (semicolon-separated)
• BIG_RAG_FAILURE_REPORT_PATH
• BIG_RAG_FORCE_REINDEX

Architecture

• src/promptPreprocessor.ts: reads config, initializes the vector store, runs retrieval, and injects RAG context.
• src/ingestion/indexManager.ts: scans, parses, chunks, embeds, and writes document chunks.
• src/parsers/: file-type parsers for HTML, PDF, EPUB, text, images, and Office documents.
• src/utils/textChunker.ts: character and sentence chunking plus token estimation.
• src/vectorstore/vectorStore.ts: sharded Vectra storage, search, stats cache, and file hash index.

Testing

npx tsc
node --test dist/tests/parseDocument.test.js dist/tests/fileExcludePatterns.test.js

Use npm run test when the LM Studio-generated .lmstudio/entry.ts file is present because that script runs the full build first.

Troubleshooting

• No results found: confirm indexing completed, lower the affinity threshold, and check that the file type is supported.
• Embedding model mismatch: use the model recorded in .big-rag-embedding.json or run a full reindex.
• Slow indexing: reduce OCR usage, keep vector store on SSD, and tune maxConcurrentFiles.
• Too much retrieval noise: raise retrievalAffinityThreshold, use sentence chunking, or increase minChunkLength.
• Context overflow: lower retrieval limit, reduce chunk size, or use a model with a larger context window.

License

ISC