README
README
π‘ RSS RAG Cache Plugin for LM Studio
Turn LM Studio into your own private, intelligent news archive. This plugin fetches your favorite RSS feeds, chunks the articles, embeds them using your local LM Studio server, and stores them in a local SQLite database for Retrieval-Augmented Generation (RAG).
Instead of searching the web, your LLM can instantly query your local cache of recent articles, blog posts, and newsletters.
β¨ Features
porter unicode61 for real BM25 word ranking) with a resident in-memory vector index (falling back to a bounded SQL scan when the matrix would exceed the memory ceiling), fused by Reciprocal Rank Fusion. Active filters (feed / author / tag / collection / date range) are resolved to an entry-ID allow-list before ranking, so both keyword lists and the vector scan only score in-scope articles β a filtered search on a large cache costs time proportional to the matches, not to the cache. If the embedding model is offline, it automatically falls back to pure keyword search.title | section breadcrumb + newline + chunk (embed style contextual-v2), so summary-only feeds and mid-article passages still produce self-contained vectors. Document/query inputs then receive model-aware asymmetric prefixes (search_document: / search_query: for nomic-embed-text-*, Represent this sentenceβ¦ for bge/mxbai/snowflake). The stored/displayed text is unchanged. Switchable via contextualChunks; caches on older embed styles are upgraded lazily by reindex_missing_embeddings.<h1>β<h3> are preserved as Markdown headings during extraction, and each chunk records the heading path that was active where the chunk ends (e.g. Title > Setup > Windows), so expanded context tells the model where a passage came from.contextChunks neighbouring chunks, so the returned snippet carries surrounding context rather than an isolated passage.Float32Array, so a semantic query is one tight dot-product loop with zero per-query allocation. The index is then maintained in place: newly indexed chunks append into recycled slots (tombstones tracked through an O(1) free list) and removed articles are tombstoned, so refreshes and retention no longer rebuild the whole matrix. It still rebuilds on a dimension change or when tombstones exceed 20% of capacity; a configurable byte ceiling (RSS_RAG_INDEX_MAX_MB, default 256 MB, with 15% headroom) falls back to the streaming scan when exceeded.manage_rss_feeds accepts a blog homepage and follows <link rel="alternate" type="application/rss+xml|atom+xml"> to the real feed, adding that instead. The granular test_feed reports any discovered alternates.feeds.json supports a per-feed retentionDays override and a collection label; retention honours the override, and search can be scoped with the collection filter.search_rss_cache, get_rss_article, get_recent_cache_articles, manage_rss_feeds β so every chat only pays the token cost of 4 tool schemas. One multiplexed admin_rss_cache tool (gated by enableAdminTools) covers diagnostics/optimization/reindexing/debug, and the legacy one-purpose tools come back with exposeGranularAdminTools.get_rss_article returns a cached article's complete text and metadata given an entryId or canonical url, with offset/maxChars paging (default 12,000 chars) and an optional chunk layout (includeChunks: true) β so the model can read past the 600-character search snippet.admin_rss_cache with action: "debug_search" (or the granular search_rss_cache_debug) explains each result β per-list rank, RRF contribution, cosine similarity, fused score β for observability without changing behaviour.npm run eval indexes a frozen corpus against a deterministic fake embedding server and reports recall@1/3/5/10, MRR, nDCG@10, and latency p50/p95, gating on regressions versus a committed baseline.<article> / content selectors, strips boilerplate, preserves paragraph breaks and heading structure, and falls back to Open Graph / meta descriptions.feeds.json): Optionally override enabled, tags, scrapeFullText, linkPatterns, maxItems, and request headers for individual feeds; tags become a search filter.days/after/before, feed, author, or tag; sort by relevance or date; paginate with limit/offset. Filters are applied before ranking, not after.cache_diagnostics (or admin_rss_cache with action: "diagnostics") reports coverage, sizes, feed health, and suggested fixes.If-None-Match / If-Modified-Since (ETag / Last-Modified) so unchanged feeds answer 304 Not Modified and cost almost no bandwidth. A manual refresh (or force: true) re-downloads everything.403 that carries Retry-After) using the header (capped at 15s). Feeds that keep failing are placed in an exponential backoff (15 min doubling to 24 h) and skipped until their retry time, so one broken feed can't slow every refresh.admin_rss_cache with action: "reembed" rebuilds the index.news, article, post, ...).maxConcurrentPerHost requests at once against the same host and spaces them by minRequestGapMs + jitter (1.5β3 s by default). HTML pages are scraped one at a time, and robots.txt is respected before scraping or fetching full article text.β οΈ BAN warning, and the offending host is paused (30 min, doubling to 12 h). Paused hosts appear as blockedHosts in the refresh result, so a block is never silent.VACUUM are available from chat through manage_rss_feeds and admin_rss_cache; the legacy granular tools remain available behind exposeGranularAdminTools.node:sqlite module (WAL mode). No C++ addons or Visual Studio build tools required!PRAGMA user_version migration runner (currently version 11), so new columns/indexes and data upgrades (like the 64-bit SimHash backfill) are applied safely to existing databases.PRAGMA optimize and VACUUMs the database only when enough free pages have accumulated (so routine refreshes stay fast).feeds.txt and cache.db in ~/.lmstudio/plugin-data/rss-rag-cache/ so they survive plugin updates.π Requirements
nomic-embed-text-v1.5) and start the local server on port 1234.π Installation
feeds.txt in your plugin-data folder on the first run.βοΈ Configuration
1. Adding Feeds (feeds.txt)
Your feeds file is located at:
C:\Users\YOUR_USERNAME\.lmstudio\plugin-data\rss-rag-cache\feeds.txt~/.lmstudio/plugin-data/rss-rag-cache/feeds.txtAdd one RSS/Atom URL per line. You can also add standard webpage URLs (the plugin will scrape them automatically). Lines starting with # are ignored.
2. Plugin Settings UI
Cache and indexing settings are global (they configure the shared on-disk
cache). Only topK is per-chat, since it governs how much context a single
conversation pulls in.
Global plugin settings:
| Setting | Default | Description |
|---|---|---|
retentionDays | 90 | How long to keep articles (30, 60, 90, or 365 days). |
refreshIntervalHours | 6 | How often the plugin automatically fetches new RSS items in the background. |
embeddingModel | nomic-embed-text-v1.5 | The exact model ID loaded in your LM Studio server. |
chunkSize | 800 | Length (in characters) of each text chunk used for embedding and search. |
embedBatchSize | 16 | How many chunks are sent to the embedding model per request (higher = faster but more VRAM). |
embedConcurrency | 3 | How many embedding requests run in parallel during indexing/backfill. |
maxConcurrentFeeds | 4 | How many RSS/Atom feeds are fetched at the same time (across all hosts). |
maxConcurrentPerHost | 2 | Cap on simultaneous requests to any single domain. Low values keep anti-bot firewalls calm. |
minRequestGapMs | 1500 | Minimum delay between two requests to the same host. Higher = safer, slower. |
requestGapJitterMs | 1500 | Random extra delay added to each per-host gap so the crawl pattern is not perfectly regular. |
respectRobotsTxt | On | Check robots.txt before scraping HTML pages or fetching full article text. |
Per-chat settings:
| Setting | Default | Description |
|---|---|---|
topK | 8 | How many unique articles to return per search. |
3. Per-Feed Options (feeds.json)
For fine-grained control, create feeds.json next to feeds.txt (or use the set_feed_options tool). Feeds listed in feeds.txt are merged with their JSON overrides:
Supported fields per feed: enabled, tags, scrapeFullText, linkPatterns, maxItems, headers, retentionDays, collection. Set enabled: false to keep a feed configured but skip fetching it, and use tags and collection to filter searches.
π οΈ LM Studio Tools
The default surface is intentionally small β 4 tools β so each chat only pays the token cost of 4 tool schemas.
Default tools (always registered)
Optional: enableAdminTools
Optional: exposeGranularAdminTools (adds admin_rss_cache plus the legacy one-purpose tools)
π§ͺ Development
Contributor notes:
π§ How It Works
π Troubleshooting
Search returns 0 matches
Ensure your embedding model is loaded and the server is running. If the model is offline, the plugin falls back to keyword search β but if your articles don't contain the exact keywords, they won't be found. Run manage_rss_feeds with action: "list" to check that chunks > 0; with enableAdminTools on, admin_rss_cache with action: "reindex" fills the gaps once the model is loaded.
HTTP 403 Forbidden on a feed
Some websites block automated requests. The plugin uses a Chrome User-Agent, robots.txt, per-host throttling, and never retries a hard 403 β but strict firewalls might still block it. When that happens the host is paused automatically and a β οΈ BAN warning names it, with the paused hosts listed under blockedHosts in the refresh result. If you control the site, set contactUserAgent so they can whitelist you; otherwise use an alternative RSS URL or an RSS Bridge service.
HTTP 429 Too Many Requests
Sites like Reddit aggressively block Node.js fetch requests. If this happens, use an RSS Bridge URL (e.g., https://rss-bridge.org/bridge01/?action=display&bridge=RedditBridge&format=Atom) instead of the native .rss link.
I changed the embedding model and search got worse
Vectors from different models aren't comparable. The plugin logs a dimension-mismatch / model-change warning and manage_rss_feeds (action: "list") shows the model and dimension the cache was built with. Run admin_rss_cache with action: "reembed" (enable enableAdminTools, or use reembed_all when granular tools are exposed) with the new model loaded to rebuild the index.
A feed keeps failing
Repeated failures trigger exponential backoff (15 min β 24 h); manage_rss_feeds with action: "list" shows each feed's failure count and next retry time. Fix the URL, or run manage_rss_feeds with action: "refresh" and force: true to retry immediately.
π¦ Changelog
1.5.0
1.4.0
1.3.0
1.2.0
1.1.0
1.0.0
π‘ RSS RAG Cache Plugin for LM Studio
Turn LM Studio into your own private, intelligent news archive. This plugin fetches your favorite RSS feeds, chunks the articles, embeds them using your local LM Studio server, and stores them in a local SQLite database for Retrieval-Augmented Generation (RAG).
Instead of searching the web, your LLM can instantly query your local cache of recent articles, blog posts, and newsletters.
β¨ Features
porter unicode61 for real BM25 word ranking) with a resident in-memory vector index (falling back to a bounded SQL scan when the matrix would exceed the memory ceiling), fused by Reciprocal Rank Fusion. Active filters (feed / author / tag / collection / date range) are resolved to an entry-ID allow-list before ranking, so both keyword lists and the vector scan only score in-scope articles β a filtered search on a large cache costs time proportional to the matches, not to the cache. If the embedding model is offline, it automatically falls back to pure keyword search.title | section breadcrumb + newline + chunk (embed style contextual-v2), so summary-only feeds and mid-article passages still produce self-contained vectors. Document/query inputs then receive model-aware asymmetric prefixes (search_document: / search_query: for nomic-embed-text-*, Represent this sentenceβ¦ for bge/mxbai/snowflake). The stored/displayed text is unchanged. Switchable via contextualChunks; caches on older embed styles are upgraded lazily by reindex_missing_embeddings.<h1>β<h3> are preserved as Markdown headings during extraction, and each chunk records the heading path that was active where the chunk ends (e.g. Title > Setup > Windows), so expanded context tells the model where a passage came from.contextChunks neighbouring chunks, so the returned snippet carries surrounding context rather than an isolated passage.Float32Array, so a semantic query is one tight dot-product loop with zero per-query allocation. The index is then maintained in place: newly indexed chunks append into recycled slots (tombstones tracked through an O(1) free list) and removed articles are tombstoned, so refreshes and retention no longer rebuild the whole matrix. It still rebuilds on a dimension change or when tombstones exceed 20% of capacity; a configurable byte ceiling (RSS_RAG_INDEX_MAX_MB, default 256 MB, with 15% headroom) falls back to the streaming scan when exceeded.manage_rss_feeds accepts a blog homepage and follows <link rel="alternate" type="application/rss+xml|atom+xml"> to the real feed, adding that instead. The granular test_feed reports any discovered alternates.feeds.json supports a per-feed retentionDays override and a collection label; retention honours the override, and search can be scoped with the collection filter.search_rss_cache, get_rss_article, get_recent_cache_articles, manage_rss_feeds β so every chat only pays the token cost of 4 tool schemas. One multiplexed admin_rss_cache tool (gated by enableAdminTools) covers diagnostics/optimization/reindexing/debug, and the legacy one-purpose tools come back with exposeGranularAdminTools.get_rss_article returns a cached article's complete text and metadata given an entryId or canonical url, with offset/maxChars paging (default 12,000 chars) and an optional chunk layout (includeChunks: true) β so the model can read past the 600-character search snippet.admin_rss_cache with action: "debug_search" (or the granular search_rss_cache_debug) explains each result β per-list rank, RRF contribution, cosine similarity, fused score β for observability without changing behaviour.npm run eval indexes a frozen corpus against a deterministic fake embedding server and reports recall@1/3/5/10, MRR, nDCG@10, and latency p50/p95, gating on regressions versus a committed baseline.<article> / content selectors, strips boilerplate, preserves paragraph breaks and heading structure, and falls back to Open Graph / meta descriptions.feeds.json): Optionally override enabled, tags, scrapeFullText, linkPatterns, maxItems, and request headers for individual feeds; tags become a search filter.days/after/before, feed, author, or tag; sort by relevance or date; paginate with limit/offset. Filters are applied before ranking, not after.cache_diagnostics (or admin_rss_cache with action: "diagnostics") reports coverage, sizes, feed health, and suggested fixes.If-None-Match / If-Modified-Since (ETag / Last-Modified) so unchanged feeds answer 304 Not Modified and cost almost no bandwidth. A manual refresh (or force: true) re-downloads everything.403 that carries Retry-After) using the header (capped at 15s). Feeds that keep failing are placed in an exponential backoff (15 min doubling to 24 h) and skipped until their retry time, so one broken feed can't slow every refresh.admin_rss_cache with action: "reembed" rebuilds the index.news, article, post, ...).maxConcurrentPerHost requests at once against the same host and spaces them by minRequestGapMs + jitter (1.5β3 s by default). HTML pages are scraped one at a time, and robots.txt is respected before scraping or fetching full article text.β οΈ BAN warning, and the offending host is paused (30 min, doubling to 12 h). Paused hosts appear as blockedHosts in the refresh result, so a block is never silent.VACUUM are available from chat through manage_rss_feeds and admin_rss_cache; the legacy granular tools remain available behind exposeGranularAdminTools.node:sqlite module (WAL mode). No C++ addons or Visual Studio build tools required!PRAGMA user_version migration runner (currently version 11), so new columns/indexes and data upgrades (like the 64-bit SimHash backfill) are applied safely to existing databases.PRAGMA optimize and VACUUMs the database only when enough free pages have accumulated (so routine refreshes stay fast).feeds.txt and cache.db in ~/.lmstudio/plugin-data/rss-rag-cache/ so they survive plugin updates.π Requirements
nomic-embed-text-v1.5) and start the local server on port 1234.π Installation
feeds.txt in your plugin-data folder on the first run.βοΈ Configuration
1. Adding Feeds (feeds.txt)
Your feeds file is located at:
C:\Users\YOUR_USERNAME\.lmstudio\plugin-data\rss-rag-cache\feeds.txt~/.lmstudio/plugin-data/rss-rag-cache/feeds.txtAdd one RSS/Atom URL per line. You can also add standard webpage URLs (the plugin will scrape them automatically). Lines starting with # are ignored.
2. Plugin Settings UI
Cache and indexing settings are global (they configure the shared on-disk
cache). Only topK is per-chat, since it governs how much context a single
conversation pulls in.
Global plugin settings:
| Setting | Default | Description |
|---|---|---|
retentionDays | 90 | How long to keep articles (30, 60, 90, or 365 days). |
refreshIntervalHours | 6 | How often the plugin automatically fetches new RSS items in the background. |
embeddingModel | nomic-embed-text-v1.5 | The exact model ID loaded in your LM Studio server. |
chunkSize | 800 | Length (in characters) of each text chunk used for embedding and search. |
embedBatchSize | 16 | How many chunks are sent to the embedding model per request (higher = faster but more VRAM). |
embedConcurrency | 3 | How many embedding requests run in parallel during indexing/backfill. |
maxConcurrentFeeds | 4 | How many RSS/Atom feeds are fetched at the same time (across all hosts). |
maxConcurrentPerHost | 2 | Cap on simultaneous requests to any single domain. Low values keep anti-bot firewalls calm. |
minRequestGapMs | 1500 | Minimum delay between two requests to the same host. Higher = safer, slower. |
requestGapJitterMs | 1500 | Random extra delay added to each per-host gap so the crawl pattern is not perfectly regular. |
respectRobotsTxt | On | Check robots.txt before scraping HTML pages or fetching full article text. |
Per-chat settings:
| Setting | Default | Description |
|---|---|---|
topK | 8 | How many unique articles to return per search. |
3. Per-Feed Options (feeds.json)
For fine-grained control, create feeds.json next to feeds.txt (or use the set_feed_options tool). Feeds listed in feeds.txt are merged with their JSON overrides:
Supported fields per feed: enabled, tags, scrapeFullText, linkPatterns, maxItems, headers, retentionDays, collection. Set enabled: false to keep a feed configured but skip fetching it, and use tags and collection to filter searches.
π οΈ LM Studio Tools
The default surface is intentionally small β 4 tools β so each chat only pays the token cost of 4 tool schemas.
Default tools (always registered)
Optional: enableAdminTools
Optional: exposeGranularAdminTools (adds admin_rss_cache plus the legacy one-purpose tools)
π§ͺ Development
Contributor notes:
π§ How It Works
π Troubleshooting
Search returns 0 matches
Ensure your embedding model is loaded and the server is running. If the model is offline, the plugin falls back to keyword search β but if your articles don't contain the exact keywords, they won't be found. Run manage_rss_feeds with action: "list" to check that chunks > 0; with enableAdminTools on, admin_rss_cache with action: "reindex" fills the gaps once the model is loaded.
HTTP 403 Forbidden on a feed
Some websites block automated requests. The plugin uses a Chrome User-Agent, robots.txt, per-host throttling, and never retries a hard 403 β but strict firewalls might still block it. When that happens the host is paused automatically and a β οΈ BAN warning names it, with the paused hosts listed under blockedHosts in the refresh result. If you control the site, set contactUserAgent so they can whitelist you; otherwise use an alternative RSS URL or an RSS Bridge service.
HTTP 429 Too Many Requests
Sites like Reddit aggressively block Node.js fetch requests. If this happens, use an RSS Bridge URL (e.g., https://rss-bridge.org/bridge01/?action=display&bridge=RedditBridge&format=Atom) instead of the native .rss link.
I changed the embedding model and search got worse
Vectors from different models aren't comparable. The plugin logs a dimension-mismatch / model-change warning and manage_rss_feeds (action: "list") shows the model and dimension the cache was built with. Run admin_rss_cache with action: "reembed" (enable enableAdminTools, or use reembed_all when granular tools are exposed) with the new model loaded to rebuild the index.
A feed keeps failing
Repeated failures trigger exponential backoff (15 min β 24 h); manage_rss_feeds with action: "list" shows each feed's failure count and next retry time. Fix the URL, or run manage_rss_feeds with action: "refresh" and force: true to retry immediately.
π¦ Changelog
1.5.0
1.4.0
1.3.0
1.2.0
1.1.0
1.0.0
contactUserAgent | (empty) | Optional contact URL/email appended to the User-Agent so site owners can reach you instead of blocking you (e.g. https://example.com/bot). |
semanticScanLimit | 50000 | Maximum chunks scanned per semantic search (higher = better recall, slower). |
htmlLinkPatterns | news,article,post,blog,stories,press | Comma-separated URL keywords used when scraping non-RSS pages. |
contextualChunks | On | Embed each chunk as title | section breadcrumb + text so vectors are self-contained (improves recall on summary-only feeds). Changing it re-embeds the cache; older embed styles upgrade lazily. |
contextChunks | 1 | How many adjacent chunks to include around each match (0β2) for extra context. |
refreshOnStartup | On | Automatically fetch feeds when the plugin loads. |
lmStudioBaseUrl | http://localhost:1234 | Where the embedding model is served. |
enableAdminTools | Off | Registers the multiplexed admin_rss_cache tool (diagnostics, optimize, prune, reindex, reembed, debug search). |
exposeGranularAdminTools | Off | Registers the legacy one-purpose tools (refresh_rss_feeds, list_rss_feeds, add_rss_feed, β¦) in addition to admin_rss_cache. |
search_rss_cache β Hybrid (BM25 keyword + semantic) search. Filter by days, after/before (ISO date), feed (URL substring), author, tag, or collection; sort by relevance (default) or date; paginate with limit/offset. Filters are resolved to an entry-ID allow-list before ranking, results are recency-weighted, de-duplicated (SimHash), and diversified (MMR). Each hit includes expanded context (via contextChunks) and a snippet windowed around the matching query terms.get_rss_article β Full article reader: returns the complete stored text and metadata (title, link, feed, author, date, tags, collection, totalChars) for a cached article given an entryId or a url. Page through long articles with offset / maxChars (default 12,000, capped at 50,000) β the response reports hasMore / nextOffset β and pass includeChunks: true to also get the chunk layout with each chunk's embed style.get_recent_cache_articles β Returns the most recent cached articles. Use this to see what news is available without guessing keywords.manage_rss_feeds β One tool for every feed operation via action: list (feeds with tags/enabled state, per-feed health β last status, failure count, next retry β cache stats, and the last refresh report), add (auto-discovers <link rel="alternate"> feeds from homepages and accepts per-feed options), remove, test, configure (per-feed feeds.json overrides), and refresh (pass force: true to re-download everything and ignore backoff).admin_rss_cache β Cache administration multiplexer via action: diagnostics (entry/chunk counts, embedding coverage and gaps, DB/WAL size, resident vector-index stats, feed backoff state, embedding model/dimension, schema version, suggested actions), optimize (WAL checkpoint, VACUUM, ANALYZE), prune (delete cached articles of feeds no longer configured), reindex (embed chunks stored text-only while the model was offline), reembed (rebuild the vector index with the current model; optional rechunk: true also re-chunks), and debug_search (the search pipeline with per-result ranking details: per-list ranks, RRF contribution, cosine similarity, fused score).refresh_rss_feeds β Refreshes all feeds listed in feeds.txt, indexes new items, generates embeddings, and applies the retention policy. Feeds are fetched conditionally (ETag / Last-Modified) and failing feeds back off; pass force: true to re-download everything and ignore backoff. Returns a report of successes, not-modified, skipped, and failed feeds.list_rss_feeds β Lists configured feeds (with their tags/enabled state) plus cache stats: entry/chunk counts, missing embeddings, DB size, the embedding model/dimension the cache was built with, per-feed health (last status, failure count, next retry), and the last refresh report.add_rss_feed β Adds an RSS/Atom feed URL or webpage URL to feeds.txt. If a homepage is given, it auto-discovers the alternate feed via <link rel="alternate"> and adds the discovered feed URL instead (returns requestedUrl, discoveredFeedUrl, type, and any candidates).remove_rss_feed β Removes a feed URL from feeds.txt (and any feeds.json override). Cached articles are kept until pruned.set_feed_options β Sets per-feed options in feeds.json (enabled, tags, scrapeFullText, linkPatterns, maxItems, headers, retentionDays, collection). Only the supplied fields change.test_feed β Checks whether a URL is a working RSS/Atom feed or scrapeable HTML page, and how many items it exposes. When the URL is an HTML page, it also returns a discovered object listing any alternate feeds found.prune_removed_feeds β Deletes cached articles whose source feed is no longer in feeds.txt.reindex_missing_embeddings β Re-embeds chunks stored as text-only while the embedding model was offline, and re-chunks any entries that ended up with no chunks. Also lazily upgrades older embed styles.reembed_all β Rebuilds the entire vector index with the currently configured embedding model. Use after switching embedding models (or when a dimension-mismatch warning appears). Optional rechunk: true also re-chunks with the current chunkSize.optimize_rss_cache β Runs a WAL checkpoint, VACUUM, and ANALYZE to reclaim disk space (e.g. after retention deletes) and refresh query statistics. A lighter version of this (PRAGMA optimize, and VACUUM when enough free pages accumulate) now runs automatically after each refresh.search_rss_cache_debug β Diagnostic version of search_rss_cache that returns per-result ranking details (_debug) including per-list ranks, RRF contribution, cosine similarity, and the fused combined score. No behaviour change to the non-debug search.cache_diagnostics β Reports cache health in one call: entry/chunk counts, embedding coverage and gaps, DB/WAL size, feed backoff state, embedding model/dimension, schema version, and suggested next actions."type": "module" to package.json. LM Studio bundles the
plugin to a CommonJS .lmstudio/dev.js; marking the package as ESM would make
Node load that bundle as ESM and break its require() calls.lib/*.js
files (Node 22.7+).RSS_RAG_DATA_DIR to relocate the data directory. The test suite uses it
to run against a throwaway SQLite database.lib/toolSurface.js
(resolveToolSurface); keep it in sync with the toolsMap in
src/toolsProvider.ts.fast-xml-parser (with a legacy regex fallback). If a URL returns HTML, it uses Cheerio to scrape the page for article links.maxConcurrentFeeds overall, and at most maxConcurrentPerHost per domain) with If-None-Match / If-Modified-Since, so unchanged feeds return 304 and send no body. Consecutive requests to the same host are spaced by minRequestGapMs + random jitter (default 1.5β3 s), so a burst never looks like a scraper. HTML pages are fetched one-by-one and, because the pacing is per host, pages on different domains no longer wait on each other. 429 is retried with backoff (honouring Retry-After); a hard 403 is not retried (re-requesting a block is what turns a soft challenge into a permanent ban) unless it carries Retry-After. A feed that keeps failing enters exponential backoff (15 min β 24 h) so it is skipped until its retry time.h1βh3, preserved as Markdown) give each chunk a section breadcrumb β the heading path active where the chunk ends.title | breadcrumb + body (contextual-v2) and sent to LM Studio's /v1/embeddings endpoint in batches, with bounded request concurrency and model-aware document/query prefixes. If the model is offline, chunks are stored text-only and a backfill (admin_rss_cache β reindex, or automatically once the model is up) completes them later; caches built with older embed styles are upgraded the same way. Repeated search queries reuse a cached (prefixed) query embedding.cache.db) through a versioned migration runner (schema version 11). Newer items list first; retention purges by publish or cache date. When an existing entry's text changes, its chunks are transparently rebuilt.porter BM25 FTS5 list, and a bounded cosine-similarity scan β all restricted to those entries β using Reciprocal Rank Fusion, deduplicates by article (collapsing syndicated duplicates via SimHash), blends in a gentle recency boost, and diversifies with MMR before returning results. Each result includes a snippet windowed around the matching query terms; the resident vector index is updated in place as articles are added or removed.feeds_state table, and per-host ban state lives in host_state. When a site blocks the crawler (HTTP 403/429, a Cloudflare/Akamai/Incapsula challenge, or a JS wall served with 200), the plugin logs a loud β οΈ BAN warning, pauses the whole host (30 min, doubling to 12 h) so every feed on that domain stays quiet, and skips it on later refreshes until the backoff expires. The active list of paused hosts is returned by refresh_rss_feeds as blockedHosts (with a banWarning summary) and is stored in feeds_state/host_state so it survives restarts. The retention pass keeps planner statistics fresh and vacuums only when warranted; admin_rss_cache (or cache_diagnostics) summarizes coverage, sizes, feed health, and suggested fixes in one call.maxConcurrentPerHost requests in flight against any single domain, spaced by minRequestGapMs + requestGapJitterMs (default 1.5β3 s). Pages on different hosts no longer wait on each other, so wide fans-out stay fast while any one domain stays polite. New settings: maxConcurrentPerHost, minRequestGapMs, requestGapJitterMs, respectRobotsTxt, contactUserAgent; maxConcurrentFeeds default lowered 8 β 4.403 is no longer retried (only a 403 carrying Retry-After), since re-requesting a block is what escalates a soft challenge into a permanent ban. 429/5xx retry behaviour is unchanged.robots.txt is honoured before scraping HTML pages and before fetching full article text (never for RSS feed URLs, which are published for machines). Missing/unreadable robots.txt means "allowed", as every major crawler assumes.β οΈ BAN warning naming the host, modal challenge, and attempt count; the whole host is then paused (30 min, doubling to 12 h) and skipped on subsequent refreshes. New host_state table (schema version 11) persists this across restarts, and the active pause list is returned as blockedHosts (with a banWarning summary) by refresh_rss_feeds, so a block is never silent. A host that answers normally again clears its own ban record.lib/rateLimiter.js (per-host limiter), lib/robots.js (robots.txt parser + cache), lib/bans.js (bot-wall classification); new tests for all three plus test/fetcher.test.js covering no-retry-on-403, 429 retry, host backoff skips, robots vetoes, and cross-host pacing.contextual-v2): chunks embed as title | section breadcrumb + body, queries/documents receive model-specific prefixes (search_query:/search_document: for nomic, Represent this sentenceβ¦ for bge/mxbai/snowflake), and <h1>β<h3> are preserved as Markdown so each chunk knows its section. Older embed styles upgrade lazily (migrations 9β10; schema version 10).search_rss_cache, get_rss_article, get_recent_cache_articles, manage_rss_feeds). New full-article reader get_rss_article (by entry ID or URL, offset/maxChars paging, optional chunk layout), new multiplexed manage_rss_feeds (list/add/remove/test/configure/refresh), and new admin_rss_cache (diagnostics/optimize/prune/reindex/reembed/debug_search). enableAdminTools and exposeGranularAdminTools opt back into the admin and granular tools.vectorIndexStats() reporting live/tombstone/capacity counts.reindex_missing_embeddings.contextChunks).Float32Array with lazy rebuilds, zero per-query allocations, and a memory ceiling (RSS_RAG_INDEX_MAX_MB) falling back to the streaming SQL scan.add_rss_feed discovers <link rel="alternate"> feeds from blog homepages; test_feed reports discovered alternates.collection labels (feeds.json), with retention honouring overrides and search supporting a collection filter.search_rss_cache_debug tool for ranking introspection; cache_diagnostics reports vector-index and embed-style coverage.lib/eval.js, npm run eval, frozen corpus, deterministic baseline) for recall@1/3/5/10, MRR, nDCG@10, and latency.porter unicode61 BM25 FTS5 list, and a bounded vector scan.feeds.json (enabled, tags, scrapeFullText, linkPatterns, maxItems, headers) and a new set_feed_options tool.days/after/before, feed, author, tag), relevance/date sorting, and limit/offset pagination.cache_diagnostics tool and a versioned PRAGMA user_version migration runner.If-None-Match / If-Modified-Since) with per-feed exponential backoff, backed by a new feeds_state table.reembed_all tool rebuilds the index after a model change.PRAGMA optimize, plus VACUUM only when enough free pages accumulate).refresh_rss_feeds accepts a force flag; list_rss_feeds reports per-feed health and the model/dimension the cache was built with.reembedAll, ranking) and added a npm run typecheck script.https://news.ycombinator.com/rss
https://www.theverge.com/rss/index.xml
# Non-RSS webpages (scraped automatically, slower):
https://example.com/news
{
"feeds": [
{ "url": "https://news.ycombinator.com/rss", "tags": ["tech", "startups"], "maxItems": 50 },
{ "url": "https://example.com/private.xml", "headers": { "Authorization": "Bearer ..." } },
{ "url": "https://example.com/news", "enabled": false }
]
}
npm install
npm test # node:test suite (lib modules + a fake embedding server)
npm run typecheck # tsc --noEmit over src/*.ts
npm run eval # Phase 0 retrieval eval (recall@k/MRR/nDCG@10 + latency)
contactUserAgent | (empty) | Optional contact URL/email appended to the User-Agent so site owners can reach you instead of blocking you (e.g. https://example.com/bot). |
semanticScanLimit | 50000 | Maximum chunks scanned per semantic search (higher = better recall, slower). |
htmlLinkPatterns | news,article,post,blog,stories,press | Comma-separated URL keywords used when scraping non-RSS pages. |
contextualChunks | On | Embed each chunk as title | section breadcrumb + text so vectors are self-contained (improves recall on summary-only feeds). Changing it re-embeds the cache; older embed styles upgrade lazily. |
contextChunks | 1 | How many adjacent chunks to include around each match (0β2) for extra context. |
refreshOnStartup | On | Automatically fetch feeds when the plugin loads. |
lmStudioBaseUrl | http://localhost:1234 | Where the embedding model is served. |
enableAdminTools | Off | Registers the multiplexed admin_rss_cache tool (diagnostics, optimize, prune, reindex, reembed, debug search). |
exposeGranularAdminTools | Off | Registers the legacy one-purpose tools (refresh_rss_feeds, list_rss_feeds, add_rss_feed, β¦) in addition to admin_rss_cache. |
search_rss_cache β Hybrid (BM25 keyword + semantic) search. Filter by days, after/before (ISO date), feed (URL substring), author, tag, or collection; sort by relevance (default) or date; paginate with limit/offset. Filters are resolved to an entry-ID allow-list before ranking, results are recency-weighted, de-duplicated (SimHash), and diversified (MMR). Each hit includes expanded context (via contextChunks) and a snippet windowed around the matching query terms.get_rss_article β Full article reader: returns the complete stored text and metadata (title, link, feed, author, date, tags, collection, totalChars) for a cached article given an entryId or a url. Page through long articles with offset / maxChars (default 12,000, capped at 50,000) β the response reports hasMore / nextOffset β and pass includeChunks: true to also get the chunk layout with each chunk's embed style.get_recent_cache_articles β Returns the most recent cached articles. Use this to see what news is available without guessing keywords.manage_rss_feeds β One tool for every feed operation via action: list (feeds with tags/enabled state, per-feed health β last status, failure count, next retry β cache stats, and the last refresh report), add (auto-discovers <link rel="alternate"> feeds from homepages and accepts per-feed options), remove, test, configure (per-feed feeds.json overrides), and refresh (pass force: true to re-download everything and ignore backoff).admin_rss_cache β Cache administration multiplexer via action: diagnostics (entry/chunk counts, embedding coverage and gaps, DB/WAL size, resident vector-index stats, feed backoff state, embedding model/dimension, schema version, suggested actions), optimize (WAL checkpoint, VACUUM, ANALYZE), prune (delete cached articles of feeds no longer configured), reindex (embed chunks stored text-only while the model was offline), reembed (rebuild the vector index with the current model; optional rechunk: true also re-chunks), and debug_search (the search pipeline with per-result ranking details: per-list ranks, RRF contribution, cosine similarity, fused score).refresh_rss_feeds β Refreshes all feeds listed in feeds.txt, indexes new items, generates embeddings, and applies the retention policy. Feeds are fetched conditionally (ETag / Last-Modified) and failing feeds back off; pass force: true to re-download everything and ignore backoff. Returns a report of successes, not-modified, skipped, and failed feeds.list_rss_feeds β Lists configured feeds (with their tags/enabled state) plus cache stats: entry/chunk counts, missing embeddings, DB size, the embedding model/dimension the cache was built with, per-feed health (last status, failure count, next retry), and the last refresh report.add_rss_feed β Adds an RSS/Atom feed URL or webpage URL to feeds.txt. If a homepage is given, it auto-discovers the alternate feed via <link rel="alternate"> and adds the discovered feed URL instead (returns requestedUrl, discoveredFeedUrl, type, and any candidates).remove_rss_feed β Removes a feed URL from feeds.txt (and any feeds.json override). Cached articles are kept until pruned.set_feed_options β Sets per-feed options in feeds.json (enabled, tags, scrapeFullText, linkPatterns, maxItems, headers, retentionDays, collection). Only the supplied fields change.test_feed β Checks whether a URL is a working RSS/Atom feed or scrapeable HTML page, and how many items it exposes. When the URL is an HTML page, it also returns a discovered object listing any alternate feeds found.prune_removed_feeds β Deletes cached articles whose source feed is no longer in feeds.txt.reindex_missing_embeddings β Re-embeds chunks stored as text-only while the embedding model was offline, and re-chunks any entries that ended up with no chunks. Also lazily upgrades older embed styles.reembed_all β Rebuilds the entire vector index with the currently configured embedding model. Use after switching embedding models (or when a dimension-mismatch warning appears). Optional rechunk: true also re-chunks with the current chunkSize.optimize_rss_cache β Runs a WAL checkpoint, VACUUM, and ANALYZE to reclaim disk space (e.g. after retention deletes) and refresh query statistics. A lighter version of this (PRAGMA optimize, and VACUUM when enough free pages accumulate) now runs automatically after each refresh.search_rss_cache_debug β Diagnostic version of search_rss_cache that returns per-result ranking details (_debug) including per-list ranks, RRF contribution, cosine similarity, and the fused combined score. No behaviour change to the non-debug search.cache_diagnostics β Reports cache health in one call: entry/chunk counts, embedding coverage and gaps, DB/WAL size, feed backoff state, embedding model/dimension, schema version, and suggested next actions."type": "module" to package.json. LM Studio bundles the
plugin to a CommonJS .lmstudio/dev.js; marking the package as ESM would make
Node load that bundle as ESM and break its require() calls.lib/*.js
files (Node 22.7+).RSS_RAG_DATA_DIR to relocate the data directory. The test suite uses it
to run against a throwaway SQLite database.lib/toolSurface.js
(resolveToolSurface); keep it in sync with the toolsMap in
src/toolsProvider.ts.fast-xml-parser (with a legacy regex fallback). If a URL returns HTML, it uses Cheerio to scrape the page for article links.maxConcurrentFeeds overall, and at most maxConcurrentPerHost per domain) with If-None-Match / If-Modified-Since, so unchanged feeds return 304 and send no body. Consecutive requests to the same host are spaced by minRequestGapMs + random jitter (default 1.5β3 s), so a burst never looks like a scraper. HTML pages are fetched one-by-one and, because the pacing is per host, pages on different domains no longer wait on each other. 429 is retried with backoff (honouring Retry-After); a hard 403 is not retried (re-requesting a block is what turns a soft challenge into a permanent ban) unless it carries Retry-After. A feed that keeps failing enters exponential backoff (15 min β 24 h) so it is skipped until its retry time.h1βh3, preserved as Markdown) give each chunk a section breadcrumb β the heading path active where the chunk ends.title | breadcrumb + body (contextual-v2) and sent to LM Studio's /v1/embeddings endpoint in batches, with bounded request concurrency and model-aware document/query prefixes. If the model is offline, chunks are stored text-only and a backfill (admin_rss_cache β reindex, or automatically once the model is up) completes them later; caches built with older embed styles are upgraded the same way. Repeated search queries reuse a cached (prefixed) query embedding.cache.db) through a versioned migration runner (schema version 11). Newer items list first; retention purges by publish or cache date. When an existing entry's text changes, its chunks are transparently rebuilt.porter BM25 FTS5 list, and a bounded cosine-similarity scan β all restricted to those entries β using Reciprocal Rank Fusion, deduplicates by article (collapsing syndicated duplicates via SimHash), blends in a gentle recency boost, and diversifies with MMR before returning results. Each result includes a snippet windowed around the matching query terms; the resident vector index is updated in place as articles are added or removed.feeds_state table, and per-host ban state lives in host_state. When a site blocks the crawler (HTTP 403/429, a Cloudflare/Akamai/Incapsula challenge, or a JS wall served with 200), the plugin logs a loud β οΈ BAN warning, pauses the whole host (30 min, doubling to 12 h) so every feed on that domain stays quiet, and skips it on later refreshes until the backoff expires. The active list of paused hosts is returned by refresh_rss_feeds as blockedHosts (with a banWarning summary) and is stored in feeds_state/host_state so it survives restarts. The retention pass keeps planner statistics fresh and vacuums only when warranted; admin_rss_cache (or cache_diagnostics) summarizes coverage, sizes, feed health, and suggested fixes in one call.maxConcurrentPerHost requests in flight against any single domain, spaced by minRequestGapMs + requestGapJitterMs (default 1.5β3 s). Pages on different hosts no longer wait on each other, so wide fans-out stay fast while any one domain stays polite. New settings: maxConcurrentPerHost, minRequestGapMs, requestGapJitterMs, respectRobotsTxt, contactUserAgent; maxConcurrentFeeds default lowered 8 β 4.403 is no longer retried (only a 403 carrying Retry-After), since re-requesting a block is what escalates a soft challenge into a permanent ban. 429/5xx retry behaviour is unchanged.robots.txt is honoured before scraping HTML pages and before fetching full article text (never for RSS feed URLs, which are published for machines). Missing/unreadable robots.txt means "allowed", as every major crawler assumes.β οΈ BAN warning naming the host, modal challenge, and attempt count; the whole host is then paused (30 min, doubling to 12 h) and skipped on subsequent refreshes. New host_state table (schema version 11) persists this across restarts, and the active pause list is returned as blockedHosts (with a banWarning summary) by refresh_rss_feeds, so a block is never silent. A host that answers normally again clears its own ban record.lib/rateLimiter.js (per-host limiter), lib/robots.js (robots.txt parser + cache), lib/bans.js (bot-wall classification); new tests for all three plus test/fetcher.test.js covering no-retry-on-403, 429 retry, host backoff skips, robots vetoes, and cross-host pacing.contextual-v2): chunks embed as title | section breadcrumb + body, queries/documents receive model-specific prefixes (search_query:/search_document: for nomic, Represent this sentenceβ¦ for bge/mxbai/snowflake), and <h1>β<h3> are preserved as Markdown so each chunk knows its section. Older embed styles upgrade lazily (migrations 9β10; schema version 10).search_rss_cache, get_rss_article, get_recent_cache_articles, manage_rss_feeds). New full-article reader get_rss_article (by entry ID or URL, offset/maxChars paging, optional chunk layout), new multiplexed manage_rss_feeds (list/add/remove/test/configure/refresh), and new admin_rss_cache (diagnostics/optimize/prune/reindex/reembed/debug_search). enableAdminTools and exposeGranularAdminTools opt back into the admin and granular tools.vectorIndexStats() reporting live/tombstone/capacity counts.reindex_missing_embeddings.contextChunks).Float32Array with lazy rebuilds, zero per-query allocations, and a memory ceiling (RSS_RAG_INDEX_MAX_MB) falling back to the streaming SQL scan.add_rss_feed discovers <link rel="alternate"> feeds from blog homepages; test_feed reports discovered alternates.collection labels (feeds.json), with retention honouring overrides and search supporting a collection filter.search_rss_cache_debug tool for ranking introspection; cache_diagnostics reports vector-index and embed-style coverage.lib/eval.js, npm run eval, frozen corpus, deterministic baseline) for recall@1/3/5/10, MRR, nDCG@10, and latency.porter unicode61 BM25 FTS5 list, and a bounded vector scan.feeds.json (enabled, tags, scrapeFullText, linkPatterns, maxItems, headers) and a new set_feed_options tool.days/after/before, feed, author, tag), relevance/date sorting, and limit/offset pagination.cache_diagnostics tool and a versioned PRAGMA user_version migration runner.If-None-Match / If-Modified-Since) with per-feed exponential backoff, backed by a new feeds_state table.reembed_all tool rebuilds the index after a model change.PRAGMA optimize, plus VACUUM only when enough free pages accumulate).refresh_rss_feeds accepts a force flag; list_rss_feeds reports per-feed health and the model/dimension the cache was built with.reembedAll, ranking) and added a npm run typecheck script.https://news.ycombinator.com/rss
https://www.theverge.com/rss/index.xml
# Non-RSS webpages (scraped automatically, slower):
https://example.com/news
{
"feeds": [
{ "url": "https://news.ycombinator.com/rss", "tags": ["tech", "startups"], "maxItems": 50 },
{ "url": "https://example.com/private.xml", "headers": { "Authorization": "Bearer ..." } },
{ "url": "https://example.com/news", "enabled": false }
]
}
npm install
npm test # node:test suite (lib modules + a fake embedding server)
npm run typecheck # tsc --noEmit over src/*.ts
npm run eval # Phase 0 retrieval eval (recall@k/MRR/nDCG@10 + latency)