README
README
Autonomous deep research for LM Studio. A swarm of specialized AI workers searches your local documents and the web, adapts its strategy, verifies claims, and synthesizes everything into a structured, confidence-scored report with auditable citations — all in one tool call.
CHANGELOG.mdQuick links:
Qwen 3.8 fast-call mode: Optimized for Qwen-class models, this mode pre-fills an empty reasoning block and appends
/no_thinkto prevent truncation of short verdicts, enabling faster reasoning calls without losing detailed traces.
New in 5.12.1: Exposes
enablePinchtabStealthtoggle in Plugin Settings (defaulttrue). When enabled, the fetch waterfall inserts a local Pinchtab headless-Chrome tier (Tier 2.5) ahead of FlareSolverr, with an SSRF guard (assertSafeUrl) and prompt-injection scrub (scrubPromptInjection) before any readable text reaches the pipeline. Zero Docker required.
New in 5.10.0: a Qwen 3.8 fast-call mode (empty
<think>prefill +/no_think) so the gatekeeper, critic, and query mutator never stall inside a reasoning block; a Tier-0 keyless JSON pre-seeder (OpenAlex + Wikipedia REST) that lands verified citations in <3.5s before any HTML engine; intra-page BM25 windowing so the critic and synthesis read the most relevant paragraphs instead of the page top; a topic-adaptive fallback planner; a KV-cache-pinned system prefix shared across planner → critic → synthesis; in-run singleflight URL dedupe; a pre-fetch SERP snippet gate; and a skill-pack fallback that ends[SKILL] none.
What to expect:
The plugin is fully functional with free scraping only. External services are opt-in and only ever add reliability — when a key is missing, every engine falls back to free scraping automatically.
| Capability | Zero-config local mode | Optional external service |
|---|---|---|
| Web search | DuckDuckGo + Bing + Brave + SearXNG (HTML scraping — SearXNG prefers your local instance, public searx.be as fallback), trusted reference sources | Serper / Brave API keys for fast, quota-backed results |
| Page fetching | Direct HTTP → got-scraping TLS → Wayback Machine (for inaccessible pages) | FlareSolverr (local headless Chrome) for aggressive JS challenges |
| Network restrictions | Falls through to other free engines + Wayback Machine | Decodo Web Scraping API fetches server-side through its proxy pool for TCP-blocked hosts |
| Local RAG | Built-in BM25 + n-gram index | — |
| Academic APIs | OpenAlex / Crossref / arXiv (keyless) | Crossref polite-pool crossrefMailto |
| Feeds | — | rssFeedUrls, telegramChannels |
| Model | Any local LM Studio model | — |
Precedence for secrets: plugin settings field → environment variable
(DECODO_API_TOKEN) → ~/.deep-swarm-research/api-keys.json.
Run the plugin once to auto-generate the file location.
Research & web
| Tool | Purpose |
|---|---|
| DeepResearch | Main entry — full Markdown report with citations, contradictions, and dimension coverage. |
| Search | Scored web results with domain-authority tiers. |
| Read Page / Multi-Read | Fetch + extract up to 10 URLs concurrently (PDFs handled automatically). |
| PDF Batch Read | Batch PDF text extraction. |
| ReadSkillFile | Load domain/formatting skills before synthesis to improve report structure |
| Use Plugin Tool | Invoke other LM Studio plugins (e.g., budget tools) with per-run budget caps |
Local library (RAG)
| Tool | Purpose |
|---|---|
| RAG Add / List / Remove / Update Library | Manage indexed folders with priority tiers. |
| RAG Search | BM25 + fuzzy n-gram hybrid search. |
| RAG Check Changes | Detect modified, deleted, or new files. |
| RAG Save / Load Index | Persist the index to disk. |
With Require Approval Before Writes on, write tools only run when the caller passes
confirmed: true.
| Shallow | Standard | Deep | Deeper | Exhaustive | |
|---|---|---|---|---|---|
| Rounds | 1 | 3 | 5 | 10 | 15 |
| Worker roles | 5 | 5 | 8 | 10 | 10 |
| Pages/worker | 5 | 8 | 12 | 18 | 25 |
| Search engines | 1 | 2 | 3 | 4 | 5 |
| Link depth | 1 | 1 | 2 | 2 | 3 |
| Content/page | 5K | 6K | 8K | 12K | 16K |
| Max sources | ~50 | ~80 | ~150 | ~250+ | ~400+ |
| Setting | Description |
|---|---|
| Data Sources | Web only, Local documents only, or Local documents and web. |
| Research Depth | Shallow → Exhaustive. Controls rounds and budgets. |
| Engine Selection Mode | Adaptive, Benchmark, or Priority. |
| Context Budget Mode | Auto (recommended) scales token limits dynamically. |
| LLM Call Budget | Standard = max 45 calls; prevents infinite loops. |
| LLM Context Isolation | Strict isolation; prevents history bleed. |
| Max Session Time | Hard wall-clock cap (minutes, 20% reserved for synthesis). 0 = no explicit cap (uses global timeout). |
| Academic APIs | Query OpenAlex, Crossref, and arXiv directly. |
| YouTube Search |
Q: Can I use this without internet? A: Yes! Use "Local documents only" mode for fully offline research.
Q: How does the citation system work? A: Every claim is linked to a specific source from the evidence ledger, with confidence scores.
Q: What's the difference between Shallow and Exhaustive depth? A: Shallow does 1 round with 5 sources; Exhaustive does 15 rounds with 25 sources per worker.
Q: Can I use multiple models? A: Currently, the plugin works with any LM Studio model you load. Different models can be used for different research tasks.
Q: How do I export the report? A: The report is generated in Markdown format. Copy-paste it into your favorite Markdown editor or convert to PDF.
Q: Does it support real-time research? A: No, the plugin is designed for batch research. Submit your query and wait for the complete report.
For most users, the default settings work well:
| Setting | Description |
|---|---|
| Data Sources | Web only, Local documents only, or Local documents and web. |
| Research Depth | Shallow → Exhaustive. Controls rounds and budgets. |
| Engine Selection Mode | Adaptive, Benchmark, or Priority. |
| Context Budget Mode | Auto (recommended) scales token limits dynamically. |
| LLM Call Budget | Standard = max 45 calls; prevents infinite loops. |
| LLM Context Isolation | Strict isolation; prevents history bleed. |
| Max Session Time | Hard wall-clock cap (minutes, 20% reserved for synthesis). 0 = no explicit cap (uses global timeout). |
| Academic APIs | Query OpenAlex, Crossref, and arXiv directly. |
| YouTube Search |
| Shallow | Standard | Deep | Deeper | Exhaustive | |
|---|---|---|---|---|---|
| Rounds | 1 | 3 | 5 | 10 | 15 |
| Worker roles | 5 | 5 | 8 | 10 | 10 |
| Pages/worker | 5 | 8 | 12 | 18 | 25 |
| Search engines | 1 | 2 | 3 | 4 | 5 |
| Link depth | 1 | 1 | 2 | 2 | 3 |
| Content/page | 5K | 6K | 8K | 12K | 16K |
| Max sources | ~50 | ~80 | ~150 | ~250+ | ~400+ |
When to use which preset:
Contributions are welcome! Areas where help is needed:
Please read the contributing guidelines before submitting pull requests.
Autonomous deep research for LM Studio. A swarm of specialized AI workers searches your local documents and the web, adapts its strategy, verifies claims, and synthesizes everything into a structured, confidence-scored report with auditable citations — all in one tool call.
CHANGELOG.mdQuick links:
Qwen 3.8 fast-call mode: Optimized for Qwen-class models, this mode pre-fills an empty reasoning block and appends
/no_thinkto prevent truncation of short verdicts, enabling faster reasoning calls without losing detailed traces.
New in 5.12.1: Exposes
enablePinchtabStealthtoggle in Plugin Settings (defaulttrue). When enabled, the fetch waterfall inserts a local Pinchtab headless-Chrome tier (Tier 2.5) ahead of FlareSolverr, with an SSRF guard (assertSafeUrl) and prompt-injection scrub (scrubPromptInjection) before any readable text reaches the pipeline. Zero Docker required.
New in 5.10.0: a Qwen 3.8 fast-call mode (empty
<think>prefill +/no_think) so the gatekeeper, critic, and query mutator never stall inside a reasoning block; a Tier-0 keyless JSON pre-seeder (OpenAlex + Wikipedia REST) that lands verified citations in <3.5s before any HTML engine; intra-page BM25 windowing so the critic and synthesis read the most relevant paragraphs instead of the page top; a topic-adaptive fallback planner; a KV-cache-pinned system prefix shared across planner → critic → synthesis; in-run singleflight URL dedupe; a pre-fetch SERP snippet gate; and a skill-pack fallback that ends[SKILL] none.
What to expect:
The plugin is fully functional with free scraping only. External services are opt-in and only ever add reliability — when a key is missing, every engine falls back to free scraping automatically.
| Capability | Zero-config local mode | Optional external service |
|---|---|---|
| Web search | DuckDuckGo + Bing + Brave + SearXNG (HTML scraping — SearXNG prefers your local instance, public searx.be as fallback), trusted reference sources | Serper / Brave API keys for fast, quota-backed results |
| Page fetching | Direct HTTP → got-scraping TLS → Wayback Machine (for inaccessible pages) | FlareSolverr (local headless Chrome) for aggressive JS challenges |
| Network restrictions | Falls through to other free engines + Wayback Machine | Decodo Web Scraping API fetches server-side through its proxy pool for TCP-blocked hosts |
| Local RAG | Built-in BM25 + n-gram index | — |
| Academic APIs | OpenAlex / Crossref / arXiv (keyless) | Crossref polite-pool crossrefMailto |
| Feeds | — | rssFeedUrls, telegramChannels |
| Model | Any local LM Studio model | — |
Precedence for secrets: plugin settings field → environment variable
(DECODO_API_TOKEN) → ~/.deep-swarm-research/api-keys.json.
Run the plugin once to auto-generate the file location.
Research & web
| Tool | Purpose |
|---|---|
| DeepResearch | Main entry — full Markdown report with citations, contradictions, and dimension coverage. |
| Search | Scored web results with domain-authority tiers. |
| Read Page / Multi-Read | Fetch + extract up to 10 URLs concurrently (PDFs handled automatically). |
| PDF Batch Read | Batch PDF text extraction. |
| ReadSkillFile | Load domain/formatting skills before synthesis to improve report structure |
| Use Plugin Tool | Invoke other LM Studio plugins (e.g., budget tools) with per-run budget caps |
Local library (RAG)
| Tool | Purpose |
|---|---|
| RAG Add / List / Remove / Update Library | Manage indexed folders with priority tiers. |
| RAG Search | BM25 + fuzzy n-gram hybrid search. |
| RAG Check Changes | Detect modified, deleted, or new files. |
| RAG Save / Load Index | Persist the index to disk. |
With Require Approval Before Writes on, write tools only run when the caller passes
confirmed: true.
| Shallow | Standard | Deep | Deeper | Exhaustive | |
|---|---|---|---|---|---|
| Rounds | 1 | 3 | 5 | 10 | 15 |
| Worker roles | 5 | 5 | 8 | 10 | 10 |
| Pages/worker | 5 | 8 | 12 | 18 | 25 |
| Search engines | 1 | 2 | 3 | 4 | 5 |
| Link depth | 1 | 1 | 2 | 2 | 3 |
| Content/page | 5K | 6K | 8K | 12K | 16K |
| Max sources | ~50 | ~80 | ~150 | ~250+ | ~400+ |
| Setting | Description |
|---|---|
| Data Sources | Web only, Local documents only, or Local documents and web. |
| Research Depth | Shallow → Exhaustive. Controls rounds and budgets. |
| Engine Selection Mode | Adaptive, Benchmark, or Priority. |
| Context Budget Mode | Auto (recommended) scales token limits dynamically. |
| LLM Call Budget | Standard = max 45 calls; prevents infinite loops. |
| LLM Context Isolation | Strict isolation; prevents history bleed. |
| Max Session Time | Hard wall-clock cap (minutes, 20% reserved for synthesis). 0 = no explicit cap (uses global timeout). |
| Academic APIs | Query OpenAlex, Crossref, and arXiv directly. |
| YouTube Search |
Q: Can I use this without internet? A: Yes! Use "Local documents only" mode for fully offline research.
Q: How does the citation system work? A: Every claim is linked to a specific source from the evidence ledger, with confidence scores.
Q: What's the difference between Shallow and Exhaustive depth? A: Shallow does 1 round with 5 sources; Exhaustive does 15 rounds with 25 sources per worker.
Q: Can I use multiple models? A: Currently, the plugin works with any LM Studio model you load. Different models can be used for different research tasks.
Q: How do I export the report? A: The report is generated in Markdown format. Copy-paste it into your favorite Markdown editor or convert to PDF.
Q: Does it support real-time research? A: No, the plugin is designed for batch research. Submit your query and wait for the complete report.
For most users, the default settings work well:
| Setting | Description |
|---|---|
| Data Sources | Web only, Local documents only, or Local documents and web. |
| Research Depth | Shallow → Exhaustive. Controls rounds and budgets. |
| Engine Selection Mode | Adaptive, Benchmark, or Priority. |
| Context Budget Mode | Auto (recommended) scales token limits dynamically. |
| LLM Call Budget | Standard = max 45 calls; prevents infinite loops. |
| LLM Context Isolation | Strict isolation; prevents history bleed. |
| Max Session Time | Hard wall-clock cap (minutes, 20% reserved for synthesis). 0 = no explicit cap (uses global timeout). |
| Academic APIs | Query OpenAlex, Crossref, and arXiv directly. |
| YouTube Search |
| Shallow | Standard | Deep | Deeper | Exhaustive | |
|---|---|---|---|---|---|
| Rounds | 1 | 3 | 5 | 10 | 15 |
| Worker roles | 5 | 5 | 8 | 10 | 10 |
| Pages/worker | 5 | 8 | 12 | 18 | 25 |
| Search engines | 1 | 2 | 3 | 4 | 5 |
| Link depth | 1 | 1 | 2 | 2 | 3 |
| Content/page | 5K | 6K | 8K | 12K | 16K |
| Max sources | ~50 | ~80 | ~150 | ~250+ | ~400+ |
When to use which preset:
Contributions are welcome! Areas where help is needed:
Please read the contributing guidelines before submitting pull requests.
format=json),
a 12s per-query timeout, max 2 concurrent local requests, and no 10-minute
cooldown when a query legitimately returns 0 results.rel scores now combine core-term overlap (0.6)
with query/dimension overlap (0.4) and cap at 0.92 instead of clamping to
1.00.web.archive.org,
archive.org, sciencedirect.com, and researchgate.net; Sciencedirect /
ResearchGate fail fast. The Feeds pass is skipped immediately when no real
RSS/Telegram targets are configured.[OpenAlex ...], Title:,
Abstract:) is stripped and retry queries are capped at 10 keywords.new LMStudioClient() inside the plugin context.New in 5.11.0: the SearXNG engine now defaults to a local instance
(http://127.0.0.1:8080, configurable via the SearXNG URL setting). The
endpoint is probed at run start — if your local SearXNG is unreachable the run
logs a single warning and transparently falls back to the public
https://searx.be, so the engine works with zero configuration either way.
Plan — the topic is decomposed into up to 10 specialized worker roles, emitted as a validated JSON plan (with a deterministic, topic-adaptive fallback).
Local first — local RAG libraries are always searched before the web; if local evidence is sufficient, the web layer is skipped.
Tier-0 JSON pre-seed — keyless OpenAlex + Wikipedia REST feeds verified reference cards into the ledger in under ~3.5s, before any HTML engine can be rate-limited, so a run never ends with zero citations.
Gap-driven web search — queries target missing dimensions. Free engines run first; paid APIs are held in reserve.
Anti-block fetching — a waterfall cascades direct fetch → got-scraping
→ FlareSolverr → Wayback Machine (plus Decodo when configured). A pre-fetch
snippet gate drops junk/login-wall URLs and in-run singleflight collapses
concurrent duplicate URL fetches.
Book discovery searches DOAB, OAPEN, Project Gutenberg, Open Library and
Library of Congress, with English-language filters. Catalog records are
listed separately as further reading, not as evidence that the book's
contents were read or verified. Access varies by catalog and title.
Critic / verifier — every evidence card is graded (deterministic pre-score + batched LLM pass); only failing claims trigger one bounded retry. Pages are BM25-windowed first, so only the most relevant paragraphs are graded.
Synthesize & verify — an isolated synthesis pass writes the report, then
an independent verifier confirms every [n] citation and URL resolves to the
ledger.
Key features explained:
| Enable YouTube scraping and transcript extraction. |
| Cache Duration | Days to reuse visited pages (default 30). 0 disables caching. |
| FlareSolverr URL | Optional: local Cloudflare bypass endpoint. |
| SearXNG URL | Optional: local SearXNG instance (default http://127.0.0.1:8080). Probed at run start; if unreachable a warning is logged once and the engine falls back to the public https://searx.be. Blank = public only. |
| Decodo Web Scraping API Token | Optional: server-side fetch for TCP-blocked hosts. |
| Allowed / Blocked Domains | Comma-separated root-domain allow/deny lists (deny wins). |
| Max PDF Download Size | Hard byte cap per PDF (0 = no cap, may use more disk space). |
| Max External Plugin Tool Calls | Per-run cap on Use Plugin Tool (0 = unlimited). |
| Require Approval Before Writes | Requires confirmed: true for RAG writes / write-capable tools. |
| Concurrent LLM Calls | Parallel predictions through the global semaphore (default 1 to prevent model overload). |
| Include Reasoning Trace | Append the collapsible reasoning/verification trace to the report. |
Issue: Some websites with aggressive anti-bot measures may still block access Workaround: Use Decodo API token or increase link depth to find alternative sources
Issue: Very large RAG indexes (>50K docs) may cause memory issues Workaround: Split your documents into multiple libraries and search them separately
Issue: PDF extraction may fail for encrypted or password-protected PDFs Workaround: Decrypt PDFs before adding them to the library
Issue: YouTube search requires additional setup Workaround: Enable YouTube Search in configuration and ensure LM Studio has internet access
| Enable YouTube scraping and transcript extraction. |
| Cache Duration | Days to reuse visited pages (default 30). 0 disables caching. |
| FlareSolverr URL | Optional: local Cloudflare bypass endpoint. |
| SearXNG URL | Optional: local SearXNG instance (default http://127.0.0.1:8080). Probed at run start; if unreachable a warning is logged once and the engine falls back to the public https://searx.be. Blank = public only. |
| Decodo Web Scraping API Token | Optional: server-side fetch for TCP-blocked hosts. |
| Allowed / Blocked Domains | Comma-separated root-domain allow/deny lists (deny wins). |
| Max PDF Download Size | Hard byte cap per PDF (0 = no cap, may use more disk space). |
| Max External Plugin Tool Calls | Per-run cap on Use Plugin Tool (0 = unlimited). |
| Require Approval Before Writes | Requires confirmed: true for RAG writes / write-capable tools. |
| Concurrent LLM Calls | Parallel predictions through the global semaphore (default 1 to prevent model overload). |
| Include Reasoning Trace | Append the collapsible reasoning/verification trace to the report. |
"Research the latest developments in quantum computing, focusing on recent papers in my local research folder"
"Compare the performance of React vs Vue for enterprise applications, citing 3-5 sources each"
"Analyze the impact of AI on software development, including both positive and negative aspects"
"Explain how neural networks work, using shallow depth for a quick overview"
// ~/.deep-swarm-research/api-keys.json (Windows: C:\Users\<you>\.deep-swarm-research\)
{
"serperApiKey": "YOUR_SERPER_KEY", // optional
"braveApiKey": "YOUR_BRAVE_KEY", // optional
"decodoApiToken": "YOUR_DECODO_TOKEN", // optional
"crossrefMailto": "you@example.com", // optional
"rssFeedUrls": ["https://feeds.bbci.co.uk/news/rss.xml"],
"telegramChannels": ["channelname"]
}
┌─────────────────────────────────────────────────────────────┐
│ Research Topic │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 1. PLAN: Decompose into specialized worker roles │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Scholar │ │ Analyst │ │ Critic │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└────────────────────┬────────────────────────────────────────┘
│
┌────────────┴────────────┐
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Local RAG │ │ Web Search │
│ (documents) │ │ (DuckDuckGo, │
│ │ │ Bing, Brave) │
└───────┬───────┘ └───────┬───────┘
│ │
└────────────┬────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 2. COLLECT: Gather evidence with citation tracking │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 3. VERIFY: Critic grades claims, retries failures │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 4. SYNTHESIZE: Combine evidence into structured report │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 5. VERIFY FINAL: Independent check of all citations │
└─────────────────────────────────────────────────────────────┘
1. PROPRIETARY local, confidential (contracts, internal memos)
2. INTERNAL shared team knowledge (wikis, docs)
3. REFERENCE curated reference material (papers, standards)
4. GENERAL miscellaneous local documents
5. WEB public internet (fills remaining gaps)
You are an expert Deep Research Analyst. Your goal is to synthesize complex information from local proprietary databases and the public web into highly structured, objective, and rigorously cited reports.
TOOL INVOCATION RULES
When a user asks for research, analysis, comparisons, or deep dives, you MUST use the "DeepResearch" tool.
- Use focusAreas to break down complex prompts into specific angles.
- Use depthOverride to control the breadth of the research.
POST-RESEARCH SYNTHESIS FRAMEWORK
When the tool returns the research report, structure your response exactly as follows:
1. Executive Summary: A concise, 3-4 sentence TL;DR.
2. Core Analysis: Detailed synthesis organized by theme, NOT by source.
3. Source Origin & Temporal Context: Note if findings rely on Local vs. Web data, and the time period.
4. Contradictions & Nuance: Highlight conflicting sources objectively.
5. Coverage Gaps & Next Steps: Identify missing info and suggest follow-ups.
STRICT CITATION & CONFIDENCE RULES
- You MUST cite every factual claim using the exact index format provided by the tool: [1], [2], etc.
- NEVER invent, guess, or hallucinate citation numbers.
- If the tool provides [High/Medium/Low Confidence] tags, preserve them.
- Prioritize proprietary/internal sources over web sources when they conflict.
TONE & STYLE
- Maintain a highly professional, objective, and analytical tone.
- Avoid filler phrases like "Based on the research provided..." Just deliver the analysis directly.
- Use Markdown formatting (bolding, lists, tables) to make dense information highly readable.
You are an expert Research Analyst. Use the "Deep Research" tool for any complex query, setting `depthOverride` based on complexity.
When the tool returns results, format your response exactly like this:
1. **Executive Summary**: 3-sentence TL;DR.
2. **Detailed Analysis**: Synthesize by theme, not by source.
3. **Source Context**: Note if findings rely on Local/Proprietary docs vs Web, and note the time period.
4. **Contradictions**: Highlight conflicting sources objectively.
5. **Gaps**: What is missing? Suggest follow-ups.
STRICT RULES:
- Cite EVERY claim using the tool's exact numbers: [1], [2], etc. NEVER hallucinate citation numbers.
- Prioritize Local/Proprietary sources over Web sources.
- Do not use filler phrases. Be direct, objective, and use Markdown tables/lists for readability.
format=json),
a 12s per-query timeout, max 2 concurrent local requests, and no 10-minute
cooldown when a query legitimately returns 0 results.rel scores now combine core-term overlap (0.6)
with query/dimension overlap (0.4) and cap at 0.92 instead of clamping to
1.00.web.archive.org,
archive.org, sciencedirect.com, and researchgate.net; Sciencedirect /
ResearchGate fail fast. The Feeds pass is skipped immediately when no real
RSS/Telegram targets are configured.[OpenAlex ...], Title:,
Abstract:) is stripped and retry queries are capped at 10 keywords.new LMStudioClient() inside the plugin context.New in 5.11.0: the SearXNG engine now defaults to a local instance
(http://127.0.0.1:8080, configurable via the SearXNG URL setting). The
endpoint is probed at run start — if your local SearXNG is unreachable the run
logs a single warning and transparently falls back to the public
https://searx.be, so the engine works with zero configuration either way.
Plan — the topic is decomposed into up to 10 specialized worker roles, emitted as a validated JSON plan (with a deterministic, topic-adaptive fallback).
Local first — local RAG libraries are always searched before the web; if local evidence is sufficient, the web layer is skipped.
Tier-0 JSON pre-seed — keyless OpenAlex + Wikipedia REST feeds verified reference cards into the ledger in under ~3.5s, before any HTML engine can be rate-limited, so a run never ends with zero citations.
Gap-driven web search — queries target missing dimensions. Free engines run first; paid APIs are held in reserve.
Anti-block fetching — a waterfall cascades direct fetch → got-scraping
→ FlareSolverr → Wayback Machine (plus Decodo when configured). A pre-fetch
snippet gate drops junk/login-wall URLs and in-run singleflight collapses
concurrent duplicate URL fetches.
Book discovery searches DOAB, OAPEN, Project Gutenberg, Open Library and
Library of Congress, with English-language filters. Catalog records are
listed separately as further reading, not as evidence that the book's
contents were read or verified. Access varies by catalog and title.
Critic / verifier — every evidence card is graded (deterministic pre-score + batched LLM pass); only failing claims trigger one bounded retry. Pages are BM25-windowed first, so only the most relevant paragraphs are graded.
Synthesize & verify — an isolated synthesis pass writes the report, then
an independent verifier confirms every [n] citation and URL resolves to the
ledger.
Key features explained:
| Enable YouTube scraping and transcript extraction. |
| Cache Duration | Days to reuse visited pages (default 30). 0 disables caching. |
| FlareSolverr URL | Optional: local Cloudflare bypass endpoint. |
| SearXNG URL | Optional: local SearXNG instance (default http://127.0.0.1:8080). Probed at run start; if unreachable a warning is logged once and the engine falls back to the public https://searx.be. Blank = public only. |
| Decodo Web Scraping API Token | Optional: server-side fetch for TCP-blocked hosts. |
| Allowed / Blocked Domains | Comma-separated root-domain allow/deny lists (deny wins). |
| Max PDF Download Size | Hard byte cap per PDF (0 = no cap, may use more disk space). |
| Max External Plugin Tool Calls | Per-run cap on Use Plugin Tool (0 = unlimited). |
| Require Approval Before Writes | Requires confirmed: true for RAG writes / write-capable tools. |
| Concurrent LLM Calls | Parallel predictions through the global semaphore (default 1 to prevent model overload). |
| Include Reasoning Trace | Append the collapsible reasoning/verification trace to the report. |
Issue: Some websites with aggressive anti-bot measures may still block access Workaround: Use Decodo API token or increase link depth to find alternative sources
Issue: Very large RAG indexes (>50K docs) may cause memory issues Workaround: Split your documents into multiple libraries and search them separately
Issue: PDF extraction may fail for encrypted or password-protected PDFs Workaround: Decrypt PDFs before adding them to the library
Issue: YouTube search requires additional setup Workaround: Enable YouTube Search in configuration and ensure LM Studio has internet access
| Enable YouTube scraping and transcript extraction. |
| Cache Duration | Days to reuse visited pages (default 30). 0 disables caching. |
| FlareSolverr URL | Optional: local Cloudflare bypass endpoint. |
| SearXNG URL | Optional: local SearXNG instance (default http://127.0.0.1:8080). Probed at run start; if unreachable a warning is logged once and the engine falls back to the public https://searx.be. Blank = public only. |
| Decodo Web Scraping API Token | Optional: server-side fetch for TCP-blocked hosts. |
| Allowed / Blocked Domains | Comma-separated root-domain allow/deny lists (deny wins). |
| Max PDF Download Size | Hard byte cap per PDF (0 = no cap, may use more disk space). |
| Max External Plugin Tool Calls | Per-run cap on Use Plugin Tool (0 = unlimited). |
| Require Approval Before Writes | Requires confirmed: true for RAG writes / write-capable tools. |
| Concurrent LLM Calls | Parallel predictions through the global semaphore (default 1 to prevent model overload). |
| Include Reasoning Trace | Append the collapsible reasoning/verification trace to the report. |
"Research the latest developments in quantum computing, focusing on recent papers in my local research folder"
"Compare the performance of React vs Vue for enterprise applications, citing 3-5 sources each"
"Analyze the impact of AI on software development, including both positive and negative aspects"
"Explain how neural networks work, using shallow depth for a quick overview"
// ~/.deep-swarm-research/api-keys.json (Windows: C:\Users\<you>\.deep-swarm-research\)
{
"serperApiKey": "YOUR_SERPER_KEY", // optional
"braveApiKey": "YOUR_BRAVE_KEY", // optional
"decodoApiToken": "YOUR_DECODO_TOKEN", // optional
"crossrefMailto": "you@example.com", // optional
"rssFeedUrls": ["https://feeds.bbci.co.uk/news/rss.xml"],
"telegramChannels": ["channelname"]
}
┌─────────────────────────────────────────────────────────────┐
│ Research Topic │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 1. PLAN: Decompose into specialized worker roles │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Scholar │ │ Analyst │ │ Critic │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└────────────────────┬────────────────────────────────────────┘
│
┌────────────┴────────────┐
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Local RAG │ │ Web Search │
│ (documents) │ │ (DuckDuckGo, │
│ │ │ Bing, Brave) │
└───────┬───────┘ └───────┬───────┘
│ │
└────────────┬────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 2. COLLECT: Gather evidence with citation tracking │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 3. VERIFY: Critic grades claims, retries failures │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 4. SYNTHESIZE: Combine evidence into structured report │
└────────────────────┬────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ 5. VERIFY FINAL: Independent check of all citations │
└─────────────────────────────────────────────────────────────┘
1. PROPRIETARY local, confidential (contracts, internal memos)
2. INTERNAL shared team knowledge (wikis, docs)
3. REFERENCE curated reference material (papers, standards)
4. GENERAL miscellaneous local documents
5. WEB public internet (fills remaining gaps)
You are an expert Deep Research Analyst. Your goal is to synthesize complex information from local proprietary databases and the public web into highly structured, objective, and rigorously cited reports.
TOOL INVOCATION RULES
When a user asks for research, analysis, comparisons, or deep dives, you MUST use the "DeepResearch" tool.
- Use focusAreas to break down complex prompts into specific angles.
- Use depthOverride to control the breadth of the research.
POST-RESEARCH SYNTHESIS FRAMEWORK
When the tool returns the research report, structure your response exactly as follows:
1. Executive Summary: A concise, 3-4 sentence TL;DR.
2. Core Analysis: Detailed synthesis organized by theme, NOT by source.
3. Source Origin & Temporal Context: Note if findings rely on Local vs. Web data, and the time period.
4. Contradictions & Nuance: Highlight conflicting sources objectively.
5. Coverage Gaps & Next Steps: Identify missing info and suggest follow-ups.
STRICT CITATION & CONFIDENCE RULES
- You MUST cite every factual claim using the exact index format provided by the tool: [1], [2], etc.
- NEVER invent, guess, or hallucinate citation numbers.
- If the tool provides [High/Medium/Low Confidence] tags, preserve them.
- Prioritize proprietary/internal sources over web sources when they conflict.
TONE & STYLE
- Maintain a highly professional, objective, and analytical tone.
- Avoid filler phrases like "Based on the research provided..." Just deliver the analysis directly.
- Use Markdown formatting (bolding, lists, tables) to make dense information highly readable.
You are an expert Research Analyst. Use the "Deep Research" tool for any complex query, setting `depthOverride` based on complexity.
When the tool returns results, format your response exactly like this:
1. **Executive Summary**: 3-sentence TL;DR.
2. **Detailed Analysis**: Synthesize by theme, not by source.
3. **Source Context**: Note if findings rely on Local/Proprietary docs vs Web, and note the time period.
4. **Contradictions**: Highlight conflicting sources objectively.
5. **Gaps**: What is missing? Suggest follow-ups.
STRICT RULES:
- Cite EVERY claim using the tool's exact numbers: [1], [2], etc. NEVER hallucinate citation numbers.
- Prioritize Local/Proprietary sources over Web sources.
- Do not use filler phrases. Be direct, objective, and use Markdown tables/lists for readability.