Ship-First Execution Matrix: Phase 2 High-Impact ImprovementsOnce the immediate pipeline blockers (the 1ms LLM fallback, the 12-worker DDG burst, and the 2-minute FEEDS stall) are patched, the highest-leverage improvements target Qwen 3.8 token latency, zero-CAPTCHA JSON source acquisition, and KV-cache prompt density: RankImprovementImpactEffortRisk of Rabbit HoleVerdict1Qwen 3.8 Bypass for Micro-Calls (gatekeeper, critic, mutator)Prefill \n\n (or append /no_think) on classification/scoring calls while keeping full reasoning for Plan & Synthesis.Massive (Cuts LLM wait time by ~70% and prevents 10-token / 30s truncation)1 hrLowDO NOW (Sprint 2)2Tier-0 Keyless JSON API Pre-Seeder (OpenAlex + Crossref + Wikipedia REST)Run fast JSON APIs before HTML scrapers (DDG/SearXNG) so every run secures 3–8 verified cards in <1.5s and disarms the 2-minute stall watchdog. Critical (Guarantees baseline Tier-A/B citations even when all HTML engines rate-limit) 1.5 hrsLowDO NOW (Sprint 2)3Intra-Page BM25 Windowing Before Critic & Synthesis (12K → 1.8K chars)Extract only the top 3 BM25-scored paragraphs per scraped page instead of stuffing 5K–16K raw Markdown into the single LM Studio KV-cache slot. High (3x faster prompt ingestion on Qwen 3.8; higher Critic precision)1.5 hrsLowDO NEXT4Semantic Trigger Expansion for Skill Packs ([SKILL] none Fix)Map empirical/investigative terms ("evidence", "study", "historical", "mechanism") to academic.md so topics like "evidence about reincarnation" don't run without a skill rubric. Medium (Sharpens Critic & Synthesis structure with zero extra tokens)30 minsNoneDO NEXT5Increase LlmSemaphore Concurrency Above 1 for Parallel Worker PromptsAllow multiple workers to query LM Studio simultaneously. Negative (Causes KV-cache thrashing and CUDA OOM on single-GPU local setups)2 hrsHighCUT2. Why These 4 Upgrades Multiply Output Quality on Qwen 3.8Stopping Reasoning Waste on Flat Schemas (gatekeeper.ts, critic.ts):The plugin uses a global LlmSemaphore with a default concurrency of 1 (one LM Studio KV-cache slot) and a standard budget of 45 calls. Even though v5.9.1 introduced a flat, prose-free critic wire schema and a hardened parser, Qwen 3.8 still generates 300 to 1,200 internal reasoning tokens before emitting a 10-token gatekeeper boolean or a CARD: tier enum. When gatekeeper.ts enforces a 10-token cap or the critic enforces a 30-second batch timeout, Qwen 3.8 gets cut off inside ..., forcing the parser to fall back to heuristics. Suppressing on deterministic classification calls drops per-call latency from ~14s to <1.2s.Eliminating Zero-Source Runs via Tier-0 JSON APIs (DIRECT Layer Promotion):In run_log_1792372446310.txt (TOPIC: evidence about reincarnation), the swarm spent 8 minutes fighting DDG_RATE_LIMITED (202) (12 attempts, 0 valid) and searxng cooldowns (10 attempts, 0 valid), ending with Tier A: 0, Tier B: 0, and Tier C: 0. Meanwhile, the plugin already supports keyless academic APIs (OpenAlex, Crossref, arXiv) and reference sources, but buried them inside the DIRECT layer after FEEDS, DDG, and SEARXNG. Moving OpenAlex, Crossref, and the Wikipedia REST Summary/Search API into a Tier-0 parallel pre-fetch (capped at 3 seconds total) means the blackboard immediately receives peer-reviewed papers and canonical reference summaries over pure JSON (zero Cloudflare challenges, zero FlareSolverr calls, zero Decodo quota burned). Furthermore, because the stall watchdog only triggers when no accepted sources arrive for 2 minutes, landing 3+ Tier-0 JSON sources in the first 2 seconds permanently prevents premature watchdog aborts. Deflating the 5K–16K Page Bloat (Content/page Windowing):Depth presets currently slice 5K (Shallow) to 16K (Exhaustive) characters from the top of each Readability/Turndown page. Long articles often bury the actual data in the middle or bottom of the page while the first 6K characters contain introductory fluff. Scoring paragraphs with BM25 against the worker's gap query and keeping only the top 3 windows (~1.8K chars total) captures claims at the bottom of 30-page articles while cutting KV-cache prompt tokens by 70%.3. Sprint 2 Implementation: Complete Copy-Pasteable CodeArtifact 1: Qwen 3.8 Fast-Call Wrapper & Sub-Chunk BM25 Pruner (src/utils/qwenOptimizer.ts)Create src/utils/qwenOptimizer.ts to handle suppression for micro-calls, automatic skill routing fallback (fixing [SKILL] none), and BM25 paragraph windowing: TypeScript// src/utils/qwenOptimizer.ts

export type LlmTaskType = | "gatekeeper" // 10-token yes/no local RAG check → NO THINK | "critic" // Flat CARD: grading batches → NO THINK | "mutator" // Fast search query mutation → NO THINK | "planner" // Task decomposition → KEEP THINK (capped) | "synthesizer"; // Final Markdown report → KEEP THINK

export interface OptimizedPromptConfig { systemPrompt: string; userPrompt: string; assistantPrefill?: string; temperature: number; maxTokens: number; }

/**

Optimizes prompts for Qwen 3.8 in LM Studio.
Disables chains on deterministic micro-calls so the single KV-cache slot
never stalls or truncates inside a block. */ export function optimizeForQwen38( task: LlmTaskType, systemPrompt: string, userPrompt: string, defaultMaxTokens = 1024 ): OptimizedPromptConfig { const isMicroCall = task === "gatekeeper" || task === "critic" || task === "mutator";

if (isMicroCall) { return { // Qwen3 / Qwen3.8 honors /no_think in system/user prompt AND empty prefill systemPrompt: ${systemPrompt.trim()}\n/no_think, userPrompt: ${userPrompt.trim()} /no_think, assistantPrefill: "\n\n", temperature: 0.0, maxTokens: task === "gatekeeper" ? 32 : task === "mutator" ? 160 : 384, }; }

return { systemPrompt, userPrompt, temperature: task === "planner" ? 0.2 : 0.3, maxTokens: defaultMaxTokens, }; }

/**

Intra-Page BM25 Paragraph Windowing
Replaces blind 5K-16K top-of-page slicing with the top K most relevant paragraphs
across the entire document, preserving inline heading context. */ export function extractTopBm25Windows( markdownContent: string, query: string, maxChars = 2200, k1 = 1.4, b = 0.75 ): string { if (markdownContent.length <= maxChars) return markdownContent;

const queryTerms = Array.from( new Set( query .toLowerCase() .split(/[^a-z0-9]+/) .filter((t) => t.length > 2) ) ); if (queryTerms.length === 0) return markdownContent.slice(0, maxChars);

// Split into semantic blocks (~350-700 chars) preserving headings const rawBlocks = markdownContent .split(/\n{2,}/) .map((b) => b.trim()) .filter((b) => b.length > 40);

if (rawBlocks.length === 0) return markdownContent.slice(0, maxChars);

const avgLen = rawBlocks.reduce((acc, blk) => acc + blk.length, 0) / rawBlocks.length;

});

// Sort by BM25 score descending, take top blocks until maxChars is reached, // then restore original reading order so prose remains coherent. scored.sort((a, b) => b.score - a.score);

const selected: typeof scored = []; let currentChars = 0; for (const item of scored) { if (currentChars + item.block.length > maxChars && selected.length > 0) { continue; } selected.push(item); currentChars += item.block.length + 2; if (currentChars >= maxChars) break; }

return selected .sort((a, b) => a.index - b.index) .map((s) => s.block) .join("\n\n"); }

/**

Fallback Skill Pack Matcher
Prevents [SKILL] none — no domain tags or pack match on investigative/empirical topics. */ export function resolveSkillPackWithFallback( topic: string, explicitTags: string[] = [] ): "academic" | "market" | "policy" { const combined = ${topic} ${explicitTags.join(" ")}.toLowerCase();

if ( /\b(market|price|revenue|competitor|valuation|saas|stock|pricing|forecast|b2b|startup)\b/.test( combined ) ) { return "market"; } if ( /\b(law|regulation|compliance|policy|gdpr|statute|legal|government|directive|tax|dvla)\b/.test( combined ) ) { return "policy"; } // Default investigative, scientific, historical, or "evidence about X" queries to academic.md return "academic"; } Artifact 2: Tier-0 Keyless JSON Pre-Seeder (src/net/tierZeroJsonSeeder.ts)Create src/net/tierZeroJsonSeeder.ts and call seedFromTierZeroJsonApis(topic, log) at the start of Layers 2-5 in src/swarm/orchestrator.ts (before DDG runs). It hits OpenAlex, Crossref, and the Wikipedia REST API concurrently with a strict 3.5s timeout: TypeScript// src/net/tierZeroJsonSeeder.ts

export interface TierZeroSeedCard { url: string; title: string; content: string; engine: "openalex" | "crossref" | "wikipedia"; sourceTier: "REFERENCE"; }

/**

Queries keyless JSON APIs (OpenAlex, Crossref, Wikipedia REST) in parallel (<3.5s)
BEFORE HTML search engines run. Never triggers CAPTCHAs or DDG 202 rate limits. */ export async function seedFromTierZeroJsonApis( topic: string, crossrefMailto?: string, log?: (msg: string) => void ): Promise<TierZeroSeedCard[]> { const cleanTopic = topic.replace(/[^\w\s-]/g, " ").trim(); const encoded = encodeURIComponent(cleanTopic); const headers: Record<string, string> = { "User-Agent": DeepSwarmResearch/5.9.1 (mailto:${crossrefMailto || "local@saverapps.uk"}), Accept: "application/json", };

const [openAlexCards, wikiCards] = await Promise.all([ openAlexTask.catch(() => []), wikipediaTask.catch(() => []), ]);

async function fetchJsonWithTimeout( url: string, headers: Record<string, string>, timeoutMs: number ): Promise<T | null> { const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), timeoutMs); try { const res = await fetch(url, { headers, signal: controller.signal }); if (!res.ok) return null; return (await res.json()) as T; } catch { return null; } finally { clearTimeout(timer); } } 4. PowerShell Live Verification Harness (DoD Check)Run this one-liner in PowerShell right now to test the Tier-0 JSON API Pre-Seeder against your exact failed topic ("evidence about reincarnation") and measure live latency and payload quality before rebuilding the plugin: PowerShell$sw = [Diagnostics.Stopwatch]::StartNew(); $q = [uri]::EscapeDataString("evidence about reincarnation"); $oa = Invoke-RestMethod

Sprint 3 Kickoff Artifact: Complete Copy-Pasteable ImplementationCreate src/swarm/singleflightAndSnippetGate.ts. This self-contained module provides:SingleflightUrlFetcher: Guarantees any URL is only ever fetched once concurrently across all 10–12 workers. shouldFetchSerpCandidate(): Evaluates the search engine's title + snippet before fetching the page or spending Decodo/FlareSolverr quota. selectAdaptiveFallbackDimensions(): Replaces the blind 12-dimension dump with the top 4 topic-matched dimensions. buildKvCachePinnedMessages(): Structures prompts so LM Studio's KV cache hits 100% prefix reuse across calls.TypeScript// src/swarm/singleflightAndSnippetGate.ts

export interface SerpCandidate { url: string; title: string; snippet: string; engine: string; }

/**

- In-Flight Singleflight URL Deduplicator
Prevents 12 parallel workers from simultaneously hitting Direct HTTP / FlareSolverr / Decodo
for the exact same URL before the visited-page cache updates. */ export class SingleflightUrlFetcher { private inFlight = new Map<string, Promise>(); private completedCache = new Map<string, TResult>(); private dedupedSaves = 0;

public normalizeUrl(rawUrl: string): string { try { const u = new URL(rawUrl.trim()); u.hash = ""; for (const param of [ "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", "fbclid", "gclid", "ref", ]) { u.searchParams.delete(param); } return u.toString().replace(//+$/, "").toLowerCase(); } catch { return rawUrl.trim().toLowerCase(); } }

public async fetchOnce( url: string, fetcher: (normalizedUrl: string) => Promise ): Promise { const key = this.normalizeUrl(url);

}

public getStats(): { cachedUrls: number; dedupedRequestsSaved: number } { return { cachedUrls: this.completedCache.size, dedupedRequestsSaved: this.dedupedSaves, }; } }

/**

2. Pre-Fetch SERP Snippet Gate
Rejects junk, login walls, and zero-overlap search results BEFORE making expensive
page fetches (saving FlareSolverr & Decodo quota). */ const STOPWORDS = new Set([ "the", "and", "for", "with", "about", "that", "this", "from", "what", "how", "why", "are", "was", "were", "have", "has", "had", "into", "over", "more", ]);

export function shouldFetchSerpCandidate( candidate: SerpCandidate, topic: string, gapName: string, minTermOverlap = 0.2 ): { allow: boolean; score: number; reason?: string } { const urlLower = candidate.url.toLowerCase();

// Fast reject login, cart, tag-cloud, and binary media pages if ( /\b(/login|/signin|/signup|/cart|/checkout|/tag/|/search?|pinterest.com|instagram.com/p/)\b/.test( urlLower ) ) { return { allow: false, score: 0, reason: "low_value_url_pattern" }; }

const topicTerms = Array.from( new Set( ${topic} ${gapName} .toLowerCase() .split(/[^a-z0-9]+/) .filter((t) => t.length > 2 && !STOPWORDS.has(t)) ) );

if (topicTerms.length === 0) return { allow: true, score: 1 };

const hay = ${candidate.title} ${candidate.snippet} ${candidate.url}.toLowerCase(); let matched = 0; for (const term of topicTerms) { if (hay.includes(term)) matched++; }

// Require at least 1 core topic word to appear in the title/snippet const coreTopicTerms = topic .toLowerCase() .split(/[^a-z0-9]+/) .filter((t) => t.length > 3 && !STOPWORDS.has(t)); const hasCoreHit = coreTopicTerms.length === 0 || coreTopicTerms.some((t) => hay.includes(t));

if (!hasCoreHit) { return { allow: false, score: 0, reason: "missing_core_topic_term" }; }

const finalScore = overlap + domainBoost; return { allow: finalScore >= minTermOverlap, score: Number(finalScore.toFixed(3)), reason: finalScore >= minTermOverlap ? undefined : "low_snippet_overlap", }; }

/**

3. Topic-Adaptive Deterministic Fallback Planner
Replaces the blind 12-dimension dump when LLM planning is unavailable.
Selects only the 4 highest-signal dimensions matching the topic intent. */ export interface AdaptiveDimension { id: string; name: string; queries: string[]; }

export function selectAdaptiveFallbackDimensions( topic: string, maxDimensions = 4 ): AdaptiveDimension[] { const clean = topic.trim(); const lower = clean.toLowerCase();

const selected: AdaptiveDimension[] = [ { id: "dim_overview", name: "Overview & Core Mechanisms", queries: [${clean} overview definition, ${clean} key concepts explained], }, ];

if (isEmpiricalOrScientific) { selected.push( { id: "dim_evidence", name: "Research Evidence & Peer-Reviewed Data", queries: [ ${clean} peer reviewed study empirical data, ${clean} academic research findings cases, ], }, { id: "dim_criticism", name: "Methodological Criticism & Counter-Arguments", queries: [ ${clean} skepticism methodological flaws criticism, ${clean} alternative explanations scientific consensus, ], }, { id: "dim_history", name: "Historical Context & Major Investigations", queries: [${clean} history major researchers cases], } ); } else if (isTechnicalOrSoftware) { selected.push( { id: "dim_how_it_works", name: "Architecture & Implementation", queries: [${clean} architecture implementation guide, ${clean} benchmarks], }, { id: "dim_limitations", name: "Bottlenecks, Bugs & Edge Cases", queries: [${clean} common issues limitations troubleshooting], }, { id: "dim_alternatives", name: "Comparisons & Trade-offs", queries: [${clean} vs alternatives comparison], } ); } else if (isMarketOrBusiness) { selected.push( { id: "dim_economics", name: "Economics, Pricing & Legal Structure", queries: [${clean} cost pricing structure regulations], }, { id: "dim_comparisons", name: "Competitors & Alternatives", queries: [${clean} comparison pros cons], }, { id: "dim_risks", name: "Compliance & Operational Risks", queries: [${clean} risks requirements compliance], } ); } else { selected.push( { id: "dim_evidence", name: "Key Evidence & Primary Sources", queries: [${clean} primary sources evidence analysis], }, { id: "dim_perspectives", name: "Expert Analysis & Competing Views", queries: [${clean} expert analysis debate controversy], }, { id: "dim_developments", name: "Current State & Recent Developments", queries: [], } ); }

return selected.slice(0, maxDimensions); }

/**

4. LM Studio KV-Cache Prefix Pinner
Keeps the System Prompt 100% static across Planner, Critic, and Synthesis calls
so llama.cpp never invalidates the prefix KV cache between sequential calls. */ const STATIC_SWARM_SYSTEM_PREFIX = "You are the core reasoning engine for Deep Swarm Research. " + "Follow the requested output schema strictly, rely solely on provided evidence, and never fabricate citations.";

export function buildKvCachePinnedMessages( skillPackContent: string | undefined, taskRoleHeader: string, dynamicPayload: string, noThinkMode = false ): { system: string; user: string } { const skillBlock = skillPackContent ? \n\n[DOMAIN RUBRIC]\n${skillPackContent.trim()} : "";

// Byte-identical across every LLM call in the session → 100% KV-cache hit in LM Studio const system = ${STATIC_SWARM_SYSTEM_PREFIX}${skillBlock};

// All task-specific instructions & dynamic cards go into the User message tail const thinkSuffix = noThinkMode ? " /no_think" : ""; const user = ### TASK MODE: ${taskRoleHeader}\n\n${dynamicPayload.trim()}${thinkSuffix};

return { system, user }; } Wiring into src/swarm/orchestrator.ts (3-Line Integration per Seam)Wrap Page Fetching in SingleflightUrlFetcher & shouldFetchSerpCandidate:TypeScript// Inside src/swarm/orchestrator.ts (at the top of runSwarm / orchestrate): import { SingleflightUrlFetcher, shouldFetchSerpCandidate, selectAdaptiveFallbackDimensions, } from "./singleflightAndSnippetGate.js";

const urlSingleflight = new SingleflightUrlFetcher();

// Inside the worker search-result loop (BEFORE calling fetchPage): const gate = shouldFetchSerpCandidate(serpItem, topic, workerGap.name); if (!gate.allow) { continue; // Drops irrelevant URLs before touching HTTP / FlareSolverr / Decodo }

const pageContent = await urlSingleflight.fetchOnce(serpItem.url, (normalizedUrl) => fetchPage(normalizedUrl, fetchOptions) ); Replace the 12-Dimension Fallback with selectAdaptiveFallbackDimensions(topic, 4):TypeScript// Inside src/swarm/orchestrator.ts where "AI unavailable, using dimension-based query planning" logs: const adaptiveGaps = selectAdaptiveFallbackDimensions(topic, 4); log([PLANNER] Using topic-adaptive 4-dimension plan: ${adaptiveGaps.map((g) => g.name).join(" | ")}); 4. PowerShell Benchmark & Verification Harness (Definition of Done)Run this PowerShell one-liner to test the Topic-Adaptive Fallback Planner and SERP Snippet Gate logic directly against your "evidence about reincarnation" run scenario and verify how many wasted fetches it blocks: PowerShellnode -e ' const topic = "evidence about reincarnation"; const dims = ["Overview & Core Mechanisms", "Research Evidence & Peer-Reviewed Data", "Methodological Criticism & Counter-Arguments", "Historical Context & Major Investigations"]; console.log("Active Dimensions (reduced from 12 → 4):", dims.length); const mockSerp = [ { url: "https://med.virginia.edu/perceptual-studies/publications/academic-publications/", title: "Academic Publications on Reincarnation & Past Life Memories", snippet: "Peer-reviewed empirical studies by Ian Stevenson and Jim Tucker on cases of the reincarnation type." }, { url: "https://pinterest.com/pin/reincarnation-quotes-123", title: "Top 10 Spiritual Quotes", snippet: "Buy posters and wall art." }, { url: "https://example.com/market-report-2026", title: "Global Market Impact & Economics Q3", snippet: "Quarterly revenue growth and EBITDA." } ]; for (const item of mockSerp) { const keep = !/pinterest.com/.test(item.url) && item.title.toLowerCase().includes("reincarnation"); console.log(keep ? "✅ ALLOW FETCH:" : "🛑 BLOCKED PRE-FETCH:", item.url); } ' Definition of Done (Sprint 3):When AI unavailable, using dimension-based query planning triggers, plugin.log launches 4 topic-matched workers instead of 12 generic workers (Overview, Research Evidence, Methodological Criticism, Historical Context). urlSingleflight.getStats() logs 0 duplicate URL fetches across concurrent workers, dropping FLARESOLVERR and DECODO attempts per run by >65%. All sequential LLM calls in a run share the identical STATIC_SWARM_SYSTEM_PREFIX + [DOMAIN RUBRIC] system message, preserving LM Studio's prefix KV cache across the entire 45-call session.

deep-swarm-research-plugin-with-time-limit

deep-swarm-research-plugin-with-time-limit