Enterprise-Grade Web Search & Retrieval Plugin for DeepSeek Harness (ctx.web)
dsh-webstack is a high-performance web capability bundle for DeepSeek Harness, integrating SearXNG for metasearch discovery and Scrapling for multi-tier, anti-bot resilient content retrieval.
Engineered specifically for local reasoning models (such as Qwen 2.5/3.8 27B and DeepSeek R1/V3) operating within bounded context windows (32k–90k tokens), dsh-webstack delivers web autonomy without tool bloat, context pollution, or security vulnerabilities.
┌───────────────────────────────────────────────────────────┐
│ DeepSeek Harness Agent │
│ Local LLM (Qwen 3.8 27B / R1) │
└─────────────────────────────┬─────────────────────────────┘
│
Native Schema │ web_search / web_fetch
▼
┌───────────────────────────────────────────────────────────┐
│ Cordis ctx.web Runtime │
└──────────────┬────────────────────────────┬───────────────┘
│ │
Provider: searxng │ Provider: scrapling │
▼ ▼
┌──────────────────────────┐ ┌───────────────────────────────┐
│ SearxngSearchProvider │ │ ScraplingFetchProvider │
│ • Clean JSON Mapping │ │ • Pre-Flight DNS Validation │
│ • Engine Deduplication │ │ • Same-Origin Redirect Lock │
└────────────┬─────────────┘ └───────────────┬───────────────┘
│ │
▼ ▼
┌──────────────────────────┐ ┌───────────────────────────────┐
│ Local SearXNG Instance │ │ Scrapling Multi-Tier Engine │
│ Privacy-preserving meta │ │ T1: curl_cffi Impersonation │
│ aggregation (P95 1.7s) │ │ T2: Headless Playwright/DOM │
└──────────────────────────┘ │ T3: Stealth WAF & Turnstile │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Markdown Economizer (GFM) │
│ 73.1% token context savings │
└───────────────────────────────┘
The model interacts exclusively with the native DeepSeek Harness web_search and web_fetch schema. Advanced scraping parameters, stealth flags, and DNS checks remain entirely encapsulated inside the provider layer.
Content fetching automatically scales across three performance tiers based on target complexity:
| Tier | Engine / Technology | Latency (Mean) | Primary Target |
|---|---|---|---|
| Tier 1 (Default) | curl_cffi (Chrome TLS fingerprint) |
531ms | Documentation, GitHub, Wikipedia, blogs, REST endpoints |
| Tier 2 (Dynamic) | Playwright Headless Chrome | 2,088ms | Client-rendered JavaScript SPAs (React, Vue, Next.js) |
| Tier 3 (Stealth) | StealthyFetcher Anti-Bot Engine |
1,669ms | Cloudflare Turnstile, interstitials, anti-bot WAF challenges |
- Turnstile Solver: Automatic detection and click-coordinate humanization for interactive Cloudflare challenges.
- Fingerprint Camouflage: Chromium canvas noise injection, WebRTC IP leakage suppression, genuine WebGL contexts, Google Search referer spoofing, and tracker blocking (~3,500 domains).
All fetch requests undergo pre-flight DNS address validation prior to connection dispatch:
- Private Subnet Denial: Immediately blocks loopback (
127.0.0.0/8,::1), RFC 1918 subnets, link-local / cloud metadata endpoints (169.254.169.254), and carrier-grade NAT (100.64.0.0/10). - Split-Horizon Protection: Hostnames resolving to mixed public/private addresses are rejected immediately.
- Strict Same-Origin Redirects: Redirects crossing origins or protocols are terminated with
WEB_REDIRECT_BLOCKED. - Zero-DNS Latency Penalty: Security checks complete in < 0.2ms before any TCP handshake or sidecar dispatch.
- 73.1% Token Reduction: Raw HTML is converted to clean, semantic GitHub-Flavored Markdown via Harness's built-in Turndown pipeline.
- 3.72x Effective Capacity: Enables local 27B models to intake nearly 4x more documentation without exceeding context bounds or suffering attention dilution.
- Hard Output Bounds:
fetchMaxOutputCharscapped at 50,000 characters to prevent context blowout.
- Dynamic Loopback Binding: The Python micro-sidecar binds to an OS-allocated ephemeral port (
127.0.0.1:0) and handshakes readiness via JSON stdout ({"status": "ready", "port": ..., "pid": ...}). - Fiber Teardown: Hooks into Cordis lifecycle events to guarantee instantaneous termination (
1.5ms) with zero orphan Python or Chromium processes.
Empirically validated on Apple Silicon (Darwin arm64) against local SearXNG, Scrapling 0.4.7, and Qwen 3.8 27B via oMLX:
| Dimension | Measured Metric | Target & Significance |
|---|---|---|
| SearXNG Search Latency | 1,148ms mean / 1,736ms P95 | 100% success rate across technical API, debugging, and general queries |
| Fast HTTP Fetch (Tier 1) | 531ms mean / 687ms P95 | High-throughput documentation retrieval with browser TLS impersonation |
| Stealth Anti-Bot Fetch (Tier 3) | 1,669ms mean | Solves Cloudflare Turnstile & interstitial challenges autonomously |
| Context Token Savings | 73.1% reduction (3.72x multiplier) | Compresses ~32.6k raw HTML tokens down to ~8.8k semantic GFM tokens |
| SSRF Threat Prevention | 100% intercepted (8/8 attack vectors) | < 0.2ms abort decision; zero network requests dispatched to private nets |
| Bridge Startup / Teardown | 114.7ms boot / 1.5ms exit | Dynamic ephemeral port assignment; 0 orphan/zombie processes |
📊 For complete per-query distributions, latency variance, and test setups, refer to the Benchmark Report.
# In your DeepSeek Harness workspace:
pnpm add dsh-webstack
# or via npm
npm install dsh-webstack- SearXNG: Deploy via Docker Compose with JSON enabled on
http://127.0.0.1:8080(see SearXNG Setup Guide). - Scrapling: Install into a virtual environment with stealth extras (see Scrapling Setup Guide):
pip install "scrapling[stealth]" && playwright install chromium
Apply cordis.patch.yml to your DeepSeek Harness profile to connect ctx.web:
- id: web
config:
searchProvider: searxng
fetchProvider: scrapling
- insert:
- id: dsh-webstack
name: dsh-webstack
config:
searxng:
baseURL: http://127.0.0.1:8080
scrapling:
pythonBinary: python3
enableDynamicFallback: true
enableStealthFallback: trueComprehensive guides and technical documentation are available in the docs/ directory:
| Guide | Description |
|---|---|
| SearXNG Setup Guide | Official Docker Compose deployment, settings.yml tuning, secret key generation, and Valkey caching. |
| Scrapling Environment Setup | Python 3.11+ virtual environment setup, Playwright Chromium installation, and sidecar verification. |
| Configuration Reference | Complete Cordis patch layer, parameter catalog for SearXNG and Scrapling, and context bounds tuning. |
| Programmatic Usage Guide | Standalone Node.js & TypeScript usage of SearxngSearchProvider and ScraplingFetchProvider without Harness. |
| Troubleshooting & FAQ | Solutions for 403 Forbidden, 429 rate limiting, missing Python paths, Playwright binaries, and SSRF blocks. |
| Benchmark Report | Empirical latency distributions, context token reduction stats, and SSRF security test telemetry. |
The codebase maintains automated test coverage across search normalization, security filters, and live integration:
# Run the complete test suite (35 tests, 10 suites)
pnpm test
# Run focused test suites
pnpm test:security # SSRF validation, private IP blocking, redirect policies
pnpm test:searxng # Query parameter encoding, response normalization
pnpm test:cancellation # Signal cancellation, child process termination
pnpm test:integration # Live HTTP fetch, redirect enforcement, Cordis seam
# Run empirical benchmark suite
pnpm run benchmarkMIT © Kuba