docs: introduce Connect section, connect-string reference, QWP stubs#444
Open
bluestreak01 wants to merge 120 commits into
Open
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 120 commits into
bluestreak01 wants to merge 120 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 549d492
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
Closed
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
The connect/QWP scaffolding rename sweep missed two files, leaving 8 internal link references pointing at moved pages and breaking the build (onBrokenLinks: throw). - changelog.mdx: ilp/overview and telegraf -> /docs/connect/... - concepts/deep-dive/sql-extensions.md: six pgwire links -> /docs/connect/compatibility/pgwire/... yarn build now passes (365 pages, 0 broken links).
The connect-string error-handling section linked to questdb/questdb-enterprise (a private repo, and the referenced sf-client.md path does not even exist there), leaving the QWP error-policy precedence rules inaccessible to public readers. Replace the dead private link with the precedence model inlined and a pointer to the public spec in questdb/java-questdb-client. Cross-checked against the client source: - on_*_error knobs and errorPolicy/errorPolicyResolver are NOT yet implemented (parser does not recognise them, no builder methods); they exist only as the SenderError.Policy spec. Reword accordingly. - errorHandler(...) and error_inbox_capacity are the only wired error-handling surfaces today (WebSocket/QWP only). - Document accepted values + defaults per category and the forced-halt behaviour for PROTOCOL_VIOLATION/UNKNOWN.
The reference client (java-questdb-client) accepts only [http, https, tcp, tcps, ws, wss, udp] and rejects qwpws/qwpwss, so the "long-form alias" claims would break any connect string a reader copies. Standardize on ws/wss across connect-string, rust, and c-and-cpp docs (C/C++ API symbol names like line_sender_qwpws_* are left intact). Also ship the wire-protocols overview as a clean page: remove the "Page in draft" admonition and both TODO stubs, fill Versioning with the actual X-QWP-Max-Version / min(clientMax, serverMax) negotiation, and fix stale "Protocols section" naming. Confirmed wss defaults to 9000 (not 443) against the parser; the explicit call-out stays.
Add a deeper, multi-agent review pass to the docs review-pr skill, modeled on the questdb repo's adversarial reviewer, in both harness variants (.pi uses subagent, .claude uses the Agent tool). - Add review levels (0-2) and a gh pr-based target step that checks out a PR or falls back to the current branch. - Add a change surface map (inbound/outbound links, sidebar neighbors, shared schema/vocabulary, reading paths) as input for the agents. - Add six review agents: structured agents for correctness, placement, information flow, and cross-page navigation/coherence (human + LLM), plus two fresh-context adversarial agents (skeptical reader and LLM one-shot consumer) that run without checklists to escape anchoring. - Add a verification pass that moves false positives to a Downgraded section before reporting.
Member
Author
Review: Connect section, connect-string reference, QWP specsReviewed at level 2 (flagship restructure: new top-level section + 25 page moves + marquee reference pages). Build run against the PR branch; findings verified directly (build, grep, link/anchor resolution, source cross-checks). Verdict: request-changesThe content is genuinely strong — the QWP wire specs and the connect-string reference are some of the best dual-audience material in the repo, the build is clean across 365 pages, and the cross-client schema drift flagged earlier has been properly fixed. But the PR ships with a missing-redirects regression on 25+ indexed pages, an internal self-contradiction in the flagship reference page, several unaddressed review items, unverified performance claims, and non-docs scope creep. These block merge. Blocking issuesCross-cutting1. [Critical] 25+ moved pages have no redirects in this repo's 2. [Major] Scope creep — non-docs artifacts bundled into a docs-content PR.
None are documentation content. Split into separate PR(s); remove 3. [Major] PR description is stale and understates scope. It calls the wire-protocol pages "stubs" and lists client-failover and store-and-forward concepts as out of scope — but the PR ships a ~1,200-line ingress spec, a full egress spec, and complete
|
The .NET client no longer accepts the `auth` key (a pre-built Authorization header value); credentials are supplied as `username`/`password` or `token`. Drop `auth` from the egress query client's Auth key table and update the mutual-exclusivity note to `username`/`password` versus `token`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restructures java.md around QuestDB.connect / Query / Completion / pooled senders, with the existing Sender and QwpQueryClient sections retained as low-level primitives. Switches connect-string examples to ws::/wss:: to match the page's stated QWP focus. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bring the Java client page in line with the current QWP-only facade (client commit 84da57d) and the single-version protocol (PR #39), and make it readable for someone building against the public API. Accuracy: - The facade takes one ws/wss connect string and uses it verbatim for both the ingress sender and the egress query client; each side reads the keys it owns. Drop the old http/https schema-translation model and references to the removed `auth=` key. - Fix wrong reference values: close_flush_timeout_millis defaults to 60s (not 5s), and a Query.submit() acquire timeout throws QueryException, not LineSenderException. - Remove the stale "ingress pins QWP v1" claim and the "v1" fallback in the failover example; the protocol is now a single negotiated version. - State that ingestion only writes to the primary (replicas reject the write upgrade), so failing over to another host needs that host to be promoted to primary. Flushing and store-and-forward: - Describe flush() as a hand-off to the always-on async send engine (in RAM by default, memory-mapped files when sf_dir is set), not a server ack. Cover memory vs store-and-forward mode, memory-grade durability (fsync modes are rejected), and the close-time drain. - Explain that the pool assigns each sender its own slot directory, the exclusive lock that keeps a live sender's data safe, restart recovery, and when to use a separate vs shared sf_dir (with drain_orphans). Readability: - De-jargonize the connection pool, Query builder, Completion, and failover sections (slot, reap, terminal, decorator, worker, "block until terminal" -> plain behavior). - Standardize directional wording on ingress/egress and give the failover section a friendlier tone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The QWP client config in the Java page was accurate on every documented default but missed three real connect-string options. Add them so the page matches the client registry: - compression: note the auto value (compress only when the server offers zstd), alongside the existing raw/zstd. - compression_level: state the default (1) and accepted range (1-22, clamped server-side to 1-9). - client_id: add a configuration-reference row for the egress label sent as the X-QWP-Client-Id header, matching the C/C++ page wording. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite the Rust client page around the v7 `QuestDb` pool API: - store-and-forward column/row senders: flush() + optional wait(ack, timeout) - db-level synchronous DataFrame/Arrow ingestion (db.flush_arrow_batch / db.flush_polars_dataframe) - query -> polars (fetch_all_polars / iter_polars / next_polars), DDL/DML, flow control (byte credits), cancellation, zstd compression - automatic failover/HA: multi-endpoint, reconnect tuning, durable replay - rewritten concurrency section (pool is Send+Sync; borrows are !Send, returned to the pool on Drop; fail-fast at pool_max) - quick start; wait() does-not-guarantee-visibility caveat Also add the C/C++ pooled API draft page + sidebar entry, the API-reference generator scripts, and fix a broken connect-string anchor (rust#flushing -> #sending-data-column-major).
`QuestDb::connect` opens no connection: it validates the connect string and starts the pool, and the first borrow opens the connection (where unreachable/ TLS/auth errors surface). Fix the docs that wrongly claimed eager pre-warming: - pool_size: "warm minimum the reaper keeps; opened lazily on first borrow, not at connect()" (was "opened eagerly at connect()") - add initial_connect_retry row (off=fail fast, on=sync retry, async=background) - Connecting: add a lazy-connect note (errors surface at first borrow) - Concurrency: drop the misleading "pre-warm" wording
Add a normative cross-client behaviour spec for QWP clients (startup, connection, failover, store-and-forward), placed under Wire Protocols alongside the QWP ingress/egress specs. Derived from the Java reference client; the behaviour/config tables are the contract all language clients are aligned to. - frontmatter + "client implementers" audience admonition + scope - reframed from a Java user doc into a normative spec (Java = reference impl) - fixed broken links: dropped the ./qwp-client-ergonomics-issues.md references and corrected #sharp-edges anchors to #known-sharp-edges - Implementation appendix marked non-normative (Java reference internals) - sidebar: Connect > Wire Protocols > QWP Client Behaviour
…erence Add a Connection pooling section (QuestDBClient handle: sender + query pools, BorrowSender/NewQuery/ExecuteSqlAsync, separate ingress/egress configs, pool tuning table); document connect-string keys present in the client but previously undocumented (connect_timeout, max_name_len, sf_* sizing, sf_durability, max_background_drainers, durable_ack_keepalive_interval_millis, ping_timeout, inbox capacities); widen compression_level to 1-22 (server clamps to 9); and fix the close_flush_timeout_millis default (60000, was mis-stated as 5000). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…HASH and failover behavior
…ery contradiction Pre-create the book table with explicit DDL so the SELECT read-back on the ts column works (auto-created tables name the designated timestamp `timestamp`), and document the auto-create naming rule. Reword the At/AtNow/Flush caution so it no longer claims a non-nil return is only ever a local error, matching the store-and-forward and Ingestion errors sections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix correctness and usability gaps flagged in the level-3 review, all verified against the qwip_go source: - m4: distinguish auto_flush=off (disables all triggers) from WithAutoFlushDisabled() (leaves the 8 MiB byte trigger live on QWP). - m6: role mismatch surfaces from Connect (the query pool prewarms query_pool_min at construction), or first BorrowQuery under lazy_connect. - m8: scope the At/AtNano conflict to "within a single flush"; cross-flush mismatches become async server rejections. - m3: document error_inbox_capacity / connection_listener_inbox_capacity (default 256, range 16-1048576, facade-forwarded) in the config table and prose, with the standalone option names. - m7: connection-event inbox is drop-oldest for all payloads; note LastTerminalError() as the reliable terminal-state backstop. - m9: make the AtNano and Decimal snippets define qs inline. - m11: add UUID-reassembly and decimal-scaling decode snippets. - nit: derive hi/lo in the QWP-only column-types snippet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al guidance Refresh the C/C++ pooled API draft (c-api-reference.md) against the current jh_conn_pool_refactor branch and grow it from a surface tour into a usable guide. API corrections (the draft predated recent renames): - borrow_column_sender/column_sender -> borrow_sf_column_sender/ sf_column_sender; column_sender_sync/ack_level -> sf_column_sender_wait/ qwpws_ack_level_* - drop the stale "no new_buffer()" note: row_sender_new_buffer / borrowed_row_sender::new_buffer() now exist and the examples use them - return-vs-drop guidance rewritten for the new contract: the return path retires terminally-latched conns itself; drop_on_return is only for in-doubt frames (must_close inspection is gone by design) New guidance sections: - Which borrow? decision table; pool sizing (borrow-at-cap fails fast, no blocking acquire); Authentication and TLS summary - Arrow ingestion via the C Data Interface (C + C++ fragments) - error-class table (failover_retry/server_rejection/... -> borrow state + action); consolidated durability & backpressure story - null-validity examples, full column-setter table, concurrency (one borrow per worker, runnable 4-thread example), orderly close, and a production-shape example (wss + token + multi-host + sf_dir)
- GO-G1: add a pre-release banner (the pooled facade + QWP query API are not yet in a tagged v4 release, so `go get .../v4` alone won't resolve them) - GO-1: fix the UuidColumn example — pass uint64 halves (int64 wouldn't compile) - GO-2: pre-create `trades TIMESTAMP(ts)` in the quick start so `ts` is the canonical designated-timestamp name; `SELECT ts ... FROM trades` no longer errors with "Invalid column: ts" - GO-3: keep `side` a SYMBOL to match the DDL; move the CharColumn demo to a genuine CHAR column - GO-4: add a FlushAndGetSequence/AwaitAckedFsn ack barrier before the read-back in the full example, so it can't race the async commit - GO-5: drop the empty `qwp-pool` RemoteRepoExample (inline quick start covers it) - GO-6/GO-7: reframe the `qwp-ingest`/`qwp-query` prose to match what the rendered standalone examples actually show - GO-8: repoint the timestamp-naming rule to the designated-timestamp concept page instead of the legacy ILP overview Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- M1: Quick start now registers WithQuestDBErrorHandler and waits for the server ack (FlushAndGetSequence + AwaitAckedFsn) before the read-back, so it no longer models the async-error and read-your-writes anti-patterns; add the async-publish/commit-race hazard to the "Read before building" list. - M2: surface the DEDUP-or-duplicate warning in the Ingestion failover section — the rule applies to every reconnecting/multi-host connection, not only store-and-forward. - M3: make the pre-release install consistent and concrete — pin to the qwip_go branch (go get ...v4@qwip_go) in both the banner and Quick start. - M4: rewrite the Full example on the canonical trades schema (symbol, side, price, amount); leave `book` to mean only the array table. Add DEDUP since the example reconnects over multi-host + sf_dir. - M5: finalize the QWP-only column-types example with AtNow, not AtNano — trades is a microsecond table, so AtNano would conflict with its resolution. Build verified green (onBrokenLinks/onBrokenAnchors: throw); both full programs pass gofmt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Curated minor pass on the Go QWP client page — the correctness/one-shot breakers plus cheap wins: - m6: bind-parameter errors surface from the first Batches() yield (Query returns only a cursor), not from Query. - m7: name the per-category on_*_error connect-string keys instead of the glob. - m8: soften "never as a Flush error" — a latched PolicyHalt rejection can resurface on a later producer call. - m9: mark the config table a curated subset; name the omitted data-loss-timing keys (close_flush_timeout_millis, sf_append_deadline_millis, auto_flush=off). - m11: document the Close ctx asymmetry (db/sender Close take ctx; query/cursor Close do not). - m13: add the read-your-writes / replica-lag caveat to durable ack. - m14: reframe the pgwire/go.md inbound link (QWP, not ILP ingestion). - m16: mark the three LineSender constructors as alternatives (paste-whole compile trap). - nit: add "WebSocket" to the page description for search parity. Build verified green (onBrokenLinks/onBrokenAnchors: throw). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…matches The Go client now retries transport outages indefinitely: reconnect_max_duration_millis bounds only the blocking sync initial connect, and the budget-exhausted terminal event kind is gone. Update the reconnect key tables, connection-event kinds, and the ingress-reconnect section accordingly; document the new WithQuestDBDrainerListener facade option and query_close_timeout_ms key. Also fix documented behavior that diverged from the code: multi-address WithAddress usage (single comma-separated string), Exec never returns QwpFailoverReset, single-addr failover retries the same host, GEOHASH has a value accessor, Query streams during iteration rather than blocking, producer backpressure applies in memory mode too, the fixed-width decimal columns are the QwpSender-only ones, and the failover example now genuinely accumulates and clears on reset.
…erminal and retriable rejections
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.Add per-language file moves and HTTP-side URL redirects in a separate restructure PR.Redirects now handled in questdb/questdb.io#2917.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code