Support ClickHouse 24.8 meta-less result streams in Table view (limited support)

[BorisTyshkevich]

Goal

Support ClickHouse 24.8 in a deliberately limited but data-safe mode when JSONStringsEachRowWithProgress / JSONEachRowWithProgress returns row records without a meta record.

The support contract for 24.8 is:

Query execution and faithful Table result display are supported. Features that require ClickHouse result-type metadata are not guaranteed on 24.8.

The primary invariant is: never discard query values because result metadata is unavailable.

Problem

applyStreamLine() currently learns both column names and ClickHouse types only from a {"meta":[...]} line. A row is converted to an array by mapping through result.columns:

if (json.meta) {
  result.columns = json.meta.map((m) => ({ name: m.name, type: m.type }));
} else if (json.row) {
  result.rows.push(result.columns.map((c) => row[c.name]));
}

On the confirmed 24.8 servers below, no meta line arrives. result.columns therefore remains empty and every successful row becomes [], so the UI silently renders an empty result despite the wire response containing the data.

Confirmed live on pinned images:

• 24.8.14.39 (OSS)
• 24.8.14.10547.altinitystable

The exact first ClickHouse 25.x release that emits meta for these formats has not been established. This issue therefore targets the observed protocol behavior rather than guessing a version boundary.

Required behavior

Meta-first streams

Existing behavior must remain unchanged when meta arrives before rows:

• column names come from meta;
• ClickHouse types come from meta;
• rows are stored in metadata column order;
• existing typed Table/KPI/chart/log behavior remains unchanged.

Meta-less streams

When a row arrives while result.columns is empty:

1. Establish columns from that row's keys, preserving Object.keys(row) order.
2. Represent unavailable ClickHouse types with the existing string field using the explicit sentinel type: ''.
3. Store the row values in that established column order.
4. Store all subsequent rows using the same established column order.
5. Do not wait or buffer rows in the hope that meta will arrive later; the reproduced 24.8 stream reaches EOF without metadata.

Conceptually:

if (json.row) {
  const row = json.row as Record<string, unknown>;

  if (result.columns.length === 0) {
    result.columns = Object.keys(row).map((name) => ({ name, type: '' }));
  }

  // existing row-cap behavior remains in force
  result.rows.push(result.columns.map((column) => row[column.name]));
}

The implementation may be structured differently, but the behavior above is required.

Degraded-functionality contract

type: '' means ClickHouse result type is unknown, not String and not an inferred type.

For a meta-less result:

• Table: supported. Column names and raw query values must be displayed faithfully.
• Type display / type-aware cell formatting: may fall back to generic/unknown behavior.
• Automatic chart role/type detection: not part of the 24.8 support guarantee.
• KPI type eligibility / tuple parsing: not part of the 24.8 support guarantee.
• Logs auto-detection: not part of the 24.8 support guarantee.
• Other consumers that require column.type must fail closed or use their existing unknown-type behavior; they must not throw, silently discard row data, or fabricate a ClickHouse type.

No value-based ClickHouse type inference is required or desired in this issue. In particular, numeric-looking strings, dates, UUIDs, decimals, large integers, enums, etc. must not be assigned synthetic ClickHouse types from their JSON representation.

Non-goals

This issue does not require:

• reconstructing full ClickHouse types from row values;
• an auxiliary metadata query;
• restoring KPI/chart/log feature parity on 24.8;
• identifying the exact first 25.x release containing the upstream metadata change;
• changing Column.type to optional.

Those can be considered separately if broader old-server support is later required.

Tests

Add regression coverage for the actual failure mode, not only row-before-meta:

• a stream containing one row and EOF with no meta ever arriving;
• multiple meta-less rows, preserving the first row's column order and every row's values;
• precision-sensitive/string values remain byte-for-byte/value-for-value intact through normalization;
• row-cap behavior still applies on meta-less streams;
• normal meta -> row behavior remains unchanged;
• representative type-dependent consumers tolerate type: '' without throwing or inventing typed behavior.

Re-run the existing live compatibility/spike harness against the pinned 24.8 images and verify ordinary Table query results now match expected values.

Documentation / support matrix

Once the regression is verified, document ClickHouse 24.8 as limited support in the support-matrix work (#71), using language equivalent to:

ClickHouse 24.8: Limited support — query execution and Table results are supported. Typed result features such as automatic charts, KPI interpretation, logs detection, and type-aware formatting may be unavailable because these servers do not provide result metadata in the streaming format used by SQL Browser.

Do not describe 24.8 as full-fidelity support.

Acceptance criteria

• Meta-less row records populate columns from row keys and preserve all row values.
• Unknown result types are represented explicitly as type: ''; no ClickHouse type is inferred from values.
• Multiple rows with no meta through EOF render correctly in Table view.
• Existing meta-first behavior and typed functionality on newer ClickHouse versions are unchanged.
• Type-dependent consumers do not crash or silently discard data when type === ''.
• Existing row-cap/progress/exception behavior is unchanged.
• Live regression verification passes on 24.8.14.39 and 24.8.14.10547.altinitystable for ordinary Table queries.
• Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71/support documentation records 24.8 as limited, not full, support with the scope above.
• ADR-0005 / ADR-0005: adopt @clickhouse/client-web behind the SQL Browser transport adapter #585 evidence cross-references this issue as the production compatibility fix, independent of the rejected @clickhouse/client-web adoption decision.

Background

This was discovered during the #585 / ADR-0005 @clickhouse/client-web validation spike. The current production transport and the official-client candidate reproduced the same incorrect empty-row behavior on 24.8, so this is a production compatibility defect rather than a client-library migration defect.

ClickHouse added metadata to the progress-bearing JSON row formats after the tested 24.8 line. Newer tested servers (26.3 / 26.6) provide metadata and do not require this fallback.

Relates to

• ADR-0005: adopt @clickhouse/client-web behind the SQL Browser transport adapter #585 / docs/ADR-0005-clickhouse-web-client.md — discovery/evidence
• Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71 — supported ClickHouse version matrix and 24.8 limited-support announcement
• Roadmap to 1.0.0 #68 — roadmap to 1.0

[BorisTyshkevich]

Triage decision updated after discussion: ClickHouse 24.8 will be supported in a limited, data-safe mode.

The implementation target is now explicit in the issue body:

• ordinary query execution and Table results are supported on 24.8;
• meta-less rows must preserve column names and all values;
• unavailable ClickHouse result types use the explicit sentinel type: '';
• no value-based ClickHouse type inference;
• type-dependent features (automatic charts, KPI interpretation, logs auto-detection, type-aware formatting) are outside the 24.8 support guarantee and must fail closed rather than fabricate semantics;
• regression coverage must include a true EOF-with-no-meta stream and live verification on the pinned 24.8 OSS and Altinity Stable images;
• Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71 should announce 24.8 as limited support, not full-fidelity support.

This replaces the earlier conditional support-floor discussion: the product decision for this issue is to support 24.8 at Level 1 degradation.

[BorisTyshkevich]

Ship log — #627

Unit	Status	PR
whole issue (unphased)	shipped	#674

#627 — handoff

• Landed: the meta-less fallback in src/core/stream.ts (one production file). When a
  row arrives while result.columns is empty, columns are established from
  Object.keys(row) with the sentinel type: ''; every later row maps through that fixed
  order. meta, progress, pct, exception, cancellation, and row-cap behavior are
  unchanged. Live-verified on both pinned 24.8 digests, evidence committed under
  docs/evidence/627/.
• Decisions taken (not spelled out in the issue):
  • No production change to any result-type consumer. The audit found every one already
    fails closed on '' (parseClickHouseType('') → null, isNumericType('') → false,
    chartStripType('') → '', logs regexes match nothing, isOptionColumnType('') →
    false, truthiness-gated type badge, autoPanel → Table). Criterion 5 is therefore
    satisfied by regressions proving fail-closed behavior, not by new guards.
  • resolveLogsShape is a deliberate exception. An explicit cfg.time/cfg.msg
    resolves by column name, never type, so an authored Logs panel now resolves against
    a meta-less result (untyped string formatted as time) where columns was previously
    always []. Permitted by the degraded-functionality contract — no throw, no discarded
    rows, no fabricated type — and now pinned by a test. The approved plan's audit called
    itself exhaustive while covering only autoPanel's auto-detect path; the conclusion
    held, the exhaustiveness claim did not.
  • Live verification is one-off with committed evidence, not a permanent harness. Two
    immutable historical digests; the ADR-0005: adopt @clickhouse/client-web behind the SQL Browser transport adapter #585 spike harness was retired in Epic: extract the Fetch-native ClickHouse client into a reusable package #630 Phase 8
    (280b54b).
  • Criterion 8 is satisfied on both sides: README gains a ## ClickHouse server compatibility section and Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71's body records the limited-support decision. Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71 stays
    open — its browser/IdP/mobile scope is untouched.
  • Roadmap Roadmap to 1.0.0 #68 unchanged — this fulfils a decision inside existing Document the supported-browser matrix (browsers, ClickHouse versions, IdP requirements) #71 work rather than
    reshaping a track.
• Deviations from the spec: none substantive. The iso-altinity Docker network did not
  pre-exist in the sandbox and was created for the live run.
• Known, deliberately unfixed: a row-then-late-meta stream would leave stored rows
  bound to row-key order while headers flip to meta order. Unreachable — meta, when
  present, is always the first line in both …WithProgress formats; the json.meta arm is
  untouched; guarding it needs the extra StreamResult state the plan forbids. Pre-Support ClickHouse 24.8 meta-less result streams in Table view (limited support) #627 the
  same sequence produced an empty row 0. No values are discarded either way.
• New invariant found in internal review: every stored row's width equals
  result.columns.length. A zero-key {"row":{}} left columns.length === 0 ambiguous
  between "not established" and "established empty", so a later real row re-established
  columns under an already-pushed zero-width row (verified: rows = [[], ['1','2']] against
  2 columns). Held pre-Support ClickHouse 24.8 meta-less result streams in Table view (limited support) #627 and for meta-first streams; the fallback opened the hole. Fixed
  in edcb8ba without adding StreamResult state, sabotage-checked.
• Deferred acceptance criteria: none. All nine claimed and proven.
• Follow-ups filed: none.
• Plan review: ChatGPT-authored, approved by Fable/high on pass 3 of 5 —
  https://chatgpt.com/c/6a7cc1bc-b294-83eb-ac4c-1d1df6e22d7f
• Internal review: 1 targeted read-only pass (Medium risk) → 3 real findings, all fixed
  in edcb8ba (zero-key invariant; resolveLogsShape coverage gap; 3 ADR-0005 passages
  still asserting the defect in the present tense). 1 further item examined and dismissed
  with reasons (late-meta, above).
• Coordinator-verified, not self-reported: full explicit gate green at every head (7550
  tests at the certified head 97d65a0; 7548 before the two review-pass tests landed),
  check-boundaries 1899 files/10 rules, build); the live 24.8 OSS run re-executed
  independently — captured stream byte-identical to the committed evidence,
  version() = 24.8.14.39, zero meta records; e2e Chromium green with one WebKit
  page.goto flake (a different test on each of two runs, spec passes 12/12 in isolation).
• Code review: 3 formal pr-mode passes in one ChatGPT conversation (seeded from the plan
  session, so plan + review share one conversation) —
  https://chatgpt.com/c/6a7cc1bc-b294-83eb-ac4c-1d1df6e22d7f
  • pass 1 (edcb8ba) → 1 accepted, 0 rejected: docs/evidence/627/README.md pinned its
    "Tested commit" to the live-capture SHA and claimed the fallback was "unchanged since",
    which edcb8ba itself falsified by changing the same json.row arm. Fixed in e94b7c5,
    which also added tests/unit/evidence-627-replay.test.ts.
    fix(#627): support ClickHouse 24.8 meta-less result streams in Table view #674 (comment)
  • pass 2 (e94b7c5) → 2 accepted, 0 rejected, both caused by pass 1's own fix: a SHA
    pinned as the branch's "final head" (in prose added by the commit after it), and a
    "byte-identical" claim that is really deep structural equality. Fixed in e9ca678.
    fix(#627): support ClickHouse 24.8 meta-less result streams in Table view #674 (comment)
  • pass 3 (97d65a0) → certified, VERDICT: SHIP, 0 accepted, 1 rejected (non-blocking:
    this PR body's own commit/test counts had gone stale — corrected by a body edit, which is
    not a commit and so did not void certification).
    fix(#627): support ClickHouse 24.8 meta-less result streams in Table view #674 (comment)
  • attemptsUsed was 1 on every pass — no retry consumed an extra session slot.
• Root cause behind passes 1-2, and the durable fix: committed evidence backing an
  evidence-gated acceptance criterion asserts things about code state, so it goes stale the
  moment a later commit touches the path it attests to. Correcting the prose twice was
  treating symptoms. The structural fix is tests/unit/evidence-627-replay.test.ts: it
  replays the committed raw captures through the real streamLines() -> applyStreamLine()
  path at whatever commit npm test runs, so a future decoder change now fails the suite
  instead of silently invalidating the evidence. 97d65a0 then removed review chronology
  (including a describe name containing "PR review pass 1") from the permanent files, on the
  coordinator's own initiative rather than from a finding — same class, third instance.
• Unresolved findings: none.
• Review outcome: certified head 97d65a0; merged as
  3b6920874095a1ce97148d8d90d86741255ce5c2; 8/8 CI checks green at the certified head
  (examples skipped by path filter); findings 3 accepted / 1 rejected / 0 unresolved.