Union absent-column tests pin the pre-LLP-0241 contract, so master is red (#820)

hypaware-core/plugins-workspace/ai-gateway/src/dataset.js

@@ -200,8 +200,12 @@ function withSchemaColumns(source) {
   // engine's streaming fast path. A partition that physically lacks the
   // requested column (the additive schema-drift case this wrapper exists
   // for) surfaces its values as `undefined` holes in the chunk; normalize
-  // them to null, the same "this partition predates the column" value the
-  // row path reads, so accumulators see one representation either way.
+  // them to null so every partition's chunk reads the same way and an
+  // accumulator sees one representation across the merged stream. This is
+  // NOT the value the row path reads: `scan` above pads an absent cell with
+  // `undefined` (LLP 0241 §alignment), and 0241 deliberately left the
+  // null/undefined split between the two paths unsettled, so nothing may
+  // branch on which one it got.
   //
   // A `where` naming a DECLARED-but-physically-absent column can't be
   // handed to the source: this wrapper is the only layer that knows the

llp/0015-query-and-datasets.spec.md

@@ -74,41 +74,45 @@ column a given partition physically lacks, and a parquet-backed source throws
 partition can't satisfy the predicate the union drops `where` for it and lets
 the engine filter. `columns` is always forwarded, which adds no failure the
 merged stream did not already have, but an absent column does **not** read as
-null. A bare identifier projection (`SELECT extra FROM t`, with or without an
-alias, a `LIMIT`, or a predicate on a column every partition has) leaves the
-drifted cell **unresolved**: `executeProject` finds no matching cell, so it emits
-a lazy `evaluateExpr` thunk that would throw, and writes no entry into the row's
-`resolved` map. `collect()` reads the pre-materialized `resolved` map for every
-advertised column and never invokes that thunk, so the key is present with the
-value `undefined`, is not `null`, and `JSON.stringify` drops it. This holds only
-under three conditions, all of them in `executeProject` and `collect()` rather
-than in the union: the consumer goes through `collect()` (as `executeQuerySql`
-does); every row reaching `collect()` carries a `resolved` map, which holds here
-because every in-repo partition's rows come from squirreling's `asyncRow`, the
-only thing that pre-materializes one; **and every output column of that
-projection is a star or a bare identifier**. `executeProject` gates on the last
-one up front (`resolveable`) and emits no `resolved` map at all when it fails, so
-a single non-identifier sibling column (an expression, a function, or even a
-literal) collapses the fast path for the whole result and the drifted column's
-thunk is invoked and throws: `SELECT extra, 1 AS n FROM t` throws even though
-nothing evaluates `extra`. A row that carries no `resolved` map at all does the
-same. Anything
-that *evaluates* the absent column throws squirreling's `ColumnNotFoundError` at
-the first row from a partition without it: a `WHERE` on it, an expression or
-function over it, `ORDER BY`, `GROUP BY`, `DISTINCT`, or an aggregate on it.
-`SELECT *` is unaffected: each partition's rows keep their own shape, so the key
-is simply absent. The column is addressable at all only because the union
-advertises the superset of partition columns; when no partition has it, planning
-fails with the same error unless a wrapper advertises the declared schema on top
-of the union (LLP 0032's `withSchemaColumns`), which keeps such a column
-addressable; the exact value a read of it then yields depends on the read path
-and is not settled here. Pinned by
+null: it reads as `undefined`. The union pads every row out to the column list
+the scan advertised
+([LLP 0241 §alignment](./0241-scan-rows-carry-advertised-columns.decision.md#alignment)),
+so a column a partition physically lacks is still a real cell on the row, and
+that cell resolves to `undefined`. The value is outside `SqlPrimitive` and
+`JSON.stringify` drops it, so a padded column renders as an absent key even
+though the row object owns it, and `Object.keys(row).length` over a star counts
+the **advertised** columns rather than the physical ones. Every **row**-path
+read agrees on that value: reading the row's pre-materialized `resolved` map
+(`collect()`'s fast path), invoking the cell directly, and evaluating the column
+above the scan in a `WHERE`, an expression or function over it, `ORDER BY`,
+`GROUP BY`, `DISTINCT`, or an aggregate. The `scanColumn` column-stream path is
+**not** part of that agreement and is not what padding changed: the union
+forwards each partition's chunks unchanged, and a wrapper above it normalizes
+the holes if it wants them uniform (LLP 0032's `withSchemaColumns` maps them to
+`null`). LLP 0241 deliberately left that `null`/`undefined` split between the
+two paths unsettled. `SELECT *` renders identically to the unpadded star,
+because the key it now owns is one `JSON.stringify` drops. The column is
+addressable at all only because the union advertises the superset of partition
+columns; when no partition has it, planning fails unless a wrapper advertises
+the declared schema on top of the union (LLP 0032's `withSchemaColumns`), which
+keeps such a column addressable; the exact value a read of it then yields
+depends on the read path and is not settled here. Pinned by
 [`test/core/union-source.test.js`](../test/core/union-source.test.js).
 
 > **Corrected (#731, PR #740).** This section previously stated that projecting
 > an absent column "reads as null, never throws". That was never true of the
-> code; the paragraph above records the measured contract. No runtime behaviour
-> changed.
+> code when it was written; the paragraph above records the measured contract.
+> No runtime behaviour changed.
+
+> **Corrected again (#820).** PR #740 was cut before, and merged after,
+> [LLP 0241](./0241-scan-rows-carry-advertised-columns.decision.md), so the text
+> it landed described a tree that no longer existed: an unresolved drifted cell,
+> `undefined` only via `collect()`'s pre-materialized fast path, and a
+> `ColumnNotFoundError` from anything that evaluated the column or merely sat
+> beside it as a non-identifier sibling. 0241's padding removed that seam, and
+> the semantic conflict turned `master` red. The paragraph above records the
+> contract as measured on the current tree. No runtime behaviour changed in this
+> correction either.
 
 > **Extended-by: [LLP 0241 §alignment](./0241-scan-rows-carry-advertised-columns.decision.md#alignment).**
 > The hints above say what a union may forward; LLP 0241 adds what shape the

src/core/query/union-source.js

@@ -102,29 +102,26 @@ export async function* alignRows(rows, columns) {
  * reading the column as null. When a partition can't satisfy the predicate we
  * drop `where` for it and let the engine filter the concatenated stream (it
  * already owns the filter via `appliedWhere: false`). `columns` is always
- * forwarded, which adds no failure the merged stream did not already have, but
- * an absent column does NOT read as null. A bare identifier projection leaves
- * the drifted cell unresolved: squirreling's `executeProject` finds no matching
- * cell, so it emits a lazy `evaluateExpr` thunk that would throw and writes no
- * entry into the row's `resolved` map. `collect()` reads the pre-materialized
- * `resolved` map for every advertised column and never invokes that thunk, so
- * the key is present with the value `undefined` (not `null`, dropped by
- * `JSON.stringify`). That holds only under three conditions, all of them in
- * `executeProject` and `collect()` rather than in the union: the consumer goes
- * through `collect()`; every row reaching `collect()` carries a `resolved` map,
- * which holds here because every in-repo partition's rows come from
- * squirreling's `asyncRow`, the only thing that pre-materializes one; and every
- * output column of that projection is a star or a bare identifier.
- * `executeProject` gates on the last one up front (`resolveable`) and emits no
- * `resolved` map at all when it fails, so a single non-identifier sibling column
- * (an expression, a function, or even a literal) collapses the fast path for the
- * whole result and the drifted column's thunk is invoked and throws:
- * `SELECT extra, 1 AS n FROM t` throws even though nothing evaluates `extra`. A
- * row that carries no `resolved` map at all does the same.
- * Anything that evaluates the column (a `WHERE` on it, an expression over it,
- * `ORDER BY`/`GROUP BY`/`DISTINCT`, an aggregate) throws `ColumnNotFoundError`
- * at the first such row. `SELECT *` keeps each partition's own row shape, so the
- * key is simply absent. `test/core/union-source.test.js` pins both halves.
+ * forwarded, which adds no failure the merged stream did not already have, and
+ * an absent column reads as `undefined` rather than `null` or a throw: every
+ * row is padded out to the scan's advertised column list by `alignRows` below,
+ * so a column a partition physically lacks is still a real cell that resolves
+ * to `undefined` (LLP 0241 §alignment). One consequence is worth stating,
+ * because it is the thing a maintainer gets wrong: `undefined` is outside
+ * `SqlPrimitive` and `JSON.stringify` drops it, so a padded column renders as
+ * an absent key even though the row object owns it. `Object.keys(row).length`
+ * over a star therefore counts the advertised columns, not the physical ones,
+ * and is not a way to discover what a partition holds. What no longer varies is
+ * which ROW path the caller took: reading `resolved`, invoking the cell, and
+ * evaluating the column in a `WHERE`, `ORDER BY`, `GROUP BY`, `DISTINCT` or an
+ * aggregate all agree, where a short row made the last group throw
+ * `ColumnNotFoundError` on the first partition lacking the column. The
+ * `scanColumn` hook below is a DIFFERENT path and padding does not touch it: it
+ * forwards each partition's chunks unchanged, so an absent column's value there
+ * is whatever the partition streams, and a wrapper above the union normalizes
+ * those holes if it wants them uniform (ai-gateway's `withSchemaColumns` maps
+ * them to `null`). LLP 0241 left that `null`/`undefined` split between the two
+ * paths unsettled on purpose. `test/core/union-source.test.js` pins these.
  *
  * Because a sub-source now emits exactly the columns it is asked for (see
  * `parquet-source.js`), forwarding `columns` also determines what the engine