docs(qwp): document canonical RFC 4122 UUID byte order

documentation/concepts/delivery-semantics.md

@@ -38,6 +38,11 @@ the server confirms a batch, the client reconnects and re-sends. If the
 server had already committed the batch but the acknowledgement was lost in
 flight, the second send produces duplicates.
 
+QWP's `wireSeq` cannot suppress this replay: it is assigned by receive order,
+exists only for response correlation on one connection, and resets after a
+reconnect. Requests carry no persistent message identifier that the server can
+use to recognize the same frame on the next connection.
+
 This path applies to every QuestDB client deployment.
 
 ### Multi-host failover replay

documentation/concepts/materialized-views.md

@@ -563,6 +563,93 @@ This happens asynchronously, minimizing write performance impact.
 
 ## Enterprise features
 
+### Restricted access with row expiry
+
+An `EXPIRE ROWS` policy on a passthrough materialized view filters expired rows
+from queries before background cleanup removes them. In Enterprise, readers
+with column-level SELECT grants also need permission on columns used to enforce
+the policy, even when those columns are absent from the query's output.
+
+#### Direct materialized-view access
+
+Grant the requested columns and the policy's predicate, partition, and order
+columns. For example, on a materialized view `mv` with columns `sym`, `k`, `v`,
+`secret`, and designated timestamp `ts`:
+
+| Expiry policy | Grants needed for `SELECT sym FROM mv` |
+| --- | --- |
+| `WHEN v < 2.0` | `SELECT ON mv(sym, v)` |
+| `WHEN ts < '2025-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` |
+| `KEEP LATEST PARTITION BY k` | `SELECT ON mv(sym, k)` |
+| `KEEP HIGHEST v PARTITION BY k` | `SELECT ON mv(sym, v, k)` |
+
+Column-level grants implicitly include the designated timestamp. A table-level
+SELECT grant covers all columns, including those needed by the policy.
+
+Review direct grants whenever you enable or change expiry. Granting a policy
+column lets the reader query that column explicitly. If it must remain hidden,
+use an ordinary SQL view as described below. Different readers can use either
+access pattern.
+
+**COUNT limitation:** `SELECT count() FROM mv` can require SELECT on unrelated
+columns as well as policy columns. With sufficient policy-column grants, use an
+explicit timestamp projection to count retained rows:
+
+```questdb-sql
+SELECT count() FROM (SELECT ts FROM mv);
+```
+
+This still requires the policy-column permissions. Direct COUNT also works with
+a table-level SELECT grant.
+
+#### Hide policy columns with an ordinary view
+
+After configuring expiry and waiting for it to apply, an authorized administrator
+can create an ordinary SQL view with only the intended output columns:
+
+```questdb-sql
+CREATE VIEW mv_public AS (SELECT sym FROM mv);
+GRANT SELECT ON mv_public TO reader;
+```
+
+The reader needs the appropriate connection permission, such as `PGWIRE` or
+`HTTP`, and SELECT on `mv_public`. They do not need any grant on `mv` or its
+policy columns. Both SELECT and COUNT through `mv_public` operate on retained
+rows, and its schema exposes only `sym`. Different output column sets can use
+separate ordinary views.
+
+#### Change expiry beneath an existing ordinary view
+
+Adding expiry, or replacing a policy with one that uses a new hidden column, can
+make reads through an existing ordinary view fail with access denied. The view's
+saved dependencies must be refreshed by reissuing its complete, unchanged
+original definition:
+
+```questdb-sql
+ALTER MATERIALIZED VIEW mv SET EXPIRE ROWS WHEN secret < 20;
+SELECT wait_wal_table('mv');
+ALTER VIEW mv_public AS (SELECT sym FROM mv);
+```
+
+Run these statements in order as an authorized administrator, waiting for each
+to complete successfully. The WAL wait ensures that the policy has been applied
+before `ALTER VIEW` collects its dependencies; an ALTER acknowledgement alone,
+including over the PostgreSQL protocol, does not establish application.
+
+The `ALTER VIEW` statement preserves existing grants on `mv_public`. Use the
+original definition for your view, including any filters and output restrictions.
+Readers can receive access-denied errors between policy application and the
+view-definition update. Reissuing the definition before policy application does
+not pick up the new dependencies.
+
+Background view compilation and `COMPILE VIEW` do not refresh these dependency
+permissions. This procedure also applies to timestamp-only expiry when the
+ordinary view predates the policy: implicit timestamp permission on a direct
+materialized-view grant does not extend to an ordinary-view-only reader.
+
+`ALTER MATERIALIZED VIEW mv DROP EXPIRE` requires no ordinary-view repair after
+it applies. Rows that have already been physically cleaned up are not restored.
+
 ### Replicated views
 
 Replication of the base table is independent of materialized view maintenance.

documentation/connect/clients/c-and-cpp.md

@@ -696,7 +696,9 @@ on_error:;
   the 2 MiB target; if 8 rows still exceed 4 MiB — which takes very large
   string, binary, or array values — the flush fails instead of splitting.
 - **Recovery depends on `in_doubt`, not on the error code.** Check
-  `line_sender_error_in_doubt` (C++: `e.in_doubt()`). False means the queue
+  `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes the
+  failed operation's input, not earlier independent flushes or replay from an
+  application checkpoint. False means the queue
   never took the frame and the chunk is intact: re-flush it. True means
   delivery is uncertain, so `wait` for what the queue already holds, and resend
   the chunk only where the table's dedup keys make duplicate rows harmless. A
@@ -776,13 +778,18 @@ See `qwp_sender.h` for the exact signatures. Complete list:
 | `column_bool` | LSB-first packed bitmap | `BOOLEAN` |
 | `column_ts` + `qwp_ts_unit` (`_micros` / `_nanos`) | int64 since epoch | `TIMESTAMP` / `TIMESTAMP_NS` |
 | `column_date` | int64 millis since epoch | `DATE` |
-| `column_uuid` | 16 bytes | `UUID` |
-| `column_long256` | 32 bytes (4 LE limbs) | `LONG256` |
+| `column_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` |
+| `column_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` |
 | `column_ipv4` | uint32 | `IPV4` |
 | `column_str` | Arrow Utf8 offsets + bytes | `VARCHAR` |
 | `column_binary` | Arrow Binary offsets + bytes | `BINARY` |
 | `symbol_i8` / `_i16` / `_i32` | dict codes + Utf8 dictionary | `SYMBOL` |
 
+`column_uuid` takes the UUID's canonical RFC 4122 bytes, exactly as they are
+written in the textual form, and byte-swaps them into QWP wire order for you.
+The row-oriented `line_sender_buffer_column_uuid` is the exception: it takes
+the two 64-bit wire halves, `(lo, hi)`.
+
 Designated timestamp (exactly once per chunk, before flush):
 `at_nanos` / `at_micros` / `at_millis` / `at_seconds` (millis and
 seconds are widened to micros on the wire). Decimals, geohash, arrays, and the
@@ -873,17 +880,137 @@ bool ingest(questdb_db* db, struct ArrowArray* array,
   caller keeps `schema`. On failure check `array->release != NULL` before
   invoking it.
 - Per-column wire-type hints (`qwp_arrow_override`: force
-  SYMBOL/VARCHAR, IPv4, char, geohash precision) steer encoding without
-  touching the Arrow schema.
+  SYMBOL/VARCHAR, IPv4, char, geohash precision, UUID, LONG256) choose the
+  wire type without touching the Arrow schema. An override wins for its
+  column over any field metadata the schema carries.
 - To append Arrow **columns** into a chunk alongside hand-built ones, use
   `qwp_chunk_append_arrow_column`, or
   `qwp_arrow_import_new` + `..._append_arrow_import` to import once
-  and slice across many chunks.
+  and slice across many chunks. Neither takes an overrides array, so an
+  IPv4, char, geohash, UUID, or LONG256 column has to carry its claim as
+  field metadata instead. The SYMBOL choice is still available on the import
+  path: `qwp_arrow_import_new` takes a `symbol_mode` argument
+  (`qwp_symbol_mode_auto`, `_symbol`, `_not_symbol`).
 - Dictionary-encoded string columns map to `SYMBOL` by default; plain Utf8 to
   `VARCHAR`. `qwp_sender.h` lists every Arrow type the client accepts, and
   the kinds it rejects (`Struct`, `Map`, `Interval`, ...); a rejected type
   fails with `line_sender_error_arrow_unsupported_column_kind`.
 
+### Binary columns: UUID, LONG256, and opaque bytes
+
+Binary Arrow columns land as `BINARY` unless the column *claims* a richer
+type. The width of a column claims nothing on its own: a bare
+`FixedSizeBinary(16)` is opaque bytes, not a UUID. A claim comes from the
+schema or from an override:
+
+| Claim | Lands as |
+| --- | --- |
+| `ARROW:extension:name = arrow.uuid` on `FixedSizeBinary(16)` | `UUID` |
+| `questdb.column_type = uuid` field metadata | `UUID` |
+| `questdb.column_type = long256` field metadata | `LONG256` |
+| `qwp_arrow_override_uuid` / `qwp_arrow_override_long256` | `UUID` / `LONG256` |
+
+UUID bytes are canonical RFC 4122 big-endian and the client byte-swaps them
+into QWP wire order; LONG256 bytes are little-endian limbs, low limb first,
+and go out verbatim. The `questdb.column_type` claims and the two overrides
+also apply to variable-length `Binary` / `LargeBinary` / `BinaryView`
+columns, where every non-null value must then be exactly 16 or 32 bytes. The
+`arrow.uuid` extension is the exception: the Arrow spec fixes its storage to
+`FixedSizeBinary(16)`, so the client rejects the label on any other type. A
+claim whose width doesn't match fails with
+`line_sender_error_arrow_ingest`.
+
+:::caution Behaviour change
+
+Before client 7.0.0 a bare `FixedSizeBinary(16)` or `(32)` column became
+`UUID` or `LONG256` on width alone, with no claim needed. It is now `BINARY`
+unless the column carries one of the claims above, so a batch that used to
+produce a `UUID` column now produces a `BINARY` one and reports no error.
+
+The UUID byte order at the API boundary changed in the same release. Code
+written against an earlier version passed QWP wire-order bytes to
+`qwp_chunk_column_uuid` and `qwp_reader_query_bind_uuid`; those values are
+now stored with their bytes reversed, also with no error.
+
+:::
+
+#### Claiming in the schema
+
+Both metadata claims are attached to the Arrow `Field`, so you make them
+wherever the batch is built. In Arrow C++:
+
+```cpp
+// The standard Arrow extension label. FixedSizeBinary(16) only.
+auto trade_id = arrow::field("trade_id", arrow::fixed_size_binary(16))
+    ->WithMetadata(arrow::key_value_metadata(
+        {"ARROW:extension:name"}, {"arrow.uuid"}));
+
+// The QuestDB claim, also valid on Binary / LargeBinary / BinaryView.
+auto order_hash = arrow::field("order_hash", arrow::fixed_size_binary(32))
+    ->WithMetadata(arrow::key_value_metadata(
+        {"questdb.column_type"}, {"long256"}));
+
+auto batch_schema = arrow::schema({
+    arrow::field("ts", arrow::timestamp(arrow::TimeUnit::NANO)),
+    trade_id,
+    order_hash});
+```
+
+`questdb.column_type = uuid` has the same shape with `uuid` as the value. Use
+it in place of `arrow.uuid` when the bytes sit in a variable-length binary
+column, which the extension label doesn't allow.
+
+Export the batch built against that schema through `arrow::ExportRecordBatch`
+and flush it exactly as above — the claims travel with it, and the flush call
+needs no extra arguments.
+
+#### Claiming at the call site
+
+An override claims the type per flush and leaves the schema alone. Fill in a
+`qwp_arrow_override` per column and pass the array to any
+`flush_arrow_batch*` call, where the example above passes no overrides:
+
+<Tabs defaultValue="cpp" groupId="c-cpp">
+<TabItem value="cpp" label="C++">
+
+```cpp
+using namespace questdb::ingress::literals;
+
+const ::qwp_arrow_override overrides[] = {
+    {"trade_id",   sizeof("trade_id") - 1,   qwp_arrow_override_uuid,    0},
+    {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0},
+};
+
+sender.flush_arrow_batch_and_wait(
+    "trades"_tn, array, schema, "ts"_cn,
+    overrides, std::size(overrides));
+```
+
+</TabItem>
+<TabItem value="c" label="C">
+
+```c
+const qwp_arrow_override overrides[] = {
+    {"trade_id",   sizeof("trade_id") - 1,   qwp_arrow_override_uuid,    0},
+    {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0},
+};
+
+bool ok = qwp_sender_flush_arrow_batch_at_column_and_wait(
+    sender, QDB_TABLE_NAME_LITERAL("trades"), array, schema,
+    QDB_COLUMN_NAME_LITERAL("ts"),
+    overrides, sizeof(overrides) / sizeof(overrides[0]),
+    qwpws_ack_level_ok, &err);
+```
+
+</TabItem>
+</Tabs>
+
+`arg` (the trailing `0`) carries the geohash precision for
+`qwp_arrow_override_geohash` and is unused by every other kind. An override
+that names a column the batch doesn't have, repeats another override's
+column, or carries an unknown kind fails with
+`line_sender_error_invalid_api_call`.
+
 ## Querying data
 
 Get a reader (QWP/WebSocket only), prepare/execute SQL, then stream batches and
@@ -1084,6 +1211,12 @@ For width-independent access, use `column::visit` and
 mantissa as little-endian two's-complement bytes. Check for null before
 decoding it.
 
+`qwp_reader_column_data_get_bytes` also serves `UUID` and `LONG256`. It hands
+back UUID values as 16 canonical RFC 4122 big-endian bytes — the decoder has
+already reversed them out of wire order, so they match what
+`qwp_chunk_column_uuid` and `bind_uuid` take — and LONG256 values as 32
+little-endian limb bytes, low limb first, verbatim from the wire.
+
 ### Parameterised queries
 
 Prepare then bind: C `qwp_reader_prepare` + `qwp_reader_query_bind_*` +
@@ -1103,8 +1236,8 @@ outlive any cursor it produces. The complete bind surface (C
 | `bind_decimal64` / `bind_decimal128` / `bind_decimal256` | unscaled value + scale | `DECIMAL` |
 | `bind_geohash` | bits + precision | `GEOHASH` |
 | `bind_varchar` | UTF-8 string | `VARCHAR` |
-| `bind_uuid` | 16 bytes | `UUID` |
-| `bind_long256` | 32 bytes | `LONG256` |
+| `bind_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` |
+| `bind_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` |
 | `bind_binary` | bytes + length | `BINARY` (not yet accepted server-side) |
 | `bind_ipv4` | uint32, host order | `IPV4` (not yet accepted server-side) |