From 13ca3df562107dad31e82d6afa02ee0f407ad6e2 Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Tue, 8 Sep 2026 11:54:07 +0200 Subject: [PATCH 1/6] Preserve and retire schema-rejected QWP batches Add REJECT_AND_CONTINUE with lease-scoped failures, bounded rejection notifications, and durable raw recovery copies before retirement. Preserve rejected ranges synchronously on the I/O thread and reuse existing queue watermark and shutdown cleanup machinery. Cover transactional and recovered groups, dictionary continuity, pool recovery, copy retries, and blocked-copy shutdown. Document compatibility and operational behavior. Validation: full core suite passed with 3,492 tests, zero failures/errors, and seven skipped; git diff --check passed. --- README.md | 52 +- .../client/LineSenderServerException.java | 1 + .../io/questdb/client/QuestDBBuilder.java | 42 +- .../main/java/io/questdb/client/Sender.java | 64 +- .../java/io/questdb/client/SenderError.java | 90 ++- .../qwp/client/QwpWebSocketSender.java | 212 +++++- .../client/sf/cursor/BackgroundDrainer.java | 139 +++- .../client/sf/cursor/CursorSendEngine.java | 41 ++ .../sf/cursor/CursorWebSocketSendLoop.java | 413 ++++++++++- .../sf/cursor/DefaultSenderErrorHandler.java | 8 +- .../qwp/client/sf/cursor/MmapSegment.java | 73 ++ .../client/sf/cursor/PersistedSymbolDict.java | 53 ++ .../sf/cursor/RejectedMiniSlotArchive.java | 486 +++++++++++++ .../qwp/client/sf/cursor/SchemaPreserver.java | 73 ++ .../sf/cursor/SchemaRejectionState.java | 254 +++++++ .../qwp/client/sf/cursor/SegmentRing.java | 15 + .../sf/cursor/SenderErrorDispatcher.java | 44 +- .../qwp/client/sf/cursor/SlotEpoch.java | 116 ++++ .../io/questdb/client/impl/PooledSender.java | 29 + .../io/questdb/client/impl/QuestDBImpl.java | 26 +- .../io/questdb/client/impl/SenderPool.java | 77 ++- .../io/questdb/client/impl/SenderSlot.java | 17 + .../questdb/client/test/SenderErrorTest.java | 3 +- .../sf/BackgroundDrainerEndToEndTest.java | 66 ++ .../BackgroundDrainerSetupFailureTest.java | 85 ++- ...WebSocketSendLoopCatchUpAlignmentTest.java | 76 +++ ...ursorWebSocketSendLoopPoisonFrameTest.java | 325 ++++++++- ...etSendLoopSchemaPreservationCloseTest.java | 142 ++++ .../qwp/client/sf/cursor/MmapSegmentTest.java | 31 + .../cursor/RejectedMiniSlotArchiveTest.java | 219 ++++++ .../sf/cursor/SchemaRejectionStateTest.java | 174 +++++ .../sf/cursor/SenderErrorDispatcherTest.java | 39 ++ .../test/impl/SchemaRejectionPoolTest.java | 350 ++++++++++ .../client/test/impl/SenderPoolSfTest.java | 9 +- design/schema-mismatch-terminal-resolution.md | 642 ++++++++++++++++++ 35 files changed, 4370 insertions(+), 116 deletions(-) create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java create mode 100644 core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java create mode 100644 design/schema-mismatch-terminal-resolution.md diff --git a/README.md b/README.md index e945c7770..f576bb881 100644 --- a/README.md +++ b/README.md @@ -141,8 +141,52 @@ try (Sender sender = db.borrowSender()) { You can also let the client flush batches for you with the `auto_flush_rows` / `auto_flush_interval` config keys, e.g. `ws::addr=localhost:9000;auto_flush_rows=10000;auto_flush_interval=1000;`. -**Confirm a batch is durably received.** Over QWP each flush returns a frame sequence number (FSN); `awaitAckedFsn` -blocks until the server has acknowledged it. With `sf_dir`, rows in the store-and-forward log replay after reconnect +**Schema errors and preserved copies.** With QuestDB 10.0.0 or later, QWP schema mismatches use +`SenderError.Policy.REJECT_AND_CONTINUE`: the owning borrowed sender fails, but +returning it and borrowing again lets the slot continue. Close and rebuild a +standalone sender after its error. Preservation runs on the I/O thread. Slow +disk I/O can delay close; if its +shutdown budget expires, cleanup retains the slot lock until that thread exits, +so rebuilding immediately may require a retry. +Other error categories keep their existing policies. Select `.schemaMismatchPolicy(SenderError.Policy.TERMINAL)` on either +builder to retain the old preserve-and-halt behavior. + +With `sf_dir`, rejected frames are copied to `//rejected/` before +retirement. A split flush can also retire valid deferred frames preceding the +bad frame; these are included in the copy. After restart, transaction mode is +unknown, so retirement conservatively includes the whole recovered commit group. +The asynchronous `errorHandler` +receives `error.getRejectedPath()` only after the directory is complete. The +producer exception carries the trigger FSN and affected range, but its path may +be null while copying is pending. Notifications are retained in a separate +256-entry queue per slot; a full queue pauses that slot's retirement. Crashes +and shutdown can still lose queued notifications. + +Use `.dlqDirectory(path)` for a different base directory or a memory-only +sender; copies go under `path//rejected/`. Memory-only senders without an +explicit destination retire without a preserved copy. `.dlqEnabled(false)` disables +preservation and accepts permanent loss of retired rows. These are builder +options. A configured destination is checked at build time; later storage +failures pause retirement and retry the copy while keeping the source frames. +A second schema rejection while an earlier range is pending, or an invalid +retirement range/dictionary, logs an error and falls back to `TERMINAL`. +Preserved copies use the +binary store-and-forward format, with rejection metadata; they are not JSON. +Copy an archive to a separate working directory before replaying it, because +normal queue cleanup removes drained data. Replay after fixing the schema can +duplicate rows that the server committed before the error. + +Copies are never automatically deleted and can contain a full symbol dictionary +each. Quarantining a damaged slot also moves its archives; use the `DATA_LOSS` +event's quarantine path to locate copies whose reported paths have moved. +Monitor `getDlqBytesWritten()`, `getDlqFilesWritten()` and free disk space +(the counters are available on `QwpWebSocketSender`). TLS does not encrypt these +files at rest. With preservation disabled, a persistent schema problem can +retire data indefinitely; keep your source data and monitor the error handler. + +**Wait for queue progress.** Over QWP each flush returns a frame sequence number (FSN); `awaitAckedFsn` +blocks until that sequence is resolved. Resolution includes server acknowledgements and locally retired rejected +frames, so observe the error handler as well; a successful wait alone does not prove every row was ingested. With `sf_dir`, rows in the store-and-forward log replay after reconnect or a producer-process restart. For periodic host-power-loss checkpoints, also configure `sf_durability=periodic;sf_sync_interval_millis=5000;`. @@ -152,8 +196,8 @@ try (Sender sender = db.borrowSender()) { sender.table("trades").symbol("symbol", t.symbol).doubleColumn("price", t.price).atNow(); } long fsn = sender.flushAndGetSequence(); // publish the batch, get its sequence number - if (sender.awaitAckedFsn(fsn, 30_000)) { // block up to 30s for the server ack - // batch acknowledged by the server + if (sender.awaitAckedFsn(fsn, 30_000)) { // block up to 30s for resolved progress + // queue resolved through fsn; check rejection notifications for ingestion errors } else { // not yet acked within the timeout; it stays buffered and replays on reconnect } diff --git a/core/src/main/java/io/questdb/client/LineSenderServerException.java b/core/src/main/java/io/questdb/client/LineSenderServerException.java index 2f23d9d0a..a6f3f9ca0 100644 --- a/core/src/main/java/io/questdb/client/LineSenderServerException.java +++ b/core/src/main/java/io/questdb/client/LineSenderServerException.java @@ -65,6 +65,7 @@ private static String buildMessage(SenderError e) { if (status != SenderError.NO_STATUS_BYTE) { sb.append(" (status=0x").append(Integer.toHexString(status & 0xFF)).append(')'); } + sb.append(" rejectedFsn=").append(e.getRejectedFsn()); sb.append(" fsn=[").append(e.getFromFsn()).append(',').append(e.getToFsn()).append(']'); if (e.getTableName() != null) { sb.append(" table=").append(e.getTableName()); diff --git a/core/src/main/java/io/questdb/client/QuestDBBuilder.java b/core/src/main/java/io/questdb/client/QuestDBBuilder.java index e846ad129..ce195817a 100644 --- a/core/src/main/java/io/questdb/client/QuestDBBuilder.java +++ b/core/src/main/java/io/questdb/client/QuestDBBuilder.java @@ -72,6 +72,9 @@ public final class QuestDBBuilder { private SenderConnectionListener connectionListener; private BackgroundDrainerListener drainerListener; private SenderErrorHandler errorHandler; + private SenderError.Policy schemaMismatchPolicy = SenderError.Policy.REJECT_AND_CONTINUE; + private boolean dlqEnabled = true; + private String dlqDir; private long housekeeperIntervalMillis = UNSET; private HttpTokenProvider httpTokenProvider; private String config; @@ -166,6 +169,41 @@ public QuestDBBuilder errorHandler(SenderErrorHandler handler) { return this; } + /** + * Select schema-mismatch handling. REJECT_AND_CONTINUE fails the owning + * handle and retires its rejected prefix; the underlying slot continues. + * TERMINAL retains queued frames and halts the slot. + */ + public QuestDBBuilder schemaMismatchPolicy(SenderError.Policy policy) { + if (policy != SenderError.Policy.TERMINAL && policy != SenderError.Policy.REJECT_AND_CONTINUE) { + throw new IllegalArgumentException("schema mismatch policy must be TERMINAL or REJECT_AND_CONTINUE"); + } + schemaMismatchPolicy = policy; + return this; + } + + /** + * Enable preserved copies before schema retirement (default: enabled for disk queues). + * Disabling preservation accepts permanent loss of retired rows. + */ + public QuestDBBuilder dlqEnabled(boolean enabled) { + dlqEnabled = enabled; + return this; + } + + /** + * Set the raw-copy base directory, including for memory-only queues. + * Copies live under directory/slot/rejected and are never automatically deleted. + * The asynchronous error names the completed directory. + */ + public QuestDBBuilder dlqDirectory(String directory) { + if (directory == null || directory.isEmpty()) { + throw new IllegalArgumentException("DLQ directory must not be empty"); + } + dlqDir = directory; + return this; + } + /** * Builds the {@link QuestDB} handle. Validates both connect strings up * front -- so a malformed config fails here even when both pools have @@ -235,10 +273,10 @@ public QuestDB build() { maxLifetimeMillis, housekeeperIntervalMillis, queryCloseTimeoutMillis, - httpTokenProvider, + null, null, httpTokenProvider, errorHandler, connectionListener, - drainerListener + drainerListener, schemaMismatchPolicy, dlqEnabled, dlqDir ); } diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 645d7b254..113211d2c 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -275,10 +275,16 @@ static Sender fromEnv() { void atNow(); /** - * Block until the server has acknowledged every frame up to {@code targetFsn}, + * Block until every frame up to {@code targetFsn} is resolved, * or until {@code timeoutMillis} elapses. Pair with {@link #flushAndGetSequence()} * to obtain {@code targetFsn} for a specific flush. *
+ * Resolution includes server acknowledgements, schema-rejected ranges retired by + * {@link SenderError.Policy#REJECT_AND_CONTINUE}, and recovered orphan tails. + * A successful wait is progress, not proof that every row was ingested. A pooled + * sender may wait for an earlier borrow's FSN; an error owned by the current + * borrow still throws. Observe the error handler for earlier rejected ranges. + *
* When {@code request_durable_ack=on} (Enterprise primary replication), {@code targetFsn} * advances after durable upload to object storage, not on the ordinary commit ACK. *
@@ -288,7 +294,7 @@ static Sender fromEnv() { * * @param targetFsn FSN to wait for; typically the return value of {@link #flushAndGetSequence()} * @param timeoutMillis upper bound on the wait; {@code <= 0} returns the current state without blocking - * @return {@code true} if the server has acknowledged up to {@code targetFsn} on return, {@code false} on timeout + * @return {@code true} if the queue has resolved up to {@code targetFsn}, {@code false} on timeout * @throws LineSenderException if the transport has latched a terminal error */ default boolean awaitAckedFsn(long targetFsn, long timeoutMillis) { @@ -503,8 +509,8 @@ default Sender decimalColumn(CharSequence name, CharSequence value) { * @param timeoutMillis upper bound on the wait; {@code <= 0} returns the * current state without blocking (the flush still * happens before the check) - * @return {@code true} if the server has acknowledged every published - * frame on return, {@code false} on timeout + * @return {@code true} if every published frame is resolved on return, + * {@code false} on timeout * @throws LineSenderException if the transport has latched a terminal error */ default boolean drain(long timeoutMillis) { @@ -610,14 +616,16 @@ default Sender geoHashColumn(CharSequence name, CharSequence value) { } /** - * Highest frame sequence number (FSN) the server has acknowledged. + * Highest contiguous resolved frame sequence number (FSN). Includes server + * acknowledgements and locally retired schema-rejected ranges or orphan tails; + * this is queue progress, not a count of successfully ingested rows. * Returns {@code -1} when no batch has been published yet, and on transports that * do not track FSNs (HTTP, TCP, UDP). *
* Snapshot accessor: for a bounded blocking wait, use * {@link #awaitAckedFsn(long, long)}. * - * @return highest acknowledged FSN, or {@code -1} if none or unsupported + * @return highest resolved FSN, or {@code -1} if none or unsupported */ default long getAckedFsn() { return -1L; @@ -1082,6 +1090,9 @@ final class LineSenderBuilder { // Optional user-supplied async error handler. When null, the sender // uses DefaultSenderErrorHandler.INSTANCE (loud-not-silent log). private io.questdb.client.SenderErrorHandler errorHandler; + private SenderError.Policy schemaMismatchPolicy = SenderError.Policy.REJECT_AND_CONTINUE; + private boolean dlqEnabled = true; + private String dlqDir; // Bounded inbox capacity for the async error dispatcher. // PARAMETER_NOT_SET_EXPLICITLY → spec default (256). private int errorInboxCapacity = PARAMETER_NOT_SET_EXPLICITLY; @@ -1714,7 +1725,8 @@ public Sender build() { actualConnectionListenerInboxCapacity, actualMaxFrameRejections, actualPoisonMinEscalationWindowMillis, - actualCatchUpCapGapMinEscalationWindowMillis + actualCatchUpCapGapMinEscalationWindowMillis, + schemaMismatchPolicy, dlqEnabled, dlqDir, transactional ); } catch (UnreplayableSlotException e) { // The one failure build() recovers from. The slot's frames reference ids @@ -2129,11 +2141,47 @@ public LineSenderBuilder enableTls() { return this; } + /** + * Select schema-mismatch handling. REJECT_AND_CONTINUE fails the owning + * handle and retires its rejected prefix; the underlying slot continues. + * TERMINAL retains queued frames and halts the slot. + */ + public LineSenderBuilder schemaMismatchPolicy(SenderError.Policy policy) { + if (policy != SenderError.Policy.TERMINAL && policy != SenderError.Policy.REJECT_AND_CONTINUE) { + throw new IllegalArgumentException("schema mismatch policy must be TERMINAL or REJECT_AND_CONTINUE"); + } + schemaMismatchPolicy = policy; + return this; + } + + /** + * Enable preserved copies before schema retirement (default: enabled for disk queues). + * Disabling preservation accepts permanent loss of retired rows. + */ + public LineSenderBuilder dlqEnabled(boolean enabled) { + dlqEnabled = enabled; + return this; + } + + /** + * Set the raw-copy base directory, including for memory-only queues. + * Copies live under directory/slot/rejected and are never automatically deleted. + * The asynchronous error names the completed directory. + */ + public LineSenderBuilder dlqDirectory(String directory) { + if (directory == null || directory.isEmpty()) { + throw new IllegalArgumentException("DLQ directory must not be empty"); + } + dlqDir = directory; + return this; + } + /** * Sets the async error handler invoked for every server-side rejection. * The handler runs on a dedicated daemon dispatcher thread, never on the * I/O thread or producer thread. Slow handlers do not stall publishing; - * if the bounded inbox fills up, surplus notifications are dropped + * schema rejections use a separate 256-entry queue that pauses retirement when full. + * For other categories, if the bounded inbox fills up, surplus notifications are dropped * (visible via {@code QwpWebSocketSender.getDroppedErrorNotifications()}). * *

WebSocket transport only; setting on other transports throws. diff --git a/core/src/main/java/io/questdb/client/SenderError.java b/core/src/main/java/io/questdb/client/SenderError.java index 3d11995e8..9c56cc3ec 100644 --- a/core/src/main/java/io/questdb/client/SenderError.java +++ b/core/src/main/java/io/questdb/client/SenderError.java @@ -36,15 +36,17 @@ *

    *
  • Asynchronously via {@link SenderErrorHandler} registered on the builder.
    • *
  • Synchronously as the payload of a {@link LineSenderServerException} thrown - * from the next producer-thread API call after a {@link Policy#TERMINAL} error has + * from the next producer-thread API call after a {@link Policy#TERMINAL} or + * owned {@link Policy#REJECT_AND_CONTINUE} error has * been latched.
    • *
* *

The {@code [fromFsn, toFsn]} span is the load-bearing correlation key — join it to * whatever the producer thread logged alongside the published-sequence value returned by - * the sender to identify the rejected data. Background orphan-drainer reports use - * {@link #NO_MESSAGE_SEQUENCE} for both bounds because those FSNs belong to another sender - * engine and must not be joined to the live producer's rows. + * the sender to identify the rejected data. Schema archive reports from background orphan drainers retain that orphan's + * local FSN span; use the archive path to identify its queue. Other background + * reports use {@link #NO_MESSAGE_SEQUENCE}. Never join an orphan's FSNs to + * the live producer's rows. * * @see SenderErrorHandler * @see LineSenderServerException @@ -69,6 +71,8 @@ public final class SenderError { private final int serverStatusByte; private final String tableName; private final long toFsn; + private final long rejectedFsn; + private final String rejectedPath; public SenderError( @NotNull Category category, @NotNull Policy appliedPolicy, @@ -96,6 +100,16 @@ private SenderError( long detectedAtNanos, @Nullable String quarantinedPath ) { + this(category, appliedPolicy, serverStatusByte, serverMessage, messageSequence, + fromFsn, toFsn, tableName, detectedAtNanos, quarantinedPath, toFsn, null); + } + + private SenderError(Category category, Policy appliedPolicy, int serverStatusByte, + String serverMessage, long messageSequence, long fromFsn, long toFsn, + String tableName, long detectedAtNanos, String quarantinedPath, + long rejectedFsn, String rejectedPath) { + this.rejectedFsn = rejectedFsn; + this.rejectedPath = rejectedPath; this.category = category; this.appliedPolicy = appliedPolicy; this.serverStatusByte = serverStatusByte; @@ -127,6 +141,37 @@ public static SenderError dataLoss(@NotNull String detail, @NotNull String quara System.nanoTime(), quarantinedPath); } + /** Local FSN named by the NACK, distinct from the full retired span. */ + public long getRejectedFsn() { + return rejectedFsn; + } + + /** Completed preserved-copy directory, or null when no copy is available yet. */ + public @Nullable String getRejectedPath() { + return rejectedPath; + } + + /** Internal copy operation used when a singleton error is resolved to a retirement span. */ + public SenderError withRejectionSpan(long first, long last) { + return new SenderError(category, Policy.REJECT_AND_CONTINUE, serverStatusByte, + serverMessage, messageSequence, first, last, tableName, detectedAtNanos, + quarantinedPath, rejectedFsn, rejectedPath); + } + + /** Returns a new error after the preserved copy has been published. */ + public SenderError withRejectedPath(String path) { + return new SenderError(category, appliedPolicy, serverStatusByte, serverMessage, + messageSequence, fromFsn, toFsn, tableName, detectedAtNanos, + quarantinedPath, rejectedFsn, path); + } + + /** Internal copy operation used when a fail-closed fallback changes policy. */ + public SenderError withAppliedPolicy(Policy policy) { + return new SenderError(category, policy, serverStatusByte, serverMessage, + messageSequence, fromFsn, toFsn, tableName, detectedAtNanos, + quarantinedPath, rejectedFsn, rejectedPath); + } + /** * @return the policy the I/O loop actually applied — RETRIABLE / RETRIABLE_OTHER means * the batch stays in the store-and-forward log and is replayed after a reconnect (no data @@ -153,8 +198,8 @@ public long getDetectedAtNanos() { /** * @return inclusive lower bound of the FSN span for the rejected batch — correlation key for producer-side logs. - * For {@link Category#DATA_LOSS} and background orphan-drainer reports this is - * {@link #NO_MESSAGE_SEQUENCE} — the span is unknown or does not belong to the live sender. + * For {@link Category#DATA_LOSS} and non-schema background reports this is + * {@link #NO_MESSAGE_SEQUENCE}. Schema archive reports retain the orphan queue's local span. */ public long getFromFsn() { return fromFsn; @@ -205,8 +250,8 @@ public int getServerStatusByte() { /** * @return inclusive upper bound of the FSN span for the rejected batch. - * For {@link Category#DATA_LOSS} and background orphan-drainer reports this is - * {@link #NO_MESSAGE_SEQUENCE} — the span is unknown or does not belong to the live sender. + * For {@link Category#DATA_LOSS} and non-schema background reports this is + * {@link #NO_MESSAGE_SEQUENCE}. Schema archive reports retain the orphan queue's local span. */ public long getToFsn() { return toFsn; @@ -302,24 +347,21 @@ public enum Category { } /** - * Policy applied by the client when a category fires. Resolution precedence (highest first): - * builder {@code errorPolicyResolver} → builder per-category {@code errorPolicy} → - * connect-string per-category {@code on_*_error} → connect-string global {@code on_server_error} - * → spec defaults. + * Policy applied by the client. Schema mismatch can be overridden through + * the schemaMismatchPolicy builder setting; other categories use their defaults. + * Reserved on_* connection-string settings do not implement a general resolver. * - *

There is no silent-drop policy by design: the client never discards - * data without telling anyone. A rejected batch is replayed - * ({@link #RETRIABLE} / {@link #RETRIABLE_OTHER}), halts the sender loudly - * with the bytes preserved on disk ({@link #TERMINAL}), or — the one case - * where the bytes can never be sent — is abandoned in place and announced - * as {@link #ABANDONED}, which is precisely what keeps the abandonment - * non-silent. + *

QWP builders default schema mismatches to {@link #REJECT_AND_CONTINUE}: + * retire the affected span after preserving it when configured, notify the + * handler, and fail its owning handle. Other errors replay, halt with bytes + * retained, or report explicit abandonment. Rejection notifications are retained + * while running, but a process crash or shutdown can lose pending callbacks. * *

{@link Category#PROTOCOL_VIOLATION} is forced {@link #TERMINAL}, * {@link Category#UNKNOWN} is forced {@link #RETRIABLE} (fail open: a * status byte from a newer server must degrade to retry, not to a dead * sender), and {@link Category#DATA_LOSS} is forced {@link #ABANDONED}; - * user overrides for these categories are ignored. + * the schema policy override cannot change these categories. */ public enum Policy { /** @@ -358,6 +400,12 @@ public enum Policy { * select it or override it away. It reports a fact about bytes already * abandoned, not a choice about how to react. */ - ABANDONED + ABANDONED, + /** + * Retire the rejected span and continue independent queued work. The owning + * handle fails until returned or rebuilt. Retirement is not server acceptance; + * a preserved copy is available only when export is enabled and completes. + */ + REJECT_AND_CONTINUE } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 41cc0a8c9..eb9e638af 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -29,6 +29,7 @@ import io.questdb.client.SenderConnectionEvent; import io.questdb.client.SenderConnectionListener; import io.questdb.client.SenderError; +import io.questdb.client.LineSenderServerException; import io.questdb.client.SenderErrorHandler; import io.questdb.client.SenderProgressHandler; import io.questdb.client.cairo.TableUtils; @@ -39,10 +40,13 @@ import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerListener; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerPool; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; import io.questdb.client.cutlass.qwp.client.sf.cursor.DefaultSenderConnectionListener; import io.questdb.client.cutlass.qwp.client.sf.cursor.DefaultSenderErrorHandler; @@ -404,6 +408,14 @@ public class QwpWebSocketSender implements Sender { // explicit flush() triggers the server-side commit. Enables accumulating // arbitrarily large datasets that exceed the server's recv buffer. private boolean transactional; + private SenderError.Policy schemaMismatchPolicy = SenderError.Policy.REJECT_AND_CONTINUE; + private boolean dlqEnabled = true; + private String dlqDir; + private SchemaPreserver schemaPreserver; + private final SchemaRejectionState schemaRejectionState = new SchemaRejectionState(); + private long schemaLeaseGeneration; + private boolean schemaLeaseStarted; + private LineSenderServerException observedSchemaFailure; // Server-advertised hard cap on QWP ingest payload bytes, captured from // X-QWP-Max-Batch-Size on each successful FOREGROUND handshake (a // background drainer's endpoint cap is irrelevant to the producer's wire). 0 when the server @@ -890,6 +902,36 @@ public static QwpWebSocketSender connectWithCredentialSupplier( int maxFrameRejections, long poisonMinEscalationWindowMillis, long catchUpCapGapMinEscalationWindowMillis + ) { + return connectWithCredentialSupplier(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, autoFlushIntervalNanos, authorizationHeaderSupplier, requestDurableAck, cursorEngine, closeFlushTimeoutMillis, reconnectMaxDurationMillis, reconnectInitialBackoffMillis, reconnectMaxBackoffMillis, initialConnectMode, errorHandler, errorInboxCapacity, durableAckKeepaliveIntervalMillis, authTimeoutMs, connectTimeoutMs, connectionListener, connectionListenerInboxCapacity, maxFrameRejections, poisonMinEscalationWindowMillis, catchUpCapGapMinEscalationWindowMillis, + SenderError.Policy.REJECT_AND_CONTINUE, true, null, false); + } + + public static QwpWebSocketSender connectWithCredentialSupplier( + List endpoints, + ClientTlsConfiguration tlsConfig, + int autoFlushRows, + int autoFlushBytes, + long autoFlushIntervalNanos, + Supplier authorizationHeaderSupplier, + boolean requestDurableAck, + CursorSendEngine cursorEngine, + long closeFlushTimeoutMillis, + long reconnectMaxDurationMillis, + long reconnectInitialBackoffMillis, + long reconnectMaxBackoffMillis, + Sender.InitialConnectMode initialConnectMode, + SenderErrorHandler errorHandler, + int errorInboxCapacity, + long durableAckKeepaliveIntervalMillis, + long authTimeoutMs, + int connectTimeoutMs, + SenderConnectionListener connectionListener, + int connectionListenerInboxCapacity, + int maxFrameRejections, + long poisonMinEscalationWindowMillis, + long catchUpCapGapMinEscalationWindowMillis, + SenderError.Policy schemaMismatchPolicy, boolean dlqEnabled, String dlqDir, boolean transactional ) { QwpWebSocketSender sender = new QwpWebSocketSender( endpoints, tlsConfig, @@ -922,6 +964,8 @@ public static QwpWebSocketSender connectWithCredentialSupplier( if (cursorEngine != null) { sender.setCursorEngine(cursorEngine, true); } + sender.setTransactional(transactional); + sender.configureSchemaMismatch(schemaMismatchPolicy, dlqEnabled, dlqDir); sender.ensureConnected(); } catch (Throwable t) { // Preserve t's IDENTITY through the rollback. Sender.build() routes on the @@ -1323,7 +1367,8 @@ private void close0(boolean[] restoreInterrupt) { // SenderError HALTs (server-side rejections like MESSAGE_TOO_BIG, // SCHEMA_MISMATCH HALT) from users who only call close() and // never call flush() afterwards. - Throwable terminalError = null; + boolean schemaFailedOnClose = hasOwnedSchemaFailure(); + Throwable terminalError = schemaFailedOnClose ? releaseFailedSchemaLease() : null; // Snapshot the exact terminal error instance that a user-thread // API call ALREADY caught (via flush()/at()) before close() ran. // If flushPendingRows/drainOnClose below also rethrow the same @@ -1343,7 +1388,7 @@ private void close0(boolean[] restoreInterrupt) { // Only drain when both the engine and the I/O loop are wired // up — close() is also called from createForTesting() teardown // and from connect() rollback paths where one or both may be null. - if (connectionError.get() == null && cursorEngine != null && cursorSendLoop != null) { + if (!schemaFailedOnClose && connectionError.get() == null && cursorEngine != null && cursorSendLoop != null) { // 1) Flush user-thread state into the engine (encoded // rows -> mmap'd / malloc'd ring). After this, the // cursor engine's publishedFsn reflects the final @@ -2166,6 +2211,33 @@ public boolean isDeltaDictEnabledForTest() { return deltaDictEnabled; } + /** Frames resolved locally after schema rejection; these were not accepted by the server. */ + public long getSchemaFramesRetired() { + CursorWebSocketSendLoop loop = cursorSendLoop; + return loop == null ? 0 : loop.getSchemaFramesRetired(); + } + + public long getSchemaRejections() { + CursorWebSocketSendLoop loop = cursorSendLoop; + return loop == null ? 0 : loop.getSchemaRejections(); + } + + public long getDlqWriteFailures() { + CursorWebSocketSendLoop loop = cursorSendLoop; + return loop == null ? 0 : loop.getDlqWriteFailures(); + } + + public long getDlqFilesWritten() { + CursorWebSocketSendLoop loop = cursorSendLoop; + return loop == null ? 0 : loop.getDlqFilesWritten(); + } + + /** Cumulative preserved bytes. Monitor together with destination free space. */ + public long getDlqBytesWritten() { + CursorWebSocketSendLoop loop = cursorSendLoop; + return loop == null ? 0 : loop.getDlqBytesWritten(); + } + /** * Total binary frames whose ACKs have been received and applied. */ @@ -2678,6 +2750,11 @@ public void setCursorSendLoopForTesting(CursorWebSocketSendLoop loop) { progressHandler, SenderProgressDispatcher.DEFAULT_CAPACITY); } loop.setConnectionDispatcher(connectionDispatcher); + if (!schemaLeaseStarted) { + beginSchemaLease(0L); + } + loop.setSchemaRejectionState(schemaRejectionState); + loop.setSchemaMismatchPolicy(schemaMismatchPolicy); loop.setErrorDispatcher(errorDispatcher); loop.setProgressDispatcher(progressDispatcher); } @@ -2738,6 +2815,129 @@ public void setErrorInboxCapacity(int capacity) { this.errorInboxCapacity = capacity; } + /** Internal recovery barrier: local retirement counts as progress, never acceptance. */ + public boolean drainResolved(long timeoutMillis) { + if (closed) { + throw new LineSenderException("Sender is closed"); + } + if (cursorEngine == null) { + return true; + } + long target = cursorEngine.publishedFsn(); + long deadline = System.nanoTime() + Math.max(0L, timeoutMillis) * 1_000_000L; + while (cursorEngine.ackedFsn() < target) { + if (closed) { + throw new LineSenderException("Sender is closed"); + } + cursorEngine.checkDurability(); + if (cursorSendLoop != null) { + cursorSendLoop.checkError(); + } + if (timeoutMillis <= 0 || System.nanoTime() >= deadline) { + return false; + } + java.util.concurrent.locks.LockSupport.parkNanos(50_000L); + } + return true; + } + + /** Internal pool lifecycle: end the initial standalone observation before borrowing. */ + public void prepareSchemaPoolSlot() { + if (schemaLeaseStarted) { + schemaRejectionState.endLease(schemaLeaseGeneration, publishedSchemaFsn()); + } + } + + public void beginSchemaLease(long generation) { + schemaLeaseGeneration = generation; + observedSchemaFailure = null; + schemaRejectionState.beginLease(generation, publishedSchemaFsn() + 1, transactional); + schemaLeaseStarted = true; + } + + public LineSenderServerException endSchemaLease() { + if (schemaLeaseStarted) { + LineSenderServerException failure = schemaRejectionState.endLease( + schemaLeaseGeneration, publishedSchemaFsn()); + return failure == observedSchemaFailure ? null : failure; + } + return null; + } + + public boolean hasOwnedSchemaFailure() { + return schemaLeaseStarted && schemaRejectionState.hasOwnedFailure(schemaLeaseGeneration); + } + + /** Discards only this failed producer's local work; the queue remains usable. */ + public LineSenderServerException releaseFailedSchemaLease() { + LineSenderServerException failure = schemaRejectionState.ownedFailure( + schemaLeaseGeneration, publishedSchemaFsn()); + resetTableBuffersAfterFlush(); + if (activeBuffer != null) { + activeBuffer.reset(); + } + hasDeferredMessages = false; + endSchemaLease(); + return failure == observedSchemaFailure ? null : failure; + } + + /** Pool return must not hide a storage or transport failure behind a lease-local rejection. */ + public void checkSchemaSlotHealth() { + LineSenderException failure = connectionError.get(); + if (failure != null) { + throw failure; + } + if (cursorEngine != null) { + cursorEngine.checkDurability(); + } + if (cursorSendLoop != null) { + cursorSendLoop.checkError(); + } + } + + private long publishedSchemaFsn() { + return cursorEngine == null ? -1L : cursorEngine.publishedFsn(); + } + + private void checkSchemaFailure() { + if (hasOwnedSchemaFailure()) { + LineSenderServerException failure = schemaRejectionState.ownedFailure( + schemaLeaseGeneration, publishedSchemaFsn()); + if (failure != null) { + observedSchemaFailure = failure; + throw failure; + } + } + } + + /** Configure before connecting so recovered data uses the selected policy. */ + public void configureSchemaMismatch(SenderError.Policy policy, boolean preserve, String directory) { + if (policy != SenderError.Policy.TERMINAL && policy != SenderError.Policy.REJECT_AND_CONTINUE) { + throw new IllegalArgumentException("schema mismatch policy must be TERMINAL or REJECT_AND_CONTINUE"); + } + this.schemaMismatchPolicy = policy; + this.dlqEnabled = preserve; + this.dlqDir = directory; + if (policy == SenderError.Policy.REJECT_AND_CONTINUE && preserve + && cursorEngine != null && (cursorEngine.sfDir() != null || directory != null)) { + String source = cursorEngine.sfDir(); + String slotId = source == null ? "memory" : java.nio.file.Paths.get(source).getFileName().toString(); + String epoch = source == null ? java.util.UUID.randomUUID().toString() + : SlotEpoch.openOrCreate(io.questdb.client.std.FilesFacade.INSTANCE, source, cursorEngine.freshFsnNamespace()); + String destination = directory == null ? source : java.nio.file.Paths.get(directory, slotId).toString(); + try { + java.nio.file.Files.createDirectories(java.nio.file.Paths.get(destination)); + } catch (java.io.IOException e) { + throw new LineSenderException(e).put("could not create schema preservation destination ").put(destination); + } + SchemaPreserver.probeDestination(io.questdb.client.std.FilesFacade.INSTANCE, destination); + io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive.cleanupTemporaryDirectories( + io.questdb.client.std.FilesFacade.INSTANCE, destination, slotId, epoch); + schemaPreserver = new SchemaPreserver(io.questdb.client.std.FilesFacade.INSTANCE, + destination, slotId, epoch); + } + } + public void setTransactional(boolean transactional) { this.transactional = transactional; } @@ -2881,6 +3081,7 @@ public synchronized void startOrphanDrainers( poisonMinEscalationWindowMillis, catchUpCapGapMinEscalationWindowMillis); ref[0] = drainer; + drainer.configureSchemaMismatch(schemaMismatchPolicy, dlqEnabled, dlqDir, errorHandler); drainerPool.submit(drainer); } } @@ -3647,6 +3848,7 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo } private void checkConnectionError() { + checkSchemaFailure(); LineSenderException error = connectionError.get(); if (error != null) { // Refresh the stack so subsequent public API calls point at the @@ -4078,6 +4280,12 @@ private void ensureConnected() { if (errorDispatcher == null) { errorDispatcher = new SenderErrorDispatcher(errorHandler, errorInboxCapacity); } + if (!schemaLeaseStarted) { + beginSchemaLease(0L); + } + cursorSendLoop.setSchemaRejectionState(schemaRejectionState); + cursorSendLoop.setSchemaMismatchPolicy(schemaMismatchPolicy); + cursorSendLoop.setSchemaPreserver(schemaPreserver); cursorSendLoop.setErrorDispatcher(errorDispatcher); // Symmetric progress dispatcher: lazy-allocated mirror of the // error path. Wired before start() for the same reason -- the diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java index facd872bb..cbc2ba03b 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java @@ -229,6 +229,10 @@ public final class BackgroundDrainer implements Runnable { // LOG -- a NOP for apps without an slf4j binding -- which is exactly the // silence this sink exists to break. private volatile SenderErrorHandler errorSink; + private volatile SenderErrorHandler schemaErrorSink; + private SenderError.Policy schemaMismatchPolicy = SenderError.Policy.TERMINAL; + private boolean schemaPreservationEnabled; + private String schemaPreservationDirectory; private volatile String lastErrorMessage; /** * Optional observer for durable-ack-unavailable transients and the @@ -930,6 +934,8 @@ public void run() { // per wire session. Closed by the finally, after loop.close(), so errors // dispatched during the loop's shutdown still reach the sink. SenderErrorDispatcher loopErrorDispatcher = null; + SchemaPreserver schemaPreserver = null; + SchemaRejectionState schemaRejectionState = null; try { // Scanner results are only snapshots. Serialize adoption against // a producer's close -> quarantine rename -> fresh-slot recreate @@ -1051,37 +1057,43 @@ public void run() { return; } engineForTesting = engine; + if (schemaMismatchPolicy == SenderError.Policy.REJECT_AND_CONTINUE) { + schemaRejectionState = new SchemaRejectionState(); + if (schemaPreservationEnabled) { + String slotId = java.nio.file.Paths.get(slotPath).getFileName().toString(); + String epoch = SlotEpoch.openOrCreate( + io.questdb.client.std.FilesFacade.INSTANCE, + slotPath, + engine.freshFsnNamespace()); + String destination = schemaPreservationDirectory == null + ? slotPath + : java.nio.file.Paths.get(schemaPreservationDirectory, slotId).toString(); + try { + java.nio.file.Files.createDirectories(java.nio.file.Paths.get(destination)); + } catch (java.io.IOException e) { + throw new SfOperationalException( + "could not create schema preservation destination " + destination, e); + } + SchemaPreserver.probeDestination( + io.questdb.client.std.FilesFacade.INSTANCE, destination); + RejectedMiniSlotArchive.cleanupTemporaryDirectories( + io.questdb.client.std.FilesFacade.INSTANCE, destination, slotId, epoch); + schemaPreserver = new SchemaPreserver( + io.questdb.client.std.FilesFacade.INSTANCE, + destination, + slotId, + epoch); + } + } if (logicalSlotLock != null) { logicalSlotLock.close(); logicalSlotLock = null; } - // A recovered deferred-only tail is an aborted transaction and can - // be retired locally once everything below it is already ACKed. - // Do this before opening a socket: auth/upgrade failures must not - // quarantine a slot that has no wire-visible work left. - engine.retireRecoveredOrphanTailIfReady(); - long target = engine.publishedFsn(); - if (engine.ackedFsn() >= target) { - LOG.info("orphan slot already drained: {} (acked={} target={})", - slotPath, engine.ackedFsn(), target); - outcome = DrainOutcome.SUCCESS; - return; - } - // Seed the progress watermark from what a previous run already durably acked, so only acks - // THIS drain earns count as progress. Seeding from the -1 field default would make the first - // poll of a partially-drained slot read as progress and hand back a budget the initial connect - // had legitimately spent. - ackProgressWatermark = engine.ackedFsn(); - client = connectWithDurableAckRetry(); - if (client == null) { - // outcome already set (FAILED or STOPPED); markFailed sentinel - // already dropped on the FAILED path. - return; - } // Read the sink once: like `listener` it is volatile because the pool // applies it at submit time and it is consumed on the drainer thread. SenderErrorHandler sink = errorSink; - if (sink != null) { + SenderErrorHandler schemaSink = schemaErrorSink; + if (sink != null || schemaSink != null) { // The I/O thread must never run the sink inline -- it is caller-supplied // code and may block -- so it reaches the sink through the same bounded, // drop-oldest, off-thread arm the foreground sender uses. @@ -1096,7 +1108,15 @@ public void run() { // (RETRIABLE / RETRIABLE_OTHER) has no such owner and is forwarded verbatim. loopErrorDispatcher = new SenderErrorDispatcher( err -> { - if (err.getAppliedPolicy() != SenderError.Policy.TERMINAL) { + if (err.getAppliedPolicy() == SenderError.Policy.REJECT_AND_CONTINUE) { + if (schemaSink != null) { + // A preserved schema report carries orphan-slot-local FSNs and the + // ready archive path. Keep both intact: this dispatcher is already + // the asynchronous delivery boundary, so another bounded hop could + // drop the report after retirement has made replay impossible. + schemaSink.onError(err); + } + } else if (err.getAppliedPolicy() != SenderError.Policy.TERMINAL && sink != null) { // This sink belongs to the live sender, while err's FSNs belong to the orphan // engine being drained. Strip that foreign correlation span before forwarding; // otherwise an operator can join it to unrelated live rows with the same FSNs. @@ -1115,6 +1135,41 @@ public void run() { SenderErrorDispatcher.DEFAULT_CAPACITY, "qdb-sf-drainer-error-dispatcher"); } + // A recovered deferred-only tail is an aborted transaction and can + // be retired locally once everything below it is already ACKed. + // Do this before opening a socket: auth/upgrade failures must not + // quarantine a slot that has no wire-visible work left. + if (schemaPreserver != null && engine.recoveredOrphanTipFsn() >= 0 + && engine.ackedFsn() >= engine.recoveredCommitBoundaryFsn()) { + SenderError recovered = schemaPreserver.findRecoveredOrphanReport( + engine.recoveredCommitBoundaryFsn() + 1L, engine.recoveredOrphanTipFsn()); + if (recovered != null && (loopErrorDispatcher == null + || !loopErrorDispatcher.tryOfferSchema(recovered))) { + lastErrorMessage = "could not retain recovered schema report before orphan retirement"; + LOG.warn("drainer slot {}: {}", slotPath, lastErrorMessage); + outcome = DrainOutcome.FAILED; + return; + } + } + engine.retireRecoveredOrphanTailIfReady(); + long target = engine.publishedFsn(); + if (engine.ackedFsn() >= target) { + LOG.info("orphan slot already drained: {} (acked={} target={})", + slotPath, engine.ackedFsn(), target); + outcome = DrainOutcome.SUCCESS; + return; + } + // Seed the progress watermark from what a previous run already durably acked, so only acks + // THIS drain earns count as progress. Seeding from the -1 field default would make the first + // poll of a partially-drained slot read as progress and hand back a budget the initial connect + // had legitimately spent. + ackProgressWatermark = engine.ackedFsn(); + client = connectWithDurableAckRetry(); + if (client == null) { + // outcome already set (FAILED or STOPPED); markFailed sentinel + // already dropped on the FAILED path. + return; + } // One iteration per wire session. Re-entered on either of the two // RECOVERABLE mid-drain terminals the recycle branch below tests // for -- a durable-ack CAPABILITY gap, or a 401/403 against a @@ -1152,6 +1207,9 @@ public void run() { // problem once SF fills. Null when no sink is installed, which // setErrorDispatcher accepts and dispatchError treats as before. loop.setErrorDispatcher(loopErrorDispatcher); + loop.setSchemaRejectionState(schemaRejectionState); + loop.setSchemaMismatchPolicy(schemaMismatchPolicy); + loop.setSchemaPreserver(schemaPreserver); loop.start(); while (!stopRequestedOrInterrupted()) { @@ -1278,6 +1336,10 @@ public void run() { lastErrorMessage = t.getMessage(); outcome = DrainOutcome.FAILED; throw t; + } catch (SfOperationalException t) { + lastErrorMessage = t.getMessage(); + LOG.error("drainer storage temporarily unavailable for slot {}: {}", slotPath, lastErrorMessage, t); + outcome = DrainOutcome.FAILED; } catch (Throwable t) { String msg = t.getMessage(); if (slotPath != null) { @@ -1383,14 +1445,13 @@ public void run() { if (engine != null) { // Failed-stop hand-off: delegateEngineClose() makes the I/O // thread run engine.close() strictly after its last engine - // access, releasing the slot lock as soon as the stuck wire - // call resolves — deferred teardown, never abandoned. The + // access, releasing the slot lock once blocked network or + // preservation I/O completes — deferred teardown, never abandoned. The // false return covers the race where the thread exited // between the failed close() and now: then it is safe (and // necessary) to close the engine here. - if (ioThreadStopped || !loop.delegateEngineClose()) { + if (ioThreadStopped || loop == null || !loop.delegateEngineClose()) { try { - // engine.close() releases the slot lock too. engine.close(); } catch (Throwable ignored) { } @@ -1416,6 +1477,26 @@ public void setErrorSink(SenderErrorHandler errorSink) { this.errorSink = errorSink; } + /** Configures schema rejection before this drainer is submitted. */ + public void configureSchemaMismatch( + SenderError.Policy policy, + boolean preserve, + String directory, + SenderErrorHandler effectiveHandler + ) { + if (policy != SenderError.Policy.TERMINAL + && policy != SenderError.Policy.REJECT_AND_CONTINUE) { + throw new IllegalArgumentException( + "schema mismatch policy must be TERMINAL or REJECT_AND_CONTINUE"); + } + this.schemaMismatchPolicy = policy; + this.schemaPreservationEnabled = preserve; + this.schemaPreservationDirectory = directory; + this.schemaErrorSink = effectiveHandler != null + ? effectiveHandler + : DefaultSenderErrorHandler.INSTANCE; + } + /** * Plug an observer for durable-ack-related events. {@code null} clears * any previously installed listener. See {@link BackgroundDrainerListener} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java index 66a0635ea..d128578e9 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java @@ -887,6 +887,30 @@ public boolean acknowledge(long seq) { return ring.acknowledge(seq); } + /** + * Reads the QWP header flags for a currently live frame. Returns {@code -1} + * when the FSN is outside the live ring or the payload is not a valid QWP + * message. The ring monitor protects the mapped bytes from trim/unmap for + * the duration of this bounded header read. + */ + public int liveQwpFrameFlags(long fsn) { + return ring.liveQwpFrameFlags(fsn); + } + + /** Returns the payload length for a currently live frame, or {@code -1}. */ + public int liveFramePayloadLength(long fsn) { + return ring.liveFramePayloadLength(fsn); + } + + /** + * Copies one currently live frame payload into caller-owned native memory. + * The copy runs under the ring monitor, so trim cannot hide or unmap the + * segment midway through it and the I/O cursor's single pin is untouched. + */ + public boolean copyLiveFrame(long fsn, long dstAddr, int dstCapacity) { + return ring.copyLiveFrame(fsn, dstAddr, dstCapacity); + } + /** * I/O thread accessor: the current active mmap'd segment. */ @@ -1278,6 +1302,18 @@ private void finishClose(boolean fullyDrained) { "could not fsync SF slot directory after segment cleanup"); } else { AckWatermark.removeOrphan(filesFacade, sfDir); + // The next engine starts a new FSN namespace. Keep its + // archive identity distinct from this drained one. + String epochPath = sfDir + '/' + SlotEpoch.FILE_NAME; + if (filesFacade.exists(epochPath)) { + if (!filesFacade.remove(epochPath)) { + durabilityFailure = new IllegalStateException( + "could not remove drained SF slot epoch"); + } else if (filesFacade.fsyncDir(sfDir) != 0) { + durabilityFailure = new IllegalStateException( + "could not fsync SF slot directory after epoch cleanup"); + } + } } } else { LOG.warn("close-time segment cleanup incomplete on slot {}; retaining the ack " @@ -1829,6 +1865,11 @@ public boolean wasRecoveredFromDisk() { return wasRecoveredFromDisk; } + /** True when construction created a new FSN namespace rather than recovering one. */ + public boolean freshFsnNamespace() { + return sfDir != null && !wasRecoveredFromDisk; + } + /** * FSN of the last commit-bearing frame in a disk-recovered ring, or * {@code -1} for fresh/memory rings. Frames above it are an orphaned diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 6643bf036..ca8b5d1ac 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -236,6 +236,8 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { * Throttle "reconnect attempt N failed" WARN logs to one per 5 s. */ private static final long RECONNECT_LOG_THROTTLE_NANOS = 5_000_000_000L; + private static final long SCHEMA_PRESERVE_RETRY_INITIAL_NANOS = 100_000_000L; + private static final long SCHEMA_PRESERVE_RETRY_MAX_NANOS = 5_000_000_000L; // Test seam: when true, recovery mirror seeding throws immediately AFTER // ensureSentDictCapacity has grown (and therefore taken ownership of) the mirror, // standing in for the copyRecoveredSymbolSuffix-adjacent failure that leaves a @@ -317,6 +319,18 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // by category. Includes both retriable and terminal outcomes — i.e. every // server-side rejection observed regardless of how the loop reacted. private final AtomicLong totalServerErrors = new AtomicLong(); + private final AtomicLong schemaFramesRetired = new AtomicLong(); + private final AtomicLong schemaRejections = new AtomicLong(); + private final AtomicLong dlqFilesWritten = new AtomicLong(); + private final AtomicLong dlqBytesWritten = new AtomicLong(); + private final AtomicLong dlqWriteFailures = new AtomicLong(); + private volatile SenderError.Policy schemaMismatchPolicy = SenderError.Policy.TERMINAL; + private volatile SchemaRejectionState schemaRejectionState; + private volatile SchemaPreserver schemaPreserver; + private SenderError preservedSchemaNotification; + private long preparedSchemaFirstFsn = -1L; + private long preparedSchemaLastFsn = -1L; + private int schemaPreserveFailures; // Delta symbol dictionary catch-up state (see swapClient). // ALWAYS active -- in memory mode, in disk mode, and (critically) even when the // per-slot persisted dictionary failed to open. sentDictCount is this loop's model @@ -348,6 +362,10 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // freed. private boolean sentDictBytesOwned; private int sentDictCount; + // Cold-path scratch used only when a locally retired range carried symbol + // deltas that later frames still reference. Reused across retirements. + private long skippedFrameScratchAddr; + private int skippedFrameScratchCapacity; // True when replay frames can start above dictionary id zero and therefore // depend on a catch-up on a fresh connection. Delta-enabled live engines // always have this dependency. A recovered delta slot whose dictionary @@ -508,6 +526,8 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // and by the I/O thread afterwards -- never concurrently. private long orphanSkipStartFsn = -1L; private long orphanSkipTipFsn = -1L; + private boolean recoveredOrphanReportLookedUp; + private SenderError recoveredOrphanReport; // Poison-frame detector state (I/O thread only). poisonFsn is the FSN of the // frame implicated by the most recent server-active rejection: the NACK-named // frame, or the OK-level head-of-line frame (highestOkFsn+1) for a @@ -557,6 +577,9 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // advance neither), so replay cannot launder the counter. Pacing only -- // this counter NEVER escalates to a terminal (Invariant B). private int zeroProgressRecycles; + // Schema retirement is local progress, not acceptance. Keep a separate + // reconnect dose until a real server ACK arrives. + private int schemaRecyclesWithoutAck; private long progressAtLastExemptRecycle = Long.MIN_VALUE; // Poison-frame detector threshold for this loop. Constructor-configured // (connect-string key max_frame_rejections); defaults to @@ -1402,6 +1425,7 @@ public synchronized void close() { releaseSentDictBytes(); } if (loopNeverRan) { + releaseSkippedFrameScratch(); freeCatchUpFrameBuffer(); } } @@ -1559,6 +1583,44 @@ public void setErrorDispatcher(SenderErrorDispatcher dispatcher) { this.errorDispatcher = dispatcher; } + public void setSchemaRejectionState(SchemaRejectionState state) { + if (state != null) { + state.setEngine(engine); + } + this.schemaRejectionState = state; + } + + public void setSchemaMismatchPolicy(SenderError.Policy policy) { + if (policy != SenderError.Policy.TERMINAL && policy != SenderError.Policy.REJECT_AND_CONTINUE) { + throw new IllegalArgumentException("schema mismatch policy must be TERMINAL or REJECT_AND_CONTINUE"); + } + this.schemaMismatchPolicy = policy; + } + + public void setSchemaPreserver(SchemaPreserver preserver) { + this.schemaPreserver = preserver; + } + + public long getDlqWriteFailures() { + return dlqWriteFailures.get(); + } + + public long getDlqFilesWritten() { + return dlqFilesWritten.get(); + } + + public long getDlqBytesWritten() { + return dlqBytesWritten.get(); + } + + public long getSchemaFramesRetired() { + return schemaFramesRetired.get(); + } + + public long getSchemaRejections() { + return schemaRejections.get(); + } + /** * Plug an async-delivery sink for ack-watermark advances. Same lifecycle * contract as {@link #setErrorDispatcher} — set once before @@ -2228,14 +2290,27 @@ private void drainPendingDurable() { releasePendingEntry(pendingDurable.pollFirst()); } if (highest != Long.MIN_VALUE) { - long fsn = fsnAtZero + highest; + long fsn = clampAckBeforeSchemaStop(fsnAtZero + highest); if (engine.acknowledge(fsn)) { totalDurableTrimAdvances.incrementAndGet(); dispatchProgress(fsn); + SchemaRejectionState state = schemaRejectionState; + if (state != null) { + state.acknowledgedThrough(fsn); + } } } } + private long clampAckBeforeSchemaStop(long fsn) { + SchemaRejectionState state = schemaRejectionState; + if (state == null) { + return fsn; + } + long stop = state.stopFsn(); + return stop >= 0 && fsn >= stop ? stop - 1L : fsn; + } + /** * Stash a wireSeq + per-table seqTxns from the current OK frame for * later durable-ack confirmation. {@link #response} must hold the OK @@ -2306,6 +2381,18 @@ private void failPaced(Throwable initial) { connectLoop(initial, "reconnect", dose); } + private void failSchemaPaced(Throwable initial) { + int level = schemaRecyclesWithoutAck++; + long dose = reconnectInitialBackoffMillis; + if (dose > 0) { + dose <<= Math.min(level, 6); + if (reconnectMaxBackoffMillis > 0 && dose > reconnectMaxBackoffMillis) { + dose = reconnectMaxBackoffMillis; + } + } + connectLoop(initial, "reconnect", dose); + } + /** * Recycle path for strike-exempt wire events: orderly closes * (NORMAL_CLOSURE / GOING_AWAY), non-orderly closes before any send on @@ -2457,6 +2544,7 @@ private void ioLoop() { if (sentDictBytesAddr != 0) { releaseSentDictBytes(); } + releaseSkippedFrameScratch(); freeCatchUpFrameBuffer(); shutdownLatch.countDown(); Runnable closeCallback = delegatedClose; @@ -2915,6 +3003,69 @@ private void releaseSentDictBytes() { sentDictCount = 0; } + /** + * Extends the reconnect dictionary mirror with deltas carried only by a + * range that is about to be skipped locally. Call before acknowledging the + * range: once trim hides it, successor frames may be impossible to replay. + * This is a rejection cold path and performs no work during normal sends. + */ + void catchUpSkippedRange(long firstFsn, long lastFsn) { + if (firstFsn < 0 || lastFsn < firstFsn) { + throw new IllegalArgumentException("invalid skipped range [first=" + + firstFsn + ", last=" + lastFsn + ']'); + } + for (long fsn = firstFsn; fsn <= lastFsn; fsn++) { + int payloadLen = engine.liveFramePayloadLength(fsn); + if (payloadLen < 0) { + throw new LineSenderException("store-and-forward frame disappeared before retirement [fsn=" + + fsn + ']'); + } + ensureSkippedFrameScratch(payloadLen); + if (!engine.copyLiveFrame(fsn, skippedFrameScratchAddr, skippedFrameScratchCapacity)) { + throw new LineSenderException("store-and-forward frame disappeared before retirement [fsn=" + + fsn + ']'); + } + int deltaStart = frameDeltaStart(skippedFrameScratchAddr, payloadLen); + if (deltaStart > sentDictCount) { + throw new LineSenderException("skipped store-and-forward frame has a symbol dictionary gap [fsn=" + + fsn + ", deltaStart=" + deltaStart + ", dictionarySize=" + sentDictCount + ']'); + } + if (deltaStart >= 0) { + accumulateSentDict(skippedFrameScratchAddr, payloadLen, deltaStart); + } + if (fsn == Long.MAX_VALUE) { + break; + } + } + } + + @TestOnly + public void catchUpSkippedRangeForTest(long firstFsn, long lastFsn) { + catchUpSkippedRange(firstFsn, lastFsn); + } + + private void ensureSkippedFrameScratch(int required) { + if (required <= skippedFrameScratchCapacity) { + return; + } + skippedFrameScratchAddr = skippedFrameScratchAddr == 0 + ? Unsafe.malloc(required, MemoryTag.NATIVE_DEFAULT) + : Unsafe.realloc( + skippedFrameScratchAddr, + skippedFrameScratchCapacity, + required, + MemoryTag.NATIVE_DEFAULT); + skippedFrameScratchCapacity = required; + } + + private void releaseSkippedFrameScratch() { + if (skippedFrameScratchAddr != 0) { + Unsafe.free(skippedFrameScratchAddr, skippedFrameScratchCapacity, MemoryTag.NATIVE_DEFAULT); + } + skippedFrameScratchAddr = 0; + skippedFrameScratchCapacity = 0; + } + /** * Decodes the varint at {@code [p, limit)} and returns {@code (value << 3) | bytes}, * or {@code -1} when it is truncated or runs past a canonical length. @@ -3269,6 +3420,20 @@ public int sentDictCount() { return sentDictCount; } + /** I/O-thread cold-path snapshot used by preserved rejection copies. */ + public byte[] snapshotSentDictionary() { + byte[] snapshot = new byte[sentDictBytesLen]; + if (sentDictBytesLen > 0) { + Unsafe.getUnsafe().copyMemory( + null, sentDictBytesAddr, snapshot, Unsafe.BYTE_OFFSET, sentDictBytesLen); + } + return snapshot; + } + + public int sentDictionaryCount() { + return sentDictCount; + } + @TestOnly public int zeroProgressRecycles() { return zeroProgressRecycles; @@ -3353,6 +3518,11 @@ public boolean trySendOneForTest() { return trySendOne(); } + @TestOnly + public boolean tryRetireSchemaRangeForTest() { + return tryRetireSchemaRange(); + } + private void ensureCatchUpFrameCapacity(int required) { if (catchUpFrameCapacity >= required) { return; @@ -3396,6 +3566,32 @@ private boolean tryReceiveAcks() { * scheduling fairness. */ private boolean trySendOne() { + SchemaRejectionState rejectionState = schemaRejectionState; + if (rejectionState != null) { + long stopFsn = rejectionState.stopFsn(); + if (stopFsn >= 0 && fsnAtZero + nextWireSeq >= stopFsn) { + if (!tryRetireSchemaRange()) { + return false; + } + if (nextWireSeq > 0) { + fail(new LineSenderException( + "recycling connection after retiring schema-rejected range")); + return false; + } + try { + positionCursorForStart(); + } catch (CatchUpSendException e) { + // Match the recovered-orphan re-anchor path below. The + // retired range changed the FSN/wire-sequence mapping, so + // a failed dictionary catch-up must recycle through the + // normal catch-up policy instead of escaping ioLoop as an + // unrelated generic reconnect failure. + fail(isCatchUpCapGap(e) ? e : e.getCause()); + return false; + } + return true; + } + } if (orphanSkipTipFsn >= 0 && fsnAtZero + nextWireSeq >= orphanSkipStartFsn) { // The send cursor reached the orphaned deferred tail. Its frames // belong to an aborted transaction and must never be transmitted @@ -3592,14 +3788,141 @@ private boolean tryRetireOrphanTail() { if (orphanSkipTipFsn < 0) { return true; } + if (engine.ackedFsn() < orphanSkipStartFsn - 1L) { + return false; + } + SchemaPreserver preserver = schemaPreserver; + if (preserver != null) { + if (!recoveredOrphanReportLookedUp) { + recoveredOrphanReport = preserver.findRecoveredOrphanReport( + orphanSkipStartFsn, orphanSkipTipFsn); + recoveredOrphanReportLookedUp = true; + } + if (recoveredOrphanReport != null) { + SenderErrorDispatcher dispatcher = errorDispatcher; + if (dispatcher == null || !dispatcher.tryOfferSchema(recoveredOrphanReport)) { + return false; + } + } + } if (!engine.retireRecoveredOrphanTailIfReady()) { return false; } orphanSkipStartFsn = -1L; orphanSkipTipFsn = -1L; + recoveredOrphanReport = null; return true; } + private boolean tryRetireSchemaRange() { + SchemaRejectionState state = schemaRejectionState; + if (state == null) { + return true; + } + SchemaRejectionState.Range range = state.sealedRange(); + if (range == null || engine.ackedFsn() < range.firstFsn - 1L) { + return false; + } + SenderErrorDispatcher dispatcher = errorDispatcher; + if (dispatcher == null) { + return false; + } + SenderError notification = preservedSchemaNotification != null + ? preservedSchemaNotification + : range.error; + SchemaPreserver preserver = schemaPreserver; + if (preserver != null && preservedSchemaNotification == null) { + try { + prepareSkippedRange(range); + } catch (LineSenderException e) { + LOG.error("could not retain dictionary coverage for schema-rejected range [{}, {}]; " + + "keeping queued bytes and stopping the sender", + range.firstFsn, range.lastFsn, e); + recordFatal(e); + dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); + return false; + } + byte[] dictionary = snapshotSentDictionary(); + final RejectedMiniSlotArchive.Result result; + try { + result = preserver.preserve(engine, range.error, + dictionary.length == 0 ? null : dictionary, sentDictCount); + } catch (LineSenderException | IllegalArgumentException | ArithmeticException e) { + LineSenderException fatal = e instanceof LineSenderException + ? (LineSenderException) e + : new LineSenderException("invalid schema-rejected preservation range", e); + LOG.error("could not preserve schema-rejected store-and-forward range [{}, {}]; " + + "keeping queued bytes and stopping the sender", + range.firstFsn, range.lastFsn, e); + recordFatal(fatal); + dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); + return false; + } catch (RuntimeException e) { + long failures = dlqWriteFailures.incrementAndGet(); + schemaPreserveFailures++; + long delay = SCHEMA_PRESERVE_RETRY_INITIAL_NANOS + << Math.min(schemaPreserveFailures - 1, 6); + LOG.warn("could not preserve schema-rejected store-and-forward range [{}, {}]; " + + "keeping source bytes stopped and retrying (failure {})", + range.firstFsn, range.lastFsn, failures, e); + parkWhileRunning(Math.min(delay, SCHEMA_PRESERVE_RETRY_MAX_NANOS)); + return false; + } + if (!result.reused) { + dlqFilesWritten.incrementAndGet(); + dlqBytesWritten.addAndGet(result.bytesWritten); + } + notification = range.error.withRejectedPath(result.path); + preservedSchemaNotification = notification; + schemaPreserveFailures = 0; + if (!running) { + // close() may have stopped the loop while the synchronous copy + // was blocked in storage. Keep the source range mapped and let + // the existing delegated I/O-thread cleanup release the engine. + return false; + } + } else { + try { + prepareSkippedRange(range); + } catch (LineSenderException e) { + LOG.error("could not retain dictionary coverage for schema-rejected range [{}, {}]; " + + "keeping queued bytes and stopping the sender", + range.firstFsn, range.lastFsn, e); + recordFatal(e); + dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); + return false; + } + } + if (!dispatcher.tryOfferSchema(notification)) { + return false; + } + engine.acknowledge(range.lastFsn); + dispatchProgress(range.lastFsn); + schemaFramesRetired.addAndGet(range.lastFsn - range.firstFsn + 1L); + preservedSchemaNotification = null; + preparedSchemaFirstFsn = -1L; + preparedSchemaLastFsn = -1L; + state.completeRetirement(range.lastFsn); + return true; + } + + private void prepareSkippedRange(SchemaRejectionState.Range range) { + if (preparedSchemaFirstFsn == range.firstFsn && preparedSchemaLastFsn == range.lastFsn) { + return; + } + catchUpSkippedRange(range.firstFsn, range.lastFsn); + preparedSchemaFirstFsn = range.firstFsn; + preparedSchemaLastFsn = range.lastFsn; + } + + private void parkWhileRunning(long nanos) { + long deadline = System.nanoTime() + nanos; + long remaining; + while (running && (remaining = deadline - System.nanoTime()) > 0L) { + LockSupport.parkNanos(remaining); + } + } + /** * Determines whether an oversized symbol-dictionary catch-up entry is always * retriable or may become terminal after the orphan settle budget. @@ -3881,6 +4204,7 @@ public void onBinaryMessage(long payloadPtr, int payloadLen) { wireSeq, highestSent); } totalAcks.incrementAndGet(); + schemaRecyclesWithoutAck = 0; long okFsn = fsnAtZero + capped; if (okFsn > highestOkFsn) { highestOkFsn = okFsn; @@ -3913,8 +4237,13 @@ public void onBinaryMessage(long payloadPtr, int payloadLen) { drainPendingDurable(); return; } - if (engine.acknowledge(fsnAtZero + capped)) { - dispatchProgress(fsnAtZero + capped); + long ackFsn = clampAckBeforeSchemaStop(fsnAtZero + capped); + if (engine.acknowledge(ackFsn)) { + dispatchProgress(ackFsn); + SchemaRejectionState state = schemaRejectionState; + if (state != null) { + state.acknowledgedThrough(ackFsn); + } } return; } @@ -4009,6 +4338,12 @@ private void handlePreSendRejection(long wireSeq, byte status, String tableName = response.getTableEntryCount() == 1 ? response.getTableName(0) : null; + // REJECT_AND_CONTINUE is legal only for an exact data frame sent + // on this connection. A pre-send NACK has no retirement target; + // fail closed while preserving every queued byte. + if (policy == SenderError.Policy.REJECT_AND_CONTINUE) { + policy = SenderError.Policy.TERMINAL; + } SenderError err = new SenderError( category, policy, @@ -4059,7 +4394,9 @@ private void handlePreSendRejection(long wireSeq, byte status, private void handleServerRejection(long wireSeq) { byte status = response.getStatus(); SenderError.Category category = classify(status); - SenderError.Policy policy = defaultPolicyFor(category); + SenderError.Policy policy = category == SenderError.Category.SCHEMA_MISMATCH + ? schemaMismatchPolicy + : defaultPolicyFor(category); // Same sanity clamp as the success branch above: do not trust a // rejection wireSeq beyond what we've actually sent. The clamped // value is only used to attribute an FSN to the error report -- @@ -4135,6 +4472,74 @@ private void handleServerRejection(long wireSeq) { ); totalServerErrors.incrementAndGet(); + if (policy == SenderError.Policy.REJECT_AND_CONTINUE) { + // Retirement requires an exact data sequence sent on this + // connection. Never feed the reporting clamp into data loss. + if (wireSeq < 0 || wireSeq > highestSent || fsn <= engine.ackedFsn()) { + SenderError terminal = new SenderError( + category, SenderError.Policy.TERMINAL, status & 0xff, + response.getErrorMessage(), wireSeq, fsn, fsn, + tableName, System.nanoTime()); + recordFatal(new LineSenderServerException(terminal)); + dispatchError(terminal); + return; + } + SchemaRejectionState state = schemaRejectionState; + if (state == null) { + SenderError terminal = err.withAppliedPolicy(SenderError.Policy.TERMINAL); + recordFatal(new LineSenderServerException(terminal)); + dispatchError(terminal); + return; + } + long floor = engine.ackedFsn() + 1L; + long first = floor; + // Walk forward once so each segment's cold lookup cache can advance + // linearly, even for a rejected prefix containing many small frames. + for (long predecessor = floor; predecessor < fsn; predecessor++) { + int flags = engine.liveQwpFrameFlags(predecessor); + if (flags < 0) { + SenderError terminal = err.withAppliedPolicy(SenderError.Policy.TERMINAL); + recordFatal(new LineSenderServerException(terminal)); + dispatchError(terminal); + return; + } + if ((flags & QwpConstants.FLAG_DEFER_COMMIT) == 0) { + first = predecessor + 1L; + } + } + final boolean installed; + try { + installed = state.reject(fsn, first, err); + } catch (IllegalStateException e) { + LOG.error("could not resolve schema-rejected store-and-forward range at fsn {}; " + + "keeping queued bytes and stopping the sender", fsn, e); + recordFatal(new LineSenderException( + "could not resolve schema-rejected store-and-forward range at fsn " + fsn, e)); + SenderError terminal = new SenderError( + category, SenderError.Policy.TERMINAL, status & 0xff, + response.getErrorMessage(), wireSeq, fsn, fsn, + tableName, System.nanoTime()); + dispatchError(terminal); + return; + } + if (!installed) { + LOG.error("received a schema rejection at fsn {} while another schema-rejected " + + "range is pending retirement; keeping queued bytes and stopping the sender", + fsn); + SenderError terminal = new SenderError( + category, SenderError.Policy.TERMINAL, status & 0xff, + response.getErrorMessage(), wireSeq, fsn, fsn, + tableName, System.nanoTime()); + recordFatal(new LineSenderServerException(terminal)); + dispatchError(terminal); + return; + } + schemaRejections.incrementAndGet(); + failSchemaPaced(new LineSenderException( + "recycling connection after schema rejection at fsn " + fsn)); + return; + } + if (policy == SenderError.Policy.TERMINAL) { // Terminal: stash the typed payload BEFORE dispatching to the // handler. The spec requires signal.terminalError to be latched diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/DefaultSenderErrorHandler.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/DefaultSenderErrorHandler.java index 47d157548..5d6b2f23b 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/DefaultSenderErrorHandler.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/DefaultSenderErrorHandler.java @@ -59,7 +59,7 @@ public void onError(SenderError e) { // Single template; SLF4J fans out the levels so the call site stays // identical and the message format is reviewable in one place. String fmt = "server rejected batch [category={}, policy={}, status=0x{}, " - + "fsn=[{},{}], table={}, seq={}, msg={}]"; + + "fsn=[{},{}], table={}, seq={}, msg={}, preserved={}]"; Object[] args = new Object[]{ e.getCategory(), e.getAppliedPolicy(), @@ -68,10 +68,12 @@ public void onError(SenderError e) { e.getToFsn(), e.getTableName() == null ? "(multi)" : e.getTableName(), e.getMessageSequence(), - e.getServerMessage() + e.getServerMessage(), + e.getRejectedPath() }; if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL - || e.getAppliedPolicy() == SenderError.Policy.ABANDONED) { + || e.getAppliedPolicy() == SenderError.Policy.ABANDONED + || e.getAppliedPolicy() == SenderError.Policy.REJECT_AND_CONTINUE) { LOG.error(fmt, args); } else { LOG.warn(fmt, args); diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java index 779c99000..b7c6dfb62 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java @@ -24,6 +24,7 @@ package io.questdb.client.cutlass.qwp.client.sf.cursor; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.std.Crc32c; import io.questdb.client.std.Files; import io.questdb.client.std.FilesFacade; @@ -106,6 +107,12 @@ public final class MmapSegment implements QuietCloseable { // ring monitor. volatile is the cheapest correct fix. private volatile long frameCount; private long mmapAddress; + // Cold live-frame lookups normally walk forward by FSN (archive/rejection + // scans). Remember one validated frame so each lookup does not rescan the + // immutable published prefix from HEADER_SIZE. These fields are accessed + // under SegmentRing's monitor; the producer never touches them. + private long liveLookupIndex; + private long liveLookupOffset = HEADER_SIZE; // publishedCursor: written by producer, read by consumer (I/O thread). Volatile // because the consumer must see writes in publication order — once the // producer bumps publishedCursor, every byte before it is fully written. @@ -752,6 +759,72 @@ public long frameCount() { return frameCount; } + int liveFramePayloadLength(long fsn) { + long offset = liveFrameOffset(fsn); + return offset < 0 ? -1 : Unsafe.getUnsafe().getInt(mmapAddress + offset + 4); + } + + boolean copyLiveFrame(long fsn, long dstAddr, int dstCapacity) { + long offset = liveFrameOffset(fsn); + if (offset < 0) { + return false; + } + int payloadLen = Unsafe.getUnsafe().getInt(mmapAddress + offset + 4); + if (payloadLen > dstCapacity) { + throw new IllegalArgumentException("destination is too small [required=" + + payloadLen + ", capacity=" + dstCapacity + ']'); + } + if (payloadLen > 0) { + Unsafe.getUnsafe().copyMemory(mmapAddress + offset + FRAME_HEADER_SIZE, dstAddr, payloadLen); + } + return true; + } + + int liveQwpFrameFlags(long fsn) { + long offset = liveFrameOffset(fsn); + if (offset < 0) { + return -1; + } + int payloadLen = Unsafe.getUnsafe().getInt(mmapAddress + offset + 4); + long payload = mmapAddress + offset + FRAME_HEADER_SIZE; + if (payloadLen < QwpConstants.HEADER_SIZE + || Unsafe.getUnsafe().getInt(payload) != QwpConstants.MAGIC_MESSAGE) { + return -1; + } + return Unsafe.getUnsafe().getByte(payload + QwpConstants.HEADER_OFFSET_FLAGS) & 0xff; + } + + private long liveFrameOffset(long fsn) { + long index = fsn - baseSeq; + long frames = frameCount; + if (index < 0 || index >= frames) { + return -1L; + } + long published = publishedCursor; + long i = 0; + long offset = HEADER_SIZE; + if (index >= liveLookupIndex) { + i = liveLookupIndex; + offset = liveLookupOffset; + } + for (; i <= index; i++) { + if (offset + FRAME_HEADER_SIZE > published) { + return -1L; + } + int payloadLen = Unsafe.getUnsafe().getInt(mmapAddress + offset + 4); + if (payloadLen < 0 || payloadLen > published - offset - FRAME_HEADER_SIZE) { + return -1L; + } + if (i == index) { + liveLookupIndex = i; + liveLookupOffset = offset; + return offset; + } + offset += FRAME_HEADER_SIZE + payloadLen; + } + return -1L; + } + /** * Bytes between the last valid frame and the file end that look like an * attempted-but-invalid frame write — set by {@link #openExisting} when diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java index 183876a71..6570c7eab 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java @@ -698,6 +698,59 @@ public long appendedBytes() { return appendOffset; } + /** + * Copies the complete committed dictionary prefix into a fresh dictionary + * file under {@code targetDir} and makes that copy durable. The source + * monitor is held while the prefix boundary and bytes are copied, so a + * concurrent symbol append is wholly before or wholly after the snapshot. + * + * @return number of bytes written, including the dictionary header + */ + public synchronized long snapshotTo(String targetDir) { + if (closed) { + throw new IllegalStateException("symbol dictionary is closed"); + } + String targetPath = targetDir + "/" + FILE_NAME; + int targetFd = ff.openRWExclusive(targetPath); + if (targetFd < 0) { + throw new SfOperationalException("could not create symbol dictionary snapshot " + targetPath); + } + long copyLen = appendOffset; + long scratch = 0L; + boolean success = false; + try { + if (!ff.allocate(targetFd, copyLen)) { + throw new SfOperationalException("could not allocate symbol dictionary snapshot " + targetPath); + } + int scratchSize = (int) Math.min(64 * 1024L, Math.max(copyLen, 1L)); + scratch = Unsafe.malloc(scratchSize, MemoryTag.NATIVE_DEFAULT); + long offset = 0L; + while (offset < copyLen) { + int chunk = (int) Math.min(scratchSize, copyLen - offset); + if (ff.read(fd, scratch, chunk, offset) != chunk) { + throw new SfOperationalException("short read copying symbol dictionary " + filePath); + } + if (ff.write(targetFd, scratch, chunk, offset) != chunk) { + throw new SfOperationalException("short write copying symbol dictionary snapshot " + targetPath); + } + offset += chunk; + } + if (ff.fsync(targetFd) != 0) { + throw new SfOperationalException("could not sync symbol dictionary snapshot " + targetPath); + } + success = true; + return copyLen; + } finally { + if (scratch != 0L) { + Unsafe.free(scratch, (int) Math.min(64 * 1024L, Math.max(copyLen, 1L)), MemoryTag.NATIVE_DEFAULT); + } + ff.close(targetFd); + if (!success) { + ff.remove(targetPath); + } + } + } + /** * Base address of the loaded entry region -- the concatenated * {@code [len][utf8]} bytes of every recovered symbol in id order, exactly as a diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java new file mode 100644 index 000000000..2180ff2c7 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java @@ -0,0 +1,486 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.Crc32c; +import io.questdb.client.std.Files; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; + +import java.nio.charset.StandardCharsets; +import java.util.UUID; + +/** Builds and validates immutable, replay-format copies of rejected SF spans. */ +public final class RejectedMiniSlotArchive { + public static final String METADATA_FILE_NAME = "rejection-meta.bin"; + public static final String SEGMENT_FILE_NAME = "rejected.sfa"; + private static final int METADATA_MAGIC = 0x314a4552; // REJ1 little-endian + private static final int METADATA_VERSION = 1; + private static final int MODE_OWNER_ONLY = 448; // 0700 + + private RejectedMiniSlotArchive() { + } + + public static Result preserve( + FilesFacade ff, + CursorSendEngine engine, + PersistedSymbolDict dictionary, + String slotDir, + String slotId, + String epoch, + SenderError error + ) { + return preserve0(ff, engine, dictionary, null, 0, slotDir, slotId, epoch, error); + } + + public static Result preserveSnapshot( + FilesFacade ff, + CursorSendEngine engine, + byte[] dictionaryEntries, + int dictionaryCount, + String slotDir, + String slotId, + String epoch, + SenderError error + ) { + if ((dictionaryEntries == null) != (dictionaryCount == 0)) { + throw new IllegalArgumentException("dictionary snapshot bytes/count mismatch"); + } + return preserve0(ff, engine, null, dictionaryEntries, dictionaryCount, + slotDir, slotId, epoch, error); + } + + private static Result preserve0( + FilesFacade ff, + CursorSendEngine engine, + PersistedSymbolDict dictionary, + byte[] dictionaryEntries, + int dictionaryCount, + String slotDir, + String slotId, + String epoch, + SenderError error + ) { + long from = error.getFromFsn(); + long to = error.getToFsn(); + if (from < 0 || to < from || error.getRejectedFsn() < from || error.getRejectedFsn() > to) { + throw new IllegalArgumentException("invalid rejection span"); + } + requirePathComponent(slotId, "slot id"); + UUID.fromString(epoch); + String rejectedRoot = slotDir + "/rejected"; + ensureDirectory(ff, rejectedRoot); + String identity = slotId + '-' + epoch + "-fsn-" + from + '-' + to; + String finalDir = rejectedRoot + '/' + identity; + Metadata expected = Metadata.from(slotId, epoch, error, + dictionary != null || dictionaryEntries != null); + if (ff.exists(finalDir)) { + validate(ff, finalDir, expected); + // Completes a previous publication whose rename succeeded but + // whose parent-directory barrier failed transiently. + if (ff.fsyncDir(rejectedRoot) != 0) { + throw new SfOperationalException("could not sync rejected mini-slot parent " + rejectedRoot); + } + return new Result(finalDir, occupiedBytes(ff, finalDir, expected.hasDictionary), true); + } + + String tempDir = rejectedRoot + "/.tmp-" + identity + '-' + UUID.randomUUID(); + ensureDirectory(ff, tempDir); + long totalSize = MmapSegment.HEADER_SIZE; + int maxPayload = 0; + for (long fsn = from; fsn <= to; fsn++) { + int len = engine.liveFramePayloadLength(fsn); + if (len < QwpConstants.HEADER_SIZE) { + throw new SfOperationalException("rejection frame is no longer live [fsn=" + fsn + ']'); + } + totalSize = Math.addExact(totalSize, MmapSegment.FRAME_HEADER_SIZE + (long) len); + maxPayload = Math.max(maxPayload, len); + if (fsn == Long.MAX_VALUE) break; + } + + long scratch = Unsafe.malloc(maxPayload, MemoryTag.NATIVE_DEFAULT); + try (MmapSegment segment = MmapSegment.create( + ff, tempDir + '/' + SEGMENT_FILE_NAME, from, totalSize, true)) { + for (long fsn = from; fsn <= to; fsn++) { + int len = engine.liveFramePayloadLength(fsn); + if (len < 0 || len > maxPayload || !engine.copyLiveFrame(fsn, scratch, maxPayload)) { + throw new SfOperationalException("rejection frame disappeared during copy [fsn=" + fsn + ']'); + } + if (fsn == to) { + long flagsAddr = scratch + QwpConstants.HEADER_OFFSET_FLAGS; + byte flags = Unsafe.getUnsafe().getByte(flagsAddr); + Unsafe.getUnsafe().putByte(flagsAddr, (byte) (flags & ~QwpConstants.FLAG_DEFER_COMMIT)); + } + if (segment.tryAppend(scratch, len) < 0) { + throw new SfOperationalException("rejection segment sizing changed during copy"); + } + if (fsn == Long.MAX_VALUE) break; + } + segment.syncPublished(); + } finally { + Unsafe.free(scratch, maxPayload, MemoryTag.NATIVE_DEFAULT); + } + + try (SfManifest ignored = SfManifest.create(ff, tempDir, from, from)) { + // create() durably writes the sole boundary record. + } + try (AckWatermark watermark = AckWatermark.open(ff, tempDir)) { + if (watermark == null) { + throw new SfOperationalException("could not create rejection ack watermark"); + } + watermark.write(from - 1L); + watermark.sync(); + } + if (dictionary != null) { + dictionary.snapshotTo(tempDir); + } else if (dictionaryEntries != null) { + long entriesAddr = Unsafe.malloc(dictionaryEntries.length, MemoryTag.NATIVE_DEFAULT); + try (PersistedSymbolDict snapshot = PersistedSymbolDict.openClean(ff, tempDir)) { + if (snapshot == null) { + throw new SfOperationalException("could not create rejected mini-slot dictionary"); + } + Unsafe.getUnsafe().copyMemory(dictionaryEntries, Unsafe.BYTE_OFFSET, null, + entriesAddr, dictionaryEntries.length); + snapshot.appendRawEntries(entriesAddr, dictionaryEntries.length, dictionaryCount); + } finally { + Unsafe.free(entriesAddr, dictionaryEntries.length, MemoryTag.NATIVE_DEFAULT); + } + } + writeMetadata(ff, tempDir, expected); + if (ff.fsyncDir(tempDir) != 0 || ff.rename(tempDir, finalDir) != 0 + || ff.fsyncDir(rejectedRoot) != 0) { + throw new SfOperationalException("could not publish rejected mini-slot " + finalDir); + } + validate(ff, finalDir, expected); + return new Result(finalDir, occupiedBytes(ff, finalDir, expected.hasDictionary), false); + } + + /** + * Copies a validated archive into a new working directory. Replay may + * consume that directory; the immutable archive is never adopted or moved. + */ + public static void copyToWorkingDirectory(FilesFacade ff, String archiveDir, String workingDir) { + Metadata metadata = readMetadata(ff, archiveDir); + validate(ff, archiveDir, metadata); + if (ff.exists(workingDir)) { + throw new IllegalArgumentException("working directory already exists: " + workingDir); + } + ensureDirectory(ff, workingDir); + copyFile(ff, archiveDir, workingDir, SEGMENT_FILE_NAME); + copyFile(ff, archiveDir, workingDir, SfManifest.FILE_NAME); + copyFile(ff, archiveDir, workingDir, AckWatermark.FILE_NAME); + copyFile(ff, archiveDir, workingDir, METADATA_FILE_NAME); + if (metadata.hasDictionary) { + copyFile(ff, archiveDir, workingDir, PersistedSymbolDict.FILE_NAME); + } + if (ff.fsyncDir(workingDir) != 0) { + throw new SfOperationalException("could not sync replay working directory " + workingDir); + } + } + + public static Metadata readMetadata(FilesFacade ff, String dir) { + String path = dir + '/' + METADATA_FILE_NAME; + long len = ff.length(path); + if (len < 76 || len > Integer.MAX_VALUE) { + throw new UnreplayableSlotException("invalid rejection metadata size " + path); + } + long mem = Unsafe.malloc(len, MemoryTag.NATIVE_DEFAULT); + int fd = ff.openRW(path); + try { + if (fd < 0 || ff.read(fd, mem, len, 0) != len) { + throw new SfOperationalException("could not read rejection metadata " + path); + } + int storedCrc = Unsafe.getUnsafe().getInt(mem + len - 4); + if (storedCrc != Crc32c.update(Crc32c.INIT, mem, len - 4)) { + throw new UnreplayableSlotException("rejection metadata CRC mismatch " + path); + } + long p = mem; + if (Unsafe.getUnsafe().getInt(p) != METADATA_MAGIC + || Unsafe.getUnsafe().getInt(p + 4) != METADATA_VERSION) { + throw new UnreplayableSlotException("unsupported rejection metadata " + path); + } + p += 8; + long rejected = Unsafe.getUnsafe().getLong(p); p += 8; + long from = Unsafe.getUnsafe().getLong(p); p += 8; + long to = Unsafe.getUnsafe().getLong(p); p += 8; + long detected = Unsafe.getUnsafe().getLong(p); p += 8; + int status = Unsafe.getUnsafe().getInt(p); p += 4; + boolean hasDictionary = Unsafe.getUnsafe().getInt(p) != 0; p += 4; + String slotId = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + String epoch = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + String category = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + String policy = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + String table = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + String message = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p); + if (p != mem + len - 4) { + throw new UnreplayableSlotException("trailing rejection metadata bytes " + path); + } + return new Metadata(slotId, epoch, rejected, from, to, detected, status, + category, policy, table, message, hasDictionary); + } finally { + if (fd >= 0) ff.close(fd); + Unsafe.free(mem, len, MemoryTag.NATIVE_DEFAULT); + } + } + + /** + * Finds the preserved report for a recovered orphan tail. Only completed + * directories belonging to this slot epoch are considered; temporary + * directories are never evidence. + */ + public static SenderError findOverlapping( + FilesFacade ff, String slotDir, String slotId, String epoch, long fromFsn, long toFsn + ) { + requirePathComponent(slotId, "slot id"); + UUID.fromString(epoch); + String rejectedRoot = slotDir + "/rejected"; + long find = ff.findFirst(rejectedRoot); + if (find <= 0) { + if (find > 0) ff.findClose(find); + return null; + } + String prefix = slotId + '-' + epoch + "-fsn-"; + try { + int rc = 1; + while (rc > 0) { + String name = Files.utf8ToString(ff.findName(find)); + int type = ff.findType(find); + rc = ff.findNext(find); + if (type != Files.DT_DIR || name == null || !name.startsWith(prefix)) continue; + String path = rejectedRoot + '/' + name; + Metadata metadata = readMetadata(ff, path); + if (!slotId.equals(metadata.slotId) || !epoch.equals(metadata.epoch) + || metadata.toFsn < fromFsn || metadata.fromFsn > toFsn) { + continue; + } + validate(ff, path, metadata); + SenderError error = new SenderError( + SenderError.Category.valueOf(metadata.category), + SenderError.Policy.valueOf(metadata.policy), metadata.status, + metadata.message, metadata.rejectedFsn, metadata.rejectedFsn, + metadata.rejectedFsn, metadata.table.isEmpty() ? null : metadata.table, + metadata.detectedAtNanos); + return error.withRejectionSpan(metadata.fromFsn, metadata.toFsn) + .withRejectedPath(path); + } + } finally { + ff.findClose(find); + } + return null; + } + + /** + * Removes incomplete publications for exactly one slot epoch. The caller + * must hold that queue's exclusive lifecycle lock; the identity prefix is + * what keeps shared memory-sender destinations from touching one another. + */ + public static void cleanupTemporaryDirectories( + FilesFacade ff, String slotDir, String slotId, String epoch + ) { + requirePathComponent(slotId, "slot id"); + UUID.fromString(epoch); + String rejectedRoot = slotDir + "/rejected"; + long find = ff.findFirst(rejectedRoot); + if (find <= 0) return; + String prefix = ".tmp-" + slotId + '-' + epoch + "-fsn-"; + java.util.ArrayList candidates = new java.util.ArrayList<>(); + try { + int rc = 1; + while (rc > 0) { + String name = Files.utf8ToString(ff.findName(find)); + int type = ff.findType(find); + rc = ff.findNext(find); + if (type == Files.DT_DIR && name != null && name.startsWith(prefix)) { + candidates.add(rejectedRoot + '/' + name); + } + } + } finally { + ff.findClose(find); + } + for (String candidate : candidates) { + removeKnownTemporaryContents(ff, candidate); + } + if (!candidates.isEmpty() && ff.fsyncDir(rejectedRoot) != 0) { + throw new SfOperationalException("could not sync rejected temporary cleanup " + rejectedRoot); + } + } + + private static void removeKnownTemporaryContents(FilesFacade ff, String dir) { + String[] names = {SEGMENT_FILE_NAME, SfManifest.FILE_NAME, AckWatermark.FILE_NAME, + PersistedSymbolDict.FILE_NAME, METADATA_FILE_NAME}; + for (String name : names) ff.remove(dir + '/' + name); + // remove() maps to unlink/rmdir. It deliberately fails if an unknown + // file appeared, preserving rather than broadening deletion scope. + ff.remove(dir); + } + + private static void validate(FilesFacade ff, String dir, Metadata expected) { + Metadata actual = readMetadata(ff, dir); + if (!expected.sameIdentity(actual)) { + throw new UnreplayableSlotException("rejected mini-slot identity mismatch " + dir); + } + try (MmapSegment segment = MmapSegment.openExisting(ff, dir + '/' + SEGMENT_FILE_NAME); + SfManifest manifest = SfManifest.open(ff, dir); + AckWatermark watermark = AckWatermark.open(ff, dir)) { + if (segment.baseSeq() != actual.fromFsn + || segment.frameCount() != actual.toFsn - actual.fromFsn + 1 + || manifest == null || manifest.headBase() != actual.fromFsn + || manifest.activeBase() != actual.fromFsn + || watermark == null || watermark.read() != actual.fromFsn - 1L) { + throw new UnreplayableSlotException("invalid rejected mini-slot boundaries " + dir); + } + } + if (actual.hasDictionary) { + try (PersistedSymbolDict ignored = PersistedSymbolDict.open(ff, dir)) { + if (ignored == null) { + throw new UnreplayableSlotException("missing rejected mini-slot dictionary " + dir); + } + } + } + } + + private static void writeMetadata(FilesFacade ff, String dir, Metadata metadata) { + byte[] slot = metadata.slotId.getBytes(StandardCharsets.UTF_8); + byte[] epoch = metadata.epoch.getBytes(StandardCharsets.UTF_8); + byte[] category = metadata.category.getBytes(StandardCharsets.UTF_8); + byte[] policy = metadata.policy.getBytes(StandardCharsets.UTF_8); + byte[] table = bytes(metadata.table); + byte[] message = bytes(metadata.message); + int len = 48 + 4 + slot.length + 4 + epoch.length + 4 + category.length + + 4 + policy.length + 4 + table.length + 4 + message.length + 4; + long mem = Unsafe.malloc(len, MemoryTag.NATIVE_DEFAULT); + int fd = -1; + String path = dir + '/' + METADATA_FILE_NAME; + try { + Unsafe.getUnsafe().setMemory(mem, len, (byte) 0); + long p = mem; + Unsafe.getUnsafe().putInt(p, METADATA_MAGIC); Unsafe.getUnsafe().putInt(p + 4, METADATA_VERSION); p += 8; + Unsafe.getUnsafe().putLong(p, metadata.rejectedFsn); p += 8; + Unsafe.getUnsafe().putLong(p, metadata.fromFsn); p += 8; + Unsafe.getUnsafe().putLong(p, metadata.toFsn); p += 8; + Unsafe.getUnsafe().putLong(p, metadata.detectedAtNanos); p += 8; + Unsafe.getUnsafe().putInt(p, metadata.status); p += 4; + Unsafe.getUnsafe().putInt(p, metadata.hasDictionary ? 1 : 0); p += 4; + p = writeString(p, slot); p = writeString(p, epoch); p = writeString(p, category); + p = writeString(p, policy); p = writeString(p, table); p = writeString(p, message); + Unsafe.getUnsafe().putInt(p, Crc32c.update(Crc32c.INIT, mem, len - 4)); + fd = ff.openRWExclusive(path); + if (fd < 0 || !ff.allocate(fd, len) || ff.write(fd, mem, len, 0) != len || ff.fsync(fd) != 0) { + throw new SfOperationalException("could not write rejection metadata " + path); + } + } finally { + if (fd >= 0) ff.close(fd); + Unsafe.free(mem, len, MemoryTag.NATIVE_DEFAULT); + } + } + + private static void copyFile(FilesFacade ff, String fromDir, String toDir, String name) { + String source = fromDir + '/' + name; + String target = toDir + '/' + name; + long len = ff.length(source); + if (len < 0) throw new SfOperationalException("missing archive file " + source); + int in = ff.openRW(source); + int out = ff.openRWExclusive(target); + long mem = Unsafe.malloc(Math.min(Math.max(len, 1), 64 * 1024), MemoryTag.NATIVE_DEFAULT); + try { + if (in < 0 || out < 0 || !ff.allocate(out, len)) throw new SfOperationalException("could not copy " + source); + for (long off = 0; off < len; ) { + long chunk = Math.min(64 * 1024L, len - off); + if (ff.read(in, mem, chunk, off) != chunk || ff.write(out, mem, chunk, off) != chunk) { + throw new SfOperationalException("short copy of " + source); + } + off += chunk; + } + if (ff.fsync(out) != 0) throw new SfOperationalException("could not sync " + target); + } finally { + if (in >= 0) ff.close(in); + if (out >= 0) ff.close(out); + Unsafe.free(mem, Math.min(Math.max(len, 1), 64 * 1024), MemoryTag.NATIVE_DEFAULT); + } + } + + private static long occupiedBytes(FilesFacade ff, String dir, boolean dict) { + long n = ff.length(dir + '/' + SEGMENT_FILE_NAME) + ff.length(dir + '/' + SfManifest.FILE_NAME) + + ff.length(dir + '/' + AckWatermark.FILE_NAME) + ff.length(dir + '/' + METADATA_FILE_NAME); + return dict ? n + ff.length(dir + '/' + PersistedSymbolDict.FILE_NAME) : n; + } + + private static void ensureDirectory(FilesFacade ff, String dir) { + if (!ff.exists(dir) && ff.mkdir(dir, MODE_OWNER_ONLY) != 0) { + throw new SfOperationalException("could not create directory " + dir); + } + } + + private static void requirePathComponent(String value, String label) { + if (value == null || value.isEmpty() || value.indexOf('/') >= 0 || value.indexOf('\\') >= 0 || value.equals(".") || value.equals("..")) { + throw new IllegalArgumentException("invalid " + label); + } + } + + private static byte[] bytes(String value) { return value == null ? new byte[0] : value.getBytes(StandardCharsets.UTF_8); } + private static long writeString(long p, byte[] value) { + Unsafe.getUnsafe().putInt(p, value.length); + if (value.length > 0) Unsafe.getUnsafe().copyMemory(value, Unsafe.BYTE_OFFSET, null, p + 4, value.length); + return p + 4 + value.length; + } + private static int utf8LengthAt(long base, long p) { return Unsafe.getUnsafe().getInt(p); } + private static String readString(long base, long limitOffset, long p) { + int len = Unsafe.getUnsafe().getInt(p); + if (len < 0 || p + 4L + len > base + limitOffset) throw new UnreplayableSlotException("invalid rejection metadata string"); + byte[] bytes = new byte[len]; + if (len > 0) Unsafe.getUnsafe().copyMemory(null, p + 4, bytes, Unsafe.BYTE_OFFSET, len); + return new String(bytes, StandardCharsets.UTF_8); + } + + public static final class Result { + public final long bytesWritten; + public final String path; + public final boolean reused; + Result(String path, long bytesWritten, boolean reused) { this.path = path; this.bytesWritten = bytesWritten; this.reused = reused; } + } + + /** Stable fields used by startup scanning to reconstruct a notification. */ + public static final class Metadata { + public final int status; + public final long detectedAtNanos, fromFsn, rejectedFsn, toFsn; + public final String category, epoch, message, policy, slotId, table; + public final boolean hasDictionary; + Metadata(String slotId, String epoch, long rejectedFsn, long fromFsn, long toFsn, + long detectedAtNanos, int status, String category, String policy, + String table, String message, boolean hasDictionary) { + this.slotId = slotId; this.epoch = epoch; this.rejectedFsn = rejectedFsn; this.fromFsn = fromFsn; + this.toFsn = toFsn; this.detectedAtNanos = detectedAtNanos; this.status = status; + this.category = category; this.policy = policy; this.table = table; + this.message = message; this.hasDictionary = hasDictionary; + } + static Metadata from(String slotId, String epoch, SenderError e, boolean hasDictionary) { + return new Metadata(slotId, epoch, e.getRejectedFsn(), e.getFromFsn(), e.getToFsn(), + e.getDetectedAtNanos(), e.getServerStatusByte(), e.getCategory().name(), + e.getAppliedPolicy().name(), emptyIfNull(e.getTableName()), + emptyIfNull(e.getServerMessage()), hasDictionary); + } + @Override public boolean equals(Object o) { + if (!(o instanceof Metadata)) return false; + Metadata m = (Metadata) o; + return rejectedFsn == m.rejectedFsn && fromFsn == m.fromFsn && toFsn == m.toFsn + && detectedAtNanos == m.detectedAtNanos && status == m.status + && hasDictionary == m.hasDictionary && eq(slotId, m.slotId) && eq(epoch, m.epoch) + && eq(category, m.category) && eq(policy, m.policy) + && eq(table, m.table) && eq(message, m.message); + } + @Override public int hashCode() { return slotId.hashCode(); } + private boolean sameIdentity(Metadata m) { + return rejectedFsn == m.rejectedFsn && fromFsn == m.fromFsn && toFsn == m.toFsn + && category.equals(m.category) && slotId.equals(m.slotId) && epoch.equals(m.epoch) + && hasDictionary == m.hasDictionary; + } + private static boolean eq(Object a, Object b) { return a == null ? b == null : a.equals(b); } + private static String emptyIfNull(String value) { return value == null ? "" : value; } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java new file mode 100644 index 000000000..c2cc4bb86 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java @@ -0,0 +1,73 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.SenderError; +import io.questdb.client.std.FilesFacade; + +import java.util.UUID; + +/** Immutable configuration for synchronous schema-rejection preservation. */ +public final class SchemaPreserver { + private final String epoch; + private final FilesFacade ff; + private final String slotDir; + private final String slotId; + + public SchemaPreserver(FilesFacade ff, String slotDir, String slotId, String epoch) { + this.ff = ff; + this.slotDir = slotDir; + this.slotId = slotId; + this.epoch = epoch; + } + + public SenderError findRecoveredOrphanReport(long fromFsn, long toFsn) { + return RejectedMiniSlotArchive.findOverlapping( + ff, slotDir, slotId, epoch, fromFsn, toFsn); + } + + /** + * Preserves the sealed range on the calling thread. A preceding failed + * publication may have left its uniquely named temporary tree behind, so + * clean only this slot and epoch's known temporary scope before retrying. + */ + public RejectedMiniSlotArchive.Result preserve( + CursorSendEngine engine, + SenderError error, + byte[] dictionaryEntries, + int dictionaryCount + ) { + RejectedMiniSlotArchive.cleanupTemporaryDirectories(ff, slotDir, slotId, epoch); + return RejectedMiniSlotArchive.preserveSnapshot( + ff, engine, dictionaryEntries, dictionaryCount, + slotDir, slotId, epoch, error); + } + + /** Build-time writability and directory-durability probe. */ + public static void probeDestination(FilesFacade ff, String slotDir) { + probeDirectory(ff, slotDir + "/rejected"); + } + + /** Probes an explicitly configured destination without assuming a slot layout. */ + public static void probeDirectory(FilesFacade ff, String directory) { + if (!ff.exists(directory) && ff.mkdir(directory, 448) != 0) { + throw new SfOperationalException("could not create schema preservation destination " + directory); + } + String probe = directory + "/.probe-" + UUID.randomUUID(); + int fd = ff.openRWExclusive(probe); + try { + if (fd < 0 || ff.fsync(fd) != 0) { + throw new SfOperationalException("schema preservation destination is not writable " + directory); + } + } finally { + if (fd >= 0) ff.close(fd); + ff.remove(probe); + } + if (ff.fsyncDir(directory) != 0) { + throw new SfOperationalException("could not sync schema preservation destination " + directory); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java new file mode 100644 index 000000000..b5be736e5 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java @@ -0,0 +1,254 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.LineSenderServerException; +import io.questdb.client.SenderError; + +import java.util.ArrayDeque; + +/** Process-local lease ownership and one pending schema-retirement range. */ +public final class SchemaRejectionState { + private final ArrayDeque leases = new ArrayDeque<>(); + private Pending pending; + private CursorSendEngine engine; + private volatile long acknowledgedFsn = -1L; + private volatile long failedGeneration = -1L; + private volatile long stopFsn = -1L; + + public void setEngine(CursorSendEngine engine) { + this.engine = engine; + } + + public synchronized void beginLease(long generation, long firstFsn, boolean transactional) { + prune(acknowledgedFsn); + Lease tail = leases.peekLast(); + if (tail != null && tail.active) { + throw new IllegalStateException("previous lease is still active"); + } + leases.addLast(new Lease(generation, firstFsn, transactional)); + } + + /** + * Ends a lease and seals an open transactional rejection. The caller is + * the sole producer and must exclude further publication before taking + * {@code publishedFsn}. + */ + public synchronized LineSenderServerException endLease(long generation, long publishedFsn) { + Lease lease = findGeneration(generation); + if (lease == null || !lease.active) { + return null; + } + lease.endFsn = publishedFsn; + lease.active = false; + if (publishedFsn < lease.firstFsn && lease.rawError == null) { + // Empty borrows carry no attribution history, even behind an unacked lease. + leases.removeLast(); + return null; + } + sealIfNeeded(lease, publishedFsn); + if (failedGeneration == generation) { + failedGeneration = -1L; + } + return lease.failure; + } + + /** + * Returns this generation's immutable owned failure. For an open + * transaction, the first producer observation seals its end at the supplied + * publication snapshot. Calls on one sender are single-producer by contract. + */ + public synchronized LineSenderServerException ownedFailure(long generation, long publishedFsn) { + Lease lease = findGeneration(generation); + if (lease == null || lease.rawError == null) { + return null; + } + sealIfNeeded(lease, publishedFsn); + return lease.failure; + } + + public boolean hasOwnedFailure(long generation) { + return failedGeneration == generation; + } + + /** I/O-thread install. Returns false while an earlier retirement is pending. */ + public synchronized boolean reject(long rejectedFsn, long spanStart, SenderError rawError) { + prune(acknowledgedFsn); + if (pending != null) { + return false; + } + Lease owner = findOwner(rejectedFsn); + if (owner == null) { + // Transaction mode is not persisted. A recovered deferred group must + // therefore be treated conservatively as transactional, regardless of + // the new producer's settings. Bound it by the recovered namespace so + // a new producer's closer cannot become part of the old transaction. + long recoveredTip = engine != null && engine.wasRecoveredFromDisk() + ? Math.max(engine.recoveredCommitBoundaryFsn(), engine.recoveredOrphanTipFsn()) + : -1L; + boolean recovered = rejectedFsn <= recoveredTip; + owner = new Lease(-1L, spanStart, recovered); + owner.active = false; + owner.endFsn = recovered ? recoveredTip : rejectedFsn; + } else if (owner.rawError == null) { + owner.rawError = rawError; + failedGeneration = owner.generation; + } + long end = rejectedFsn; + if (owner.transactional) { + long tip = owner.active && engine != null ? engine.publishedFsn() : owner.endFsn; + long closer = firstCommitFsn(rejectedFsn, tip); + end = closer >= 0 ? closer : owner.active ? -1L : tip; + } + pending = new Pending(spanStart, end, owner, rawError); + stopFsn = spanStart; + if (end >= spanStart) { + finishFailure(end); + } + return true; + } + + public long stopFsn() { + return stopFsn; + } + + public synchronized Range sealedRange() { + if (pending == null || pending.error == null) { + return null; + } + return new Range(pending.firstFsn, pending.lastFsn, pending.error); + } + + public synchronized void completeRetirement(long lastFsn) { + if (pending == null || pending.lastFsn != lastFsn) { + throw new IllegalStateException("retirement range changed"); + } + acknowledgedThrough(lastFsn); + pending.owner.retired = true; + pending = null; + stopFsn = -1L; + prune(lastFsn); + } + + public void acknowledgedThrough(long fsn) { + if (fsn > acknowledgedFsn) { + acknowledgedFsn = fsn; + } + } + + private void sealIfNeeded(Lease lease, long publishedFsn) { + if (pending == null || pending.owner != lease || pending.error != null) { + return; + } + long end = pending.lastFsn; + if (lease.transactional) { + long closer = firstCommitFsn(pending.rawError.getRejectedFsn(), publishedFsn); + end = closer >= 0 ? closer : publishedFsn; + } + finishFailure(end); + } + + private long firstCommitFsn(long first, long last) { + if (engine == null) { + return -1L; + } + for (long fsn = first; fsn <= last; fsn++) { + int flags = engine.liveQwpFrameFlags(fsn); + if (flags < 0) { + throw new IllegalStateException("missing frame while resolving transaction at FSN " + fsn); + } + if ((flags & io.questdb.client.cutlass.qwp.protocol.QwpConstants.FLAG_DEFER_COMMIT) == 0) { + return fsn; + } + } + return -1L; + } + + private void finishFailure(long lastFsn) { + pending.lastFsn = lastFsn; + pending.error = pending.rawError.withRejectionSpan(pending.firstFsn, lastFsn); + Lease lease = pending.owner; + if (lease.failure == null && lease.generation >= 0) { + lease.failure = new LineSenderServerException(pending.error); + } + } + + private Lease findGeneration(long generation) { + Lease tail = leases.peekLast(); + if (tail != null && tail.generation == generation) { + return tail; + } + for (Lease lease : leases) { + if (lease.generation == generation) { + return lease; + } + } + return null; + } + + private Lease findOwner(long fsn) { + for (Lease lease : leases) { + if (fsn >= lease.firstFsn && (lease.active || fsn <= lease.endFsn)) { + return lease; + } + } + return null; + } + + private void prune(long fsn) { + while (true) { + Lease head = leases.peekFirst(); + if (head == null || head.active || (head.rawError != null && !head.retired) || head.endFsn > fsn) { + return; + } + leases.removeFirst(); + } + } + + public static final class Range { + public final SenderError error; + public final long firstFsn; + public final long lastFsn; + + private Range(long firstFsn, long lastFsn, SenderError error) { + this.firstFsn = firstFsn; + this.lastFsn = lastFsn; + this.error = error; + } + } + + private static final class Pending { + private final long firstFsn; + private long lastFsn; + private final Lease owner; + private final SenderError rawError; + private SenderError error; + + private Pending(long firstFsn, long lastFsn, Lease owner, SenderError rawError) { + this.firstFsn = firstFsn; + this.lastFsn = lastFsn; + this.owner = owner; + this.rawError = rawError; + } + } + + private static final class Lease { + private final long firstFsn; + private final long generation; + private final boolean transactional; + private boolean active = true; + private long endFsn = -1L; + private LineSenderServerException failure; + private SenderError rawError; + private boolean retired; + + private Lease(long generation, long firstFsn, boolean transactional) { + this.generation = generation; + this.firstFsn = firstFsn; + this.transactional = transactional; + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentRing.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentRing.java index 6fd7f5aaf..9bc9429eb 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentRing.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SegmentRing.java @@ -1299,6 +1299,21 @@ synchronized MmapSegment pinSegmentContaining(long fsn) { return segment; } + synchronized int liveFramePayloadLength(long fsn) { + MmapSegment segment = findSegmentContaining0(fsn); + return segment == null ? -1 : segment.liveFramePayloadLength(fsn); + } + + synchronized boolean copyLiveFrame(long fsn, long dstAddr, int dstCapacity) { + MmapSegment segment = findSegmentContaining0(fsn); + return segment != null && segment.copyLiveFrame(fsn, dstAddr, dstCapacity); + } + + synchronized int liveQwpFrameFlags(long fsn) { + MmapSegment segment = findSegmentContaining0(fsn); + return segment == null ? -1 : segment.liveQwpFrameFlags(fsn); + } + /** * Oldest sealed segment, or {@code null} if the sealed list is empty. * Used by the I/O loop's "current was trimmed out from under us" diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SenderErrorDispatcher.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SenderErrorDispatcher.java index fb796712a..976acf44d 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SenderErrorDispatcher.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SenderErrorDispatcher.java @@ -32,6 +32,8 @@ import org.slf4j.LoggerFactory; import java.util.concurrent.LinkedBlockingDeque; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; @@ -104,6 +106,9 @@ public final class SenderErrorDispatcher implements QuietCloseable { // the sole producer, the dispatcher is the sole consumer; close() also // enqueues POISON, but only once and under `lock`. private final LinkedBlockingDeque inbox; + private final ArrayBlockingQueue schemaInbox = new ArrayBlockingQueue<>(DEFAULT_CAPACITY); + // Includes the callback currently executing; ordinary deque overflow cannot evict these. + private final AtomicInteger schemaPending = new AtomicInteger(); // Threads are started lazily under this monitor; takes the same role as // SegmentManager.start() — first offer() that observes a null thread // wins the race to spawn it. @@ -162,7 +167,7 @@ public void close() { //noinspection ResultOfMethodCallIgnored inbox.offer(POISON); Thread t = dispatcherThread; - if (t != null) { + if (t != null && t != Thread.currentThread()) { long deadline = System.nanoTime() + DRAIN_DEADLINE_NANOS; long remainingMillis; while ((remainingMillis = (deadline - System.nanoTime()) / 1_000_000L) > 0) { @@ -306,11 +311,38 @@ public boolean offer(SenderError error) { return true; } + /** Retains a schema notification without dropping; false leaves retirement pending. */ + public boolean tryOfferSchema(SenderError error) { + if (closed || error == null) { + return false; + } + int count; + do { + count = schemaPending.get(); + if (count >= DEFAULT_CAPACITY) { + return false; + } + } while (!schemaPending.compareAndSet(count, count + 1)); + if (closed || !schemaInbox.offer(error)) { + schemaPending.decrementAndGet(); + return false; + } + startDispatcherIfNeeded(); + return true; + } + + public int getPendingSchemaNotifications() { + return schemaPending.get(); + } + private void dispatchLoop() { - while (!closed || !inbox.isEmpty()) { - SenderError err; + while (!closed || !inbox.isEmpty() || !schemaInbox.isEmpty()) { + SenderError err = schemaInbox.poll(); + boolean schema = err != null; try { - err = inbox.poll(100, TimeUnit.MILLISECONDS); + if (err == null) { + err = inbox.poll(10, TimeUnit.MILLISECONDS); + } } catch (InterruptedException e) { if (closed) { return; @@ -344,6 +376,10 @@ private void dispatchLoop() { h.onError(err); } catch (Throwable t) { LOG.error("SenderErrorHandler threw on {}: {}", err, t.getMessage(), t); + } finally { + if (schema) { + schemaPending.decrementAndGet(); + } } } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java new file mode 100644 index 000000000..67cec0912 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java @@ -0,0 +1,116 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.std.Crc32c; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; + +import java.util.UUID; + +/** Durable identity for one lifecycle of an SF slot's FSN namespace. */ +public final class SlotEpoch { + public static final String FILE_NAME = ".slot-epoch"; + private static final int CRC_OFFSET = 28; + private static final int FILE_SIZE = 32; + private static final int MAGIC = 0x31455053; // SPE1 little-endian + private static final int VERSION = 1; + + private SlotEpoch() { + } + + /** + * Opens or creates the slot epoch. The caller must hold the slot's + * exclusive {@link SlotLock}; this method publishes a newly-created epoch + * with file and directory durability before returning it. + */ + public static String openOrCreate(FilesFacade ff, String slotDir) { + return openOrCreate(ff, slotDir, false); + } + + /** + * Opens the epoch for a recovered FSN namespace, or replaces it when the + * caller has proved that this is a fresh namespace. The latter check is + * required even though clean close normally removes the sidecar: a crash + * or unlink failure after durable segment removal can leave only the old + * epoch behind. + */ + public static String openOrCreate(FilesFacade ff, String slotDir, boolean freshFsnNamespace) { + String path = slotDir + "/" + FILE_NAME; + if (ff.exists(path) && !freshFsnNamespace) { + String epoch = read(ff, path); + // Also completes a prior create whose rename succeeded but whose + // directory barrier reported a transient failure. + if (ff.fsyncDir(slotDir) != 0) { + throw new SfOperationalException("could not sync durable slot epoch " + path); + } + return epoch; + } + UUID uuid = UUID.randomUUID(); + String value = uuid.toString(); + String temp = path + ".tmp-" + UUID.randomUUID(); + long mem = Unsafe.malloc(FILE_SIZE, MemoryTag.NATIVE_DEFAULT); + int fd = -1; + boolean published = false; + try { + Unsafe.getUnsafe().setMemory(mem, FILE_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(mem, MAGIC); + Unsafe.getUnsafe().putInt(mem + 4, VERSION); + Unsafe.getUnsafe().putLong(mem + 8, uuid.getMostSignificantBits()); + Unsafe.getUnsafe().putLong(mem + 16, uuid.getLeastSignificantBits()); + Unsafe.getUnsafe().putInt(mem + CRC_OFFSET, Crc32c.update(Crc32c.INIT, mem, CRC_OFFSET)); + fd = ff.openRWExclusive(temp); + if (fd < 0 || !ff.allocate(fd, FILE_SIZE) + || ff.write(fd, mem, FILE_SIZE, 0) != FILE_SIZE + || ff.fsync(fd) != 0) { + throw new SfOperationalException("could not create durable slot epoch " + path); + } + ff.close(fd); + fd = -1; + if (freshFsnNamespace && ff.exists(path) && !ff.remove(path)) { + throw new SfOperationalException("could not replace stale slot epoch " + path); + } + if (ff.rename(temp, path) != 0 || ff.fsyncDir(slotDir) != 0) { + throw new SfOperationalException("could not publish durable slot epoch " + path); + } + published = true; + return value; + } finally { + if (fd >= 0) { + ff.close(fd); + } + Unsafe.free(mem, FILE_SIZE, MemoryTag.NATIVE_DEFAULT); + if (!published) { + ff.remove(temp); + } + } + } + + public static String read(FilesFacade ff, String path) { + if (ff.length(path) != FILE_SIZE) { + throw new UnreplayableSlotException("invalid slot epoch size " + path); + } + long mem = Unsafe.malloc(FILE_SIZE, MemoryTag.NATIVE_DEFAULT); + int fd = ff.openRW(path); + try { + if (fd < 0 || ff.read(fd, mem, FILE_SIZE, 0) != FILE_SIZE + || Unsafe.getUnsafe().getInt(mem) != MAGIC + || Unsafe.getUnsafe().getInt(mem + 4) != VERSION + || Unsafe.getUnsafe().getInt(mem + CRC_OFFSET) + != Crc32c.update(Crc32c.INIT, mem, CRC_OFFSET)) { + throw new UnreplayableSlotException("invalid slot epoch " + path); + } + return new UUID(Unsafe.getUnsafe().getLong(mem + 8), + Unsafe.getUnsafe().getLong(mem + 16)).toString(); + } finally { + if (fd >= 0) { + ff.close(fd); + } + Unsafe.free(mem, FILE_SIZE, MemoryTag.NATIVE_DEFAULT); + } + } +} diff --git a/core/src/main/java/io/questdb/client/impl/PooledSender.java b/core/src/main/java/io/questdb/client/impl/PooledSender.java index 7b4e5f802..f108844ff 100644 --- a/core/src/main/java/io/questdb/client/impl/PooledSender.java +++ b/core/src/main/java/io/questdb/client/impl/PooledSender.java @@ -25,6 +25,8 @@ package io.questdb.client.impl; import io.questdb.client.Sender; +import io.questdb.client.LineSenderServerException; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; import io.questdb.client.std.Decimal128; @@ -154,6 +156,23 @@ public void close() { if (generation != slot.generation()) { return; } + Sender delegate = slot.live(generation); + QwpWebSocketSender qwp = delegate instanceof QwpWebSocketSender ? (QwpWebSocketSender) delegate : null; + if (qwp != null && qwp.hasOwnedSchemaFailure()) { + LineSenderServerException failure; + try { + failure = qwp.releaseFailedSchemaLease(); + qwp.checkSchemaSlotHealth(); + } catch (RuntimeException | Error operationalFailure) { + slot.pool().discardBroken(this); + throw operationalFailure; + } + slot.pool().giveBack(this); + if (failure != null) { + throw failure; + } + return; + } // Track normal completion rather than catching a specific throwable // type. flush() can exit abnormally with an Error (AssertionError // under -ea, OutOfMemoryError, ...) as well as a RuntimeException; @@ -169,6 +188,16 @@ public void close() { slot.live(generation).flush(); flushed = true; } finally { + if (!flushed && qwp != null && qwp.hasOwnedSchemaFailure()) { + try { + qwp.releaseFailedSchemaLease(); + qwp.checkSchemaSlotHealth(); + flushed = true; + } catch (RuntimeException | Error operationalFailure) { + slot.pool().discardBroken(this); + throw operationalFailure; + } + } if (flushed) { slot.pool().giveBack(this); } else { diff --git a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java index 574d6b59e..4a2b8a734 100644 --- a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java +++ b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java @@ -148,6 +148,29 @@ public QuestDBImpl( SenderErrorHandler errorHandler, SenderConnectionListener connectionListener, BackgroundDrainerListener drainerListener + ) { + this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, housekeeperIntervalMillis, queryCloseTimeoutMillis, senderFactory, connectHook, tokenProvider, errorHandler, connectionListener, drainerListener, io.questdb.client.SenderError.Policy.REJECT_AND_CONTINUE, true, null); + } + + public QuestDBImpl( + String ingestConfig, + String queryConfig, + int senderMin, + int senderMax, + int queryMin, + int queryMax, + long acquireTimeoutMillis, + long idleTimeoutMillis, + long maxLifetimeMillis, + long housekeeperIntervalMillis, + long queryCloseTimeoutMillis, + IntFunction senderFactory, + Consumer connectHook, + HttpTokenProvider tokenProvider, + SenderErrorHandler errorHandler, + SenderConnectionListener connectionListener, + BackgroundDrainerListener drainerListener, + io.questdb.client.SenderError.Policy schemaMismatchPolicy, boolean dlqEnabled, String dlqDir ) { SenderPool builtSenderPool = null; QueryClientPool builtQueryPool = null; @@ -160,7 +183,8 @@ public QuestDBImpl( // build() never blocks on a slow / reachable-but-not-acking // server; the housekeeper drives it via runStartupRecoveryStep(). true, - errorHandler, connectionListener, drainerListener, tokenProvider); + errorHandler, connectionListener, drainerListener, tokenProvider, + schemaMismatchPolicy, dlqEnabled, dlqDir); builtQueryPool = new QueryClientPool( queryConfig, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, connectHook, null, tokenProvider); diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java index 912d3b444..0636719c3 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderPool.java +++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java @@ -34,6 +34,7 @@ import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerListener; import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher; +import io.questdb.client.cutlass.qwp.client.sf.cursor.DefaultSenderErrorHandler; import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLock; import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLockContentionException; import io.questdb.client.std.Files; @@ -154,6 +155,9 @@ public final class SenderPool implements AutoCloseable { private final SenderConnectionListener connectionListener; private final BackgroundDrainerListener drainerListener; private final SenderErrorHandler errorHandler; + private final SenderError.Policy schemaMismatchPolicy; + private final boolean dlqEnabled; + private final String dlqDir; private final long idleTimeoutMillis; private final HttpTokenProvider tokenProvider; // Delivery channel for recovery-delegate errors that pass the @@ -483,6 +487,18 @@ public static SenderPool createWithRecoveryControlsForTesting( drainerListener, null, null, null, tokenProvider, null); } + SenderPool(String configurationString, int minSize, int maxSize, + long acquireTimeoutMillis, long idleTimeoutMillis, long maxLifetimeMillis, + IntFunction senderFactory, boolean deferStartupRecovery, + SenderErrorHandler errorHandler, SenderConnectionListener connectionListener, + BackgroundDrainerListener drainerListener, HttpTokenProvider tokenProvider, + SenderError.Policy schemaMismatchPolicy, boolean dlqEnabled, String dlqDir) { + this(configurationString, minSize, maxSize, acquireTimeoutMillis, idleTimeoutMillis, + maxLifetimeMillis, senderFactory, deferStartupRecovery, errorHandler, + connectionListener, drainerListener, null, null, null, tokenProvider, null, + schemaMismatchPolicy, dlqEnabled, dlqDir); + } + private SenderPool( String configurationString, int minSize, @@ -500,10 +516,35 @@ private SenderPool( Runnable recoveryWaiter, HttpTokenProvider tokenProvider, Runnable beforeFailedRecoveryJoinHook + ) { + this(configurationString, minSize, maxSize, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, senderFactory, deferStartupRecovery, errorHandler, connectionListener, drainerListener, postFactoryHook, recoveryThreadFactory, recoveryWaiter, tokenProvider, beforeFailedRecoveryJoinHook, SenderError.Policy.REJECT_AND_CONTINUE, true, null); + } + + private SenderPool( + String configurationString, + int minSize, + int maxSize, + long acquireTimeoutMillis, + long idleTimeoutMillis, + long maxLifetimeMillis, + IntFunction senderFactory, + boolean deferStartupRecovery, + SenderErrorHandler errorHandler, + SenderConnectionListener connectionListener, + BackgroundDrainerListener drainerListener, + Runnable postFactoryHook, + ThreadFactory recoveryThreadFactory, + Runnable recoveryWaiter, + HttpTokenProvider tokenProvider, + Runnable beforeFailedRecoveryJoinHook, + SenderError.Policy schemaMismatchPolicy, boolean dlqEnabled, String dlqDir ) { if (minSize < 0 || maxSize < 1 || minSize > maxSize) { throw new IllegalArgumentException("invalid pool sizing: min=" + minSize + ", max=" + maxSize); } + this.schemaMismatchPolicy = schemaMismatchPolicy; + this.dlqEnabled = dlqEnabled; + this.dlqDir = dlqDir; this.errorHandler = errorHandler; this.connectionListener = connectionListener; this.drainerListener = drainerListener; @@ -544,9 +585,21 @@ private SenderPool( this.storeAndForward = probe.isStoreAndForwardEnabled(); this.slotBaseId = this.storeAndForward ? probe.getConfiguredSenderId() : null; this.sfDir = this.storeAndForward ? probe.getConfiguredSfDir() : null; + if (schemaMismatchPolicy == SenderError.Policy.REJECT_AND_CONTINUE + && dlqEnabled && (dlqDir != null || sfDir != null)) { + String destination = dlqDir != null ? dlqDir : sfDir; + try { + java.nio.file.Files.createDirectories(java.nio.file.Paths.get(destination)); + } catch (java.io.IOException e) { + throw new io.questdb.client.cutlass.line.LineSenderException(e) + .put("could not create schema preservation destination ").put(destination); + } + io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver.probeDirectory( + io.questdb.client.std.FilesFacade.INSTANCE, destination); + } this.slotInUse = this.storeAndForward ? new boolean[maxSize] : null; - this.recoveryErrorDispatcher = (errorHandler != null && this.storeAndForward) - ? new SenderErrorDispatcher(errorHandler, SenderErrorDispatcher.DEFAULT_CAPACITY, + this.recoveryErrorDispatcher = this.storeAndForward + ? new SenderErrorDispatcher(errorHandler != null ? errorHandler : DefaultSenderErrorHandler.INSTANCE, SenderErrorDispatcher.DEFAULT_CAPACITY, "qdb-sf-pool-recovery-errors") : null; // Pre-warm minSize connections. Pre-warm runs single-threaded in the @@ -1123,7 +1176,9 @@ private RecoveryDrainOutcome drainCandidateSlotForRecovery(int slotIndex, String // on a timeout: a server that fails to ack within the budget // will very likely do the same for every remaining slot -- the // same reasoning as the build-failure case above. - if (!recoverer.delegate().drain(remainingMillis)) { + if (!(recoverer.delegate() instanceof QwpWebSocketSender + ? ((QwpWebSocketSender) recoverer.delegate()).drainResolved(remainingMillis) + : recoverer.delegate().drain(remainingMillis))) { if (warnSlotOnce(slotIndex)) { LOG.warn("startup SF recovery: drain did not ack slot {} " + "within {}ms; deferring this and remaining slots", @@ -1228,6 +1283,7 @@ public PooledSender borrow() { // wrapper handed out can be told apart from any prior, // now-stale borrow of the same slot. s.bumpGeneration(); + s.beginSchemaLease(); return new PooledSender(s, s.generation()); } if (all.size() + inFlightCreations + closingSlots + leakedSlots + recoveringSlots < maxSize) { @@ -1296,6 +1352,7 @@ public PooledSender borrow() { } all.add(created); created.bumpGeneration(); + created.beginSchemaLease(); inFlightCreations--; creationFinished.signalAll(); return new PooledSender(created, created.generation()); @@ -1649,11 +1706,15 @@ public void giveBack(PooledSender ps) { // twice and hand it to two borrowers writing into one delegate. return; } + io.questdb.client.LineSenderServerException schemaFailure = s.endSchemaLease(); s.bumpGeneration(); s.markIdleAt(System.currentTimeMillis()); assert !available.contains(s) : "slot already present in available deque on giveBack"; available.addLast(s); slotReleased.signal(); + if (schemaFailure != null) { + throw schemaFailure; + } return; } } finally { @@ -2039,7 +2100,11 @@ private Sender.LineSenderBuilder applyRecoveryCallbacks(Sender.LineSenderBuilder builder.errorHandler(new SenderErrorHandler() { @Override public void onError(SenderError error) { - if (isRecoveryEventUserRelevant(error)) { + if (error.getAppliedPolicy() == SenderError.Policy.REJECT_AND_CONTINUE) { + // Already on the delegate's reliable dispatcher: a second lossy hop + // would defeat the schema FIFO's delivery guarantee. + (errorHandler != null ? errorHandler : DefaultSenderErrorHandler.INSTANCE).onError(error); + } else if (isRecoveryEventUserRelevant(error)) { recoveryErrorDispatcher.offer(error); } } @@ -2049,6 +2114,10 @@ public void onError(SenderError error) { } private Sender.LineSenderBuilder applyTokenProvider(Sender.LineSenderBuilder builder) { + builder.schemaMismatchPolicy(schemaMismatchPolicy).dlqEnabled(dlqEnabled); + if (dlqDir != null) { + builder.dlqDirectory(dlqDir); + } if (tokenProvider != null) { builder.httpTokenProvider(tokenProvider); } diff --git a/core/src/main/java/io/questdb/client/impl/SenderSlot.java b/core/src/main/java/io/questdb/client/impl/SenderSlot.java index 5d9ed6ff7..b2bf3f848 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderSlot.java +++ b/core/src/main/java/io/questdb/client/impl/SenderSlot.java @@ -25,6 +25,7 @@ package io.questdb.client.impl; import io.questdb.client.Sender; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; /** * One reusable {@link SenderPool} slot: owns a real {@link Sender} delegate, its @@ -68,6 +69,9 @@ final class SenderSlot { this.slotIndex = slotIndex; this.createdAtMillis = System.currentTimeMillis(); this.idleSinceMillis = this.createdAtMillis; + if (delegate instanceof QwpWebSocketSender) { + ((QwpWebSocketSender) delegate).prepareSchemaPoolSlot(); + } } /** @@ -79,6 +83,19 @@ void bumpGeneration() { generation++; } + void beginSchemaLease() { + if (delegate instanceof QwpWebSocketSender) { + ((QwpWebSocketSender) delegate).beginSchemaLease(generation); + } + } + + io.questdb.client.LineSenderServerException endSchemaLease() { + if (delegate instanceof QwpWebSocketSender) { + return ((QwpWebSocketSender) delegate).endSchemaLease(); + } + return null; + } + long createdAtMillis() { return createdAtMillis; } diff --git a/core/src/test/java/io/questdb/client/test/SenderErrorTest.java b/core/src/test/java/io/questdb/client/test/SenderErrorTest.java index 10e91d139..416880d90 100644 --- a/core/src/test/java/io/questdb/client/test/SenderErrorTest.java +++ b/core/src/test/java/io/questdb/client/test/SenderErrorTest.java @@ -57,11 +57,12 @@ public void testAllCategoriesEnumerable() { @Test public void testAllPoliciesEnumerable() { SenderError.Policy[] policies = SenderError.Policy.values(); - Assert.assertEquals(4, policies.length); + Assert.assertEquals(5, policies.length); Assert.assertEquals(SenderError.Policy.RETRIABLE, SenderError.Policy.valueOf("RETRIABLE")); Assert.assertEquals(SenderError.Policy.RETRIABLE_OTHER, SenderError.Policy.valueOf("RETRIABLE_OTHER")); Assert.assertEquals(SenderError.Policy.TERMINAL, SenderError.Policy.valueOf("TERMINAL")); Assert.assertEquals(SenderError.Policy.ABANDONED, SenderError.Policy.valueOf("ABANDONED")); + Assert.assertEquals(SenderError.Policy.REJECT_AND_CONTINUE, SenderError.Policy.valueOf("REJECT_AND_CONTINUE")); } @Test diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/BackgroundDrainerEndToEndTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/BackgroundDrainerEndToEndTest.java index 0a147ca18..8e4065416 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/BackgroundDrainerEndToEndTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/BackgroundDrainerEndToEndTest.java @@ -25,6 +25,8 @@ package io.questdb.client.test.cutlass.qwp.client.sf; import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.WebSocketResponse; import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; import io.questdb.client.std.Files; import io.questdb.client.test.cutlass.qwp.client.QwpWireTestUtils; @@ -42,8 +44,10 @@ import java.nio.file.Paths; import java.util.ArrayList; import java.util.List; +import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; /** * End-to-end coverage of the background drainer adopting an orphan slot. @@ -73,6 +77,68 @@ public void tearDown() { if (sfDir != null) rmDirRec(sfDir); } + @Test + public void testDrainerPreservesAndReportsSchemaRejectedSpan() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) { + silent.start(); + Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS)); + String ghostConfig = "ws::addr=localhost:" + silent.getPort() + + ";sf_dir=" + sfDir + + ";sender_id=ghost;close_flush_timeout_millis=0;"; + try (Sender ghost = Sender.fromConfig(ghostConfig)) { + ghost.table("bad").stringColumn("value", "wrong").atNow(); + Assert.assertEquals(0L, ghost.flushAndGetSequence()); + } + } + + CountDownLatch reported = new CountDownLatch(1); + AtomicReference captured = new AtomicReference<>(); + AtomicLong nextSequence = new AtomicLong(); + try (TestWebSocketServer rejecting = new TestWebSocketServer( + new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = nextSequence.getAndIncrement(); + try { + client.sendBinary(QwpWireTestUtils.buildNack( + sequence, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + })) { + rejecting.start(); + Assert.assertTrue(rejecting.awaitStart(5, TimeUnit.SECONDS)); + String primaryConfig = "ws::addr=localhost:" + rejecting.getPort() + + ";sf_dir=" + sfDir + + ";sender_id=primary;drain_orphans=true;max_background_drainers=1;"; + try (Sender ignored = Sender.builder(primaryConfig) + .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE) + .dlqEnabled(true) + .errorHandler(error -> { + if (error.getAppliedPolicy() == SenderError.Policy.REJECT_AND_CONTINUE) { + captured.set(error); + reported.countDown(); + } + }) + .build()) { + Assert.assertTrue("schema rejection callback", reported.await(10, TimeUnit.SECONDS)); + } + } + + SenderError error = captured.get(); + Assert.assertNotNull(error); + Assert.assertEquals(SenderError.Category.SCHEMA_MISMATCH, error.getCategory()); + Assert.assertEquals(0L, error.getRejectedFsn()); + Assert.assertEquals(0L, error.getFromFsn()); + Assert.assertEquals(0L, error.getToFsn()); + Assert.assertNotNull("archive must be published before callback", error.getRejectedPath()); + Assert.assertTrue(java.nio.file.Files.isDirectory(Paths.get(error.getRejectedPath()))); + Assert.assertFalse(Files.exists(sfDir + "/ghost/" + OrphanScanner.FAILED_SENTINEL_NAME)); + }); + } + @Test public void testDrainerEmptiesOrphanSlotAgainstAckServer() throws Exception { TestUtils.assertMemoryLeak(() -> { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java index a8bd96c9d..6eff9e600 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java @@ -24,6 +24,12 @@ package io.questdb.client.test.cutlass.qwp.client.sf.cursor; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.FilesFacade; +import java.util.concurrent.atomic.AtomicReference; import io.questdb.client.cutlass.qwp.client.sf.cursor.AckWatermark; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; @@ -58,6 +64,83 @@ public void tearDown() { removeRecursive(slotPath); } + @Test + public void testPreservedOrphanRetiresAndReportsWithoutConnecting() throws Exception { + assertOrphanRetiresOffline(true); + } + + @Test + public void testUnreportedOrphanRetiresWithoutConnectingWhenPreservationEnabled() throws Exception { + assertOrphanRetiresOffline(false); + } + + private void assertOrphanRetiresOffline(boolean archive) throws Exception { + TestUtils.assertMemoryLeak(() -> { + String archivedPath = null; + try (CursorSendEngine original = new CursorSendEngine(slotPath, SEGMENT_BYTES)) { + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, + QwpConstants.FLAG_DEFER_COMMIT); + original.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + if (archive) { + String epoch = SlotEpoch.openOrCreate( + FilesFacade.INSTANCE, slotPath, original.freshFsnNamespace()); + SenderError error = new SenderError( + SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "bad schema", 0, 0, 0, null, 1); + archivedPath = RejectedMiniSlotArchive.preserve( + FilesFacade.INSTANCE, original, null, slotPath, + Paths.get(slotPath).getFileName().toString(), epoch, error).path; + } + } + AtomicReference report = new AtomicReference<>(); + AtomicReference callbackThread = new AtomicReference<>(); + BackgroundDrainer drainer = new BackgroundDrainer(slotPath, SEGMENT_BYTES, Long.MAX_VALUE, + () -> { throw new AssertionError("orphan-only retirement must not connect"); }, + 5_000L, 1L, 10L, true, 200L); + drainer.configureSchemaMismatch(SenderError.Policy.REJECT_AND_CONTINUE, true, null, + e -> { report.set(e); callbackThread.set(Thread.currentThread()); }); + drainer.run(); + Assert.assertEquals(BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + Assert.assertFalse(OrphanScanner.isCandidateOrphan(slotPath)); + if (archive) { + Assert.assertNotNull(report.get()); + Assert.assertEquals(archivedPath, report.get().getRejectedPath()); + Assert.assertNotSame(Thread.currentThread(), callbackThread.get()); + Assert.assertTrue(java.nio.file.Files.isDirectory(Paths.get(archivedPath))); + } else { + Assert.assertNull(report.get()); + } + }); + } + + @Test + public void testPreservationDestinationFailureDoesNotQuarantine() throws Exception { + TestUtils.assertMemoryLeak(() -> { + seedUnackedFrame(); + String blocked = slotPath + "/blocked-destination"; + java.nio.file.Files.createFile(Paths.get(blocked)); + BackgroundDrainer drainer = new BackgroundDrainer(slotPath, SEGMENT_BYTES, + Long.MAX_VALUE, () -> { throw new AssertionError("must fail before connect"); }, + 5_000L, 1L, 10L, true, 200L); + drainer.configureSchemaMismatch(SenderError.Policy.REJECT_AND_CONTINUE, + true, blocked, null); + drainer.run(); + Assert.assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + Assert.assertFalse(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + Assert.assertTrue("storage outage must leave source recoverable", OrphanScanner.isCandidateOrphan(slotPath)); + try (CursorSendEngine ignored = new CursorSendEngine(slotPath, SEGMENT_BYTES)) { + Assert.assertTrue(ignored.publishedFsn() >= 0); + } + }); + } + @Test public void testConnectErrorPropagatesWithoutQuarantine() throws Exception { TestUtils.assertMemoryLeak(() -> { @@ -243,7 +326,7 @@ public void testSealedResidueFirstSightHealsAndDoesNotQuarantine() throws Except // client's reseal-after-recovery did. int fd = Files.openRW(p0Path); Assert.assertTrue("openRW must succeed", fd >= 0); - long junk = Unsafe.malloc(12, MemoryTag.NATIVE_DEFAULT); + long junk = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); try { for (int i = 0; i < 3; i++) { Unsafe.getUnsafe().putInt(junk + i * 4L, 0xCAFEBABE); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java index ec828aae2..5f0328abd 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java @@ -25,6 +25,7 @@ package io.questdb.client.test.cutlass.qwp.client.sf.cursor; import io.questdb.client.DefaultHttpClientConfiguration; +import io.questdb.client.SenderError; import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.QwpRoleMismatchException; @@ -32,6 +33,8 @@ import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; import io.questdb.client.cutlass.qwp.client.sf.cursor.PersistedSymbolDict; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.network.PlainSocketFactory; import io.questdb.client.std.Files; @@ -129,6 +132,48 @@ public void testCatchUpFrameAckDoesNotAdvanceTrimWatermark() throws Exception { }); } + @Test + public void testDiskSuccessorDictionarySurvivesSkippedDeltaCarrier() throws Exception { + assertSkippedDeltaCarrierKeepsSuccessorReplayable(false); + } + + @Test + public void testMemorySuccessorDictionarySurvivesSkippedDeltaCarrier() throws Exception { + assertSkippedDeltaCarrierKeepsSuccessorReplayable(true); + } + + @Test + public void testSealedSchemaRangeStopsAndSelfAcknowledges() throws Exception { + TestUtils.assertMemoryLeak(() -> { + CatchUpCapturingClient client = new CatchUpCapturingClient(0); + try (CursorSendEngine engine = new CursorSendEngine(null, 16_384)) { + appendDeltaDictFrame(engine, 0, 'a'); + appendDeltaDictFrame(engine, 1, 'b'); + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(1, 0, false); + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 7, "mismatch", 1, + 1, 1, "tab", System.nanoTime()); + assertTrue(state.reject(1, 0, error)); + SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(ignored -> { }); + CursorWebSocketSendLoop loop = newLoop(engine, client); + try { + loop.setSchemaRejectionState(state); + loop.setErrorDispatcher(dispatcher); + loop.positionCursorForStartForTest(); + assertTrue(loop.trySendOneForTest()); + assertEquals(1, engine.ackedFsn()); + assertEquals(2, loop.getSchemaFramesRetired()); + assertEquals(-1, state.stopFsn()); + assertEquals(Arrays.asList("a", "b"), readMirrorSymbols(loop)); + } finally { + loop.close(); + dispatcher.close(); + } + } + }); + } + @Test public void testSplitCatchUpFramesAcksDoNotAdvanceTrimWatermark() throws Exception { // A small advertised cap splits the dictionary across several catch-up @@ -1198,6 +1243,37 @@ private void assertUnrelatedReconnectStateRestartsCapGapEpisode(boolean roleReje }); } + private void assertSkippedDeltaCarrierKeepsSuccessorReplayable(boolean memory) throws Exception { + TestUtils.assertMemoryLeak(() -> { + CatchUpCapturingClient client = new CatchUpCapturingClient(0); + try (CursorSendEngine closeableEngine = memory + ? new CursorSendEngine(null, 16_384) + : newEngine()) { + appendDeltaDictFrame(closeableEngine, 0, 'a'); // carrier to retire + appendDeltaDictFrame(closeableEngine, 1, 'b'); // surviving successor + assertEquals(QwpConstants.FLAG_DELTA_SYMBOL_DICT & 0xff, + closeableEngine.liveQwpFrameFlags(0)); + assertEquals(-1, closeableEngine.liveQwpFrameFlags(2)); + CursorWebSocketSendLoop loop = newLoop(closeableEngine, client); + try { + loop.catchUpSkippedRangeForTest(0, 0); + assertEquals(Arrays.asList("a"), readMirrorSymbols(loop)); + + // The successor starts at id 1. Folding it after the skipped + // carrier is the same contiguity check trySendOne applies and + // proves the carrier's symbol was not lost from catch-up state. + loop.catchUpSkippedRangeForTest(1, 1); + assertEquals(Arrays.asList("a", "b"), readMirrorSymbols(loop)); + + invokeSetWireBaselineWithCatchUp(loop, 2L); + assertCatchUpReassembles(client, "a", "b"); + } finally { + loop.close(); + } + } + }); + } + // Builds a QWP delta frame [12-byte header][deltaStart varint][deltaCount // varint][ [len varint][utf8] ... ] for the given symbols. accumulateSentDict // skips the header, so its content is irrelevant; the caller frees the frame. diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java index 1f446a113..6cdf61f44 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java @@ -29,24 +29,34 @@ import io.questdb.client.SenderError; import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.http.client.WebSocketFrameHandler; +import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.WebSocketResponse; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; import io.questdb.client.network.PlainSocketFactory; -import io.questdb.client.std.Files; +import io.questdb.client.std.FilesFacade; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Unsafe; +import io.questdb.client.test.tools.DelegatingFilesFacade; import io.questdb.client.test.tools.TestUtils; -import org.junit.After; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; import java.nio.charset.StandardCharsets; -import java.nio.file.Paths; import java.util.ArrayList; import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -76,32 +86,13 @@ public class CursorWebSocketSendLoopPoisonFrameTest { private String tmpDir; - @Before - public void setUp() { - tmpDir = Paths.get(System.getProperty("java.io.tmpdir"), - "qdb-cursor-poison-" + System.nanoTime()).toString(); - assertEquals(0, Files.mkdir(tmpDir, Files.DIR_MODE_DEFAULT)); - } + @Rule + public final TemporaryFolder temp = new TemporaryFolder(); - @After - public void tearDown() { - if (tmpDir == null) return; - long find = Files.findFirst(tmpDir); - if (find > 0) { - try { - int rc = 1; - while (rc > 0) { - String name = Files.utf8ToString(Files.findName(find)); - if (name != null && !".".equals(name) && !"..".equals(name)) { - Files.remove(tmpDir + "/" + name); - } - rc = Files.findNext(find); - } - } finally { - Files.findClose(find); - } - } - Files.remove(tmpDir); + @Before + public void setUp() throws Exception { + // Preservation tests create nested archives; the rule cleans those too. + tmpDir = temp.newFolder("slot").getAbsolutePath(); } @Test @@ -155,6 +146,229 @@ public void testDurableModeDetectorFiresDespiteReplayReOks() throws Exception { }); } + @Test + public void testSecondSchemaNackWhileRetirementPendingFailsClosedAfterDurableReplayOk() throws Exception { + TestUtils.assertMemoryLeak(() -> { + List clients = new ArrayList<>(); + try (CursorSendEngine engine = newEngine()) { + appendFrames(engine, 2); + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1L, 0L, false); + SenderError first = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "first mismatch", 1L, + 1L, 1L, null, System.nanoTime()); + assertTrue(state.reject(1L, 1L, first)); + try (CursorWebSocketSendLoop loop = newDurableLoop(engine, clients)) { + loop.setSchemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE); + loop.setSchemaRejectionState(state); + + assertEquals("retirement must wait for the durable predecessor", -1L, engine.ackedFsn()); + + setSentCount(loop, 2); + deliverOk(loop, 0L, names("trades"), txns(7L)); + assertEquals("an OK without its durable ACK must not release the predecessor", + -1L, engine.ackedFsn()); + deliverSchemaNack(loop, 1L, "second mismatch"); + + try { + loop.checkError(); + fail("a second schema rejection cannot replace an unresolved retirement range"); + } catch (LineSenderServerException e) { + assertEquals(SenderError.Category.SCHEMA_MISMATCH, + e.getServerError().getCategory()); + } + assertEquals("fail-closed fallback must preserve every source frame", + -1L, engine.ackedFsn()); + assertEquals(1L, state.stopFsn()); + } + } finally { + closeAll(clients); + } + }); + } + + @Test + public void testSkippedRangeFailureLatchesInsteadOfReconnect() throws Exception { + TestUtils.assertMemoryLeak(() -> { + List clients = new ArrayList<>(); + try (CursorSendEngine engine = newEngine()) { + appendFrames(engine, 1); + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1L, 0L, false); + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "missing range frame", 1L, + 1L, 1L, null, System.nanoTime()); + assertTrue(state.reject(1L, 1L, error)); + assertTrue(engine.acknowledge(0L)); + AtomicReference reported = new AtomicReference<>(); + try (CursorWebSocketSendLoop loop = newDurableLoop(engine, clients); + io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher dispatcher = + new io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher(reported::set)) { + loop.setSchemaRejectionState(state); + loop.setErrorDispatcher(dispatcher); + assertFalse(loop.tryRetireSchemaRangeForTest()); + + try { + loop.checkError(); + fail("a skipped-range inconsistency must latch terminal"); + } catch (LineSenderException e) { + assertTrue(e.getMessage().contains("frame disappeared before retirement")); + } + assertEquals("the inconsistent range must remain unacknowledged", + 0L, engine.ackedFsn()); + long deadline = System.nanoTime() + 5_000_000_000L; + while (reported.get() == null && System.nanoTime() < deadline) { + Thread.yield(); + } + assertEquals("the callback must describe the fail-closed policy actually applied", + SenderError.Policy.TERMINAL, reported.get().getAppliedPolicy()); + assertEquals(1L, reported.get().getFromFsn()); + assertEquals(1L, reported.get().getToFsn()); + } + } finally { + closeAll(clients); + } + }); + } + + @Test + public void testPreservationFailureRetriesThenRetiresWithoutTerminal() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (CursorSendEngine engine = newEngine()) { + appendFrames(engine, 1); + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1L, 0L, false); + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "mismatch", 0L, + 0L, 0L, null, System.nanoTime()); + assertTrue(state.reject(0L, 0L, error)); + FailFirstTemporaryMkdirFacade ff = new FailFirstTemporaryMkdirFacade(); + String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, tmpDir); + AtomicReference reported = new AtomicReference<>(); + SchemaPreserver preserver = new SchemaPreserver( + ff, tmpDir, "retry-slot", epoch); + try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(reported::set); + CursorWebSocketSendLoop loop = newDurableLoop(engine, new ArrayList<>())) { + loop.setSchemaRejectionState(state); + loop.setSchemaPreserver(preserver); + loop.setErrorDispatcher(dispatcher); + + assertFalse(loop.tryRetireSchemaRangeForTest()); + assertEquals(1L, loop.getDlqWriteFailures()); + assertEquals(-1L, engine.ackedFsn()); + assertEquals(null, loop.getTerminalError()); + assertEquals(null, reported.get()); + + long successDeadline = System.nanoTime() + 5_000_000_000L; + assertTrue(loop.tryRetireSchemaRangeForTest()); + assertEquals(0L, engine.ackedFsn()); + assertEquals(null, loop.getTerminalError()); + while (reported.get() == null && System.nanoTime() < successDeadline) Thread.yield(); + assertEquals(SenderError.Policy.REJECT_AND_CONTINUE, + reported.get().getAppliedPolicy()); + assertTrue(reported.get().getRejectedPath() != null); + } + } + }); + } + + @Test + public void testFullSchemaNotificationFifoDoesNotRepeatPreservation() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (CursorSendEngine engine = newEngine()) { + appendFrames(engine, 1); + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1L, 0L, false); + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "mismatch", 0L, + 0L, 0L, null, System.nanoTime()); + assertTrue(state.reject(0L, 0L, error)); + CountingPreserveFacade ff = new CountingPreserveFacade(tmpDir + "/rejected"); + String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, tmpDir); + CountDownLatch handlerEntered = new CountDownLatch(1); + CountDownLatch releaseHandler = new CountDownLatch(1); + try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(ignored -> { + handlerEntered.countDown(); + try { + releaseHandler.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }); CursorWebSocketSendLoop loop = newDurableLoop(engine, new ArrayList<>())) { + assertTrue(dispatcher.tryOfferSchema(error)); + assertTrue(handlerEntered.await(5, TimeUnit.SECONDS)); + for (int i = 1; i < SenderErrorDispatcher.DEFAULT_CAPACITY; i++) { + assertTrue(dispatcher.tryOfferSchema(error)); + } + loop.setSchemaRejectionState(state); + loop.setSchemaPreserver(new SchemaPreserver(ff, tmpDir, "fifo-slot", epoch)); + loop.setErrorDispatcher(dispatcher); + + assertFalse(loop.tryRetireSchemaRangeForTest()); + int syncsAfterPreserve = ff.rejectedRootSyncs; + assertTrue(syncsAfterPreserve > 0); + assertFalse(loop.tryRetireSchemaRangeForTest()); + assertEquals("a cached notification must avoid archive validation and fsync", + syncsAfterPreserve, ff.rejectedRootSyncs); + assertEquals(-1L, engine.ackedFsn()); + + releaseHandler.countDown(); + long deadline = System.nanoTime() + 5_000_000_000L; + while (dispatcher.getPendingSchemaNotifications() + >= SenderErrorDispatcher.DEFAULT_CAPACITY + && System.nanoTime() < deadline) { + Thread.yield(); + } + assertTrue(loop.tryRetireSchemaRangeForTest()); + assertEquals(0L, engine.ackedFsn()); + } finally { + releaseHandler.countDown(); + } + } + }); + } + + @Test + public void testTransactionalCloserScanFailureFromNackLatchesWithoutReconnect() throws Exception { + TestUtils.assertMemoryLeak(() -> { + List clients = new ArrayList<>(); + try (CursorSendEngine engine = newEngine()) { + appendDeferredFrame(engine); + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1L, 0L, true); + state.endLease(1L, 1L); // recovered/advertised tail includes missing frame 1 + AtomicReference reported = new AtomicReference<>(); + try (CursorWebSocketSendLoop loop = newDurableLoop(engine, clients); + SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(reported::set)) { + loop.setSchemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE); + loop.setSchemaRejectionState(state); + loop.setErrorDispatcher(dispatcher); + setSentCount(loop, 1L); + deliverSchemaNack(loop, 0L, "transaction mismatch"); + try { + loop.checkError(); + fail("transactional closer scan failure must latch terminal"); + } catch (LineSenderException e) { + assertTrue(e.getMessage().contains("could not resolve schema-rejected")); + } + assertEquals("terminal path must not enter reconnect", 0L, + loop.getTotalReconnectAttempts()); + assertEquals(-1L, engine.ackedFsn()); + long deadline = System.nanoTime() + 5_000_000_000L; + while (reported.get() == null && System.nanoTime() < deadline) Thread.yield(); + assertEquals(SenderError.Policy.TERMINAL, reported.get().getAppliedPolicy()); + } + } finally { + closeAll(clients); + } + }); + } + @Test public void testPoisonTerminalNamesTheRejectedFsn() throws Exception { // The escalated terminal must name the frame the server rejected, not @@ -880,6 +1094,36 @@ public void testNotWritableRecycleFirstImmediateThenPaced() throws Exception { // harness // --------------------------------------------------------------------- + private static final class FailFirstTemporaryMkdirFacade extends DelegatingFilesFacade { + private boolean failed; + + @Override + public int mkdir(String path, int mode) { + if (!failed && path.contains("/rejected/.tmp-retry-slot-")) { + failed = true; + return -1; + } + return super.mkdir(path, mode); + } + } + + private static final class CountingPreserveFacade extends DelegatingFilesFacade { + private final String rejectedRoot; + private int rejectedRootSyncs; + + private CountingPreserveFacade(String rejectedRoot) { + this.rejectedRoot = rejectedRoot; + } + + @Override + public int fsyncDir(String path) { + if (rejectedRoot.equals(path)) { + rejectedRootSyncs++; + } + return super.fsyncDir(path); + } + } + /** * In-memory transport emulating a healthy server that deterministically * NACKs the head frame: accepts the connection, waits for one send on @@ -1056,6 +1300,19 @@ private static void appendFrames(CursorSendEngine engine, int count) { } } + private static void appendDeferredFrame(CursorSendEngine engine) { + long buf = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(buf, 16, (byte) 0); + Unsafe.getUnsafe().putInt(buf, io.questdb.client.cutlass.qwp.protocol.QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(buf + io.questdb.client.cutlass.qwp.protocol.QwpConstants.HEADER_OFFSET_FLAGS, + io.questdb.client.cutlass.qwp.protocol.QwpConstants.FLAG_DEFER_COMMIT); + engine.appendBlocking(buf, 16); + } finally { + Unsafe.free(buf, 16, MemoryTag.NATIVE_DEFAULT); + } + } + private static long buildErrorPayload(long wireSeq, byte status, String message) { // Error frame: status(1) + sequence(8) + msgLen(2) + bytes byte[] msg = message.getBytes(StandardCharsets.UTF_8); @@ -1140,6 +1397,18 @@ private static void deliverRetriableNack(CursorWebSocketSendLoop loop, long wire } } + private static void deliverSchemaNack(CursorWebSocketSendLoop loop, long wireSeq, + String msg) throws Exception { + long packed = buildErrorPayload(wireSeq, WebSocketResponse.STATUS_SCHEMA_MISMATCH, msg); + long ptr = packed & 0xFFFFFFFFFFFFL; + int size = (int) (packed >>> 48); + try { + invokeOnBinaryMessage(loop, ptr, size); + } finally { + Unsafe.free(ptr, size, MemoryTag.NATIVE_DEFAULT); + } + } + private static void closeAll(List clients) { // swapClient already closed all but the most recently installed // client; close() is idempotent, so sweep them all. diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java new file mode 100644 index 000000000..1de91badd --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java @@ -0,0 +1,142 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.DefaultHttpClientConfiguration; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.http.client.WebSocketClient; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLock; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.network.PlainSocketFactory; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import io.questdb.client.test.tools.DelegatingFilesFacade; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +import static org.junit.Assert.*; + +public class CursorWebSocketSendLoopSchemaPreservationCloseTest { + @Rule + public final TemporaryFolder temp = new TemporaryFolder(); + + @Test(timeout = 30_000L) + public void testBlockedCopyRetainsEngineUntilIoThreadCleanup() throws Exception { + TestUtils.assertMemoryLeak(() -> { + String directory = temp.newFolder("slot").getAbsolutePath(); + BlockingArchiveFacade ff = new BlockingArchiveFacade(); + CursorSendEngine engine = new CursorSendEngine(directory, 4096); + WebSocketClient client = new WebSocketClient( + DefaultHttpClientConfiguration.INSTANCE, PlainSocketFactory.INSTANCE) { + @Override + protected void ioWait(int timeout, int operation) { + } + + @Override + protected void setupIoWait() { + } + + @Override + public void closeTraffic() { + // No real socket; closing network traffic cannot cancel disk I/O. + } + }; + CursorWebSocketSendLoop loop = new CursorWebSocketSendLoop(client, engine, 0, + CursorWebSocketSendLoop.DEFAULT_PARK_NANOS, null, 1000, 5000, false); + CountDownLatch cleaned = new CountDownLatch(1); + try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(error -> { })) { + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + engine.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1, 0, false); + assertTrue(state.reject(0, 0, new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "mismatch", 0, + 0, 0, "tab", System.nanoTime()))); + loop.setSchemaRejectionState(state); + loop.setSchemaPreserver(new SchemaPreserver(ff, directory, "slot", + SlotEpoch.openOrCreate(FilesFacade.INSTANCE, directory))); + loop.setErrorDispatcher(dispatcher); + loop.setShutdownAwaitTimeoutMillis(25); + loop.start(); + assertTrue("I/O thread did not begin preservation", ff.entered.await(5, TimeUnit.SECONDS)); + assertEquals("qdb-cursor-ws-io", ff.copyThread.get().getName()); + assertEquals(-1, engine.ackedFsn()); + try { + loop.close(); + fail("blocked disk I/O must exhaust the shutdown budget"); + } catch (LineSenderException expected) { + assertTrue(expected.getMessage().contains("timed out")); + } + assertTrue(loop.delegateClose(() -> { + engine.close(); + cleaned.countDown(); + })); + assertEquals(1, cleaned.getCount()); + try (SlotLock ignored = SlotLock.acquire(directory)) { + fail("engine lock was released while the copy still uses its memory"); + } catch (IllegalStateException expected) { + // The I/O thread retains engine ownership until disk access ends. + } + ff.release.countDown(); + assertTrue("deferred engine cleanup did not finish", cleaned.await(5, TimeUnit.SECONDS)); + try (SlotLock ignored = SlotLock.acquire(directory)) { + // Rebuild can now acquire the same queue. + } + } finally { + ff.release.countDown(); + loop.close(); + engine.close(); + client.close(); + } + }); + } + + private static final class BlockingArchiveFacade extends DelegatingFilesFacade { + private final CountDownLatch entered = new CountDownLatch(1); + private final CountDownLatch release = new CountDownLatch(1); + private final AtomicReference copyThread = new AtomicReference<>(); + + @Override + public int mkdir(String path, int mode) { + if (path.contains("/rejected/.tmp-")) { + copyThread.set(Thread.currentThread()); + entered.countDown(); + boolean interrupted = false; + while (true) { + try { + release.await(); + break; + } catch (InterruptedException ignored) { + interrupted = true; + } + } + if (interrupted) Thread.currentThread().interrupt(); + } + return super.mkdir(path, mode); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java index 32473cd39..e978ad157 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java @@ -220,6 +220,37 @@ public void testHeaderShapeMatchesTheDocumentedLayout() throws Exception { }); } + @Test + public void testLiveFrameLookupCacheRetainsBoundsAndCorruptionChecks() throws Exception { + TestUtils.assertMemoryLeak(() -> { + long payload = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT); + try (MmapSegment segment = MmapSegment.createInMemory(10L, 4096L)) { + java.lang.reflect.Method payloadLength = MmapSegment.class + .getDeclaredMethod("liveFramePayloadLength", long.class); + payloadLength.setAccessible(true); + for (int i = 0; i < 4; i++) { + assertTrue(segment.tryAppend(payload, i + 1) >= 0); + } + + assertEquals(1, ((Integer) payloadLength.invoke(segment, 10L)).intValue()); + assertEquals(3, ((Integer) payloadLength.invoke(segment, 12L)).intValue()); + assertEquals(3, ((Integer) payloadLength.invoke(segment, 12L)).intValue()); + assertEquals(2, ((Integer) payloadLength.invoke(segment, 11L)).intValue()); + assertEquals(4, ((Integer) payloadLength.invoke(segment, 13L)).intValue()); + assertEquals(-1, ((Integer) payloadLength.invoke(segment, 9L)).intValue()); + assertEquals(-1, ((Integer) payloadLength.invoke(segment, 14L)).intValue()); + + // A cached offset is still validated before use. + long fourthOffset = MmapSegment.HEADER_SIZE + + 3L * MmapSegment.FRAME_HEADER_SIZE + 1L + 2L + 3L; + Unsafe.getUnsafe().putInt(segment.address() + fourthOffset + 4, Integer.MAX_VALUE); + assertEquals(-1, ((Integer) payloadLength.invoke(segment, 13L)).intValue()); + } finally { + Unsafe.free(payload, 16, MemoryTag.NATIVE_DEFAULT); + } + }); + } + @Test public void testSegmentWithAnUnknownVersionIsRefused() throws Exception { // The version byte survives the barrier removal precisely so a real future diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java new file mode 100644 index 000000000..0cec14ad7 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java @@ -0,0 +1,219 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegment; +import io.questdb.client.cutlass.qwp.client.sf.cursor.PersistedSymbolDict; +import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Comparator; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.fail; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertNotNull; + +public class RejectedMiniSlotArchiveTest { + private Path root; + + @Before + public void setUp() throws Exception { + root = Files.createTempDirectory("qdb-rejected-mini-slot-"); + } + + @After + public void tearDown() throws Exception { + if (root != null) { + try (java.util.stream.Stream paths = Files.walk(root)) { + paths.sorted(Comparator.reverseOrder()).forEach(p -> { + try { Files.deleteIfExists(p); } catch (Exception ignored) { } + }); + } + } + } + + @Test + public void testEpochSurvivesReopenAndRejectsCorruption() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String slot = Files.createDirectory(root.resolve("slot")).toString(); + String first = SlotEpoch.openOrCreate(ff, slot); + assertEquals(first, SlotEpoch.openOrCreate(ff, slot)); + assertEquals(first, SlotEpoch.read(ff, slot + '/' + SlotEpoch.FILE_NAME)); + String reset = SlotEpoch.openOrCreate(ff, slot, true); + assertFalse(first.equals(reset)); + assertEquals(reset, SlotEpoch.openOrCreate(ff, slot, false)); + } + + @Test + public void testCleanEngineCloseEndsEpochLifecycle() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String slot = Files.createDirectory(root.resolve("clean-close-slot")).toString(); + try (CursorSendEngine engine = new CursorSendEngine(slot, 4096)) { + SlotEpoch.openOrCreate(ff, slot, engine.freshFsnNamespace()); + assertTrue(ff.exists(slot + '/' + SlotEpoch.FILE_NAME)); + } + assertFalse(ff.exists(slot + '/' + SlotEpoch.FILE_NAME)); + } + + @Test + public void testRecoveryFindsOnlyCurrentEpochAndScopedTempCleanup() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("recovery-source")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + appendDeltaFrame(engine, 0, true, "zero"); + SenderError error = rejection(0).withRejectionSpan(0, 0); + RejectedMiniSlotArchive.Result result = RejectedMiniSlotArchive.preserve( + ff, engine, null, source, "slot-recovery", epoch, error); + SenderError recovered = RejectedMiniSlotArchive.findOverlapping( + ff, source, "slot-recovery", epoch, 0, 0); + assertNotNull(recovered); + assertEquals(result.path, recovered.getRejectedPath()); + assertEquals(0, recovered.getRejectedFsn()); + assertEquals(null, RejectedMiniSlotArchive.findOverlapping( + ff, source, "slot-recovery", java.util.UUID.randomUUID().toString(), 0, 0)); + + Path rejected = Path.of(source, "rejected"); + Path ours = Files.createDirectory(rejected.resolve( + ".tmp-slot-recovery-" + epoch + "-fsn-0-0-dead")); + Files.createFile(ours.resolve(RejectedMiniSlotArchive.SEGMENT_FILE_NAME)); + Path other = Files.createDirectory(rejected.resolve( + ".tmp-other-" + epoch + "-fsn-0-0-live")); + RejectedMiniSlotArchive.cleanupTemporaryDirectories( + ff, source, "slot-recovery", epoch); + assertFalse(Files.exists(ours)); + assertTrue(Files.exists(other)); + } + } + + @Test + public void testPreservedSubsetReopensWithDictionarySupersetAndWorkingCopyKeepsArchive() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("source")).toString(); + String dictDir = Files.createDirectory(root.resolve("dict")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096); + PersistedSymbolDict dictionary = PersistedSymbolDict.openClean(dictDir)) { + dictionary.appendSymbol("zero"); + dictionary.appendSymbol("one"); + dictionary.appendSymbol("unused-superset-entry"); + appendDeltaFrame(engine, 0, true, "zero"); + appendDeltaFrame(engine, 1, true, "one"); + String serverMessage = "column mismatch ".repeat(2048); + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, serverMessage, 1, + 0, 1, "tab", 42).withRejectionSpan(0, 1); + RejectedMiniSlotArchive.Result result = RejectedMiniSlotArchive.preserve( + ff, engine, dictionary, source, "slot-0", epoch, error); + assertFalse(result.reused); + assertTrue(result.bytesWritten > 0); + + RejectedMiniSlotArchive.Metadata metadata = RejectedMiniSlotArchive.readMetadata(ff, result.path); + assertEquals(0, metadata.fromFsn); + assertEquals(1, metadata.toFsn); + assertEquals(serverMessage, metadata.message); + + try (MmapSegment segment = MmapSegment.openExisting(result.path + '/' + + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)) { + long second = MmapSegment.HEADER_SIZE; + second += MmapSegment.FRAME_HEADER_SIZE + + Unsafe.getUnsafe().getInt(segment.address() + second + 4); + long payload = segment.address() + second + MmapSegment.FRAME_HEADER_SIZE; + assertEquals(0, Unsafe.getUnsafe().getByte(payload + QwpConstants.HEADER_OFFSET_FLAGS) + & QwpConstants.FLAG_DEFER_COMMIT); + } + + RejectedMiniSlotArchive.Result reused = RejectedMiniSlotArchive.preserve( + ff, engine, dictionary, source, "slot-0", epoch, error); + assertTrue(reused.reused); + + String working = root.resolve("working").toString(); + RejectedMiniSlotArchive.copyToWorkingDirectory(ff, result.path, working); + assertTrue(ff.exists(result.path + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)); + try (CursorSendEngine replay = new CursorSendEngine(working, 4096)) { + assertEquals(1, replay.publishedFsn()); + assertEquals(-1, replay.ackedFsn()); + } + assertTrue(ff.exists(result.path + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)); + } + } + + @Test + public void testSynchronousPreserverReturnsCompleteCopy() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("sync-source")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + appendDeltaFrame(engine, 0, true, "zero"); + SchemaPreserver preserver = new SchemaPreserver(ff, source, "slot-sync", epoch); + RejectedMiniSlotArchive.Result result = preserver.preserve(engine, + rejection(0), new byte[]{4, 'z', 'e', 'r', 'o'}, 1); + assertNotNull(RejectedMiniSlotArchive.readMetadata(ff, result.path)); + try (PersistedSymbolDict dictionary = PersistedSymbolDict.open(ff, result.path)) { + assertNotNull(dictionary); + assertEquals(1, dictionary.size()); + } + } + } + + @Test + public void testSynchronousPreserverPropagatesFailure() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("sync-failure")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + SchemaPreserver preserver = new SchemaPreserver(ff, source, "slot-failure", epoch); + try { + preserver.preserve(engine, rejection(0), null, 0); + fail("missing source frame must fail preservation"); + } catch (io.questdb.client.cutlass.qwp.client.sf.cursor.SfOperationalException expected) { + assertEquals(-1, engine.ackedFsn()); + } + } + } + + private static SenderError rejection(long fsn) { + return new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "column mismatch", fsn, + fsn, fsn, "tab", 42); + } + + private static void appendDeltaFrame(CursorSendEngine engine, int deltaStart, + boolean deferred, String symbol) { + byte[] utf8 = symbol.getBytes(java.nio.charset.StandardCharsets.UTF_8); + int size = QwpConstants.HEADER_SIZE + 2 + 1 + utf8.length; + long buf = Unsafe.malloc(size, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(buf, size, (byte) 0); + Unsafe.getUnsafe().putInt(buf, QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(buf + QwpConstants.HEADER_OFFSET_FLAGS, + (byte) (QwpConstants.FLAG_DELTA_SYMBOL_DICT + | (deferred ? QwpConstants.FLAG_DEFER_COMMIT : 0))); + long p = buf + QwpConstants.HEADER_SIZE; + Unsafe.getUnsafe().putByte(p, (byte) deltaStart); + Unsafe.getUnsafe().putByte(p + 1, (byte) 1); + Unsafe.getUnsafe().putByte(p + 2, (byte) utf8.length); + Unsafe.getUnsafe().copyMemory(utf8, Unsafe.BYTE_OFFSET, null, p + 3, utf8.length); + engine.appendBlocking(buf, size); + } finally { + Unsafe.free(buf, size, MemoryTag.NATIVE_DEFAULT); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java new file mode 100644 index 000000000..c117ac349 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java @@ -0,0 +1,174 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.LineSenderServerException; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import org.junit.Test; +import org.junit.Rule; +import org.junit.rules.TemporaryFolder; + +import static org.junit.Assert.*; + +public class SchemaRejectionStateTest { + @Rule + public final TemporaryFolder temp = new TemporaryFolder(); + + @Test + public void testRecoveredGroupRetiresThroughCloserRegardlessOfNewLeaseMode() throws Exception { + String path = temp.newFolder("recovered-group").getAbsolutePath(); + try (CursorSendEngine original = new CursorSendEngine(path, 4096)) { + append(original, true); + append(original, true); + append(original, false); + append(original, false); + } + try (CursorSendEngine recovered = new CursorSendEngine(path, 4096)) { + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(recovered); + state.beginLease(1, 4, false); + append(recovered, false); + assertTrue(state.reject(1, 0, error(1))); + assertEquals(2, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(1)); + } + } + + @Test + public void testRecoveredOpenTailCannotUseNewProducerCloser() throws Exception { + String path = temp.newFolder("recovered-tail").getAbsolutePath(); + try (CursorSendEngine original = new CursorSendEngine(path, 4096)) { + append(original, true); + append(original, true); + } + try (CursorSendEngine recovered = new CursorSendEngine(path, 4096)) { + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(recovered); + state.beginLease(1, 2, false); + append(recovered, false); + assertTrue(state.reject(0, 0, error(0))); + assertEquals(1, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(1)); + } + } + + private static void append(CursorSendEngine engine, boolean deferred) { + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, + (byte) (deferred ? QwpConstants.FLAG_DEFER_COMMIT : 0)); + engine.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + } + + @Test + public void testReturnCapturesRejectionArrivingAfterFinalProducerCall() { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(7, 10, true); + assertTrue(state.reject(12, 10, error(12))); + LineSenderServerException failure = state.endLease(7, 15); + assertNotNull(failure); + assertEquals(15, failure.getServerError().getToFsn()); + assertNull("duplicate return must not throw again", state.endLease(7, 15)); + } + + @Test + public void testOpenTransactionSealsOnFirstProducerObservation() { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(7, 10, true); + assertTrue(state.reject(12, 10, error(12))); + assertTrue(state.hasOwnedFailure(7)); + assertNull(state.sealedRange()); + + LineSenderServerException first = state.ownedFailure(7, 15); + assertNotNull(first); + assertSame(first, state.ownedFailure(7, 99)); + assertEquals(10, first.getServerError().getFromFsn()); + assertEquals(15, first.getServerError().getToFsn()); + assertEquals(15, state.sealedRange().lastFsn); + } + + @Test + public void testOrdinaryRangeIsImmediatelySealedAtRejectedFrame() { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(2, 3, false); + assertTrue(state.reject(5, 4, error(5))); + assertEquals(5, state.sealedRange().lastFsn); + assertEquals(5, state.ownedFailure(2, 8).getServerError().getToFsn()); + } + + @Test + public void testRecoveredFrameWithoutLeaseStillRetiresAndReports() { + SchemaRejectionState state = new SchemaRejectionState(); + assertTrue(state.reject(4, 2, error(4))); + SchemaRejectionState.Range range = state.sealedRange(); + assertNotNull(range); + assertEquals(2, range.firstFsn); + assertEquals(4, range.lastFsn); + assertEquals(4, range.error.getRejectedFsn()); + } + + @Test + public void testReturnedUnackedTransactionalLeaseRetainsOwnershipAndFinalEnd() { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(7, 10, true); + state.endLease(7, 15); + + assertTrue(state.reject(12, 10, error(12))); + assertEquals(10, state.stopFsn()); + assertEquals(15, state.sealedRange().lastFsn); + LineSenderServerException failure = state.ownedFailure(7, 99); + assertNotNull(failure); + assertEquals(10, failure.getServerError().getFromFsn()); + assertEquals(15, failure.getServerError().getToFsn()); + } + + @Test + public void testClosedTransactionDoesNotRetireNextTransactionInSameLease() { + try (CursorSendEngine engine = new CursorSendEngine(null, 4096)) { + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + // Two closed transactions share one sender lease. + for (int i = 0; i < 4; i++) { + Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, (byte) ((i & 1) == 0 ? QwpConstants.FLAG_DEFER_COMMIT : 0)); + engine.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1, 0, true); + assertTrue(state.reject(0, 0, error(0))); + assertEquals(1, state.sealedRange().lastFsn); + assertEquals(1, state.ownedFailure(1, 3).getServerError().getToFsn()); + state.completeRetirement(1); + // A second rejected transaction has its own notification span; + // the handle keeps the first immutable exception. + assertTrue(state.reject(2, 2, error(2))); + assertEquals(2, state.sealedRange().error.getRejectedFsn()); + assertEquals(3, state.sealedRange().lastFsn); + assertEquals(1, state.ownedFailure(1, 3).getServerError().getToFsn()); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + } + } + + private static SenderError error(long fsn) { + return new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 7, "mismatch", 1, + fsn, fsn, "tab", System.nanoTime()); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SenderErrorDispatcherTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SenderErrorDispatcherTest.java index 69725fb7e..e27e0dfd6 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SenderErrorDispatcherTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SenderErrorDispatcherTest.java @@ -37,6 +37,45 @@ public class SenderErrorDispatcherTest { + @Test + public void testSchemaCapacityIncludesExecutingCallbackAndSurvivesOverflow() throws Exception { + CountDownLatch entered = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch delivered = new CountDownLatch(256); + AtomicInteger schemaCalls = new AtomicInteger(); + try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(error -> { + if (error.getAppliedPolicy() == SenderError.Policy.REJECT_AND_CONTINUE) { + if (schemaCalls.getAndIncrement() == 0) { + entered.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + delivered.countDown(); + } + }, 2)) { + try { + Assert.assertTrue(dispatcher.tryOfferSchema(buildError(0).withRejectionSpan(0, 0))); + Assert.assertTrue(entered.await(5, TimeUnit.SECONDS)); + for (int i = 1; i < 256; i++) { + Assert.assertTrue(dispatcher.tryOfferSchema(buildError(i).withRejectionSpan(i, i))); + } + Assert.assertEquals(256, dispatcher.getPendingSchemaNotifications()); + Assert.assertFalse(dispatcher.tryOfferSchema(buildError(256).withRejectionSpan(256, 256))); + for (int i = 0; i < 100; i++) { + dispatcher.offer(buildError(i)); + } + Assert.assertEquals(256, dispatcher.getPendingSchemaNotifications()); + } finally { + release.countDown(); + } + Assert.assertTrue(delivered.await(5, TimeUnit.SECONDS)); + Assert.assertEquals(256, schemaCalls.get()); + } + } + @Test public void testCloseDrainsRemainingEntries() { // After close(), entries already in the queue should still be diff --git a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java new file mode 100644 index 000000000..86bffdb96 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java @@ -0,0 +1,350 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.test.impl; + +import io.questdb.client.QuestDB; +import io.questdb.client.LineSenderServerException; +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.WebSocketResponse; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import io.questdb.client.test.cutlass.qwp.client.QwpWireTestUtils; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +public class SchemaRejectionPoolTest { + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test + public void testRecoveredPreservedOrphanReportsAsynchronouslyBeforeRetirement() throws Exception { + String base = temp.newFolder("recovered").getAbsolutePath(); + String slot = Files.createDirectory(java.nio.file.Paths.get(base, "saved")).toString(); + String archive; + try (CursorSendEngine engine = new CursorSendEngine(slot, 1 << 20)) { + String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, slot, engine.freshFsnNamespace()); + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, QwpConstants.FLAG_DEFER_COMMIT); + engine.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "schema rejected", 0, 0, 0, null, 1); + archive = RejectedMiniSlotArchive.preserve(FilesFacade.INSTANCE, engine, null, + slot, "saved", epoch, error).path; + } + CountDownLatch reported = new CountDownLatch(1); + AtomicReference error = new AtomicReference<>(); + AtomicReference callbackThread = new AtomicReference<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + throw new AssertionError("orphan frames must be retired without sending"); + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.builder("ws::addr=localhost:" + server.getPort() + ";sf_dir=" + base + ";") + .senderId("saved").errorHandler(e -> { + error.set(e); + callbackThread.set(Thread.currentThread()); + reported.countDown(); + }).build()) { + Assert.assertTrue(reported.await(5, TimeUnit.SECONDS)); + Assert.assertNotSame(Thread.currentThread(), callbackThread.get()); + Assert.assertEquals(archive, error.get().getRejectedPath()); + Assert.assertEquals(0, sender.getAckedFsn()); + } + } + } + + @Test + public void testLazyPoolValidatesDestinationBeforeFirstBorrow() throws Exception { + String file = temp.newFile("not-a-directory").getAbsolutePath(); + try (QuestDB ignored = QuestDB.builder().fromConfig("ws::addr=localhost:1;") + .senderPoolMin(0).senderPoolMax(1).queryPoolMin(0).queryPoolMax(1) + .dlqDirectory(file).build()) { + Assert.fail("build must reject a destination that cannot hold archives"); + } catch (io.questdb.client.cutlass.line.LineSenderException expected) { + Assert.assertTrue(expected.getMessage().contains("schema preservation destination")); + } + } + + @Test + public void testDiskQueuePreservesBeforeRetirementAndContinuation() throws Exception { + CountDownLatch reported = new CountDownLatch(1); + AtomicReference rejection = new AtomicReference<>(); + AtomicReference firstConnection = new AtomicReference<>(); + Map sequences = new ConcurrentHashMap<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = sequences.merge(client, 1L, Long::sum) - 1; + try { + if (firstConnection.compareAndSet(null, client)) { + client.sendBinary(QwpWireTestUtils.buildNack(sequence, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + } else if (firstConnection.get() != client) { + client.sendBinary(QwpWireTestUtils.buildAck(sequence)); + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + String sfDir = temp.newFolder("sf").getAbsolutePath(); + try (QuestDB db = QuestDB.builder() + .fromConfig("ws::addr=localhost:" + server.getPort() + ";sf_dir=" + sfDir + + ";close_flush_timeout_millis=0;") + .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) + .errorHandler(error -> { rejection.set(error); reported.countDown(); }).build()) { + Sender failed = db.borrowSender(); + failed.table("bad").stringColumn("value", "wrong").atNow(); + long rejectedFsn = failed.flushAndGetSequence(); + Assert.assertTrue(reported.await(10, TimeUnit.SECONDS)); + try { + failed.awaitAckedFsn(rejectedFsn, 0); + Assert.fail("owning handle must fail after preserved rejection"); + } catch (LineSenderServerException expected) { + // Mark the lease-local failure observed so close only returns the slot. + } + failed.close(); + SenderError error = rejection.get(); + Assert.assertEquals(SenderError.Policy.REJECT_AND_CONTINUE, error.getAppliedPolicy()); + Assert.assertNotNull(error.getRejectedPath()); + Assert.assertTrue(Files.isDirectory(java.nio.file.Paths.get(error.getRejectedPath()))); + try (Sender healthy = db.borrowSender()) { + healthy.table("good").longColumn("value", 42).atNow(); + long target = healthy.flushAndGetSequence(); + Assert.assertTrue(healthy.awaitAckedFsn(target, 10_000)); + } + } + } + } + + @Test + public void testObservedFailedHandleCloseReturnsSlot() throws Exception { + CountDownLatch rejected = new CountDownLatch(1); + AtomicReference firstConnection = new AtomicReference<>(); + Map sequences = new ConcurrentHashMap<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = sequences.merge(client, 1L, Long::sum) - 1; + try { + if (firstConnection.compareAndSet(null, client)) { + client.sendBinary(QwpWireTestUtils.buildNack(sequence, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + rejected.countDown(); + } else if (firstConnection.get() != client) { + client.sendBinary(QwpWireTestUtils.buildAck(sequence)); + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = newPool(server)) { + Sender failed = db.borrowSender(); + failed.table("bad").stringColumn("value", "wrong").atNow(); + long rejectedFsn = failed.flushAndGetSequence(); + Assert.assertTrue(rejected.await(5, TimeUnit.SECONDS)); + try { + failed.awaitAckedFsn(rejectedFsn, 10_000); + Assert.fail("owning handle must observe schema rejection"); + } catch (LineSenderServerException expected) { + Assert.assertEquals(rejectedFsn, expected.getServerError().getRejectedFsn()); + } + failed.close(); + + try (Sender healthy = db.borrowSender()) { + healthy.table("good").longColumn("value", 42).atNow(); + long target = healthy.flushAndGetSequence(); + Assert.assertTrue("returned slot must remain usable", healthy.awaitAckedFsn(target, 10_000)); + } + } + } + } + + @Test + public void testTransactionalDeferredRejectionRetiresThroughPublishedTail() throws Exception { + CountDownLatch firstReceived = new CountDownLatch(1); + CountDownLatch rejectNow = new CountDownLatch(1); + AtomicReference firstConnection = new AtomicReference<>(); + Map sequences = new ConcurrentHashMap<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = sequences.merge(client, 1L, Long::sum) - 1; + try { + if (firstConnection.compareAndSet(null, client)) { + firstReceived.countDown(); + if (!rejectNow.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("test did not release NACK"); + } + client.sendBinary(QwpWireTestUtils.buildNack(sequence, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + } else if (firstConnection.get() != client) { + client.sendBinary(QwpWireTestUtils.buildAck(sequence)); + } + } catch (IOException | InterruptedException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.builder() + .fromConfig("ws::addr=localhost:" + server.getPort() + + ";auto_flush_rows=1;auto_flush_bytes=off;transaction=on;close_flush_timeout_millis=0;") + .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) + .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE).dlqEnabled(false).build()) { + Sender failed = db.borrowSender(); + failed.table("bad").longColumn("value", 1).atNow(); + Assert.assertTrue(firstReceived.await(5, TimeUnit.SECONDS)); + failed.table("bad").longColumn("value", 2).atNow(); + rejectNow.countDown(); + try { + failed.awaitAckedFsn(1, 10_000); + Assert.fail("transaction owner must fail"); + } catch (LineSenderServerException expected) { + Assert.assertEquals(0, expected.getServerError().getFromFsn()); + Assert.assertEquals(1, expected.getServerError().getToFsn()); + } + failed.close(); + try (Sender healthy = db.borrowSender()) { + healthy.table("good").longColumn("value", 3).atNow(); + } + } finally { + rejectNow.countDown(); + } + } + } + + @Test + public void testMalformedSchemaSequenceFailsClosed() throws Exception { + CountDownLatch rejected = new CountDownLatch(1); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + try { + client.sendBinary(QwpWireTestUtils.buildNack(99, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + rejected.countDown(); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + Sender sender = Sender.builder("ws::addr=localhost:" + server.getPort() + + ";close_flush_timeout_millis=0;") + .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE) + .dlqEnabled(false) + .build(); + try { + sender.table("bad").longColumn("value", 1).atNow(); + long target = sender.flushAndGetSequence(); + Assert.assertTrue(rejected.await(5, TimeUnit.SECONDS)); + try { + sender.awaitAckedFsn(target, 10_000); + Assert.fail("out-of-range NACK must fail closed"); + } catch (LineSenderServerException expected) { + Assert.assertEquals(SenderError.Policy.TERMINAL, + expected.getServerError().getAppliedPolicy()); + } + } finally { + try { + sender.close(); + } catch (LineSenderServerException ignored) { + } + } + } + } + + @Test + public void testRejectionAfterReturnDoesNotFailNextBorrow() throws Exception { + CountDownLatch firstReceived = new CountDownLatch(1); + CountDownLatch rejectNow = new CountDownLatch(1); + CountDownLatch reported = new CountDownLatch(1); + AtomicReference rejection = new AtomicReference<>(); + AtomicReference rejectedConnection = new AtomicReference<>(); + Map sequences = new ConcurrentHashMap<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = sequences.merge(client, 1L, Long::sum) - 1; + try { + if (rejectedConnection.compareAndSet(null, client)) { + firstReceived.countDown(); + if (!rejectNow.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("test did not release NACK"); + } + client.sendBinary(QwpWireTestUtils.buildNack(sequence, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + } else if (rejectedConnection.get() != client) { + client.sendBinary(QwpWireTestUtils.buildAck(sequence)); + } + } catch (IOException | InterruptedException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.builder() + .fromConfig("ws::addr=localhost:" + server.getPort() + ";close_flush_timeout_millis=0;") + .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) + .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE).dlqEnabled(false) + .errorHandler(error -> { rejection.set(error); reported.countDown(); }).build()) { + try (Sender a = db.borrowSender()) { + a.table("bad").stringColumn("value", "wrong").atNow(); + a.flush(); + Assert.assertTrue(firstReceived.await(5, TimeUnit.SECONDS)); + } + try (Sender b = db.borrowSender()) { + b.table("good").longColumn("value", 42).atNow(); + long target = b.flushAndGetSequence(); + rejectNow.countDown(); + Assert.assertTrue("later borrow must drain past old rejection", b.awaitAckedFsn(target, 10_000)); + Assert.assertTrue(reported.await(5, TimeUnit.SECONDS)); + Assert.assertEquals(SenderError.Policy.REJECT_AND_CONTINUE, rejection.get().getAppliedPolicy()); + Assert.assertEquals(0, rejection.get().getRejectedFsn()); + } + } finally { + rejectNow.countDown(); + } + } + } + + private static QuestDB newPool(TestWebSocketServer server) { + return QuestDB.builder() + .fromConfig("ws::addr=localhost:" + server.getPort() + ";close_flush_timeout_millis=0;") + .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) + .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE).dlqEnabled(false) + .build(); + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java index 24e60e49b..83ded197a 100644 --- a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java @@ -2615,8 +2615,8 @@ public void testConcurrentBorrowReturnStress() throws Exception { @Test public void testConcurrentFirstBorrowsWithMinZeroRaceOnSfDir() throws Exception { // C2 regression: senderPoolMin(0) means no single-threaded pre-warm, - // so the shared parent sf_dir is NOT created at construction (the - // constructor probe only parses the config). The first concurrent + // so no slot is created at construction. Remove the empty directory + // left by the eager DLQ destination probe to exercise concurrent // borrows then race into build() -> Files.mkdir(sfDir) outside the // pool lock. Pre-fix, the mkdir loser got a non-zero rc (EEXIST) and // its borrow() threw "could not create sf_dir" on a perfectly healthy @@ -2629,9 +2629,10 @@ public void testConcurrentFirstBorrowsWithMinZeroRaceOnSfDir() throws Exception Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); String config = "ws::addr=localhost:" + port + ";sf_dir=" + sfDir + ";"; - // minSize=0 -> no pre-warm -> sf_dir absent until first borrow. + // minSize=0 -> no slots; the destination probe creates only the root. try (SenderPool pool = new SenderPool(config, 0, 4, 10_000, Long.MAX_VALUE, Long.MAX_VALUE)) { - Assert.assertFalse("sf_dir must not exist before the first borrow", + java.nio.file.Files.delete(java.nio.file.Paths.get(sfDir)); + Assert.assertFalse("test must restore the first-directory creation race", Files.exists(sfDir)); final int threads = 4; diff --git a/design/schema-mismatch-terminal-resolution.md b/design/schema-mismatch-terminal-resolution.md new file mode 100644 index 000000000..679677c2a --- /dev/null +++ b/design/schema-mismatch-terminal-resolution.md @@ -0,0 +1,642 @@ +# Schema rejection: report, retire, continue + +Status: Java phases one and two implemented and validated locally. Updated: 2026-09-08. Owner: Jaromir Hamala. +Supersedes the whole-group/journal revision; see Appendix B for what was +dropped and why. + +This amends [`qwp-nack-policy-v2.md`](qwp-nack-policy-v2.md). Today a +schema-invalid batch latches a `TERMINAL` error, stays in the store-and-forward +log, is replayed and rejected again on every restart, and parks the slot. The +new selectable policy is `REJECT_AND_CONTINUE`: fail the handle that owns the data, +report the rejection, retire the affected frames from replay, and keep +independent data moving. Phase one retains `TERMINAL` as the default; phase two +flips the default in the same release as preserved copies. + +## Decisions + +1. Validate the NACK against frames actually sent on this connection. Never + clamp an invalid sequence into a retirement candidate. +2. Compute the rejection span by scanning frame flags on the ring, not from a + tracked scalar. Ordinary flushes retire the deferred prefix through the + rejected frame. Transactional senders retire the whole transaction. +3. Retire in memory using the existing orphan-tail machinery in the send loop: + stop before the sealed span, retain a non-dropping in-memory notification, + wait for lower ACKs, self-acknowledge through the span, recycle, continue. + Callback completion is not a retirement gate. +4. No journal. A crash before retirement replays the frames and the server + can reject them again. Phase one has the crash-reporting exceptions below; + it does not guarantee a callback for every observed NACK across restart. +5. The handle that published the span fails until returned or rebuilt. Its + waits report the rejection. Other handles never see it. +6. Phase two preserves the span's bytes in the existing segment and dictionary + formats. Phase three adds an offline reader. Neither gates phase one. + +Retirement is not acceptance and not proof of rollback. Per-table force +commits, auto-created tables and auto-added columns can survive a rejection. +Application resubmission can duplicate rows. + +| User | Phase-one default: `TERMINAL` | Explicit phase-one `REJECT_AND_CONTINUE` / phase-two default | +|---|---|---| +| Pooled producer | Existing preserve-and-halt behavior; the slot can remain poisoned. | Return and reborrow; the slot progresses. Ordinary unrejected prefix rows are retired too: lost in phase one, copied in phase two when export is enabled. | +| Standalone producer | Rebuild may encounter the same rejection. | Handle state resets on rebuild; unretired frames may re-reject. | +| Transactional producer after a crash | Existing recovery behavior remains unchanged; do not assume every open tail is re-sent. | A crash before preservation/notification can yield orphan retirement, no callback and no preserved bytes. | +| Handler author | Existing bounded best-effort notifications. | Non-dropping while running, but crash/shutdown can lose pending notifications even after retirement. Phase two supplies a ready preserved-copy path. | +| Operator | Must explicitly opt in to retirement without a copy. | Phase two defaults to preservation before retirement; export opt-out accepts payload loss. | + +## Shipping phases + +| Phase | Ships | Boundary | +|---|---|---| +| 1 | NACK validation, span computation, in-memory retirement/notifications, selectable `REJECT_AND_CONTINUE`, failed-handle ownership, recovery drain fix, two counters. | `TERMINAL` remains default. No new on-disk state; process-local queue identity only. Opt-in retirement discards prefix bytes and requires the source. | +| 2 | Span copy in existing segment/dictionary formats, durable slot epoch, build-time destination probe, opt-out and storage counters. | Flip the schema default to `REJECT_AND_CONTINUE` only when this ships; copy is default-on with `sf_dir` and gates retirement. | +| 3 | Offline reader over copied files; JSONL export. | Separate release. | + +Phase one does not fix the pooled poison demo out of the box: its default +`TERMINAL` policy still parks the slot. The demo progresses only when explicitly +configured with `REJECT_AND_CONTINUE`, until phase two changes the default. + +The default flip and preserved-copy support ship together, not in separate +releases. Explicit phase-one users accept that ordinary A/B/C/D split-flush +rejection at C retires A and B as well as C, while D can commit. Phase two does +not recover bytes already discarded by phase one. Memory-only phase-two users +without a destination, and export opt-outs, retain their own source data. + +## Terms + +| Term | Meaning | +|---|---| +| Frame / FSN | One locally published ingest frame and its sequence number. | +| Commit-bearing frame | A frame without `FLAG_DEFER_COMMIT`. Commits it and every deferred frame before it. | +| Group start | The frame after the last commit-bearing frame below the rejected FSN. | +| Rejection span | Inclusive FSN range retired for one rejection. | +| Retired | Locally acknowledged without server acceptance; never sent again. | +| Owning lease | The borrow (or the standalone sender's lifetime) that published the span. | + +## Retirement mechanism + +### Validate the NACK + +`handleServerRejection` currently clamps out-of-range wire sequences so that an +error can still be attributed. Keep that for reporting; never feed a clamped or +pre-send sequence into retirement. A NACK retires data only when its sequence +maps to a data frame sent on this connection at or above the replay start. + +### Compute the span + +Add a read-only engine API for live per-FSN QWP header flags; recovery-only +`RecoveredFrameAnalysis` is not such an API. Pin or otherwise protect the read +against trimming and validate the frame. Scan forward from a captured safe floor +of `ackedFsn + 1` to just before the rejected FSN, retaining the latest +commit-bearing boundary. The group starts just after that boundary, or at the +floor if none exists. A cold lookup cache makes this scan and archive copying +linear without indexing frames during healthy publication. The latest published commit boundary is the wrong scalar: a NACK can +arrive after a later closer was published. + +Ordinary senders retire `[group start, rejected FSN]`. Valid deferred +predecessors remain in the ring after server rollback, but this policy deliberately +retires them along with the rejected frame. Published successors replay and may +commit as a partial flush. The split-flush javadoc's partial-publication caveat +does not itself authorize that disposal; it is an explicit policy trade-off, +opt-in until preserved copies ship. + +Transactional senders retire from group start through their published closer, +or through their published open tail if no closer exists. Capture transaction +mode with the owning generation; never use a later borrower's settings. A closed +span ends at the first commit-bearing frame at or after the rejection, not the +lease's last published frame: one lease can contain several transactions. It +can retire without waiting for lease return. An unclosed transaction follows +this producer-side sealing protocol: + +1. On rejection, fail the owning generation and install a replay stop at the + span's start. Do not report a provisional end as final. +2. The next producer call that observes the failure, or lease return, excludes + further publication. Sender methods already require one producer thread. +3. Read `publishedFsn` and look for the first closer after the rejected frame. + Use that closer, or the published tip if still open. This snapshot is the + sealing point; the first exception carries immutable final bounds. +4. On return, skip flush and discard staged rows, then advance the generation + and make the slot available to the next borrower. +5. Queue the final-span notification and let retirement proceed once lower ACKs + and, in phase two, the preserved copy are ready. + +A held or leaked failed lease that neither observes its failure nor returns +can stall its unclosed span indefinitely. Either producer observation or return +seals it; callback completion does not. A later borrower's +closer cannot commit its rows because the replay stop and retirement precede +sending beyond the sealed span. Validate the publication race protocol under +the phase-one gate in Open decisions. + +Recovery does not persist the previous producer's transaction mode. Treat a +rejected recovered group conservatively as transactional: retire through its +first recovered commit-bearing frame, or its recovered open tail. Never use a +new producer's closer. This also retires the tail of an ordinary split flush +after restart; phase two preserves those rows in the copy. + +### Retire in the send loop + +`CursorWebSocketSendLoop` already stops at a recovered orphan tail, and +`retireRecoveredOrphanTailIfReady` self-acknowledges it once every lower frame +is acknowledged, then recycles the connection to re-anchor the arithmetic +wire-sequence-to-FSN mapping. Generalize that range so it can be set live by +the I/O thread on a NACK. Retain the final-span notification before advancing +the watermark, but do not wait for its callback. Phase two additionally waits +for the preserved copy to be durably published. + +I/O-side order on a NACK (return-side sealing and callback execution are separate): + +1. Validate. Compute the span. Latch the error on the owning lease. +2. Disconnect and recycle immediately; do not wait for lease return or + notification capacity. +3. Replay from the watermark. Independent frames below the span send and + acknowledge normally. Stop before the span. +4. Once the span is sealed and lower frames have their configured ACKs, retain + its notification in the FIFO and signal the dispatcher. If the FIFO is full, + stop here until capacity is available. Then advance the watermark through + the span without waiting for callback completion. In phase two, the copy + must be durable before notification enqueue and watermark advancement. +5. Recycle, continue with frames above the span. + +A second schema rejection below an already pending retirement range cannot be +merged safely by this implementation. Log the second rejected FSN and fall back +to `TERMINAL`, retaining source bytes. This can occur when a predecessor is +re-rejected while replaying for durable ACKs. Invalid closer scans or failed +skipped-range dictionary reconstruction also fail closed instead of reconnecting +forever. + +No sparse ACK map, no second watermark file, no change to the watermark or +segment encodings. Each connection still sends one contiguous range. The general path uses two +recycles, after NACK and after retirement; each may ship dictionary catch-up. +Setup can sometimes retire before any wire sequence is consumed, but do not +assume that shortcut when estimating rejection cost. + +Retired frames may be the only carriers of dictionary deltas referenced by +successors. Disk-backed catch-up must use the persisted dictionary covering +those deltas, not only replayed frames. Memory mode uses the live dictionary +mirror/snapshot; verify it covers skipped, possibly unsent transactional frames +before reclaiming them. Full-dictionary frames remain self-sufficient. Neither +mode may lose symbols because their carrier frame was retired. This is new live +retirement behavior: recovered orphan tails have no successors requiring those +skipped deltas. Spike this first in memory and disk modes before committing to +the phase-one schedule; existing orphan tests are not sufficient evidence. + +### Crash behaviour + +Phase one does not guarantee at-least-once callbacks across restart. A pending +notification is memory-only and retirement does not wait for invocation: + +- Before durable retirement, a closed span may replay and re-reject if the + schema is unchanged, producing another callback opportunity. +- Without a closer, recovery may retire the open tail without sending it: + zero callbacks and no preserved bytes are possible. This is not presented as + a reporting guarantee equivalent to repeated `TERMINAL` rejection. +- After durable retirement but before callback invocation, a crash loses the + notification and there is no replay to reconstruct it. +- A changed schema may allow replay to succeed without recreating the original + rejection. Before watermark durability, duplicate reports remain possible. + +Phase two preserves evidence once its directory is durably published; recovery +can report from that copy. A crash before copy publication retains the open-tail +reporting gap. Memory-only queues cannot reconstruct data lost with the process. +The default phase-one `TERMINAL` path is unchanged; these are the guarantees of +the explicit retirement policy, not reasons to claim unconditional delivery. + +## Ownership and handle contract + +A rejection belongs to the lease whose published FSN range contains the +rejected FSN. Each slot records the lease generation and its first published +FSN at borrow; failed/observed state and the return-side end snapshot above +are also required. Use the highest already published FSN plus one as the start. Row-level calls in +`QwpWebSocketSender.checkConnectionError` and the `PooledSender` wrapper both +check it, since row calls poll the delegate directly. Empty borrows retain no +lease record. Publishing leases remain until resolved progress passes them; +active-lease lookup is constant time, while historical rejection attribution +scans retained leases. A long ACK stall can therefore retain many small records. + +An owned rejection fails the handle. Every subsequent publish, wait and drain +on that handle throws the same `LineSenderServerException` until the handle is +returned. A failed pooled lease needs a distinct `PooledSender.close()` path: +skip `flush()`, discard its staged rows, seal any pending span, and give the slot +back; only then throw the owned error if not already observed. Do not route this +known lease-local rejection through `discardBroken`. That path remains for real +sender/storage/cleanup failures. A stale or repeated close is a no-op. A healthy +close racing a newly owned rejection must take this same cleanup path rather +than discard the slot simply because its flush observed that rejection. + +A standalone sender is one lifetime lease. Its failed close seals/discards local +work and releases resources but does not wait for rejection retirement. Skip +`drainOnClose` for this failure. Rebuild clears public handle state; if retirement +was not durable, replay and another rejection are allowed. Close is not a promise +that the old rejection can never recur. Preservation runs on the I/O thread. +Close waits for that thread within its +existing shutdown budget. If disk I/O outlasts the budget, close reports the +shutdown failure and the existing I/O-thread cleanup fallback retains the +engine and slot lock until the thread exits. Immediate rebuild can therefore +still encounter lock contention. There is no separate preservation worker or +preserver-specific deferred cleanup. + +Callback completion no longer participates in retirement, so handler-initiated +close creates no callback/retirement dependency cycle. Preserve ordinary safe +dispatcher shutdown (never join the current thread); no synthetic callback-return +signal or special retirement-completion protocol is needed. Close/rebuild can +still encounter any frame whose retirement was not durable. + +Waits on a failed handle throw even for FSNs the server accepted. That is +deliberate: a wait returning true for a retired FSN would be a false delivery +confirmation, which is worse than a spurious throw. The exception carries the +span so the caller can tell which batches are actually affected. + +Healthy handles keep today's slot-wide wait and drain semantics. Retirement +advances the watermark, so a healthy borrower's drain completes past another +borrower's retired span rather than hanging. Rejections from earlier borrows +never fail a later handle. Consequently a healthy B waiting on A's retired FSN +can return true: for cross-borrow targets this is resolved progress, not delivery +confirmation. Acceptance conclusions are limited to the caller's own publications +while its handle is healthy. This ownership limit is part of the API contract; +it is why clearing A's owned failure would be different from letting B progress. +Do not describe a slot-wide wait on an arbitrary old FSN as proof of ingestion. + +Pool startup recovery treats a retired span as progress. Today it reports +`RecoveryDrainOutcome.FAILED`, retries, and parks the slot on a failure streak; +that path is what poisons the scan. + +## Reporting + +Retain schema notifications in a per-slot sticky FIFO with capacity **256 +entries**, separate from the ordinary drop-oldest deque. This is a fixed initial +implementation limit, not a new public setting. Count the callback currently in +progress against the 256; release its capacity only when invocation completes, +including when the handler throws. Ordinary overflow cannot evict schema entries. +Keep the same entry across reconnect attempts for an in-progress retirement. +Signal the dispatcher without waiting for user code. Callbacks run outside I/O, +pool and queue locks; log and contain thrown exceptions. + +A stuck handler permits up to 256 retained rejections in that slot, including +its current invocation. The next rejection cannot retire until an entry completes +and frees capacity. Keep that rejection in the live retirement state and leave +its frames unretired; do not drop or overwrite a notification. The I/O loop +remains responsive and independent slots have their own FIFO capacity. A shared +dispatcher can still delay their handlers, eventually filling their FIFOs too. + +Thus slow-handler behavior degrades to a callback-dependent retirement stall +only once the 256-entry allowance is exhausted. Failure to allocate/retain an +entry also leaves the span unretired. This bounds retained notification count, +not arbitrary server-message bytes. Shutdown may abandon the volatile backlog; +no durable delivery is promised. + +Install an effective handler and dispatcher for every reporting path: custom +when supplied, otherwise the default one-line logger. `SenderPool` currently +creates its recovery dispatcher only with a custom handler and SFA enabled; +wire the default path too. Startup schema callbacks run asynchronously, never +on the thread calling `build()`. Recovery does not wait for callback completion while its FIFO has capacity. + +Phase-one identity is slot ID + process-local queue instance + trigger FSN. +Allocate a new instance identity on queue recreation; it is not a cross-restart +deduplication key. Durable slot epoch and legacy initialization are phase-two +work because the preserved directory needs stable identity. No epoch sidecar or +migration is introduced in phase one. + +The error carries: category, policy `REJECT_AND_CONTINUE`, rejected FSN, span +from/to, server message, table when known, and in phase two the preserved file +path once it is published. It states that the span will not be retried, that +server side effects may remain, and that resubmission needs the source. + +## Policy and API + +- Add `SenderError.Policy.REJECT_AND_CONTINUE`. Do not reuse `TERMINAL` + (bytes preserved, sender halted) or `ABANDONED` (no throw, `DATA_LOSS` only). + Update the default handler's log line and policy switches. +- Phase one keeps `TERMINAL` as the schema default; phase two switches to + `REJECT_AND_CONTINUE` with preserved-copy support. Add one builder method, provisionally + `schemaMismatchPolicy(TERMINAL | REJECT_AND_CONTINUE)`, as the escape hatch. + Do not implement the resolver precedence chain or wire `on_schema_error` in + this change; the connect-string key stays a consumed no-op as today. Correct + `Policy` javadoc to describe the actually implemented builder override and + default, removing the non-existent resolver/precedence claim. Document that + the reserved connection-string key does not enable the escape hatch. +- `LineSenderServerException.getServerError()` exposes the rejected FSN via a + new accessor plus the existing `getFromFsn()` and `getToFsn()` span. Include + all three in the message. +- `getAckedFsn()` is documented as the resolved watermark: acceptance or local + retirement. It already advances for recovered orphan tails. No second + accessor. + +Other categories are unchanged. `PARSE_ERROR` and `SECURITY_ERROR` remain +`TERMINAL` and can still poison a persistent slot; enabling them needs their +own attribution review, since malformed input can compromise the flag scan +that computes spans. + +## Server baseline + +The mechanism relies on two server behaviours, both present in the inspected +source (`QwpIngressUpgradeProcessor.handleBinaryMessage`): deferred rows are rolled +back before the NACK is sent, and frames after a NACK are consumed without +processing or reply until disconnect. `QwpSenderE2ETest.testDeferredCommitSchemaMismatchRollsBack` +covers the first. Post-NACK reply handling is whatever the existing retriable +recycle does today; this change adds no new handling. The server does not close +the connection after a NACK: client-side disconnect is required to resume. Older +servers without the unresolved-sequence gate can apply successors on that same +connection; ignoring their replies and replaying can duplicate rows. The minimum +supported-release note must explicitly exclude that behavior. + +The supported baseline for this policy is QuestDB 10.0.0 or later. The 10.0.0 +source tag contains both the unresolved-sequence gate and rollback before NACK. +The rollback E2E test passes against the local 10.0.1-SNAPSHOT checkout at +`496b24d996ea321015c7cfeabcbfc7e563e053e7`, using the current client artifact. +Its harness now needs to observe the owning handle's failure before close; +receiving the callback alone does not consume that failure. A temporary test +adaptation verified the final span and allowed the unchanged database rollback +assertions to run; the server checkout was then restored. +No capability negotiation is added. + +## Phase two: preserve the span + +Introduce a durable slot epoch for directory/deduplication identity. Preserve it +across restart of the same queue and change it when a clean queue restarts FSNs. +Initialize/migrate under the exclusive slot lock before creating copies; existing +identity metadata may be reused only if it provides that lifecycle. This sidecar +is identity metadata, not a phase-one rejection journal. + +Before either retirement or ready-path callback enqueue, copy the span into +`//rejected/--fsn--/` as a mini slot: one +segment file in the existing `MmapSegment` format holding the span's frames in +order, plus a frozen full dictionary snapshot covering the copied frames. The +I/O mirror folds skipped deltas before taking the snapshot, including in memory +mode. It may contain unused later entries; preserving this superset avoids a +decoder and reconstructing historical dictionary versions. Both +formats already carry magic, version and CRC and have Java/Rust fixtures. +Add rejection metadata sufficient for the promised recovery callback (epoch, +trigger/span, category/status and message); existing segment headers alone do +not contain it. Define and validate that metadata file without calling it an +existing SFA field. Final-directory publication covers all constituent files. + +Rewrite the last copied frame's QWP flags to clear `FLAG_DEFER_COMMIT` and +recompute that frame's CRC, so the copy is a closed unit. Without that, the +existing recovery reader classifies a deferred-only copy as an orphan tail and +retires it unsent. Validation must confirm that a copied subset with its +dictionary snapshot replays through the existing reader; this is validation of +an existing format, not a new one. + +Write into a process-unique temporary directory, sync, rename, sync the parent. +A directory at its final name is complete. Under the queue lifecycle lock, +remove unfinished directories matching that exact slot and epoch. Do not remove +other epochs' temporary directories: a shared memory-only destination cannot +prove that another queue has stopped. Such leftovers require operator cleanup. +Reuse an existing final directory +with the same identity on re-rejection after a crash. Copy synchronously on the +I/O thread with bounded buffers, after lower frames are acknowledged and the +rejection span is sealed. No data above the span may be sent until the copy is +complete, so a separate copy worker cannot advance ingestion. A large span, +full dictionary or slow filesystem delays I/O-thread responsiveness, including +keepalives and shutdown; reconnect handles an expired connection. Healthy +publication is unchanged. + +Keep the completed-copy notification until the bounded FIFO admits it, avoiding +repeated archive validation and directory sync while the handler queue is full. +On copy failure keep the frames unretired, log the failure, and wait using the +stop-aware backoff before retrying, with bounded exponential pacing. A failed copy does not latch a +fatal sender error. Before retry, remove unfinished temporary directories for +this exact slot and epoch, so repeated failures do not accumulate partial copies. + +Java settings are builder methods on `Sender` and `QuestDB`: preservation is +enabled by default with `sf_dir`; `dlqEnabled(false)` opts out; +`dlqDirectory(path)` selects `path//rejected/` and gives memory-only +senders a destination. No new connection-string keys are introduced. Probe the +destination at build time, including for a lazy pool with no warm connections. Files are never deleted by the client. TLS does not protect these +bytes at rest. Quarantining a damaged slot also moves its `rejected/` directory; +previously reported paths then change. Use the quarantine path reported by the +`DATA_LOSS` event to locate those copies. A ready path guarantees completeness +when delivered, not a permanent location. Document alongside existing `sf_dir` behaviour and use +restrictive permissions. Each copy may carry the full dictionary; that is the +cost of not decoding and is counted in `dlq_bytes_written_total`; alarm on sustained growth of that +counter and filesystem free space, since no automatic retention bounds it. + +Recovery: a preserved directory whose span overlaps an orphan tail at startup +may mean a crash before callback completion or retirement; it does not prove +that the callback never ran. Dispatch the callback from +the directory's rejection metadata and retire the tail. For an orphan-only slot, +retain that notification and retire locally before attempting a connection. An +unreachable server must not prevent this socket-free cleanup; callback execution +remains asynchronous. + +With export disabled, a persistent schema fault retires indefinitely with one +paced report per span and no circuit breaker. That is the accepted cost of +opting out. + +## Phase three: offline reader + +A CLI or static helper opens a preserved directory with the existing recovery +reader and, once an ingest-frame decoder exists, exports JSONL: one metadata +line, one row per line, a row-count trailer. Resubmission after a schema fix +first copies the preserved directory to a separate working slot using +`RejectedMiniSlotArchive.copyToWorkingDirectory`, then opens that working slot. +Opening the original directly would let normal drain cleanup destroy the evidence. +This reuses +the replay path and needs no decoder. Decoder failures never touch ingestion. + +## Rust and C/C++ + +Rust's SFA path under `questdb-rs/src/ingress/sender/` publishes non-deferred +frames (`qwp_ws_publisher.rs::encode_to_scratch` passes `false` to the defer +argument); `qwp_ws_sfa_queue.rs` is the backing queue. Its spans are singletons. +Do not generalize this to `column_sender/sender.rs`: that file also contains a +direct path with deferred split prefixes. Its `rebase_lease_observation` method +is the lease-rebase citation, not `db.rs`. Implement the same ownership, +failed-handle, callback-independent retirement and phased policy surface. `Drop` releases +the lease but cannot throw; the callback is the only report there. Phase one +adds no on-disk state, so a Java slot after retirement is readable by any +current client, subject to the existing format contract. Phase-two epoch +metadata does not encode replay decisions. Phase two's `rejected/` directory is ignorable by clients that +do not know it. No cross-client gate is needed. + +## Pacing, metrics, healthy path + +- Reconnect after a rejection uses the existing reconnect backoff, reset on + any real ACK. Distinct rejected FSNs do not accumulate their own strike count; + the same-FSN poison detector is unchanged and still escalates a frame that is + rejected without ever being retired. +The Java observation methods are on `QwpWebSocketSender`: + +| Counter | Accessor | Meaning | +|---|---|---| +| `schema_frames_retired_total` | `getSchemaFramesRetired()` | Frames locally retired; not accepted rows. | +| `schema_rejections_total` | `getSchemaRejections()` | Attributed schema NACKs. | +| `dlq_files_written_total` | `getDlqFilesWritten()` | Newly published archive directories; reuse is not counted. | +| `dlq_bytes_written_total` | `getDlqBytesWritten()` | Bytes in newly published archives, including dictionary copies. | +| `dlq_write_failures_total` | `getDlqWriteFailures()` | Preservation failures; source frames remain queued. | + +- Healthy publication stays allocation-free per row and takes no new lock per + frame. Generation boundaries are written at borrow/return; prove race safety for + failure checks and sealing without a new per-frame lock. Existing ingestion benchmarks gate the change. + +## Validation + +- Reject a singleton, a middle deferred frame and a closing frame; span matches + the commit-boundary scan, including when a later closer was already published. +- Ordinary split flush: prefix retired, successors commit as a partial batch, + documented. Transactional: whole transaction retired, staged rows discarded, + no partial commit, including when the closer was published before the NACK. +- Interrupted split flush from a failed lease: its deferred tail is retired; + B's first closer never commits A's rows. +- Lower independent frames acknowledge before self-ack; frames above the span + replay after recycle; dictionary catch-up sequences re-anchor correctly. +- Invalid, pre-send and catch-up NACKs never retire. A second schema NACK + during pending retirement reports `TERMINAL` and retains the stopped range. +- Invalid transaction scans and skipped dictionary reconstruction fail closed; + transient catch-up send failures preserve the existing cap-gap bookkeeping. +- Recovered rejected groups include their original closer or recovered tip, + irrespective of the new producer's mode; a new closer is never included. +- Orphan-only slots retire without connecting, with a preserved metadata report + retained asynchronously first when present. Copy failures retry with pacing + and temporary cleanup, without advancing the watermark or latching fatal. +- Ownership: A's rejection after B borrowed fails neither B nor B's waits. + Failed handle: publish, wait, empty wait and drain throw; first close throws + only if unobserved, skips flush and returns the slot; + repeated close is idempotent; reborrow is healthy. Standalone: rebuild on the + same `sf_dir` clears handle state but may replay unretired frames. +- Slow/throwing handlers do not gate retirement below FIFO capacity; pending entries + survive ordinary inbox overflow, remain distinct and dispatch when unblocked. + Failure to retain an entry prevents that span's retirement without dropping + earlier notifications. With one blocked callback, 256 retained entries permit + progress; the next span waits until callback completion frees capacity. Test + shutdown/crash loss of the volatile backlog. +- Crash before self-ack with closer published: replay, re-reject, callback + opportunity. Crash after durable self-ack but before invocation can lose the + callback. Crash without closer: orphan retirement, and in phase two the + callback from the preserved directory. Crash after self-ack: no replay. +- Pool recovery with and without custom handlers drains through bad, bad, good. +- Handler closes pooled/standalone senders and the pool without self-join or + retirement waits; rebuild before durable retirement can re-reject. +- Unclosed transactional span waits for return; generation/end snapshot excludes + B's frames. Confirm pending span callbacks cannot report a provisional end. +- Live header reads respect the watermark floor and concurrent trimming; + memory and disk catch-up include deltas carried only by retired frames. +- Phase-one process-local identities distinguish recreated queues without disk + state. Phase-two epoch survives restart but changes on FSN reset. +- Phase-one defaults preserve/halt; explicit retirement loses ordinary prefix + rows as documented. Phase-two default flips only with durable-copy support. +- A healthy cross-borrow wait reports resolution, not acceptance of retired data. +- Phase two: copy replays through the existing reader; flag rewrite and CRC; + atomic publish; reuse on re-rejection; disk full; opt-out; probe. +- Both ACK levels; force commits and surviving schema side effects; benchmark + non-regression. + +## Local validation (2026-09-07) + +- Full core suite: 3,480 tests, zero failures/errors, seven skipped. +- Final targeted integration run after the last boundary and lookup changes: + 181 tests, zero failures/errors. Includes pool return/reborrow, standalone + orphan draining, preserved-tail startup reporting, dictionary continuity, + archive reuse/recovery, bounded callbacks and trim-safe frame lookup. +- Examples reactor package succeeds. +- Server rollback E2E succeeds with the new owning-handle observation noted + above. No server implementation changes are required. +- Fixed-work healthy producer check: one million rows, batches of 1,000, + three alternating runs per build. Both builds measure zero producer bytes + allocated per row; observed times are approximately 39–43 ns/row. This is a + narrow memory-mode check, not a disk or end-to-end throughput claim. +- The rejected-range lookup previously scaled quadratically (1,000/2,000/4,000 + lookups took about 1.1/4.2/15.3 ms). The cold forward lookup cache removes the + repeated scans, with no healthy publish-path index, lock or allocation. + +### Review follow-up (2026-09-08) + +- Full core suite: 3,489 tests, zero failures/errors, seven skipped. +- Final affected-suite run after the last test and policy-reporting corrections: + 98 tests, zero failures/errors. Covers recovered group boundaries, + socket-free orphan retirement with and without preserved metadata, second + schema NACK under durable ACK replay, invalid closer scans, missing skipped + frames, preservation failure followed by successful retry, and existing + dictionary, archive, orphan-tail and pool regressions. +- `git diff --check` passes. + +### Synchronous preservation follow-up (2026-09-08) + +- Removed the preservation worker, request/completion handshake and separate + shutdown coordination. The I/O thread now owns copying and retirement. +- Retained a completed-copy notification while the FIFO is full, stop-aware + retry pacing, and the existing I/O-thread cleanup fallback. +- Full core suite: 3,492 tests, zero failures/errors, seven skipped. Regression + coverage includes successful retry, no repeated archive sync while the FIFO + is full, and a blocked synchronous copy retaining the engine lock through a + shutdown timeout until delegated cleanup finishes. +- `git diff --check` passes. + +## Open decisions + +| Item | Owner | Gate | +|---|---|---| +| Minimum server release: 10.0.0; local rollback E2E passed on 10.0.1-SNAPSHOT. | Server/QWP maintainer | Resolved | +| Java builder method is `schemaMismatchPolicy`; Rust/C/C++ implementation remains a separate client deliverable. | Client API maintainers | Java resolved | +| Copied subset/dictionary recovery, last-flag CRC and archive reuse verified by tests. | Persistence maintainer | Resolved | +| Skipped dictionary carrier recovery verified in memory and disk modes. | Persistence/I/O maintainers | Resolved | +| Live-header API, first-closer bounds and atomic return-side failure capture tested; healthy stop/ACK checks use volatile fields. | Persistence/pool maintainers | Resolved | +| CRC-protected `.slot-epoch`, fresh-namespace rotation under the slot lock, restart/reuse tests. | Persistence maintainer | Resolved | +| `RejectedMiniSlotArchive` metadata version 1, CRC, enum names, span and trigger. | Persistence maintainer | Resolved | +| Diagnostics for full notification FIFOs and unreturned failed leases. | Design owner | Phase 1 | + +## Appendix A: source evidence + +- `QwpWebSocketSender.flushPendingRowsSplit`: per-table frames with + `FLAG_DEFER_COMMIT` on all but the last; its javadoc states the split is not + atomic and can deliver a prefix twice. +- `QwpWebSocketSender.transactional`: auto-flush defers, explicit `flush()` + commits; documented as committing atomically per table. +- `QwpWebSocketSender.checkConnectionError`: row-level calls poll the delegate, + so ownership checks cannot live only in `PooledSender`. +- `CursorSendEngine.retireRecoveredOrphanTailIfReady` and + `CursorWebSocketSendLoop.trySendOne`: existing stop, self-ack and recycle for + a recovered deferred tail. This is the machinery phase one generalizes. +- `CursorWebSocketSendLoop.handleServerRejection`: clamps invalid NACK + sequences for attribution; unsafe as a retirement input. +- `SenderPool` recovery drain and failure streak: the poisoned-scan path. +- `MmapSegment`, `PersistedSymbolDict`: formats reused for phase two; both + versioned with CRC, and a foreign version fails recovery without quarantine. +- Server: `QwpIngressUpgradeProcessor.handleBinaryMessage` withholds ACKs for deferred + frames, clears state before the NACK, and consumes later frames without + reply. `QwpSenderE2ETest.testDeferredCommitSchemaMismatchRollsBack`. +- Rust: `questdb-rs/src/ingress/sender/qwp_ws_publisher.rs::encode_to_scratch` + passes defer=false for SFA; `sender/qwp_ws_sfa_queue.rs` stores the frames. + `column_sender/sender.rs::rebase_lease_observation` rebases lease observation; + its direct split path uses deferred prefixes and is a different path. + +## Appendix B: alternatives dropped + +**Whole-group retirement for every ordinary split flush.** Prefix retirement +allows ordinary successors to deliver instead of discarding the entire flush. +It does not eliminate the publication/return linearization and tail sealing +needed for unclosed transactional or interrupted spans. + +**Retire only the ordinary rejected frame.** Could preserve valid predecessors +still on the ring when an independent closer survives. It is not equivalent to +prefix disposal and is worth a separate policy decision. If the rejected frame +is itself the only closer, replayed predecessors have no closer; letting a later +borrow commit them changes the result again. Define that case, interruption and +no-successor behavior before selecting this alternative. This revision retains +prefix retirement explicitly rather than claiming those predecessors are lost +under either choice. + +**Rejection journal.** Gave retained at-least-once callbacks across crashes and +handler-independent retirement, at the cost of a versioned side file, +compaction, delivery markers, and a downgrade and cross-client gate that +released readers cannot honor without a segment version bump. Re-rejection is +idempotent, so the journal bought a stronger callback guarantee than the data +path needs. Revisit if the documented zero-notification crash windows are unacceptable. + +**Throw once, then clear.** Simpler handle lifecycle, but a slot-wide wait on +the rejected FSN would then return true after retirement: a false delivery +confirmation for its own failed publication. Rejected. A later healthy borrow +may still observe slot-wide resolution for that FSN; the ownership distinction +is intentional and does not establish acceptance of old data. + +**New DLQ container format.** Replaced by a copy in the existing segment and +dictionary formats, which already have versioning, CRC and fixtures, and which +the existing recovery reader can replay after a one-byte flag rewrite. + +**Server capability gate and distinct-FSN pacer.** Replaced by a stated +minimum server release and the existing reconnect backoff. + +**Callback-gated retirement and phase-one epoch.** Callback completion is not a +retirement requirement: it adds handler-dependent stalls without durable payload +recovery. Phase one retains notifications in memory and accepts crash loss. +Durable epoch identity moves to phase two with preserved directories. Removing the per-rejection gate requires the 256-entry sticky FIFO; only +capacity exhaustion makes retirement wait for callback completion. From f2c6d586fca529c55c2634e98ee99ebd5953b170 Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Tue, 8 Sep 2026 12:56:19 +0200 Subject: [PATCH 2/6] fix(test): keep rejection archive tests compatible with Java 8 Replace Path.of and String.repeat with Java 8-compatible equivalents, preserving the long-message round-trip coverage. Validated on Temurin Java 8: reactor compilation, six archive tests, examples, and Javadoc packaging passed. --- .../qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java index 0cec14ad7..7b5965009 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java @@ -16,12 +16,14 @@ import io.questdb.client.std.FilesFacade; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Unsafe; +import io.questdb.client.test.tools.TestUtils; import org.junit.After; import org.junit.Before; import org.junit.Test; import java.nio.file.Files; import java.nio.file.Path; +import java.nio.file.Paths; import java.util.Comparator; import static org.junit.Assert.assertEquals; @@ -90,7 +92,7 @@ public void testRecoveryFindsOnlyCurrentEpochAndScopedTempCleanup() throws Excep assertEquals(null, RejectedMiniSlotArchive.findOverlapping( ff, source, "slot-recovery", java.util.UUID.randomUUID().toString(), 0, 0)); - Path rejected = Path.of(source, "rejected"); + Path rejected = Paths.get(source, "rejected"); Path ours = Files.createDirectory(rejected.resolve( ".tmp-slot-recovery-" + epoch + "-fsn-0-0-dead")); Files.createFile(ours.resolve(RejectedMiniSlotArchive.SEGMENT_FILE_NAME)); @@ -116,7 +118,7 @@ public void testPreservedSubsetReopensWithDictionarySupersetAndWorkingCopyKeepsA dictionary.appendSymbol("unused-superset-entry"); appendDeltaFrame(engine, 0, true, "zero"); appendDeltaFrame(engine, 1, true, "one"); - String serverMessage = "column mismatch ".repeat(2048); + String serverMessage = TestUtils.repeat("column mismatch ", 2048); SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, SenderError.Policy.REJECT_AND_CONTINUE, 3, serverMessage, 1, 0, 1, "tab", 42).withRejectionSpan(0, 1); From 984ed350b2de15a90d50d2cbfa12dd4e12b6f294 Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Tue, 8 Sep 2026 13:07:58 +0200 Subject: [PATCH 3/6] fix(test): compare recovered archive paths portably Compare Path objects instead of raw strings so equivalent Windows path separators do not fail the recovery notification assertion. Validation: 22 related tests passed locally on Linux; Windows CI confirmation remains pending. --- .../io/questdb/client/test/impl/SchemaRejectionPoolTest.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java index 86bffdb96..fadbb6ddc 100644 --- a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java @@ -75,7 +75,8 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat }).build()) { Assert.assertTrue(reported.await(5, TimeUnit.SECONDS)); Assert.assertNotSame(Thread.currentThread(), callbackThread.get()); - Assert.assertEquals(archive, error.get().getRejectedPath()); + Assert.assertEquals(java.nio.file.Paths.get(archive), + java.nio.file.Paths.get(error.get().getRejectedPath())); Assert.assertEquals(0, sender.getAckedFsn()); } } From 729c212b448689e4182e5c5a25dc15ae02001661 Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Tue, 8 Sep 2026 16:13:22 +0200 Subject: [PATCH 4/6] Fix schema lease retention and rejected archive recovery --- .../qwp/client/QwpWebSocketSender.java | 30 +++- .../sf/cursor/RejectedMiniSlotArchive.java | 29 +++- .../sf/cursor/SchemaRejectionState.java | 45 +++++- .../client/RejectedArchiveRecoveryTest.java | 147 ++++++++++++++++++ .../cursor/RejectedMiniSlotArchiveTest.java | 87 +++++++++++ .../sf/cursor/SchemaRejectionStateTest.java | 127 +++++++++++++++ .../test/impl/SchemaRejectionPoolTest.java | 85 +++++++++- design/schema-mismatch-terminal-resolution.md | 22 ++- 8 files changed, 553 insertions(+), 19 deletions(-) create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index eb9e638af..77f634b92 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -412,7 +412,7 @@ public class QwpWebSocketSender implements Sender { private boolean dlqEnabled = true; private String dlqDir; private SchemaPreserver schemaPreserver; - private final SchemaRejectionState schemaRejectionState = new SchemaRejectionState(); + private SchemaRejectionState schemaRejectionState; private long schemaLeaseGeneration; private boolean schemaLeaseStarted; private LineSenderServerException observedSchemaFailure; @@ -2849,6 +2849,12 @@ public void prepareSchemaPoolSlot() { } public void beginSchemaLease(long generation) { + if (schemaMismatchPolicy != SenderError.Policy.REJECT_AND_CONTINUE) { + return; + } + if (schemaRejectionState == null) { + schemaRejectionState = new SchemaRejectionState(); + } schemaLeaseGeneration = generation; observedSchemaFailure = null; schemaRejectionState.beginLease(generation, publishedSchemaFsn() + 1, transactional); @@ -4312,12 +4318,26 @@ private void ensureConnected() { // frees the mirror via its loopNeverRan path; it also closes the shared // client, so the client.close() below is a safe idempotent no-op. if (cursorSendLoop != null) { - cursorSendLoop.close(); - cursorSendLoop = null; + try { + cursorSendLoop.close(); + cursorSendLoop = null; + } catch (Throwable closeFailure) { + if (closeFailure != t) t.addSuppressed(closeFailure); + } } if (client != null) { - client.close(); - client = null; + try { + client.close(); + client = null; + } catch (Throwable closeFailure) { + if (closeFailure != t) t.addSuppressed(closeFailure); + } + } + if (t instanceof UnreplayableSlotException) { + // Startup also validates archives needed by orphan retirement. + // Sender.build() must receive this type to quarantine the slot; + // wrapping it would make every build retry fail on the same bytes. + throw (UnreplayableSlotException) t; } Endpoint ep = currentEndpoint(); LineSenderException ex = new LineSenderException(t); diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java index 2180ff2c7..4b3a01dbb 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java @@ -253,11 +253,29 @@ public static SenderError findOverlapping( int type = ff.findType(find); rc = ff.findNext(find); if (type != Files.DT_DIR || name == null || !name.startsWith(prefix)) continue; + int separator = name.indexOf('-', prefix.length()); + if (separator < 0) continue; + long archiveFrom; + long archiveTo; + try { + archiveFrom = Long.parseLong(name.substring(prefix.length(), separator)); + archiveTo = Long.parseLong(name.substring(separator + 1)); + } catch (NumberFormatException e) { + continue; + } + // Completed archives have canonical range names. Filter before + // opening metadata: a damaged, already-drained archive is not + // evidence about this recovered tail and must not block startup. + if (archiveFrom < 0 || archiveTo < archiveFrom + || !name.equals(prefix + archiveFrom + '-' + archiveTo) + || archiveTo < fromFsn || archiveFrom > toFsn) { + continue; + } String path = rejectedRoot + '/' + name; Metadata metadata = readMetadata(ff, path); if (!slotId.equals(metadata.slotId) || !epoch.equals(metadata.epoch) - || metadata.toFsn < fromFsn || metadata.fromFsn > toFsn) { - continue; + || metadata.fromFsn != archiveFrom || metadata.toFsn != archiveTo) { + throw new UnreplayableSlotException("rejected mini-slot directory identity mismatch " + path); } validate(ff, path, metadata); SenderError error = new SenderError( @@ -335,6 +353,13 @@ private static void validate(FilesFacade ff, String dir, Metadata expected) { || watermark == null || watermark.read() != actual.fromFsn - 1L) { throw new UnreplayableSlotException("invalid rejected mini-slot boundaries " + dir); } + } catch (MmapSegmentCorruptionException e) { + // Positively identified archive corruption is a terminal recovery + // verdict too. Operational read/mmap failures retain their type. + UnreplayableSlotException failure = new UnreplayableSlotException( + "corrupt rejected mini-slot " + dir + ": " + e.getMessage()); + failure.initCause(e); + throw failure; } if (actual.hasDictionary) { try (PersistedSymbolDict ignored = PersistedSymbolDict.open(ff, dir)) { diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java index b5be736e5..b8f9b3d5e 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java @@ -7,6 +7,7 @@ import io.questdb.client.LineSenderServerException; import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import java.util.ArrayDeque; @@ -29,6 +30,18 @@ public synchronized void beginLease(long generation, long firstFsn, boolean tran if (tail != null && tail.active) { throw new IllegalStateException("previous lease is still active"); } + if (tail != null) { + // The returned handle can no longer observe an owned exception. Keep + // only the range needed to classify a delayed rejection, and combine + // completed borrows instead of retaining one object per borrow until ACK. + if (failedGeneration == tail.generation) { + failedGeneration = -1L; + } + tail.generation = -1L; + tail.failure = null; + tail.rawError = null; + mergeReturnedTail(); + } leases.addLast(new Lease(generation, firstFsn, transactional)); } @@ -49,6 +62,9 @@ public synchronized LineSenderServerException endLease(long generation, long pub leases.removeLast(); return null; } + int flags = lease.transactional && engine != null ? engine.liveQwpFrameFlags(publishedFsn) : -1; + lease.endsWithCommit = !lease.transactional + || (flags >= 0 && (flags & QwpConstants.FLAG_DEFER_COMMIT) == 0); sealIfNeeded(lease, publishedFsn); if (failedGeneration == generation) { failedGeneration = -1L; @@ -93,7 +109,7 @@ public synchronized boolean reject(long rejectedFsn, long spanStart, SenderError owner = new Lease(-1L, spanStart, recovered); owner.active = false; owner.endFsn = recovered ? recoveredTip : rejectedFsn; - } else if (owner.rawError == null) { + } else if (owner.generation >= 0 && owner.rawError == null) { owner.rawError = rawError; failedGeneration = owner.generation; } @@ -127,7 +143,6 @@ public synchronized void completeRetirement(long lastFsn) { throw new IllegalStateException("retirement range changed"); } acknowledgedThrough(lastFsn); - pending.owner.retired = true; pending = null; stopFsn = -1L; prune(lastFsn); @@ -160,7 +175,7 @@ private long firstCommitFsn(long first, long last) { if (flags < 0) { throw new IllegalStateException("missing frame while resolving transaction at FSN " + fsn); } - if ((flags & io.questdb.client.cutlass.qwp.protocol.QwpConstants.FLAG_DEFER_COMMIT) == 0) { + if ((flags & QwpConstants.FLAG_DEFER_COMMIT) == 0) { return fsn; } } @@ -198,10 +213,26 @@ private Lease findOwner(long fsn) { return null; } + private void mergeReturnedTail() { + Lease tail = leases.removeLast(); + Lease previous = leases.peekLast(); + if (previous != null && previous.transactional == tail.transactional + && previous.endFsn + 1 == tail.firstFsn && previous.endsWithCommit + && (pending == null || pending.owner != previous)) { + // Normal pool return publishes a commit before ending the lease. + // Its frame flags preserve each transaction's boundary in a merged + // range. An unfinished failed transaction must keep its own end, + // and an in-flight retirement must retain its owner object. + tail.firstFsn = previous.firstFsn; + leases.removeLast(); + } + leases.addLast(tail); + } + private void prune(long fsn) { while (true) { Lease head = leases.peekFirst(); - if (head == null || head.active || (head.rawError != null && !head.retired) || head.endFsn > fsn) { + if (head == null || head.active || (pending != null && pending.owner == head) || head.endFsn > fsn) { return; } leases.removeFirst(); @@ -236,14 +267,14 @@ private Pending(long firstFsn, long lastFsn, Lease owner, SenderError rawError) } private static final class Lease { - private final long firstFsn; - private final long generation; + private long firstFsn; + private long generation; private final boolean transactional; private boolean active = true; private long endFsn = -1L; + private boolean endsWithCommit; private LineSenderServerException failure; private SenderError rawError; - private boolean retired; private Lease(long generation, long firstFsn, boolean transactional) { this.generation = generation; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java new file mode 100644 index 000000000..31dad9ec9 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java @@ -0,0 +1,147 @@ +/******************************************************************************* + * Copyright (c) 2014-2026 QuestDB + * Licensed under the Apache License, Version 2.0. + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.std.FilesFacade; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.Assert.*; + +public class RejectedArchiveRecoveryTest { + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testDamagedDrainedArchiveDoesNotBlockRepeatedBuilds() throws Exception { + assertRepeatedBuilds(RejectedMiniSlotArchive.METADATA_FILE_NAME, false); + } + + @Test(timeout = 30_000) + public void testDamagedOverlappingArchiveQuarantinesOnceAndBuildsContinue() throws Exception { + assertRepeatedBuilds(RejectedMiniSlotArchive.METADATA_FILE_NAME, true); + } + + @Test(timeout = 30_000) + public void testIntactOverlappingArchiveReportsAndBuildsContinue() throws Exception { + assertRepeatedBuilds(null, true); + } + + @Test(timeout = 30_000) + public void testDamagedOverlappingArchiveSegmentQuarantinesOnce() throws Exception { + assertRepeatedBuilds(RejectedMiniSlotArchive.SEGMENT_FILE_NAME, true); + } + + private void assertRepeatedBuilds(String damagedFile, boolean overlaps) throws Exception { + boolean damaged = damagedFile != null; + Path base = temp.newFolder().toPath(); + Path slot = Files.createDirectory(base.resolve("saved")); + String archive; + try (CursorSendEngine engine = new CursorSendEngine(slot.toString(), 4096)) { + String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, slot.toString(), engine.freshFsnNamespace()); + append(engine, false); + archive = preserve(engine, slot, epoch, 0); + assertTrue(engine.acknowledge(0)); + append(engine, true); // Uncommitted orphan tail at FSN 1 forces the startup archive scan. + if (overlaps) archive = preserve(engine, slot, epoch, 1); + } + Path archiveFile = Paths.get(archive, damaged ? damagedFile : RejectedMiniSlotArchive.METADATA_FILE_NAME); + byte[] archiveBytes = Files.readAllBytes(archiveFile); + if (damaged) { + archiveBytes[0] ^= 1; // Corrupt metadata CRC or segment magic, retaining the remaining bytes. + Files.write(archiveFile, archiveBytes); + } + AtomicInteger quarantines = new AtomicInteger(); + CountDownLatch schemaReported = new CountDownLatch(1); + Map sequences = new ConcurrentHashMap<>(); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long sequence = sequences.merge(client, 1L, Long::sum) - 1; + try { + client.sendBinary(QwpWireTestUtils.buildAck(sequence)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + })) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + List failures = new ArrayList<>(); + for (int attempt = 0; attempt < 3; attempt++) { + try (Sender sender = Sender.builder("ws::addr=localhost:" + server.getPort() + + ";sf_dir=" + base + ";close_flush_timeout_millis=0;") + .senderId("saved").errorHandler(error -> { + if (error.getCategory() == SenderError.Category.DATA_LOSS) quarantines.incrementAndGet(); + if (error.getCategory() == SenderError.Category.SCHEMA_MISMATCH) schemaReported.countDown(); + }).build()) { + sender.table("healthy").longColumn("value", attempt).atNow(); + long target = sender.flushAndGetSequence(); + assertTrue("new rows must drain after recovery", sender.awaitAckedFsn(target, 5_000)); + if (attempt == 0 && !damaged && overlaps) { + assertTrue(schemaReported.await(5, TimeUnit.SECONDS)); + } + } catch (RuntimeException e) { + failures.add(e); + } + } + assertTrue("all three builds must succeed: " + failures, failures.isEmpty()); + } + Path quarantined = base.resolve("saved.unreplayable-0"); + if (damaged && overlaps) { + assertEquals(1, quarantines.get()); + assertTrue(Files.exists(quarantined.resolve(".failed"))); + assertArrayEquals(archiveBytes, Files.readAllBytes(quarantined.resolve(slot.relativize(archiveFile)))); + assertFalse(Files.exists(base.resolve("saved.unreplayable-1"))); + } else { + assertEquals(0, quarantines.get()); + assertFalse(Files.exists(quarantined)); + assertArrayEquals(archiveBytes, Files.readAllBytes(archiveFile)); + } + } + + private static String preserve(CursorSendEngine engine, Path slot, String epoch, long fsn) { + SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH, + SenderError.Policy.REJECT_AND_CONTINUE, 3, "schema rejected", fsn, fsn, fsn, null, 1); + return RejectedMiniSlotArchive.preserve(FilesFacade.INSTANCE, engine, null, + slot.toString(), "saved", epoch, error).path; + } + + private static void append(CursorSendEngine engine, boolean deferred) { + long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + try { + Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0); + Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE); + Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, + (byte) (deferred ? QwpConstants.FLAG_DEFER_COMMIT : 0)); + engine.appendBlocking(frame, QwpConstants.HEADER_SIZE); + } finally { + Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java index 7b5965009..5d00e8b3d 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java @@ -9,14 +9,17 @@ import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegment; +import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegmentException; import io.questdb.client.cutlass.qwp.client.sf.cursor.PersistedSymbolDict; import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.client.sf.cursor.UnreplayableSlotException; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.std.FilesFacade; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Unsafe; import io.questdb.client.test.tools.TestUtils; +import io.questdb.client.test.tools.DelegatingFilesFacade; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -31,6 +34,8 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; public class RejectedMiniSlotArchiveTest { private Path root; @@ -105,6 +110,88 @@ public void testRecoveryFindsOnlyCurrentEpochAndScopedTempCleanup() throws Excep } } + @Test + public void testRecoveryDoesNotReadDamagedArchivesOutsideRequestedRange() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("unrelated-archives")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + for (int i = 0; i < 3; i++) appendDeltaFrame(engine, i, true, "symbol" + i); + for (long fsn : new long[]{0, 2}) { + String archive = RejectedMiniSlotArchive.preserve( + ff, engine, null, source, "slot", epoch, rejection(fsn)).path; + Files.write(Paths.get(archive, RejectedMiniSlotArchive.METADATA_FILE_NAME), new byte[]{0}); + } + assertNull(RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 1, 1)); + // Corruption still fails closed when its range is actually needed. + try { + RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 0, 0); + fail("overlapping damaged metadata must not be ignored"); + } catch (UnreplayableSlotException expected) { + assertTrue(expected.getMessage().contains("invalid rejection metadata size")); + } + } + } + + @Test + public void testRecoveryRequiresMetadataToMatchDirectoryRange() throws Exception { + FilesFacade ff = FilesFacade.INSTANCE; + String source = Files.createDirectory(root.resolve("mismatched-archive")).toString(); + String epoch = SlotEpoch.openOrCreate(ff, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + appendDeltaFrame(engine, 0, true, "zero"); + String archive = RejectedMiniSlotArchive.preserve( + ff, engine, null, source, "slot", epoch, rejection(0)).path; + Files.move(Paths.get(archive), Paths.get(source, "rejected", "slot-" + epoch + "-fsn-0-1")); + try { + RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 1, 1); + fail("overlapping directory with contradictory metadata must fail closed"); + } catch (UnreplayableSlotException expected) { + assertTrue(expected.getMessage().contains("directory identity mismatch")); + } + } + } + + @Test + public void testRecoveryIgnoresNoncanonicalDirectoryNames() throws Exception { + String source = Files.createDirectory(root.resolve("noncanonical-archives")).toString(); + String epoch = java.util.UUID.randomUUID().toString(); + Path rejected = Files.createDirectory(Paths.get(source, "rejected")); + String prefix = "slot-" + epoch + "-fsn-"; + for (String suffix : new String[]{"", "0", "0-0-extra", "-1-0", "2-1", "00-1", "+0-1", + "0-9223372036854775808"}) { + Files.createDirectory(rejected.resolve(prefix + suffix)); + } + assertNull(RejectedMiniSlotArchive.findOverlapping( + FilesFacade.INSTANCE, source, "slot", epoch, 0, Long.MAX_VALUE)); + } + + @Test + public void testArchiveReadFailureIsNotReclassifiedAsCorruption() throws Exception { + String source = Files.createDirectory(root.resolve("archive-read-failure")).toString(); + String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, source); + try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) { + appendDeltaFrame(engine, 0, true, "zero"); + String archive = RejectedMiniSlotArchive.preserve( + FilesFacade.INSTANCE, engine, null, source, "slot", epoch, rejection(0)).path; + MmapSegmentException failure = new MmapSegmentException("injected operational read failure"); + FilesFacade ff = new DelegatingFilesFacade() { + @Override + public int openRW(String path) { + if (path.equals(archive + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)) throw failure; + return super.openRW(path); + } + }; + try { + RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 0, 0); + fail("operational read failure must propagate"); + } catch (MmapSegmentException expected) { + assertSame(failure, expected); + } + assertTrue(Files.isDirectory(Paths.get(archive))); + } + } + @Test public void testPreservedSubsetReopensWithDictionarySupersetAndWorkingCopyKeepsArchive() throws Exception { FilesFacade ff = FilesFacade.INSTANCE; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java index c117ac349..1aa9d2322 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java @@ -16,12 +16,139 @@ import org.junit.Rule; import org.junit.rules.TemporaryFolder; +import java.lang.reflect.Field; +import java.util.Collection; + import static org.junit.Assert.*; public class SchemaRejectionStateTest { @Rule public final TemporaryFolder temp = new TemporaryFolder(); + @Test + public void testUnackedOrdinaryBorrowsRetainBoundedHistory() throws Exception { + SchemaRejectionState state = new SchemaRejectionState(); + for (int i = 0; i < 20_000; i++) { + state.beginLease(i, i, false); + state.endLease(i, i); + } + assertEquals(2, retainedRanges(state)); + state.beginLease(20_000, 20_000, false); + assertTrue(state.reject(10_000, 10_000, error(10_000))); + assertEquals(10_000, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(20_000)); + assertNull(state.ownedFailure(20_000, 20_000)); + } + + @Test + public void testUnackedTransactionalBorrowsPreserveCommitBoundaries() throws Exception { + try (CursorSendEngine engine = new CursorSendEngine(null, 4096)) { + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + for (int i = 0; i < 20_000; i++) { + state.beginLease(i, 2L * i, true); + append(engine, true); + append(engine, false); + state.endLease(i, 2L * i + 1); + } + assertEquals(2, retainedRanges(state)); + state.beginLease(20_000, 40_000, true); + assertTrue(state.reject(20_000, 20_000, error(20_000))); + assertEquals(20_001, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(20_000)); + state.completeRetirement(20_001); + assertTrue(state.reject(20_002, 20_002, error(20_002))); + assertEquals(20_003, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(20_000)); + } + } + + @Test + public void testUnfinishedReturnedTransactionCannotConsumeLaterBorrow() throws Exception { + try (CursorSendEngine engine = new CursorSendEngine(null, 4096)) { + SchemaRejectionState state = new SchemaRejectionState(); + state.setEngine(engine); + state.beginLease(1, 0, true); + append(engine, true); + append(engine, true); + state.endLease(1, 1); + state.beginLease(2, 2, true); + append(engine, false); + state.endLease(2, 2); + state.beginLease(3, 3, true); + assertTrue(state.reject(0, 0, error(0))); + assertEquals(1, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(3)); + } + } + + @Test + public void testPendingRetirementSurvivesBorrowHistoryCompaction() throws Exception { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(1, 0, false); + state.endLease(1, 0); + assertTrue(state.reject(0, 0, error(0))); + for (int i = 2; i < 20_000; i++) { + state.beginLease(i, i - 1, false); + state.endLease(i, i - 1); + } + assertEquals(3, retainedRanges(state)); + assertEquals(0, state.sealedRange().lastFsn); + state.completeRetirement(0); + state.beginLease(20_000, 19_999, false); + assertEquals(2, retainedRanges(state)); + assertTrue(state.reject(1, 1, error(1))); + assertEquals(1, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(20_000)); + } + + @Test + public void testCompactionIntoPendingOwnerPreservesItsRangeAndNextBorrowFailure() { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(1, 0, false); + state.endLease(1, 0); + state.beginLease(2, 1, false); + assertTrue(state.reject(1, 1, error(1))); + LineSenderServerException failure = state.ownedFailure(2, 1); + assertSame(failure, state.endLease(2, 1)); + + // The failed returned lease absorbs the older range while its + // retirement is still pending. The notification must remain [1, 1]. + state.beginLease(3, 2, false); + assertEquals(1, state.sealedRange().firstFsn); + assertEquals(1, state.sealedRange().lastFsn); + assertFalse(state.hasOwnedFailure(2)); + assertFalse(state.hasOwnedFailure(3)); + state.completeRetirement(1); + assertTrue(state.reject(2, 2, error(2))); + assertTrue(state.hasOwnedFailure(3)); + LineSenderServerException nextFailure = state.ownedFailure(3, 2); + assertNotSame(failure, nextFailure); + assertEquals(2, nextFailure.getServerError().getRejectedFsn()); + } + + @Test + public void testEmptyBorrowsDoNotRetainHistoryBehindUnackedRange() throws Exception { + SchemaRejectionState state = new SchemaRejectionState(); + state.beginLease(0, 0, false); + state.endLease(0, 0); + for (int i = 1; i < 20_000; i++) { + state.beginLease(i, 1, false); + state.endLease(i, 0); + } + assertEquals(1, retainedRanges(state)); + state.acknowledgedThrough(0); + state.beginLease(20_000, 1, false); + state.endLease(20_000, 0); + assertEquals(0, retainedRanges(state)); + } + + private static int retainedRanges(SchemaRejectionState state) throws Exception { + Field field = SchemaRejectionState.class.getDeclaredField("leases"); + field.setAccessible(true); + return ((Collection) field.get(state)).size(); + } + @Test public void testRecoveredGroupRetiresThroughCloserRegardlessOfNewLeaseMode() throws Exception { String path = temp.newFolder("recovered-group").getAbsolutePath(); diff --git a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java index fadbb6ddc..442289c6b 100644 --- a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java @@ -25,7 +25,9 @@ import org.junit.rules.TemporaryFolder; import java.io.IOException; +import java.lang.reflect.Field; import java.nio.file.Files; +import java.util.Collection; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.CountDownLatch; @@ -36,6 +38,69 @@ public class SchemaRejectionPoolTest { @Rule public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + @Test + public void testNoAckBorrowsRetainBoundedSchemaHistory() throws Exception { + assertNoAckBorrowHistory(SenderError.Policy.REJECT_AND_CONTINUE, false); + assertNoAckBorrowHistory(SenderError.Policy.REJECT_AND_CONTINUE, true); + } + + @Test + public void testTerminalPolicyDoesNotAllocateSchemaHistory() throws Exception { + assertNoAckBorrowHistory(SenderError.Policy.TERMINAL, false); + assertNoAckBorrowHistory(SenderError.Policy.TERMINAL, true); + } + + private void assertNoAckBorrowHistory(SenderError.Policy policy, boolean transactional) throws Exception { + CountDownLatch received = new CountDownLatch(1); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + received.countDown(); // Deliberately never ACK. + } + })) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Exercise both memory queues and disk-backed outage buffering. + String storage = transactional ? "sf_dir=" + temp.newFolder().getAbsolutePath() + + ";sf_durability=periodic;sf_sync_interval_millis=1000;" : ""; + try (QuestDB db = QuestDB.builder() + .fromConfig("ws::addr=localhost:" + server.getPort() + + ";close_flush_timeout_millis=0;auto_flush_rows=1;auto_flush_bytes=off;transaction=" + + (transactional ? "on" : "off") + ";" + storage) + .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) + .schemaMismatchPolicy(policy).dlqEnabled(false).build()) { + Object delegate = null; + for (int i = 0; i < 20_000; i++) { + try (Sender sender = db.borrowSender()) { + Object borrowedDelegate = field(field(sender, "slot"), "delegate"); + if (delegate == null) { + delegate = borrowedDelegate; + } else { + Assert.assertSame("all borrows must reuse the same slot", delegate, borrowedDelegate); + } + sender.table("unacked").longColumn("value", i).atNow(); + } + } + Assert.assertTrue(received.await(5, TimeUnit.SECONDS)); + CursorSendEngine engine = (CursorSendEngine) field(delegate, "cursorEngine"); + Assert.assertEquals(-1, engine.ackedFsn()); + Assert.assertTrue("every borrow must publish data", engine.publishedFsn() >= 19_999); + Object state = field(delegate, "schemaRejectionState"); + if (policy == SenderError.Policy.TERMINAL) { + Assert.assertNull(state); + } else { + Assert.assertEquals(2, ((Collection) field(state, "leases")).size()); + } + } + } + } + + private static Object field(Object object, String name) throws Exception { + Field field = object.getClass().getDeclaredField(name); + field.setAccessible(true); + return field.get(object); + } + @Test public void testRecoveredPreservedOrphanReportsAsynchronouslyBeforeRetirement() throws Exception { String base = temp.newFolder("recovered").getAbsolutePath(); @@ -289,6 +354,11 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat @Test public void testRejectionAfterReturnDoesNotFailNextBorrow() throws Exception { + assertRejectionAfterReturnDoesNotFailNextBorrow(false); + assertRejectionAfterReturnDoesNotFailNextBorrow(true); + } + + private static void assertRejectionAfterReturnDoesNotFailNextBorrow(boolean transactional) throws Exception { CountDownLatch firstReceived = new CountDownLatch(1); CountDownLatch rejectNow = new CountDownLatch(1); CountDownLatch reported = new CountDownLatch(1); @@ -317,7 +387,9 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat server.start(); Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); try (QuestDB db = QuestDB.builder() - .fromConfig("ws::addr=localhost:" + server.getPort() + ";close_flush_timeout_millis=0;") + .fromConfig("ws::addr=localhost:" + server.getPort() + + ";close_flush_timeout_millis=0;auto_flush_rows=1;auto_flush_bytes=off;transaction=" + + (transactional ? "on" : "off") + ";") .senderPoolSize(1).queryPoolMin(0).queryPoolMax(1) .schemaMismatchPolicy(SenderError.Policy.REJECT_AND_CONTINUE).dlqEnabled(false) .errorHandler(error -> { rejection.set(error); reported.countDown(); }).build()) { @@ -326,14 +398,23 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat a.flush(); Assert.assertTrue(firstReceived.await(5, TimeUnit.SECONDS)); } + // Move the rejected borrow into a compacted historical range. + for (int i = 0; i < 20; i++) { + try (Sender intervening = db.borrowSender()) { + intervening.table("good").longColumn("value", i).atNow(); + } + } try (Sender b = db.borrowSender()) { b.table("good").longColumn("value", 42).atNow(); - long target = b.flushAndGetSequence(); + b.flush(); + long target = ((CursorSendEngine) field(field(field(b, "slot"), "delegate"), "cursorEngine")) + .publishedFsn(); rejectNow.countDown(); Assert.assertTrue("later borrow must drain past old rejection", b.awaitAckedFsn(target, 10_000)); Assert.assertTrue(reported.await(5, TimeUnit.SECONDS)); Assert.assertEquals(SenderError.Policy.REJECT_AND_CONTINUE, rejection.get().getAppliedPolicy()); Assert.assertEquals(0, rejection.get().getRejectedFsn()); + Assert.assertEquals(transactional ? 1 : 0, rejection.get().getToFsn()); } } finally { rejectNow.countDown(); diff --git a/design/schema-mismatch-terminal-resolution.md b/design/schema-mismatch-terminal-resolution.md index 679677c2a..0ce6ab784 100644 --- a/design/schema-mismatch-terminal-resolution.md +++ b/design/schema-mismatch-terminal-resolution.md @@ -208,9 +208,17 @@ FSN at borrow; failed/observed state and the return-side end snapshot above are also required. Use the highest already published FSN plus one as the start. Row-level calls in `QwpWebSocketSender.checkConnectionError` and the `PooledSender` wrapper both check it, since row calls poll the delegate directly. Empty borrows retain no -lease record. Publishing leases remain until resolved progress passes them; -active-lease lookup is constant time, while historical rejection attribution -scans retained leases. A long ACK stall can therefore retain many small records. +lease record. `TERMINAL` does not allocate schema ownership state. Under +`REJECT_AND_CONTINUE`, the next borrow removes the returned handle's generation +and exception and coalesces adjacent returned ranges with the same transaction +mode. Ordinary ranges can always coalesce; transactional ranges can coalesce +only across a commit-bearing return boundary. The queued frame flags still +identify each closed transaction's end. An unfinished failed transaction keeps +its own end boundary, and a pending retirement keeps its owner object until +completion. Thus normal publishing borrows during an ACK stall retain one +historical range plus the current or most recently returned lease, rather than +one record per borrow. Active-lease lookup remains constant time. Resolved +progress prunes obsolete ranges. An owned rejection fails the handle. Every subsequent publish, wait and drain on that handle throws the same `LineSenderServerException` until the handle is @@ -418,6 +426,14 @@ retain that notification and retire locally before attempting a connection. An unreachable server must not prevent this socket-free cleanup; callback execution remains asynchronous. +Startup filters completed archive directories by their canonical slot, epoch +and FSN range before opening metadata. Damage in an unrelated, already-drained +archive cannot block recovery. An overlapping archive must pass metadata, +directory identity and replay-file validation. Proven corruption preserves the +`UnreplayableSlotException` type through startup cleanup so `Sender.build()` +can quarantine the whole slot, report `DATA_LOSS` and continue on a fresh one. +Operational storage failures do not become corruption verdicts. + With export disabled, a persistent schema fault retires indefinitely with one paced report per span and no circuit breaker. That is the accepted cost of opting out. From 26e3c3acccac21374d8fec4742d165037827695b Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Tue, 8 Sep 2026 16:48:21 +0200 Subject: [PATCH 5/6] fix(test): persist ACK watermark in archive recovery fixture --- .../qwp/client/RejectedArchiveRecoveryTest.java | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java index 31dad9ec9..93b484856 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java @@ -7,6 +7,7 @@ import io.questdb.client.Sender; import io.questdb.client.SenderError; +import io.questdb.client.cutlass.qwp.client.sf.cursor.AckWatermark; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; @@ -63,6 +64,8 @@ private void assertRepeatedBuilds(String damagedFile, boolean overlaps) throws E Path slot = Files.createDirectory(base.resolve("saved")); String archive; try (CursorSendEngine engine = new CursorSendEngine(slot.toString(), 4096)) { + // Keep fixture construction independent of the manager's ACK-persistence tick. + engine.getManagerForTesting().close(); String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, slot.toString(), engine.freshFsnNamespace()); append(engine, false); archive = preserve(engine, slot, epoch, 0); @@ -70,6 +73,15 @@ private void assertRepeatedBuilds(String damagedFile, boolean overlaps) throws E append(engine, true); // Uncommitted orphan tail at FSN 1 forces the startup archive scan. if (overlaps) archive = preserve(engine, slot, epoch, 1); } + // FSN 0 is the drained prefix of this fixture. acknowledge() only + // advances the live ring; a partially drained close need not persist + // that watermark. Write it explicitly so orphan validation happens + // during build(), rather than after replay ACKs on the I/O thread. + try (AckWatermark watermark = AckWatermark.open(slot.toString())) { + assertNotNull(watermark); + watermark.write(0); + watermark.sync(); + } Path archiveFile = Paths.get(archive, damaged ? damagedFile : RejectedMiniSlotArchive.METADATA_FILE_NAME); byte[] archiveBytes = Files.readAllBytes(archiveFile); if (damaged) { @@ -100,6 +112,8 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat if (error.getCategory() == SenderError.Category.DATA_LOSS) quarantines.incrementAndGet(); if (error.getCategory() == SenderError.Category.SCHEMA_MISMATCH) schemaReported.countDown(); }).build()) { + assertEquals("quarantine must complete during build", damaged && overlaps ? 1 : 0, + quarantines.get()); sender.table("healthy").longColumn("value", attempt).atNow(); long target = sender.flushAndGetSequence(); assertTrue("new rows must drain after recovery", sender.awaitAckedFsn(target, 5_000)); From 1c40cf4c20c2c24f239ae6ecbbacb3c33d6680a2 Mon Sep 17 00:00:00 2001 From: Jaromir Hamala Date: Wed, 9 Sep 2026 11:00:10 +0200 Subject: [PATCH 6/6] refactor(qwp): simplify schema rejection recovery --- README.md | 19 +- .../java/io/questdb/client/SenderError.java | 30 +- .../qwp/client/QwpWebSocketSender.java | 20 +- .../client/sf/cursor/BackgroundDrainer.java | 22 +- .../client/sf/cursor/CursorSendEngine.java | 17 - .../sf/cursor/CursorWebSocketSendLoop.java | 53 +- .../qwp/client/sf/cursor/MmapSegment.java | 22 +- .../client/sf/cursor/PersistedSymbolDict.java | 53 -- .../sf/cursor/RejectedMiniSlotArchive.java | 639 +++++--------- .../qwp/client/sf/cursor/SchemaPreserver.java | 73 -- .../sf/cursor/SchemaRejectionState.java | 132 +-- .../qwp/client/sf/cursor/SlotEpoch.java | 116 --- .../io/questdb/client/impl/SenderPool.java | 2 +- .../client/RejectedArchiveRecoveryTest.java | 92 +- .../BackgroundDrainerSetupFailureTest.java | 26 +- ...ursorWebSocketSendLoopPoisonFrameTest.java | 22 +- ...etSendLoopSchemaPreservationCloseTest.java | 7 +- .../qwp/client/sf/cursor/MmapSegmentTest.java | 35 +- .../cursor/RejectedMiniSlotArchiveTest.java | 390 ++++----- .../sf/cursor/SchemaRejectionStateTest.java | 92 +- .../test/impl/SchemaRejectionPoolTest.java | 58 +- design/schema-mismatch-terminal-resolution.md | 818 ++++-------------- examples/POOLED_SF_POISON_DEMO.md | 49 ++ .../sender/WsPooledSchemaPoisonDemo.java | 256 ++++++ 24 files changed, 1188 insertions(+), 1855 deletions(-) delete mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java delete mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java create mode 100644 examples/POOLED_SF_POISON_DEMO.md create mode 100644 examples/src/main/java/com/example/sender/WsPooledSchemaPoisonDemo.java diff --git a/README.md b/README.md index f576bb881..1e6b8b492 100644 --- a/README.md +++ b/README.md @@ -170,14 +170,25 @@ options. A configured destination is checked at build time; later storage failures pause retirement and retry the copy while keeping the source frames. A second schema rejection while an earlier range is pending, or an invalid retirement range/dictionary, logs an error and falls back to `TERMINAL`. -Preserved copies use the -binary store-and-forward format, with rejection metadata; they are not JSON. +Preserved payloads use the binary store-and-forward format; `rejection.properties` +contains human-readable error metadata. A source queue namespace, the source +segment's persisted generation token, and the exact FSN range determine the archive +directory. A retry or restart for that same live range removes its exact crashed +staging directory and reuses a structurally valid completed copy. + +Startup does not scan archive directories. If recovery finds an orphan tail, it +checks only that range's deterministic archive path. A completed copy whose +metadata, segment, manifest, watermark, and optional dictionary validate produces +an asynchronous `SenderError` before the tail retires. A missing or damaged copy +is ignored so archive output cannot block live-queue recovery. Unrelated and +legacy `.tmp-*` directories are left untouched. A crash before publication or +after retirement but before callback delivery can still lose the notification. Copy an archive to a separate working directory before replaying it, because normal queue cleanup removes drained data. Replay after fixing the schema can duplicate rows that the server committed before the error. -Copies are never automatically deleted and can contain a full symbol dictionary -each. Quarantining a damaged slot also moves its archives; use the `DATA_LOSS` +Completed copies are never automatically deleted and can contain a full symbol dictionary +each. Quarantining a damaged live slot also moves its archives; use the `DATA_LOSS` event's quarantine path to locate copies whose reported paths have moved. Monitor `getDlqBytesWritten()`, `getDlqFilesWritten()` and free disk space (the counters are available on `QwpWebSocketSender`). TLS does not encrypt these diff --git a/core/src/main/java/io/questdb/client/SenderError.java b/core/src/main/java/io/questdb/client/SenderError.java index 9c56cc3ec..93ed50fc8 100644 --- a/core/src/main/java/io/questdb/client/SenderError.java +++ b/core/src/main/java/io/questdb/client/SenderError.java @@ -43,10 +43,12 @@ * *

The {@code [fromFsn, toFsn]} span is the load-bearing correlation key — join it to * whatever the producer thread logged alongside the published-sequence value returned by - * the sender to identify the rejected data. Schema archive reports from background orphan drainers retain that orphan's - * local FSN span; use the archive path to identify its queue. Other background - * reports use {@link #NO_MESSAGE_SEQUENCE}. Never join an orphan's FSNs to - * the live producer's rows. + * the sender to identify the rejected data. A schema report recovered from a completed + * preserved copy retains the recovered queue's local FSN span; use the archive path to + * identify its queue. Such a report is reconstructed only for the exact still-live orphan + * range whose deterministic archive exists and passes structural validation. Other + * background reports use {@link #NO_MESSAGE_SEQUENCE}. Never join an orphan's FSNs to the + * live producer's rows. * * @see SenderErrorHandler * @see LineSenderServerException @@ -146,7 +148,7 @@ public long getRejectedFsn() { return rejectedFsn; } - /** Completed preserved-copy directory, or null when no copy is available yet. */ + /** Completed preserved-copy directory, or null when this report has no available copy. */ public @Nullable String getRejectedPath() { return rejectedPath; } @@ -190,7 +192,9 @@ public SenderError withAppliedPolicy(Policy policy) { } /** - * @return wall-clock-independent receipt time on the I/O thread, from {@link System#nanoTime()}. + * @return the value of {@link System#nanoTime()} when the original process received the + * rejection. A report reconstructed from a preserved copy retains that raw value; it cannot + * be compared or ordered against {@code nanoTime()} values from the recovering process. */ public long getDetectedAtNanos() { return detectedAtNanos; @@ -199,15 +203,17 @@ public long getDetectedAtNanos() { /** * @return inclusive lower bound of the FSN span for the rejected batch — correlation key for producer-side logs. * For {@link Category#DATA_LOSS} and non-schema background reports this is - * {@link #NO_MESSAGE_SEQUENCE}. Schema archive reports retain the orphan queue's local span. + * {@link #NO_MESSAGE_SEQUENCE}. Recovered schema reports retain the orphan queue's local span. */ public long getFromFsn() { return fromFsn; } /** - * @return server's per-frame messageSequence as mirrored back in the rejection frame, or - * {@link #NO_MESSAGE_SEQUENCE} for {@link Category#PROTOCOL_VIOLATION} (WS close frames carry no QWP sequence). + * @return the server's per-frame message sequence mirrored in a live rejection, or + * {@link #NO_MESSAGE_SEQUENCE} when no QWP sequence exists. A schema report reconstructed + * from a preserved copy uses its persisted rejected FSN here because the original wire + * sequence is not stored; use {@link #getRejectedFsn()} for that local correlation value. */ public long getMessageSequence() { return messageSequence; @@ -251,7 +257,7 @@ public int getServerStatusByte() { /** * @return inclusive upper bound of the FSN span for the rejected batch. * For {@link Category#DATA_LOSS} and non-schema background reports this is - * {@link #NO_MESSAGE_SEQUENCE}. Schema archive reports retain the orphan queue's local span. + * {@link #NO_MESSAGE_SEQUENCE}. Recovered schema reports retain the orphan queue's local span. */ public long getToFsn() { return toFsn; @@ -355,7 +361,9 @@ public enum Category { * retire the affected span after preserving it when configured, notify the * handler, and fail its owning handle. Other errors replay, halt with bytes * retained, or report explicit abandonment. Rejection notifications are retained - * while running, but a process crash or shutdown can lose pending callbacks. + * while running. On restart, a completed preserved copy reconstructs a notification + * only for its exact still-live recovered orphan range; a crash or shutdown can still + * lose callbacks before publication or after retirement. * *

{@link Category#PROTOCOL_VIOLATION} is forced {@link #TERMINAL}, * {@link Category#UNKNOWN} is forced {@link #RETRIABLE} (fail open: a diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 77f634b92..51dbdb9a2 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -40,8 +40,7 @@ import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; -import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver; -import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch; +import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerListener; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerPool; @@ -411,7 +410,7 @@ public class QwpWebSocketSender implements Sender { private SenderError.Policy schemaMismatchPolicy = SenderError.Policy.REJECT_AND_CONTINUE; private boolean dlqEnabled = true; private String dlqDir; - private SchemaPreserver schemaPreserver; + private RejectedMiniSlotArchive schemaPreserver; private SchemaRejectionState schemaRejectionState; private long schemaLeaseGeneration; private boolean schemaLeaseStarted; @@ -2928,19 +2927,16 @@ public void configureSchemaMismatch(SenderError.Policy policy, boolean preserve, && cursorEngine != null && (cursorEngine.sfDir() != null || directory != null)) { String source = cursorEngine.sfDir(); String slotId = source == null ? "memory" : java.nio.file.Paths.get(source).getFileName().toString(); - String epoch = source == null ? java.util.UUID.randomUUID().toString() - : SlotEpoch.openOrCreate(io.questdb.client.std.FilesFacade.INSTANCE, source, cursorEngine.freshFsnNamespace()); String destination = directory == null ? source : java.nio.file.Paths.get(directory, slotId).toString(); + String archiveNamespace = RejectedMiniSlotArchive.namespaceForSource(source); try { java.nio.file.Files.createDirectories(java.nio.file.Paths.get(destination)); } catch (java.io.IOException e) { throw new LineSenderException(e).put("could not create schema preservation destination ").put(destination); } - SchemaPreserver.probeDestination(io.questdb.client.std.FilesFacade.INSTANCE, destination); - io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive.cleanupTemporaryDirectories( - io.questdb.client.std.FilesFacade.INSTANCE, destination, slotId, epoch); - schemaPreserver = new SchemaPreserver(io.questdb.client.std.FilesFacade.INSTANCE, - destination, slotId, epoch); + RejectedMiniSlotArchive.probeDestination(io.questdb.client.std.FilesFacade.INSTANCE, destination); + schemaPreserver = new RejectedMiniSlotArchive(io.questdb.client.std.FilesFacade.INSTANCE, + destination, archiveNamespace); } } @@ -4291,7 +4287,7 @@ private void ensureConnected() { } cursorSendLoop.setSchemaRejectionState(schemaRejectionState); cursorSendLoop.setSchemaMismatchPolicy(schemaMismatchPolicy); - cursorSendLoop.setSchemaPreserver(schemaPreserver); + cursorSendLoop.setRejectionArchive(schemaPreserver); cursorSendLoop.setErrorDispatcher(errorDispatcher); // Symmetric progress dispatcher: lazy-allocated mirror of the // error path. Wired before start() for the same reason -- the @@ -4334,7 +4330,7 @@ private void ensureConnected() { } } if (t instanceof UnreplayableSlotException) { - // Startup also validates archives needed by orphan retirement. + // Preserve typed failures from live-queue recovery. // Sender.build() must receive this type to quarantine the slot; // wrapping it would make every build retry fail on the same bytes. throw (UnreplayableSlotException) t; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java index cbc2ba03b..f4fd8b681 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java @@ -934,7 +934,7 @@ public void run() { // per wire session. Closed by the finally, after loop.close(), so errors // dispatched during the loop's shutdown still reach the sink. SenderErrorDispatcher loopErrorDispatcher = null; - SchemaPreserver schemaPreserver = null; + RejectedMiniSlotArchive schemaPreserver = null; SchemaRejectionState schemaRejectionState = null; try { // Scanner results are only snapshots. Serialize adoption against @@ -1061,10 +1061,6 @@ public void run() { schemaRejectionState = new SchemaRejectionState(); if (schemaPreservationEnabled) { String slotId = java.nio.file.Paths.get(slotPath).getFileName().toString(); - String epoch = SlotEpoch.openOrCreate( - io.questdb.client.std.FilesFacade.INSTANCE, - slotPath, - engine.freshFsnNamespace()); String destination = schemaPreservationDirectory == null ? slotPath : java.nio.file.Paths.get(schemaPreservationDirectory, slotId).toString(); @@ -1074,15 +1070,11 @@ public void run() { throw new SfOperationalException( "could not create schema preservation destination " + destination, e); } - SchemaPreserver.probeDestination( + RejectedMiniSlotArchive.probeDestination( io.questdb.client.std.FilesFacade.INSTANCE, destination); - RejectedMiniSlotArchive.cleanupTemporaryDirectories( - io.questdb.client.std.FilesFacade.INSTANCE, destination, slotId, epoch); - schemaPreserver = new SchemaPreserver( - io.questdb.client.std.FilesFacade.INSTANCE, - destination, - slotId, - epoch); + schemaPreserver = new RejectedMiniSlotArchive( + io.questdb.client.std.FilesFacade.INSTANCE, destination, + RejectedMiniSlotArchive.namespaceForSource(slotPath)); } } if (logicalSlotLock != null) { @@ -1142,7 +1134,7 @@ public void run() { if (schemaPreserver != null && engine.recoveredOrphanTipFsn() >= 0 && engine.ackedFsn() >= engine.recoveredCommitBoundaryFsn()) { SenderError recovered = schemaPreserver.findRecoveredOrphanReport( - engine.recoveredCommitBoundaryFsn() + 1L, engine.recoveredOrphanTipFsn()); + engine, engine.recoveredCommitBoundaryFsn() + 1L, engine.recoveredOrphanTipFsn()); if (recovered != null && (loopErrorDispatcher == null || !loopErrorDispatcher.tryOfferSchema(recovered))) { lastErrorMessage = "could not retain recovered schema report before orphan retirement"; @@ -1209,7 +1201,7 @@ public void run() { loop.setErrorDispatcher(loopErrorDispatcher); loop.setSchemaRejectionState(schemaRejectionState); loop.setSchemaMismatchPolicy(schemaMismatchPolicy); - loop.setSchemaPreserver(schemaPreserver); + loop.setRejectionArchive(schemaPreserver); loop.start(); while (!stopRequestedOrInterrupted()) { diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java index d128578e9..a82e2c9e2 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java @@ -1302,18 +1302,6 @@ private void finishClose(boolean fullyDrained) { "could not fsync SF slot directory after segment cleanup"); } else { AckWatermark.removeOrphan(filesFacade, sfDir); - // The next engine starts a new FSN namespace. Keep its - // archive identity distinct from this drained one. - String epochPath = sfDir + '/' + SlotEpoch.FILE_NAME; - if (filesFacade.exists(epochPath)) { - if (!filesFacade.remove(epochPath)) { - durabilityFailure = new IllegalStateException( - "could not remove drained SF slot epoch"); - } else if (filesFacade.fsyncDir(sfDir) != 0) { - durabilityFailure = new IllegalStateException( - "could not fsync SF slot directory after epoch cleanup"); - } - } } } else { LOG.warn("close-time segment cleanup incomplete on slot {}; retaining the ack " @@ -1865,11 +1853,6 @@ public boolean wasRecoveredFromDisk() { return wasRecoveredFromDisk; } - /** True when construction created a new FSN namespace rather than recovering one. */ - public boolean freshFsnNamespace() { - return sfDir != null && !wasRecoveredFromDisk; - } - /** * FSN of the last commit-bearing frame in a disk-recovered ring, or * {@code -1} for fresh/memory rings. Frames above it are an orphaned diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index ca8b5d1ac..25f097426 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -326,7 +326,7 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { private final AtomicLong dlqWriteFailures = new AtomicLong(); private volatile SenderError.Policy schemaMismatchPolicy = SenderError.Policy.TERMINAL; private volatile SchemaRejectionState schemaRejectionState; - private volatile SchemaPreserver schemaPreserver; + private volatile RejectedMiniSlotArchive schemaPreserver; private SenderError preservedSchemaNotification; private long preparedSchemaFirstFsn = -1L; private long preparedSchemaLastFsn = -1L; @@ -1597,7 +1597,7 @@ public void setSchemaMismatchPolicy(SenderError.Policy policy) { this.schemaMismatchPolicy = policy; } - public void setSchemaPreserver(SchemaPreserver preserver) { + public void setRejectionArchive(RejectedMiniSlotArchive preserver) { this.schemaPreserver = preserver; } @@ -2294,10 +2294,6 @@ private void drainPendingDurable() { if (engine.acknowledge(fsn)) { totalDurableTrimAdvances.incrementAndGet(); dispatchProgress(fsn); - SchemaRejectionState state = schemaRejectionState; - if (state != null) { - state.acknowledgedThrough(fsn); - } } } } @@ -3791,11 +3787,11 @@ private boolean tryRetireOrphanTail() { if (engine.ackedFsn() < orphanSkipStartFsn - 1L) { return false; } - SchemaPreserver preserver = schemaPreserver; - if (preserver != null) { + RejectedMiniSlotArchive archive = schemaPreserver; + if (archive != null) { if (!recoveredOrphanReportLookedUp) { - recoveredOrphanReport = preserver.findRecoveredOrphanReport( - orphanSkipStartFsn, orphanSkipTipFsn); + recoveredOrphanReport = archive.findRecoveredOrphanReport( + engine, orphanSkipStartFsn, orphanSkipTipFsn); recoveredOrphanReportLookedUp = true; } if (recoveredOrphanReport != null) { @@ -3830,18 +3826,18 @@ private boolean tryRetireSchemaRange() { SenderError notification = preservedSchemaNotification != null ? preservedSchemaNotification : range.error; - SchemaPreserver preserver = schemaPreserver; + RejectedMiniSlotArchive preserver = schemaPreserver; + try { + prepareSkippedRange(range); + } catch (LineSenderException e) { + LOG.error("could not retain dictionary coverage for schema-rejected range [{}, {}]; " + + "keeping queued bytes and stopping the sender", + range.firstFsn, range.lastFsn, e); + recordFatal(e); + dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); + return false; + } if (preserver != null && preservedSchemaNotification == null) { - try { - prepareSkippedRange(range); - } catch (LineSenderException e) { - LOG.error("could not retain dictionary coverage for schema-rejected range [{}, {}]; " - + "keeping queued bytes and stopping the sender", - range.firstFsn, range.lastFsn, e); - recordFatal(e); - dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); - return false; - } byte[] dictionary = snapshotSentDictionary(); final RejectedMiniSlotArchive.Result result; try { @@ -3881,17 +3877,6 @@ private boolean tryRetireSchemaRange() { // the existing delegated I/O-thread cleanup release the engine. return false; } - } else { - try { - prepareSkippedRange(range); - } catch (LineSenderException e) { - LOG.error("could not retain dictionary coverage for schema-rejected range [{}, {}]; " - + "keeping queued bytes and stopping the sender", - range.firstFsn, range.lastFsn, e); - recordFatal(e); - dispatchError(range.error.withAppliedPolicy(SenderError.Policy.TERMINAL)); - return false; - } } if (!dispatcher.tryOfferSchema(notification)) { return false; @@ -4240,10 +4225,6 @@ public void onBinaryMessage(long payloadPtr, int payloadLen) { long ackFsn = clampAckBeforeSchemaStop(fsnAtZero + capped); if (engine.acknowledge(ackFsn)) { dispatchProgress(ackFsn); - SchemaRejectionState state = schemaRejectionState; - if (state != null) { - state.acknowledgedThrough(ackFsn); - } } return; } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java index b7c6dfb62..41c6893e6 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/MmapSegment.java @@ -36,7 +36,9 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.security.SecureRandom; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; /** * One mmap-backed SF segment file. The user thread (the single producer) @@ -49,7 +51,7 @@ * On-disk layout — header and frame format: * 

  *   [u32 magic 'SF01'] [u8 ver=1] [u8 flags]   [u16 reserved=0]
- *   [u64 baseSeq]      [u64 createdMicros]                        24-byte header
+ *   [u64 baseSeq]      [u64 generationToken]                     24-byte header
  *   frame, frame, ...                                              each frame:
  *                                                                  [u32 crc32c]
  *                                                                  [u32 payloadLen]
@@ -76,6 +78,11 @@ public final class MmapSegment implements QuietCloseable {
     // soft downgrade (see syncPublished) and must not spam the log once per
     // barrier when RLIMIT_MEMLOCK or the platform says no.
     private static final AtomicBoolean MLOCK_REFUSAL_WARNED = new AtomicBoolean();
+    // Consecutive values cannot repeat within one JVM before 64-bit wrap. A
+    // cryptographically random starting point makes a collision with a token
+    // persisted by another process a 1-in-2^64 event for any fixed token.
+    private static final AtomicLong NEXT_GENERATION_TOKEN =
+            new AtomicLong(new SecureRandom().nextLong());
     private static final int RECOVERY_BUFFER_SIZE = 64 * 1024;
 
     private final FilesFacade filesFacade;
@@ -248,7 +255,7 @@ static MmapSegment create(FilesFacade ff, long pathPtr, String displayPath, long
             Unsafe.getUnsafe().putByte(addr + 5, manifestRequired ? MANIFEST_REQUIRED_FLAG : (byte) 0); // flags
             Unsafe.getUnsafe().putShort(addr + 6, (short) 0); // reserved
             Unsafe.getUnsafe().putLong(addr + 8, baseSeq);
-            Unsafe.getUnsafe().putLong(addr + 16, Os.currentTimeMicros());
+            Unsafe.getUnsafe().putLong(addr + 16, nextGenerationToken());
             return new MmapSegment(ff, displayPath, fd, addr, sizeBytes, baseSeq,
                     HEADER_SIZE, 0, false, 0L);
         } catch (Throwable t) {
@@ -286,7 +293,7 @@ public static MmapSegment createInMemory(long baseSeq, long sizeBytes) {
             Unsafe.getUnsafe().putByte(addr + 5, (byte) 0);
             Unsafe.getUnsafe().putShort(addr + 6, (short) 0);
             Unsafe.getUnsafe().putLong(addr + 8, baseSeq);
-            Unsafe.getUnsafe().putLong(addr + 16, Os.currentTimeMicros());
+            Unsafe.getUnsafe().putLong(addr + 16, nextGenerationToken());
             return new MmapSegment(null, null, -1, addr, sizeBytes, baseSeq,
                     HEADER_SIZE, 0, true, 0L);
         } catch (Throwable t) {
@@ -759,6 +766,15 @@ public long frameCount() {
         return frameCount;
     }
 
+    /** Immutable, opaque segment generation token stored in the segment header. */
+    public long generationToken() {
+        return Unsafe.getUnsafe().getLong(mmapAddress + 16);
+    }
+
+    private static long nextGenerationToken() {
+        return NEXT_GENERATION_TOKEN.getAndIncrement();
+    }
+
     int liveFramePayloadLength(long fsn) {
         long offset = liveFrameOffset(fsn);
         return offset < 0 ? -1 : Unsafe.getUnsafe().getInt(mmapAddress + offset + 4);
diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java
index 6570c7eab..183876a71 100644
--- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java
+++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java
@@ -698,59 +698,6 @@ public long appendedBytes() {
         return appendOffset;
     }
 
-    /**
-     * Copies the complete committed dictionary prefix into a fresh dictionary
-     * file under {@code targetDir} and makes that copy durable. The source
-     * monitor is held while the prefix boundary and bytes are copied, so a
-     * concurrent symbol append is wholly before or wholly after the snapshot.
-     *
-     * @return number of bytes written, including the dictionary header
-     */
-    public synchronized long snapshotTo(String targetDir) {
-        if (closed) {
-            throw new IllegalStateException("symbol dictionary is closed");
-        }
-        String targetPath = targetDir + "/" + FILE_NAME;
-        int targetFd = ff.openRWExclusive(targetPath);
-        if (targetFd < 0) {
-            throw new SfOperationalException("could not create symbol dictionary snapshot " + targetPath);
-        }
-        long copyLen = appendOffset;
-        long scratch = 0L;
-        boolean success = false;
-        try {
-            if (!ff.allocate(targetFd, copyLen)) {
-                throw new SfOperationalException("could not allocate symbol dictionary snapshot " + targetPath);
-            }
-            int scratchSize = (int) Math.min(64 * 1024L, Math.max(copyLen, 1L));
-            scratch = Unsafe.malloc(scratchSize, MemoryTag.NATIVE_DEFAULT);
-            long offset = 0L;
-            while (offset < copyLen) {
-                int chunk = (int) Math.min(scratchSize, copyLen - offset);
-                if (ff.read(fd, scratch, chunk, offset) != chunk) {
-                    throw new SfOperationalException("short read copying symbol dictionary " + filePath);
-                }
-                if (ff.write(targetFd, scratch, chunk, offset) != chunk) {
-                    throw new SfOperationalException("short write copying symbol dictionary snapshot " + targetPath);
-                }
-                offset += chunk;
-            }
-            if (ff.fsync(targetFd) != 0) {
-                throw new SfOperationalException("could not sync symbol dictionary snapshot " + targetPath);
-            }
-            success = true;
-            return copyLen;
-        } finally {
-            if (scratch != 0L) {
-                Unsafe.free(scratch, (int) Math.min(64 * 1024L, Math.max(copyLen, 1L)), MemoryTag.NATIVE_DEFAULT);
-            }
-            ff.close(targetFd);
-            if (!success) {
-                ff.remove(targetPath);
-            }
-        }
-    }
-
     /**
      * Base address of the loaded entry region -- the concatenated
      * {@code [len][utf8]} bytes of every recovered symbol in id order, exactly as a
diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java
index 4b3a01dbb..168fef816 100644
--- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java
+++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchive.java
@@ -7,115 +7,154 @@
 
 import io.questdb.client.SenderError;
 import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
-import io.questdb.client.std.Crc32c;
-import io.questdb.client.std.Files;
 import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
 import io.questdb.client.std.Unsafe;
 
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.LinkOption;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Properties;
 import java.util.UUID;
 
-/** Builds and validates immutable, replay-format copies of rejected SF spans. */
+/** Writes immutable rejected ranges in the existing SFA replay format. */
 public final class RejectedMiniSlotArchive {
-    public static final String METADATA_FILE_NAME = "rejection-meta.bin";
+    public static final String METADATA_FILE_NAME = "rejection.properties";
     public static final String SEGMENT_FILE_NAME = "rejected.sfa";
-    private static final int METADATA_MAGIC = 0x314a4552; // REJ1 little-endian
-    private static final int METADATA_VERSION = 1;
     private static final int MODE_OWNER_ONLY = 448; // 0700
 
-    private RejectedMiniSlotArchive() {
+    private final String directory;
+    private final FilesFacade ff;
+    private final String namespace;
+    // A published directory awaiting its parent fsync. Retry only that barrier.
+    private Result pending;
+
+    public RejectedMiniSlotArchive(FilesFacade ff, String directory) {
+        this(ff, directory, namespaceForSource(directory));
     }
 
-    public static Result preserve(
-            FilesFacade ff,
-            CursorSendEngine engine,
-            PersistedSymbolDict dictionary,
-            String slotDir,
-            String slotId,
-            String epoch,
-            SenderError error
-    ) {
-        return preserve0(ff, engine, dictionary, null, 0, slotDir, slotId, epoch, error);
+    public RejectedMiniSlotArchive(FilesFacade ff, String directory, String namespace) {
+        this.ff = ff;
+        this.directory = directory;
+        this.namespace = UUID.fromString(namespace).toString();
     }
 
-    public static Result preserveSnapshot(
-            FilesFacade ff,
-            CursorSendEngine engine,
-            byte[] dictionaryEntries,
-            int dictionaryCount,
-            String slotDir,
-            String slotId,
-            String epoch,
-            SenderError error
-    ) {
-        if ((dictionaryEntries == null) != (dictionaryCount == 0)) {
-            throw new IllegalArgumentException("dictionary snapshot bytes/count mismatch");
+    /** Stable for a disk queue path; unique for each memory-only queue. */
+    public static String namespaceForSource(String source) {
+        if (source == null) return UUID.randomUUID().toString();
+        String path = Paths.get(source).toAbsolutePath().normalize().toString();
+        return UUID.nameUUIDFromBytes(path.getBytes(StandardCharsets.UTF_8)).toString();
+    }
+
+    /** Returns a structurally valid report for exactly this live range, if one was published. */
+    public SenderError findRecoveredOrphanReport(CursorSendEngine engine, long fromFsn, long toFsn) {
+        if (fromFsn < 0 || toFsn < fromFsn) return null;
+        try {
+            PathsForRange paths = paths(engine, fromFsn, toFsn);
+            removeKnownDirectory(paths.temp);
+            return readReport(Paths.get(paths.completed), fromFsn, toFsn);
+        } catch (IOException | RuntimeException ignored) {
+            // Archive output must never prevent live-queue recovery.
+            return null;
         }
-        return preserve0(ff, engine, null, dictionaryEntries, dictionaryCount,
-                slotDir, slotId, epoch, error);
     }
 
-    private static Result preserve0(
-            FilesFacade ff,
-            CursorSendEngine engine,
-            PersistedSymbolDict dictionary,
-            byte[] dictionaryEntries,
-            int dictionaryCount,
-            String slotDir,
-            String slotId,
-            String epoch,
-            SenderError error
-    ) {
+    /** I/O-thread only. Source frames remain live until this returns successfully. */
+    public Result preserve(CursorSendEngine engine, SenderError error,
+                           byte[] dictionaryEntries, int dictionaryCount) {
+        if (pending == null) pending = write(engine, error, dictionaryEntries, dictionaryCount);
+        if (ff.fsyncDir(directory + "/rejected") != 0) {
+            throw new SfOperationalException("could not sync rejection archive parent " + directory);
+        }
+        Result result = pending;
+        pending = null;
+        return result;
+    }
+
+    private Result write(CursorSendEngine engine, SenderError error,
+                         byte[] dictionaryEntries, int dictionaryCount) {
+        if ((dictionaryEntries == null) != (dictionaryCount == 0)) {
+            throw new IllegalArgumentException("dictionary snapshot bytes/count mismatch");
+        }
         long from = error.getFromFsn();
         long to = error.getToFsn();
         if (from < 0 || to < from || error.getRejectedFsn() < from || error.getRejectedFsn() > to) {
             throw new IllegalArgumentException("invalid rejection span");
         }
-        requirePathComponent(slotId, "slot id");
-        UUID.fromString(epoch);
-        String rejectedRoot = slotDir + "/rejected";
-        ensureDirectory(ff, rejectedRoot);
-        String identity = slotId + '-' + epoch + "-fsn-" + from + '-' + to;
-        String finalDir = rejectedRoot + '/' + identity;
-        Metadata expected = Metadata.from(slotId, epoch, error,
-                dictionary != null || dictionaryEntries != null);
-        if (ff.exists(finalDir)) {
-            validate(ff, finalDir, expected);
-            // Completes a previous publication whose rename succeeded but
-            // whose parent-directory barrier failed transiently.
-            if (ff.fsyncDir(rejectedRoot) != 0) {
-                throw new SfOperationalException("could not sync rejected mini-slot parent " + rejectedRoot);
+        String rejectedRoot = directory + "/rejected";
+        ensureDirectory(rejectedRoot);
+        if (ff.fsyncDir(directory) != 0) {
+            throw new SfOperationalException("could not sync archive destination " + directory);
+        }
+        PathsForRange paths = paths(engine, from, to);
+        try {
+            if (readReport(Paths.get(paths.completed), from, to) != null) {
+                return new Result(paths.completed, 0, true);
             }
-            return new Result(finalDir, occupiedBytes(ff, finalDir, expected.hasDictionary), true);
+        } catch (IOException | RuntimeException ignored) {
+            // Replace only this exact range identity while its source is still live.
+        }
+        removeKnownDirectory(paths.completed);
+        removeKnownDirectory(paths.temp);
+        if (ff.exists(paths.completed) || ff.exists(paths.temp)
+                || ff.mkdir(paths.temp, MODE_OWNER_ONLY) != 0) {
+            throw new SfOperationalException("could not create archive staging directory " + paths.temp);
         }
 
-        String tempDir = rejectedRoot + "/.tmp-" + identity + '-' + UUID.randomUUID();
-        ensureDirectory(ff, tempDir);
-        long totalSize = MmapSegment.HEADER_SIZE;
-        int maxPayload = 0;
-        for (long fsn = from; fsn <= to; fsn++) {
-            int len = engine.liveFramePayloadLength(fsn);
-            if (len < QwpConstants.HEADER_SIZE) {
-                throw new SfOperationalException("rejection frame is no longer live [fsn=" + fsn + ']');
+        boolean published = false;
+        try {
+            int maxPayload = 0;
+            long totalSize = MmapSegment.HEADER_SIZE;
+            for (long fsn = from; fsn <= to; fsn++) {
+                int len = engine.liveFramePayloadLength(fsn);
+                if (len < QwpConstants.HEADER_SIZE) {
+                    throw new SfOperationalException("rejection frame is no longer live [fsn=" + fsn + ']');
+                }
+                totalSize = Math.addExact(totalSize, MmapSegment.FRAME_HEADER_SIZE + (long) len);
+                maxPayload = Math.max(maxPayload, len);
+                if (fsn == Long.MAX_VALUE) break;
+            }
+            copyFrames(engine, paths.temp, from, to, totalSize, maxPayload);
+            try (SfManifest ignored = SfManifest.create(ff, paths.temp, from, from)) {
+                // create() durably writes the sole boundary record.
+            }
+            try (AckWatermark watermark = AckWatermark.open(ff, paths.temp)) {
+                if (watermark == null) throw new SfOperationalException("could not create rejection ack watermark");
+                watermark.write(from - 1L);
+                watermark.sync();
             }
-            totalSize = Math.addExact(totalSize, MmapSegment.FRAME_HEADER_SIZE + (long) len);
-            maxPayload = Math.max(maxPayload, len);
-            if (fsn == Long.MAX_VALUE) break;
+            if (dictionaryEntries != null) writeDictionary(paths.temp, dictionaryEntries, dictionaryCount);
+            writeMetadata(paths.temp, error, dictionaryEntries != null);
+            Result result = new Result(paths.completed,
+                    occupiedBytes(paths.temp, dictionaryEntries != null), false);
+            if (ff.fsyncDir(paths.temp) != 0 || ff.rename(paths.temp, paths.completed) != 0) {
+                throw new SfOperationalException("could not publish rejected mini-slot " + paths.completed);
+            }
+            published = true;
+            return result;
+        } finally {
+            if (!published) removeKnownDirectory(paths.temp);
         }
+    }
 
+    private void copyFrames(CursorSendEngine engine, String target, long from, long to,
+                            long totalSize, int maxPayload) {
         long scratch = Unsafe.malloc(maxPayload, MemoryTag.NATIVE_DEFAULT);
         try (MmapSegment segment = MmapSegment.create(
-                ff, tempDir + '/' + SEGMENT_FILE_NAME, from, totalSize, true)) {
+                ff, target + '/' + SEGMENT_FILE_NAME, from, totalSize, true)) {
             for (long fsn = from; fsn <= to; fsn++) {
                 int len = engine.liveFramePayloadLength(fsn);
                 if (len < 0 || len > maxPayload || !engine.copyLiveFrame(fsn, scratch, maxPayload)) {
                     throw new SfOperationalException("rejection frame disappeared during copy [fsn=" + fsn + ']');
                 }
                 if (fsn == to) {
-                    long flagsAddr = scratch + QwpConstants.HEADER_OFFSET_FLAGS;
-                    byte flags = Unsafe.getUnsafe().getByte(flagsAddr);
-                    Unsafe.getUnsafe().putByte(flagsAddr, (byte) (flags & ~QwpConstants.FLAG_DEFER_COMMIT));
+                    long flags = scratch + QwpConstants.HEADER_OFFSET_FLAGS;
+                    Unsafe.getUnsafe().putByte(flags, (byte) (Unsafe.getUnsafe().getByte(flags)
+                            & ~QwpConstants.FLAG_DEFER_COMMIT));
                 }
                 if (segment.tryAppend(scratch, len) < 0) {
                     throw new SfOperationalException("rejection segment sizing changed during copy");
@@ -126,386 +165,168 @@ private static Result preserve0(
         } finally {
             Unsafe.free(scratch, maxPayload, MemoryTag.NATIVE_DEFAULT);
         }
-
-        try (SfManifest ignored = SfManifest.create(ff, tempDir, from, from)) {
-            // create() durably writes the sole boundary record.
-        }
-        try (AckWatermark watermark = AckWatermark.open(ff, tempDir)) {
-            if (watermark == null) {
-                throw new SfOperationalException("could not create rejection ack watermark");
-            }
-            watermark.write(from - 1L);
-            watermark.sync();
-        }
-        if (dictionary != null) {
-            dictionary.snapshotTo(tempDir);
-        } else if (dictionaryEntries != null) {
-            long entriesAddr = Unsafe.malloc(dictionaryEntries.length, MemoryTag.NATIVE_DEFAULT);
-            try (PersistedSymbolDict snapshot = PersistedSymbolDict.openClean(ff, tempDir)) {
-                if (snapshot == null) {
-                    throw new SfOperationalException("could not create rejected mini-slot dictionary");
-                }
-                Unsafe.getUnsafe().copyMemory(dictionaryEntries, Unsafe.BYTE_OFFSET, null,
-                        entriesAddr, dictionaryEntries.length);
-                snapshot.appendRawEntries(entriesAddr, dictionaryEntries.length, dictionaryCount);
-            } finally {
-                Unsafe.free(entriesAddr, dictionaryEntries.length, MemoryTag.NATIVE_DEFAULT);
-            }
-        }
-        writeMetadata(ff, tempDir, expected);
-        if (ff.fsyncDir(tempDir) != 0 || ff.rename(tempDir, finalDir) != 0
-                || ff.fsyncDir(rejectedRoot) != 0) {
-            throw new SfOperationalException("could not publish rejected mini-slot " + finalDir);
-        }
-        validate(ff, finalDir, expected);
-        return new Result(finalDir, occupiedBytes(ff, finalDir, expected.hasDictionary), false);
-    }
-
-    /**
-     * Copies a validated archive into a new working directory. Replay may
-     * consume that directory; the immutable archive is never adopted or moved.
-     */
-    public static void copyToWorkingDirectory(FilesFacade ff, String archiveDir, String workingDir) {
-        Metadata metadata = readMetadata(ff, archiveDir);
-        validate(ff, archiveDir, metadata);
-        if (ff.exists(workingDir)) {
-            throw new IllegalArgumentException("working directory already exists: " + workingDir);
-        }
-        ensureDirectory(ff, workingDir);
-        copyFile(ff, archiveDir, workingDir, SEGMENT_FILE_NAME);
-        copyFile(ff, archiveDir, workingDir, SfManifest.FILE_NAME);
-        copyFile(ff, archiveDir, workingDir, AckWatermark.FILE_NAME);
-        copyFile(ff, archiveDir, workingDir, METADATA_FILE_NAME);
-        if (metadata.hasDictionary) {
-            copyFile(ff, archiveDir, workingDir, PersistedSymbolDict.FILE_NAME);
-        }
-        if (ff.fsyncDir(workingDir) != 0) {
-            throw new SfOperationalException("could not sync replay working directory " + workingDir);
-        }
-    }
-
-    public static Metadata readMetadata(FilesFacade ff, String dir) {
-        String path = dir + '/' + METADATA_FILE_NAME;
-        long len = ff.length(path);
-        if (len < 76 || len > Integer.MAX_VALUE) {
-            throw new UnreplayableSlotException("invalid rejection metadata size " + path);
-        }
-        long mem = Unsafe.malloc(len, MemoryTag.NATIVE_DEFAULT);
-        int fd = ff.openRW(path);
-        try {
-            if (fd < 0 || ff.read(fd, mem, len, 0) != len) {
-                throw new SfOperationalException("could not read rejection metadata " + path);
-            }
-            int storedCrc = Unsafe.getUnsafe().getInt(mem + len - 4);
-            if (storedCrc != Crc32c.update(Crc32c.INIT, mem, len - 4)) {
-                throw new UnreplayableSlotException("rejection metadata CRC mismatch " + path);
-            }
-            long p = mem;
-            if (Unsafe.getUnsafe().getInt(p) != METADATA_MAGIC
-                    || Unsafe.getUnsafe().getInt(p + 4) != METADATA_VERSION) {
-                throw new UnreplayableSlotException("unsupported rejection metadata " + path);
-            }
-            p += 8;
-            long rejected = Unsafe.getUnsafe().getLong(p); p += 8;
-            long from = Unsafe.getUnsafe().getLong(p); p += 8;
-            long to = Unsafe.getUnsafe().getLong(p); p += 8;
-            long detected = Unsafe.getUnsafe().getLong(p); p += 8;
-            int status = Unsafe.getUnsafe().getInt(p); p += 4;
-            boolean hasDictionary = Unsafe.getUnsafe().getInt(p) != 0; p += 4;
-            String slotId = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            String epoch = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            String category = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            String policy = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            String table = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            String message = readString(mem, len - 4, p); p += 4 + utf8LengthAt(mem, p);
-            if (p != mem + len - 4) {
-                throw new UnreplayableSlotException("trailing rejection metadata bytes " + path);
-            }
-            return new Metadata(slotId, epoch, rejected, from, to, detected, status,
-                    category, policy, table, message, hasDictionary);
-        } finally {
-            if (fd >= 0) ff.close(fd);
-            Unsafe.free(mem, len, MemoryTag.NATIVE_DEFAULT);
-        }
-    }
-
-    /**
-     * Finds the preserved report for a recovered orphan tail. Only completed
-     * directories belonging to this slot epoch are considered; temporary
-     * directories are never evidence.
-     */
-    public static SenderError findOverlapping(
-            FilesFacade ff, String slotDir, String slotId, String epoch, long fromFsn, long toFsn
-    ) {
-        requirePathComponent(slotId, "slot id");
-        UUID.fromString(epoch);
-        String rejectedRoot = slotDir + "/rejected";
-        long find = ff.findFirst(rejectedRoot);
-        if (find <= 0) {
-            if (find > 0) ff.findClose(find);
-            return null;
-        }
-        String prefix = slotId + '-' + epoch + "-fsn-";
-        try {
-            int rc = 1;
-            while (rc > 0) {
-                String name = Files.utf8ToString(ff.findName(find));
-                int type = ff.findType(find);
-                rc = ff.findNext(find);
-                if (type != Files.DT_DIR || name == null || !name.startsWith(prefix)) continue;
-                int separator = name.indexOf('-', prefix.length());
-                if (separator < 0) continue;
-                long archiveFrom;
-                long archiveTo;
-                try {
-                    archiveFrom = Long.parseLong(name.substring(prefix.length(), separator));
-                    archiveTo = Long.parseLong(name.substring(separator + 1));
-                } catch (NumberFormatException e) {
-                    continue;
-                }
-                // Completed archives have canonical range names. Filter before
-                // opening metadata: a damaged, already-drained archive is not
-                // evidence about this recovered tail and must not block startup.
-                if (archiveFrom < 0 || archiveTo < archiveFrom
-                        || !name.equals(prefix + archiveFrom + '-' + archiveTo)
-                        || archiveTo < fromFsn || archiveFrom > toFsn) {
-                    continue;
-                }
-                String path = rejectedRoot + '/' + name;
-                Metadata metadata = readMetadata(ff, path);
-                if (!slotId.equals(metadata.slotId) || !epoch.equals(metadata.epoch)
-                        || metadata.fromFsn != archiveFrom || metadata.toFsn != archiveTo) {
-                    throw new UnreplayableSlotException("rejected mini-slot directory identity mismatch " + path);
-                }
-                validate(ff, path, metadata);
-                SenderError error = new SenderError(
-                        SenderError.Category.valueOf(metadata.category),
-                        SenderError.Policy.valueOf(metadata.policy), metadata.status,
-                        metadata.message, metadata.rejectedFsn, metadata.rejectedFsn,
-                        metadata.rejectedFsn, metadata.table.isEmpty() ? null : metadata.table,
-                        metadata.detectedAtNanos);
-                return error.withRejectionSpan(metadata.fromFsn, metadata.toFsn)
-                        .withRejectedPath(path);
-            }
-        } finally {
-            ff.findClose(find);
-        }
-        return null;
     }
 
-    /**
-     * Removes incomplete publications for exactly one slot epoch. The caller
-     * must hold that queue's exclusive lifecycle lock; the identity prefix is
-     * what keeps shared memory-sender destinations from touching one another.
-     */
-    public static void cleanupTemporaryDirectories(
-            FilesFacade ff, String slotDir, String slotId, String epoch
-    ) {
-        requirePathComponent(slotId, "slot id");
-        UUID.fromString(epoch);
-        String rejectedRoot = slotDir + "/rejected";
-        long find = ff.findFirst(rejectedRoot);
-        if (find <= 0) return;
-        String prefix = ".tmp-" + slotId + '-' + epoch + "-fsn-";
-        java.util.ArrayList candidates = new java.util.ArrayList<>();
-        try {
-            int rc = 1;
-            while (rc > 0) {
-                String name = Files.utf8ToString(ff.findName(find));
-                int type = ff.findType(find);
-                rc = ff.findNext(find);
-                if (type == Files.DT_DIR && name != null && name.startsWith(prefix)) {
-                    candidates.add(rejectedRoot + '/' + name);
-                }
-            }
+    private void writeDictionary(String target, byte[] entries, int count) {
+        long address = Unsafe.malloc(entries.length, MemoryTag.NATIVE_DEFAULT);
+        try (PersistedSymbolDict dictionary = PersistedSymbolDict.openClean(ff, target)) {
+            if (dictionary == null) throw new SfOperationalException("could not create rejected mini-slot dictionary");
+            Unsafe.getUnsafe().copyMemory(entries, Unsafe.BYTE_OFFSET, null, address, entries.length);
+            dictionary.appendRawEntries(address, entries.length, count);
         } finally {
-            ff.findClose(find);
-        }
-        for (String candidate : candidates) {
-            removeKnownTemporaryContents(ff, candidate);
+            Unsafe.free(address, entries.length, MemoryTag.NATIVE_DEFAULT);
         }
-        if (!candidates.isEmpty() && ff.fsyncDir(rejectedRoot) != 0) {
-            throw new SfOperationalException("could not sync rejected temporary cleanup " + rejectedRoot);
-        }
-    }
-
-    private static void removeKnownTemporaryContents(FilesFacade ff, String dir) {
-        String[] names = {SEGMENT_FILE_NAME, SfManifest.FILE_NAME, AckWatermark.FILE_NAME,
-                PersistedSymbolDict.FILE_NAME, METADATA_FILE_NAME};
-        for (String name : names) ff.remove(dir + '/' + name);
-        // remove() maps to unlink/rmdir. It deliberately fails if an unknown
-        // file appeared, preserving rather than broadening deletion scope.
-        ff.remove(dir);
     }
 
-    private static void validate(FilesFacade ff, String dir, Metadata expected) {
-        Metadata actual = readMetadata(ff, dir);
-        if (!expected.sameIdentity(actual)) {
-            throw new UnreplayableSlotException("rejected mini-slot identity mismatch " + dir);
+    private SenderError readReport(Path archive, long fromFsn, long toFsn) throws IOException {
+        LinkOption[] noFollow = {LinkOption.NOFOLLOW_LINKS};
+        Path metadataFile = archive.resolve(METADATA_FILE_NAME);
+        if (!java.nio.file.Files.isDirectory(archive, noFollow)
+                || !java.nio.file.Files.isRegularFile(metadataFile, noFollow)
+                || java.nio.file.Files.size(metadataFile) > 64 * 1024L) return null;
+        Properties metadata = new Properties();
+        try (InputStream input = java.nio.file.Files.newInputStream(metadataFile)) {
+            metadata.load(input);
         }
+        if (!"1".equals(metadata.getProperty("version"))
+                || Long.parseLong(metadata.getProperty("fromFsn")) != fromFsn
+                || Long.parseLong(metadata.getProperty("toFsn")) != toFsn
+                || !SenderError.Category.SCHEMA_MISMATCH.name().equals(metadata.getProperty("category"))
+                || !SenderError.Policy.REJECT_AND_CONTINUE.name().equals(metadata.getProperty("policy"))) return null;
+        long rejectedFsn = Long.parseLong(metadata.getProperty("rejectedFsn"));
+        if (rejectedFsn < fromFsn || rejectedFsn > toFsn) return null;
+        String dictionary = metadata.getProperty("dictionary");
+        if (!("true".equals(dictionary) || "false".equals(dictionary))) return null;
+        String dir = archive.toString();
+        if (Boolean.parseBoolean(dictionary) != ff.exists(dir + '/' + PersistedSymbolDict.FILE_NAME)) return null;
         try (MmapSegment segment = MmapSegment.openExisting(ff, dir + '/' + SEGMENT_FILE_NAME);
              SfManifest manifest = SfManifest.open(ff, dir);
              AckWatermark watermark = AckWatermark.open(ff, dir)) {
-            if (segment.baseSeq() != actual.fromFsn
-                    || segment.frameCount() != actual.toFsn - actual.fromFsn + 1
-                    || manifest == null || manifest.headBase() != actual.fromFsn
-                    || manifest.activeBase() != actual.fromFsn
-                    || watermark == null || watermark.read() != actual.fromFsn - 1L) {
-                throw new UnreplayableSlotException("invalid rejected mini-slot boundaries " + dir);
-            }
-        } catch (MmapSegmentCorruptionException e) {
-            // Positively identified archive corruption is a terminal recovery
-            // verdict too. Operational read/mmap failures retain their type.
-            UnreplayableSlotException failure = new UnreplayableSlotException(
-                    "corrupt rejected mini-slot " + dir + ": " + e.getMessage());
-            failure.initCause(e);
-            throw failure;
+            if (segment.baseSeq() != fromFsn || segment.frameCount() != toFsn - fromFsn + 1L
+                    || manifest == null || manifest.headBase() != fromFsn || manifest.activeBase() != fromFsn
+                    || watermark == null || watermark.read() != fromFsn - 1L) return null;
         }
-        if (actual.hasDictionary) {
-            try (PersistedSymbolDict ignored = PersistedSymbolDict.open(ff, dir)) {
-                if (ignored == null) {
-                    throw new UnreplayableSlotException("missing rejected mini-slot dictionary " + dir);
-                }
+        if (Boolean.parseBoolean(dictionary)) {
+            try (PersistedSymbolDict persisted = PersistedSymbolDict.open(ff, dir)) {
+                if (persisted == null) return null;
             }
         }
+        String detected = metadata.getProperty("detectedAtNanos");
+        return new SenderError(SenderError.Category.SCHEMA_MISMATCH,
+                SenderError.Policy.REJECT_AND_CONTINUE,
+                Integer.parseInt(metadata.getProperty("status")), metadata.getProperty("message"),
+                rejectedFsn, rejectedFsn, rejectedFsn, metadata.getProperty("table"),
+                detected == null ? 0L : Long.parseLong(detected))
+                .withRejectionSpan(fromFsn, toFsn).withRejectedPath(dir);
     }
 
-    private static void writeMetadata(FilesFacade ff, String dir, Metadata metadata) {
-        byte[] slot = metadata.slotId.getBytes(StandardCharsets.UTF_8);
-        byte[] epoch = metadata.epoch.getBytes(StandardCharsets.UTF_8);
-        byte[] category = metadata.category.getBytes(StandardCharsets.UTF_8);
-        byte[] policy = metadata.policy.getBytes(StandardCharsets.UTF_8);
-        byte[] table = bytes(metadata.table);
-        byte[] message = bytes(metadata.message);
-        int len = 48 + 4 + slot.length + 4 + epoch.length + 4 + category.length
-                + 4 + policy.length + 4 + table.length + 4 + message.length + 4;
-        long mem = Unsafe.malloc(len, MemoryTag.NATIVE_DEFAULT);
+    private void writeMetadata(String dir, SenderError error, boolean dictionary) {
+        Properties metadata = new Properties();
+        metadata.setProperty("version", "1");
+        metadata.setProperty("fromFsn", Long.toString(error.getFromFsn()));
+        metadata.setProperty("toFsn", Long.toString(error.getToFsn()));
+        metadata.setProperty("rejectedFsn", Long.toString(error.getRejectedFsn()));
+        metadata.setProperty("status", Integer.toString(error.getServerStatusByte()));
+        metadata.setProperty("detectedAtNanos", Long.toString(error.getDetectedAtNanos()));
+        metadata.setProperty("category", error.getCategory().name());
+        metadata.setProperty("policy", error.getAppliedPolicy().name());
+        metadata.setProperty("dictionary", Boolean.toString(dictionary));
+        if (error.getTableName() != null) metadata.setProperty("table", error.getTableName());
+        if (error.getServerMessage() != null) metadata.setProperty("message", error.getServerMessage());
+        ByteArrayOutputStream output = new ByteArrayOutputStream();
+        try {
+            metadata.store(output, "Preserved schema rejection; replay a working copy after fixing the schema");
+        } catch (IOException e) {
+            throw new SfOperationalException("could not encode rejection metadata", e);
+        }
+        byte[] bytes = output.toByteArray();
+        long address = Unsafe.malloc(bytes.length, MemoryTag.NATIVE_DEFAULT);
         int fd = -1;
-        String path = dir + '/' + METADATA_FILE_NAME;
         try {
-            Unsafe.getUnsafe().setMemory(mem, len, (byte) 0);
-            long p = mem;
-            Unsafe.getUnsafe().putInt(p, METADATA_MAGIC); Unsafe.getUnsafe().putInt(p + 4, METADATA_VERSION); p += 8;
-            Unsafe.getUnsafe().putLong(p, metadata.rejectedFsn); p += 8;
-            Unsafe.getUnsafe().putLong(p, metadata.fromFsn); p += 8;
-            Unsafe.getUnsafe().putLong(p, metadata.toFsn); p += 8;
-            Unsafe.getUnsafe().putLong(p, metadata.detectedAtNanos); p += 8;
-            Unsafe.getUnsafe().putInt(p, metadata.status); p += 4;
-            Unsafe.getUnsafe().putInt(p, metadata.hasDictionary ? 1 : 0); p += 4;
-            p = writeString(p, slot); p = writeString(p, epoch); p = writeString(p, category);
-            p = writeString(p, policy); p = writeString(p, table); p = writeString(p, message);
-            Unsafe.getUnsafe().putInt(p, Crc32c.update(Crc32c.INIT, mem, len - 4));
-            fd = ff.openRWExclusive(path);
-            if (fd < 0 || !ff.allocate(fd, len) || ff.write(fd, mem, len, 0) != len || ff.fsync(fd) != 0) {
-                throw new SfOperationalException("could not write rejection metadata " + path);
+            Unsafe.getUnsafe().copyMemory(bytes, Unsafe.BYTE_OFFSET, null, address, bytes.length);
+            fd = ff.openRWExclusive(dir + '/' + METADATA_FILE_NAME);
+            if (fd < 0 || !ff.allocate(fd, bytes.length)
+                    || ff.write(fd, address, bytes.length, 0) != bytes.length || ff.fsync(fd) != 0) {
+                throw new SfOperationalException("could not write rejection metadata " + dir);
             }
         } finally {
             if (fd >= 0) ff.close(fd);
-            Unsafe.free(mem, len, MemoryTag.NATIVE_DEFAULT);
+            Unsafe.free(address, bytes.length, MemoryTag.NATIVE_DEFAULT);
         }
     }
 
-    private static void copyFile(FilesFacade ff, String fromDir, String toDir, String name) {
-        String source = fromDir + '/' + name;
-        String target = toDir + '/' + name;
-        long len = ff.length(source);
-        if (len < 0) throw new SfOperationalException("missing archive file " + source);
-        int in = ff.openRW(source);
-        int out = ff.openRWExclusive(target);
-        long mem = Unsafe.malloc(Math.min(Math.max(len, 1), 64 * 1024), MemoryTag.NATIVE_DEFAULT);
-        try {
-            if (in < 0 || out < 0 || !ff.allocate(out, len)) throw new SfOperationalException("could not copy " + source);
-            for (long off = 0; off < len; ) {
-                long chunk = Math.min(64 * 1024L, len - off);
-                if (ff.read(in, mem, chunk, off) != chunk || ff.write(out, mem, chunk, off) != chunk) {
-                    throw new SfOperationalException("short copy of " + source);
-                }
-                off += chunk;
-            }
-            if (ff.fsync(out) != 0) throw new SfOperationalException("could not sync " + target);
-        } finally {
-            if (in >= 0) ff.close(in);
-            if (out >= 0) ff.close(out);
-            Unsafe.free(mem, Math.min(Math.max(len, 1), 64 * 1024), MemoryTag.NATIVE_DEFAULT);
-        }
+    private PathsForRange paths(CursorSendEngine engine, long fromFsn, long toFsn) {
+        MmapSegment source = engine.findSegmentContaining(fromFsn);
+        if (source == null) throw new SfOperationalException("rejection range is no longer live");
+        String name = namespace + "-seg-" + source.generationToken() + "-fsn-" + fromFsn + '-' + toFsn;
+        String root = directory + "/rejected/";
+        return new PathsForRange(root + name, root + ".tmp-" + name);
     }
 
-    private static long occupiedBytes(FilesFacade ff, String dir, boolean dict) {
-        long n = ff.length(dir + '/' + SEGMENT_FILE_NAME) + ff.length(dir + '/' + SfManifest.FILE_NAME)
-                + ff.length(dir + '/' + AckWatermark.FILE_NAME) + ff.length(dir + '/' + METADATA_FILE_NAME);
-        return dict ? n + ff.length(dir + '/' + PersistedSymbolDict.FILE_NAME) : n;
+    private void removeKnownDirectory(String dir) {
+        Path path = Paths.get(dir);
+        if (!java.nio.file.Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS)) return;
+        String[] names = {SEGMENT_FILE_NAME, SfManifest.FILE_NAME, AckWatermark.FILE_NAME,
+                PersistedSymbolDict.FILE_NAME, METADATA_FILE_NAME};
+        for (String name : names) ff.remove(dir + '/' + name);
+        // Unknown contents keep the directory in place rather than broadening deletion scope.
+        ff.remove(dir);
     }
 
-    private static void ensureDirectory(FilesFacade ff, String dir) {
+    private void ensureDirectory(String dir) {
         if (!ff.exists(dir) && ff.mkdir(dir, MODE_OWNER_ONLY) != 0) {
             throw new SfOperationalException("could not create directory " + dir);
         }
     }
 
-    private static void requirePathComponent(String value, String label) {
-        if (value == null || value.isEmpty() || value.indexOf('/') >= 0 || value.indexOf('\\') >= 0 || value.equals(".") || value.equals("..")) {
-            throw new IllegalArgumentException("invalid " + label);
-        }
+    private long occupiedBytes(String dir, boolean dictionary) {
+        long n = ff.length(dir + '/' + SEGMENT_FILE_NAME) + ff.length(dir + '/' + SfManifest.FILE_NAME)
+                + ff.length(dir + '/' + AckWatermark.FILE_NAME) + ff.length(dir + '/' + METADATA_FILE_NAME);
+        return dictionary ? n + ff.length(dir + '/' + PersistedSymbolDict.FILE_NAME) : n;
     }
 
-    private static byte[] bytes(String value) { return value == null ? new byte[0] : value.getBytes(StandardCharsets.UTF_8); }
-    private static long writeString(long p, byte[] value) {
-        Unsafe.getUnsafe().putInt(p, value.length);
-        if (value.length > 0) Unsafe.getUnsafe().copyMemory(value, Unsafe.BYTE_OFFSET, null, p + 4, value.length);
-        return p + 4 + value.length;
+    public static void probeDestination(FilesFacade ff, String slotDir) {
+        probeDirectory(ff, slotDir + "/rejected");
     }
-    private static int utf8LengthAt(long base, long p) { return Unsafe.getUnsafe().getInt(p); }
-    private static String readString(long base, long limitOffset, long p) {
-        int len = Unsafe.getUnsafe().getInt(p);
-        if (len < 0 || p + 4L + len > base + limitOffset) throw new UnreplayableSlotException("invalid rejection metadata string");
-        byte[] bytes = new byte[len];
-        if (len > 0) Unsafe.getUnsafe().copyMemory(null, p + 4, bytes, Unsafe.BYTE_OFFSET, len);
-        return new String(bytes, StandardCharsets.UTF_8);
+
+    public static void probeDirectory(FilesFacade ff, String directory) {
+        if (!ff.exists(directory) && ff.mkdir(directory, MODE_OWNER_ONLY) != 0) {
+            throw new SfOperationalException("could not create schema preservation destination " + directory);
+        }
+        String probe = directory + "/.probe-" + UUID.randomUUID();
+        int fd = ff.openRWExclusive(probe);
+        try {
+            if (fd < 0 || ff.fsync(fd) != 0) {
+                throw new SfOperationalException("schema preservation destination is not writable " + directory);
+            }
+        } finally {
+            if (fd >= 0) ff.close(fd);
+            ff.remove(probe);
+        }
+        if (ff.fsyncDir(directory) != 0) {
+            throw new SfOperationalException("could not sync schema preservation destination " + directory);
+        }
     }
 
     public static final class Result {
         public final long bytesWritten;
         public final String path;
         public final boolean reused;
-        Result(String path, long bytesWritten, boolean reused) { this.path = path; this.bytesWritten = bytesWritten; this.reused = reused; }
-    }
 
-    /** Stable fields used by startup scanning to reconstruct a notification. */
-    public static final class Metadata {
-        public final int status;
-        public final long detectedAtNanos, fromFsn, rejectedFsn, toFsn;
-        public final String category, epoch, message, policy, slotId, table;
-        public final boolean hasDictionary;
-        Metadata(String slotId, String epoch, long rejectedFsn, long fromFsn, long toFsn,
-                 long detectedAtNanos, int status, String category, String policy,
-                 String table, String message, boolean hasDictionary) {
-            this.slotId = slotId; this.epoch = epoch; this.rejectedFsn = rejectedFsn; this.fromFsn = fromFsn;
-            this.toFsn = toFsn; this.detectedAtNanos = detectedAtNanos; this.status = status;
-            this.category = category; this.policy = policy; this.table = table;
-            this.message = message; this.hasDictionary = hasDictionary;
-        }
-        static Metadata from(String slotId, String epoch, SenderError e, boolean hasDictionary) {
-            return new Metadata(slotId, epoch, e.getRejectedFsn(), e.getFromFsn(), e.getToFsn(),
-                    e.getDetectedAtNanos(), e.getServerStatusByte(), e.getCategory().name(),
-                    e.getAppliedPolicy().name(), emptyIfNull(e.getTableName()),
-                    emptyIfNull(e.getServerMessage()), hasDictionary);
-        }
-        @Override public boolean equals(Object o) {
-            if (!(o instanceof Metadata)) return false;
-            Metadata m = (Metadata) o;
-            return rejectedFsn == m.rejectedFsn && fromFsn == m.fromFsn && toFsn == m.toFsn
-                    && detectedAtNanos == m.detectedAtNanos && status == m.status
-                    && hasDictionary == m.hasDictionary && eq(slotId, m.slotId) && eq(epoch, m.epoch)
-                    && eq(category, m.category) && eq(policy, m.policy)
-                    && eq(table, m.table) && eq(message, m.message);
+        Result(String path, long bytesWritten, boolean reused) {
+            this.path = path;
+            this.bytesWritten = bytesWritten;
+            this.reused = reused;
         }
-        @Override public int hashCode() { return slotId.hashCode(); }
-        private boolean sameIdentity(Metadata m) {
-            return rejectedFsn == m.rejectedFsn && fromFsn == m.fromFsn && toFsn == m.toFsn
-                    && category.equals(m.category) && slotId.equals(m.slotId) && epoch.equals(m.epoch)
-                    && hasDictionary == m.hasDictionary;
+    }
+
+    private static final class PathsForRange {
+        final String completed;
+        final String temp;
+
+        PathsForRange(String completed, String temp) {
+            this.completed = completed;
+            this.temp = temp;
         }
-        private static boolean eq(Object a, Object b) { return a == null ? b == null : a.equals(b); }
-        private static String emptyIfNull(String value) { return value == null ? "" : value; }
     }
 }
diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java
deleted file mode 100644
index c2cc4bb86..000000000
--- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaPreserver.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/*******************************************************************************
- * Copyright (c) 2014-2026 QuestDB
- * Licensed under the Apache License, Version 2.0.
- ******************************************************************************/
-
-package io.questdb.client.cutlass.qwp.client.sf.cursor;
-
-import io.questdb.client.SenderError;
-import io.questdb.client.std.FilesFacade;
-
-import java.util.UUID;
-
-/** Immutable configuration for synchronous schema-rejection preservation. */
-public final class SchemaPreserver {
-    private final String epoch;
-    private final FilesFacade ff;
-    private final String slotDir;
-    private final String slotId;
-
-    public SchemaPreserver(FilesFacade ff, String slotDir, String slotId, String epoch) {
-        this.ff = ff;
-        this.slotDir = slotDir;
-        this.slotId = slotId;
-        this.epoch = epoch;
-    }
-
-    public SenderError findRecoveredOrphanReport(long fromFsn, long toFsn) {
-        return RejectedMiniSlotArchive.findOverlapping(
-                ff, slotDir, slotId, epoch, fromFsn, toFsn);
-    }
-
-    /**
-     * Preserves the sealed range on the calling thread. A preceding failed
-     * publication may have left its uniquely named temporary tree behind, so
-     * clean only this slot and epoch's known temporary scope before retrying.
-     */
-    public RejectedMiniSlotArchive.Result preserve(
-            CursorSendEngine engine,
-            SenderError error,
-            byte[] dictionaryEntries,
-            int dictionaryCount
-    ) {
-        RejectedMiniSlotArchive.cleanupTemporaryDirectories(ff, slotDir, slotId, epoch);
-        return RejectedMiniSlotArchive.preserveSnapshot(
-                ff, engine, dictionaryEntries, dictionaryCount,
-                slotDir, slotId, epoch, error);
-    }
-
-    /** Build-time writability and directory-durability probe. */
-    public static void probeDestination(FilesFacade ff, String slotDir) {
-        probeDirectory(ff, slotDir + "/rejected");
-    }
-
-    /** Probes an explicitly configured destination without assuming a slot layout. */
-    public static void probeDirectory(FilesFacade ff, String directory) {
-        if (!ff.exists(directory) && ff.mkdir(directory, 448) != 0) {
-            throw new SfOperationalException("could not create schema preservation destination " + directory);
-        }
-        String probe = directory + "/.probe-" + UUID.randomUUID();
-        int fd = ff.openRWExclusive(probe);
-        try {
-            if (fd < 0 || ff.fsync(fd) != 0) {
-                throw new SfOperationalException("schema preservation destination is not writable " + directory);
-            }
-        } finally {
-            if (fd >= 0) ff.close(fd);
-            ff.remove(probe);
-        }
-        if (ff.fsyncDir(directory) != 0) {
-            throw new SfOperationalException("could not sync schema preservation destination " + directory);
-        }
-    }
-}
diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java
index b8f9b3d5e..213bc472c 100644
--- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java
+++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SchemaRejectionState.java
@@ -9,14 +9,11 @@
 import io.questdb.client.SenderError;
 import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
 
-import java.util.ArrayDeque;
-
-/** Process-local lease ownership and one pending schema-retirement range. */
+/** Current borrow and one pending retirement. Returned borrows have no observation history. */
 public final class SchemaRejectionState {
-    private final ArrayDeque leases = new ArrayDeque<>();
+    private Lease current;
     private Pending pending;
     private CursorSendEngine engine;
-    private volatile long acknowledgedFsn = -1L;
     private volatile long failedGeneration = -1L;
     private volatile long stopFsn = -1L;
 
@@ -25,24 +22,16 @@ public void setEngine(CursorSendEngine engine) {
     }
 
     public synchronized void beginLease(long generation, long firstFsn, boolean transactional) {
-        prune(acknowledgedFsn);
-        Lease tail = leases.peekLast();
-        if (tail != null && tail.active) {
-            throw new IllegalStateException("previous lease is still active");
-        }
-        if (tail != null) {
-            // The returned handle can no longer observe an owned exception. Keep
-            // only the range needed to classify a delayed rejection, and combine
-            // completed borrows instead of retaining one object per borrow until ACK.
-            if (failedGeneration == tail.generation) {
-                failedGeneration = -1L;
+        if (current != null) {
+            if (current.active) {
+                throw new IllegalStateException("previous lease is still active");
+            }
+            if (current.transactional != transactional) {
+                throw new IllegalStateException("transaction mode must remain fixed for a sender");
             }
-            tail.generation = -1L;
-            tail.failure = null;
-            tail.rawError = null;
-            mergeReturnedTail();
         }
-        leases.addLast(new Lease(generation, firstFsn, transactional));
+        failedGeneration = -1L;
+        current = new Lease(generation, firstFsn, transactional);
     }
 
     /**
@@ -51,21 +40,27 @@ public synchronized void beginLease(long generation, long firstFsn, boolean tran
      * {@code publishedFsn}.
      */
     public synchronized LineSenderServerException endLease(long generation, long publishedFsn) {
-        Lease lease = findGeneration(generation);
-        if (lease == null || !lease.active) {
+        Lease lease = current;
+        if (lease == null || lease.generation != generation || !lease.active) {
             return null;
         }
         lease.endFsn = publishedFsn;
-        lease.active = false;
-        if (publishedFsn < lease.firstFsn && lease.rawError == null) {
-            // Empty borrows carry no attribution history, even behind an unacked lease.
-            leases.removeLast();
-            return null;
-        }
-        int flags = lease.transactional && engine != null ? engine.liveQwpFrameFlags(publishedFsn) : -1;
-        lease.endsWithCommit = !lease.transactional
-                || (flags >= 0 && (flags & QwpConstants.FLAG_DEFER_COMMIT) == 0);
         sealIfNeeded(lease, publishedFsn);
+        // Pool return must close normal transactions. A failed open tail may
+        // return only after sealing the range which prevents its resurrection
+        // by the next borrow's commit. Check before giving up producer ownership.
+        if (engine != null && lease.transactional && publishedFsn >= lease.firstFsn
+                && publishedFsn > engine.ackedFsn()
+                && (pending == null || pending.lastFsn < publishedFsn)) {
+            int flags = engine.liveQwpFrameFlags(publishedFsn);
+            // ACK/trim can race this cold lookup. An already resolved closer
+            // needs no longer to be present in the ring.
+            if ((flags < 0 && publishedFsn > engine.ackedFsn())
+                    || (flags >= 0 && (flags & QwpConstants.FLAG_DEFER_COMMIT) != 0)) {
+                throw new IllegalStateException("returned transaction has no commit or rejection boundary");
+            }
+        }
+        lease.active = false;
         if (failedGeneration == generation) {
             failedGeneration = -1L;
         }
@@ -78,8 +73,8 @@ public synchronized LineSenderServerException endLease(long generation, long pub
      * publication snapshot. Calls on one sender are single-producer by contract.
      */
     public synchronized LineSenderServerException ownedFailure(long generation, long publishedFsn) {
-        Lease lease = findGeneration(generation);
-        if (lease == null || lease.rawError == null) {
+        Lease lease = current;
+        if (lease == null || !lease.active || lease.generation != generation || lease.rawError == null) {
             return null;
         }
         sealIfNeeded(lease, publishedFsn);
@@ -92,11 +87,10 @@ public boolean hasOwnedFailure(long generation) {
 
     /** I/O-thread install. Returns false while an earlier retirement is pending. */
     public synchronized boolean reject(long rejectedFsn, long spanStart, SenderError rawError) {
-        prune(acknowledgedFsn);
         if (pending != null) {
             return false;
         }
-        Lease owner = findOwner(rejectedFsn);
+        Lease owner = current != null && current.active && rejectedFsn >= current.firstFsn ? current : null;
         if (owner == null) {
             // Transaction mode is not persisted. A recovered deferred group must
             // therefore be treated conservatively as transactional, regardless of
@@ -106,9 +100,10 @@ public synchronized boolean reject(long rejectedFsn, long spanStart, SenderError
                     ? Math.max(engine.recoveredCommitBoundaryFsn(), engine.recoveredOrphanTipFsn())
                     : -1L;
             boolean recovered = rejectedFsn <= recoveredTip;
-            owner = new Lease(-1L, spanStart, recovered);
+            owner = new Lease(-1L, spanStart, recovered || (current != null && current.transactional));
             owner.active = false;
-            owner.endFsn = recovered ? recoveredTip : rejectedFsn;
+            owner.endFsn = recovered ? recoveredTip : current == null ? rejectedFsn
+                    : current.active ? current.firstFsn - 1L : current.endFsn;
         } else if (owner.generation >= 0 && owner.rawError == null) {
             owner.rawError = rawError;
             failedGeneration = owner.generation;
@@ -142,16 +137,8 @@ public synchronized void completeRetirement(long lastFsn) {
         if (pending == null || pending.lastFsn != lastFsn) {
             throw new IllegalStateException("retirement range changed");
         }
-        acknowledgedThrough(lastFsn);
         pending = null;
         stopFsn = -1L;
-        prune(lastFsn);
-    }
-
-    public void acknowledgedThrough(long fsn) {
-        if (fsn > acknowledgedFsn) {
-            acknowledgedFsn = fsn;
-        }
     }
 
     private void sealIfNeeded(Lease lease, long publishedFsn) {
@@ -191,54 +178,6 @@ private void finishFailure(long lastFsn) {
         }
     }
 
-    private Lease findGeneration(long generation) {
-        Lease tail = leases.peekLast();
-        if (tail != null && tail.generation == generation) {
-            return tail;
-        }
-        for (Lease lease : leases) {
-            if (lease.generation == generation) {
-                return lease;
-            }
-        }
-        return null;
-    }
-
-    private Lease findOwner(long fsn) {
-        for (Lease lease : leases) {
-            if (fsn >= lease.firstFsn && (lease.active || fsn <= lease.endFsn)) {
-                return lease;
-            }
-        }
-        return null;
-    }
-
-    private void mergeReturnedTail() {
-        Lease tail = leases.removeLast();
-        Lease previous = leases.peekLast();
-        if (previous != null && previous.transactional == tail.transactional
-                && previous.endFsn + 1 == tail.firstFsn && previous.endsWithCommit
-                && (pending == null || pending.owner != previous)) {
-            // Normal pool return publishes a commit before ending the lease.
-            // Its frame flags preserve each transaction's boundary in a merged
-            // range. An unfinished failed transaction must keep its own end,
-            // and an in-flight retirement must retain its owner object.
-            tail.firstFsn = previous.firstFsn;
-            leases.removeLast();
-        }
-        leases.addLast(tail);
-    }
-
-    private void prune(long fsn) {
-        while (true) {
-            Lease head = leases.peekFirst();
-            if (head == null || head.active || (pending != null && pending.owner == head) || head.endFsn > fsn) {
-                return;
-            }
-            leases.removeFirst();
-        }
-    }
-
     public static final class Range {
         public final SenderError error;
         public final long firstFsn;
@@ -267,12 +206,11 @@ private Pending(long firstFsn, long lastFsn, Lease owner, SenderError rawError)
     }
 
     private static final class Lease {
-        private long firstFsn;
-        private long generation;
+        private final long firstFsn;
+        private final long generation;
         private final boolean transactional;
         private boolean active = true;
         private long endFsn = -1L;
-        private boolean endsWithCommit;
         private LineSenderServerException failure;
         private SenderError rawError;
 
diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java
deleted file mode 100644
index 67cec0912..000000000
--- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/SlotEpoch.java
+++ /dev/null
@@ -1,116 +0,0 @@
-/*******************************************************************************
- * Copyright (c) 2014-2026 QuestDB
- * Licensed under the Apache License, Version 2.0.
- ******************************************************************************/
-
-package io.questdb.client.cutlass.qwp.client.sf.cursor;
-
-import io.questdb.client.std.Crc32c;
-import io.questdb.client.std.FilesFacade;
-import io.questdb.client.std.MemoryTag;
-import io.questdb.client.std.Unsafe;
-
-import java.util.UUID;
-
-/** Durable identity for one lifecycle of an SF slot's FSN namespace. */
-public final class SlotEpoch {
-    public static final String FILE_NAME = ".slot-epoch";
-    private static final int CRC_OFFSET = 28;
-    private static final int FILE_SIZE = 32;
-    private static final int MAGIC = 0x31455053; // SPE1 little-endian
-    private static final int VERSION = 1;
-
-    private SlotEpoch() {
-    }
-
-    /**
-     * Opens or creates the slot epoch. The caller must hold the slot's
-     * exclusive {@link SlotLock}; this method publishes a newly-created epoch
-     * with file and directory durability before returning it.
-     */
-    public static String openOrCreate(FilesFacade ff, String slotDir) {
-        return openOrCreate(ff, slotDir, false);
-    }
-
-    /**
-     * Opens the epoch for a recovered FSN namespace, or replaces it when the
-     * caller has proved that this is a fresh namespace.  The latter check is
-     * required even though clean close normally removes the sidecar: a crash
-     * or unlink failure after durable segment removal can leave only the old
-     * epoch behind.
-     */
-    public static String openOrCreate(FilesFacade ff, String slotDir, boolean freshFsnNamespace) {
-        String path = slotDir + "/" + FILE_NAME;
-        if (ff.exists(path) && !freshFsnNamespace) {
-            String epoch = read(ff, path);
-            // Also completes a prior create whose rename succeeded but whose
-            // directory barrier reported a transient failure.
-            if (ff.fsyncDir(slotDir) != 0) {
-                throw new SfOperationalException("could not sync durable slot epoch " + path);
-            }
-            return epoch;
-        }
-        UUID uuid = UUID.randomUUID();
-        String value = uuid.toString();
-        String temp = path + ".tmp-" + UUID.randomUUID();
-        long mem = Unsafe.malloc(FILE_SIZE, MemoryTag.NATIVE_DEFAULT);
-        int fd = -1;
-        boolean published = false;
-        try {
-            Unsafe.getUnsafe().setMemory(mem, FILE_SIZE, (byte) 0);
-            Unsafe.getUnsafe().putInt(mem, MAGIC);
-            Unsafe.getUnsafe().putInt(mem + 4, VERSION);
-            Unsafe.getUnsafe().putLong(mem + 8, uuid.getMostSignificantBits());
-            Unsafe.getUnsafe().putLong(mem + 16, uuid.getLeastSignificantBits());
-            Unsafe.getUnsafe().putInt(mem + CRC_OFFSET, Crc32c.update(Crc32c.INIT, mem, CRC_OFFSET));
-            fd = ff.openRWExclusive(temp);
-            if (fd < 0 || !ff.allocate(fd, FILE_SIZE)
-                    || ff.write(fd, mem, FILE_SIZE, 0) != FILE_SIZE
-                    || ff.fsync(fd) != 0) {
-                throw new SfOperationalException("could not create durable slot epoch " + path);
-            }
-            ff.close(fd);
-            fd = -1;
-            if (freshFsnNamespace && ff.exists(path) && !ff.remove(path)) {
-                throw new SfOperationalException("could not replace stale slot epoch " + path);
-            }
-            if (ff.rename(temp, path) != 0 || ff.fsyncDir(slotDir) != 0) {
-                throw new SfOperationalException("could not publish durable slot epoch " + path);
-            }
-            published = true;
-            return value;
-        } finally {
-            if (fd >= 0) {
-                ff.close(fd);
-            }
-            Unsafe.free(mem, FILE_SIZE, MemoryTag.NATIVE_DEFAULT);
-            if (!published) {
-                ff.remove(temp);
-            }
-        }
-    }
-
-    public static String read(FilesFacade ff, String path) {
-        if (ff.length(path) != FILE_SIZE) {
-            throw new UnreplayableSlotException("invalid slot epoch size " + path);
-        }
-        long mem = Unsafe.malloc(FILE_SIZE, MemoryTag.NATIVE_DEFAULT);
-        int fd = ff.openRW(path);
-        try {
-            if (fd < 0 || ff.read(fd, mem, FILE_SIZE, 0) != FILE_SIZE
-                    || Unsafe.getUnsafe().getInt(mem) != MAGIC
-                    || Unsafe.getUnsafe().getInt(mem + 4) != VERSION
-                    || Unsafe.getUnsafe().getInt(mem + CRC_OFFSET)
-                    != Crc32c.update(Crc32c.INIT, mem, CRC_OFFSET)) {
-                throw new UnreplayableSlotException("invalid slot epoch " + path);
-            }
-            return new UUID(Unsafe.getUnsafe().getLong(mem + 8),
-                    Unsafe.getUnsafe().getLong(mem + 16)).toString();
-        } finally {
-            if (fd >= 0) {
-                ff.close(fd);
-            }
-            Unsafe.free(mem, FILE_SIZE, MemoryTag.NATIVE_DEFAULT);
-        }
-    }
-}
diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java
index 0636719c3..2f1acf316 100644
--- a/core/src/main/java/io/questdb/client/impl/SenderPool.java
+++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java
@@ -594,7 +594,7 @@ private SenderPool(
                 throw new io.questdb.client.cutlass.line.LineSenderException(e)
                         .put("could not create schema preservation destination ").put(destination);
             }
-            io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver.probeDirectory(
+            io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive.probeDirectory(
                     io.questdb.client.std.FilesFacade.INSTANCE, destination);
         }
         this.slotInUse = this.storeAndForward ? new boolean[maxSize] : null;
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java
index 93b484856..a0ee42943 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/RejectedArchiveRecoveryTest.java
@@ -10,7 +10,6 @@
 import io.questdb.client.cutlass.qwp.client.sf.cursor.AckWatermark;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
 import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
 import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
@@ -31,6 +30,7 @@
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 
 import static org.junit.Assert.*;
 
@@ -44,52 +44,76 @@ public void testDamagedDrainedArchiveDoesNotBlockRepeatedBuilds() throws Excepti
     }
 
     @Test(timeout = 30_000)
-    public void testDamagedOverlappingArchiveQuarantinesOnceAndBuildsContinue() throws Exception {
+    public void testDamagedOverlappingArchiveDoesNotAffectQueue() throws Exception {
         assertRepeatedBuilds(RejectedMiniSlotArchive.METADATA_FILE_NAME, true);
     }
 
     @Test(timeout = 30_000)
-    public void testIntactOverlappingArchiveReportsAndBuildsContinue() throws Exception {
+    public void testPublishedArchiveReconstructsCallbackBeforeOrphanRetirement() throws Exception {
         assertRepeatedBuilds(null, true);
     }
 
     @Test(timeout = 30_000)
-    public void testDamagedOverlappingArchiveSegmentQuarantinesOnce() throws Exception {
+    public void testDamagedOverlappingArchiveSegmentDoesNotAffectQueue() throws Exception {
         assertRepeatedBuilds(RejectedMiniSlotArchive.SEGMENT_FILE_NAME, true);
     }
 
+    @Test(timeout = 30_000)
+    public void testMismatchedArchiveBoundaryDoesNotAffectQueue() throws Exception {
+        assertRepeatedBuilds(AckWatermark.FILE_NAME, true);
+    }
+
     private void assertRepeatedBuilds(String damagedFile, boolean overlaps) throws Exception {
         boolean damaged = damagedFile != null;
         Path base = temp.newFolder().toPath();
         Path slot = Files.createDirectory(base.resolve("saved"));
+        // Residue from the original PR and a crashed writer is output only.
+        Path oldEpoch = slot.resolve(".slot-epoch");
+        Files.write(oldEpoch, new byte[]{0});
+        Path staging = Files.createDirectories(slot.resolve("rejected/.tmp-legacy-writer"));
+        Path oldMetadata = staging.resolve("rejection-meta.bin");
+        Files.write(oldMetadata, new byte[]{1});
         String archive;
         try (CursorSendEngine engine = new CursorSendEngine(slot.toString(), 4096)) {
             // Keep fixture construction independent of the manager's ACK-persistence tick.
             engine.getManagerForTesting().close();
-            String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, slot.toString(), engine.freshFsnNamespace());
             append(engine, false);
-            archive = preserve(engine, slot, epoch, 0);
+            archive = preserve(engine, slot, 0);
             assertTrue(engine.acknowledge(0));
-            append(engine, true); // Uncommitted orphan tail at FSN 1 forces the startup archive scan.
-            if (overlaps) archive = preserve(engine, slot, epoch, 1);
+            append(engine, true); // Uncommitted orphan tail at FSN 1.
+            if (overlaps) archive = preserve(engine, slot, 1);
         }
         // FSN 0 is the drained prefix of this fixture. acknowledge() only
         // advances the live ring; a partially drained close need not persist
-        // that watermark. Write it explicitly so orphan validation happens
-        // during build(), rather than after replay ACKs on the I/O thread.
+        // that watermark. Write it explicitly so the orphan can retire during
+        // build(), rather than after replay ACKs on the I/O thread.
         try (AckWatermark watermark = AckWatermark.open(slot.toString())) {
             assertNotNull(watermark);
             watermark.write(0);
             watermark.sync();
         }
         Path archiveFile = Paths.get(archive, damaged ? damagedFile : RejectedMiniSlotArchive.METADATA_FILE_NAME);
-        byte[] archiveBytes = Files.readAllBytes(archiveFile);
         if (damaged) {
-            archiveBytes[0] ^= 1; // Corrupt metadata CRC or segment magic, retaining the remaining bytes.
-            Files.write(archiveFile, archiveBytes);
+            if (RejectedMiniSlotArchive.SEGMENT_FILE_NAME.equals(damagedFile)) {
+                // Keep the valid file size while corrupting the segment header.
+                byte[] archiveBytes = Files.readAllBytes(archiveFile);
+                archiveBytes[0] ^= 1;
+                Files.write(archiveFile, archiveBytes);
+            } else if (AckWatermark.FILE_NAME.equals(damagedFile)) {
+                try (AckWatermark archiveWatermark = AckWatermark.open(archive)) {
+                    assertNotNull(archiveWatermark);
+                    archiveWatermark.write(1); // Structurally valid, but expected boundary is 0.
+                    archiveWatermark.sync();
+                }
+            } else {
+                Files.write(archiveFile, new byte[]{0});
+            }
         }
+        byte[] archiveBytes = Files.readAllBytes(archiveFile);
         AtomicInteger quarantines = new AtomicInteger();
+        AtomicInteger schemaReports = new AtomicInteger();
         CountDownLatch schemaReported = new CountDownLatch(1);
+        AtomicReference recoveredReport = new AtomicReference<>();
         Map sequences = new ConcurrentHashMap<>();
         try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() {
             @Override
@@ -110,16 +134,18 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat
                                 + ";sf_dir=" + base + ";close_flush_timeout_millis=0;")
                         .senderId("saved").errorHandler(error -> {
                             if (error.getCategory() == SenderError.Category.DATA_LOSS) quarantines.incrementAndGet();
-                            if (error.getCategory() == SenderError.Category.SCHEMA_MISMATCH) schemaReported.countDown();
+                            if (error.getCategory() == SenderError.Category.SCHEMA_MISMATCH) {
+                                recoveredReport.set(error);
+                                schemaReports.incrementAndGet();
+                                schemaReported.countDown();
+                            }
                         }).build()) {
-                    assertEquals("quarantine must complete during build", damaged && overlaps ? 1 : 0,
+                    assertEquals("archives must never quarantine a live queue", 0,
                             quarantines.get());
                     sender.table("healthy").longColumn("value", attempt).atNow();
                     long target = sender.flushAndGetSequence();
                     assertTrue("new rows must drain after recovery", sender.awaitAckedFsn(target, 5_000));
-                    if (attempt == 0 && !damaged && overlaps) {
-                        assertTrue(schemaReported.await(5, TimeUnit.SECONDS));
-                    }
+
                 } catch (RuntimeException e) {
                     failures.add(e);
                 }
@@ -127,23 +153,31 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat
             assertTrue("all three builds must succeed: " + failures, failures.isEmpty());
         }
         Path quarantined = base.resolve("saved.unreplayable-0");
-        if (damaged && overlaps) {
-            assertEquals(1, quarantines.get());
-            assertTrue(Files.exists(quarantined.resolve(".failed")));
-            assertArrayEquals(archiveBytes, Files.readAllBytes(quarantined.resolve(slot.relativize(archiveFile))));
-            assertFalse(Files.exists(base.resolve("saved.unreplayable-1")));
+        assertEquals(0, quarantines.get());
+        if (overlaps && !damaged) {
+            assertTrue("published archive report must precede orphan retirement",
+                    schemaReported.await(5, TimeUnit.SECONDS));
+            assertEquals(1, schemaReports.get());
+            SenderError report = recoveredReport.get();
+            assertNotNull(report);
+            assertEquals(1, report.getFromFsn());
+            assertEquals(1, report.getToFsn());
+            assertEquals(1, report.getRejectedFsn());
+            assertEquals(archive, report.getRejectedPath());
         } else {
-            assertEquals(0, quarantines.get());
-            assertFalse(Files.exists(quarantined));
-            assertArrayEquals(archiveBytes, Files.readAllBytes(archiveFile));
+            assertEquals(1, schemaReported.getCount());
+            assertEquals(0, schemaReports.get());
         }
+        assertFalse(Files.exists(quarantined));
+        assertArrayEquals(archiveBytes, Files.readAllBytes(archiveFile));
+        assertArrayEquals(new byte[]{0}, Files.readAllBytes(oldEpoch));
+        assertArrayEquals(new byte[]{1}, Files.readAllBytes(oldMetadata));
     }
 
-    private static String preserve(CursorSendEngine engine, Path slot, String epoch, long fsn) {
+    private static String preserve(CursorSendEngine engine, Path slot, long fsn) {
         SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH,
                 SenderError.Policy.REJECT_AND_CONTINUE, 3, "schema rejected", fsn, fsn, fsn, null, 1);
-        return RejectedMiniSlotArchive.preserve(FilesFacade.INSTANCE, engine, null,
-                slot.toString(), "saved", epoch, error).path;
+        return new RejectedMiniSlotArchive(FilesFacade.INSTANCE, slot.toString()).preserve(engine, error, null, 0).path;
     }
 
     private static void append(CursorSendEngine engine, boolean deferred) {
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java
index 6eff9e600..6bb9d14bb 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerSetupFailureTest.java
@@ -25,17 +25,15 @@
 package io.questdb.client.test.cutlass.qwp.client.sf.cursor;
 
 import io.questdb.client.SenderError;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
-import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
-import io.questdb.client.std.FilesFacade;
-import java.util.concurrent.atomic.AtomicReference;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.AckWatermark;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegment;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
+import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
 import io.questdb.client.std.Files;
+import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
 import io.questdb.client.std.Unsafe;
 import io.questdb.client.test.tools.TestUtils;
@@ -46,6 +44,7 @@
 
 import java.nio.file.Paths;
 import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 
 public class BackgroundDrainerSetupFailureTest {
 
@@ -65,7 +64,7 @@ public void tearDown() {
     }
 
     @Test
-    public void testPreservedOrphanRetiresAndReportsWithoutConnecting() throws Exception {
+    public void testPreservedOrphanReportsBeforeRetirementWithoutConnecting() throws Exception {
         assertOrphanRetiresOffline(true);
     }
 
@@ -89,14 +88,10 @@ private void assertOrphanRetiresOffline(boolean archive) throws Exception {
                     Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT);
                 }
                 if (archive) {
-                    String epoch = SlotEpoch.openOrCreate(
-                            FilesFacade.INSTANCE, slotPath, original.freshFsnNamespace());
                     SenderError error = new SenderError(
                             SenderError.Category.SCHEMA_MISMATCH,
                             SenderError.Policy.REJECT_AND_CONTINUE, 3, "bad schema", 0, 0, 0, null, 1);
-                    archivedPath = RejectedMiniSlotArchive.preserve(
-                            FilesFacade.INSTANCE, original, null, slotPath,
-                            Paths.get(slotPath).getFileName().toString(), epoch, error).path;
+                    archivedPath = new RejectedMiniSlotArchive(FilesFacade.INSTANCE, slotPath).preserve(original, error, null, 0).path;
                 }
             }
             AtomicReference report = new AtomicReference<>();
@@ -110,9 +105,12 @@ private void assertOrphanRetiresOffline(boolean archive) throws Exception {
             Assert.assertEquals(BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome());
             Assert.assertFalse(OrphanScanner.isCandidateOrphan(slotPath));
             if (archive) {
-                Assert.assertNotNull(report.get());
-                Assert.assertEquals(archivedPath, report.get().getRejectedPath());
-                Assert.assertNotSame(Thread.currentThread(), callbackThread.get());
+                SenderError recovered = report.get();
+                Assert.assertNotNull(recovered);
+                Assert.assertEquals(0, recovered.getFromFsn());
+                Assert.assertEquals(0, recovered.getToFsn());
+                Assert.assertEquals(archivedPath, recovered.getRejectedPath());
+                Assert.assertNotEquals(Thread.currentThread(), callbackThread.get());
                 Assert.assertTrue(java.nio.file.Files.isDirectory(Paths.get(archivedPath)));
             } else {
                 Assert.assertNull(report.get());
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java
index 6cdf61f44..4834763fc 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopPoisonFrameTest.java
@@ -34,11 +34,9 @@
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
 import io.questdb.client.network.PlainSocketFactory;
-import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
 import io.questdb.client.std.Unsafe;
 import io.questdb.client.test.tools.DelegatingFilesFacade;
@@ -246,14 +244,13 @@ public void testPreservationFailureRetriesThenRetiresWithoutTerminal() throws Ex
                         0L, 0L, null, System.nanoTime());
                 assertTrue(state.reject(0L, 0L, error));
                 FailFirstTemporaryMkdirFacade ff = new FailFirstTemporaryMkdirFacade();
-                String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, tmpDir);
                 AtomicReference reported = new AtomicReference<>();
-                SchemaPreserver preserver = new SchemaPreserver(
-                        ff, tmpDir, "retry-slot", epoch);
+                RejectedMiniSlotArchive preserver = new RejectedMiniSlotArchive(
+                        ff, tmpDir);
                 try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(reported::set);
                      CursorWebSocketSendLoop loop = newDurableLoop(engine, new ArrayList<>())) {
                     loop.setSchemaRejectionState(state);
-                    loop.setSchemaPreserver(preserver);
+                    loop.setRejectionArchive(preserver);
                     loop.setErrorDispatcher(dispatcher);
 
                     assertFalse(loop.tryRetireSchemaRangeForTest());
@@ -288,8 +285,7 @@ public void testFullSchemaNotificationFifoDoesNotRepeatPreservation() throws Exc
                         0L, 0L, null, System.nanoTime());
                 assertTrue(state.reject(0L, 0L, error));
                 CountingPreserveFacade ff = new CountingPreserveFacade(tmpDir + "/rejected");
-                String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, tmpDir);
-                CountDownLatch handlerEntered = new CountDownLatch(1);
+                        CountDownLatch handlerEntered = new CountDownLatch(1);
                 CountDownLatch releaseHandler = new CountDownLatch(1);
                 try (SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(ignored -> {
                     handlerEntered.countDown();
@@ -305,7 +301,7 @@ public void testFullSchemaNotificationFifoDoesNotRepeatPreservation() throws Exc
                         assertTrue(dispatcher.tryOfferSchema(error));
                     }
                     loop.setSchemaRejectionState(state);
-                    loop.setSchemaPreserver(new SchemaPreserver(ff, tmpDir, "fifo-slot", epoch));
+                    loop.setRejectionArchive(new RejectedMiniSlotArchive(ff, tmpDir));
                     loop.setErrorDispatcher(dispatcher);
 
                     assertFalse(loop.tryRetireSchemaRangeForTest());
@@ -339,9 +335,9 @@ public void testTransactionalCloserScanFailureFromNackLatchesWithoutReconnect()
             try (CursorSendEngine engine = newEngine()) {
                 appendDeferredFrame(engine);
                 SchemaRejectionState state = new SchemaRejectionState();
-                state.setEngine(engine);
                 state.beginLease(1L, 0L, true);
-                state.endLease(1L, 1L); // recovered/advertised tail includes missing frame 1
+                state.endLease(1L, 1L); // Inject an advertised tail including missing frame 1.
+                state.setEngine(engine);
                 AtomicReference reported = new AtomicReference<>();
                 try (CursorWebSocketSendLoop loop = newDurableLoop(engine, clients);
                      SenderErrorDispatcher dispatcher = new SenderErrorDispatcher(reported::set)) {
@@ -1099,7 +1095,7 @@ private static final class FailFirstTemporaryMkdirFacade extends DelegatingFiles
 
         @Override
         public int mkdir(String path, int mode) {
-            if (!failed && path.contains("/rejected/.tmp-retry-slot-")) {
+            if (!failed && path.contains("/rejected/.tmp-")) {
                 failed = true;
                 return -1;
             }
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java
index 1de91badd..e911f7bc8 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopSchemaPreservationCloseTest.java
@@ -11,14 +11,12 @@
 import io.questdb.client.cutlass.line.LineSenderException;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaRejectionState;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLock;
 import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
 import io.questdb.client.network.PlainSocketFactory;
-import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
 import io.questdb.client.std.Unsafe;
 import io.questdb.client.test.tools.DelegatingFilesFacade;
@@ -77,8 +75,7 @@ public void closeTraffic() {
                         SenderError.Policy.REJECT_AND_CONTINUE, 3, "mismatch", 0,
                         0, 0, "tab", System.nanoTime())));
                 loop.setSchemaRejectionState(state);
-                loop.setSchemaPreserver(new SchemaPreserver(ff, directory, "slot",
-                        SlotEpoch.openOrCreate(FilesFacade.INSTANCE, directory)));
+                loop.setRejectionArchive(new RejectedMiniSlotArchive(ff, directory));
                 loop.setErrorDispatcher(dispatcher);
                 loop.setShutdownAwaitTimeoutMillis(25);
                 loop.start();
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java
index e978ad157..47943cca6 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/MmapSegmentTest.java
@@ -204,8 +204,10 @@ public void testHeaderShapeMatchesTheDocumentedLayout() throws Exception {
         assertEquals(1, MmapSegment.VERSION);
         TestUtils.assertMemoryLeak(() -> {
             String path = tmpDir + "/seg-header.sfa";
+            long generationToken;
             try (MmapSegment seg = MmapSegment.create(path, 7L, 4096L)) {
                 assertEquals(7L, seg.baseSeq());
+                generationToken = seg.generationToken();
             }
             byte[] bytes = java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(path));
             java.nio.ByteBuffer header = java.nio.ByteBuffer
@@ -216,7 +218,38 @@ public void testHeaderShapeMatchesTheDocumentedLayout() throws Exception {
             assertEquals("flags", 0, header.get(5));
             assertEquals("reserved", 0, header.getShort(6));
             assertEquals(7L, header.getLong(8));
-            assertTrue("createdMicros must be stamped", header.getLong(16) > 0L);
+            assertEquals(generationToken, header.getLong(16));
+        });
+    }
+
+    @Test
+    public void testFreshSegmentsHaveDistinctGenerationTokens() throws Exception {
+        TestUtils.assertMemoryLeak(() -> {
+            try (MmapSegment firstDisk = MmapSegment.create(tmpDir + "/generation-1.sfa", 0L, 4096L);
+                 MmapSegment secondDisk = MmapSegment.create(tmpDir + "/generation-2.sfa", 0L, 4096L);
+                 MmapSegment firstMemory = MmapSegment.createInMemory(0L, 4096L);
+                 MmapSegment secondMemory = MmapSegment.createInMemory(0L, 4096L)) {
+                assertNotEquals(firstDisk.generationToken(), secondDisk.generationToken());
+                assertNotEquals(firstMemory.generationToken(), secondMemory.generationToken());
+            }
+        });
+    }
+
+    @Test
+    public void testLegacyCreationTimestampIsReadAsOpaqueGenerationToken() throws Exception {
+        TestUtils.assertMemoryLeak(() -> {
+            String path = tmpDir + "/legacy-generation.sfa";
+            try (MmapSegment ignored = MmapSegment.create(path, 7L, 4096L)) {
+                // Close before replacing the token with a legacy timestamp value.
+            }
+            long legacyTimestamp = 1_234_567_890L;
+            try (RandomAccessFile file = new RandomAccessFile(path, "rw")) {
+                file.seek(16L);
+                file.writeLong(Long.reverseBytes(legacyTimestamp));
+            }
+            try (MmapSegment reopened = MmapSegment.openExisting(path)) {
+                assertEquals(legacyTimestamp, reopened.generationToken());
+            }
         });
     }
 
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java
index 5d00e8b3d..88e3ae497 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/RejectedMiniSlotArchiveTest.java
@@ -6,274 +6,242 @@
 package io.questdb.client.test.cutlass.qwp.client.sf.cursor;
 
 import io.questdb.client.SenderError;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SchemaPreserver;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegment;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegmentException;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.PersistedSymbolDict;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.MmapSegmentCorruptionException;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.UnreplayableSlotException;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.SfOperationalException;
+import io.questdb.client.cutlass.qwp.client.sf.cursor.SfRecoveryException;
 import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
 import io.questdb.client.std.FilesFacade;
 import io.questdb.client.std.MemoryTag;
 import io.questdb.client.std.Unsafe;
-import io.questdb.client.test.tools.TestUtils;
 import io.questdb.client.test.tools.DelegatingFilesFacade;
-import org.junit.After;
-import org.junit.Before;
+import io.questdb.client.test.tools.TestUtils;
+import org.junit.Rule;
 import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
 
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.Comparator;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
 
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertSame;
+import static org.junit.Assert.*;
 
 public class RejectedMiniSlotArchiveTest {
-    private Path root;
-
-    @Before
-    public void setUp() throws Exception {
-        root = Files.createTempDirectory("qdb-rejected-mini-slot-");
-    }
-
-    @After
-    public void tearDown() throws Exception {
-        if (root != null) {
-            try (java.util.stream.Stream paths = Files.walk(root)) {
-                paths.sorted(Comparator.reverseOrder()).forEach(p -> {
-                    try { Files.deleteIfExists(p); } catch (Exception ignored) { }
-                });
-            }
-        }
-    }
-
-    @Test
-    public void testEpochSurvivesReopenAndRejectsCorruption() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String slot = Files.createDirectory(root.resolve("slot")).toString();
-        String first = SlotEpoch.openOrCreate(ff, slot);
-        assertEquals(first, SlotEpoch.openOrCreate(ff, slot));
-        assertEquals(first, SlotEpoch.read(ff, slot + '/' + SlotEpoch.FILE_NAME));
-        String reset = SlotEpoch.openOrCreate(ff, slot, true);
-        assertFalse(first.equals(reset));
-        assertEquals(reset, SlotEpoch.openOrCreate(ff, slot, false));
-    }
-
-    @Test
-    public void testCleanEngineCloseEndsEpochLifecycle() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String slot = Files.createDirectory(root.resolve("clean-close-slot")).toString();
-        try (CursorSendEngine engine = new CursorSendEngine(slot, 4096)) {
-            SlotEpoch.openOrCreate(ff, slot, engine.freshFsnNamespace());
-            assertTrue(ff.exists(slot + '/' + SlotEpoch.FILE_NAME));
-        }
-        assertFalse(ff.exists(slot + '/' + SlotEpoch.FILE_NAME));
-    }
+    @Rule
+    public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build();
 
     @Test
-    public void testRecoveryFindsOnlyCurrentEpochAndScopedTempCleanup() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("recovery-source")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
+    public void testPreservedSubsetReplaysWithDictionaryAndKeepsArchive() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        byte[] dict = {4, 'z', 'e', 'r', 'o', 3, 'o', 'n', 'e', 6, 'u', 'n', 'u', 's', 'e', 'd'};
         try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
             appendDeltaFrame(engine, 0, true, "zero");
-            SenderError error = rejection(0).withRejectionSpan(0, 0);
-            RejectedMiniSlotArchive.Result result = RejectedMiniSlotArchive.preserve(
-                    ff, engine, null, source, "slot-recovery", epoch, error);
-            SenderError recovered = RejectedMiniSlotArchive.findOverlapping(
-                    ff, source, "slot-recovery", epoch, 0, 0);
-            assertNotNull(recovered);
-            assertEquals(result.path, recovered.getRejectedPath());
-            assertEquals(0, recovered.getRejectedFsn());
-            assertEquals(null, RejectedMiniSlotArchive.findOverlapping(
-                    ff, source, "slot-recovery", java.util.UUID.randomUUID().toString(), 0, 0));
-
-            Path rejected = Paths.get(source, "rejected");
-            Path ours = Files.createDirectory(rejected.resolve(
-                    ".tmp-slot-recovery-" + epoch + "-fsn-0-0-dead"));
-            Files.createFile(ours.resolve(RejectedMiniSlotArchive.SEGMENT_FILE_NAME));
-            Path other = Files.createDirectory(rejected.resolve(
-                    ".tmp-other-" + epoch + "-fsn-0-0-live"));
-            RejectedMiniSlotArchive.cleanupTemporaryDirectories(
-                    ff, source, "slot-recovery", epoch);
-            assertFalse(Files.exists(ours));
-            assertTrue(Files.exists(other));
+            appendDeltaFrame(engine, 1, true, "one");
+            String message = TestUtils.repeat("mismatch: \u00e9\n\\=", 8192);
+            SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH,
+                    SenderError.Policy.REJECT_AND_CONTINUE, 3, message, 1, 0, 1, "tab", 42);
+            RejectedMiniSlotArchive writer = new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source);
+            RejectedMiniSlotArchive.Result result = writer.preserve(engine, error, dict, 3);
+            assertTrue(result.bytesWritten > 0);
+            Properties metadata = new Properties();
+            try (java.io.InputStream input = Files.newInputStream(Paths.get(result.path,
+                    RejectedMiniSlotArchive.METADATA_FILE_NAME))) {
+                metadata.load(input);
+            }
+            assertEquals("0", metadata.getProperty("fromFsn"));
+            assertEquals("1", metadata.getProperty("toFsn"));
+            assertEquals(message, metadata.getProperty("message"));
+            try (MmapSegment segment = MmapSegment.openExisting(result.path + '/'
+                    + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)) {
+                long second = MmapSegment.HEADER_SIZE;
+                second += MmapSegment.FRAME_HEADER_SIZE
+                        + Unsafe.getUnsafe().getInt(segment.address() + second + 4);
+                long payload = segment.address() + second + MmapSegment.FRAME_HEADER_SIZE;
+                assertEquals(0, Unsafe.getUnsafe().getByte(payload + QwpConstants.HEADER_OFFSET_FLAGS)
+                        & QwpConstants.FLAG_DEFER_COMMIT);
+            }
+            String working = Paths.get(source, "working").toString();
+            copyArchive(result.path, working);
+            try (CursorSendEngine replay = new CursorSendEngine(working, 4096)) {
+                assertEquals(1, replay.publishedFsn());
+                assertEquals(-1, replay.ackedFsn());
+                assertEquals(3, replay.getPersistedSymbolDict().size());
+                replay.acknowledge(1);
+            }
+            assertTrue(Files.exists(Paths.get(result.path, RejectedMiniSlotArchive.SEGMENT_FILE_NAME)));
+            assertEquals(-1, engine.ackedFsn());
         }
     }
 
     @Test
-    public void testRecoveryDoesNotReadDamagedArchivesOutsideRequestedRange() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("unrelated-archives")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
+    public void testSameRangeIsReusedAcrossWriterRestart() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        String namespace = RejectedMiniSlotArchive.namespaceForSource(source);
+        RejectedMiniSlotArchive.Result first;
         try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
-            for (int i = 0; i < 3; i++) appendDeltaFrame(engine, i, true, "symbol" + i);
-            for (long fsn : new long[]{0, 2}) {
-                String archive = RejectedMiniSlotArchive.preserve(
-                        ff, engine, null, source, "slot", epoch, rejection(fsn)).path;
-                Files.write(Paths.get(archive, RejectedMiniSlotArchive.METADATA_FILE_NAME), new byte[]{0});
-            }
-            assertNull(RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 1, 1));
-            // Corruption still fails closed when its range is actually needed.
-            try {
-                RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 0, 0);
-                fail("overlapping damaged metadata must not be ignored");
-            } catch (UnreplayableSlotException expected) {
-                assertTrue(expected.getMessage().contains("invalid rejection metadata size"));
+            appendDeltaFrame(engine, 0, false, "zero");
+            first = new RejectedMiniSlotArchive(
+                    FilesFacade.INSTANCE, source, namespace).preserve(engine, rejection(0), null, 0);
+            assertFalse(first.reused);
+        }
+        try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
+            RejectedMiniSlotArchive.Result second = new RejectedMiniSlotArchive(
+                    FilesFacade.INSTANCE, source, namespace).preserve(engine, rejection(0), null, 0);
+            assertTrue(second.reused);
+            assertEquals(first.path, second.path);
+            assertEquals(0, second.bytesWritten);
+            try (java.util.stream.Stream archives = Files.list(Paths.get(source, "rejected"))) {
+                assertEquals(1, archives.count());
             }
         }
     }
 
     @Test
-    public void testRecoveryRequiresMetadataToMatchDirectoryRange() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("mismatched-archive")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
+    public void testFreshFsnNamespaceDoesNotReuseOldRange() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        String namespace = RejectedMiniSlotArchive.namespaceForSource(source);
+        String first;
+        long firstGeneration;
         try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
-            appendDeltaFrame(engine, 0, true, "zero");
-            String archive = RejectedMiniSlotArchive.preserve(
-                    ff, engine, null, source, "slot", epoch, rejection(0)).path;
-            Files.move(Paths.get(archive), Paths.get(source, "rejected", "slot-" + epoch + "-fsn-0-1"));
-            try {
-                RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 1, 1);
-                fail("overlapping directory with contradictory metadata must fail closed");
-            } catch (UnreplayableSlotException expected) {
-                assertTrue(expected.getMessage().contains("directory identity mismatch"));
-            }
+            appendDeltaFrame(engine, 0, false, "first");
+            firstGeneration = engine.findSegmentContaining(0).generationToken();
+            first = new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source, namespace)
+                    .preserve(engine, rejection(0), null, 0).path;
+            engine.acknowledge(0);
+        }
+        try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
+            appendDeltaFrame(engine, 0, false, "second");
+            assertNotEquals(firstGeneration, engine.findSegmentContaining(0).generationToken());
+            RejectedMiniSlotArchive writer = new RejectedMiniSlotArchive(
+                    FilesFacade.INSTANCE, source, namespace);
+            assertNull(writer.findRecoveredOrphanReport(engine, 0, 0));
+            RejectedMiniSlotArchive.Result second = writer.preserve(engine, rejection(0), null, 0);
+            assertFalse(second.reused);
+            assertNotEquals(first, second.path);
         }
     }
 
     @Test
-    public void testRecoveryIgnoresNoncanonicalDirectoryNames() throws Exception {
-        String source = Files.createDirectory(root.resolve("noncanonical-archives")).toString();
-        String epoch = java.util.UUID.randomUUID().toString();
-        Path rejected = Files.createDirectory(Paths.get(source, "rejected"));
-        String prefix = "slot-" + epoch + "-fsn-";
-        for (String suffix : new String[]{"", "0", "0-0-extra", "-1-0", "2-1", "00-1", "+0-1",
-                "0-9223372036854775808"}) {
-            Files.createDirectory(rejected.resolve(prefix + suffix));
+    public void testRecoveredLookupCleansOnlyExactStagingDirectory() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
+            appendDeltaFrame(engine, 0, false, "zero");
+            RejectedMiniSlotArchive writer = new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source);
+            Path completed = Paths.get(writer.preserve(engine, rejection(0), null, 0).path);
+            Path otherWriter = Paths.get(new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source,
+                    RejectedMiniSlotArchive.namespaceForSource(null))
+                    .preserve(engine, rejection(0), null, 0).path);
+            Path staging = completed.resolveSibling(".tmp-" + completed.getFileName());
+            Files.move(completed, staging);
+            Path unrelated = Files.createDirectories(Paths.get(source, "rejected", ".tmp-unrelated"));
+            Files.write(unrelated.resolve("keep"), new byte[]{1});
+
+            assertNull(new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source)
+                    .findRecoveredOrphanReport(engine, 0, 0));
+            assertFalse(Files.exists(staging));
+            assertTrue(Files.isDirectory(otherWriter));
+            assertArrayEquals(new byte[]{1}, Files.readAllBytes(unrelated.resolve("keep")));
         }
-        assertNull(RejectedMiniSlotArchive.findOverlapping(
-                FilesFacade.INSTANCE, source, "slot", epoch, 0, Long.MAX_VALUE));
+        assertNotEquals(RejectedMiniSlotArchive.namespaceForSource(null),
+                RejectedMiniSlotArchive.namespaceForSource(null));
     }
 
     @Test
-    public void testArchiveReadFailureIsNotReclassifiedAsCorruption() throws Exception {
-        String source = Files.createDirectory(root.resolve("archive-read-failure")).toString();
-        String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, source);
+    public void testParentSyncRetryDoesNotCopyAgain() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        AtomicInteger publications = new AtomicInteger();
+        AtomicInteger rootSyncs = new AtomicInteger();
+        FilesFacade ff = new DelegatingFilesFacade() {
+            @Override
+            public int rename(String from, String to) {
+                int result = super.rename(from, to);
+                if (result == 0) publications.incrementAndGet();
+                return result;
+            }
+            @Override
+            public int fsyncDir(String dir) {
+                if (dir.equals(source + "/rejected") && rootSyncs.incrementAndGet() <= 2) return -1;
+                return super.fsyncDir(dir);
+            }
+        };
         try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
-            appendDeltaFrame(engine, 0, true, "zero");
-            String archive = RejectedMiniSlotArchive.preserve(
-                    FilesFacade.INSTANCE, engine, null, source, "slot", epoch, rejection(0)).path;
-            MmapSegmentException failure = new MmapSegmentException("injected operational read failure");
-            FilesFacade ff = new DelegatingFilesFacade() {
-                @Override
-                public int openRW(String path) {
-                    if (path.equals(archive + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)) throw failure;
-                    return super.openRW(path);
+            appendDeltaFrame(engine, 0, false, "zero");
+            RejectedMiniSlotArchive writer = new RejectedMiniSlotArchive(ff, source);
+            for (int attempt = 0; attempt < 2; attempt++) {
+                try {
+                    writer.preserve(engine, rejection(0), null, 0);
+                    fail("directory barrier must gate successful preservation");
+                } catch (SfOperationalException expected) {
+                    assertEquals(-1, engine.ackedFsn());
                 }
-            };
-            try {
-                RejectedMiniSlotArchive.findOverlapping(ff, source, "slot", epoch, 0, 0);
-                fail("operational read failure must propagate");
-            } catch (MmapSegmentException expected) {
-                assertSame(failure, expected);
             }
-            assertTrue(Files.isDirectory(Paths.get(archive)));
+            RejectedMiniSlotArchive.Result result = writer.preserve(engine, rejection(0), null, 0);
+            assertEquals(1, publications.get());
+            assertTrue(Files.exists(Paths.get(result.path, RejectedMiniSlotArchive.METADATA_FILE_NAME)));
         }
     }
 
     @Test
-    public void testPreservedSubsetReopensWithDictionarySupersetAndWorkingCopyKeepsArchive() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("source")).toString();
-        String dictDir = Files.createDirectory(root.resolve("dict")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
-        try (CursorSendEngine engine = new CursorSendEngine(source, 4096);
-             PersistedSymbolDict dictionary = PersistedSymbolDict.openClean(dictDir)) {
-            dictionary.appendSymbol("zero");
-            dictionary.appendSymbol("one");
-            dictionary.appendSymbol("unused-superset-entry");
-            appendDeltaFrame(engine, 0, true, "zero");
-            appendDeltaFrame(engine, 1, true, "one");
-            String serverMessage = TestUtils.repeat("column mismatch ", 2048);
-            SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH,
-                    SenderError.Policy.REJECT_AND_CONTINUE, 3, serverMessage, 1,
-                    0, 1, "tab", 42).withRejectionSpan(0, 1);
-            RejectedMiniSlotArchive.Result result = RejectedMiniSlotArchive.preserve(
-                    ff, engine, dictionary, source, "slot-0", epoch, error);
-            assertFalse(result.reused);
-            assertTrue(result.bytesWritten > 0);
-
-            RejectedMiniSlotArchive.Metadata metadata = RejectedMiniSlotArchive.readMetadata(ff, result.path);
-            assertEquals(0, metadata.fromFsn);
-            assertEquals(1, metadata.toFsn);
-            assertEquals(serverMessage, metadata.message);
-
-            try (MmapSegment segment = MmapSegment.openExisting(result.path + '/'
-                    + RejectedMiniSlotArchive.SEGMENT_FILE_NAME)) {
-                long second = MmapSegment.HEADER_SIZE;
-                second += MmapSegment.FRAME_HEADER_SIZE
-                        + Unsafe.getUnsafe().getInt(segment.address() + second + 4);
-                long payload = segment.address() + second + MmapSegment.FRAME_HEADER_SIZE;
-                assertEquals(0, Unsafe.getUnsafe().getByte(payload + QwpConstants.HEADER_OFFSET_FLAGS)
-                        & QwpConstants.FLAG_DEFER_COMMIT);
+    public void testFailedCopyCleansOnlyItsOwnStagingDirectory() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
+        Path rejected = Files.createDirectory(Paths.get(source, "rejected"));
+        Path other = Files.createDirectory(rejected.resolve(".tmp-another-writer"));
+        Files.write(other.resolve("keep"), new byte[]{1});
+        AtomicInteger attempts = new AtomicInteger();
+        FilesFacade ff = new DelegatingFilesFacade() {
+            @Override
+            public int openRWExclusive(String path) {
+                if (path.endsWith(RejectedMiniSlotArchive.METADATA_FILE_NAME) && attempts.getAndIncrement() == 0) return -1;
+                return super.openRWExclusive(path);
             }
-
-            RejectedMiniSlotArchive.Result reused = RejectedMiniSlotArchive.preserve(
-                    ff, engine, dictionary, source, "slot-0", epoch, error);
-            assertTrue(reused.reused);
-
-            String working = root.resolve("working").toString();
-            RejectedMiniSlotArchive.copyToWorkingDirectory(ff, result.path, working);
-            assertTrue(ff.exists(result.path + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME));
-            try (CursorSendEngine replay = new CursorSendEngine(working, 4096)) {
-                assertEquals(1, replay.publishedFsn());
-                assertEquals(-1, replay.ackedFsn());
+        };
+        try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
+            appendDeltaFrame(engine, 0, false, "zero");
+            RejectedMiniSlotArchive writer = new RejectedMiniSlotArchive(ff, source);
+            try {
+                writer.preserve(engine, rejection(0), null, 0);
+                fail("injected metadata write failure");
+            } catch (SfOperationalException expected) {
+                assertEquals(-1, engine.ackedFsn());
             }
-            assertTrue(ff.exists(result.path + '/' + RejectedMiniSlotArchive.SEGMENT_FILE_NAME));
+            try (java.util.stream.Stream children = Files.list(rejected)) {
+                assertEquals(1, children.count());
+            }
+            writer.preserve(engine, rejection(0), null, 0);
+            assertArrayEquals(new byte[]{1}, Files.readAllBytes(other.resolve("keep")));
         }
     }
 
     @Test
-    public void testSynchronousPreserverReturnsCompleteCopy() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("sync-source")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
+    public void testExistingRecoveryReaderRejectsDamagedArchiveCopy() throws Exception {
+        String source = temp.newFolder().getAbsolutePath();
         try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
-            appendDeltaFrame(engine, 0, true, "zero");
-            SchemaPreserver preserver = new SchemaPreserver(ff, source, "slot-sync", epoch);
-            RejectedMiniSlotArchive.Result result = preserver.preserve(engine,
-                    rejection(0), new byte[]{4, 'z', 'e', 'r', 'o'}, 1);
-            assertNotNull(RejectedMiniSlotArchive.readMetadata(ff, result.path));
-            try (PersistedSymbolDict dictionary = PersistedSymbolDict.open(ff, result.path)) {
-                assertNotNull(dictionary);
-                assertEquals(1, dictionary.size());
+            appendDeltaFrame(engine, 0, false, "zero");
+            String archive = new RejectedMiniSlotArchive(FilesFacade.INSTANCE, source)
+                    .preserve(engine, rejection(0), null, 0).path;
+            Path segment = Paths.get(archive, RejectedMiniSlotArchive.SEGMENT_FILE_NAME);
+            byte[] bytes = Files.readAllBytes(segment);
+            bytes[0] ^= 1;
+            Files.write(segment, bytes);
+            String working = source + "/working";
+            copyArchive(archive, working);
+            try (CursorSendEngine ignored = new CursorSendEngine(working, 4096)) {
+                fail("damaged archive must not replay");
+            } catch (MmapSegmentCorruptionException | SfRecoveryException expected) {
+                assertArrayEquals(bytes, Files.readAllBytes(segment));
             }
         }
     }
 
-    @Test
-    public void testSynchronousPreserverPropagatesFailure() throws Exception {
-        FilesFacade ff = FilesFacade.INSTANCE;
-        String source = Files.createDirectory(root.resolve("sync-failure")).toString();
-        String epoch = SlotEpoch.openOrCreate(ff, source);
-        try (CursorSendEngine engine = new CursorSendEngine(source, 4096)) {
-            SchemaPreserver preserver = new SchemaPreserver(ff, source, "slot-failure", epoch);
-            try {
-                preserver.preserve(engine, rejection(0), null, 0);
-                fail("missing source frame must fail preservation");
-            } catch (io.questdb.client.cutlass.qwp.client.sf.cursor.SfOperationalException expected) {
-                assertEquals(-1, engine.ackedFsn());
+    private static void copyArchive(String archive, String working) throws Exception {
+        Path target = Files.createDirectory(Paths.get(working));
+        try (java.util.stream.Stream files = Files.list(Paths.get(archive))) {
+            for (Path file : (Iterable) files::iterator) {
+                Files.copy(file, target.resolve(file.getFileName()));
             }
         }
     }
diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java
index 1aa9d2322..990784df5 100644
--- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java
+++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/SchemaRejectionStateTest.java
@@ -16,8 +16,6 @@
 import org.junit.Rule;
 import org.junit.rules.TemporaryFolder;
 
-import java.lang.reflect.Field;
-import java.util.Collection;
 
 import static org.junit.Assert.*;
 
@@ -26,13 +24,12 @@ public class SchemaRejectionStateTest {
     public final TemporaryFolder temp = new TemporaryFolder();
 
     @Test
-    public void testUnackedOrdinaryBorrowsRetainBoundedHistory() throws Exception {
+    public void testLateOrdinaryRejectionAfterManyBorrows() throws Exception {
         SchemaRejectionState state = new SchemaRejectionState();
         for (int i = 0; i < 20_000; i++) {
             state.beginLease(i, i, false);
             state.endLease(i, i);
         }
-        assertEquals(2, retainedRanges(state));
         state.beginLease(20_000, 20_000, false);
         assertTrue(state.reject(10_000, 10_000, error(10_000)));
         assertEquals(10_000, state.sealedRange().lastFsn);
@@ -41,7 +38,7 @@ public void testUnackedOrdinaryBorrowsRetainBoundedHistory() throws Exception {
     }
 
     @Test
-    public void testUnackedTransactionalBorrowsPreserveCommitBoundaries() throws Exception {
+    public void testLateTransactionalRejectionAfterManyBorrows() throws Exception {
         try (CursorSendEngine engine = new CursorSendEngine(null, 4096)) {
             SchemaRejectionState state = new SchemaRejectionState();
             state.setEngine(engine);
@@ -51,8 +48,7 @@ public void testUnackedTransactionalBorrowsPreserveCommitBoundaries() throws Exc
                 append(engine, false);
                 state.endLease(i, 2L * i + 1);
             }
-            assertEquals(2, retainedRanges(state));
-            state.beginLease(20_000, 40_000, true);
+                state.beginLease(20_000, 40_000, true);
             assertTrue(state.reject(20_000, 20_000, error(20_000)));
             assertEquals(20_001, state.sealedRange().lastFsn);
             assertFalse(state.hasOwnedFailure(20_000));
@@ -71,19 +67,20 @@ public void testUnfinishedReturnedTransactionCannotConsumeLaterBorrow() throws E
             state.beginLease(1, 0, true);
             append(engine, true);
             append(engine, true);
+            assertTrue(state.reject(0, 0, error(0)));
+            assertNull(state.sealedRange());
             state.endLease(1, 1);
             state.beginLease(2, 2, true);
             append(engine, false);
             state.endLease(2, 2);
             state.beginLease(3, 3, true);
-            assertTrue(state.reject(0, 0, error(0)));
             assertEquals(1, state.sealedRange().lastFsn);
             assertFalse(state.hasOwnedFailure(3));
         }
     }
 
     @Test
-    public void testPendingRetirementSurvivesBorrowHistoryCompaction() throws Exception {
+    public void testPendingRetirementSurvivesManyBorrows() throws Exception {
         SchemaRejectionState state = new SchemaRejectionState();
         state.beginLease(1, 0, false);
         state.endLease(1, 0);
@@ -92,18 +89,16 @@ public void testPendingRetirementSurvivesBorrowHistoryCompaction() throws Except
             state.beginLease(i, i - 1, false);
             state.endLease(i, i - 1);
         }
-        assertEquals(3, retainedRanges(state));
         assertEquals(0, state.sealedRange().lastFsn);
         state.completeRetirement(0);
         state.beginLease(20_000, 19_999, false);
-        assertEquals(2, retainedRanges(state));
         assertTrue(state.reject(1, 1, error(1)));
         assertEquals(1, state.sealedRange().lastFsn);
         assertFalse(state.hasOwnedFailure(20_000));
     }
 
     @Test
-    public void testCompactionIntoPendingOwnerPreservesItsRangeAndNextBorrowFailure() {
+    public void testPendingRangeAndNextBorrowHaveIndependentFailures() {
         SchemaRejectionState state = new SchemaRejectionState();
         state.beginLease(1, 0, false);
         state.endLease(1, 0);
@@ -112,8 +107,7 @@ public void testCompactionIntoPendingOwnerPreservesItsRangeAndNextBorrowFailure(
         LineSenderServerException failure = state.ownedFailure(2, 1);
         assertSame(failure, state.endLease(2, 1));
 
-        // The failed returned lease absorbs the older range while its
-        // retirement is still pending. The notification must remain [1, 1].
+        // Reborrowing cannot replace the pending notification's original range.
         state.beginLease(3, 2, false);
         assertEquals(1, state.sealedRange().firstFsn);
         assertEquals(1, state.sealedRange().lastFsn);
@@ -128,7 +122,7 @@ public void testCompactionIntoPendingOwnerPreservesItsRangeAndNextBorrowFailure(
     }
 
     @Test
-    public void testEmptyBorrowsDoNotRetainHistoryBehindUnackedRange() throws Exception {
+    public void testEmptyBorrowsDoNotOwnEarlierPublications() throws Exception {
         SchemaRejectionState state = new SchemaRejectionState();
         state.beginLease(0, 0, false);
         state.endLease(0, 0);
@@ -136,17 +130,8 @@ public void testEmptyBorrowsDoNotRetainHistoryBehindUnackedRange() throws Except
             state.beginLease(i, 1, false);
             state.endLease(i, 0);
         }
-        assertEquals(1, retainedRanges(state));
-        state.acknowledgedThrough(0);
         state.beginLease(20_000, 1, false);
         state.endLease(20_000, 0);
-        assertEquals(0, retainedRanges(state));
-    }
-
-    private static int retainedRanges(SchemaRejectionState state) throws Exception {
-        Field field = SchemaRejectionState.class.getDeclaredField("leases");
-        field.setAccessible(true);
-        return ((Collection) field.get(state)).size();
     }
 
     @Test
@@ -248,7 +233,7 @@ public void testRecoveredFrameWithoutLeaseStillRetiresAndReports() {
     }
 
     @Test
-    public void testReturnedUnackedTransactionalLeaseRetainsOwnershipAndFinalEnd() {
+    public void testReturnedUnackedTransactionalLeaseReportsWithoutOwnedFailure() {
         SchemaRejectionState state = new SchemaRejectionState();
         state.beginLease(7, 10, true);
         state.endLease(7, 15);
@@ -256,10 +241,8 @@ public void testReturnedUnackedTransactionalLeaseRetainsOwnershipAndFinalEnd() {
         assertTrue(state.reject(12, 10, error(12)));
         assertEquals(10, state.stopFsn());
         assertEquals(15, state.sealedRange().lastFsn);
-        LineSenderServerException failure = state.ownedFailure(7, 99);
-        assertNotNull(failure);
-        assertEquals(10, failure.getServerError().getFromFsn());
-        assertEquals(15, failure.getServerError().getToFsn());
+        assertFalse(state.hasOwnedFailure(7));
+        assertNull(state.ownedFailure(7, 99));
     }
 
     @Test
@@ -293,6 +276,57 @@ public void testClosedTransactionDoesNotRetireNextTransactionInSameLease() {
         }
     }
 
+    @Test
+    public void testUnclosedReturnDoesNotReleaseProducerOwnership() throws Exception {
+        try (CursorSendEngine engine = new CursorSendEngine(null, 4096)) {
+            SchemaRejectionState state = new SchemaRejectionState();
+            state.setEngine(engine);
+            state.beginLease(1, 0, true);
+            append(engine, true);
+            try {
+                state.endLease(1, 0);
+                fail("normal return needs a closer");
+            } catch (IllegalStateException expected) {
+                assertTrue(expected.getMessage().contains("no commit or rejection boundary"));
+            }
+            // Producer ownership remains intact; only sealing its rejection
+            // permits this unfinished transaction to be followed by a new borrow.
+            assertTrue(state.reject(0, 0, error(0)));
+            assertNotNull(state.endLease(1, 0));
+            state.beginLease(2, 1, true);
+            append(engine, false);
+            assertEquals(0, state.sealedRange().lastFsn);
+            assertFalse(state.hasOwnedFailure(2));
+        }
+    }
+
+    @Test
+    public void testReturnRacingRejectionNeverFailsNextBorrow() throws Exception {
+        java.util.concurrent.ExecutorService io = java.util.concurrent.Executors.newSingleThreadExecutor();
+        try {
+            for (int i = 0; i < 500; i++) {
+                SchemaRejectionState state = new SchemaRejectionState();
+                state.beginLease(1, 0, false);
+                java.util.concurrent.CyclicBarrier start = new java.util.concurrent.CyclicBarrier(2);
+                java.util.concurrent.Future rejected = io.submit(() -> {
+                    start.await(5, java.util.concurrent.TimeUnit.SECONDS);
+                    return state.reject(0, 0, error(0));
+                });
+                start.await(5, java.util.concurrent.TimeUnit.SECONDS);
+                LineSenderServerException returned = state.endLease(1, 0);
+                state.beginLease(2, 1, false);
+                assertTrue(rejected.get(5, java.util.concurrent.TimeUnit.SECONDS));
+                if (returned != null) assertEquals(0, returned.getServerError().getRejectedFsn());
+                assertFalse(state.hasOwnedFailure(2));
+                assertNull(state.ownedFailure(2, 1));
+                assertEquals(0, state.sealedRange().lastFsn);
+            }
+        } finally {
+            io.shutdownNow();
+            assertTrue(io.awaitTermination(5, java.util.concurrent.TimeUnit.SECONDS));
+        }
+    }
+
     private static SenderError error(long fsn) {
         return new SenderError(SenderError.Category.SCHEMA_MISMATCH,
                 SenderError.Policy.REJECT_AND_CONTINUE, 7, "mismatch", 1,
diff --git a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java
index 442289c6b..7c34d7e5e 100644
--- a/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java
+++ b/core/src/test/java/io/questdb/client/test/impl/SchemaRejectionPoolTest.java
@@ -11,12 +11,6 @@
 import io.questdb.client.SenderError;
 import io.questdb.client.cutlass.qwp.client.WebSocketResponse;
 import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.RejectedMiniSlotArchive;
-import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotEpoch;
-import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
-import io.questdb.client.std.FilesFacade;
-import io.questdb.client.std.MemoryTag;
-import io.questdb.client.std.Unsafe;
 import io.questdb.client.test.cutlass.qwp.client.QwpWireTestUtils;
 import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer;
 import org.junit.Assert;
@@ -27,7 +21,6 @@
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.nio.file.Files;
-import java.util.Collection;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CountDownLatch;
@@ -89,7 +82,8 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat
                 if (policy == SenderError.Policy.TERMINAL) {
                     Assert.assertNull(state);
                 } else {
-                    Assert.assertEquals(2, ((Collection) field(state, "leases")).size());
+                    Assert.assertNotNull(field(state, "current"));
+                    Assert.assertNull(field(state, "pending"));
                 }
             }
         }
@@ -101,52 +95,6 @@ private static Object field(Object object, String name) throws Exception {
         return field.get(object);
     }
 
-    @Test
-    public void testRecoveredPreservedOrphanReportsAsynchronouslyBeforeRetirement() throws Exception {
-        String base = temp.newFolder("recovered").getAbsolutePath();
-        String slot = Files.createDirectory(java.nio.file.Paths.get(base, "saved")).toString();
-        String archive;
-        try (CursorSendEngine engine = new CursorSendEngine(slot, 1 << 20)) {
-            String epoch = SlotEpoch.openOrCreate(FilesFacade.INSTANCE, slot, engine.freshFsnNamespace());
-            long frame = Unsafe.malloc(QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT);
-            try {
-                Unsafe.getUnsafe().setMemory(frame, QwpConstants.HEADER_SIZE, (byte) 0);
-                Unsafe.getUnsafe().putInt(frame, QwpConstants.MAGIC_MESSAGE);
-                Unsafe.getUnsafe().putByte(frame + QwpConstants.HEADER_OFFSET_FLAGS, QwpConstants.FLAG_DEFER_COMMIT);
-                engine.appendBlocking(frame, QwpConstants.HEADER_SIZE);
-            } finally {
-                Unsafe.free(frame, QwpConstants.HEADER_SIZE, MemoryTag.NATIVE_DEFAULT);
-            }
-            SenderError error = new SenderError(SenderError.Category.SCHEMA_MISMATCH,
-                    SenderError.Policy.REJECT_AND_CONTINUE, 3, "schema rejected", 0, 0, 0, null, 1);
-            archive = RejectedMiniSlotArchive.preserve(FilesFacade.INSTANCE, engine, null,
-                    slot, "saved", epoch, error).path;
-        }
-        CountDownLatch reported = new CountDownLatch(1);
-        AtomicReference error = new AtomicReference<>();
-        AtomicReference callbackThread = new AtomicReference<>();
-        try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() {
-            public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) {
-                throw new AssertionError("orphan frames must be retired without sending");
-            }
-        })) {
-            server.start();
-            Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS));
-            try (Sender sender = Sender.builder("ws::addr=localhost:" + server.getPort() + ";sf_dir=" + base + ";")
-                    .senderId("saved").errorHandler(e -> {
-                        error.set(e);
-                        callbackThread.set(Thread.currentThread());
-                        reported.countDown();
-                    }).build()) {
-                Assert.assertTrue(reported.await(5, TimeUnit.SECONDS));
-                Assert.assertNotSame(Thread.currentThread(), callbackThread.get());
-                Assert.assertEquals(java.nio.file.Paths.get(archive),
-                        java.nio.file.Paths.get(error.get().getRejectedPath()));
-                Assert.assertEquals(0, sender.getAckedFsn());
-            }
-        }
-    }
-
     @Test
     public void testLazyPoolValidatesDestinationBeforeFirstBorrow() throws Exception {
         String file = temp.newFile("not-a-directory").getAbsolutePath();
@@ -398,7 +346,7 @@ public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] dat
                     a.flush();
                     Assert.assertTrue(firstReceived.await(5, TimeUnit.SECONDS));
                 }
-                // Move the rejected borrow into a compacted historical range.
+                // Let several borrowers publish before the old rejection arrives.
                 for (int i = 0; i < 20; i++) {
                     try (Sender intervening = db.borrowSender()) {
                         intervening.table("good").longColumn("value", i).atNow();
diff --git a/design/schema-mismatch-terminal-resolution.md b/design/schema-mismatch-terminal-resolution.md
index 0ce6ab784..7a20e4fe2 100644
--- a/design/schema-mismatch-terminal-resolution.md
+++ b/design/schema-mismatch-terminal-resolution.md
@@ -1,658 +1,174 @@
-# Schema rejection: report, retire, continue
+# Schema rejection: preserve, retire, continue
 
-Status: Java phases one and two implemented and validated locally. Updated: 2026-09-08. Owner: Jaromir Hamala.
-Supersedes the whole-group/journal revision; see Appendix B for what was
-dropped and why.
+Status: uncommitted simplification experiment based on `26e3c3a` (PR #94).
+Updated: 2026-09-09. No server or wire-format changes.
 
-This amends [`qwp-nack-policy-v2.md`](qwp-nack-policy-v2.md). Today a
-schema-invalid batch latches a `TERMINAL` error, stays in the store-and-forward
-log, is replayed and rejected again on every restart, and parks the slot. The
-new selectable policy is `REJECT_AND_CONTINUE`: fail the handle that owns the data,
-report the rejection, retire the affected frames from replay, and keep
-independent data moving. Phase one retains `TERMINAL` as the default; phase two
-flips the default in the same release as preserved copies.
+## Goal and scope
 
-## Decisions
+A schema-rejected QWP batch must not permanently disable a pooled sender slot.
+Fail the active borrow which published the rejected data, preserve the affected
+queued frames when configured, retire that range, and continue independent
+queued work. Returning a failed borrow allows the next borrower to use the slot.
+A standalone handle remains failed until closed and rebuilt.
 
-1. Validate the NACK against frames actually sent on this connection. Never
-   clamp an invalid sequence into a retirement candidate.
-2. Compute the rejection span by scanning frame flags on the ring, not from a
-   tracked scalar. Ordinary flushes retire the deferred prefix through the
-   rejected frame. Transactional senders retire the whole transaction.
-3. Retire in memory using the existing orphan-tail machinery in the send loop:
-   stop before the sealed span, retain a non-dropping in-memory notification,
-   wait for lower ACKs, self-acknowledge through the span, recycle, continue.
-   Callback completion is not a retirement gate.
-4. No journal. A crash before retirement replays the frames and the server
-   can reject them again. Phase one has the crash-reporting exceptions below;
-   it does not guarantee a callback for every observed NACK across restart.
-5. The handle that published the span fails until returned or rebuilt. Its
-   waits report the rejection. Other handles never see it.
-6. Phase two preserves the span's bytes in the existing segment and dictionary
-   formats. Phase three adds an offline reader. Neither gates phase one.
+This experiment preserves the PR's rejection-range rules, default policy,
+notification capacity, and preserve-before-retirement ordering. It simplifies
+borrow ownership and replaces general archive discovery with an exact lookup for
+a recovered orphan range. It does not implement whole-group rejection for
+ordinary split flushes or set aside entire queues.
 
-Retirement is not acceptance and not proof of rollback. Per-table force
-commits, auto-created tables and auto-added columns can survive a rejection.
-Application resubmission can duplicate rows.
+## Behavioral changes from PR #94
 
-| User | Phase-one default: `TERMINAL` | Explicit phase-one `REJECT_AND_CONTINUE` / phase-two default |
-|---|---|---|
-| Pooled producer | Existing preserve-and-halt behavior; the slot can remain poisoned. | Return and reborrow; the slot progresses. Ordinary unrejected prefix rows are retired too: lost in phase one, copied in phase two when export is enabled. |
-| Standalone producer | Rebuild may encounter the same rejection. | Handle state resets on rebuild; unretired frames may re-reject. |
-| Transactional producer after a crash | Existing recovery behavior remains unchanged; do not assume every open tail is re-sent. | A crash before preservation/notification can yield orphan retirement, no callback and no preserved bytes. |
-| Handler author | Existing bounded best-effort notifications. | Non-dropping while running, but crash/shutdown can lose pending notifications even after retirement. Phase two supplies a ready preserved-copy path. |
-| Operator | Must explicitly opt in to retirement without a copy. | Phase two defaults to preservation before retirement; export opt-out accepts payload loss. |
-
-## Shipping phases
-
-| Phase | Ships | Boundary |
-|---|---|---|
-| 1 | NACK validation, span computation, in-memory retirement/notifications, selectable `REJECT_AND_CONTINUE`, failed-handle ownership, recovery drain fix, two counters. | `TERMINAL` remains default. No new on-disk state; process-local queue identity only. Opt-in retirement discards prefix bytes and requires the source. |
-| 2 | Span copy in existing segment/dictionary formats, durable slot epoch, build-time destination probe, opt-out and storage counters. | Flip the schema default to `REJECT_AND_CONTINUE` only when this ships; copy is default-on with `sf_dir` and gates retirement. |
-| 3 | Offline reader over copied files; JSONL export. | Separate release. |
-
-Phase one does not fix the pooled poison demo out of the box: its default
-`TERMINAL` policy still parks the slot. The demo progresses only when explicitly
-configured with `REJECT_AND_CONTINUE`, until phase two changes the default.
-
-The default flip and preserved-copy support ship together, not in separate
-releases. Explicit phase-one users accept that ordinary A/B/C/D split-flush
-rejection at C retires A and B as well as C, while D can commit. Phase two does
-not recover bytes already discarded by phase one. Memory-only phase-two users
-without a destination, and export opt-outs, retain their own source data.
-
-## Terms
-
-| Term | Meaning |
+| Area | Experiment |
 |---|---|
-| Frame / FSN | One locally published ingest frame and its sequence number. |
-| Commit-bearing frame | A frame without `FLAG_DEFER_COMMIT`. Commits it and every deferred frame before it. |
-| Group start | The frame after the last commit-bearing frame below the rejected FSN. |
-| Rejection span | Inclusive FSN range retired for one rejection. |
-| Retired | Locally acknowledged without server acceptance; never sent again. |
-| Owning lease | The borrow (or the standalone sender's lifetime) that published the span. |
-
-## Retirement mechanism
-
-### Validate the NACK
-
-`handleServerRejection` currently clamps out-of-range wire sequences so that an
-error can still be attributed. Keep that for reporting; never feed a clamped or
-pre-send sequence into retirement. A NACK retires data only when its sequence
-maps to a data frame sent on this connection at or above the replay start.
-
-### Compute the span
-
-Add a read-only engine API for live per-FSN QWP header flags; recovery-only
-`RecoveredFrameAnalysis` is not such an API. Pin or otherwise protect the read
-against trimming and validate the frame. Scan forward from a captured safe floor
-of `ackedFsn + 1` to just before the rejected FSN, retaining the latest
-commit-bearing boundary. The group starts just after that boundary, or at the
-floor if none exists. A cold lookup cache makes this scan and archive copying
-linear without indexing frames during healthy publication. The latest published commit boundary is the wrong scalar: a NACK can
-arrive after a later closer was published.
-
-Ordinary senders retire `[group start, rejected FSN]`. Valid deferred
-predecessors remain in the ring after server rollback, but this policy deliberately
-retires them along with the rejected frame. Published successors replay and may
-commit as a partial flush. The split-flush javadoc's partial-publication caveat
-does not itself authorize that disposal; it is an explicit policy trade-off,
-opt-in until preserved copies ship.
-
-Transactional senders retire from group start through their published closer,
-or through their published open tail if no closer exists. Capture transaction
-mode with the owning generation; never use a later borrower's settings. A closed
-span ends at the first commit-bearing frame at or after the rejection, not the
-lease's last published frame: one lease can contain several transactions. It
-can retire without waiting for lease return. An unclosed transaction follows
-this producer-side sealing protocol:
-
-1. On rejection, fail the owning generation and install a replay stop at the
-   span's start. Do not report a provisional end as final.
-2. The next producer call that observes the failure, or lease return, excludes
-   further publication. Sender methods already require one producer thread.
-3. Read `publishedFsn` and look for the first closer after the rejected frame.
-   Use that closer, or the published tip if still open. This snapshot is the
-   sealing point; the first exception carries immutable final bounds.
-4. On return, skip flush and discard staged rows, then advance the generation
-   and make the slot available to the next borrower.
-5. Queue the final-span notification and let retirement proceed once lower ACKs
-   and, in phase two, the preserved copy are ready.
-
-A held or leaked failed lease that neither observes its failure nor returns
-can stall its unclosed span indefinitely. Either producer observation or return
-seals it; callback completion does not. A later borrower's
-closer cannot commit its rows because the replay stop and retirement precede
-sending beyond the sealed span. Validate the publication race protocol under
-the phase-one gate in Open decisions.
-
-Recovery does not persist the previous producer's transaction mode. Treat a
-rejected recovered group conservatively as transactional: retire through its
-first recovered commit-bearing frame, or its recovered open tail. Never use a
-new producer's closer. This also retires the tail of an ordinary split flush
-after restart; phase two preserves those rows in the copy.
-
-### Retire in the send loop
-
-`CursorWebSocketSendLoop` already stops at a recovered orphan tail, and
-`retireRecoveredOrphanTailIfReady` self-acknowledges it once every lower frame
-is acknowledged, then recycles the connection to re-anchor the arithmetic
-wire-sequence-to-FSN mapping. Generalize that range so it can be set live by
-the I/O thread on a NACK. Retain the final-span notification before advancing
-the watermark, but do not wait for its callback. Phase two additionally waits
-for the preserved copy to be durably published.
-
-I/O-side order on a NACK (return-side sealing and callback execution are separate):
-
-1. Validate. Compute the span. Latch the error on the owning lease.
-2. Disconnect and recycle immediately; do not wait for lease return or
-   notification capacity.
-3. Replay from the watermark. Independent frames below the span send and
-   acknowledge normally. Stop before the span.
-4. Once the span is sealed and lower frames have their configured ACKs, retain
-   its notification in the FIFO and signal the dispatcher. If the FIFO is full,
-   stop here until capacity is available. Then advance the watermark through
-   the span without waiting for callback completion. In phase two, the copy
-   must be durable before notification enqueue and watermark advancement.
-5. Recycle, continue with frames above the span.
-
-A second schema rejection below an already pending retirement range cannot be
-merged safely by this implementation. Log the second rejected FSN and fall back
-to `TERMINAL`, retaining source bytes. This can occur when a predecessor is
-re-rejected while replaying for durable ACKs. Invalid closer scans or failed
-skipped-range dictionary reconstruction also fail closed instead of reconnecting
-forever.
-
-No sparse ACK map, no second watermark file, no change to the watermark or
-segment encodings. Each connection still sends one contiguous range. The general path uses two
-recycles, after NACK and after retirement; each may ship dictionary catch-up.
-Setup can sometimes retire before any wire sequence is consumed, but do not
-assume that shortcut when estimating rejection cost.
-
-Retired frames may be the only carriers of dictionary deltas referenced by
-successors. Disk-backed catch-up must use the persisted dictionary covering
-those deltas, not only replayed frames. Memory mode uses the live dictionary
-mirror/snapshot; verify it covers skipped, possibly unsent transactional frames
-before reclaiming them. Full-dictionary frames remain self-sufficient. Neither
-mode may lose symbols because their carrier frame was retired. This is new live
-retirement behavior: recovered orphan tails have no successors requiring those
-skipped deltas. Spike this first in memory and disk modes before committing to
-the phase-one schedule; existing orphan tests are not sufficient evidence.
-
-### Crash behaviour
-
-Phase one does not guarantee at-least-once callbacks across restart. A pending
-notification is memory-only and retirement does not wait for invocation:
-
-- Before durable retirement, a closed span may replay and re-reject if the
-  schema is unchanged, producing another callback opportunity.
-- Without a closer, recovery may retire the open tail without sending it:
-  zero callbacks and no preserved bytes are possible. This is not presented as
-  a reporting guarantee equivalent to repeated `TERMINAL` rejection.
-- After durable retirement but before callback invocation, a crash loses the
-  notification and there is no replay to reconstruct it.
-- A changed schema may allow replay to succeed without recreating the original
-  rejection. Before watermark durability, duplicate reports remain possible.
-
-Phase two preserves evidence once its directory is durably published; recovery
-can report from that copy. A crash before copy publication retains the open-tail
-reporting gap. Memory-only queues cannot reconstruct data lost with the process.
-The default phase-one `TERMINAL` path is unchanged; these are the guarantees of
-the explicit retirement policy, not reasons to claim unconditional delivery.
-
-## Ownership and handle contract
-
-A rejection belongs to the lease whose published FSN range contains the
-rejected FSN. Each slot records the lease generation and its first published
-FSN at borrow; failed/observed state and the return-side end snapshot above
-are also required. Use the highest already published FSN plus one as the start. Row-level calls in
-`QwpWebSocketSender.checkConnectionError` and the `PooledSender` wrapper both
-check it, since row calls poll the delegate directly. Empty borrows retain no
-lease record. `TERMINAL` does not allocate schema ownership state. Under
-`REJECT_AND_CONTINUE`, the next borrow removes the returned handle's generation
-and exception and coalesces adjacent returned ranges with the same transaction
-mode. Ordinary ranges can always coalesce; transactional ranges can coalesce
-only across a commit-bearing return boundary. The queued frame flags still
-identify each closed transaction's end. An unfinished failed transaction keeps
-its own end boundary, and a pending retirement keeps its owner object until
-completion. Thus normal publishing borrows during an ACK stall retain one
-historical range plus the current or most recently returned lease, rather than
-one record per borrow. Active-lease lookup remains constant time. Resolved
-progress prunes obsolete ranges.
-
-An owned rejection fails the handle. Every subsequent publish, wait and drain
-on that handle throws the same `LineSenderServerException` until the handle is
-returned. A failed pooled lease needs a distinct `PooledSender.close()` path:
-skip `flush()`, discard its staged rows, seal any pending span, and give the slot
-back; only then throw the owned error if not already observed. Do not route this
-known lease-local rejection through `discardBroken`. That path remains for real
-sender/storage/cleanup failures. A stale or repeated close is a no-op. A healthy
-close racing a newly owned rejection must take this same cleanup path rather
-than discard the slot simply because its flush observed that rejection.
-
-A standalone sender is one lifetime lease. Its failed close seals/discards local
-work and releases resources but does not wait for rejection retirement. Skip
-`drainOnClose` for this failure. Rebuild clears public handle state; if retirement
-was not durable, replay and another rejection are allowed. Close is not a promise
-that the old rejection can never recur. Preservation runs on the I/O thread.
-Close waits for that thread within its
-existing shutdown budget. If disk I/O outlasts the budget, close reports the
-shutdown failure and the existing I/O-thread cleanup fallback retains the
-engine and slot lock until the thread exits. Immediate rebuild can therefore
-still encounter lock contention. There is no separate preservation worker or
-preserver-specific deferred cleanup.
-
-Callback completion no longer participates in retirement, so handler-initiated
-close creates no callback/retirement dependency cycle. Preserve ordinary safe
-dispatcher shutdown (never join the current thread); no synthetic callback-return
-signal or special retirement-completion protocol is needed. Close/rebuild can
-still encounter any frame whose retirement was not durable.
-
-Waits on a failed handle throw even for FSNs the server accepted. That is
-deliberate: a wait returning true for a retired FSN would be a false delivery
-confirmation, which is worse than a spurious throw. The exception carries the
-span so the caller can tell which batches are actually affected.
-
-Healthy handles keep today's slot-wide wait and drain semantics. Retirement
-advances the watermark, so a healthy borrower's drain completes past another
-borrower's retired span rather than hanging. Rejections from earlier borrows
-never fail a later handle. Consequently a healthy B waiting on A's retired FSN
-can return true: for cross-borrow targets this is resolved progress, not delivery
-confirmation. Acceptance conclusions are limited to the caller's own publications
-while its handle is healthy. This ownership limit is part of the API contract;
-it is why clearing A's owned failure would be different from letting B progress.
-Do not describe a slot-wide wait on an arbitrary old FSN as proof of ingestion.
-
-Pool startup recovery treats a retired span as progress. Today it reports
-`RecoveryDrainOutcome.FAILED`, retries, and parks the slot on a failure streak;
-that path is what poisons the scan.
-
-## Reporting
-
-Retain schema notifications in a per-slot sticky FIFO with capacity **256
-entries**, separate from the ordinary drop-oldest deque. This is a fixed initial
-implementation limit, not a new public setting. Count the callback currently in
-progress against the 256; release its capacity only when invocation completes,
-including when the handler throws. Ordinary overflow cannot evict schema entries.
-Keep the same entry across reconnect attempts for an in-progress retirement.
-Signal the dispatcher without waiting for user code. Callbacks run outside I/O,
-pool and queue locks; log and contain thrown exceptions.
-
-A stuck handler permits up to 256 retained rejections in that slot, including
-its current invocation. The next rejection cannot retire until an entry completes
-and frees capacity. Keep that rejection in the live retirement state and leave
-its frames unretired; do not drop or overwrite a notification. The I/O loop
-remains responsive and independent slots have their own FIFO capacity. A shared
-dispatcher can still delay their handlers, eventually filling their FIFOs too.
-
-Thus slow-handler behavior degrades to a callback-dependent retirement stall
-only once the 256-entry allowance is exhausted. Failure to allocate/retain an
-entry also leaves the span unretired. This bounds retained notification count,
-not arbitrary server-message bytes. Shutdown may abandon the volatile backlog;
-no durable delivery is promised.
-
-Install an effective handler and dispatcher for every reporting path: custom
-when supplied, otherwise the default one-line logger. `SenderPool` currently
-creates its recovery dispatcher only with a custom handler and SFA enabled;
-wire the default path too. Startup schema callbacks run asynchronously, never
-on the thread calling `build()`. Recovery does not wait for callback completion while its FIFO has capacity.
-
-Phase-one identity is slot ID + process-local queue instance + trigger FSN.
-Allocate a new instance identity on queue recreation; it is not a cross-restart
-deduplication key. Durable slot epoch and legacy initialization are phase-two
-work because the preserved directory needs stable identity. No epoch sidecar or
-migration is introduced in phase one.
-
-The error carries: category, policy `REJECT_AND_CONTINUE`, rejected FSN, span
-from/to, server message, table when known, and in phase two the preserved file
-path once it is published. It states that the span will not be retried, that
-server side effects may remain, and that resubmission needs the source.
-
-## Policy and API
-
-- Add `SenderError.Policy.REJECT_AND_CONTINUE`. Do not reuse `TERMINAL`
-  (bytes preserved, sender halted) or `ABANDONED` (no throw, `DATA_LOSS` only).
-  Update the default handler's log line and policy switches.
-- Phase one keeps `TERMINAL` as the schema default; phase two switches to
-  `REJECT_AND_CONTINUE` with preserved-copy support. Add one builder method, provisionally
-  `schemaMismatchPolicy(TERMINAL | REJECT_AND_CONTINUE)`, as the escape hatch.
-  Do not implement the resolver precedence chain or wire `on_schema_error` in
-  this change; the connect-string key stays a consumed no-op as today. Correct
-  `Policy` javadoc to describe the actually implemented builder override and
-  default, removing the non-existent resolver/precedence claim. Document that
-  the reserved connection-string key does not enable the escape hatch.
-- `LineSenderServerException.getServerError()` exposes the rejected FSN via a
-  new accessor plus the existing `getFromFsn()` and `getToFsn()` span. Include
-  all three in the message.
-- `getAckedFsn()` is documented as the resolved watermark: acceptance or local
-  retirement. It already advances for recovered orphan tails. No second
-  accessor.
-
-Other categories are unchanged. `PARSE_ERROR` and `SECURITY_ERROR` remain
-`TERMINAL` and can still poison a persistent slot; enabling them needs their
-own attribution review, since malformed input can compromise the flag scan
-that computes spans.
-
-## Server baseline
-
-The mechanism relies on two server behaviours, both present in the inspected
-source (`QwpIngressUpgradeProcessor.handleBinaryMessage`): deferred rows are rolled
-back before the NACK is sent, and frames after a NACK are consumed without
-processing or reply until disconnect. `QwpSenderE2ETest.testDeferredCommitSchemaMismatchRollsBack`
-covers the first. Post-NACK reply handling is whatever the existing retriable
-recycle does today; this change adds no new handling. The server does not close
-the connection after a NACK: client-side disconnect is required to resume. Older
-servers without the unresolved-sequence gate can apply successors on that same
-connection; ignoring their replies and replaying can duplicate rows. The minimum
-supported-release note must explicitly exclude that behavior.
-
-The supported baseline for this policy is QuestDB 10.0.0 or later. The 10.0.0
-source tag contains both the unresolved-sequence gate and rollback before NACK.
-The rollback E2E test passes against the local 10.0.1-SNAPSHOT checkout at
-`496b24d996ea321015c7cfeabcbfc7e563e053e7`, using the current client artifact.
-Its harness now needs to observe the owning handle's failure before close;
-receiving the callback alone does not consume that failure. A temporary test
-adaptation verified the final span and allowed the unchanged database rollback
-assertions to run; the server checkout was then restored.
-No capability negotiation is added.
-
-## Phase two: preserve the span
-
-Introduce a durable slot epoch for directory/deduplication identity. Preserve it
-across restart of the same queue and change it when a clean queue restarts FSNs.
-Initialize/migrate under the exclusive slot lock before creating copies; existing
-identity metadata may be reused only if it provides that lifecycle. This sidecar
-is identity metadata, not a phase-one rejection journal.
-
-Before either retirement or ready-path callback enqueue, copy the span into
-`//rejected/--fsn--/` as a mini slot: one
-segment file in the existing `MmapSegment` format holding the span's frames in
-order, plus a frozen full dictionary snapshot covering the copied frames. The
-I/O mirror folds skipped deltas before taking the snapshot, including in memory
-mode. It may contain unused later entries; preserving this superset avoids a
-decoder and reconstructing historical dictionary versions. Both
-formats already carry magic, version and CRC and have Java/Rust fixtures.
-Add rejection metadata sufficient for the promised recovery callback (epoch,
-trigger/span, category/status and message); existing segment headers alone do
-not contain it. Define and validate that metadata file without calling it an
-existing SFA field. Final-directory publication covers all constituent files.
-
-Rewrite the last copied frame's QWP flags to clear `FLAG_DEFER_COMMIT` and
-recompute that frame's CRC, so the copy is a closed unit. Without that, the
-existing recovery reader classifies a deferred-only copy as an orphan tail and
-retires it unsent. Validation must confirm that a copied subset with its
-dictionary snapshot replays through the existing reader; this is validation of
-an existing format, not a new one.
-
-Write into a process-unique temporary directory, sync, rename, sync the parent.
-A directory at its final name is complete. Under the queue lifecycle lock,
-remove unfinished directories matching that exact slot and epoch. Do not remove
-other epochs' temporary directories: a shared memory-only destination cannot
-prove that another queue has stopped. Such leftovers require operator cleanup.
-Reuse an existing final directory
-with the same identity on re-rejection after a crash. Copy synchronously on the
-I/O thread with bounded buffers, after lower frames are acknowledged and the
-rejection span is sealed. No data above the span may be sent until the copy is
-complete, so a separate copy worker cannot advance ingestion. A large span,
-full dictionary or slow filesystem delays I/O-thread responsiveness, including
-keepalives and shutdown; reconnect handles an expired connection. Healthy
-publication is unchanged.
-
-Keep the completed-copy notification until the bounded FIFO admits it, avoiding
-repeated archive validation and directory sync while the handler queue is full.
-On copy failure keep the frames unretired, log the failure, and wait using the
-stop-aware backoff before retrying, with bounded exponential pacing. A failed copy does not latch a
-fatal sender error. Before retry, remove unfinished temporary directories for
-this exact slot and epoch, so repeated failures do not accumulate partial copies.
-
-Java settings are builder methods on `Sender` and `QuestDB`: preservation is
-enabled by default with `sf_dir`; `dlqEnabled(false)` opts out;
-`dlqDirectory(path)` selects `path//rejected/` and gives memory-only
-senders a destination. No new connection-string keys are introduced. Probe the
-destination at build time, including for a lazy pool with no warm connections. Files are never deleted by the client. TLS does not protect these
-bytes at rest. Quarantining a damaged slot also moves its `rejected/` directory;
-previously reported paths then change. Use the quarantine path reported by the
-`DATA_LOSS` event to locate those copies. A ready path guarantees completeness
-when delivered, not a permanent location. Document alongside existing `sf_dir` behaviour and use
-restrictive permissions. Each copy may carry the full dictionary; that is the
-cost of not decoding and is counted in `dlq_bytes_written_total`; alarm on sustained growth of that
-counter and filesystem free space, since no automatic retention bounds it.
-
-Recovery: a preserved directory whose span overlaps an orphan tail at startup
-may mean a crash before callback completion or retirement; it does not prove
-that the callback never ran. Dispatch the callback from
-the directory's rejection metadata and retire the tail. For an orphan-only slot,
-retain that notification and retire locally before attempting a connection. An
-unreachable server must not prevent this socket-free cleanup; callback execution
-remains asynchronous.
-
-Startup filters completed archive directories by their canonical slot, epoch
-and FSN range before opening metadata. Damage in an unrelated, already-drained
-archive cannot block recovery. An overlapping archive must pass metadata,
-directory identity and replay-file validation. Proven corruption preserves the
-`UnreplayableSlotException` type through startup cleanup so `Sender.build()`
-can quarantine the whole slot, report `DATA_LOSS` and continue on a fresh one.
-Operational storage failures do not become corruption verdicts.
-
-With export disabled, a persistent schema fault retires indefinitely with one
-paced report per span and no circuit breaker. That is the accepted cost of
-opting out.
-
-## Phase three: offline reader
-
-A CLI or static helper opens a preserved directory with the existing recovery
-reader and, once an ingest-frame decoder exists, exports JSONL: one metadata
-line, one row per line, a row-count trailer. Resubmission after a schema fix
-first copies the preserved directory to a separate working slot using
-`RejectedMiniSlotArchive.copyToWorkingDirectory`, then opens that working slot.
-Opening the original directly would let normal drain cleanup destroy the evidence.
-This reuses
-the replay path and needs no decoder. Decoder failures never touch ingestion.
-
-## Rust and C/C++
-
-Rust's SFA path under `questdb-rs/src/ingress/sender/` publishes non-deferred
-frames (`qwp_ws_publisher.rs::encode_to_scratch` passes `false` to the defer
-argument); `qwp_ws_sfa_queue.rs` is the backing queue. Its spans are singletons.
-Do not generalize this to `column_sender/sender.rs`: that file also contains a
-direct path with deferred split prefixes. Its `rebase_lease_observation` method
-is the lease-rebase citation, not `db.rs`. Implement the same ownership,
-failed-handle, callback-independent retirement and phased policy surface. `Drop` releases
-the lease but cannot throw; the callback is the only report there. Phase one
-adds no on-disk state, so a Java slot after retirement is readable by any
-current client, subject to the existing format contract. Phase-two epoch
-metadata does not encode replay decisions. Phase two's `rejected/` directory is ignorable by clients that
-do not know it. No cross-client gate is needed.
-
-## Pacing, metrics, healthy path
-
-- Reconnect after a rejection uses the existing reconnect backoff, reset on
-  any real ACK. Distinct rejected FSNs do not accumulate their own strike count;
-  the same-FSN poison detector is unchanged and still escalates a frame that is
-  rejected without ever being retired.
-The Java observation methods are on `QwpWebSocketSender`:
-
-| Counter | Accessor | Meaning |
-|---|---|---|
-| `schema_frames_retired_total` | `getSchemaFramesRetired()` | Frames locally retired; not accepted rows. |
-| `schema_rejections_total` | `getSchemaRejections()` | Attributed schema NACKs. |
-| `dlq_files_written_total` | `getDlqFilesWritten()` | Newly published archive directories; reuse is not counted. |
-| `dlq_bytes_written_total` | `getDlqBytesWritten()` | Bytes in newly published archives, including dictionary copies. |
-| `dlq_write_failures_total` | `getDlqWriteFailures()` | Preservation failures; source frames remain queued. |
-
-- Healthy publication stays allocation-free per row and takes no new lock per
-  frame. Generation boundaries are written at borrow/return; prove race safety for
-  failure checks and sealing without a new per-frame lock. Existing ingestion benchmarks gate the change.
-
-## Validation
-
-- Reject a singleton, a middle deferred frame and a closing frame; span matches
-  the commit-boundary scan, including when a later closer was already published.
-- Ordinary split flush: prefix retired, successors commit as a partial batch,
-  documented. Transactional: whole transaction retired, staged rows discarded,
-  no partial commit, including when the closer was published before the NACK.
-- Interrupted split flush from a failed lease: its deferred tail is retired;
-  B's first closer never commits A's rows.
-- Lower independent frames acknowledge before self-ack; frames above the span
-  replay after recycle; dictionary catch-up sequences re-anchor correctly.
-- Invalid, pre-send and catch-up NACKs never retire. A second schema NACK
-  during pending retirement reports `TERMINAL` and retains the stopped range.
-- Invalid transaction scans and skipped dictionary reconstruction fail closed;
-  transient catch-up send failures preserve the existing cap-gap bookkeeping.
-- Recovered rejected groups include their original closer or recovered tip,
-  irrespective of the new producer's mode; a new closer is never included.
-- Orphan-only slots retire without connecting, with a preserved metadata report
-  retained asynchronously first when present. Copy failures retry with pacing
-  and temporary cleanup, without advancing the watermark or latching fatal.
-- Ownership: A's rejection after B borrowed fails neither B nor B's waits.
-  Failed handle: publish, wait, empty wait and drain throw; first close throws
-  only if unobserved, skips flush and returns the slot;
-  repeated close is idempotent; reborrow is healthy. Standalone: rebuild on the
-  same `sf_dir` clears handle state but may replay unretired frames.
-- Slow/throwing handlers do not gate retirement below FIFO capacity; pending entries
-  survive ordinary inbox overflow, remain distinct and dispatch when unblocked.
-  Failure to retain an entry prevents that span's retirement without dropping
-  earlier notifications. With one blocked callback, 256 retained entries permit
-  progress; the next span waits until callback completion frees capacity. Test
-  shutdown/crash loss of the volatile backlog.
-- Crash before self-ack with closer published: replay, re-reject, callback
-  opportunity. Crash after durable self-ack but before invocation can lose the
-  callback. Crash without closer: orphan retirement, and in phase two the
-  callback from the preserved directory. Crash after self-ack: no replay.
-- Pool recovery with and without custom handlers drains through bad, bad, good.
-- Handler closes pooled/standalone senders and the pool without self-join or
-  retirement waits; rebuild before durable retirement can re-reject.
-- Unclosed transactional span waits for return; generation/end snapshot excludes
-  B's frames. Confirm pending span callbacks cannot report a provisional end.
-- Live header reads respect the watermark floor and concurrent trimming;
-  memory and disk catch-up include deltas carried only by retired frames.
-- Phase-one process-local identities distinguish recreated queues without disk
-  state. Phase-two epoch survives restart but changes on FSN reset.
-- Phase-one defaults preserve/halt; explicit retirement loses ordinary prefix
-  rows as documented. Phase-two default flips only with durable-copy support.
-- A healthy cross-borrow wait reports resolution, not acceptance of retired data.
-- Phase two: copy replays through the existing reader; flag rewrite and CRC;
-  atomic publish; reuse on re-rejection; disk full; opt-out; probe.
-- Both ACK levels; force commits and surviving schema side effects; benchmark
-  non-regression.
-
-## Local validation (2026-09-07)
-
-- Full core suite: 3,480 tests, zero failures/errors, seven skipped.
-- Final targeted integration run after the last boundary and lookup changes:
-  181 tests, zero failures/errors. Includes pool return/reborrow, standalone
-  orphan draining, preserved-tail startup reporting, dictionary continuity,
-  archive reuse/recovery, bounded callbacks and trim-safe frame lookup.
-- Examples reactor package succeeds.
-- Server rollback E2E succeeds with the new owning-handle observation noted
-  above. No server implementation changes are required.
-- Fixed-work healthy producer check: one million rows, batches of 1,000,
-  three alternating runs per build. Both builds measure zero producer bytes
-  allocated per row; observed times are approximately 39–43 ns/row. This is a
-  narrow memory-mode check, not a disk or end-to-end throughput claim.
-- The rejected-range lookup previously scaled quadratically (1,000/2,000/4,000
-  lookups took about 1.1/4.2/15.3 ms). The cold forward lookup cache removes the
-  repeated scans, with no healthy publish-path index, lock or allocation.
-
-### Review follow-up (2026-09-08)
-
-- Full core suite: 3,489 tests, zero failures/errors, seven skipped.
-- Final affected-suite run after the last test and policy-reporting corrections:
-  98 tests, zero failures/errors. Covers recovered group boundaries,
-  socket-free orphan retirement with and without preserved metadata, second
-  schema NACK under durable ACK replay, invalid closer scans, missing skipped
-  frames, preservation failure followed by successful retry, and existing
-  dictionary, archive, orphan-tail and pool regressions.
-- `git diff --check` passes.
-
-### Synchronous preservation follow-up (2026-09-08)
-
-- Removed the preservation worker, request/completion handshake and separate
-  shutdown coordination. The I/O thread now owns copying and retirement.
-- Retained a completed-copy notification while the FIFO is full, stop-aware
-  retry pacing, and the existing I/O-thread cleanup fallback.
-- Full core suite: 3,492 tests, zero failures/errors, seven skipped. Regression
-  coverage includes successful retry, no repeated archive sync while the FIFO
-  is full, and a blocked synchronous copy retaining the engine lock through a
-  shutdown timeout until delegated cleanup finishes.
-- `git diff --check` passes.
-
-## Open decisions
-
-| Item | Owner | Gate |
-|---|---|---|
-| Minimum server release: 10.0.0; local rollback E2E passed on 10.0.1-SNAPSHOT. | Server/QWP maintainer | Resolved |
-| Java builder method is `schemaMismatchPolicy`; Rust/C/C++ implementation remains a separate client deliverable. | Client API maintainers | Java resolved |
-| Copied subset/dictionary recovery, last-flag CRC and archive reuse verified by tests. | Persistence maintainer | Resolved |
-| Skipped dictionary carrier recovery verified in memory and disk modes. | Persistence/I/O maintainers | Resolved |
-| Live-header API, first-closer bounds and atomic return-side failure capture tested; healthy stop/ACK checks use volatile fields. | Persistence/pool maintainers | Resolved |
-| CRC-protected `.slot-epoch`, fresh-namespace rotation under the slot lock, restart/reuse tests. | Persistence maintainer | Resolved |
-| `RejectedMiniSlotArchive` metadata version 1, CRC, enum names, span and trigger. | Persistence maintainer | Resolved |
-| Diagnostics for full notification FIFOs and unreturned failed leases. | Design owner | Phase 1 |
-
-## Appendix A: source evidence
-
-- `QwpWebSocketSender.flushPendingRowsSplit`: per-table frames with
-  `FLAG_DEFER_COMMIT` on all but the last; its javadoc states the split is not
-  atomic and can deliver a prefix twice.
-- `QwpWebSocketSender.transactional`: auto-flush defers, explicit `flush()`
-  commits; documented as committing atomically per table.
-- `QwpWebSocketSender.checkConnectionError`: row-level calls poll the delegate,
-  so ownership checks cannot live only in `PooledSender`.
-- `CursorSendEngine.retireRecoveredOrphanTailIfReady` and
-  `CursorWebSocketSendLoop.trySendOne`: existing stop, self-ack and recycle for
-  a recovered deferred tail. This is the machinery phase one generalizes.
-- `CursorWebSocketSendLoop.handleServerRejection`: clamps invalid NACK
-  sequences for attribution; unsafe as a retirement input.
-- `SenderPool` recovery drain and failure streak: the poisoned-scan path.
-- `MmapSegment`, `PersistedSymbolDict`: formats reused for phase two; both
-  versioned with CRC, and a foreign version fails recovery without quarantine.
-- Server: `QwpIngressUpgradeProcessor.handleBinaryMessage` withholds ACKs for deferred
-  frames, clears state before the NACK, and consumes later frames without
-  reply. `QwpSenderE2ETest.testDeferredCommitSchemaMismatchRollsBack`.
-- Rust: `questdb-rs/src/ingress/sender/qwp_ws_publisher.rs::encode_to_scratch`
-  passes defer=false for SFA; `sender/qwp_ws_sfa_queue.rs` stores the frames.
-  `column_sender/sender.rs::rebase_lease_observation` rebases lease observation;
-  its direct split path uses deferred prefixes and is a different path.
-
-## Appendix B: alternatives dropped
-
-**Whole-group retirement for every ordinary split flush.** Prefix retirement
-allows ordinary successors to deliver instead of discarding the entire flush.
-It does not eliminate the publication/return linearization and tail sealing
-needed for unclosed transactional or interrupted spans.
-
-**Retire only the ordinary rejected frame.** Could preserve valid predecessors
-still on the ring when an independent closer survives. It is not equivalent to
-prefix disposal and is worth a separate policy decision. If the rejected frame
-is itself the only closer, replayed predecessors have no closer; letting a later
-borrow commit them changes the result again. Define that case, interruption and
-no-successor behavior before selecting this alternative. This revision retains
-prefix retirement explicitly rather than claiming those predecessors are lost
-under either choice.
-
-**Rejection journal.** Gave retained at-least-once callbacks across crashes and
-handler-independent retirement, at the cost of a versioned side file,
-compaction, delivery markers, and a downgrade and cross-client gate that
-released readers cannot honor without a segment version bump. Re-rejection is
-idempotent, so the journal bought a stronger callback guarantee than the data
-path needs. Revisit if the documented zero-notification crash windows are unacceptable.
-
-**Throw once, then clear.** Simpler handle lifecycle, but a slot-wide wait on
-the rejected FSN would then return true after retirement: a false delivery
-confirmation for its own failed publication. Rejected. A later healthy borrow
-may still observe slot-wide resolution for that FSN; the ownership distinction
-is intentional and does not establish acceptance of old data.
-
-**New DLQ container format.** Replaced by a copy in the existing segment and
-dictionary formats, which already have versioning, CRC and fixtures, and which
-the existing recovery reader can replay after a one-byte flag rewrite.
-
-**Server capability gate and distinct-FSN pacer.** Replaced by a stated
-minimum server release and the existing reconnect backoff.
-
-**Callback-gated retirement and phase-one epoch.** Callback completion is not a
-retirement requirement: it adds handler-dependent stalls without durable payload
-recovery. Phase one retains notifications in memory and accepts crash loss.
-Durable epoch identity moves to phase two with preserved directories. Removing the per-rejection gate requires the 256-entry sticky FIFO; only
-capacity exhaustion makes retirement wait for callback completion.
+| Completed borrow history | Keep only the current borrow and one pending rejection. No deque, coalescing, pruning, or per-ACK ownership maintenance. |
+| Transaction mode | Fixed for a sender's borrow lifecycle, as configured by the builders. Switching it between borrows is rejected. |
+| Archive identity | Deterministic from the source namespace, source segment generation token, and exact FSN range; no durable `.slot-epoch`. |
+| Restart | Recover the live SFA queue, then check only the deterministic archive path for an exact recovered orphan range. No directory scan. |
+| Callback reconstruction | A structurally valid completed copy for that still-live orphan range is queued before retirement. Other archives do not reconstruct callbacks. |
+| Duplicate copies | A retry or restart for the same live identity and range reuses the completed copy. |
+| Metadata | `rejection.properties` replaces the custom CRC-protected `rejection-meta.bin` format. Payload segments and dictionary retain their existing checked formats. |
+| Interrupted copies | Recovery or preservation removes the deterministic staging tree for that exact live range. Unrelated and legacy staging trees remain untouched. |
+
+There is no public archive reader or replay-copy API in the ingestion client.
+Copy an archive directory to a separate working directory and use the existing
+SFA recovery reader. Both old and new copies retain the same SFA payload formats;
+the metadata filenames differ. Old PR archives and `.slot-epoch` files are left
+untouched and cannot block startup. The internal exact-path lookup reads the
+properties only to validate and reconstruct the matching orphan report; there is
+no archive index or general notification log.
+
+## Rejection boundaries
+
+An FSN is a local frame sequence number. A commit-bearing frame has
+`FLAG_DEFER_COMMIT` clear. The group starts after the last commit-bearing frame
+below the rejected FSN, bounded below by the unresolved queue floor.
+
+- Ordinary flush: retire the deferred prefix through the rejected frame.
+  Published successors remain eligible for replay and may produce a partial flush.
+- Transactional sender: retire through the first commit-bearing frame at or after
+  the rejection. If no closer has been published, the producer must seal its
+  published tail when it observes the failure or returns the borrow.
+- Recovered data: transaction mode is unknown, so retire conservatively through
+  the recovered group's closer or recovered open tail. Never use a new
+  producer's closer to determine an old group's end.
+
+Retirement does not imply server rollback. Schema changes and previously forced
+commits can survive rejection, so manually replaying a copy can duplicate rows.
+
+## Ownership without history
+
+`SchemaRejectionState` holds the current borrow and one pending rejection.
+The current record contains generation, first FSN, transaction mode, active/end
+state, and its first immutable failure. A pending rejection may retain the record
+of a returned failed borrow until its range retires; it does not retain any
+other historical borrows.
+
+The NACK must first be validated against frames actually sent on the current
+connection. An active borrow owns it only if the rejected FSN is at or above
+that borrow's first FSN. Older and recovered data produce asynchronous reports
+without failing the current borrower. A returned borrow cannot receive another
+producer-side exception, even if no new borrower has arrived yet.
+
+Begin, return, rejection installation, and open-tail sealing synchronize on the
+same state object. `failedGeneration` and `stopFsn` remain volatile observations
+for the producer and I/O loop. An open rejected range is sealed before returning
+its producer ownership. Successful normal pool return flushes a commit boundary;
+an unclosed transactional return without a covering rejection is refused.
+The check tolerates ACK/trim racing the frame lookup.
+
+Historical transaction ends are recoverable from queued commit flags. For an
+old live rejection, the scan stops before the active borrow, or at the most
+recent return when the slot is idle. Recovered ranges use the engine's original
+recovery boundary. Consequently no scan borrows a new producer's transaction
+closer. This relies on normal returns closing transactions and exceptional open
+returns sealing their rejection before reuse; arbitrary unclosed returns are
+not a supported state transition.
+
+## Retirement and preservation
+
+The I/O loop performs the following sequence:
+
+1. Validate the NACK and identify its range. Ambiguous or out-of-range responses
+   never become local retirement targets.
+2. Latch the owning borrow's failure and reconnect. Replay independent lower
+   frames until they have their configured ACKs, stopping before the rejection.
+3. Wait for the range to be sealed. Preserve dictionary deltas from every
+   skipped frame, including unsent transaction successors, for later replay.
+4. When preservation is configured, copy the range into its deterministic staging
+   directory using existing segment, manifest, watermark, and dictionary formats.
+   Clear the final copied frame's defer flag so the copy can replay independently.
+5. Write diagnostic properties, sync the files and directory, rename the staging
+   directory, and sync its parent. Only then is preservation complete.
+6. Retain the final notification in the separate 256-entry schema queue. A full
+   queue pauses retirement; callback completion otherwise does not gate it.
+7. Advance the existing resolved watermark through the range, then resume from
+   the next FSN on a correctly reanchored connection.
+
+`RejectedMiniSlotArchive` writes copies and can read the one deterministic path
+for an exact live range. There is no directory scan, separate preserver wrapper,
+or slot-epoch lifecycle. The writer is used by one I/O thread. After rename
+succeeds, it retains the result until the parent-directory sync succeeds, so a
+transient barrier failure retries without creating repeated copies. A later
+retry or restart reuses a completed valid copy and removes the exact crashed
+staging tree. Earlier write failures retain source frames and retry with the send
+loop's existing bounded backoff.
+
+A damaged completed archive cannot block live queue startup. For an exact
+recovered orphan range, the internal lookup validates the properties, segment,
+manifest, watermark, and optional dictionary. A missing, malformed, unreadable,
+or mismatched copy produces no report and the live orphan still retires. For
+manual replay, always use a working copy because queue cleanup can remove drained
+files and existing corruption handling can quarantine damaged working data.
+
+## Crash and shutdown boundaries
+
+- Before preservation completes: source frames remain queued. Closed groups can
+  replay and be rejected again. Recovery may discard an unclosed orphan tail
+  without another NACK; there is no universal callback or archive guarantee.
+- After archive publication but before durable retirement: the copy survives;
+  restart reuses it. If the exact range is a recovered orphan tail, its report is
+  retained before retirement; a closed range can replay and be rejected again.
+- After retirement but before callback delivery: a crash or bounded dispatcher
+  shutdown can lose the callback. With no still-live range, startup does not
+  reconstruct it from the archive.
+- A blocked filesystem call can exceed the close budget. Existing delegated
+  I/O-thread cleanup retains the source engine and lock until the worker exits.
+
+With disk buffering, preservation defaults to the slot's `rejected/` directory.
+An explicit DLQ base also supports memory queues. Memory queues without a
+configured destination, or preservation disabled explicitly, retire without
+copies. Completed archives have no automatic retention policy.
+
+`getAckedFsn`, `awaitAckedFsn`, and `drain` retain PR #94's resolved-progress
+semantics: locally retired data counts as progress, not server acceptance.
+The owning handle still throws for its rejected publication. Other error
+categories retain their existing policies, and `TERMINAL` remains selectable.
+
+## Validation and remaining limits
+
+Focused coverage includes delayed NACKs after many ordinary/transactional borrows,
+return-versus-NACK races, failed open-tail sealing, recovered group boundaries,
+successor dictionary continuity, source retention during write failure,
+parent-sync retry without recopying, notification saturation, blocked-copy close,
+offline archive replay, deterministic archive reuse and staging cleanup, recovered
+orphan notification reconstruction, and repeated startup with damaged archived
+copies.
+
+Validation on OpenJDK 25:
+
+- Before the final segment-token substitution, the full core suite ran 3,507
+  tests with zero failures or errors and seven skipped. A broader schema-focused
+  run at that stage passed 66 tests.
+- After the segment-token change, focused `MmapSegmentTest`,
+  `RejectedMiniSlotArchiveTest`, and `RejectedArchiveRecoveryTest` coverage passed
+  39 tests.
+- Examples compiled successfully. `git diff --check` passed.
+
+The current production diff removes 507 lines relative to `26e3c3a`. The archive
+class is 332 lines, down from 511, with the 73-line preserver wrapper and 116-line
+slot-epoch class deleted. The stronger return invariant and explicit
+archive behavior changes still require design review before adoption. No measured
+throughput improvement is claimed. The real server rollback behavior and minimum
+server version remain the PR's existing assumptions; local protocol tests use
+the repository's test server.
diff --git a/examples/POOLED_SF_POISON_DEMO.md b/examples/POOLED_SF_POISON_DEMO.md
new file mode 100644
index 000000000..8d5e51622
--- /dev/null
+++ b/examples/POOLED_SF_POISON_DEMO.md
@@ -0,0 +1,49 @@
+# Pooled store-and-forward schema-rejection demo
+
+This demo shows the default `REJECT_AND_CONTINUE` behavior for a one-slot
+pooled WebSocket sender. A borrow that publishes a schema-mismatched row fails,
+but returning that borrow releases the slot. A later borrow from the same pool
+then publishes a valid row successfully.
+
+Before retiring the rejected range, a disk-backed sender preserves its raw QWP
+frames and dictionary state in a completed archive. The owning borrow receives
+a synchronous `LineSenderServerException` identifying the rejected FSN range.
+The configured asynchronous error handler receives a `SenderError` containing
+the completed archive path. The demo checks both signals and verifies that the
+valid row reaches QuestDB.
+
+It uses a real QuestDB server with QWP available at `localhost:9000`. Start an
+ephemeral test server:
+
+```bash
+docker run --rm -d --name qdb-java-sfa-poison-demo \
+  -p 9000:9000 questdb/questdb:nightly
+```
+
+The demo drops and recreates only the table `java_sfa_poison_demo`; do not point
+it at a production database.
+
+Build the current client and choose a new temporary directory. A source build
+needs CMake, NASM, a C/C++ compiler, and the checked-out zstd submodule:
+
+```bash
+git submodule update --init --recursive
+cmake -DCMAKE_BUILD_TYPE=Release -B core/cmake-build-release -S core
+cmake --build core/cmake-build-release --config Release
+mvn -pl core -Dmaven.test.skip=true install
+mvn -f examples/pom.xml -DskipTests compile
+export QDB_POISON_DEMO_SF_DIR="$(mktemp -d)"
+```
+
+Run the demo once with that empty directory:
+
+```bash
+mvn -f examples/pom.xml \
+  org.codehaus.mojo:exec-maven-plugin:3.5.0:java \
+  -Dexec.mainClass=com.example.sender.WsPooledSchemaPoisonDemo \
+  -Dexec.args="${QDB_POISON_DEMO_SF_DIR}"
+```
+
+The output names the failed borrow's FSN range, the preserved-copy directory,
+and the successful valid row. Pass `host:port` as the final argument to use a
+server other than `localhost:9000`.
diff --git a/examples/src/main/java/com/example/sender/WsPooledSchemaPoisonDemo.java b/examples/src/main/java/com/example/sender/WsPooledSchemaPoisonDemo.java
new file mode 100644
index 000000000..4bc2dd148
--- /dev/null
+++ b/examples/src/main/java/com/example/sender/WsPooledSchemaPoisonDemo.java
@@ -0,0 +1,256 @@
+package com.example.sender;
+
+import io.questdb.client.LineSenderServerException;
+import io.questdb.client.QuestDB;
+import io.questdb.client.Sender;
+import io.questdb.client.SenderError;
+import io.questdb.client.SenderErrorHandler;
+import io.questdb.client.cutlass.qwp.client.QwpColumnBatch;
+import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler;
+import io.questdb.client.cutlass.qwp.client.QwpQueryClient;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Shows that a schema rejection fails one pooled-sender borrow without
+ * poisoning the underlying store-and-forward slot.
+ *
+ * See {@code examples/POOLED_SF_POISON_DEMO.md} for setup and invocation.
+ */
+public final class WsPooledSchemaPoisonDemo {
+
+    private static final String SENDER_ID = "java-schema-poison-demo";
+    private static final String TABLE = "java_sfa_poison_demo";
+    private static final long WAIT_MILLIS = 10_000;
+
+    private WsPooledSchemaPoisonDemo() {
+    }
+
+    public static void main(String[] args) throws Exception {
+        if (args.length < 1 || args.length > 2) {
+            printUsageAndExit();
+        }
+
+        Path sfDir = Paths.get(args[0]).toAbsolutePath().normalize();
+        String address = args.length == 2 ? args[1] : "localhost:9000";
+        rejectConfigSeparators(sfDir, address);
+
+        Path slotDir = sfDir.resolve(SENDER_ID + "-0");
+        if (Files.exists(slotDir)) {
+            throw new IllegalStateException(
+                    "The demo slot already exists: " + slotDir + ". Use a new empty directory.");
+        }
+
+        resetTable(address);
+        runDemo(address, sfDir);
+    }
+
+    private static void runDemo(String address, Path sfDir) throws Exception {
+        CountDownLatch reportReady = new CountDownLatch(1);
+        AtomicReference report = new AtomicReference<>();
+
+        try (QuestDB db = createOneSlotPool(address, sfDir, error -> {
+            if (error.getCategory() == SenderError.Category.SCHEMA_MISMATCH) {
+                report.compareAndSet(null, error);
+                reportReady.countDown();
+            }
+        })) {
+            System.out.println("First borrow: sending a STRING into the LONG column.");
+            LineSenderServerException rejection = sendRejectedRow(db);
+            requireSchemaMismatch(rejection);
+            printRejection("Owning borrow failed", rejection.getServerError());
+
+            if (!reportReady.await(WAIT_MILLIS, TimeUnit.MILLISECONDS)) {
+                throw new IllegalStateException("Timed out waiting for the schema-rejection report");
+            }
+            SenderError preserved = report.get();
+            if (preserved == null || preserved.getRejectedPath() == null
+                    || preserved.getAppliedPolicy() != SenderError.Policy.REJECT_AND_CONTINUE
+                    || preserved.getFromFsn() != rejection.getServerError().getFromFsn()
+                    || preserved.getToFsn() != rejection.getServerError().getToFsn()
+                    || !Files.isDirectory(Paths.get(preserved.getRejectedPath()))) {
+                throw new IllegalStateException("The rejected range was not preserved", rejection);
+            }
+            printRejection("Asynchronous preserved-copy report", preserved);
+            System.out.println("Rejected bytes were preserved at " + preserved.getRejectedPath());
+
+            System.out.println("Second borrow: sending a valid LONG row through the same one-slot pool.");
+            try (Sender healthy = db.borrowSender()) {
+                healthy.table(TABLE)
+                        .longColumn("value", 42)
+                        .symbol("marker", "good-after-rejection")
+                        .atNow();
+                long fsn = healthy.flushAndGetSequence();
+                if (!healthy.awaitAckedFsn(fsn, WAIT_MILLIS)) {
+                    throw new IllegalStateException("Timed out waiting for the valid row [fsn=" + fsn + ']');
+                }
+            }
+        }
+
+        long delivered = awaitGoodRows(address);
+        if (delivered != 1) {
+            throw new IllegalStateException("Expected one valid row [count=" + delivered + ']');
+        }
+        System.out.println("SUCCESS: returning the failed borrow kept the slot usable; the valid row was delivered.");
+    }
+
+    private static QuestDB createOneSlotPool(
+            String address,
+            Path sfDir,
+            SenderErrorHandler errorHandler
+    ) {
+        String config = "ws::addr=" + address + ';'
+                + "sf_dir=" + sfDir + ';'
+                + "sender_id=" + SENDER_ID + ';'
+                + "sf_durability=periodic;"
+                + "sf_sync_interval_millis=1;"
+                + "close_flush_timeout_millis=0;";
+
+        return QuestDB.builder()
+                .fromConfig(config)
+                .senderPoolSize(1)
+                .queryPoolMin(0)
+                .queryPoolMax(1)
+                .acquireTimeoutMillis(3_000)
+                .errorHandler(errorHandler)
+                .build();
+    }
+
+    private static long awaitGoodRows(String address) throws InterruptedException {
+        long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(WAIT_MILLIS);
+        long count;
+        do {
+            count = countGoodRows(address);
+            if (count > 0) {
+                return count;
+            }
+            Thread.sleep(50);
+        } while (System.nanoTime() < deadline);
+        return count;
+    }
+
+    private static LineSenderServerException sendRejectedRow(QuestDB db) {
+        Sender sender = db.borrowSender();
+        LineSenderServerException rejection = null;
+        try {
+            sender.table(TABLE)
+                    .stringColumn("value", "not-a-long")
+                    .symbol("marker", "bad")
+                    .atNow();
+            long fsn = sender.flushAndGetSequence();
+            if (!sender.awaitAckedFsn(fsn, WAIT_MILLIS)) {
+                throw new IllegalStateException("Timed out waiting for schema rejection [fsn=" + fsn + ']');
+            }
+            throw new IllegalStateException("The bad row was unexpectedly accepted");
+        } catch (LineSenderServerException expected) {
+            rejection = expected;
+        } finally {
+            try {
+                sender.close();
+            } catch (LineSenderServerException closeRejection) {
+                if (rejection == null) {
+                    rejection = closeRejection;
+                }
+            }
+        }
+        return rejection;
+    }
+
+    private static void resetTable(String address) {
+        execute(address, "DROP TABLE IF EXISTS " + TABLE, new NoRowsHandler());
+        execute(address,
+                "CREATE TABLE " + TABLE
+                        + " (value LONG, marker SYMBOL, ts TIMESTAMP)"
+                        + " TIMESTAMP(ts) PARTITION BY DAY WAL",
+                new NoRowsHandler());
+    }
+
+    private static long countGoodRows(String address) {
+        final long[] count = {Long.MIN_VALUE};
+        execute(address,
+                "SELECT count() FROM " + TABLE + " WHERE marker = 'good-after-rejection'",
+                new QwpColumnBatchHandler() {
+                    @Override
+                    public void onBatch(QwpColumnBatch batch) {
+                        if (batch.getRowCount() > 0) {
+                            count[0] = batch.getLongValue(0, 0);
+                        }
+                    }
+
+                    @Override
+                    public void onEnd(long totalRows) {
+                    }
+
+                    @Override
+                    public void onError(byte status, String message) {
+                        throw new IllegalStateException(
+                                String.format("Verification query failed [status=0x%02X, message=%s]",
+                                        status & 0xFF,
+                                        message));
+                    }
+                });
+        if (count[0] == Long.MIN_VALUE) {
+            throw new IllegalStateException("The verification query returned no count");
+        }
+        return count[0];
+    }
+
+    private static void execute(String address, String sql, QwpColumnBatchHandler handler) {
+        try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=" + address + ';')) {
+            client.connect();
+            client.execute(sql, handler);
+        }
+    }
+
+    private static void requireSchemaMismatch(LineSenderServerException rejection) {
+        if (rejection == null
+                || rejection.getServerError().getCategory() != SenderError.Category.SCHEMA_MISMATCH
+                || rejection.getServerError().getAppliedPolicy() != SenderError.Policy.REJECT_AND_CONTINUE) {
+            throw new IllegalStateException("The first borrow did not fail with REJECT_AND_CONTINUE", rejection);
+        }
+    }
+
+    private static void printRejection(String prefix, SenderError error) {
+        System.out.printf(
+                "%s: category=%s policy=%s fsn=[%d..%d] message=%s%n",
+                prefix,
+                error.getCategory(),
+                error.getAppliedPolicy(),
+                error.getFromFsn(),
+                error.getToFsn(),
+                error.getServerMessage());
+    }
+
+    private static void rejectConfigSeparators(Path sfDir, String address) {
+        if (sfDir.toString().indexOf(';') >= 0 || address.indexOf(';') >= 0) {
+            throw new IllegalArgumentException("The address and sf-dir must not contain ';'");
+        }
+    }
+
+    private static void printUsageAndExit() {
+        System.err.println("Usage: WsPooledSchemaPoisonDemo  [host:port]");
+        System.exit(2);
+    }
+
+    private static final class NoRowsHandler implements QwpColumnBatchHandler {
+        @Override
+        public void onBatch(QwpColumnBatch batch) {
+            throw new IllegalStateException("DDL unexpectedly returned rows");
+        }
+
+        @Override
+        public void onEnd(long totalRows) {
+        }
+
+        @Override
+        public void onError(byte status, String message) {
+            throw new IllegalStateException(
+                    String.format("DDL failed [status=0x%02X, message=%s]", status & 0xFF, message));
+        }
+    }
+}