Add pluggable page-level AEAD encryption support by HTHou · Pull Request #866 · apache/tsfile

HTHou · 2026-07-15T10:22:24Z

Summary

add an optional file encryption header carrying provider, profile, key, wrapped data key, and file identity metadata
introduce a provider-neutral AEAD SPI and registry without embedding a concrete cryptographic implementation
encrypt compressed page bodies for non-aligned, aligned, and table-model write paths, and decrypt them on the corresponding read paths
preserve existing unencrypted TsFile behavior and propagate encryption metadata through append, recovery, lazy loading, and sketch tooling
add compatibility, round-trip, tamper-detection, append, and recovery tests

Motivation

TsFile currently has legacy encryption interfaces but no self-describing file-level context for pluggable page-level authenticated encryption. This change establishes the format and I/O integration points while leaving algorithm and key-management implementations to external providers.

Compatibility

Existing unencrypted TsFiles keep their current layout and remain readable.
Encrypted files require a registered provider matching the identifiers stored in the encryption header.
Page metadata remains readable while compressed page bodies are protected with AEAD.
The encrypted-file format and public API are still under discussion and are not intended as a compatibility commitment in this draft.

Validation

./mvnw spotless:check -P with-java -pl java/tsfile -am
./mvnw test -P with-java -pl java/tsfile -am -Dtest='PageCryptoContextTest,FileEncryptionHeaderTest,TDEPageAeadTsFileTest,UnClosedTsFileReaderTest,ForceAppendTsFileWriterTest,RestorableTsFileIOWriterTest,TimePageWriterTest' -Dsurefire.failIfNoSpecifiedTests=false
30 targeted tests passed with no failures or errors

Draft discussion points

finalize the encryption-header schema and format-version semantics
bind page authentication data to an unambiguous chunk/page identity
refine page-index ownership and the separation between legacy encryption parameters and file AEAD context
confirm append/recovery behavior and backward-compatibility expectations before stabilizing the format

codecov-commenter · 2026-07-15T10:44:40Z

Codecov Report

❌ Patch coverage is 82.50319% with 274 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.91%. Comparing base (d33e640) to head (ecb0b6b).
⚠️ Report is 9 commits behind head on develop.

Files with missing lines	Patch %	Lines
...pache/tsfile/file/header/FileEncryptionHeader.java	78.78%	21 Missing ⚠️
...e/read/reader/block/SingleDeviceTsBlockReader.java	74.07%	21 Missing ⚠️
...org/apache/tsfile/write/writer/TsFileIOWriter.java	81.13%	20 Missing ⚠️
...sfile/write/v4/AbstractTableModelTsFileWriter.java	40.00%	18 Missing ⚠️
...che/tsfile/encrypt/EncryptionProviderRegistry.java	67.30%	17 Missing ⚠️
...a/org/apache/tsfile/read/TsFileSequenceReader.java	86.72%	15 Missing ⚠️
...a/org/apache/tsfile/encrypt/PageCryptoContext.java	77.77%	14 Missing ⚠️
...g/apache/tsfile/read/reader/chunk/ChunkReader.java	68.88%	14 Missing ⚠️
...ava/org/apache/tsfile/file/header/ChunkHeader.java	89.16%	13 Missing ⚠️
.../read/query/timegenerator/TsFileTimeGenerator.java	54.16%	11 Missing ⚠️
... and 26 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #866      +/-   ##
===========================================
+ Coverage    64.53%   65.91%   +1.38%     
===========================================
  Files          760      767       +7     
  Lines        53314    54864    +1550     
  Branches      8453     8722     +269     
===========================================
+ Hits         34404    36163    +1759     
+ Misses       17258    17055     -203     
+ Partials      1652     1646       -6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…x/tsfile-tde-page-aead # Conflicts: # java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java # java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java

…age-aead

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

InputStream#readNBytes(...) is only available starting in Java 9. If this project targets Java 8… · New
getAssociatedData() always allocates a new byte[] copy. Since this is called on every page… · New
For page-AEAD encrypted files, decrypting requires both the correct pageIndex and a nonnegative… · New
registerProvider(...) trims provider ids before storing them, but unregisterProvider(...) does not.… · New

What changed in this PR

This PR introduces pluggable page-level AEAD encryption for TsFile by adding a self-describing file encryption header, a provider-neutral encryption SPI/registry, and integrating page body encrypt/decrypt across write and read paths while preserving unencrypted TsFile compatibility.

Changes:

Added FileEncryptionHeader (written after the TsFile version byte) and propagated encryption context through writers/readers, append, and recovery flows.
Introduced provider-neutral AEAD SPI (IEncryptProvider, EncryptionProviderRegistry) and page-associated-data binding (PageCryptoContext) with per-chunk ordinals and per-page indices.
Added/updated tests to validate round-trips, tamper detection, ordinal continuity across append/recovery, and reader behavior.

File	Description
java/tsfile/src/test/java/org/apache/tsfile/write/writer/TimePageWriterTest.java	Adds an AEAD encrypt/decrypt page round-trip test using the test provider.
java/tsfile/src/test/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriterTest.java	Adds recovery/ownership tests for encrypted headers and chunk ordinal continuation.
java/tsfile/src/test/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriterTest.java	Adds force-append coverage for encrypted files and chunk ordinal uniqueness.
java/tsfile/src/test/java/org/apache/tsfile/write/TDEPageAeadTsFileTest.java	New integration tests for encrypted read/write paths, tamper detection, and chunk reuse rejection.
java/tsfile/src/test/java/org/apache/tsfile/read/UnClosedTsFileReaderTest.java	Adds reading encrypted pages from file header in unclosed reader scenario.
java/tsfile/src/test/java/org/apache/tsfile/file/header/FileEncryptionHeaderTest.java	New round-trip and validation tests for the file encryption header format.
java/tsfile/src/test/java/org/apache/tsfile/encrypt/TestAeadEncryptionProvider.java	Adds a JDK-only AES-GCM test provider implementing the new SPI.
java/tsfile/src/test/java/org/apache/tsfile/encrypt/PageCryptoContextTest.java	Adds tests for AAD binding/tamper detection and ordinal allocation behavior.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java	Writes encryption header before data, threads encryption context through chunk headers, and blocks encrypted chunk copy.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriter.java	Loads encryption header on recovery, resumes ordinal allocation, and closes owned AEAD parameters.
java/tsfile/src/main/java/org/apache/tsfile/write/writer/ForceAppendTsFileWriter.java	Loads encryption header for append, resumes ordinal allocation, and closes owned AEAD parameters.
java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java	Propagates file-level AEAD context and ensures owned AEAD parameters are closed.
java/tsfile/src/main/java/org/apache/tsfile/write/page/ValuePageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/TimePageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/PageWriter.java	Uses centralized `PageBodyEncoder` and adds pageIndex/chunkOrdinal support.
java/tsfile/src/main/java/org/apache/tsfile/write/page/PageBodyEncoder.java	New helper to compress + encrypt page bodies (including AEAD AAD binding).
java/tsfile/src/main/java/org/apache/tsfile/write/page/EncodedPageBody.java	New lightweight holder for encoded page body bytes (data/offset/size).
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ValueChunkWriter.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/TimeChunkWriter.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/chunk/ChunkWriterImpl.java	Allocates per-chunk ordinals and passes pageIndex/chunkOrdinal to page writers + chunk headers.
java/tsfile/src/main/java/org/apache/tsfile/write/TsFileWriter.java	Propagates file-level AEAD context and closes owned AEAD parameters on close.
java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileSketchTool.java	Uses `reader.getDataStartOffset()` to handle optional encryption header.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/page/LazyLoadPageData.java	Adds AEAD decrypt path using pageIndex/chunkOrdinal and validates plaintext sizes.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/ChunkReader.java	Threads pageIndex/chunkOrdinal into page readers and adds AEAD page decrypt/uncompress overloads.
java/tsfile/src/main/java/org/apache/tsfile/read/reader/chunk/AbstractAlignedChunkReader.java	Threads pageIndex/chunkOrdinal through aligned chunk page reads.
java/tsfile/src/main/java/org/apache/tsfile/read/common/Chunk.java	Rejects merging AEAD-encrypted chunks and ensures ordinal allocation during rewrite.
java/tsfile/src/main/java/org/apache/tsfile/read/UnClosedTsFileReader.java	Uses file-level encrypt param directly for unclosed reader.
java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java	Parses optional encryption header, tracks data start offset, records chunk ordinals, and supports AEAD page decryption with AAD.
java/tsfile/src/main/java/org/apache/tsfile/file/header/FileEncryptionHeader.java	New persisted encryption header format (versioned, size-bounded, wrapped key only).
java/tsfile/src/main/java/org/apache/tsfile/file/header/ChunkHeader.java	Adds optional chunk ordinal field for AEAD binding and updates (de)serialization accordingly.
java/tsfile/src/main/java/org/apache/tsfile/file/MetaMarker.java	Adds `ENCRYPTION_HEADER` marker for file-level encryption header.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/PageCryptoContext.java	New canonical AEAD page context with stable associated data construction.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptor.java	Adds encryptor selection for AEAD params and `encryptPage` default method.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncryptProvider.java	New SPI interface for provider implementations.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IEncrypt.java	Extends `AutoCloseable` and adds `getPageBodyOverhead` contract for AEAD profiles.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/IDecryptor.java	Adds decryptor selection for AEAD params and `decryptPage` default method.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptionProviderRegistry.java	New registry (and service loader integration) for stable provider ids.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptUtils.java	Adds AEAD-aware `getEncryptParameter` and `getEncrypt` overload.
java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptParameter.java	Extends parameters for AEAD (provider/profile ids, wrapped key, file id, chunk ordinal allocation, destroy/close).
java/common/src/main/resources/org/apache/tsfile/i18n/messages_zh.properties	Adds i18n strings for new encryption header/provider/AAD errors.
java/common/src/main/resources/org/apache/tsfile/i18n/messages.properties	Adds i18n strings for new encryption header/provider/AAD errors.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Unresolved context-validation, fail-open SPI, and missing last-reader decryption paths can produce insecure or unreadable files.

4 open findings

4 resolved since last review

InputStream#readNBytes(...) is only available starting in Java 9. If this project targets Java 8…
For page-AEAD encrypted files, decrypting requires both the correct pageIndex and a nonnegative…
getAssociatedData() always allocates a new byte[] copy. Since this is called on every page…
registerProvider(...) trims provider ids before storing them, but unregisterProvider(...) does not.…

Previously missed (1)

In code that hasn't changed since last review

Decrypt aligned BLOB/OBJECT pages before last-point decoding

java/tsfile/src/main/java/org/apache/tsfile/write/page/PageBodyEncoder.java:69

Encrypting every page body leaves TsFileLastReader.readAlignedLastPoint incompatible with encrypted aligned BLOB/OBJECT columns: that path slices the stored body and passes it directly to IUnCompressor/ValuePageReader without decryption (TsFileLastReader.java:179-216). It will feed ciphertext to the decoder (or decompressor), so last-point reads fail for these supported types. Route that path through the AEAD-aware page deserialization while tracking the selected page index and chunk ordinal.

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Headerless recovery can create unreadable mixed-format files, and encryption-context validation and lifetime issues remain unresolved.

2 open findings

4 resolved since last review

Previously missed (2)

In code that hasn't changed since last review

Normalize provider IDs during lookup

java/tsfile/src/main/java/org/apache/tsfile/encrypt/EncryptionProviderRegistry.java:58

Provider IDs are trimmed when registering and unregistering, but lookup uses the untrimmed persisted value. A provider whose declared ID contains surrounding whitespace is therefore registered successfully under the trimmed key yet can never be resolved from a parameter/header carrying that declared ID. Apply the same normalization during lookup (or reject non-canonical IDs consistently).

Preserve encryption context for materialized chunks

java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java:2417

The same EncryptParameter instance is attached to every Chunk returned by readMemChunk() and to its lazy page readers. Destroying it here means an already materialized in-memory chunk becomes unusable as soon as the sequence reader is closed; subsequent page loading fails with parameter_destroyed. Give returned chunks an independently owned context or otherwise coordinate the context lifetime instead of invalidating it with the file handle.

🧠 Review effort: Balanced

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

JackieTien97

Thanks for working on this. The overall direction looks good: a self-describing file header, a provider SPI, and per-page AEAD with the AAD bound to fileCryptoId / chunk ordinal / page index. The fail-closed checks on chunk copy/merge, the immutable file context and the ordinal validation are nice, and the lifecycle tests cover a lot of edge paths.

Validation: I ran the 7 test classes listed in the description plus the two new lifecycle test classes (72 tests), and the full java/tsfile suite (1038 tests, 0 failures, 10 skipped), on b58e5c37. The only change since then, fc88fb87, adds a pageWriter null check in reset() and doesn't affect the comments below. I reproduced each issue with a small probe test; the repro steps are in the inline comments.

Summary of the inline comments:

Blocking: page-level raw copy (writePageHeaderAndDataIntoBuff(ByteBuffer, PageHeader)) is not fail-closed for page AEAD and silently writes undecryptable files. This is the IoTDB fast-compaction path.
The AAD binds pages to chunk ordinals, but nothing binds chunks to their series: swapping whole chunks between s1 and s2 goes undetected.
TsFileReader now strongly retains every QueryDataSet, which changes the contract for existing users of unencrypted files.
With the reference-counted Chunk / ChunkReader ownership, cleanup only happens if every caller closes every chunk.
Recovery and unsealed reads: headerless file + AEAD parameter, a torn header surfacing as an unchecked exception, and the getEncryptParam() fallback.
Parameter ownership, SPI hardening (reachability of the legacy encrypt/decrypt, per-file key generation, nonce guidance), and a few nits.

JackieTien97 · 2026-10-12T08:12:35Z

+      }
+      return;
+    }
+    throw new IOException(Messages.get("error.write.encrypted_chunk_copy_unsupported"));


Page-level raw copy isn't guarded like chunk-level copy, so it silently writes unreadable files.

Chunk copy is rejected here, but the page-level raw-copy APIs still accept encrypted page bodies when the target writer uses page AEAD:

ChunkWriterImpl#writePageHeaderAndDataIntoBuff(ByteBuffer, PageHeader)

TimeChunkWriter#writePageHeaderAndDataIntoBuff(ByteBuffer, PageHeader)

ValueChunkWriter#writePageHeaderAndDataIntoBuff(ByteBuffer, PageHeader)

AlignedChunkWriterImpl#writePageHeaderAndDataIntoTimeBuff / #writePageHeaderAndDataIntoValueBuff

The AAD binds the copied ciphertext to the source file's fileCryptoId, chunk ordinal and page index, so it can never authenticate inside the new chunk.

Repro:

Write file A with page AEAD.

Read a raw page body with readMemChunk + ChunkReader.readCompressedPageData.

Feed it to new ChunkWriterImpl(schema, targetParam).writePageHeaderAndDataIntoBuff(raw, pageHeader) for file B, then call writeToFileWriter and endFile(). All of these succeed.

Query B: it fails with AEADBadTagException: Tag mismatch.

This is the path IoTDB's fast compaction uses (AbstractCompactionWriter#flushNonAlignedPageToChunkWriter / #flushAlignedPageToChunkWriter, InstantPageLoader). Compaction deletes the source files afterwards, so this ends in data loss.

Suggestion: at minimum, reject raw page copy when the target writer has a chunk ordinal (fail closed, like writeChunk). Better: offer a re-encrypting copy path that decrypts with the source context and re-encrypts the still-compressed plaintext with the target chunk ordinal and page index. No decompress/decode is needed, so fast compaction stays cheap for encrypted files.

Fixed in d391241. The scalar, time and value raw-page copy entry points now reject page-AEAD targets before changing the page buffer; aligned copy delegates to those guarded paths. Regression tests exercise the rejection. A compressed-plaintext re-encryption API is not introduced in this patch; the database integration must use a safe rewrite path instead of copying authenticated ciphertext across contexts.

JackieTien97 · 2026-10-12T08:12:35Z

+    }
+  }
+
+  private static byte[] buildAssociatedData(


The AAD binds a page to (fileCryptoId, chunkOrdinal, pageIndex, sizes), but nothing binds a chunk to its series.

measurementId, dataType, encoding and compressionType (and the device) are not in the associated data. The ordinal lives in the unauthenticated chunk header, so it moves together with the chunk.

Repro: write d1.s1 = 11 and d1.s2 = 22 (PLAIN/UNCOMPRESSED, one point each, so both chunks are 79 bytes). Swap the two chunks' whole byte ranges (header + pages) in the file. Querying d1.s1 returns 22, with no authentication error.

Truncating trailing pages or chunks, or editing the plaintext statistics so pages get skipped, also goes undetected. testSwappingEqualSizedPagesBetweenChunksFailsAuthentication only covers swapping pages between chunks.

Suggestion: put the fields that are fixed when a page is encrypted (dataType, encoding, compressionType, time/value column mask) into the AAD. measurementId and the device need more thought, because names can be remapped at flush time (measurementNameRemapper) and chunk writers don't know the device. Either way, please document that page AEAD does not provide chunk-level integrity (deletion, truncation, reordering).

Addressed the whole-chunk swap repro in 2f2a452 with a separate position proof in encrypted Chunk Headers. Immediately before writing the header, the provider authenticates a domain/version, fileCryptoId, the actual absolute chunk-marker offset, and chunkOrdinal. Sequential and positional readers verify the proof before using the ordinal or page body. The proof is 28 opaque bytes; the concrete TimechoDB provider uses a fresh 12-byte IV plus a 16-byte GCM tag over empty plaintext. Page AAD and page-buffer encryption timing remain unchanged. ChunkPositionAuthenticationTest covers equal-sized whole-chunk swaps, both read paths, query, empty chunks and recovery. Scope clarification: this protects relocation of an existing chunk, not general metadata integrity. Measurement/device mapping, codec fields, statistics and indexes remain unauthenticated; pruning based on altered statistics, deletion/truncation and index remapping are not claimed to be detected by this design.

JackieTien97 · 2026-10-12T08:12:35Z

+        tsfileExecutor.execute(queryExpression, partitionStartOffset, partitionEndOffset));
+  }
+
+  private QueryDataSet retainQuery(QueryDataSet dataSet) {


This silently changes the contract of TsFileReader#query for existing users, including users of unencrypted files.

Before this PR, QueryDataSet had no close() and wasn't AutoCloseable. The only (and correct) way to use it was to drop the reference and let GC reclaim it. Now every returned dataset is held here until it is fully iterated or explicitly closed. Existing code that stops early (reads only the first rows, breaks, or throws while iterating) now piles up datasets for the reader's lifetime, each pinning its current chunk buffer. Nothing warns at compile time.

Repro on an unencrypted file: 1000 × (query(...), next() once, drop the reference) leaves queryDataSets.size() == 1000.

The tracking is only needed to release page-AEAD references, but it applies to every file. Suggestion: only track datasets when fileReader.hasFileEncryptionHeader() is true. That keeps the unencrypted path fully backward compatible, and requiring close() for the new encrypted feature is a reasonable new contract. If the intent is to require close() for all files, please document it as a breaking change on query() and in the release notes.

Minor: this HashSet isn't thread-safe if one TsFileReader is shared by threads issuing queries concurrently.

Fixed in d391241: only datasets for files with a page-AEAD encryption header are retained for cleanup. Plaintext queries are not added to the tracking set, preserving their existing lifetime behavior. The tracking set is synchronized, including iteration during reader close. Added a plaintext early-abandonment regression test.

JackieTien97 · 2026-10-12T08:12:35Z

+ * A materialized chunk owning a page-AEAD reference that survives file reader closure. Close it
+ * after use, or transfer it to a chunk reader that releases the reference when closed.
+ */
+public class Chunk implements AutoCloseable {


With reference counting, provider cleanup only happens if every caller closes every Chunk, and existing callers don't.

Every Chunk built with a page-AEAD parameter now retain()s it (copying the plaintext key if the parameter carries one) and must be closed. Existing callers of readMemChunk don't close chunks. For example, IoTDB's ChunkCache caches the Chunk returned by readMemChunk and doesn't close it on eviction. With the cache disabled, it re-wraps the loaded chunk in constructChunk and never closes the original. In practice the count rarely reaches zero, so "the provider is closed after the last reference" doesn't hold. Meanwhile every downstream call site has to learn a new close contract.

Suggestion: decouple the provider context's lifetime from object lifetimes. For example, keep a bounded process-wide cache of IEncrypt keyed by (providerId, profileId, keyId, keyVersion, wrappedDataKey) that closes entries on eviction, and let Chunk hold only the immutable file encryption descriptor. Then Chunk doesn't need to be closeable, and unencrypted paths stay untouched.

We are keeping explicit ownership for the new page-AEAD feature rather than adding a process-wide crypto cache to TsFile. Chunk retains the shared file-level provider context, and ChunkReader owns/releases the supplied chunk; retained chunks do not independently unwrap a key on each read. The plaintext path does not acquire crypto references. The consuming TimechoDB branch has been adapted: ChunkCache closes entries on eviction, closes temporary loaded chunks after creating retained query views, and handles eviction/retain races; query readers release their owned views. This downstream adaptation is required before enabling page AEAD. We do not claim that arbitrary unmodified downstream callers will release encrypted resources automatically.

JackieTien97 · 2026-10-12T08:12:35Z

+  }
+
+  @Override
+  public void close() {


Closing a ChunkReader now destroys the encryption reference of the caller's Chunk.

Before this PR, close() here was a no-op. Now it closes the supplied chunk.

Repro: Chunk c = reader.readMemChunk(meta), read it through new ChunkReader(c), then close(). Afterwards c.getEncryptParam().isDestroyed() is true. Re-wrapping the chunk's data with its parameter, new Chunk(c.getHeader(), data, c.getEncryptParam()) (the ChunkCache#constructChunk pattern), throws Encryption parameters have been destroyed. That worked before this PR and still works for unencrypted chunks. It's a behavior change worth documenting, or avoiding with the approach suggested on Chunk.

Also, ChunkReader and AbstractAlignedChunkReader now own resources but aren't AutoCloseable, so try-with-resources can't be used.

The ownership transfer for page-AEAD chunks is intentional and documented on ChunkReader: closing it releases the supplied chunk's reference. A cache must pass an independently retained Chunk view, not its own cache entry. The TimechoDB integration now follows that rule and closes cache-owned references on eviction. Plaintext chunks do not have this crypto lifetime requirement. In d391241, IChunkReader also extends AutoCloseable, so scalar and aligned readers can be used with try-with-resources.

JackieTien97 · 2026-10-12T08:12:35Z

+        numOfPages,
+        mask,
+        chunkOrdinal,
+        null);


nit: this public overload passes a null chunk parameter, so in a page-AEAD file it always fails the sharesPageAeadFileContext check, and the ordinal only matters for page AEAD. The 5-arg LazyLoadPageData(..., int pageIndex) constructor has the same problem: it fixes chunkOrdinal to -1. Neither can be used successfully with page AEAD; consider removing them.

Removed both unusable PR-added overloads in d391241. The original legacy overloads remain, and the full AEAD-aware overloads require the necessary chunk context/ordinal.

JackieTien97 · 2026-10-12T08:12:35Z

+        return;
+      }
+      marker.flip();
+      if (marker.get() != MetaMarker.ENCRYPTION_HEADER) {


nit: every reader open, including for unencrypted files, now does an extra 1-byte positional read at offset 7. For remote or object-storage TsFileInputs that's an extra round trip. It could be merged with the version-byte read in loadFileVersion.

Acknowledged as an I/O optimization opportunity. This patch still performs a separate marker read when checking the optional encryption header; it does not yet combine that read with the version read. The metadata-offset constructors retain their existing no-checked-exception signature and load the optional encryption header lazily. I am not marking the extra-read nit as fixed.

JackieTien97 · 2026-10-12T08:12:35Z

+    }
+    FileEncryptionHeader.serialize(encryptParameter, out.wrapAsStream());
+    // A recoverable encrypted file must persist its wrapped data key before any encrypted page.
+    out.force();


question: this adds an fsync for every new encrypted file. Is it needed, given that unflushed data is covered by the caller's WAL? The magic/version of a plaintext file isn't forced either.

Removed the per-encrypted-file force() in d391241. The header is flushed for normal visibility, but TsFile does not impose an additional fsync policy. The database integration is responsible for ordering file durability against its WAL; the TimechoDB flush path now forces encrypted TsFile data before memtable release/WAL completion callbacks.

JackieTien97 · 2026-10-12T08:12:35Z

+  }
+
+  private static void loadServices() {
+    if (SERVICES_LOADED.compareAndSet(false, true)) {


nit: SERVICES_LOADED is set before iterating. If one provider throws (a duplicate id with a different class, a ServiceConfigurationError, ...), the remaining providers are never registered and loading is never retried. Consider registering the rest and reporting the failures instead.

Fixed in d391241. Service discovery is synchronized through registration; the loaded flag is published only after successful completion. A per-provider failure does not stop registration of subsequent providers, and failed discovery remains retryable. Added regression tests for continued registration and retry after failure.

JackieTien97 · 2026-10-12T08:12:35Z

    return measurementID;
  }

+  public long getChunkOrdinal() {


nit: toString() doesn't include chunkOrdinal, which would help when debugging encrypted files (e.g. in sketch output).

Acknowledged. chunkOrdinal is not yet included in ChunkHeader.toString() in this update, so this debug-output nit remains open. Authentication proof bytes are intentionally not printed.

JackieTien97

Please address the two inline findings before merging: page authentication failures can be silently converted into incomplete table-query results, and concurrent first use can fail while service-provider registration is still in progress. Both issues were reproduced locally on b58e5c3; the affected code paths are unchanged in fc88fb8.

JackieTien97 · 2026-10-12T08:17:38Z

+      unCompressor.uncompress(
+          decryptedPageData, 0, decryptedPageData.length, uncompressedPageData, 0);
    } catch (Exception e) {
      throw new IOException(


[P1] Propagate page authentication failures to the query caller

This AEAD overload wraps authentication failures in IOException, which the table-query stack treats as a recoverable read failure: DeviceOrderedTsBlockReader.hasNext() catches it during reader construction, logs it, and skips the device; SingleDeviceTsBlockReader.hasNext() similarly returns false for an IOException during iteration. Consequently, a tampered encrypted file can produce an apparently successful empty or partial query result.

I reproduced this by writing a three-row encrypted table, verifying that it returned all three rows, and flipping one byte in the time-page body. The log contained AEADBadTagException: Tag mismatch, but ResultSet.next() returned false without throwing, and the query returned zero rows. Please preserve and propagate authentication failures through both table-reader paths instead of treating them as end-of-data, and add a regression test at the public table-query API.

Fixed in d391241. Both DeviceOrderedTsBlockReader and SingleDeviceTsBlockReader propagate encryption failures (including causes wrapped by IOException) instead of treating them as a skipped device/end-of-data. Provider runtime failures are normalized at the page-AEAD boundary without losing the cause. Public table-query regression tests cover tampered time-page construction and a later value-page failure: both must throw rather than return an apparently successful empty/partial result.

JackieTien97 · 2026-10-12T08:17:39Z

+    if (SERVICES_LOADED.compareAndSet(false, true)) {
+      for (IEncryptProvider provider : ServiceLoader.load(IEncryptProvider.class)) {
+        registerProvider(provider);
+      }


[P2] Wait for service discovery to finish before publishing the loaded state

SERVICES_LOADED becomes true before the providers have been registered. If a second thread creates an encryption context while the first thread is still constructing a service-loaded provider, it skips loadServices() and looks up an incomplete PROVIDERS map, incorrectly throwing provider_not_found for a valid installed provider.

I reproduced this deterministically by pausing a provider constructor with a latch and creating a second context concurrently: the second call failed with "Encryption provider is not registered", while the first completed successfully after registration resumed. Please make concurrent callers wait for initialization to complete and publish the loaded state only after registration succeeds.

Fixed in d391241. Concurrent callers wait for service discovery/registration to complete, and SERVICES_LOADED is set only after successful registration. EncryptionProviderRegistryTest.testConcurrentFirstUseWaitsForServiceRegistration uses a latched provider constructor to reproduce the race deterministically and verifies both callers succeed.

jt2594838 · 2026-10-14T07:54:52Z

+  /**
+   * Returns a fixed-size opaque proof authenticating a chunk's physical position. The proof is
+   * exactly {@link #getChunkAuthenticationSize()} bytes. Providers using AEAD must use a fresh
+   * nonce under the file DEK, independently of page encryption calls.
+   */
+  default byte[] authenticateChunk(ChunkCryptoContext context) {
+    throw new EncryptException(Messages.get("error.encrypt.chunk_authentication_unsupported"));
+  }
+
+  /** Verifies the proof before a chunk identity is used; unsupported providers fail closed. */
+  default void verifyChunkAuthentication(byte[] authentication, ChunkCryptoContext context) {
+    throw new EncryptException(Messages.get("error.encrypt.chunk_authentication_unsupported"));
+  }


May provide the implementation name.

jt2594838 · 2026-10-14T08:00:28Z

+      if (marker != MetaMarker.ENCRYPTION_HEADER) {
+        switch ((byte) marker) {
+          case MetaMarker.CHUNK_GROUP_HEADER:
+          case MetaMarker.CHUNK_HEADER:
+          case MetaMarker.SEPARATOR:
+          case MetaMarker.VERSION:
+          case MetaMarker.OPERATION_INDEX_RANGE:
+          case MetaMarker.ONLY_ONE_PAGE_CHUNK_HEADER:
+          case MetaMarker.TIME_CHUNK_HEADER:
+          case MetaMarker.VALUE_CHUNK_HEADER:
+          case MetaMarker.ONLY_ONE_PAGE_TIME_CHUNK_HEADER:
+          case MetaMarker.ONLY_ONE_PAGE_VALUE_CHUNK_HEADER:
+            break;
+          default:
+            MetaMarker.handleUnexpectedMarker((byte) marker);
+        }


May use enhanced switch to report an error during compilation.

jt2594838 · 2026-10-14T08:00:35Z

      if (ioSizeRecorder != null) {
        ioSizeRecorder.accept(dataStartOffset - BASE_FILE_HEADER_SIZE);
      }
+    } catch (org.apache.tsfile.exception.encrypt.EncryptException e) {


mind the import

Add pluggable page-level AEAD encryption support

76754e0

JackieTien97 and others added 5 commits August 6, 2026 16:37

Bump version to 2.4.1-260806-SNAPSHOT

7d25839

Merge remote-tracking branch 'origin/snapshot/2.4.1-260806' into code…

dd20f4d

…x/tsfile-tde-page-aead # Conflicts: # java/tsfile/src/main/java/org/apache/tsfile/write/v4/AbstractTableModelTsFileWriter.java # java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java

Merge develop into TsFile TDE branch

f7b5789

Merge remote-tracking branch 'origin/develop' into codex/tsfile-tde-p…

b940382

…age-aead

Bind AEAD pages to file-unique chunk ordinals

dbd0609

HTHou marked this pull request as ready for review October 9, 2026 09:35

HTHou requested a balanced review from Copilot October 9, 2026 09:35

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Copilot stopped reviewing on behalf of HTHou due to an error October 9, 2026 10:22
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Address page AEAD review feedback

f2f58a7

HTHou requested a balanced review from Copilot October 10, 2026 02:39

Copilot started reviewing on behalf of HTHou October 10, 2026 02:40

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Fail closed for page AEAD and validate TsFile encryption context

0eb25b0

HTHou requested a balanced review from Copilot October 10, 2026 03:07

Copilot started reviewing on behalf of HTHou October 10, 2026 03:08

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Comment thread java/tsfile/src/main/java/org/apache/tsfile/write/writer/RestorableTsFileIOWriter.java Outdated

Comment thread java/tsfile/src/main/java/org/apache/tsfile/write/writer/TsFileIOWriter.java

HTHou added 3 commits October 10, 2026 16:36

Fix page AEAD append and reader lifecycle edge cases

517b06b

Manage shared page AEAD contexts across reader lifecycles

59203d2

Release page AEAD contexts in table block readers

b58e5c3

HTHou requested a balanced review from Copilot October 12, 2026 04:05

Copilot AI reviewed Oct 12, 2026

Fix encrypted chunk flush after clearing page writers

fc88fb8

JackieTien97 reviewed Oct 12, 2026

View reviewed changes

JackieTien97 requested changes Oct 12, 2026

View reviewed changes

HTHou added 2 commits October 13, 2026 10:10

Harden page AEAD recovery, ownership and provider contracts

d391241

Authenticate encrypted chunk positions before page reads

2f2a452

jt2594838 reviewed Oct 13, 2026

View reviewed changes

HTHou added 2 commits October 14, 2026 10:31

Address page AEAD review feedback

0f5413d

Support profile-defined chunk authentication lengths

ecb0b6b

jt2594838 approved these changes Oct 14, 2026

View reviewed changes

Conversation

HTHou commented Jul 15, 2026

Summary

Motivation

Compatibility

Validation

Draft discussion points

Uh oh!

codecov-commenter commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jul 15, 2026 •

edited

Loading