Add optional targetDocsPerChunk cap to VarByteChunkForwardIndexWriterV6

[xiangfu0]

Summary

VarByteChunkForwardIndexWriterV6 currently closes a chunk only when the next
entry would overflow the chunkSize-byte buffer. This PR adds an optional
targetDocsPerChunk parameter so a chunk can additionally be bounded by document
count, letting callers control chunk granularity independently of the byte budget.

• New 4-arg constructor VarByteChunkForwardIndexWriterV6(file, compressionType, chunkSize, targetDocsPerChunk). The existing 3-arg constructor delegates with
  targetDocsPerChunk = -1 (DISABLE_DOCS_PER_CHUNK).
• When targetDocsPerChunk > 0, a chunk is flushed once it holds that many docs,
  even if the byte buffer isn't full; otherwise behavior is unchanged.
• The buffer-overflow flush predicate is extracted into a protected
  shouldStartNewChunk(int) hook in VarByteChunkForwardIndexWriterV4 (mirroring
  the existing writeChunkHeader hook), so V6 adds the cap without duplicating
  putBytes().

Motivation

For raw string/bytes columns, the ZSTD compression ratio depends heavily on how
many repeated values fall within a single chunk (the dedup window). Being able to
bound a chunk by document count — not only by bytes — gives finer control over the
size/granularity tradeoff for repetitive columns.

Constructor examples

// Default (3-arg): byte-driven flush only — identical to the prior behavior.
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize);

// Cap each chunk at 1000 docs: flush at 1000 docs OR when the chunkSize buffer
// fills, whichever comes first.
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize, 1000);

// Explicitly disable the docs cap (equivalent to the 3-arg constructor).
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize,
    VarByteChunkForwardIndexWriterV6.DISABLE_DOCS_PER_CHUNK);

Configuring V6 with a chunk size (table config)

Select this writer and the chunk size per column via fieldConfigList:

{
  "fieldConfigList": [
    {
      "name": "myColumn",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "ZSTANDARD",
          "rawIndexWriterVersion": 6,
          "targetMaxChunkSize": "1MB",
          "targetDocsPerChunk": -1
        }
      }
    }
  ]
}

• rawIndexWriterVersion: 6 selects this writer.
• targetMaxChunkSize is the chunk byte budget. The effective chunk size is
  max(min(maxLength × targetDocsPerChunk, targetMaxChunkSize), 4KB), so
  targetDocsPerChunk: -1 makes the chunk size driven purely by targetMaxChunkSize
  (otherwise a narrow column is capped at maxLength × targetDocsPerChunk, e.g.
  ~32 KB for a 32-byte column at the default 1000).

Note: the table-config targetDocsPerChunk derives the chunk byte size; the new
constructor parameter targetDocsPerChunk (a hard per-chunk doc cap) is a
writer-level API and is not wired into table config in this PR.

Backward compatibility

• The default -1 reproduces the exact current behavior.
• The on-disk format and writer version (6) are unchanged; the target chunk size
  remains self-describing in the file header, so existing and new indexes stay
  mutually readable.

Testing

• New VarByteChunkV6Test#testTargetDocsPerChunkCapsChunk asserts each capped chunk
  holds exactly targetDocsPerChunk docs and that values round-trip.
• All inherited V4/V5/V6 read/write tests pass (the -1 default path).

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 64.77%. Comparing base (e8bb63f) to head (32584c5).

Files with missing lines	Patch %	Lines
.../writer/impl/VarByteChunkForwardIndexWriterV6.java	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #18772   +/-   ##
=========================================
  Coverage     64.76%   64.77%           
  Complexity     1322     1322           
=========================================
  Files          3393     3393           
  Lines        211079   211087    +8     
  Branches      33159    33161    +2     
=========================================
+ Hits         136707   136732   +25     
+ Misses        63341    63322   -19     
- Partials      11031    11033    +2     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	64.77% <88.88%> (+<0.01%)	⬆️
temurin	64.77% <88.88%> (+<0.01%)	⬆️
unittests	64.77% <88.88%> (+<0.01%)	⬆️
unittests1	56.97% <22.22%> (+<0.01%)	⬆️
unittests2	37.17% <88.88%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds an optional document-count cap for chunk flushing in VarByteChunkForwardIndexWriterV6, allowing callers to control chunk granularity by both byte budget (chunkSize) and a new targetDocsPerChunk parameter, while keeping the on-disk format/version unchanged.

Changes:

• Introduces a protected shouldStartNewChunk(int sizeRequired) hook (and a helper for current-chunk doc count) in V4 so subclasses can add extra chunk-flush predicates without duplicating putBytes().
• Extends V6 with a 4-arg constructor and a docs-per-chunk cap that triggers chunk flushes once the current chunk reaches the configured doc count.
• Adds a V6 unit test intended to validate docs-per-chunk behavior and round-trip correctness.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkForwardIndexWriterV4.java	Extracts the “start new chunk” predicate into an overridable hook and exposes current-chunk doc count for subclasses.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkForwardIndexWriterV6.java	Adds optional targetDocsPerChunk support via shouldStartNewChunk override and a new constructor overload.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/VarByteChunkV6Test.java	Adds a test for docs-per-chunk capping and value round-tripping.