Allow non-immutable ingestion transforms for offline tables

[xiangfu0]

Summary

Allow table-level ingestionConfig.transformConfigs on OFFLINE tables to use registered scalar functions from any volatility category. REALTIME table-level transforms continue to require IMMUTABLE functions.

Expression parsing and the existing transform validation remain unchanged. Schema-level transforms and postPartialUpsertTransformConfigs also retain their existing volatility checks.

Motivation

Some OFFLINE upsert inputs, such as daily partial Parquet drops, do not contain a physical timestamp or row-version column. This lets ingestion populate a declared comparison column with a transform such as now() without rewriting the input file first.

Usage

Declare the generated column in the schema:

{
  "dateTimeFieldSpecs": [
    {
      "name": "ingestionTime",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}

Use it as the OFFLINE upsert comparison column:

{
  "upsertConfig": {
    "mode": "FULL",
    "comparisonColumns": ["ingestionTime"]
  },
  "ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "ingestionTime",
        "transformFunction": "now()"
      }
    ]
  }
}

The remaining OFFLINE upsert requirements, including primary-key partitioning and strict replica-group routing, are unchanged.

Scope

This is a validation-policy change only. It does not add a new SPI, segment format, serializer, or execution path. Non-immutable functions retain their existing ingestion execution semantics.

Validation

• TableConfigUtilsTest
• SchemaUtilsTest
• PinotTableRestletResourceTest
• TableConfigsRestletResourceTest
• 44 focused tests passed on JDK 25
• Spotless, Checkstyle, and license checks passed for the affected modules

[Copilot]

Pull request overview

This pull request relaxes validation for table-level ingestion transforms so that OFFLINE tables may use registered scalar functions across all volatility categories, while keeping REALTIME ingestion transforms restricted to IMMUTABLE functions (with legacy/grandfathering behavior preserved for unchanged existing configs).

Changes:

• Skip ingestion-transform volatility validation for TableType.OFFLINE while retaining it for TableType.REALTIME.
• Update SPI documentation strings/comments to describe the updated volatility policy and its implications.
• Expand/adjust unit and REST-layer tests to cover: OFFLINE non-immutable transforms allowed, REALTIME rejection + grandfathering, and offline-upsert comparison-column transform usage.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/IngestionConfig.java	Updates the JSON property description for transformConfigs to document OFFLINE vs REALTIME volatility policy and instability risks.
pinot-spi/src/main/java/org/apache/pinot/spi/annotations/ScalarFunction.java	Updates @ScalarFunction volatility documentation to reflect context-specific (OFFLINE vs REALTIME) transform policies.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java	Changes validation so only REALTIME table-level ingestion transforms enforce IMMUTABLE volatility.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java	Updates/extends validation tests for OFFLINE acceptance, REALTIME rejection + grandfathering, offline-upsert transform validation, and post-partial-upsert volatility checks.
pinot-core/src/test/java/org/apache/pinot/core/util/SchemaUtilsTest.java	Adjusts the “grandfather existing non-deterministic transform” test to model REALTIME behavior post-policy change.
pinot-controller/src/test/java/org/apache/pinot/controller/api/TableConfigsRestletResourceTest.java	Updates REST tests to model legacy REALTIME transform grandfathering and adds OFFLINE non-immutable create/update coverage.
pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTableRestletResourceTest.java	Updates REST API tests to allow OFFLINE non-immutable transforms and keep REALTIME behavior unchanged (reject + legacy update path).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.96%. Comparing base (ffa97d9) to head (e5d287d).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19244      +/-   ##
============================================
+ Coverage     66.94%   66.96%   +0.02%     
  Complexity     1423     1423              
============================================
  Files          3453     3453              
  Lines        218858   218868      +10     
  Branches      34787    34790       +3     
============================================
+ Hits         146512   146567      +55     
+ Misses        60621    60583      -38     
+ Partials      11725    11718       -7     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	66.96% <100.00%> (+0.02%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	66.96% <100.00%> (+0.02%)	⬆️
unittests	66.96% <100.00%> (+0.02%)	⬆️
unittests1	57.71% <100.00%> (+0.01%)	⬆️
unittests2	39.02% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Documentation follow-up: pinot-contrib/pinot-docs#983 (merged).