Apply ingestion Groovy policy consistently

[xiangfu0]

Summary

• Centralize ingestion Groovy policy enforcement before evaluator construction.
• Apply the policy consistently to schema FieldSpec and table-config transforms during controller validation.
• Propagate the resolved policy through realtime server ingestion, minion tasks, and offline segment generation, including standalone, Spark, and Hadoop paths.
• Reject disabled Groovy transforms with a clear configuration error instead of compiling, executing, or silently ignoring them.
• Preserve explicitly enabled Groovy, non-Groovy built-in transforms, legacy map transforms, and query-time Groovy behavior.

Root cause

The existing ingestion Groovy setting was enforced for table-config validation, but schema FieldSpec validation and several runtime evaluator-construction paths did not receive or apply the same policy. A schema persisted by an older deployment could therefore also reach ingestion-time evaluator construction without the controller-side check.

This change makes evaluator construction the common enforcement boundary and retains explicit checks at validation and runtime call sites as defense in depth.

How to reproduce

1. Leave controller.disable.ingestion.groovy unset, which uses the disabled default.
2. Attempt a schema create, update, or validation operation, or a combined tableConfigs operation, whose schema contains a Groovy-backed FieldSpec transform.
3. Before this change, the transform could reach evaluator construction despite the disabled policy. A schema persisted by an older deployment could reach the same path during ingestion.
4. After this change, Pinot rejects the transform with a configuration error before Groovy compilation. Explicitly setting the policy to false preserves the existing opt-in behavior.

Behavior

• Only an explicit controller.disable.ingestion.groovy=false enables ingestion Groovy.
• Missing, blank, or invalid values fail closed.
• Schema create, update, validation, and combined tableConfigs operations apply the same policy as table-config transforms.
• Authorization is checked before semantic validation on affected validation paths, so denied requests cannot trigger evaluator construction.
• Realtime ingestion and offline/minion segment-generation paths enforce the policy for persisted legacy schemas.
• Built-in transforms continue to work when Groovy is disabled.
• Conflicting controller, server, or minion settings fail with an actionable configuration error once the cluster policy is present.

Rolling upgrade and rollback

Upgrade an API-serving controller before servers and minions. The first API-serving controller atomically publishes the resolved policy to the cluster config; once present, that cluster value is authoritative for controller, server, and minion processes.

For a legacy cluster where ingestion Groovy is intentionally enabled, configure API-serving controllers consistently with controller.disable.ingestion.groovy=false before upgrading them. If a new server or minion must start before an upgraded controller publishes the cluster value, temporarily configure the same explicit opt-in on that instance. An unconfigured new runtime process fails closed while the cluster value is absent.

Coordinate later policy changes through the cluster configuration before restarting services. Rolling back runtime roles removes the new defense-in-depth enforcement from those older processes, so keep the cluster policy consistent and verify persisted schemas before rollback.

SPI compatibility

This change adds an immutable IngestionGroovyPolicy value and a RecordEnricherCreationContext for policy-aware enricher construction. RecordEnricherFactory retains the existing creation method; the new context-aware overload delegates to it by default, and the new security-policy hook is also a default method. Existing enricher implementations therefore require no source changes, while policy-sensitive implementations can opt into the new context and validation hook.

Test coverage

Targeted tests cover:

• Default-disabled and explicit-opt-in evaluator behavior.
• Schema create, update, and validation operations.
• Combined tableConfigs create, update, and validation operations.
• Authorization-before-validation ordering.
• Table-config and FieldSpec transform policy consistency.
• Built-in transforms under the disabled policy.
• Realtime ingestion with persisted legacy schemas.
• Offline segment generation through common, standalone, Spark, Hadoop, and minion paths.
• Groovy filters, enrichers, and transform-function partitioners.
• Concurrent controller startup and authoritative cluster-policy propagation.
• Existing hybrid and stale-segment integration scenarios with explicit Groovy opt-in.

Affected-module formatting, checkstyle, license-format, and license checks pass. The JDK 25 warning-enabled affected-module compile passes with -Xlint:all,-deprecation.

The exact JDK 25 -Xlint:all invocation remains blocked on current master because zstd-jni references its provided JetBrains @NotNull dependency from the unchanged ZstandardDecompressor source.

Scope

This change does not alter authorization actions or query-time Groovy policy. This PR is ready for coordinated review.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.03792% with 95 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.03%. Comparing base (2aa3aad) to head (c15263a).

Files with missing lines	Patch %	Lines
...ealtime/writer/StatelessRealtimeSegmentWriter.java	0.00%	10 Missing ⚠️
...sks/refreshsegment/RefreshSegmentTaskExecutor.java	0.00%	7 Missing ⚠️
...upsertcompaction/UpsertCompactionTaskExecutor.java	0.00%	7 Missing ⚠️
...he/pinot/segment/local/utils/TableConfigUtils.java	87.27%	6 Missing and 1 partial ⚠️
...richer/function/CustomFunctionEnricherFactory.java	53.84%	3 Missing and 3 partials ⚠️
...ache/pinot/common/utils/ServiceStartableUtils.java	90.00%	0 Missing and 3 partials ⚠️
...mmender/realtime/provisioning/MemoryEstimator.java	75.00%	3 Missing ⚠️
...che/pinot/controller/util/FileIngestionHelper.java	50.00%	3 Missing ⚠️
...sing/partitioner/TransformFunctionPartitioner.java	50.00%	3 Missing ⚠️
...local/recordtransformer/ExpressionTransformer.java	78.57%	3 Missing ⚠️
... and 25 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19235      +/-   ##
============================================
+ Coverage     66.97%   67.03%   +0.06%     
- Complexity     1423     1432       +9     
============================================
  Files          3453     3457       +4     
  Lines        218936   219292     +356     
  Branches      34802    34843      +41     
============================================
+ Hits         146638   147011     +373     
+ Misses        60588    60570      -18     
- Partials      11710    11711       +1     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.03% <81.03%> (+0.06%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.03% <81.03%> (+0.06%)	⬆️
unittests	67.03% <81.03%> (+0.06%)	⬆️
unittests1	57.73% <54.68%> (-0.01%)	⬇️
unittests2	39.11% <70.25%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.