RANGER-5719: Shared partition-plan library and SPIFFE header auth utilities by ramackri · Pull Request #1137 · apache/ranger

ramackri · 2026-08-06T02:23:32Z

Summary

First PR in the RANGER-5655 series — Admin-managed Kafka audit partition plan.

Split from closed #1135.

This PR adds the partition plan library only (17 files). It does not change runtime behavior, SPIFFE auth, Admin REST, ingestor, or DB schema. Safe to merge independently.

Outbound SPIFFE / trusted-header auth for the audit-server destination is in #1139 (RANGER-5723).

Partition plan library (`agents-common`)

Canonical JSON model stored in Admin x_ranger_global_state (RangerAuditPartitionPlan):

Component	Purpose
`PartitionPlan`, `PluginEntry`, `BufferEntry`	JSON model: topic, version, plugin → partition map, buffer slots
`PartitionPlanAllocator`	Assigns Kafka partition numbers when plugins are promoted from buffer
`PartitionPlanValidator`	Validates plan shape, version, and partition ranges
`PartitionPlanRoutingUtils`	Resolves plugin ID → Kafka partition at ingestor runtime
`PolicyDownloadAuthUsersUtil`	Extracts `policy.download.auth.users` per service for ingestor allow-lists
`AuditPartitionPlanAdminConfig`	Reads Admin partition-count settings (`partitions.per.plugin`, per-plugin overrides)
`AuditPartitionPlanConstants`	Shared defaults (`ranger_audits` topic, initial version, property names)

Tests: allocator, validator, routing, JSON round-trip, allow-list extraction.

Related PRs — RANGER-5655 merge order

PR	JIRA	Scope	Depends on
This PR	RANGER-5719	Partition plan library only	—
#1138	RANGER-5720	DB patch 078	— (parallel)
#1139	RANGER-5723	SPIFFE utilities + audit-server destination outbound	— (parallel)
(future)	RANGER-5721	Admin REST + `AuditPartitionPlanMgr`	#1137, #1138
(future)	RANGER-5722	Ingestor routing, inbound SPIFFE, Admin poller	#1137, #1138, #1139

AuditPluginIdResolver deferred to RANGER-5721.

Test plan

mvn -pl agents-common -am test — partition plan unit tests pass
CI: build-17, plugins-docker-build, services-docker-build (commit b0a5a1f67)

Review follow-up

Addressed @rameeshm's Aug 14 comments in b0a5a1f67:

Contiguous 1..topicPartitionCount partition ID validation
Simplified appendTailPartitions
Null-safe allow-list normalization in PolicyDownloadAuthUsersUtil
Append-only plugin check via subList

Wire PluginHeaderAuthConfig into RangerAuditServerDestination so plugins send X-Spiffe-Id on audit POSTs when xasecure.audit.destination.auditserver.authn.header.enabled=true. Requires RANGER-5719 (apache#1137) for PluginHeaderAuthConfig and setTrustedAuthHeaders().

…uting Add agents-common partition plan model, allocator, validator, routing helpers, and PolicyDownloadAuthUsersUtil for RANGER-5655. SPIFFE header utilities move to RANGER-5723 (apache#1139).

Drop scalePlugin, replacePlan, isPromoteAlreadyApplied, sameContentAs, toAllowedUserSets, hashToSlotIndex, and related helpers that no production caller uses in the 5721/5722 design; can reintroduce when scaling or bulk replace is wired.

Remove PluginEntry.empty(), the four-arg promotePlugin overload, and DEFAULT_AUDIT_TOPIC (tests use the literal topic name).

No callers in apache#1137, rangerRelease 5721/5722, or tests; plans are built via plugins(map).

Copilot

Pull request overview

Introduces a shared “audit partition plan” library under agents-common to model, validate, allocate, and route Admin-managed Kafka audit partitions, with accompanying unit tests to lock down expected behaviors and JSON compatibility.

Changes:

Added canonical JSON model types for the partition plan (PartitionPlan, PluginEntry, BufferEntry) plus JSON round-trip support.
Added core utilities for plan lifecycle and runtime use: allocation/mutation (PartitionPlanAllocator), validation (PartitionPlanValidator), and routing helpers (PartitionPlanRoutingUtils).
Added supporting utilities/constants (AuditPartitionPlanConstants, AuditPartitionPlanAdminConfig, PolicyDownloadAuthUsersUtil) and a focused unit test suite.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
agents-common/src/main/java/org/apache/ranger/audit/partition/AuditPartitionPlanAdminConfig.java	Reads Admin configuration to determine per-plugin partition allocation defaults/overrides.
agents-common/src/main/java/org/apache/ranger/audit/partition/AuditPartitionPlanConstants.java	Defines shared constants for the partition plan feature (defaults + property keys).
agents-common/src/main/java/org/apache/ranger/audit/partition/PartitionPlanAllocator.java	Implements append-only mutation helpers for promoting plugins/onboarding services/updating allow-lists.
agents-common/src/main/java/org/apache/ranger/audit/partition/PartitionPlanRoutingUtils.java	Converts logical plan partition IDs to Kafka producer partition indices (0-based) with clamping.
agents-common/src/main/java/org/apache/ranger/audit/partition/PartitionPlanValidator.java	Validates plan structure and enforces append-only constraints across updates.
agents-common/src/main/java/org/apache/ranger/audit/partition/PolicyDownloadAuthUsersUtil.java	Parses/normalizes `policy.download.auth.users` values used for ingestor authorization metadata.
agents-common/src/main/java/org/apache/ranger/audit/partition/exception/PartitionPlanException.java	Defines a dedicated runtime exception type for plan validation/allocation failures.
agents-common/src/main/java/org/apache/ranger/audit/partition/model/BufferEntry.java	Models buffer partitions portion of the plan for future plugin promotions.
agents-common/src/main/java/org/apache/ranger/audit/partition/model/PartitionPlan.java	Core JSON model + (de)serialization entrypoints and builder.
agents-common/src/main/java/org/apache/ranger/audit/partition/model/PluginEntry.java	Models per-plugin partitions and associated Ranger service repos.
agents-common/src/test/java/org/apache/ranger/audit/partition/AuditPartitionPlanAdminConfigTest.java	Verifies default/override resolution behavior for Admin config.
agents-common/src/test/java/org/apache/ranger/audit/partition/PartitionPlanAllocatorTest.java	Exercises allocator mutations (promotion/onboarding/removal/allow-list updates).
agents-common/src/test/java/org/apache/ranger/audit/partition/PartitionPlanRoutingUtilsTest.java	Verifies logical-id → Kafka-index mapping and clamping behavior.
agents-common/src/test/java/org/apache/ranger/audit/partition/PartitionPlanTestSupport.java	Provides reusable seed/pre-assigned plan fixtures for tests.
agents-common/src/test/java/org/apache/ranger/audit/partition/PartitionPlanValidatorTest.java	Covers validator rejection/acceptance scenarios and append-only constraints.
agents-common/src/test/java/org/apache/ranger/audit/partition/PolicyDownloadAuthUsersUtilTest.java	Verifies parsing/normalization rules for allow-list values.
agents-common/src/test/java/org/apache/ranger/audit/partition/model/PartitionPlanJsonTest.java	Validates JSON round-trip stability for seed/onboarded/allow-list variants.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- Reject null plugin entries, non-contiguous partition ids, and ineffective allow-lists - Split append-only validation messages; allow metadata updates at same partition count - Simplify tail partition allocation; harden allow-list normalization against null elements

rameeshm · 2026-08-24T18:04:47Z

+
+/** Parses {@code policy.download.auth.users} service config for audit ingestor authorization. */
+public final class PolicyDownloadAuthUsersUtil {
+    public static final String CONFIG_NAME = "policy.download.auth.users";


Do we need to rely only on "policy.download.auth.users"? I feel we should have a new param "service.access.audit.user" which will be a keytab user (by default service user can be there e.g hive, rangeraz, hdfs, etc) and if its not there then check for Service admin and if no there then look for "policy.download.auth.users". What do you say? @mneethiraj need your input on this. Thanks.

@ramackri - if this is to determine the users authorized to report audits for a service, we should only recognize service.admin.users and service.admin.groups. Picking up users frompolicy.download.auth.users is not correct.

This was referenced Aug 6, 2026

RANGER-5720: DB patch 078 for audit partition plan global state #1138

Open

RANGER-5723: Plugin SPIFFE outbound auth for audit-server destination #1139

Merged

RANGER-5655: Shared partition plan library, DB patch 078, plugin SPIFFE auth #1135

Closed

ramackri requested review from mneethiraj and rameeshm August 6, 2026 02:31

RANGER-5719: Shared partition-plan library for admin-managed audit ro…

0ced753

…uting Add agents-common partition plan model, allocator, validator, routing helpers, and PolicyDownloadAuthUsersUtil for RANGER-5655. SPIFFE header utilities move to RANGER-5723 (apache#1139).

ramackri force-pushed the RANGER-5719-patch branch from c4cf544 to 0ced753 Compare August 6, 2026 03:15

ramk added 3 commits August 6, 2026 13:02

RANGER-5719: Trim remaining unused partition-plan helpers

645fa69

Remove PluginEntry.empty(), the four-arg promotePlugin overload, and DEFAULT_AUDIT_TOPIC (tests use the literal topic name).

RANGER-5719: Remove unused PartitionPlan.Builder.putPlugin

37348ab

No callers in apache#1137, rangerRelease 5721/5722, or tests; plans are built via plugins(map).

ramackri requested a review from kumaab August 7, 2026 16:38

rameeshm requested a lite review from Copilot August 14, 2026 20:30

Copilot started reviewing on behalf of rameeshm August 14, 2026 20:30 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

rameeshm reviewed Aug 14, 2026

View reviewed changes

ramackri changed the title ~~RANGER-5719: Shared partition-plan library and SPIFFE header auth utilities~~ RANGER-5719: Shared partition-plan library for admin-managed audit routing Aug 18, 2026

ramackri changed the title ~~RANGER-5719: Shared partition-plan library for admin-managed audit routing~~ RANGER-5719: Shared partition-plan library and SPIFFE header auth utilities Aug 18, 2026

rameeshm reviewed Aug 26, 2026

View reviewed changes

Navigation Menu

Uh oh!

RANGER-5719: Shared partition-plan library and SPIFFE header auth utilities - #1137

RANGER-5719: Shared partition-plan library and SPIFFE header auth utilities#1137
ramackri wants to merge 5 commits into
apache:masterfrom
ramackri:RANGER-5719-patch

ramackri commented Aug 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rameeshm Aug 24, 2026

Uh oh!

mneethiraj Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ramackri commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Partition plan library (agents-common)

Related PRs — RANGER-5655 merge order

Test plan

Review follow-up

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rameeshm Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

mneethiraj Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ramackri commented Aug 6, 2026 •

edited

Loading

Partition plan library (`agents-common`)