Skip to content

SOLR-18062: Support arbitrary Kafka properties via env and sysprops#4185

Open
openworld-maker wants to merge 1 commit intoapache:mainfrom
openworld-maker:SOLR-18062-kafka-additional-properties
Open

SOLR-18062: Support arbitrary Kafka properties via env and sysprops#4185
openworld-maker wants to merge 1 commit intoapache:mainfrom
openworld-maker:SOLR-18062-kafka-additional-properties

Conversation

@openworld-maker
Copy link
Contributor

Description

This PR completes SOLR-18062 by hardening and clarifying CrossDC Kafka pass-through property behavior.

Jira: https://issues.apache.org/jira/browse/SOLR-18062

main already included the core SOLR-18062 support for arbitrary Kafka properties via:

  • SOLR_CROSSDC_KAFKA_* environment variables
  • solr.crossdc.kafka.* system properties

This change focuses on missing edge-case handling, precedence clarity, and testability.

What Changed

1) Hardened pass-through extraction in ConfUtil

  • Added a testable helper:
    • addAdditionalKafkaProperties(Map<String,Object> properties, Map<String,String> env, Properties sysProps)
  • Preserved existing explicit config values already present in properties.
  • Added explicit precedence for pass-through collisions:
    • solr.crossdc.kafka.* system properties override SOLR_CROSSDC_KAFKA_* env values for the same normalized Kafka key.
  • Added edge-case filtering:
    • ignore blank / empty values
    • ignore empty normalized keys (for example bare prefixes)

2) Added focused unit coverage in ConfUtilTest

  • env mapping (SOLR_CROSSDC_KAFKA_FOO_BAR -> foo.bar)
  • sysprop mapping (solr.crossdc.kafka.foo.bar -> foo.bar)
  • precedence for env vs sysprop conflicts
  • explicit-property preservation (pass-through does not overwrite explicit value)
  • blank value and empty key filtering
  • verified pass-through values are preserved through KafkaCrossDcConf.getAdditionalProperties()

3) Updated CrossDC documentation

  • Clarified key normalization behavior
  • Documented precedence (sysprop > env for same pass-through key)
  • Documented explicit setting preservation and blank-value/empty-key ignoring

Behavior / Compatibility

  • Backward compatible with current SOLR-18062 prefixes and behavior.
  • No changes to required CrossDC properties.
  • No change to producer/consumer wiring (getAdditionalProperties() use remains as-is).

Validation

Ran locally:

  • ./gradlew :solr:modules:cross-dc:test --tests org.apache.solr.crossdc.common.ConfUtilTest
  • ./gradlew tidy :solr:modules:cross-dc:check -x test :solr:solr-ref-guide:check -x test

@github-actions github-actions bot added documentation Improvements or additions to documentation tests labels Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants