Skip to content

add defaultMetrics for logging emitter#19030

Open
sarangv wants to merge 3 commits intoapache:masterfrom
sarangv:logging-emitter-allowlist
Open

add defaultMetrics for logging emitter#19030
sarangv wants to merge 3 commits intoapache:masterfrom
sarangv:logging-emitter-allowlist

Conversation

@sarangv
Copy link

@sarangv sarangv commented Feb 18, 2026

Fixes 19021.

Description

LoggingEmitter can restrict which metrics are written to the log. When this behavior is turned on, only ServiceMetricEvent metrics that appear in a configured permitted set are emitted; alerts and other non-metric events are never dropped. When the feature is off, the emitter behaves as before (all events logged).

Runtime properties

  • druid.emitter.logging.shouldFilterMetrics (default false): If true, logging is limited to metrics named in the permitted set. If false, every event is logged.
  • druid.emitter.logging.allowedMetricsPath (optional): File path for the permitted-metrics definition. Omitted or empty → the in-JAR resource defaultMetrics.json is used. If a path is given but the file does not exist, a warning is logged and the in-JAR resource is used instead (startup continues; operators can spot the warning).

Permitted-metrics file shape

The file must be a JSON object whose keys are metric names; values are placeholders (e.g. []), e.g. {"query/time": [], "query/bytes": []}. This shape matches other emitter default configs (e.g. Prometheus) and leaves room for per-metric metadata later. On invalid content, operators see: "Allowed metrics file must be a JSON object with metric names as keys".

Rationale

  • DruidException: Errors use DruidException.forPersona(Persona.OPERATOR) with NOT_FOUND (default resource absent) and RUNTIME_FAILURE (parse/read failure), matching the project’s guidance for operator-facing failures.
  • Missing custom file: Rather than failing startup, the code logs a WARN and uses the in-JAR default so a wrong path does not prevent the process from starting; the log still exposes the mistake.
  • Object format: A key-value object was chosen over a JSON array so the schema can later carry extra fields per metric without a breaking change.
  • Emitters.java: The two new keys are written into the legacy logging map only when the matching property exists, preserving Java defaults and avoiding odd empty-string behavior.
  • No set when off: With filtering disabled, allowedMetrics stays null; no extra collection is created. Contains checks stay O(1).

Release note

Operators can limit which metrics the logging emitter writes by setting druid.emitter.logging.shouldFilterMetrics=true and, if desired, druid.emitter.logging.allowedMetricsPath to a JSON object file (keys = metric names). A missing custom file results in a warning and use of the bundled defaultMetrics.json. Alerts and other non-metric events are always logged.


Key changed/added classes in this PR
  • LoggingEmitter – load permitted set (classpath or file; use default if file missing), apply in emit(), DruidException on bad/missing config
  • LoggingEmitterConfigshouldFilterMetrics, allowedMetricsPath plus Javadoc
  • Emitters – map the new keys from legacy properties only when set
  • processing/src/main/resources/defaultMetrics.json – in-JAR permitted set (object form)
  • processing/src/test/resources/defaultMetrics.json – test resource
  • LoggingEmitterTest – seven cases: feature off, default resource, custom file, missing file uses default, non-metrics pass through, empty set, path ignored when off; @After tearDown for close
  • LoggingEmitterConfigTest – new config fields covered for defaults and legacy mapping

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@sarangv sarangv force-pushed the logging-emitter-allowlist branch from d795910 to b5a2d02 Compare February 18, 2026 21:33
Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Contributor

@abhishekrb19 abhishekrb19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @sarangv! There's also #19026
from @nozjkoitop, which has a similar implementation. Would you both be open to collaborating on the reviews and aligning on any implementation differences? 🙂

I see this PR has a bit more test coverage and the defaultMetrics file schema is a bit more extensible if we want to add additional dimensions in the future similar to other emitter types.

|`druid.emitter.logging.loggerClass`|The class used for logging.|`org.apache.druid.java.util.emitter.core.LoggingEmitter`|
|`druid.emitter.logging.logLevel`|Choices: debug, info, warn, error. The log level at which message are logged.|info|
|`druid.emitter.logging.shouldFilterMetrics`|When true, only metrics listed in the allowlist are emitted; non-metric events (e.g. alerts) are always emitted. When false, all events are logged (backward-compatible).|false|
|`druid.emitter.logging.allowedMetricsPath`|Path to a JSON file whose keys are the allowed metric names. Only used when `shouldFilterMetrics` is true. If null or empty, the bundled classpath resource `defaultMetrics.json` is used. If a path is set but the file is missing, a warning is logged and the emitter falls back to the default classpath resource.|null|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a path is set but the file is missing, a warning is logged and the emitter falls back to the default classpath resource.

IMO, in this case, we should fail fast by throwing an exception instead of falling back in the event of a misconfiguration, since an operator would be intentionally opting in to enable metric filtering and provide a custom path

@nozjkoitop
Copy link
Contributor

Sure thing! I updated #19026 to match the deltas noted here (more test coverage and extensible default-metrics schema). @sarangv could you take a quick look at the updated #19026 and leave any review notes there? I’d prefer we keep review in a single PR to avoid duplicating effort

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow filtering of metrics in the LoggingEmitter

3 participants

Comments