add defaultMetrics for logging emitter#19030
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
d795910 to
b5a2d02
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Thanks for the contribution, @sarangv! There's also #19026
from @nozjkoitop, which has a similar implementation. Would you both be open to collaborating on the reviews and aligning on any implementation differences? 🙂
I see this PR has a bit more test coverage and the defaultMetrics file schema is a bit more extensible if we want to add additional dimensions in the future similar to other emitter types.
| |`druid.emitter.logging.loggerClass`|The class used for logging.|`org.apache.druid.java.util.emitter.core.LoggingEmitter`| | ||
| |`druid.emitter.logging.logLevel`|Choices: debug, info, warn, error. The log level at which message are logged.|info| | ||
| |`druid.emitter.logging.shouldFilterMetrics`|When true, only metrics listed in the allowlist are emitted; non-metric events (e.g. alerts) are always emitted. When false, all events are logged (backward-compatible).|false| | ||
| |`druid.emitter.logging.allowedMetricsPath`|Path to a JSON file whose keys are the allowed metric names. Only used when `shouldFilterMetrics` is true. If null or empty, the bundled classpath resource `defaultMetrics.json` is used. If a path is set but the file is missing, a warning is logged and the emitter falls back to the default classpath resource.|null| |
There was a problem hiding this comment.
If a path is set but the file is missing, a warning is logged and the emitter falls back to the default classpath resource.
IMO, in this case, we should fail fast by throwing an exception instead of falling back in the event of a misconfiguration, since an operator would be intentionally opting in to enable metric filtering and provide a custom path
Fixes 19021.
Description
LoggingEmittercan restrict which metrics are written to the log. When this behavior is turned on, onlyServiceMetricEventmetrics that appear in a configured permitted set are emitted; alerts and other non-metric events are never dropped. When the feature is off, the emitter behaves as before (all events logged).Runtime properties
druid.emitter.logging.shouldFilterMetrics(defaultfalse): Iftrue, logging is limited to metrics named in the permitted set. Iffalse, every event is logged.druid.emitter.logging.allowedMetricsPath(optional): File path for the permitted-metrics definition. Omitted or empty → the in-JAR resourcedefaultMetrics.jsonis used. If a path is given but the file does not exist, a warning is logged and the in-JAR resource is used instead (startup continues; operators can spot the warning).Permitted-metrics file shape
The file must be a JSON object whose keys are metric names; values are placeholders (e.g.
[]), e.g.{"query/time": [], "query/bytes": []}. This shape matches other emitter default configs (e.g. Prometheus) and leaves room for per-metric metadata later. On invalid content, operators see: "Allowed metrics file must be a JSON object with metric names as keys".Rationale
DruidException.forPersona(Persona.OPERATOR)withNOT_FOUND(default resource absent) andRUNTIME_FAILURE(parse/read failure), matching the project’s guidance for operator-facing failures.allowedMetricsstaysnull; no extra collection is created. Contains checks stay O(1).Release note
Operators can limit which metrics the logging emitter writes by setting
druid.emitter.logging.shouldFilterMetrics=trueand, if desired,druid.emitter.logging.allowedMetricsPathto a JSON object file (keys = metric names). A missing custom file results in a warning and use of the bundleddefaultMetrics.json. Alerts and other non-metric events are always logged.Key changed/added classes in this PR
LoggingEmitter– load permitted set (classpath or file; use default if file missing), apply inemit(), DruidException on bad/missing configLoggingEmitterConfig–shouldFilterMetrics,allowedMetricsPathplus JavadocEmitters– map the new keys from legacy properties only when setprocessing/src/main/resources/defaultMetrics.json– in-JAR permitted set (object form)processing/src/test/resources/defaultMetrics.json– test resourceLoggingEmitterTest– seven cases: feature off, default resource, custom file, missing file uses default, non-metrics pass through, empty set, path ignored when off;@AftertearDown for closeLoggingEmitterConfigTest– new config fields covered for defaults and legacy mappingThis PR has: