[Scheduler] Add new scheduler metrics#10402
Conversation
This reverts commit 517781d.
| newTaggedMetricsHandler(h.metricsHandler, scheduler). | ||
| Counter(metrics.ScheduleInvokerExecuteInvalidated.Name()). | ||
| Record(int64(count), metrics.ReasonTag(invokerExecuteInvalidatedAlreadyRecorded)) | ||
| newTaggedLogger(h.baseLogger, scheduler).Debug( |
There was a problem hiding this comment.
This'll be moved to event log shortly, I just don't want too many PRs stacked.
|
Couple suggestions: 1. Use an Several of these new metrics follow a 2. Take it further with a Rather than one metric per scheduler component, a single Instead of 13 separate metrics: One metric: This makes common queries much simpler: Cardinality impact is negligible --
|
New metrics
All counters are tagged with
namespaceandschedule_backendvianewTaggedMetricsHandler. Reason-tagged counters use areasontag with limited cardinality.Generator
schedule_generator_ticksschedule_generator_paused_ticksscheduler_generator_loop_completedIdle
schedule_idle_task_firedschedule_idle_task_invalidatedreason:held_open/expiration_shift/closedInvoker
schedule_invoker_process_buffer_firedschedule_invoker_process_buffer_invalidatedreason:stale_hwmschedule_invoker_execute_firedschedule_invoker_execute_invalidatedreason:no_work(Validate) oralready_recorded(concurrent ExecuteTask already wroteRunId)schedule_buffered_start_droppedreason:missed_catchup_windoworpaused_or_limitedBackfiller
schedule_backfiller_firedschedule_backfiller_invalidatedreason:stale_hwmschedule_backfiller_completed