Add consumer-group rebalance, empty-group, and metadata signals to kafka_consumer

[piochelepiotr]

What does this PR do?

Enriches the kafka_consumer cluster-monitoring consumer-group collection (enable_cluster_monitoring) with signals that cannot be derived by grouping existing tagged metrics in the Datadog UI:

• kafka.consumer_group.rebalancing (gauge, 1/0): detected two ways — group state (PreparingRebalance/CompletingRebalance) for classic groups, and assignment != target_assignment for KIP-848 consumer-protocol groups.
• kafka.consumer_group.empty (gauge, 1/0): 1 when the group is in the EMPTY state (committed offsets but no active members — an orphaned/abandoned group).

Dimensional group metadata is added as tags on existing gauges (no new metrics), per review preference:

• partition_assignor, consumer_group_type, is_simple_consumer_group on kafka.consumer_group.members
• group_instance_id (static membership) on kafka.consumer_group.member.partitions

All data is sourced from fields describe_consumer_groups already returns in the pinned confluent-kafka==2.13.2 client — no new API calls.

This is PR 1 of a sequenced plan to close the "consumer groups are second-class" gap identified in the DSM-Kafka vendor evaluation. Lag rollups were intentionally excluded: kafka.consumer_lag is already tagged by consumer_group, so per-group lag is a UI group-by, not a new metric.

Motivation

The DSM-Kafka product has no first-class consumer-group view; group health (rebalancing, orphaned groups, assignment strategy) was not observable. These signals are not derivable from existing metrics and require their own series to be alertable.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

🤖 Generated with Claude Code

[datadog-prod-us1-6]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 88.50% (+1.04%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 625cfcd | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run