[GSoC 2026] Kafka Streams runner: surface SDK-harness metrics as MetricResults

[junaiddshaukat]

Summary

Collects the MonitoringInfos the SDK harness reports with each bundle and exposes them as Beam MetricResults. Part of #18479, first step toward the @ValidatesRunner suite -- PAssert verifies its assertions ran via a success counter it reads from PipelineResult.metrics(), which previously threw not-implemented.

Follows the Flink pattern (FlinkExecutableStageFunction):

• The translation context owns a MetricsContainerStepMap, one container per executable stage.
• ExecutableStageProcessor replaces BundleProgressHandler.ignored() with a handler that folds the harness's final MonitoringInfos into its stage's container as each bundle completes. Only the completion response is applied — progress responses carry cumulative mid-bundle snapshots and the container's update adds counter values, so folding both would double-count.
• KafkaStreamsPortablePipelineResult.metrics() returns asAttemptedOnlyMetricResults; attempted values only for now -- committed metrics need to be folded into the exactly-once commit, which belongs with the durability work.
• KafkaStreamsTestRunner.run now returns the MetricResults so tests can assert on user counters.

Test: MetricsTest runs Create.of(1, 2, 3) through a counting DoFn in the EMBEDDED harness and asserts the user counter reads 3 through the runner's MetricResults.

Next up (separate PRs): a TestPipeline-dispatchable runner harness, then the validatesRunner gradle task with the first enabled tests.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enables the Kafka Streams runner to surface SDK-harness metrics as MetricResults. By integrating MonitoringInfos collection into the bundle processing lifecycle, the runner can now report attempted metrics, which is a critical prerequisite for supporting the @ValidatesRunner test suite and PAssert verification.

Highlights

• Metrics Integration: Implemented collection of MonitoringInfos from the SDK harness and exposed them as Beam MetricResults in the Kafka Streams runner.
• Architecture: Adopted the Flink pattern by using a MetricsContainerStepMap in the translation context, allowing executable stage processors to update metrics as bundles complete.
• Testing: Added a new MetricsTest to verify that user counters from a DoFn correctly surface through the runner's MetricResults.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request implements user metrics reporting from the SDK harness in the Kafka Streams runner by capturing cumulative bundle metrics on completion and exposing them via MetricsContainerStepMap. The reviewer identified a critical concurrency issue where parallel tasks in a multi-partition topology would overwrite each other's metrics if they shared a single global map. To address this, the reviewer provided a comprehensive set of suggestions to instantiate, register, and merge task-specific metrics accumulators across all partitions when querying metrics.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[github-actions]

Assigning reviewers:

R: @shunping added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[je-ik]

Cool, let me know if you want to add the log of the responses, if not I'm ready to merge this as is.