Add test reporting for Karate v2

[daniel-mohedano]

What Does This Do

• Adds a new karate-2.0 instrumentation module to implement test visibility for Karate 2.0+.
  • The implementation follows the same approach used for the original instrumentation in the karate-1.0 module.
  • Also implements skip-based features (ITR and disabled tests), as part of the tracing listener.

Motivation

Karate v2 is a complete ground-up rewrite of the framework. Because of this, the original karate-1.0 module cannot instrument it.

Additional Notes

A following PR will contain the changes regarding execution instrumentation for Karate v2.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: SDTEST-3816

[cit-pr-commenter-54b7da]

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	53.89	55.43	$\color{green}{\blacktriangledown}$ -1.54	54.33	$\color{green}{\blacktriangledown}$ -0.44	74/234
agentEvpProxy	53.85	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	20.77	22.09	$\color{green}{\blacktriangledown}$ -1.32	21.65	$\color{green}{\blacktriangledown}$ -0.88	38/121
agentless	17.83	18.82	$\color{green}{\blacktriangledown}$ -0.99	18.82	$\color{green}{\blacktriangledown}$ -0.99	37/119
agentlessCodeCoverage	18.91	20.39	$\color{green}{\blacktriangledown}$ -1.48	19.99	$\color{green}{\blacktriangledown}$ -1.08	37/118
agentlessLineCoverage	29.05	30.42	$\color{green}{\blacktriangledown}$ -1.37	29.82	$\color{green}{\blacktriangledown}$ -0.77	36/117

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.61	37.15	$\color{green}{\blacktriangledown}$ -0.54	36.42	$\color{red}{\blacktriangle}$ +0.19	37/114
agentless	37.56	36.42	$\color{red}{\blacktriangle}$ +1.14	36.42	$\color{red}{\blacktriangle}$ +1.14	37/114
agentlessCodeCoverage	45.01	45.38	$\color{green}{\blacktriangledown}$ -0.37	44.48	$\color{red}{\blacktriangle}$ +0.53	37/114
agentlessLineCoverage	75.15	74.82	$\color{red}{\blacktriangle}$ +0.33	74.82	$\color{red}{\blacktriangle}$ +0.33	36/113

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	7.95	9.35	$\color{green}{\blacktriangledown}$ -1.40	10.33	$\color{green}{\blacktriangledown}$ -2.38	37/114
agentless	10.60	10.33	$\color{red}{\blacktriangle}$ +0.27	10.13	$\color{red}{\blacktriangle}$ +0.47	37/114
agentlessCodeCoverage	16.96	16.69	$\color{red}{\blacktriangle}$ +0.27	17.38	$\color{green}{\blacktriangledown}$ -0.42	36/112

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.33	9.54	$\color{red}{\blacktriangle}$ +2.79	9.54	$\color{red}{\blacktriangle}$ +2.79	34/34

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.42	12.87	$\color{red}{\blacktriangle}$ +0.55	13.13	$\color{red}{\blacktriangle}$ +0.29	35/115
agentless	12.05	12.37	$\color{green}{\blacktriangledown}$ -0.32	12.12	$\color{green}{\blacktriangledown}$ -0.07	35/114
agentlessCodeCoverage	15.87	15.11	$\color{red}{\blacktriangle}$ +0.76	15.11	$\color{red}{\blacktriangle}$ +0.76	35/114
agentlessLineCoverage	19.46	18.82	$\color{red}{\blacktriangle}$ +0.64	18.82	$\color{red}{\blacktriangle}$ +0.64	35/115

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 57.03% (-0.01%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e459f4b | Docs | Datadog PR Page | Give us feedback!}

[cit-pr-commenter-54b7da]

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	92.23	93.23	$\color{green}{\blacktriangledown}$ -1.00	93.23	$\color{green}{\blacktriangledown}$ -1.00	41/124
agentless	89.79	89.58	$\color{red}{\blacktriangle}$ +0.21	89.58	$\color{red}{\blacktriangle}$ +0.21	38/120
agentlessCodeCoverage	99.08	99.00	$\color{red}{\blacktriangle}$ +0.08	99.00	$\color{red}{\blacktriangle}$ +0.08	40/119
agentlessLineCoverage	98.89	99.00	$\color{green}{\blacktriangledown}$ -0.11	99.00	$\color{green}{\blacktriangledown}$ -0.11	37/117

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	87.85	87.80	$\color{red}{\blacktriangle}$ +0.05	87.80	$\color{red}{\blacktriangle}$ +0.05	38/120
agentless	81.71	81.05	$\color{red}{\blacktriangle}$ +0.66	81.05	$\color{red}{\blacktriangle}$ +0.66	38/119
agentlessCodeCoverage	95.83	97.04	$\color{green}{\blacktriangledown}$ -1.21	95.12	$\color{red}{\blacktriangle}$ +0.71	38/117
agentlessLineCoverage	112.74	111.62	$\color{red}{\blacktriangle}$ +1.12	111.62	$\color{red}{\blacktriangle}$ +1.12	37/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.77	19.20	$\color{red}{\blacktriangle}$ +2.57	19.20	$\color{red}{\blacktriangle}$ +2.57	40/126
agentless	18.19	19.20	$\color{green}{\blacktriangledown}$ -1.01	19.20	$\color{green}{\blacktriangledown}$ -1.01	38/123
agentlessCodeCoverage	22.37	22.54	$\color{green}{\blacktriangledown}$ -0.17	22.09	$\color{red}{\blacktriangle}$ +0.28	38/121
agentlessLineCoverage	45.29	44.48	$\color{red}{\blacktriangle}$ +0.81	44.48	$\color{red}{\blacktriangle}$ +0.81	38/127

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	15.97	16.36	$\color{green}{\blacktriangledown}$ -0.39	16.04	$\color{green}{\blacktriangledown}$ -0.07	37/113
agentless	9.71	9.73	$\color{green}{\blacktriangledown}$ -0.02	9.73	$\color{green}{\blacktriangledown}$ -0.02	37/114
agentlessCodeCoverage	13.52	13.67	$\color{green}{\blacktriangledown}$ -0.15	13.40	$\color{red}{\blacktriangle}$ +0.12	38/113
agentlessLineCoverage	33.33	32.95	$\color{red}{\blacktriangle}$ +0.38	32.95	$\color{red}{\blacktriangle}$ +0.38	37/113

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.75 s	14.65 s	[-0.2%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	13.69 s	13.72 s	[-1.0%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	16.88 s	16.14 s	[+0.1%; +9.0%] (maybe worse)
startup:petclinic:iast:Agent	16.86 s	16.91 s	[-1.4%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.78 s	16.73 s	[-1.1%; +1.7%] (no difference)
startup:petclinic:sca:Agent	16.93 s	16.39 s	[-1.4%; +8.0%] (no difference)
startup:petclinic:tracing:Agent	15.80 s	15.64 s	[-4.8%; +6.9%] (unstable)

Commit: e459f4bc · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	-6.99	13.40	$\color{green}{\blacktriangledown}$ -20.39	14.81	$\color{green}{\blacktriangledown}$ -21.80	35/119
agentless	-3.47	12.62	$\color{green}{\blacktriangledown}$ -16.09	17.03	$\color{green}{\blacktriangledown}$ -20.50	35/118
agentlessCodeCoverage	94.47	79.45	$\color{red}{\blacktriangle}$ +15.02	86.07	$\color{red}{\blacktriangle}$ +8.40	35/118
agentlessLineCoverage	150.78	125.85	$\color{red}{\blacktriangle}$ +24.93	139.09	$\color{red}{\blacktriangle}$ +11.69	35/118

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[daniel-mohedano]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2599ffbfcf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[daniel-mohedano]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e459f4bc8e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

More details

The karate-2.0 instrumentation is behaviourally correct for its stated scope (basic test reporting + ITR skipping). The most non-obvious risk — that SCENARIO_EXIT fires even when the listener vetoes SCENARIO_ENTER — was confirmed safe: Karate v2 populates the result with skipped-step entries, so afterScenario does not spuriously call onTestSkip, and the unconditional onTestFinish call is a no-op because onTestIgnore already removed the test from inProgressTests. Tag names are stored without the @ prefix, confirming the datadog_itr_unskippable guard works correctly.

Was this helpful? React 👍 or 👎

_{📊 Validated against 4 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit e459f4b · What is Autotest? · Any feedback? Reach out in #autotest}