Report outbound hosts to backend endpoint to forward to splunk/kusto

[AbhishekBhaskar]

What are you trying to accomplish?

The egress-allowlist observability data currently only lands in proxy logs, which aren't forwarded to Splunk/Kusto — so we can't analyze the outbound hosts jobs actually contact in order to tune the allowlist before flipping enforce on.

This adds a proxy-side reporter that batches the outbound hosts observed during a job and POSTs them to a new dependabot-api endpoint, POST /update_jobs/{job_id}/record_egress_hosts (alongside record_metrics), which forwards them to Splunk/Kusto.

Depends on the corresponding record_egress_hosts endpoint in dependabot-api.

Anything you want to highlight for special attention from reviewers?

• What's reported: every observed host with an allowlisted flag (not just non-allowlisted), so we get the full egress picture per ecosystem. Payload shape: {"data":[{host, allowlisted, count, package_manager}, ...]}. Easy to narrow to non-allowlisted-only if preferred.
• Reuses existing patterns: the egress.Collector mirrors metrics.CollectorClient (buffered, 1-min flush + flush-on-close, skips when the API endpoint is empty for smoke tests); RecordEgressHosts reuses the API client's retry/backoff doRequest.
• Gated by the existing flags: recording only happens when proxy_egress_observe/proxy_egress_enforce is active; fail-open mode records nothing. Distinct hosts are capped (10k) to bound memory.
• The EgressAllowlistHandler now takes an EgressHostRecorder interface (nil-safe), keeping the handler decoupled from the collector.

How will you know you've accomplished your goal?

• New unit tests: apiclient hits /update_jobs/{job_id}/record_egress_hosts; collector aggregates counts, clears its buffer, skips empty-endpoint/empty-host, and flushes on close; handler records observed hosts with correct allowlisted status and records nothing when disabled.
• Full suite + gofmt/go vet pass locally (go test ./...).
• End-to-end: once the API endpoint is live, observed hosts appear in Splunk/Kusto keyed by package_manager.

Checklist

• I have run the complete test suite to ensure all tests and linters pass.
• I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
• I have written clear and descriptive commit messages.
• I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
• I have ensured that the code is well-documented and easy to understand.

[Copilot]

Copilot review overview

🟡 Changes recommended

Shutdown synchronization and failed-batch retention must be addressed to prevent buffered telemetry loss.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite (auto)
Findings: 2

Note

Copilot is running an experiment and ran this review at Lite.

New issues introduced by this change (2)

Severity	Finding
	internal/​egress/​collector.go — Shutdown can exit before the final egress flush View comment
	internal/​egress/​collector.go — Failed reports permanently discard the buffered hosts View comment

What changed in this PR

Adds batched outbound-host reporting from the proxy to the Dependabot API for Splunk/Kusto observability.

Changes:

• Collects and aggregates observed egress hosts.
• Integrates recording with allowlist handling and proxy lifecycle.
• Adds API support and unit tests for reporting.

File	Summary
proxy.go	Wires and stops the egress collector; shutdown does not wait for final flushing.
internal/​metrics/​collector_client_test.go	Updates the API client mock.
internal/​handlers/​egress_allowlist.go	Records observed hosts and allowlist status.
internal/​handlers/​egress_allowlist_test.go	Tests recording behavior.
internal/​egress/​collector.go	Implements batching and reporting; requires shutdown synchronization and failed-batch retention.
internal/​egress/​collector_test.go	Tests aggregation and flushing.
internal/​apiclient/​client.go	Adds the egress reporting endpoint.
internal/​apiclient/​client_test.go	Tests endpoint requests and payload handling.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.