Add system-tests for agentless functionality - #7607
Draft
bwoebi wants to merge 12 commits into
Draft
Conversation
…urations Agentless remote config has no agent to relay client state via /v0.7/config: the native RC client polls the backend endpoint directly and reports its per-config apply state inline on that same request. Add the protobuf message definitions and wire them into the proxy's request deserializer so that state is visible to the test harness.
TUF derives a key's keyid from the canonical JSON of the key object, not from the raw public key bytes. libdatadog's rust-tuf recomputes the keyid this way when verifying the root, and previously failed to match our test signature (metadata root signature threshold not met). Fix the derivation and regenerate the two pinned fixtures that embed the old (wrong) keyid. Also: TUF metadata version 0 is spec-invalid, but was used as an empty-state sentinel for the mocked backend; real clients (correctly) reject it. Move the sentinel to version 1, and add an opt-in agent_refresh_interval field to targets metadata (default None/omitted, a no-op for all existing callers) so agentless scenarios can override the native RC client's 60s default poll interval down to something tests can wait on.
When context.scenario.include_agent is False, there is no agent to relay client state via /v0.7/config: watch /api/v0.1/configurations directly (via interfaces.datadog_direct) and read the client's reported state out of the request instead of a follow-up response, normalizing protobuf's stringified uint64 fields back to int along the way. Also re-sign real backend config pushes at version + 1 and set agent_refresh_interval=1, to stay strictly greater than the version-1 empty-state sentinel and keep the agentless client's poll interval fast enough for tests to wait on (see previous commit).
Replace hardcoded interfaces.agent/_LOGS_PATH/_DEBUGGER_PATH/etc. references throughout BaseDebuggerTest with overridable class attributes (_backend_interface, _snapshot_paths, _traces_path, _telemetry_path, _symbols_interface, _symbols_path), defaulting to today's agent-mode values so no existing test changes behavior. Add AgentlessBaseDebuggerTest, which points these at interfaces.datadog_direct and the single unified agentless debugger-intake path (libdatadog's agentless debugger sender collapses logs/snapshots/diagnostics/symdb onto one path, unlike the agent-relayed protocol). Span-decoration collection (_collect_span_decoration) calls get_spans_list(), which only interfaces.agent implements (it parses /v0.4/traces msgpack; agentless traces use a different wire format). Gate that call behind hasattr() so it's skipped for agentless backends rather than raising, per decision: span-decoration assertions stay agent-only for now.
AgentlessEndToEndScenario hardcoded FFE's own MockFFEAgentlessBackendServer into its base __init__/configure(). Replace that with a _create_mock_backend() hook (default None, meaning no mock backend), structurally typed via a small AgentlessBackendServer protocol, so other agentless scenarios can reuse this base without carrying FFE-specific setup. FeatureFlaggingAgentlessEndToEndScenario now overrides the hook and keeps its own env/extra_hosts wiring. Move the generic 'capture direct egress' wiring (DD_SITE=mock-intake.invalid, HTTPS_PROXY through the proxy container, mounting the mitmproxy CA bundle so the weblog trusts the intercepted TLS, watchdog/teardown registration for interfaces.datadog_direct) up from FeatureFlaggingAgentlessEndToEndScenario into the shared base, gated on a new capture_direct_egress constructor param (default True). Add pass-through rc_api_enabled/rc_backend_enabled params: when rc_backend_enabled is set, wire the same TUF test root env vars already used to make the real Datadog Agent trust the mocked RC backend (utils/_context/containers.py::AgentContainer), so the native agentless RC client trusts it too. Override _wait_for_app_readiness() to skip the interfaces.library.ready wait when capturing direct egress: that readiness signal only ever fires on agent-facing traffic, which agentless mode never sends, so it would time out on every run otherwise. Also fix a resource leak in _start_mock_backend(): the mock backend was only assigned to self._mock_backend after backend.reset() succeeded, so a failing reset() skipped cleanup in configure()'s except handler and leaked the backend.
APM_TRACING_AGENTLESS (AgentlessEndToEndScenario) covers direct-to-intake trace submission, client-side stats, and Remote Configuration under DD_AGENTLESS_ENABLED, with no Datadog Agent. DEBUGGER_AGENTLESS (new DebuggerAgentlessScenario, in its own module) covers agentless Dynamic Instrumentation (probe upload/logs/snapshots) and Symbol DB the same way, reusing the TUF-trust and mocked RC backend wiring generalized in the previous commit. Both are excluded from the tracer_release scenario group, same as the other agentless scenarios; add them to test_group_rules.py's exclusion list.
…er, SymDB) tests/test_agentless.py (APM_TRACING_AGENTLESS): trace submission, client-side stats, and Remote Configuration all land directly on the intake with no Datadog Agent, each asserted on host/path/headers, and RC additionally drives a real config push through to ACKNOWLEDGED via send_apm_tracing_command(). tests/debugger/test_debugger_agentless.py (DEBUGGER_AGENTLESS): a log probe installed via the agentless native RC client emits a snapshot to the agentless debugger intake; Symbol DB (forced via _DD_SYMBOL_DATABASE_FORCE_UPLOAD) uploads to the same unified path. Both verified end-to-end against a live weblog build of the (unmerged) dd-trace-py bob/agentless-setting branch. Endpoints/shapes are derived from reading tracer/libdatadog source, since that branch isn't released; re-confirm against real captures once it merges.
Scenario and test code is generic (no python-only guards), scoped to python for now via manifest entries only, so dropping these entries is enough once another tracer implements DD_AGENTLESS_ENABLED.
Test_Agentless_Trace_Submission and Test_Agentless_Stats only checked headers and non-empty content, not the actual shape of the trace/stats payloads. Add real assertions on captured payloads (root span for GET /: service, type, error, http.method, http.status_code; matching stats bucket entry: AgentHostname, ClientComputed, Service, Type, Hits, TopLevelHits, Errors), derived from and verified against live-captured traffic against the branch. Verified live: APM_TRACING_AGENTLESS still 3/3 passing with these assertions.
Trace submission, stats, RC, debugger snapshot, and SymDB requests each now assert their full required header set (dd-api-key, content-type, user-agent, datadog-meta-* tracer identity, dd-evp-origin, etc.), not just dd-api-key/content-type as before. Caught a real discrepancy along the way: SymDB uploads share dd-evp-origin: agent-debugger with logs/snapshots/diagnostics rather than a distinct agent-symdb value, and don't send x-datadog-additional-tags at all - fixed the assertion to match confirmed live behavior.
Trace submission now asserts content-encoding: zstd and that the wire size is genuinely smaller than the decoded body, following an audit of the agentless Rust encoder against the real Agent that found compression had been silently disabled (now fixed on the dd-trace-py branch). New Test_Agentless_Stats_Multi_Flush polls for two same-runtime stats flushes and asserts ClientStatsPayload.Sequence increments by exactly 1 per flush - a guarantee agent-mode never gives a real signal for, since the Agent's own re-aggregation always resets Sequence to 0 on relay.
Contributor
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding tests to assert basic functionality of sending stuff to the intake.
Handling traces, stats, RC, debugger and SymDB.