feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery#2117
Draft
leoromanovsky wants to merge 3 commits into
Draft
feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery#2117leoromanovsky wants to merge 3 commits into
leoromanovsky wants to merge 3 commits into
Conversation
…ure gate - Add telemetry/flagevaluation.rs with FfeFlagEvaluationBatch and FfeFlagEvaluationEvent types modeled on exposures.rs - Required fields: timestamp, flag.key, first_evaluation, last_evaluation, evaluation_count - Optional fields use skip_serializing_if = Option::is_none for omitempty semantics (reviewer concern #2 review:4477935835) - Context pruning helper prune_context() enforces 256 fields / 256 chars skip-not-truncate (reviewer concern #1 review:4477935835) - New Cargo feature flagevaluation-evp gates the module (parallel to exposure-events) - Gate telemetry module in lib.rs to include flagevaluation-evp feature - 30 tests pass (25 existing + 5 new); OTel evaluation_metrics.rs and exposures.rs are byte-for-byte unchanged
…P flusher - Add ffe_flagevaluation_flusher.rs: structural copy of ffe_exposures_flusher.rs with path constant EVP_FLAGEVALUATIONS_PATH = /evp_proxy/v2/api/v2/flagevaluations and batch type FfeFlagEvaluationBatch; fire-and-forget send_batch with non-2xx warn+drop and timeout via biased tokio::select! - Add SidecarAction::FfeFlagEvaluationBatch variant to mod.rs enum; re-export FfeFlagEvaluationBatch from datadog-ffe telemetry module - Route SidecarAction::FfeFlagEvaluationBatch in sidecar_server.rs to ffe_flagevaluation_flusher::send_batch (parallel to FfeExposureBatch arm) - Add exhaust arm for new variant in telemetry.rs process_actions match - Enable flagevaluation-evp feature on datadog-ffe dep in sidecar Cargo.toml - 40 sidecar tests pass (3 new: posts_to_evp_proxy, non_2xx_does_not_panic, timeout_returns_without_waiting); OTel FfeEvaluationMetric path untouched
The worker->sidecar IPC serializes SidecarAction with bincode (non-self-describing). serde_json::Value (deserialize_any) and #[serde(skip_serializing_if)] both make bincode deserialize fail, so the sidecar silently dropped every FfeFlagEvaluationBatch ('IPC serve: failed to decode request') while the worker enqueue still returned ok.
- Carry pruned context as a JSON-object string (Option<String>); remove all skip_serializing_if from the wire types (keep #[serde(default)] for deserialize).
- Re-expand the context string into a JSON object and strip null/false/empty placeholders in ffe_flagevaluation_flusher::build_payload (POST shape unchanged; degraded tier carries no null placeholders).
- Add enqueue_actions_reliable (checked blocking send + reconnect-retry) for one-shot FFE batches; best-effort enqueue_actions left unchanged for high-volume telemetry.
- Add a bincode round-trip test for FfeFlagEvaluationBatch (mixed Some/None fields) to lock the wire-codec contract.
Contributor
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
|
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Adds the native foundation for server-side EVP
flagevaluationemission consumed bydd-trace-php(part of the cross-SDK flag-evaluation effort that mirrors the Go reference [dd-trace-go#4886]). The PHP tracer aggregates flag evaluations in the native bridge and delivers the batch through the sidecar's EVP proxy path; this PR provides the payload types, the two-tier-aware serialization, and the sidecar flusher + action that make that delivery work — bincode-safe over the worker→sidecar IPC.What's added
datadog-ffe/src/telemetry/flagevaluation.rs— the EVPflagevaluationpayload model (FfeFlagEvaluationBatch/FfeFlagEvaluationEventwith{key}sub-objects,FlagEvalEventContext),prune_context(≤256 fields / ≤256-char string values, oversized skipped not truncated), feature-gated behindflagevaluation-evp.datadog-sidecar—SidecarAction::FfeFlagEvaluationBatch, theffe_flagevaluation_flusher(POST/evp_proxy/v2/api/v2/flagevaluationswithX-Datadog-EVP-Subdomain: event-platform-intake), and a reliable enqueue path for one-shot FFE batches.The wire-codec fix (the substantive part)
The worker→sidecar IPC serializes
SidecarActionwith bincode (datadog-ipccodec). Two serde idioms that are correct for the JSON POST are fatal over bincode and caused the sidecar to silently drop every flag-evaluation batch (IPC serve: failed to decode request, while the worker's enqueue still returnedok):serde_json::Valuefields — bincode is non-self-describing and cannot serviceserde_json::Value'sdeserialize_any.#[serde(skip_serializing_if = …)]— serialize omits the field, but bincode's positional deserialize still expects it, so every subsequent field misaligns.Resolution — decouple the wire from the POST:
FlagEvalEventContext.evaluation: Option<String>), and no field usesskip_serializing_if(all fields always serialized;#[serde(default)]retained for deserialize).build_payload: it re-expands the context string into a JSON object and strips null/false/empty-string/empty-object placeholders (reproducing the oldskip_serializing_ifsemantics so the degraded tier carries no null placeholders).FfeFlagEvaluationBatchwith mixedSome/Nonefields →bincode::deserialize(bincode::serialize(x)) == x) locks this so the incompatibility can't silently return.Reliable one-shot delivery
blocking::enqueue_actionsis best-effort (load-shedding + unchecked send; on reconnect only registered metrics are replayed). FFE batches are low-volume, important, one-shot events, so this addsenqueue_actions_reliable(a checked blocking send wrapped in reconnect-retry-once viawith_retry), used by the PHP RSHUTDOWN flush. The existing best-effortenqueue_actions(relied on by high-volume telemetry) is unchanged.Validation
Proven end-to-end via
dd-trace-phpagainstffe-dogfoodingmock-intake (app-php7): negative-control → green (count 0 → N),context.evaluationarrives as a JSON object, degraded tier has no null placeholders, sidecar logssent flag evaluation batch, status=202. Unit tests: bincode round-trip,build_payloadobject-expansion + placeholder-stripping, flusher send.cargo check/nextestgreen ondatadog-ffe+datadog-sidecar.🚧 Draft — companion to the
dd-trace-phpEVP flagevaluation PR (which bumps the libdatadog submodule to this change).