feat(go): Add explicit ForceSample marker to the span-kind sampler - #738
Draft
ccschmitz-launchdarkly wants to merge 1 commit into
Draft
feat(go): Add explicit ForceSample marker to the span-kind sampler#738ccschmitz-launchdarkly wants to merge 1 commit into
ccschmitz-launchdarkly wants to merge 1 commit into
Conversation
The dogfood tracer has no way to say "always keep this span" except by setting a SpanKind the sampling map happens to map to 1.0. That coupling caused INC-241: adding the correct Producer kind to kafka.submit raised self-telemetry ~159x, which drove ingest autoscaling into its task cap and exhausted subnet addresses and Postgres connections. Add launchdarkly.sampling.force plus a ForceSample() span start option that the sampler honors for any kind, so callers can express the intent directly and SpanKind can go back to being purely semantic. The marker outranks the unsampled-parent drop from #722: that drop exists to stop a child's kind from silently re-rolling a decision the trace already made, whereas an explicit per-call marker is a deliberate statement. The paths that need it are leaf event spans carrying their payload in their own attributes, where being orphaned is harmless but being dropped loses data. Only a genuine boolean true counts. OTel packs bool and int into the same numeric field, so Value.AsBool() alone would read Int(key, 1) as true and make the marker as easy to trip by accident as the SpanKind it replaces. The attribute is left on the exported span on purpose, so a server-side sampling rule can match it and dial every force-sampled span back down without a deploy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
launchdarkly.sampling.forceand aForceSample()span start option that the Go sampler honors for any span kind, so callers can say "always keep this span" without abusingSpanKindto mean it — the coupling that caused INC-241.Groundwork for O11Y-1880; the consuming change lands in
observability.Two decisions worth a look:
truecounts. OTel packs bool and int into the same numeric field, soValue.AsBool()alone readsInt(key, 1)as true — which would make the marker as easy to trip accidentally as theSpanKindit replaces. A test covers this.The attribute stays on the exported span deliberately, so a server-side sampling rule can match it and dial every force-sampled span down without a deploy.