Port upstream coroot-node-agent v1.29.0 → v1.36.2

Tracking issue for porting coroot/coroot-node-agent v1.29.0 → v1.36.2 into this fork.

Where we are

Everything up to upstream v1.28.3 is already here. Merge upstream coroot/coroot-node-agent changes #201 brought it in as a squash, so git doesn't know: git merge-base still points at 2025-06-03. That's why a plain git merge upstream/main conflicts everywhere.
49 upstream commits since v1.28.3. Each was cherry-picked on its own onto main and the conflict hunks counted, ignoring the generated ebpftracer/ebpf.go. The counts are in the tables below.
We already have equivalents of two of them:
- coroot/coroot-node-agent@7efe671 (OpenSSL fd capture from syscalls) is fix(ebpf): capture OpenSSL traffic regardless of BIO type and OpenSSL version #346 here, implemented with different BPF maps.
- coroot/coroot-node-agent@91bb874: our Dockerfile already pins snapshot.debian.org.
rustls, Java TLS and inbound L7 can't be cherry-picked. Upstream builds them on 7efe671's maps (ssl_pending, *_last_read_fd). Here they have to be ported onto fix(ebpf): capture OpenSSL traffic regardless of BIO type and OpenSSL version #346's ssl_write_pending / ssl_read_pending / ssl_fds.
Upstream's Windows refactor (coroot/coroot-node-agent@34de3a0) moved files. It split flags/, node/ and gpu/ into *_linux.go / *_windows.go. Later upstream commits that touch flags/flags_linux.go or windows/* need their paths remapped by hand.
The upstream log commits need logparser ≥ v1.4.2. Our logparser fork is 6 upstream commits behind.

How to work through it

One PR per batch, in order.
git cherry-pick -x so every commit message keeps the upstream SHA.
After any .c change, regenerate with cd ebpftracer && make build.
Each PR needs a local end-to-end run, and a rollout to a test node checked for up == 0 and duplicate-series gather errors before the next batch starts.
Compare cumulative counters, not short windows.

Batches

B1: push path and service lifecycle (Go only)

Done when:

Remote-write to a store behind TLS signed by a private CA works with CA_FILE.
An empty or corrupt file in WAL_DIR is discarded and sending resumes.
systemctl restart <unit> keeps the unit's series continuous.

B2: cardinality guards

coroot/coroot-node-agent@33c46ec: cap unique FQDNs per container in container_dns_requests_total; the overflow goes to ~other (4) (fix: cap DNS domain labels per container, add opt-in --min-container-age (B2 of #369) #371)
coroot/coroot-node-agent@34ea61f: --min-container-age (3) (fix: cap DNS domain labels per container, add opt-in --min-container-age (B2 of #369) #371)

Decided: 50 FQDNs. --min-container-age ships at 0 (off), changed from upstream's 30s after review: with 30s, crash-looping pods at the maximum backoff and jobs under 30s never reported.

Done when: the series count per node is measured before and after, and containers living longer than 30s keep all their series.

B3: L7 parser correctness (eBPF)

coroot/coroot-node-agent@2d8fb1b: RabbitMQ/ClickHouse protocol misclassification (5)
coroot/coroot-node-agent@b53b38d: stronger ClickHouse detection (1)
coroot/coroot-node-agent@a8a084f: unbounded allocations and lost queries in the ClickHouse parser (10)
coroot/coroot-node-agent@556154b: bounded offset in the ClickHouse query parser (fixes program load) (2)
coroot/coroot-node-agent@6d5addb: drop L7 events with unknown protocols (4)

Done when:

The programs load on a 5.8 kernel and on a current kernel.
Verified instruction counts stay within limits.
A ClickHouse client and a RabbitMQ client are classified correctly.
L7 drop and unknown counters don't regress over 24h.

B4: uprobe lifecycle (port; overlaps #349, #352, #354)

coroot/coroot-node-agent@f8d437a: deduplicate uprobes (23)
coroot/coroot-node-agent@2fdd10d: uprobe leak from a race between the instrument goroutine and Process.Close (9)
coroot/coroot-node-agent@2c72586: register uprobes before reporting running processes (1)
coroot/coroot-node-agent@44e3e8e: startup delay for Python GIL / Node.js event-loop instrumentation (2)
coroot/coroot-node-agent@0ae7f5a: apply the startup delay to TLS uprobes too (8)

Decided: the TLS attach delay ships at 0. Upstream's 30s would drop the first 30s of every process's TLS traffic, and all of it for processes that live less than 30s. Measure attach CPU with #352 in place before changing that. Upstream uses a single flag for both delays, so keeping Python/Node at 30s with TLS at 0 needs a separate flag; settle that in the PR.

Done when:

The uprobe count on a test node stays flat over 24h.
TLS-attach CPU is no worse than before.
Plaintext captured for long-lived processes doesn't drop.

B5: TLS runtimes (port onto the #346 hooks)

coroot/coroot-node-agent@913e9d4: rustls plaintext capture (7, plus rework onto our maps)
coroot/coroot-node-agent@579eef8: Java TLS through a Java agent + uprobes (11, plus jar and native assets)

The Java agent is injected into running JVMs. It stays opt-in, off by default.

Done when:

A rustls client and a Java HttpClient calling HTTPS produce HTTP metrics with the right status, and spans with paths.
With the Java agent enabled, the JVM stays stable under load.

B6: inbound L7 (port; depends on B3–B5)

coroot/coroot-node-agent@fa6f500: instrument inbound L7 requests (46 hunks, plus the B5 files)

It adds container_*_inbound_* metrics labelled status (plus method for RabbitMQ/NATS). It has to be re-checked against our variable-length ring records (#356) and the HTTP/2 resync and stream-cap changes (#364, #365).

Done when:

A request from an uninstrumented client produces container_http_inbound_requests_total.
With both sides instrumented, server-side MySQL, Redis and Postgres counts match client-side counts within 1%.
Ring-buffer drops don't regress.
The programs still load on 5.8.

B7: logs (logparser fork synced first)

Sync the 6 missing upstream logparser commits into our logparser fork
coroot/coroot-node-agent@cd65ce5: retry opening a container log file that doesn't exist yet (1)
coroot/coroot-node-agent@e4be150: find the log file when the open() fd is already gone (4)
coroot/coroot-node-agent@ab9b049: rate-limit log pattern extraction under log storms (8; drop the windows/ parts)
coroot/coroot-node-agent@e1cd714: extract attributes from JSON logs (6; drop the windows/ parts)
coroot/coroot-node-agent@18cfe3c: OTLP trace context from trace_id/span_id in JSON logs (2)

Done when:

A log storm into one container doesn't push the agent's CPU past the cap.
JSON log lines become OTLP records with attributes and trace context.

B8: process start and boot time

coroot/coroot-node-agent@27bc75d: process start time from /proc/<pid>/stat instead of taskstats (1)
coroot/coroot-node-agent@8c14700 and coroot/coroot-node-agent@a9fd101: read the host boot time once (2 + 3)

Done when: process start times are unchanged and the taskstats calls are gone from the CPU profile.

After the port

Once every commit above is ported or skipped, open a PR with git merge -s ours <upstream tag>. It changes no files; it gives git the right merge base, so the next sync starts from v1.36 instead of 2025-06-03.

Not porting (for now)

Commits	Reason
34de3a0, 481feb0, and the `windows/*` parts of later commits	Windows support is out of scope
e5818fc, e73d36c, b9fc7fd, 675758e, 835ae85, 0c934cd, 8992a5b	JVM async-profiler and Go heap profiling: revisit with the profiling work. The async-profiler is also injected into JVMs
f6813ff	docker → moby client: do it when the docker client deprecation requires it
de67715, ae3e2e6	group AWS/GCP endpoints by FQDN: overlaps the FQDN resolution in #313–#315; compare naming first
f97a3f8	ping timestamping: the pinger is off by default here
80d4530	RHEL 7 / kernel 3.10: our L7 path needs BPF ring buffers (5.8+, #363), so this needs a perf-buffer fallback design. First test whether RHEL 8's backported ring buffer loads our programs
2de271a	removes the Vagrant test setup, which we don't have
7efe671, 91bb874	already covered (#346, Dockerfile)

B1 is in #370. Two changes to the plan:

coroot/coroot-node-agent@1e261c3 is dropped. fix: log each agent message once, and skip untracked sockets before the map lookup #358 already logs each message once.
The createdAt fallback from coroot/coroot-node-agent@75d6656 moves to B2. It only applies with --min-container-age.

75d6656 also turned out to be mostly covered here already: #353 registers processes on exec, listen and connect events. B1 takes the rest of it: registration for every event type, and a locked lookup.

B2 is in #371, stacked on #370.

coroot/coroot-node-agent@33c46ec is re-implemented in L7Stats, because this fork records DNS with destination labels.
coroot/coroot-node-agent@34ea61f ships with the 30s default. It now includes the createdAt fallback held back from 75d6656.
A pid is treated as exited only when /proc/<pid> is gone.

Also found during the e2e (not part of B2): on every build, a burst of new systemd units started within about a minute of the agent starting can go undetected, 1–2 of 10 per burst.

fix: port upstream push-path and container-tracking fixes (B1 of #369) (

B2 change after review: --min-container-age ships with a default of 0 (off), not upstream's 30s. A pod in CrashLoopBackOff at the 5-minute maximum backoff is removed between restarts and comes back as a new container each time, so with 30s it never reported. Short jobs were hidden too. The flag stays available for installs that want it.

fix: cap DNS domain labels per container, add opt-in --min-container-…

B3 is in #377: coroot/coroot-node-agent@a8a084f (new ClickHouse parser), b53b38d and 556154b (stricter eBPF detection), plus regenerated ebpf.go.

Two B3 commits aren't needed here:

2d8fb1b, eBPF half: this fork already classifies every AMQP frame on 5672 as RabbitMQ before the port-gated ClickHouse check.
6d5addb: L7Stats already ignores protocols that have no metrics.

In the e2e, main lost the text of every query that carried settings (empty db.statement); #377 keeps it. The programs load on 5.10 and 6.1.

B4 is split.

B4a is #378.

2c72586 is ported: uprobe programs are registered before anything is attached.
44e3e8e is in with --instrumentation-delay defaulting to 0. The measured saving was about 2.6 millicores at 300 short-lived Python processes a minute, because Node.js tracing is already opt-in here.
2fdd10d isn't needed: fix(tls): count TLS capture losses and fix the gaps they exposed #353 already closes links that arrive after Close.

B4b, still to do: f8d437a (uprobe deduplication) and 0ae7f5a (delay for TLS uprobes). This reworks TLS attach here, so it needs a spec first.

Design:
- Attach each uprobe set once per binary (dev + inode), refcounted per process, with no pid filter. Today every attach carries a pid filter (elf.go, symbol_cache.go).
- go_tls_offsets_map stays keyed by TGID. Each new process of an already-attached binary still needs its entry written, but no attach.
- libssl.so is shared by many binaries, which makes it the biggest win.
- The guarantees from fix(tls): count TLS capture losses and fix the gaps they exposed #353 (attach before the first connect, exec re-check) and feat(tls): attribute TLS losses per container and find sockets behind wrapped Go connections #354 (wrapper walk) must hold.
Measure first: TLS-attach CPU and capture rate, using the short-lived-client scenario (about 390 clients, baseline 388–390 captured) and an attach-share CPU profile on a busy node.
Done when:
- the short-lived capture rate is at or above baseline;
- the number of uprobe links scales with distinct binaries, not processes;
- TLS-attach CPU is lower;
- the link count returns to baseline after process churn (no leaks);
- long-lived TLS capture is unchanged;
- the programs load on 5.10 and 6.1.
0ae7f5a ships with a TLS delay of 0 unless the measurement justifies more.

B8 is in #379: 27bc75d, 8c14700 and a9fd101, plus a test for GetStartTime. Process registration no longer depends on taskstats. At about 100 process starts per second, the agent used 3667 ms of CPU over 60s, against 4251 ms on main.

B7 is in #380, on top of nudgebee/logparser#34, which merges coroot/logparser v1.4.2 into our fork. Merge #34 first, with a merge commit.

All five log commits are in.
JSON logs: exported with the message as the body, the other fields as attributes, and the trace context set.
--log-pattern-extraction-limit defaults to 0. In a 59k-line storm, 100/s saved about 4.6 millicores but hid 93% of the storm under the sampled pattern.
Over-limit messages are still exported, not dropped as upstream does.

Specs for the remaining large batches

Each of these is a re-port onto this fork's eBPF code, not a cherry-pick, so each needs its own design pass.

B5: rustls and Java TLS (913e9d4, 579eef8)

Design

How upstream attaches: both feed plaintext through upstream's own maps (ssl_pending, rustls_last_read_fd, java_tls_last_read_fd), consumed in its syscall read/write path.
What has to change here: this fork's OpenSSL capture (fix(ebpf): capture OpenSSL traffic regardless of BIO type and OpenSSL version #346) uses ssl_write_pending, ssl_read_pending and ssl_fds, correlating the plaintext call with the next socket write or read on the same thread. Both runtimes have to be re-hooked onto those.
rustls: it's sans-IO. Plaintext goes into rustls's buffer, and TLS records are written to the socket later, possibly on another thread under an async runtime such as tokio. Thread correlation has to be checked there, not just assumed.
Java: a Java agent (jar plus native lib) is loaded into running JVMs with jattach. That makes it opt-in (--enable-java-tls, default off), and it needs a security review before the default changes.

Done when

reqwest with rustls, both blocking and tokio, has its HTTPS requests counted with the right status, and its spans carry paths.
A Java HttpClient over HTTPS is captured with the flag on and not loaded with it off.
The JVM stays stable under load with the agent attached.
OpenSSL and Go capture rates don't regress.

B6: inbound (server-side) L7 (fa6f500)

Design

What it adds: server-side parsing for HTTP/1, HTTP/2, gRPC, Postgres, MySQL, Redis, Mongo, Kafka and the other protocols, on connections the process accepted.
New metrics: container_*_inbound_*, labelled status (plus method for RabbitMQ/NATS). They're low-cardinality and separate from the client-side series, so nothing is double-counted.
Order: it builds on B5's TLS hooks for server-side TLS. It has to be fitted into this fork's variable-length ring records (fix(ebpf): size L7 ring buffer records to their content #356), HTTP/2 resync (fix: decode HTTP/2 headers reliably (HPACK, preface detection, duplicate Go TLS writes) #364, fix(l7): resynchronize HTTP/2 parsing after any lost header block (#364 review) #365) and stream cap (fix(l7): track up to 1000 HTTP/2 streams per connection #368).

Done when

For local servers (HTTP/1, HTTP/2, gRPC, Postgres, MySQL, Redis), inbound counts match the client-side counts within 1%.
Outbound counts are unchanged.
Ring-buffer drops are no worse (compared on cumulative counters).
The programs load on 5.8+ and on a RHEL 8 4.18 kernel.
The agent's CPU overhead is measured.
Reference: in a side-by-side run, OBI captured server-side requests that this agent doesn't see today; that run is the baseline to compare against.

fix: port upstream ClickHouse parser and detection fixes (B3 of #369) (…

fix: register uprobe programs before attaching, add opt-in --instrume…

fix: process start time from /proc/<pid>/stat instead of taskstats (B…

fix: port upstream log fixes, JSON log attributes and trace context (…

Navigation Menu

Repository navigation

Port upstream coroot-node-agent v1.29.0 → v1.36.2 #369

Where we are

How to work through it

Batches

B1: push path and service lifecycle (Go only)

B2: cardinality guards

B3: L7 parser correctness (eBPF)

B4: uprobe lifecycle (port; overlaps #349, #352, #354)

B5: TLS runtimes (port onto the #346 hooks)

B6: inbound L7 (port; depends on B3–B5)

B7: logs (logparser fork synced first)

B8: process start and boot time

After the port

Not porting (for now)

Specs for the remaining large batches

B5: rustls and Java TLS (913e9d4, 579eef8)

B6: inbound (server-side) L7 (fa6f500)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Port upstream coroot-node-agent v1.29.0 → v1.36.2 #369

Description

Where we are

How to work through it

Batches

B1: push path and service lifecycle (Go only)

B2: cardinality guards

B3: L7 parser correctness (eBPF)

B4: uprobe lifecycle (port; overlaps #349, #352, #354)

B5: TLS runtimes (port onto the #346 hooks)

B6: inbound L7 (port; depends on B3–B5)

B7: logs (logparser fork synced first)

B8: process start and boot time

After the port

Not porting (for now)

Activity

mayankpande88 commented on Oct 7, 2026

mayankpande88 commented on Oct 7, 2026

mayankpande88 commented on Oct 8, 2026

mayankpande88 commented on Oct 8, 2026

mayankpande88 commented on Oct 8, 2026

mayankpande88 commented on Oct 8, 2026

mayankpande88 commented on Oct 8, 2026

mayankpande88 commented on Oct 8, 2026

Specs for the remaining large batches

B5: rustls and Java TLS (913e9d4, 579eef8)

B6: inbound (server-side) L7 (fa6f500)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions