Read Kubernetes metadata in the background, off the threads analyzing ALS, eBPF access logs and MAL by wu-sheng · Pull Request #14135 · apache/skywalking

wu-sheng · 2026-10-14T02:51:38Z

Fix Kubernetes metadata lookups calling the API server on the threads analyzing the data

Add a unit test to verify that the fix works.
Explain briefly why the bug exists and how to fix it.

Why. Every IP → pod / service / Istio ServiceEntry lookup went through Guava LoadingCaches whose loaders made synchronous Kubernetes REST calls on the caller's thread: a cluster-wide pods?fieldSelector=status.podIP list per IP, pod and service GETs, and the whole-cluster endpoints / services / nodes / ServiceEntry lists. The callers are the gRPC handler threads of Envoy ALS (k8s-mesh) and of the eBPF access log, and MAL (retagByK8sMeta, process naming) through K8sInfoRegistry. Each cache miss, each expireAfterWrite expiry, and each failed call (failures were not cached) repeated the call on that thread. When the API server is slow or unreachable, every access log stream touching an uncached IP holds a handler thread, with the batch it was analyzing, for up to the client timeout and fabric8's retries; gRPC's default executor is unbounded before JDK 25, so threads and retained batches pile up until OAP runs out of memory.

Fix. The API calls, their scopes and the matching logic stay as they were; only where and when they run changes.

A new LookupCache (in library-kubernetes-support) replaces the per-IP / per-pod caches of K8SServiceRegistry, IstioServiceEntryRegistry and K8sInfoRegistry. A key not cached is read once in the background; a key has one pending read at most, whether it is read for the first time or again. The reads are queued to two lanes, each running one read at a time on one background thread (a platform thread that ends once idle before JDK 25, a virtual thread on JDK 25+), so a backlog never turns into a burst of concurrent API requests: K8sLookup-waiting for the first reads MAL waits for, and K8sLookup for every other read, including all reads again. A lookup shares any read already pending for its key, including one in the other lane.
Envoy ALS and the eBPF access log never wait: an IP seen for the first time is unknown, and the lookups after its read find it. An eBPF connection whose addresses were unknown resolves them again on its later messages, which do not carry the connection again, instead of keeping the first, unknown, resolution for the rest of the stream; a resolved address is kept. MAL retagByK8sMeta first queues the reads of the pods of its samples not read yet, then waits for them up to core.default.malMetadataReadTimeout in all, however many pods it has; a cached pod is not waited for, even while it is read again. The waiting of a pod counts from when a step first waited for it within twice the timeout, so the retagByK8sMeta of one Export over the same pods wait that long at most in all, whatever happens to their reads meanwhile (a full lane admitting them later, an entry expiring); its lookups after that never wait. Process naming reads its pod and service the same way. The existing K8sInfoRegistry lookup methods keep their names and signatures.
A read that fails (a 403, a 5xx, the API server not answering or refusing the connection) is told apart from one that finds nothing, and each is retried by its own rule (table below). The cluster lists (endpoints, services, nodes, ServiceEntries) are read on the background thread once three minutes old; a pod not in the endpoints read, or not selected by the services read, may be newer than them, so they are read again for it when older than the not-found retry period. A list read that fails keeps the list read before; with none, the failure is kept, so the reads needing the list fail without reading it. A resource the API server does not serve (ServiceEntries without Istio) is an empty list. After a failure or such a 404, a list is not read again for three minutes, even for a pod not in it. Without the nodes, such as without the permission to list them, no IP is taken for a node's, as before.
The reads use a copy of the shared client without request retries (Client.newClient(RequestConfig), sharing its connections and threads), so a failing read holds its lane for one attempt, up to the client's timeouts, instead of through fabric8's retries and backoff (up to 10 retries).
The HTTP and TCP access log analyzers share one K8SServiceRegistry and one IstioServiceEntryRegistry, as they look up the same IPs with the same configuration.
Failed reads are summarized per read (list pods, get services, ...) as one ERROR line a minute after the first failure and then every five minutes, with a recovery line, instead of a line per failure; a 404 of a resource the API server does not serve is not logged and ends an earlier failure of the same read. Reads not queued as a lane is full are summarized the same way at WARN.
Istio ServiceEntries: read through the shared Kubernetes client instead of a new DefaultIstioClient per read; hostname resolutions use Armeria's default backoff instead of withMaxAttempts(3), which retried without delay once exhausted, and their failures are summarized at WARN instead of a line per attempt; the matching uses sequential streams instead of nested parallelStreams on the common pool, and skips null and IPv6 CIDR addresses that threw.
Pod by IP prefers a pod that has not terminated, as a completed pod keeps its IP after the IP is reused.

Lookup policy. Times are the defaults; keys are in core.default.

	Envoy ALS `k8s-mesh` (and Istio ServiceEntries), eBPF access log	MAL `retagByK8sMeta`, process naming
Key not cached	unknown now; the read is queued in `K8sLookup`; an eBPF connection resolves it again on its later messages	the reads of all the step's keys are queued in `K8sLookup-waiting`, then waited for up to `malMetadataReadTimeout` (2000 ms) in all — the steps of one Export over the same keys together; a key not read by then gets an empty service
Data cached	the data; read again in the background on the first use `k8sMetadataRefreshPeriod` (3 min) after it was written; a read again returning data, the same or not, writes it again	the same
Nothing found (e.g. an IP outside the cluster, a pod newer than the lists read)	unknown; read again on the first use `k8sMetadataNotFoundRetryPeriod` (1 min) after it was written; the endpoints are read again for a pod not in them once older than that	empty service; read again the same way; the services are read again for a pod no service selects once older than that
Read failed (403, 5xx, not answering, refused)	unknown; retried on the first use `k8sMetadataRefreshPeriod` (3 min) after it failed	empty service; the same
Read again of data finds nothing or fails	the data is kept, read again on the first use `k8sMetadataRefreshPeriod` later	the same
Dropped	`k8sMetadataCacheTime` `als` (21 min) after it was last written	`k8sMetadataCacheTime` `mal` (21 min) after it was last written
Lane full (`k8sMetadataReadQueueSize`)	unknown, not cached; queued again by the next lookup	empty service at once, not cached; the same

Configuration. Each has a default in the code, for an application.yml written before it, and is written out in application.yml with the same value.

Key	Env	Default	Meaning
`malMetadataReadTimeout`	`SW_CORE_MAL_METADATA_READ_TIMEOUT`	`2000` (ms)	how long one MAL lookup step, such as one `retagByK8sMeta` over the samples of a metric, waits in all; kept under the 5-second export timeout of the OpenTelemetry Collector's OTLP exporter, which sends the data again when it times out
`k8sMetadataCacheTime`	`SW_CORE_K8S_METADATA_CACHE_TIME`	`als=21,mal=21` (min)	how long an entry is kept after it was last written, per cache; longer than the refresh period
`k8sMetadataRefreshPeriod`	`SW_CORE_K8S_METADATA_REFRESH_PERIOD`	`3` (min)	how long data is used before it is read again, and a failed read waits before it is retried
`k8sMetadataNotFoundRetryPeriod`	`SW_CORE_K8S_METADATA_NOT_FOUND_RETRY_PERIOD`	`1` (min)	how long finding nothing is used before it is read again
`k8sMetadataReadQueueSize`	`SW_CORE_K8S_METADATA_READ_QUEUE_SIZE`	empty, no limit	the most reads waiting at once per lane, as `als=<size>,mal=<size>`

A value not in its form stops OAP at boot with a message showing the form, a bare number (such as 1000) included.

Docs. als_setting.md (a policy section with the table above), mal.md, configuration-vocabulary.md; the storage e2e config-dump.yml expects the new keys.

Tests.

LookupCacheTest: a lookup never waits and the read serves the next lookups; one pending read per key, shared across lanes and across expiry; data used after the refresh period is read again once while lookups keep getting it, and a read again of the same data keeps it alive; an entry expires the cache time after its last write; a read again finding nothing or failing keeps the data and is retried a refresh period later; finding nothing is read again after the not-found retry period, a failed read only after the refresh period, and a read again of an entry without data writes what it finds or that it failed; a full lane refuses a key that a later lookup queues, and a timed lookup whose read is refused returns at once; a waiting lookup does not queue behind the reads nobody waits for; a timed lookup waits at most the timeout since its read was queued; each lane has its own limit; the expiry is longer than the refresh period.
CachedReadTest (read again once older than the caller accepts; a 404 kept as empty; a failed read keeps what was read before; a failed first read kept so callers do not read again), KubernetesApiReadsTest (404 is nothing and not reported; 403 and other failures throw and are summarized; and, against a local HTTP server, a 503 sent once by a client configured with 10 retries), K8sMetadataReadSettingsTest (the als=…,mal=… forms, defaults of a lane left out, the cache time longer than the refresh period, a bare number typed by the config loader rejected with the form, nothing published when a setting is invalid), SerialReaderTest, FailureSummaryLoggerTest.
K8SServiceRegistryTest (the endpoints read again for a pod not in them; a failed read is a failure, not finding nothing) and IstioServiceEntryRegistryTest cover the readers, the registry shared by the analyzers of one configuration, and that a lookup never waits for a read.
Existing tests (K8sTagTest, MALExpressionExecutionTest, LALExpressionExecutionTest, K8SALSServiceMeshHTTPAnalysisTest, FieldFormatterTest, KubernetesCoordinatorTest) pass unchanged.

Verified locally against a real API server (kind, with the API server's audit log recording each request OAP sent):

Envoy ALS never waits: thousands of entries over 5 streams answered in 0.2–0.6 s with the API server healthy, unresponsive (docker pause) or refusing (403), on JDK 11 and 25; HTTP and TCP, k8s-mesh alone and mx-mesh,k8s-mesh; pod and STATIC / CIDR / DNS ServiceEntry names resolved.
One read per key, one request at a time per lane, nothing read again before it is due; a 403 for ServiceEntries is listed once, not once per IP.
A new pod: first lookup 1 s after it started finds it but not in the endpoints read 24 s earlier; the use 66 s later reads the endpoints again and resolves it (before: up to 3–6 minutes).
403: nothing read again at +5 s or at +75 s; read again at +186 s. Failures are one ERROR summary.
MAL with the API server unresponsive: one Export of 50 pods not read yet, through four retagByK8sMeta, took 2.2 s (before: 5 s per pod); the same pods again 0.2 s; healthy, 41 pods not read yet 0.3 s.
With a 1-minute refresh and 2-minute cache time: pods in use were read again at +66 s and +129 s and stayed cached (0.19 s with the API server paused); pods unused for 2.5 minutes had expired (2.2 s, read again).
2000 IPs not read yet over 50 streams with the API server paused, JDK 11: gRPC executor threads followed the 50 streams and did not grow on a second wave; no thread but the two readers in Kubernetes code; back to 2 after recovery.
Every invalid setting, including a bare number for k8sMetadataReadQueueSize or k8sMetadataCacheTime, stops OAP at boot with a message showing the form.
If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the CHANGES log.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

This is still an O(#services) scan for each resolve (including during startup when hasSynced() is… · New
The API/docs here say callers should pass “JSON paths”, but the trimming logic (ResourceTrimmer)… · New
The Javadoc states that if a handler is registered after the store started, it is “first called on… · New
address.contains("/") will throw a NullPointerException if the addresses list contains a null… · New

What changed in this PR

This PR shifts Kubernetes/Istio metadata lookups used by ALS/MAL/LAL from synchronous API-server calls on request threads to watch-maintained local stores, reducing tail latency and preventing thread/memory blowups when the API server is slow/unreachable.

Changes:

Introduces ResourceInformer + trimming/indexing infrastructure in library-kubernetes-support and migrates Pods/Services/Endpoints/Nodes/ServiceEntries to informer-backed stores.
Updates Envoy ALS K8s/Istio registries (and MAL K8sInfoRegistry) to resolve purely from local stores and avoid caching misses before the first sync.
Adds extensive unit tests for informer behavior, trimming correctness, and rule-field parsing; updates docs and CHANGES.

File	Description
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/k8s/ServiceNameFormatterFieldPathsTest.java	Adds tests for extracting rule-read field paths from naming rules.
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/k8s/K8SServiceRegistryTest.java	Verifies K8s registry starts watches with needed fields and miss-caching behavior pre/post sync.
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/istio/IstioServiceEntryRegistryTest.java	Tests ServiceEntry matching (static/DNS) and hostname resolver creation behavior.
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/k8s/ServiceNameFormatter.java	Adds `fieldPathsOf()` extraction to drive store field retention from naming rules.
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/k8s/K8SServiceRegistry.java	Switches service resolution to local stores; starts watches at boot; avoids caching before sync.
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/istio/IstioServiceEntryRegistry.java	Switches to local ServiceEntry store; moves DNS resolver creation off lookup threads; avoids caching before sync.
oap-server/server-library/library-kubernetes-support/src/test/java/org/apache/skywalking/library/kubernetes/ResourceTrimmerTest.java	Tests trimmed copies retain the values naming rules read.
oap-server/server-library/library-kubernetes-support/src/test/java/org/apache/skywalking/library/kubernetes/ResourceInformerTest.java	End-to-end informer/store tests (sync, retries, watch errors, ordering, trimming).
oap-server/server-library/library-kubernetes-support/src/test/java/org/apache/skywalking/library/kubernetes/LookupFieldsTest.java	Validates index values and trimmed-field retention across stores.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/TrimmingItemStore.java	Adds item store wrapper to persist trimmed copies while preserving informer index behavior.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/SharedKubernetesClient.java	Tracks/halts informers at shutdown before closing the shared client.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/ResourceTrimmer.java	Implements mapping from rule bean paths to serialized JSON paths and selective field retention.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/ResourceInformer.java	Adds generic informer-backed local store with retrying start and non-blocking reads.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/KubernetesServices.java	Migrates Services to informer-backed store with trimming and watch start.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/KubernetesPods.java	Migrates Pods to informer-backed store with IP indexing and pod selection logic.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/KubernetesNodes.java	Introduces informer-backed Node address store for node-IP detection.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/KubernetesEndpoints.java	Migrates Endpoints to informer-backed store with ready-address IP indexing.
oap-server/server-library/library-kubernetes-support/src/main/java/org/apache/skywalking/library/kubernetes/IstioServiceEntries.java	Migrates ServiceEntries to informer-backed store and exposes change event handlers.
oap-server/server-library/library-kubernetes-support/pom.xml	Adds Awaitility/beanutils/log bridge test dependencies for new tests.
oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/k8s/K8sInfoRegistry.java	Avoids caching misses before sync and routes lookups to local stores.
docs/en/setup/envoy/als_setting.md	Documents required RBAC and startup “unknown until synced” behavior.
docs/en/concepts-and-designs/mal.md	Documents RBAC and local-store behavior for MAL K8s metadata usage.
docs/en/changes/changes.md	Adds release note describing the metadata lookup/stores change and implications.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

wu-sheng · 2026-10-15T04:35:25Z

Updated lookup rules

This revision changes how the Kubernetes metadata lookups retry, cache and wait. The PR description is updated to match.

What changed

A failed read is no longer cached like "nothing found". A failure (403, 5xx, the API server not answering or refusing the connection) is retried on the first use 3 minutes later. A read that succeeds and matches nothing is read again on the first use 1 minute later, as the key may be new to the API server or to the lists OAP matched it against.
New pods are found within about a minute. A pod found by IP but not in the endpoints read (Envoy ALS), or selected by no service in the services read (MAL), makes the reader read that list again once it is older than 1 minute. Before, the 3-minute lists could hide a new pod for 3–6 minutes.
Cached data lives longer. It is still read again on its first use 3 minutes after it was written, but is now dropped 21 minutes after its last write (7 refresh periods), per cache: als for Envoy ALS, mal for MAL and the eBPF access log. A read again that finds nothing or fails keeps the data and is retried a refresh period later.
MAL waits at most 2 seconds per retagByK8sMeta, not per pod. The step queues the reads of all its pods first, then waits for them within one malMetadataReadTimeout, now 2000 ms by default. A pod's wait counts from when a step first waited for it, so the several retagByK8sMeta of one Export over the same pods wait 2 s at most in all, whatever happens to their reads meanwhile (a full lane admitting them later, an entry expiring). A cached pod is not waited for, even while it is read again. Its lookups after that never wait. That keeps an Export under the OpenTelemetry Collector's 5-second export timeout, after which it sends the data again. Pods not read by then get an empty service for this Export, and are read in the background for the next one.
Failed lists are not read once per IP. A list the API server does not serve (404, e.g. no Istio) is an empty list. A failed list read keeps the list read before; with none, the failure is kept. After either, the list is not read again for 3 minutes, even for a pod not in it. So a 403 on ServiceEntries is read once, not once per new IP.
An eBPF connection resolves its unknown addresses again. The later messages of a connection do not carry it again; a connection whose addresses were unknown now resolves them again on those messages once their metadata is read, instead of staying unknown (or dropped as same-service traffic) for the rest of the stream. A resolved address is kept.
Every setting has a code default and is written out in application.yml with the same value. An invalid value, including a bare number for an als=…,mal=… key, stops OAP at boot with a message showing the form.

Lookup rules (defaults; keys in core.default)

	Envoy ALS `k8s-mesh` (and Istio ServiceEntries), eBPF access log	MAL `retagByK8sMeta`, process naming
Key not cached	unknown now; the read is queued (`K8sLookup` lane); an eBPF connection resolves it again on its later messages	all the step's reads queued first (`K8sLookup-waiting` lane), then waited for up to `malMetadataReadTimeout` (2000 ms) in all, the steps of one Export together
Data cached	the data; read again in the background on the first use `k8sMetadataRefreshPeriod` (3 min) after it was written	the same
Nothing found	unknown; read again on the first use `k8sMetadataNotFoundRetryPeriod` (1 min) after it was written; endpoints read again for a pod not in them once that old	empty service; the same, with the services list
Read failed (403, 5xx, not answering, refused)	unknown; retried on the first use `k8sMetadataRefreshPeriod` (3 min) after it failed	empty service; the same
Read again of data finds nothing or fails	data kept; retried on a use a refresh period later	the same
Dropped	`k8sMetadataCacheTime` `als` (21 min) after the last write	`k8sMetadataCacheTime` `mal` (21 min) after the last write
Lane full (`k8sMetadataReadQueueSize`)	unknown, not cached; queued again by the next lookup	empty service at once; the same

Settings

Key	Env	Default
`malMetadataReadTimeout`	`SW_CORE_MAL_METADATA_READ_TIMEOUT`	`2000` ms
`k8sMetadataCacheTime`	`SW_CORE_K8S_METADATA_CACHE_TIME`	`als=21,mal=21` minutes
`k8sMetadataRefreshPeriod`	`SW_CORE_K8S_METADATA_REFRESH_PERIOD`	`3` minutes
`k8sMetadataNotFoundRetryPeriod`	`SW_CORE_K8S_METADATA_NOT_FOUND_RETRY_PERIOD`	`1` minute
`k8sMetadataReadQueueSize`	`SW_CORE_K8S_METADATA_READ_QUEUE_SIZE`	empty (no limit)

Two reader threads serve all of it, one per lane. Each runs one read at a time, is created on demand and ends after 60 s idle (a virtual thread on JDK 25+). Refreshes are triggered by use, not by a timer.

Verified against a real API server (kind, with the API server's audit log as the record of each request OAP sent):

New pod: a new pod's IP was first looked up 1 s after the pod started, and found nothing, as the endpoints read 24 s earlier lacked it. The use 66 s later read the endpoints again and resolved it.
403: nothing was read again at +5 s or +75 s; everything was read again at +186 s. ServiceEntries were listed once, not once per IP.
MAL with the API server unresponsive: one Export of 50 unread pods, through four retagByK8sMeta, took 2.2 s; the same pods again took 0.2 s; 41 cached pods took 0.2 s. With mal=1, 6 unread pods took 2.3 s.
Short cache (1-minute refresh, 2-minute cache time): pods in use were read again at +66 s and +129 s and stayed cached. Pods unused for 2.5 minutes had expired.
Envoy ALS never waited: 0.2–0.6 s per batch, healthy, paused or 403, on JDK 11 and 25.
Load with the API server unresponsive: 2000 unread IPs over 50 streams, JDK 11. No thread grew; only the two readers were in Kubernetes code.
Boot: every invalid setting stopped OAP with a message showing the expected form.

… the data The lookups of pods, services, endpoints, nodes and Istio ServiceEntries behind Envoy ALS k8s-mesh, the eBPF access log and MAL made synchronous Kubernetes API calls on the gRPC handler threads, on every cache miss, cache expiry and failed call. A slow or unreachable API server held one handler thread, with the batch it was analyzing, per access log stream, until OAP ran out of memory. The same API calls are now made in the background, one at a time in each of two lanes (the first reads MAL waits for, and the rest), and only for what the lookups use. Envoy ALS and the eBPF access log never wait: an IP seen for the first time is unknown while it is read, and is found by the later lookups; an eBPF connection resolves its unknown addresses again on its later messages. A MAL retagByK8sMeta queues the reads of its pods first and waits for them up to core.default.malMetadataReadTimeout (2 seconds) in all; its lookups do not wait. Data is read again when used core.default.k8sMetadataRefreshPeriod (3 minutes) after it was written, and dropped core.default.k8sMetadataCacheTime (als=21,mal=21 minutes) after its last write. Finding nothing is read again when used core.default.k8sMetadataNotFoundRetryPeriod (1 minute) later, and the endpoints or services read are read again for a pod not in them once that old, as the pod may be new. A failed read, such as for a 403 or the API server not answering, is retried when used the refresh period later and keeps the data read before. The reads waiting at once are of distinct keys and can be limited per lane by core.default.k8sMetadataReadQueueSize (als=<size>,mal=<size>). The reads are sent without the client's retries, so a failing read holds its lane for one attempt, and the HTTP and TCP access log analyzers share their lookups. Failed and refused reads are logged as periodic summaries instead of a line per failure, and a resource the API server does not serve is not logged. Istio ServiceEntry hostname resolutions retry with a growing backoff instead of immediately, and ServiceEntries are read through the shared Kubernetes client.

wu-sheng requested a balanced review from Copilot October 14, 2026 02:51

Copilot AI reviewed Oct 14, 2026

View reviewed changes

wu-sheng added this to the 11.1.0 milestone Oct 14, 2026

wu-sheng added backend OAP backend related. enhancement Enhancement on performance or codes labels Oct 14, 2026

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from 2b54561 to 6dc25d2 Compare October 14, 2026 03:23

Copilot stopped reviewing on behalf of wu-sheng due to an error October 14, 2026 04:06
The job was not acquired by a runner even after multiple attempts.

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from 6dc25d2 to 3462bb8 Compare October 14, 2026 06:04

wu-sheng requested a review from kezhenxu94 October 14, 2026 06:32

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from 3462bb8 to d35a164 Compare October 14, 2026 08:59

wu-sheng changed the title ~~Serve Kubernetes metadata from watch-maintained local stores instead of API calls on gRPC threads~~ Read Kubernetes metadata in the background, off the threads analyzing ALS, eBPF access logs and MAL Oct 14, 2026

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch 2 times, most recently from c39a431 to fc68169 Compare October 14, 2026 10:23

kezhenxu94 previously approved these changes Oct 14, 2026

View reviewed changes

wu-sheng dismissed kezhenxu94’s stale review via e4d8253 October 14, 2026 23:57

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from ef6bf9a to e4d8253 Compare October 14, 2026 23:57

kezhenxu94 previously approved these changes Oct 15, 2026

View reviewed changes

wu-sheng dismissed kezhenxu94’s stale review via 735aaae October 15, 2026 04:35

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from e4d8253 to 735aaae Compare October 15, 2026 04:35

Dev-next-gen mentioned this pull request Oct 15, 2026

Fix ES metrics reads locating indices with superDatasetDayStep instead of dayStep #14139

Merged

3 of 4 tasks

kezhenxu94 previously approved these changes Oct 15, 2026

View reviewed changes

wu-sheng dismissed kezhenxu94’s stale review via 70f22a4 October 15, 2026 05:50

wu-sheng force-pushed the fix/k8s-metadata-watch-index branch from 735aaae to 70f22a4 Compare October 15, 2026 05:50

kezhenxu94 approved these changes Oct 15, 2026

View reviewed changes

wu-sheng merged commit d13cf73 into master Oct 15, 2026
479 of 481 checks passed

wu-sheng deleted the fix/k8s-metadata-watch-index branch October 15, 2026 06:45

Navigation Menu

Repository navigation

Read Kubernetes metadata in the background, off the threads analyzing ALS, eBPF access logs and MAL - #14135

Read Kubernetes metadata in the background, off the threads analyzing ALS, eBPF access logs and MAL#14135
wu-sheng merged 1 commit into
masterfrom
fix/k8s-metadata-watch-index

wu-sheng commented Oct 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wu-sheng commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wu-sheng commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix Kubernetes metadata lookups calling the API server on the threads analyzing the data

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wu-sheng commented Oct 15, 2026

Updated lookup rules

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wu-sheng commented Oct 14, 2026 •

edited

Loading