[ML] Periodically report pytorch_inference RSS

[edsavage]

Summary

Adds a periodic memory reporter to pytorch_inference so Elasticsearch can track the process's real resident set size (RSS) for a trained model deployment, independent of inference traffic.

• A dedicated timer thread emits process stats every 10s (a single RSS read; negligible cost), so memory is observed even when a deployment is idle or between requests.
• Each report carries both the current RSS (memory_rss) and the OS peak / high-water mark (memory_max_rss). The peak is transmitted explicitly rather than derived on the ES side from the sample stream, so transient spikes between reports are not lost — this is the signal used to keep model assignment and adaptive scaling OOM-safe.
• The process-stats document key is renamed from process_stats to stats to match the Elasticsearch PyTorchResult / InferenceProcessStats parser.
• The reporter thread is shut down promptly (condition-variable signalled) before the rest of the process tears down, so it can never write to a closing stream.

Relates #2885

Release-ordering note (important)

ES's InferenceProcessStats parser is strict and treats an unknown field as a fatal parse error for the result stream. Because this PR (re)introduces memory_max_rss, the Elasticsearch-side change that accepts memory_max_rss as an optional field must be merged before this PR is merged into ml-cpp. See the companion Elasticsearch PR.

Merging this PR before the ES-side fix is not simply "safe with the constraint on the version bump" — once this PR lands, the nightly ml-cpp snapshot will start emitting memory_max_rss. The automatic ES version bump PR will pick up that snapshot and run ES integration tests and QAF tests against an ES parser that still rejects unknown fields, causing those tests to fail with a fatal parse error. The version bump PR will be stuck until the ES-side fix lands. To avoid that: merge the ES-side PR first, then merge this one.

Test plan

• CResultWriterTest.testWriteProcessStats updated for the stats schema + memory_max_rss; passes (all 7 CResultWriterTest cases pass).
• Manual: run pytorch_inference with a model and confirm periodic stats docs are emitted ~every 10s with memory_rss and memory_max_rss. Observed {"request_id":"ignore","stats":{"memory_rss":0,"memory_max_rss":93339648}} after one 10s tick on macOS (memory_rss is 0 on macOS — platform-specific; both fields will be non-zero on Linux).
• Confirm prompt shutdown (no stray writes / hangs on exit). Process exits with code 0 cleanly after stdin closes.

Made with Cursor

[edsavage]

buildkite run_serverless_tests ES_SERVERLESS_BRANCH=feature/ml-cpp-repo-override

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @edsavage, I've created a changelog YAML for you.

[edsavage]

buildkite run_serverless_tests ES_SERVERLESS_BRANCH=feature/ml-cpp-repo-override

[edsavage]

buildkite run_serverless_tests

[Copilot]

Pull request overview

Adds periodic current and peak RSS reporting for pytorch_inference, with updated stats serialization and evaluation tooling.

Changes:

• Adds a stoppable 10-second memory reporter thread.
• Renames process-stat output to stats and includes RSS fields.
• Updates tests, evaluator parsing, and changelog documentation.

Reviewed changes

Copilot reviewed 2 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
docs/changelog/3160.yaml	Adds changelog metadata.
docs/CHANGELOG.asciidoc	Adds the release-note entry.
bin/pytorch_inference/unittest/CResultWriterTest.cc	Updates stats serialization expectations.
bin/pytorch_inference/Main.cc	Adds periodic memory reporting and shutdown handling.
bin/pytorch_inference/evaluate.py	Parses and displays updated memory statistics.
bin/pytorch_inference/CResultWriter.h	Documents memory stats output.
bin/pytorch_inference/CResultWriter.cc	Updates the stats output schema.

The periodic stats documents use request_id ignore, but evaluate.py counts them as request-triggered samples, which can misassociate request sizes and cause an IndexError. These samples must be distinguished before incrementing stats_count.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[edsavage]

buildkite build this