feat(turbomind): restore metrics reporting#4768
Open
lvhan028 wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restores and extends TurboMind metrics reporting by wiring scheduler-level cache/sequence metrics and request-level cached-token accounting through the C++ engine, pybind layer, and Python metrics pipeline, with accompanying tests and documentation updates.
Changes:
- Add new TurboMind metrics fields (
cache_usage,prefix_cache_hit_rate,cached_tokens) and expose them to Python. - Rework TurboMind scheduler metrics computation to report live cache-object usage and prefix-cache hit rate.
- Add/extend unit tests for scheduler stats conversion and Prometheus logger recording; update metrics docs (EN/ZH).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_lmdeploy/test_metrics_stats.py | New tests for SchedulerStats cache-usage logic and TurboMind request metrics events/cached_tokens. |
| tests/test_lmdeploy/test_metrics_loggers.py | Adds Prometheus logger test coverage for scheduler metrics gauges. |
| src/turbomind/utils/metrics.h | Extends ScheduleMetrics/RequestMetrics with cache usage and cached-token fields and logging output. |
| src/turbomind/python/bind.cpp | Exposes new metrics fields (cached_tokens, cache_usage, prefix_cache_hit_rate) via pybind. |
| src/turbomind/memory/test_memory.cc | Adds test ensuring allocator stats track live object bytes. |
| src/turbomind/memory/stats.h | Adds live_bytes to MemoryStats. |
| src/turbomind/memory/stats.cc | Updates formatted cache stats output to include live bytes and utilization. |
| src/turbomind/memory/object.cc | Computes live_bytes from slab allocator stats. |
| src/turbomind/engine/request.h | Adds first_schedule_recorded to ensure first-admission observability is counted once. |
| src/turbomind/engine/engine.cc | Implements schedule metrics computation (live objects/bytes, prefix hit rate) and stores per-request cached tokens on first schedule. |
| lmdeploy/turbomind/turbomind.py | Plumbs new TurboMind schedule metrics fields and adds cached_tokens + conditional SCHEDULED event emission. |
| lmdeploy/metrics/stats.py | Uses explicit cache_usage when available, otherwise falls back to block-based usage. |
| lmdeploy/metrics/metrics_processor.py | Removes None guard for schedule metrics updates now that TurboMind reports metrics again. |
| lmdeploy/messages.py | Adds `cache_usage: float |
| docs/zh_cn/advance/metrics.md | Documents TurboMind gpu_cache_usage_perc meaning and prefix-cache hit metric semantics (ZH). |
| docs/en/advance/metrics.md | Documents TurboMind gpu_cache_usage_perc meaning and prefix-cache hit metric semantics (EN). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+947
to
949
| const MemoryStats memory = object_allocator_.Stats(); | ||
| TM_CHECK_LE(active_blocks.size(), memory.live_allocations); | ||
|
|
Comment on lines
+510
to
+514
| const int64_t cached_tokens = std::clamp<int64_t>(c.history_len, 0, c.prompt_len); | ||
| if (!is_warm_up_ && param_.enable_prefix_caching) { | ||
| prefix_query_tokens_ += c.prompt_len; | ||
| prefix_hit_tokens_ += cached_tokens; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.