Skip to content

feat(turbomind): integrate LMCache for KV cache lookup, store, and retrieve - #4983

Draft
irexyc wants to merge 4 commits into
InternLM:mainfrom
irexyc:lmcache
Draft

irexyc wants to merge 4 commits into
InternLM:mainfrom
irexyc:lmcache

Conversation

@irexyc

@irexyc irexyc commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add optional LMCache multiprocess-server integration through --lmcache-addr and TurbomindEngineConfig.lmcache_addr.
  • Implement asynchronous LOOKUP, chunk-based STORE, and RETRIEVE using ZeroMQ, MessagePack, and CUDA IPC.
  • Support attention KV caches and GDN checkpoints through unified cache layout planning and chunk-aligned checkpoint capture.
  • Integrate retrieval with native prefix caching and scheduler allocation, preserving local cache contents and coordinating completion across TP ranks.
  • Use RAII for transfer pins and lookup-lock ownership, with cancellation, failure fallback, and shutdown draining.
  • Reuse the existing TP communicator and keep cache coordination encapsulated in the LMCache component.
  • Exclude request modes requiring full prompt computation and incompatible dynamic-RoPE cache reuse.

f'moe_a2a_backend={engine_config.moe_a2a_backend}')

model_comm = _tm.TurboMind.create(model_dir='', engine_config=ec)
model_comm = _tm.TurboMind.create(model_dir=model_path, engine_config=ec)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

model_path is passed to lmcache as model_name. Is there a better way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants