Conversation
The DistServe proxy used the OpenAI response id as remote_session_id. That id is the client session_id (default -1) or a chatcmpl-* string, not the Prefill scheduler key, so cache-free logged invalid free and leaked preserved Prefill sessions. Return cache_session_id from Prefill and use it for migration/free. Fixes InternLM#4967 Signed-off-by: GokayAI <60583610+gokay-ai@users.noreply.github.com>
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
DistServe Proxy
/v1/completionscan leak Prefill scheduler sessions until the Prefill ZMQ engine dies.After Prefill, the proxy set
migration_request.remote_session_idfrom the OpenAI responseid. That field is the clientsession_id(default-1), not the internal Prefill scheduler session. Decode then cache-frees-1, Prefill logsinvalid free, and thepreserve_cachesequence stays inscheduler.sessions/seq_manager. Chat has the same mismatch (chatcmpl-*is not an int).Fixes #4967
Modification
/v1/completionsand/v1/chat/completionsreturncache_session_id(engine/scheduler session id) with the DistServe cache fields.remote_session_idand PD shelf/unshelf. Missing or invalid owners are rejected instead of falling back to the publicid.Related: #4977 also plumbs a cache owner id as part of a broader DistServe change. This PR is scoped to the #4967 free-path mismatch only.
BC-breaking (Optional)
Proxy and Prefill servers should be upgraded together: DistServe proxy now requires
cache_session_idfrom Prefill. Public OpenAI responseidis unchanged.Checklist