Currently, we call reset on every forward call in llms. This causes, especially in multimodal llms (Gemma 4 with multiple audio inputs), that we need to encode everything from the context from scratch. This makes TTFT grows on every call. We need to handle smart strategy to cache already encoded info.
Currently, we call reset on every forward call in llms. This causes, especially in multimodal llms (Gemma 4 with multiple audio inputs), that we need to encode everything from the context from scratch. This makes TTFT grows on every call. We need to handle smart strategy to cache already encoded info.