vLLM inference on finetuned model

[projan-shakya]

It seems like you have used your finetuned model for inference in vLLM. When i try my finetuned model, sometimes there is gibberish audio or some of the audio tokens are skipped. Is there any workaround that?

[SebastianBodza]

Yes my model has a different chat template as i accidentally changed it a bit. Overall the standard transformers inference is working correctly?

As i do not have a GPU present, I just threw GPT-5.4 at it you can give the branch https://github.com/SebastianBodza/Orpheus_Distributed_FastAPI/tree/official-setup a try. LMK if that works.

[projan-shakya]

it is still gibberish or skips some words when the audio is generating. The issue seems to persist in every inference engine with fine tuned models, but your finetuned model seems to work fine.

[SebastianBodza]

Then you probably have an issue during training with the chat template setup vs. the inference. You might want to double check your template setup