Significant performance discrepancy on LVBench reproduction #3
Description
First of all, thank you for your impressive work and for open-sourcing the code.
I am currently trying to reproduce the results reported in your paper, specifically the performance on the LVBench dataset.
Following the instructions in the README and strictly following the default settings, I evaluated the model using the provided checkpoints stage3. However, I observed a significant gap between my reproduction results and the scores reported in the paper:
Reported Score:
Overall 48
My Reproduction:
Overall 26.0813
cartoon 24.3816
documentary 24.8731
live 24.0458
selfmedia 26.183
sport 32.9167
tv 24.4
My environment: pytorch: 2.7.1+cu118, 4*A100 GPU
Could you please clarify if there are any specific settings, prompt templates, or post-processing steps used for the LVBench evaluation that might not be documented in the README? Or could you verify if the evaluation script requires specific arguments to match the paper's setting?