fix(rollout): honor per-call args in generate_rollout_async (GenerateState caches first args)

[sdc17]

Summary

generate_rollout_async(args, ...) does not honor its args argument for task dispatch. GenerateState is a global singleton (SingletonMeta), so its __init__ (and therefore self.args) runs only on the first instantiation.GenerateState.submit_generate_tasks then dispatches generate_and_rm_group(self.args, ...) using that cached self.args. Any rollout invoked after the first with a different args object silently runs with the first call's args.

Impact

This occurs when eval and train are invoked with different args, a supported pattern via --eval-function-path.

A concrete example using the framework's own --group-rm:

• The dedicated eval path asserts group RM is off (assert not args.group_rm in eval_rollout and
  eval_rollout_single_dataset). So to evaluate while training with --group-rm, the natural
  approach is a custom --eval-function-path that runs eval on a copy of args with group_rm=False.
• If eval runs before training (a baseline eval), that eval creates the GenerateState singleton with
  group_rm=False. Every subsequent training rollout then reuses the singleton and is dispatched with
  group_rm=False, so group RM is silently disabled for the whole run, with no error, because the
  training side args.group_rm is still True. Running the same config with --skip-eval-before-train
  works, because training creates the singleton first. That contrast is the tell.

More generally, any field a custom eval or rollout function changes on its args is ignored after the first rollout.

Root cause

class GenerateState(metaclass=SingletonMeta):             
    def __init__(self, args):
        self.args = args                       # set once, on first singleton creation                                                                                                                                                          
    def submit_generate_tasks(self, samples):                                                                          
        for group in samples:                                                                                                                                                                                                                   
            ... generate_and_rm_group(self.args, group, ...)   # uses the cached self.args                             

generate_rollout_async(args, ...) calls GenerateState(args), which returns the existing singleton (its __init__ is skipped), then calls submit_generate_tasks, which dispatches with the stale self.args.

Fix

Refresh the singleton's args to the current call's args at the start of each rollout:

state = GenerateState(args)
state.args = args                                                                                                                                                                                                                               

self.args has exactly one consumer, submit_generate_tasks, and it is called only from generate_rollout_async. The eval path (eval_rollout_single_dataset) dispatches generate_and_rm(args, ...) with its own per call args and never reads self.args, so refreshing self.args here covers every consumer.

Reproduction

Train with --group-rm plus a custom --eval-function-path that sets group_rm=False on a copy of args, and run eval before training. Group RM is silently off during training, while the same config with --skip-eval-before-train is correct.

sA = GenerateState(args_A)        # args_A.group_rm = True                                                             
sB = GenerateState(args_B)        # args_B.group_rm = False                                                                                                                                                                                     
assert sA is sB                   # singleton                                                                          
assert sB.args.group_rm is True   # first creation args win; args_B is ignored