[Harbor] Support agents beyond terminus 2 such as open hands

[CharlieFRuan]

Currently we only support RL on Terminus2:

SkyRL/examples/train_integrations/harbor/harbor_trial_config/default.yaml

Line 31 in 168b20f

name: terminus-2

We'd want to RL on other agent harness via Harbor as well.

This might include plumbing Harbor and make changes if needed in SkyRL internals.

Need to be especially careful about whether open hands does off policy things that make chat history non-strictly appending (e.g. summarization).

We should support RL on other agents with strictly appending chat history first, and then we can support step-wise training for all agents.

Final deliverable: a curve on perhaps CodeContest, compare against Terminus2

Hardware needed:

• 1xA100 for development (final curve can be run by SkyRL maintainers)
• Modal/Daytona (8 sandbox concurrency should be enough)

[arteemg]

I can take this on!

[MrIridescent]

Hi! I'd like to take a look at this. I'm new to the project but I've fixed similar issues before. I'll submit a PR shortly.

[CharlieFRuan]

@MrIridescent Thank you! Could you work on mini swe agent via Harbor, and @arteemg can look at Open Hands? So that there's no duplicated effort

[CharlieFRuan]

Will close this issue for now. After this PR, we can only support step-wise training for Harbor (crucial for fully async). Harbor only book keeps collect_rollout_details for Terminus 2, and hence that is the only agent harness that can be properly supported.

For more see #1542

In future, we need to do SkyRL-side book keeping in order to support other agent harnesses