SimplerEnv exists to check whether a simulated evaluation predicts a real one. I maintain an Apache-2.0 harness that perturbs the instruction a VLA policy receives and reports the attack success rate with a matched benign arm, and I have the opposite problem: everything I have is simulation, and I have never run on hardware.
So the question is specific. Your correlation results are measured on benign task execution. Is there any reason to expect the same correlation to hold when the failure is driven by a reframed instruction rather than by visual or physical mismatch?
I ask because one family transfers on a real model in our work and the rest do not. A reframed instruction drove a SmolVLA policy off its benign task on 44 of 50 matched pairs across ten libero_object tasks, against 0 benign twins at the same seed, McNemar exact p = 4.6e-13. A harmless reword fired 1 of 50, and nonsense fired 0 of 50. That is one family out of 17. The other 14 have only ever run against a stub.
What I would like to do, if you are open to it: run the instruction-perturbation arm inside SimplerEnv on the tasks where you already have a measured real-world correlation, and report whether the sim-to-real relationship survives the perturbation. If it does not, that is the most useful negative result I could publish, because it would mean simulation-based instruction attacks do not predict real behaviour and everyone doing what I do should say so.
Can you please tell me whether the existing correlation data is at a granularity that would let a second arm be compared against it, or whether that would need a fresh real-world run?
Harness: https://github.com/provael/provael
Sim-to-real status, including the zero: https://www.provael.com/sim-to-real/
SimplerEnv exists to check whether a simulated evaluation predicts a real one. I maintain an Apache-2.0 harness that perturbs the instruction a VLA policy receives and reports the attack success rate with a matched benign arm, and I have the opposite problem: everything I have is simulation, and I have never run on hardware.
So the question is specific. Your correlation results are measured on benign task execution. Is there any reason to expect the same correlation to hold when the failure is driven by a reframed instruction rather than by visual or physical mismatch?
I ask because one family transfers on a real model in our work and the rest do not. A reframed instruction drove a SmolVLA policy off its benign task on 44 of 50 matched pairs across ten libero_object tasks, against 0 benign twins at the same seed, McNemar exact p = 4.6e-13. A harmless reword fired 1 of 50, and nonsense fired 0 of 50. That is one family out of 17. The other 14 have only ever run against a stub.
What I would like to do, if you are open to it: run the instruction-perturbation arm inside SimplerEnv on the tasks where you already have a measured real-world correlation, and report whether the sim-to-real relationship survives the perturbation. If it does not, that is the most useful negative result I could publish, because it would mean simulation-based instruction attacks do not predict real behaviour and everyone doing what I do should say so.
Can you please tell me whether the existing correlation data is at a granularity that would let a second arm be compared against it, or whether that would need a fresh real-world run?
Harness: https://github.com/provael/provael
Sim-to-real status, including the zero: https://www.provael.com/sim-to-real/