Different results of configuration evaluation during simulation

[vsfreeman]

Hello!
Thank you very much for your work.

I run the script rt1x_bridge.sh and my results don't match yours.
My results:

• 1 experiment: 
  

1.     PutCarrotOnPlateInScene-v0; Average success: 0.08333333333333333
   

2.     StackGreenCubeOnYellowCubeBakedTexInScene-v0; Average success: 0.0
   

3.     PutSpoonOnTableClothInScene-v0; Average success: 0.041666666666666664
   

4.     PutEggplantInBasketScene-v0; Average success: 0.0
   

• 2 experiment:
  

1.     PutCarrotOnPlateInScene-v0; Average success: 0.041666666666666664
   

2.     StackGreenCubeOnYellowCubeBakedTexInScene-v0; Average success: 0.0
   

3.     PutSpoonOnTableClothInScene-v0; Average success: 0.041666666666666664
   

4.     PutEggplantInBasketScene-v0; Average success: 0.0
   

Your results(from TABLE V of your paper):
Put Carrot on Plate; Success: 0.042
Stack Green Block on Yellow Block; Success: 0.000
Put Spoon on Towel; Success: 0.000
Put Eggplant in Yellow Basket; Success: 0.000

Questions:

1. Why might my results differ from yours?
2. Why do the names of the indicators differ? ( Put Spoon on Towel / PutSpoonOnTable)

[xuanlinli17]

There are some variance. Typically on the Bridge envs one needs to evaluate for at least 75 trials (repeat the evaluation 3 times) and take the average (but rt-1x's output is deterministic so idk why, probably because simulation itself is not deterministic). Default instructions in Simplerenv match the corresponding evals in the real world.

[vsfreeman]

ok, thanks for your answer