WIP: Prepare Experiment.load to handle async out of order trial completion

[bpkroth]

Pull Request

Title

Prepare Experiment.load to handle async out of order trial completion

---

Description

In the case of ParallelTrialScheduler (#971) where Trials may complete out of order relative to what they were submitted in, we can no longer rely only on _last_trial_id to fetch the latest Trial results when registering finished Trial scores with the Optimizer since that may cause us to skip trial_ids that are less than that that haven't completed yet.

This change enables us to handle that situation by doing the following:

1. Track the a set() of registered_trial_ids and allow omitting them from being returned by the Experiment.load() operation.
2. As an optimization (and since that set could otherwise grow large and cause SQL query text construction issues), we also track the first unfinished trial_id in the Experiment. This allows us to remove all trial_ids < first_unfinished_trial_id from the registered_trial_ids set.

• Needs tests.

• Split changes out

  • mock_trial_data - Introduce mock_trial_data for MockEnv and add some basic Scheduler tests #980
  • Scheduler test improvements - Introduce mock_trial_data for MockEnv and add some basic Scheduler tests #980

---

Type of Change

• 🛠️ Bug fix
• ✨ New feature
• 🔄 Refactor
• 🧪 Tests

---

Testing

• New unit tests.
• End to end tests as a part of WIP: Parallel Trial Scheduler #971

---

Additional Notes (optional)

This should have no impact on the SyncScheduler.

• This should be merged after the following PRs:
  • Rename and reorganize some Scheduler methods #975
  • Delay entering TrialRunner context until run_trial #970
  • Add tests for Status type conversion #976

---