Introduce mock_trial_data for MockEnv and add some basic Scheduler tests by bpkroth · Pull Request #980 · microsoft/MLOS

bpkroth · 2025-05-19T22:37:20Z

Pull Request

Title

Introduce mock_trial_data and add some basic Scheduler tests

Description

Adds mock_trial_data to MockEnv to allow testing changes to the base class more explicitly.
- Supports sleeping and exceptions and different status returns so that we can check how ParallelTrial will respond in WIP: Parallel Trial Scheduler #971
Adds some very basic Scheduler tests.
- Especially for Scheduler book keeping and basic behavior (e.g., run the trial)

TODO: Split out:

Fix DB Schema for MySQL to store fractional seconds
Type coherce metrics and params as numerics instead of only strings upon retrieval

Type of Change

✨ New feature
🔄 Refactor
🧪 Tests

Testing

Adds new tests and existing ones.

Additional Notes (optional)

Split out from WIP: Prepare Experiment.load to handle async out of order trial completion #973 in order to allow testing changes to base Scheduler class more explicitly
Merge Refactor Status.parse #982 first.

for more information, see https://pre-commit.ci

…cheduler config examples.

for more information, see https://pre-commit.ci

…r-and-tests

…uler-and-tests

…cheduler config examples (#981) # Pull Request ## Title Refactor some test fixtures for better reuse so we can test loading Scheduler config examples ______________________________________________________________________ ## Description - Moves temporary file based sqlite Storage fixture to a reusable location. - Move Optimizer fixtures to reusable location following Storage fixtures pattern. - Uses both in setting up TrialRunner fixtures for testing Scheduler config examples. - Also be sure to load all test configs too. ______________________________________________________________________ ## Type of Change - 🔄 Refactor - 🧪 Tests ______________________________________________________________________ ## Testing - Introduces new CI test for Scheduler configs that follows the usual pattern for other objects (Storage, Optimizer, Environments, Services, etc.) ______________________________________________________________________ ## Additional Notes (optional) - Split out from #980, #973 ______________________________________________________________________ --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>

for more information, see https://pre-commit.ci

mlos_bench/mlos_bench/tests/config/schedulers/test_load_scheduler_config_examples.py

…ler_config_examples.py

…-tests

bpkroth · 2025-05-22T22:37:49Z

mlos_bench/mlos_bench/tests/schedulers/test_scheduler.py

+            trial_data,
+        ), f"Trial {trial_id} was not registered in the optimizer."
+
+    # TODO: And check the intermediary results.


Thinking I will push this part off to #973. Thoughts?

for more information, see https://pre-commit.ci

bpkroth · 2025-05-27T21:45:04Z

mlos_bench/mlos_bench/storage/sql/experiment.py

 from mlos_bench.storage.sql.trial import Trial
 from mlos_bench.tunables.tunable_groups import TunableGroups
-from mlos_bench.util import utcify_timestamp
+from mlos_bench.util import try_parse_val, utcify_timestamp


Work to return numeric values for metrics and configs if possible.

TODO: Split this change out to a separate PR.

bpkroth · 2025-05-27T21:45:48Z

mlos_bench/mlos_bench/storage/sql/schema.py

-            Column("ts_start", DateTime),
-            Column("ts_end", DateTime),
+            Column("ts_start", DateTime().with_variant(mysql.DATETIME(fsp=6), "mysql")),
+            Column("ts_end", DateTime().with_variant(mysql.DATETIME(fsp=6), "mysql")),


Work to allow MySQL to store fraction time.
TODO: Split this out to its own PR

bpkroth · 2025-05-27T21:46:27Z

mlos_bench/mlos_bench/tests/__init__.py

-    "trial_runner_id": None,
+    "experiment_id": "SomeExperimentName",
+    "trial_id": 1,
+    "trial_runner_id": 0,


Changes to handle a non-null trial_id requirement for MockEnv

TODO: Maybe split this out to its own PR too.

# Pull Request ## Title Small refactor to `Status.parse` ______________________________________________________________________ ## Description - Rename `Status.from_dict` -> `Status.parse` - Allow `Status.parse` to accept a `Status` as an argument and return it as is. (complete convenience for some python comprehension code elsewhere) - Add a test for that. ______________________________________________________________________ ## Type of Change - 🔄 Refactor ______________________________________________________________________ ## Testing Small new CI check and existing tests. ______________________________________________________________________ ## Additional Notes (optional) For convenience in another PR - #980 ______________________________________________________________________ --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>

bpkroth added 6 commits May 19, 2025 13:57

Start introducing MockScheduler for better Scheduler testing

cd8b88f

fixup scheduler schemas refactor

2c9c968

reorg optimizer fixtures for reuse

2f4a82e

reorg files based storage fixture

24ccf5a

adding basic scheduler config example loader test

bdfd9b0

Refactor Scheduler schema definitions to make it easier to add new ones.

b66e17a

bpkroth added the WIP Work in progress - do not merge yet label May 19, 2025

pre-commit-ci bot and others added 2 commits May 19, 2025 22:38

[pre-commit.ci] auto fixes from pre-commit.com hooks

30191c0

for more information, see https://pre-commit.ci

Refactor some test fixtures for better reuse so we can test loading S…

bc8dc8f

…cheduler config examples.

bpkroth mentioned this pull request May 19, 2025

Refactor some test fixtures for better reuse so we can test loading Scheduler config examples #981

Merged

pre-commit-ci bot and others added 14 commits May 19, 2025 22:46

[pre-commit.ci] auto fixes from pre-commit.com hooks

ea7b3ff

for more information, see https://pre-commit.ci

fixup

7563b8a

lint

b03e31c

apply suggestions

8eb14c0

revert

8ad4c3d

load test configs too

d4d5153

format

1eb1acb

Merge branch 'refactor/tests' into refactor/mock-scheduler-and-tests

947b759

Merge branch 'refactor/scheduler-schemas' into refactor/mock-schedule…

0c3c805

…r-and-tests

list in __all__ so we load it as a part of tests schemas checking

1112af4

Merge branch 'main' into refactor/mock-scheduler-and-tests

973ad2b

new mock scheduler schema and test configs

7a0d087

refactor to split steps for easier hooking

dded243

Merge remote-tracking branch 'upstream/main' into refactor/mock-sched…

c305196

…uler-and-tests

bpkroth and others added 2 commits May 19, 2025 19:59

Merge branch 'main' into refactor/mock-scheduler-and-tests

b16db23

[pre-commit.ci] auto fixes from pre-commit.com hooks

f1fe022

for more information, see https://pre-commit.ci

bpkroth commented May 20, 2025

View reviewed changes

mlos_bench/mlos_bench/tests/config/schedulers/test_load_scheduler_config_examples.py Outdated Show resolved Hide resolved

bpkroth and others added 2 commits May 19, 2025 20:02

Update mlos_bench/mlos_bench/tests/config/schedulers/test_load_schedu…

83ff70f

…ler_config_examples.py

fixup

71420e6

bpkroth added 2 commits May 22, 2025 17:26

doc tweaks

f5cb468

Merge branch 'refactor/parse-status' into refactor/mock-scheduler-and…

c047bc2

…-tests

bpkroth added ready for review Ready for review and removed WIP Work in progress - do not merge yet labels May 22, 2025

fixup

dc7a7b2

bpkroth changed the title ~~WIP: Introduce mock_trial_data for MockEnv and add some basic Scheduler tests~~ Introduce mock_trial_data for MockEnv and add some basic Scheduler tests May 22, 2025

bpkroth commented May 22, 2025

View reviewed changes

bpkroth and others added 17 commits May 22, 2025 17:40

fixup

76e94ed

fixups

7f8522a

remove old files

9e03e63

add some common globals

f9e3339

[pre-commit.ci] auto fixes from pre-commit.com hooks

a05b5ae

for more information, see https://pre-commit.ci

simplify

94e31b5

rename

21e49db

fixup

46be004

always try and convert metrics back to numerics if possible

01f7a02

wip: tell mysql to use a datetime that can store fractional seconds

b035e57

add more trial data for testing

ca101ef

:type fixups

98ec1cc

check the telemetry

82f602c

[pre-commit.ci] auto fixes from pre-commit.com hooks

176480f

for more information, see https://pre-commit.ci

revert the bulk conversion one

22f4e79

debugging

aa9f728

[pre-commit.ci] auto fixes from pre-commit.com hooks

6289216

for more information, see https://pre-commit.ci

bpkroth commented May 27, 2025

View reviewed changes

bpkroth removed the ready for review Ready for review label Jun 2, 2025

Merge branch 'main' into refactor/mock-scheduler-and-tests

5be025e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce mock_trial_data for MockEnv and add some basic Scheduler tests#980

Introduce mock_trial_data for MockEnv and add some basic Scheduler tests#980
bpkroth wants to merge 84 commits intomicrosoft:mainfrom
bpkroth:refactor/mock-scheduler-and-tests

bpkroth commented May 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

bpkroth May 22, 2025

Uh oh!

bpkroth May 27, 2025

Uh oh!

bpkroth May 27, 2025

Uh oh!

bpkroth May 27, 2025

Uh oh!

bpkroth May 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bpkroth commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Title

Description

Type of Change

Testing

Additional Notes (optional)

Uh oh!

Uh oh!

bpkroth May 22, 2025

Choose a reason for hiding this comment

Uh oh!

bpkroth May 27, 2025

Choose a reason for hiding this comment

Uh oh!

bpkroth May 27, 2025

Choose a reason for hiding this comment

Uh oh!

bpkroth May 27, 2025

Choose a reason for hiding this comment

Uh oh!

bpkroth May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bpkroth commented May 19, 2025 •

edited

Loading