Skip to content

fix(evals): pass the eval self-tests on native Windows - #4553

Merged
ScriptedAlchemy merged 2 commits into
masterfrom
devin/1792964118-windows-test-portability
Oct 26, 2026
Merged

ScriptedAlchemy merged 2 commits into
masterfrom
devin/1792964118-windows-test-portability

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Native Windows bazel test --config=ci //... on 393edb94 failed four Python test targets. No open PR touches these files (#4543 changes only .bazelrc/MODULE.bazel toolchain pins).

Target Cause Fix
//scripts:ambient_tokio_check_test str(path.relative_to(root)) reported findings as crates\demo\... .as_posix()
//evals/agent_adoption:selftest print/report.md in cp1252 can't encode Δ; ["bash", ...] resolves to System32\bash.exe (the WSL launcher, since CreateProcess searches System32 before PATH); carry: copy is read-only expected 0o400, but on Windows the only mode is the read-only bit (0o444) UTF-8 report + stdout; bash() resolves Git Bash through git; read-only expectation per platform
//scripts:build_eval_binary_test Same WSL bash; Git's bin/bash.exe wrapper puts /usr/bin ahead of the fake uname/bazel; fakes written with CRLF; the script printed MSYS paths (/tmp/...) that native callers can't open The test runs MSYS usr/bin/bash.exe with its dir behind the fakes, pins uname to Linux by default, and writes fakes with \n. build-eval-binary.sh now prints both branches through absolute() (pwd -W on MSYS, else pwd), so re-entry with TRACEDECAY_BIN=<output> returns the same string
//evals/memory:real_model_runner_test The runner owns agents and daemons through os.killpg/setsid (scripts/lib/portable_process.py). Python on Windows has neither, and the test timed out at 300s target_compatible_with marks the test incompatible on Windows, the same way as //evals/hermetic:run_test

On Linux and macOS, pwd -W is an invalid option, so absolute() falls back to pwd and the output does not change.

Verified on Windows Server (Bazel 9.3.1): bazel test --config=ci on the three compatible targets passes. The fourth is reported as incompatible. Each script also passes when run directly with python.

Link to Devin session: https://app.devin.ai/sessions/b7efb5226c9845d097483ac9b2e3d9c3
Open in Devin Desktop: https://app.devin.ai/desktop/session/b7efb5226c9845d097483ac9b2e3d9c3?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

Report ambient-tokio findings with POSIX paths, write the adoption report as UTF-8, run shell fixtures under Git's MSYS bash instead of the System32 WSL launcher, and print build-eval-binary paths in a form native Windows callers can open. The real-model runner owns agents through POSIX process groups, so its test is incompatible on Windows like evals/hermetic:run_test.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: fadecbd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@ScriptedAlchemy
ScriptedAlchemy merged commit 430e447 into master Oct 26, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant