Conversation
| xcodebuild clean test \ | ||
| -project 'ScreenObject.xcodeproj' \ | ||
| -scheme 'ScreenObject' \ | ||
| -destination "platform=iOS Simulator,name=$SIMULATOR_NAME,OS=14.5" |
There was a problem hiding this comment.
I decided to:
- Keep this simple and don't integrate Fastlane just to run the tests, nor
xcprettyorxcbeautifyto format the output. All of those options would require extra dependency and CI setup that doesn't seem granted at this time - Hardcode the iOS version here, rather than pass it to the script, so we have a single source of truth. This makes the syntax a bit less clear in the pipeline, though. Given the scope of this project, I think we can deal with it for the moment. I can also see a future in which we might want to run tests against multiple iOS version, if we get to it, we can start passing the value in the same way as we do with
$SIMULATOR_NAME.
There was a problem hiding this comment.
Same feedback as in the companion XCUITestHelpers PR here
| env: | ||
| IMAGE_ID: xcode-12.5.1 | ||
| agents: | ||
| queue: 'mac' |
There was a problem hiding this comment.
@jkmassel @AliSoftware TIL that Buildkite offers a native way to specify default env and agents. I think we should adopt this approach in our other pipelines instead of the custom anchor for the env value.
I haven't tested this, but I expect that if we define a default env with a FOO variable and then define a step-specific env with a BAR variable, the step will have access to both FOO and BAR.
There was a problem hiding this comment.
I like the pipeline level default and agree it makes sense to use it.
But unlike you my guess would be that any step-specific env would completely override the default. Otherwise there wouldn't be a way to undefine FOO in some steps… so in any case definitively worth checking the behavior in such a case.
There was a problem hiding this comment.
@mokagio Huh, actually this might not behave like either of us expect, as according to the BuildKite documentation, the env set at the top level… always takes precedence over the ones set on each step?!
Top level pipeline environment variables will override what is set in the env attribute of an individual step.
Which is a strange choice from them imho, and iiuc means that we wouldn't be able to provide e.g. a different IMAGE_ID just for a specific step while using the top-level-defined one as a default, because the top-level one would always override them all… 😒
There was a problem hiding this comment.
🤔 I'm guessing that's a typo in the docs. I run a test and the results is inline with my expectation.
There was a problem hiding this comment.
I opened a PR in their docs to suggest an update.
This option works fine for this pipeline, but we should wait to see what Buildkite replies in terms of what the expected behavior is before considering updating our other pipelines.
pachlava
left a comment
pachlava
left a comment
There was a problem hiding this comment.
Same as my comment for Automattic/XCUITestHelpers/pull/4 (minus the error part, which did not appear here). Thank you! 
|
Oh shoot my bad I should have disabled the auto-merge after commenting this… 😓 Gio, feel free to open a subsequent PR if you think my wording nitpick/suggestion is worth it 🙂 |
|
My bad too... Didn't realize my approval would trigger auto-merge, sorry Gio! |
|
No worries @AliSoftware @pachlava 👍 👌 Thank for noticing and caring. I'll followup on the wording next week, as I'm currently on support rotation. |
3 participants
Integrates Buildkite to run the UI tests against iPhone 12 and latest iPad Pro.
We may add more Simulator targets in the future, e.g. if we discover fiddly methods that behave differently across Simulator.
To test, verify that CI run and passed.