[HOTE-803] feat: Add some improvements to E2E tests#355
Conversation
Lambdas Coverage Report
|
UI Coverage Report
|
There was a problem hiding this comment.
Pull request overview
Updates the Playwright E2E GitHub Actions workflow to better align local CI startup with the repo’s local environment scripts and to surface flaky tests (tests that pass on retry) in CI output.
Changes:
- Switches local startup command from
npm run starttonpm run local:startin the E2E workflow. - Adds a “Detect flaky tests” step that parses Playwright JSON results and emits GitHub warnings.
- Extends the job summary to include flaky test counts and details.
| playwright-${{ runner.os }}- | ||
|
|
||
| - name: "Install Playwright browsers" | ||
| timeout-minutes: 3 |
There was a problem hiding this comment.
The 3-minute step timeout for npx playwright install --with-deps is likely too low on cache-miss runners (apt deps + browser downloads can exceed this), causing avoidable CI failures; increase/remove the timeout or apply it only to the download portion when cache is warm.
| timeout-minutes: 3 |
| FLAKY_TESTS=$(jq -r ' | ||
| [.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | | ||
| if length > 0 then | ||
| .[] | "\(.title) (\(.location.file):\(.location.line))" | ||
| else | ||
| empty | ||
| end | ||
| ' "$RESULTS_FILE" 2>/dev/null) |
There was a problem hiding this comment.
The flaky-test detection suppresses all jq parse/runtime errors (2>/dev/null), which can silently hide a broken/changed Playwright JSON schema and report “No flaky tests” incorrectly; capture jq’s exit code and emit a warning (or fail the step) when parsing fails.
| FLAKY_COUNT=$(jq '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | length' "$RESULTS_FILE" 2>/dev/null || echo 0) | ||
| if [ "$FLAKY_COUNT" -gt 0 ]; then | ||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| echo "### :warning: Flaky Tests ($FLAKY_COUNT)" >> $GITHUB_STEP_SUMMARY | ||
| echo "These tests failed initially but passed on retry:" >> $GITHUB_STEP_SUMMARY | ||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| jq -r '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | .[] | "- **\(.title)** (\(.location.file):\(.location.line))"' "$RESULTS_FILE" 2>/dev/null >> $GITHUB_STEP_SUMMARY |
There was a problem hiding this comment.
In the summary generation, jq ... 2>/dev/null || echo 0 treats JSON parse errors the same as “0 flaky tests”, which can make the job summary misleading; handle jq failures explicitly and surface a warning when the results file can’t be parsed.
| FLAKY_COUNT=$(jq '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | length' "$RESULTS_FILE" 2>/dev/null || echo 0) | |
| if [ "$FLAKY_COUNT" -gt 0 ]; then | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| echo "### :warning: Flaky Tests ($FLAKY_COUNT)" >> $GITHUB_STEP_SUMMARY | |
| echo "These tests failed initially but passed on retry:" >> $GITHUB_STEP_SUMMARY | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| jq -r '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | .[] | "- **\(.title)** (\(.location.file):\(.location.line))"' "$RESULTS_FILE" 2>/dev/null >> $GITHUB_STEP_SUMMARY | |
| if FLAKY_COUNT=$(jq '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | length' "$RESULTS_FILE" 2>/dev/null); then | |
| if [ "$FLAKY_COUNT" -gt 0 ]; then | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| echo "### :warning: Flaky Tests ($FLAKY_COUNT)" >> $GITHUB_STEP_SUMMARY | |
| echo "These tests failed initially but passed on retry:" >> $GITHUB_STEP_SUMMARY | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| if FLAKY_DETAILS=$(jq -r '[.suites[]? | .. | .tests?[]? | select(.status == "flaky")] | .[] | "- **\(.title)** (\(.location.file):\(.location.line))"' "$RESULTS_FILE" 2>/dev/null); then | |
| printf '%s\n' "$FLAKY_DETAILS" >> $GITHUB_STEP_SUMMARY | |
| else | |
| echo ":warning: Could not parse flaky test details from $RESULTS_FILE" >> $GITHUB_STEP_SUMMARY | |
| fi | |
| fi | |
| else | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| echo ":warning: Could not parse flaky test results from $RESULTS_FILE" >> $GITHUB_STEP_SUMMARY |
|
Description
Context
Type of changes
Checklist
Sensitive Information Declaration
To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.