Skip to content

test: force headless Matplotlib backend for local + CI test runs - #197

Open
petercorke wants to merge 1 commit into
rai-opensource:masterfrom
petercorke:fix/headless-tests-locally
Open

test: force headless Matplotlib backend for local + CI test runs#197
petercorke wants to merge 1 commit into
rai-opensource:masterfrom
petercorke:fix/headless-tests-locally

Conversation

@petercorke

Copy link
Copy Markdown
Collaborator

Problem

Running pytest locally pops up real Matplotlib windows, and a couple of
animation tests hang outright. CI already sets MPLBACKEND=Agg in the
workflow env, but that never applied to local runs — several modules
(geom2d, geom3d, spline, animate) import matplotlib.pyplot at
module load time, so whichever backend is active at that point sticks for
the whole session, and locally that's the platform's interactive backend.

Fix

  1. tests/conftest.py (new): os.environ.setdefault("MPLBACKEND", "Agg")
    before anything else runs. conftest.py is guaranteed to load before
    pytest imports any test module, so this closes the gap for local runs
    without touching the CI workflow (redundant there now, not conflicting).
    setdefault(), not a hard override, so MPLBACKEND=MacOSX pytest ...
    still shows a real plot when you actually want to eyeball one.

  2. Forcing Agg everywhere surfaced a real, previously-masked hang:
    tranimate(..., wait=True) busy-loops on plt.pause() waiting for
    FuncAnimation's timer callback to deregister itself — driven by the
    GUI event loop, which Agg doesn't have. Confirmed with a hard-alarm
    timeout: the affected test ate a full 12s budget with zero output vs.
    completing normally under a real backend. These tests were already
    skipped in CI via os.environ.get("CI") == "true", which happened to
    also mask this locally-only-when-Agg-is-forced hang — but only in CI;
    locally (pre-Agg) they ran fine because a real backend genuinely pumps
    the event loop.

    Broadened the three affected skip conditions (test_animate in
    test_transforms3d_plot.py; the two animate=True test_visualize
    tests and the inline guard in test_spline.py) to additionally skip
    when plt.get_backend() == "agg" — the actual root cause, rather than
    the CI-env-var proxy for it. Purely additive, no existing condition
    removed, so CI behaviour is unchanged. Everything else (plain,
    non-animated plotting tests) now safely runs headless both locally and
    in CI — strictly more coverage than before.

Testing

  • Full suite, CI unset (simulating a local run): 339 passed, 3 skipped,
    1.6s, no windows, no hangs.
  • Full suite, CI=true (simulating CI): 338 passed, 4 skipped — matches
    prior CI behaviour.
  • black --check clean at the pinned 23.10.0.

Local `pytest` runs were popping up real windows and, for a few tests,
hanging outright. Two separate problems, one root cause:

1. CI sets MPLBACKEND=Agg via the workflow env, but nothing did that
   for local runs. Several modules (geom2d, geom3d, spline, animate)
   import matplotlib.pyplot at module load time, so whatever backend
   is active when they're first imported sticks for the whole
   session. Locally that's the platform's interactive backend, hence
   the popups.

   Fix: tests/conftest.py sets os.environ.setdefault("MPLBACKEND",
   "Agg") before anything else runs. conftest.py is guaranteed to
   load before pytest imports any test module, so this closes the gap
   for local runs without touching the CI workflow (now redundant
   there, not conflicting). setdefault(), not a hard override, keeps
   an escape hatch: `MPLBACKEND=MacOSX pytest ...` still shows you a
   real plot when you actually want one.

2. Forcing Agg everywhere surfaced a latent bug: Twist/pose animation
   tests use tranimate(..., wait=True), which busy-loops on
   `plt.pause()` waiting for FuncAnimation's timer callback to
   deregister itself. That deregistration is driven by the GUI event
   loop, which Agg doesn't have — so under Agg the loop never exits.
   Confirmed with a hard-alarm timeout: the affected test consumed a
   full 12s budget with zero output, versus completing normally under
   a real backend.

   These tests were already skipped in CI (via an `os.environ.get(
   "CI") == "true"` check) for a related but distinct reason — no
   display in the CI runner. That check happened to also mask this
   hang, but only in CI; locally, before this change, it ran fine
   because a real backend really does pump the event loop. Once Agg
   is forced locally too, the same hang would trigger there.

   Fix: broadened the three affected skip conditions (test_animate in
   test_transforms3d_plot.py; the two animate=True visualize tests
   and the inline guard in test_spline.py) to additionally skip
   whenever plt.get_backend() == "agg", which is the actual root
   cause rather than the CI-env-var proxy for it. Purely additive —
   no existing skip condition was removed, so CI behaviour is
   unchanged. Everything else (plain, non-animated plotting tests)
   now safely runs headless both locally and in CI, which is strictly
   more coverage than before.

Verified: full suite passes locally with CI unset (339 passed, 3
skipped, 1.6s, no windows, no hangs) and with CI=true set (338 passed,
4 skipped, matching prior CI behaviour).
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@petercorke

Copy link
Copy Markdown
Collaborator Author

Merge-order suggestion across the currently open PRs, based on a full file-overlap check across all of them. Nothing here is a conflict with anything already merged — these are parallel PRs sharing some source files with each other, so a rebase will likely be needed wherever paths cross, regardless of order.

Recommended order:

  1. Fixes to base.plot_box() #179 (fix/base-plot-box) — already approved, CI green, ready now
  2. Fix/twist pitch clean #196 (fix/twist-pitch-clean) — isolated to twist.py
  3. test: force headless Matplotlib backend for local + CI test runs #197 (fix/headless-tests-locally) — isolated to test files only (this one)
  4. docs: fix Sphinx docstring formatting and reST syntax errors #195 (docs/sphinx-docstring-fixes) — touches ~18 files, overlaps most of the PRs below; docstring-only/low-risk, so merging it here means the functional PRs below each only rebase against it once
  5. feat(pose,quaternion): add __imatmul__ (@=) operator #199 (feat/imatmul) — only conflicts with refactor: consistent printline() across pose and quaternion classes #200
  6. fix(base): export tr2pos2, pos2tr2, tr2adjoint2; rename tradjoint2 #194 (fix/export-base-functions) — conflicts with perf: defer Matplotlib import until something actually plots #198 and refactor: consistent printline() across pose and quaternion classes #200
  7. perf: defer Matplotlib import until something actually plots #198 (perf/lazy-matplotlib-import) — conflicts with fix(base): export tr2pos2, pos2tr2, tr2adjoint2; rename tradjoint2 #194 and refactor: consistent printline() across pose and quaternion classes #200
  8. refactor: consistent printline() across pose and quaternion classes #200 (refactor/printline) — touches the most shared surface (conflicts with fix(base): export tr2pos2, pos2tr2, tr2adjoint2; rename tradjoint2 #194, perf: defer Matplotlib import until something actually plots #198, feat(pose,quaternion): add __imatmul__ (@=) operator #199); merging it last means it absorbs one final rebase instead of three others rebasing against it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants