Skip to content

Adds lazy loading + streaming queue population for minitest-queue#375

Open
danielahrnsbrak wants to merge 1 commit intomainfrom
danielahrnsbrak/lazy-loading-and-streaming
Open

Adds lazy loading + streaming queue population for minitest-queue#375
danielahrnsbrak wants to merge 1 commit intomainfrom
danielahrnsbrak/lazy-loading-and-streaming

Conversation

@danielahrnsbrak
Copy link

Summary

This PR adds lazy loading + streaming queue population for minitest-queue to improve startup behavior and memory profile for large suites, while keeping existing eager mode intact.

What changed

  • Added lazy-load execution flow for Minitest:
    • New lazy queue population strategy and on-demand test resolution.
    • Leader streams discovered tests to Redis in batches while non-leaders begin consuming immediately.
  • Added new queue entry encoding/parsing (test_id + metadata), and updated Redis Lua scripts to operate on entry payloads safely.
  • Added runtime components for lazy mode:
    • ClassResolver
    • FileLoader
    • QueueEntry
    • LazyEntryResolver
    • LazyTestDiscovery
  • Extended Redis worker behavior:
    • stream_populate support
    • streaming state management (streaming/ready)
    • stale-stream detection with timeout handling
    • worker profile recording improvements
  • Extended CLI/config support:
    • --lazy-load
    • --lazy-load-stream-batch-size
    • --lazy-load-stream-timeout
    • --test-files
    • CI_QUEUE_LAZY_LOAD_TEST_HELPERS
  • Documentation updates:
    • Expanded lazy-load docs and usage guidance.
    • Explicitly clarified lazy loading/streaming is currently supported only for minitest-queue (not rspec-queue).

Why

Large test suites pay a high boot cost when every worker eagerly loads all test files.
This change allows:

  • faster time-to-first-test for non-leader workers,
  • reduced per-worker memory pressure,
  • better behavior for very large file lists (--test-files).

Test plan

  • bundle exec rake test
  • Verified no test failures: 195 tests, 642 assertions, 0 failures, 0 errors, 0 skips
  • Verified coverage report generation

Notes

  • Lazy loading and streaming are Minitest-only in this PR.
  • Existing non-lazy execution path remains supported.

@danielahrnsbrak danielahrnsbrak force-pushed the danielahrnsbrak/lazy-loading-and-streaming branch from b4e596a to 8fe8edd Compare February 12, 2026 19:45
@danielahrnsbrak danielahrnsbrak marked this pull request as ready for review February 12, 2026 19:52
@danielahrnsbrak danielahrnsbrak force-pushed the danielahrnsbrak/lazy-loading-and-streaming branch from 8fe8edd to 41ab5d8 Compare February 12, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant