[Bug] BlockFileCacheTest flaky: background thread interference and async open timeout

[heguanhui]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

BlockFileCacheTest has two types of flaky failures:

---

Type 1: Background thread interference — ttl_modify failure

The test_file_cache() helper creates a BlockFileCache that starts background threads. These threads asynchronously modify cache state between test assertions, causing file_block->state() to return EMPTY instead of SKIP_CACHE:

[ RUN      ] BlockFileCacheTest.ttl_modify
be/test/io/cache/block_file_cache_test.cpp:447: Failure
Expected: file_block->state() == io::FileBlock::State::SKIP_CACHE
Actual:    EMPTY == SKIP_CACHE

Root cause: the background evict_in_advance thread evicts releasable DOWNLOADED blocks, freeing space so that try_reserve() unexpectedly succeeds, keeping the state as EMPTY instead of transitioning to SKIP_CACHE.

---

Type 2: Background thread interference — io_error failure

Same root cause as Type 1, but manifests as incorrect block count because EMPTY blocks are not removed due to use_count>2 from background thread references:

[ RUN      ] BlockFileCacheTest.io_error
be/test/io/cache/block_file_cache_test.cpp:530: Failure
Expected: mgr.get_file_blocks_num(key) == 9
Actual:    10 == 9

Root cause: when a FileBlocksHolder destructor tries to remove EMPTY blocks (use_count()==2), the background thread or another holder still holds a reference (use_count()>2), preventing removal and leaving the block in the queue.

---

Type 3: Insufficient async open timeout — evict_privilege_order_for_ttl failure

initialize() starts a background disk I/O loading thread that sets _async_open_done=true only on completion. Tests wait only 100ms (100 iterations × 1ms), which is insufficient under high CPU load:

[ RUN      ] BlockFileCacheTest.evict_privilege_order_for_ttl
be/test/io/cache/block_file_cache_test.cpp:6980: Failure
Expected: cache.get_or_set(key1, offset, 100000, context1) to succeed
  (async open not completed, cache not ready)

Root cause: the 100ms total timeout is too short when the system is under load. The background loading thread needs more time to complete disk I/O and set _async_open_done=true.

---

What You Expected?

Tests should be deterministic and not affected by background threads or timing issues.

How to Reproduce?

Run BlockFileCacheTest repeatedly under CPU load. The flaky failures appear intermittently.

Anything Else?

All three are test defects, not business code defects. The fix:

1. For Type 1 & 2: Save and restore config values, set all background thread intervals to 10000000ms during test_file_cache and test_file_cache_memory_storage to prevent background thread interference.
2. For Type 3: Extract wait_for_async_open() helper with 1000 iterations × 10ms (10s total timeout), replace all 91 inline wait loops. The 10ms sleep interval also avoids exacerbating CPU pressure under load compared to the original 1ms interval.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_20b5b0e4212d

I checked the live issue state and current upstream master code for the reported BlockFileCacheTest failures.

Initial assessment:

• This looks like a BE unit-test determinism problem, not evidence of a production cache correctness bug from the current report.
• BlockFileCache::initialize_unlocked() starts background monitor, GC, evict-in-advance, LRU update, dump, and replay threads for each cache instance. The test_file_cache() and test_file_cache_memory_storage() helpers assert exact internal block state/counts while those maintenance threads are live.
• The ttl_modify failure is consistent with the code path: run_background_evict_in_advance() can call try_evict_in_advance(), which removes releasable downloaded blocks through the LRU reservation path and can change the capacity condition that the later SKIP_CACHE assertion depends on.
• The io_error block-count failure also matches the cleanup rule in FileBlocksHolder::~FileBlocksHolder(): an EMPTY or deleting block is removed only when use_count() == 2; any extra live shared pointer during the assertion can leave the block in cache metadata.
• The async-open timeout is also plausible. Filesystem-backed cache storage starts a background loading thread and sets _async_open_done only after the load finishes, while the reported test still uses a short polling loop before asserting readiness.

Current public state:

• PR [fix](be) Fix flaky BlockFileCacheTest by disabling background threads and increasing async open timeout #64190 is already open for this issue. Its overall direction matches the code path: isolate these tests from background maintenance timing and centralize/increase the async-open wait.
• Two details are worth checking during PR review: the async wait helper should fail or return an assertion result on timeout so tests do not continue silently after the longer wait, and temporary global config changes should be restored with an RAII/scope guard so an early ASSERT_* inside a helper cannot leak modified intervals to later tests.

Missing validation:

• Please attach or link a flaky CI job / stress-run output if available, and verify PR [fix](be) Fix flaky BlockFileCacheTest by disabling background threads and increasing async open timeout #64190 with repeated BlockFileCacheTest runs under CPU load.
• At minimum, rerun the directly affected cases (ttl_modify, io_error, and evict_privilege_order_for_ttl) repeatedly in the same build mode where the flake was observed.
• If any flake remains after [fix](be) Fix flaky BlockFileCacheTest by disabling background threads and increasing async open timeout #64190, the useful follow-up evidence would be the full gtest log around the failed case, the exact BE test command, sanitizer/build type, host load, and the file-cache test temp directory state.