Finalize recordings asynchronously with stop-time catch-up

.github/workflows/build.yml

@@ -38,10 +38,10 @@ jobs:
       fail-fast: false
       matrix:
         config:
-          - { name: "Ubuntu 22.04", os: ubuntu-22.04 }
-          - { name: "Ubuntu 24.04", os: ubuntu-24.04 }
-          - { name: "macOS", os: macos-14, cmake_extra: '-DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"' }
-          - { name: "Windows", os: windows-latest }
+          - { name: "Ubuntu 22.04", os: ubuntu-22.04, cli: "install/bin/LabRecorderCLI" }
+          - { name: "Ubuntu 24.04", os: ubuntu-24.04, cli: "install/bin/LabRecorderCLI" }
+          - { name: "macOS", os: macos-14, cmake_extra: '-DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"', cli: "install/LabRecorderCLI" }
+          - { name: "Windows", os: windows-latest, cli: "install/LabRecorderCLI.exe" }
 
     steps:
       - name: Checkout
@@ -80,6 +80,7 @@ jobs:
           -DCMAKE_BUILD_TYPE=${{ env.BUILD_TYPE }}
           -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/install
           -DLSL_FETCH_IF_MISSING=ON
+          -DLABRECORDER_BUILD_TESTING=ON
           ${{ matrix.config.cmake_extra }}
 
       # -----------------------------------------------------------------------
@@ -88,6 +89,9 @@ jobs:
       - name: Build
         run: cmake --build build --config ${{ env.BUILD_TYPE }} --parallel
 
+      - name: Test recording finalization
+        run: ctest --test-dir build -C ${{ env.BUILD_TYPE }} --output-on-failure
+
       # -----------------------------------------------------------------------
       # Install
       # -----------------------------------------------------------------------
@@ -97,17 +101,27 @@ jobs:
       # -----------------------------------------------------------------------
       # Test CLI
       # -----------------------------------------------------------------------
-      - name: Test CLI (Linux)
-        if: runner.os == 'Linux'
-        run: ./install/bin/LabRecorderCLI --help || true
+      - name: Test CLI
+        shell: bash
+        run: ./${{ matrix.config.cli }} --help || true
+
+      # -----------------------------------------------------------------------
+      # Integration test: teardown latency and XDF integrity
+      # -----------------------------------------------------------------------
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
 
-      - name: Test CLI (macOS)
-        if: runner.os == 'macOS'
-        run: ./install/LabRecorderCLI --help || true
+      - name: Install integration test dependencies
+        run: python -m pip install --upgrade pip pylsl pyxdf
 
-      - name: Test CLI (Windows)
-        if: runner.os == 'Windows'
-        run: ./install/LabRecorderCLI.exe --help || true
+      # pylsl brings its own liblsl; it only has to speak the same wire protocol as the liblsl
+      # bundled with the recorder, not be the same build. Set PYLSL_LIB here if that ever stops
+      # holding.
+      - name: Test recording teardown
+        shell: bash
+        run: python scripts/test_recording_teardown.py --bin "${{ matrix.config.cli }}"
 
       # -----------------------------------------------------------------------
       # Package

.gitignore

@@ -18,3 +18,4 @@ liblsl.deb
 install-qt.sh
 .DS_Store
 .codegraph/
+__pycache__/

BUILD.md

@@ -92,6 +92,29 @@ The command line install feature does not put build products in the sample place
 If any significant changes are made to the project (such as changing Qt or Visual Stuido version) it is recommended that you delete or rename the build folder and start over. Various partial cleaning processes do not work well.
 
 
+## Recording tests
+
+Enable the deterministic shutdown tests when configuring, then build and run CTest:
+
+```sh
+cmake -S . -B build -DLABRECORDER_BUILD_TESTING=ON
+cmake --build build --config Release
+ctest --test-dir build -C Release --output-on-failure
+```
+
+These tests compile the recording implementation and XDF writer against a controlled inlet.
+They cover delayed clock measurements, stopping during an unavailable measurement, and a
+worker that exceeds the join warning deadline. Completion is checked only after the file is
+closed; stop requests remain nonblocking. A subprocess test verifies bounded CLI exit with a
+worker that never returns, and GUI builds test event-loop responsiveness, stalled-state controls,
+remote status, restart rejection, and deferred window close. The Qt test uses the offscreen platform.
+
+The real-stream integration suite additionally requires `pylsl` and `pyxdf`:
+
+```sh
+python scripts/test_recording_teardown.py --bin /path/to/LabRecorderCLI
+```
+
 ## Linux
 
     * Ubuntu (/Debian)
@@ -122,4 +145,3 @@ If any significant changes are made to the project (such as changing Qt or Visua
 1. You may need to specify additional cmake options.
  . Build everything and copy the files to the `install` folder:
     * `cmake --build . --target install`
-

CMakeLists.txt

@@ -30,6 +30,7 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 # Build Options
 # =============================================================================
 option(LABRECORDER_BUILD_GUI "Build the GUI application (requires Qt6)" ON)
+option(LABRECORDER_BUILD_TESTING "Build deterministic recording tests" OFF)
 
 # =============================================================================
 # LSL Discovery Options
@@ -218,6 +219,49 @@ target_link_libraries(${PROJECT_NAME}CLI PRIVATE
     LSL::lsl
 )
 
+if(LABRECORDER_BUILD_TESTING)
+    enable_testing()
+    add_executable(test_recording_finalization
+        tests/recording_finalization.cpp
+        src/recording.cpp
+    )
+    # Compile the real recording implementation against a controlled inlet. Do not link
+    # liblsl into this target: its timeout behavior is supplied by the test double.
+    target_include_directories(test_recording_finalization PRIVATE tests/fake_lsl src)
+    target_link_libraries(test_recording_finalization PRIVATE xdfwriter Threads::Threads)
+    foreach(scenario IN ITEMS delayed unavailable stalled failed abandoned catchup cutoff clocksync unknown_clock finish_now independent_progress nonadvancing)
+        add_test(NAME recording_finalization_${scenario}
+            COMMAND test_recording_finalization ${scenario})
+        set_tests_properties(recording_finalization_${scenario} PROPERTIES TIMEOUT 20)
+    endforeach()
+    find_package(Python3 REQUIRED COMPONENTS Interpreter)
+    add_executable(test_recording_cli src/clirecorder.cpp src/recording.cpp)
+    target_include_directories(test_recording_cli PRIVATE tests/fake_lsl src)
+    target_link_libraries(test_recording_cli PRIVATE xdfwriter Threads::Threads)
+    add_test(NAME recording_cli_timeout
+        COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/cli_finalization.py
+            $<TARGET_FILE:test_recording_cli>)
+    set_tests_properties(recording_cli_timeout PROPERTIES TIMEOUT 25)
+    if(LABRECORDER_BUILD_GUI)
+        add_executable(test_gui_finalization tests/gui_finalization.cpp tests/gui_recording_stub.cpp
+            src/mainwindow.cpp src/mainwindow.h src/mainwindow.ui src/tcpinterface.cpp src/tcpinterface.h)
+        target_include_directories(test_gui_finalization PRIVATE src)
+        target_link_libraries(test_gui_finalization PRIVATE Qt6::Widgets Qt6::Network LSL::lsl Threads::Threads)
+        if(WIN32)
+            add_custom_command(TARGET test_gui_finalization POST_BUILD
+                COMMAND ${CMAKE_COMMAND} -E copy_if_different
+                    $<TARGET_FILE:LSL::lsl> $<TARGET_FILE_DIR:test_gui_finalization>)
+        endif()
+        add_test(NAME recording_gui_finalization COMMAND test_gui_finalization)
+        add_test(NAME recording_gui_finish_now COMMAND test_gui_finalization --finish-now)
+        add_test(NAME recording_gui_force_exit
+            COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/gui_force_exit.py
+                $<TARGET_FILE:test_gui_finalization>)
+        set_tests_properties(recording_gui_finalization recording_gui_force_exit recording_gui_finish_now PROPERTIES TIMEOUT 25
+            ENVIRONMENT "QT_QPA_PLATFORM=offscreen")
+    endif()
+endif()
+
 # =============================================================================
 # Copy config file to build directory for testing
 # =============================================================================

README.md

@@ -63,9 +63,17 @@ The Block/Task field can be overwriten or selected among a list of items found i
 
 <!--If the checkbox "Enable scripted actions" is checked, then scripted actions that are defined in your current config file will be automatically invoked when you click Start, Stop, or select a block. This check box is by normally unchecked unless you have custom-tailored a configuration to your experiment or experimentation environment.-->
 
-Click "Start" to start a recording. If everything goes well, the status bar will now display the time since you started the recording, and more importantly, the current file size (the number before the kb) will grow slowly. This is a way to check whether you are still in fact recording data. The recording program cannot be closed while you are recording (as a safety measure).
+Click "Start" to start a recording. If everything goes well, the status bar will now display the time since you started the recording, and more importantly, the current file size (the number before the kb) will grow slowly. This is a way to check whether you are still in fact recording data. Closing the window requests Stop and waits for finalization while keeping the interface responsive.
 
-When you are done recording, click the "Stop" button. You can now close the program. See [the xdf repository](https://github.com/sccn/xdf) for tools and information on how to use the XDF files.
+When you are done recording, click **Stop**. This fixes a cutoff in the recorder's clock. Inlets stay subscribed to catch up on delayed data timestamped at or before that cutoff. The interface distinguishes **Catching up**, **Waiting for pre-stop data**, and **Closing recording file**. A new recording cannot start until the file is flushed and closed.
+
+Catch-up continues while pre-stop timestamps advance, even if it takes longer than five seconds. Each stream gets a **two-second inactivity grace**, measured since its last advancing pre-stop timestamp (or the start of catch-up). Samples arriving out of order are accepted during that grace; receiving a post-stop sample does not close the inlet immediately. Silent marker streams, disconnected sources, and non-advancing timestamps therefore cannot hold collection open indefinitely. Data arriving after the grace expires may be missed: neither silence nor crossing the cutoff proves that an outlet's buffers are empty.
+
+Comparison uses an LSL clock-correction estimate, without modifying the timestamps written to XDF or correcting an already clock-synchronized inlet twice. Clock estimates and source timestamps can be imperfect. If no correction is available, arrivals are preserved during a finite grace, potentially including post-stop samples, and the footer records this limitation. Each stream footer's `collection_end` records the recorder-clock `stop_time`, the grace interval, and a reason: `cutoff_observed`, `inactivity_timeout`, `clock_unavailable`, `unreliable_timestamps`, `user_requested`, or `transfer_error`. `cutoff_observed` means a later timestamp was seen and the grace elapsed; it is not a guarantee against arbitrarily delayed or reordered samples.
+
+**Finish now…** lets you end catch-up early and finalize the samples already saved; its confirmation explains that additional data may be missed. If a worker or file operation makes no progress for five seconds, **Force quit…** becomes available. Progress on one stream cannot hide a stalled worker on another. Force quitting may leave an incomplete file or lose buffered samples; it never reports successful finalization. Completed chunks are flushed periodically and footers immediately to improve recovery. **Stopped — file finalized** means the workers finished and the file was flushed and closed; the status also notes streams that ended without observing the cutoff.
+
+`LabRecorderCLI` uses a five-second **inactivity** timeout after Enter, extended by progress rather than elapsed time alone. Use `--stop-timeout SECONDS` before the output filename to change it (greater than zero, at most 3600; values below the two-second grace can interrupt normal catch-up). Exit status **0** means the file finalized, subject to its footer's collection-end reasons; **3** means a worker exceeded the inactivity timeout and the file may be incomplete; **4** means recording or finalization failed. A continuously advancing backlog has no fixed overall deadline. See [the xdf repository](https://github.com/sccn/xdf) for tools and information on how to use the XDF files.
 
 ## Preparing a Full Study
 
@@ -89,9 +97,12 @@ Currently supported commands include:
 * `select none`
 * `start`
 * `stop`
+* `status`
 * `update`
 * `filename ...`
 
+`stop` acknowledges the request with `OK`; this is not a completion notification. Poll `status` for a newline-terminated state: `recording`, `finishing` (initial stop request), `catching_up`, `waiting`, `closing`, `stalled`, `stopped`, or `error`. Wait for `stopped` before restarting or using the file. `start` is rejected with `ERROR <state>` while a recording is active or finalizing.
+
 `filename` is followed by a series of space-delimited options enclosed in curly braces. e.g. {root:C:\root_data_dir}
 * `root` - Sets the root data directory.
 * `template` - sets the File Name / Template. Will unselect BIDS option. May contain wildcards.