fix: incorrect annotation of rewards

docs/trials_table_mapping.md

@@ -30,6 +30,42 @@ Earlier mapping used `Response.json` (`SoftwareEvents`) for lick times (where
 Lick times now come from the `Behavior/Lickometer` stream, and reward delivery
 times use the Harp valve open times.
 
+### Reward-delivery annotations
+
+Each reward-delivery timestamp carries a label in the series' `data` field:
+
+| Label | Meaning |
+| --- | --- |
+| `earned` | Water the animal worked for: the matched trial has no free water (`is_auto_reward_right` is `None`). |
+| `auto` | Free water: the matched trial has `is_auto_reward_right` set. Scheduled autowater and the anti-bias intervention share that channel and are **not** split here — `auto_waterL` / `auto_waterR` and `anti_bias_left_water` / `anti_bias_right_water` record the mechanism per trial. |
+| `manual` | The delivery is the closest valve opening to a `GiveManualWater` software event for this port. Takes precedence over the other labels, since manual water is not aligned to a go cue. |
+
+Two properties of this series are worth stating explicitly, because both differ
+from "every time the valve opened":
+
+**Deliveries are matched to trials by the `Response` timestamp**, not the
+`TrialOutcome` timestamp. `TrialOutcome` fires at the *end* of a trial, after the
+reward-consumption and ITI periods, so a delivery can sit nearer the *previous*
+trial's outcome and inherit its `is_auto_reward_right`. The valve opens within
+milliseconds of the response, so the response anchors each delivery to its own
+trial. Verified on `864253_2026-08-11_12-52-08`, `864253_2026-08-04_12-54-55`,
+and `864253_2026-08-10_13-09-30`: nearest-`Response` agrees with the trial whose
+`[quiescent_start_time, ITI_start_time)` window contains the delivery on all 859
+valve openings, and every `earned` delivery follows a lick on that same port
+(median 2.8 ms, max 32 ms).
+
+**The series is reward-keyed**: a delivery whose matched trial reports
+`is_rewarded=False` is dropped rather than annotated, so the retained count
+equals the metadata mapper's `sum(is_rewarded)` (248 / 298 / 288 on those three
+sessions, against 261 / 298 / 300 valve openings). Manual water is
+experimenter-driven, unrelated to the trial outcome, and is never dropped. Free
+water is triggered immediately at the go cue and the trial then continues
+normally, so `is_rewarded` reports the outcome of the animal's *own choice* — a
+separate event from the free water. The consequence is that free water delivered
+on a trial that did not pay out is real water the animal received and is **not**
+in this series; the trials table's ungated `auto_water*` /`anti_bias_*` columns
+still record it.
+
 ## Trials Table
 
 Columns are grouped by the raw source they map from.
@@ -61,16 +97,16 @@ Columns are grouped by the raw source they map from.
 
 | Trials column | Mapping |
 | --- | --- |
-| `auto_waterL` / `auto_waterR` | From `is_auto_reward_right`. `1` on the auto-responded side; `0` on the other side, when there was no auto-response (`None`), or when the trial is missing. |
-| `anti_bias_left_water` / `anti_bias_right_water` | Boolean. `True` when the anti-bias algorithm delivered a water intervention to that side — i.e. `trial.metadata.extra.is_bias_water_intervention` is `True` **and** `is_auto_reward_right` points to that side (`False` → left, `True` → right). The anti-bias water uses the same auto-response channel as ordinary autowater, so the `is_bias_water_intervention` flag is what distinguishes it. `False` otherwise. |
+| `auto_waterL` / `auto_waterR` | **Scheduled autowater only**: `1` when `trial.metadata.extra.is_autowater` is `True` **and** `is_auto_reward_right` points to that side. `0` otherwise, including when the trial's free water came from the anti-bias algorithm — that is reported by `anti_bias_left_water` / `anti_bias_right_water`. `is_auto_reward_right` is only the delivery *channel* (free water fired, and to which side); the mechanism comes from the metadata, so the two columns are mutually exclusive. Not gated on `is_rewarded`: the column records what the task did, and free water fires at the go cue regardless of how the animal's own choice resolves. Note this is narrower than the legacy `dynamic-foraging-task` column of the same name, which was the ungated channel ("Autowater given at Left", straight from `B_AutoWaterTrial`) and predates anti-bias water. |
+| `anti_bias_left_water` / `anti_bias_right_water` | Boolean. `True` when the anti-bias algorithm delivered a water intervention to that side — i.e. `trial.metadata.extra.is_bias_water_intervention` is `True` **and** `is_auto_reward_right` points to that side (`False` → left, `True` → right). The anti-bias water uses the same auto-response channel as scheduled autowater, so the `is_bias_water_intervention` flag is what distinguishes it and the two columns are mutually exclusive. `False` otherwise. Like `auto_water*`, **not** gated on `is_rewarded`: these columns record what the algorithm did, and the intervention fires at the go cue regardless of how the animal's own choice resolves. The reward-delivery series *is* reward-keyed, so this column can exceed the series' `auto` count. |
 | `anti_bias_lickspout_movement` | Signed horizontal displacement (mm, positive is rightward) the anti-bias algorithm moved the lickspouts on this trial: `trial.lickspout_offset_delta` when `trial.metadata.extra.is_bias_stage_intervention` is `True`, else `0.0`. |
 | `bait_left` / `bait_right` | Boolean. `bait_right` is `True` if `p_reward_right == 1` and `is_auto_reward_right` is `None` or `False`. `bait_left` is `True` if `p_reward_left == 1` and `is_auto_reward_right` is `None` or `True`. |
 | `response_duration` | `response_deadline_duration`. |
 | `reward_consumption_duration` | `Trial -> reward_consumption_duration`. |
 | `reward_probabilityL` / `reward_probabilityR` | The **block** probability from `Trial -> metadata -> p_reward_left` / `p_reward_right`. The top-level `trial.p_reward_left` / `p_reward_right` is the per-trial probability, not the block probability, so it is not used here. `None` when the trial or its metadata is missing. |
 | `reward_size_left` | `Trial -> reward_size.left` — the reward volume (uL) at the left port. Defaults to `2.0` when not set on the trial. `None` when the trial is missing. |
 | `reward_size_right` | `Trial -> reward_size.right` — the reward volume (uL) at the right port. Defaults to `2.0` when not set on the trial. `None` when the trial is missing. |
-| `rewarded_historyL` / `rewarded_historyR` | **Earned** reward only: filter `is_rewarded == True`, then on `is_right_choice`. `False` on both sides when `is_auto_reward_right` is set (either side) — that trial's water is autowater and is reported by `auto_waterL` / `auto_waterR`. |
+| `rewarded_historyL` / `rewarded_historyR` | **Earned** reward only: filter `is_rewarded == True`, then on `is_right_choice`. `False` on both sides when `is_auto_reward_right` is set (either side) — that trial's water is free water, reported by `auto_waterL` / `auto_waterR` (scheduled autowater) or `anti_bias_left_water` / `anti_bias_right_water` (anti-bias intervention). This matches the `earned` / `auto` split in the reward-delivery series: per side, `rewarded_history*` equals that series' `earned` count exactly. |
 
 ### From `TrialGeneratorSpec.json` (`SoftwareEvents` stream)
 
@@ -191,3 +227,7 @@ These were mapped during exploration but are no longer in scope:
 | 2026-08-06 | Confirmed and documented that the legacy `delay_*` columns describe the acquisition software's **quiescence period**: `delay_start_time` is the `QuiescentPeriod` timestamp (always equal to the new `quiescent_start_time`) and `delay_duration` / `delay_beta` / `delay_min` / `delay_max` summarize `quiescence_period_duration`. `delay_duration` is the *configured* duration — each lick restarts the quiescent period, so the realized `quiescent_stop_time - quiescent_start_time` can be longer. Column descriptions updated accordingly. |
 | 2026-08-12 | `rewarded_historyL` / `rewarded_historyR` now record **earned** reward only: an auto-reward trial (`is_auto_reward_right` set to either side) is `False` on *both* sides, since `TrialOutcome.is_rewarded` is `True` for autowater too and that water is already reported by `auto_waterL` / `auto_waterR`. This matches the `earned` / `automatic` split used for the NWB reward-delivery annotations. |
 | 2026-08-12 | `min_reward_each_block` is now `0` rather than `NULL` when the trial generator exposes no `min_block_reward` — no per-block minimum is a floor of zero, not an unknown. The column is non-nullable (`float`, default `0`). |
+| 2026-08-17 | Reward-delivery annotations now match each delivery to its trial by the `Response` software-event timestamp rather than the `TrialOutcome` timestamp. `TrialOutcome` fires at the *end* of a trial (after the reward-consumption and ITI periods), so a delivery could land nearer the *previous* trial's outcome and inherit its `is_auto_reward_right`, flipping `earned` and `auto`. The valve opens within milliseconds of the response, so the response anchors the delivery to its own trial. Verified 0-mismatch against trial-window containment on 859 valve openings across three sessions. |
+| 2026-08-17 | The reward-delivery series is now reward-keyed: a delivery whose matched trial reports `is_rewarded=False` is dropped rather than annotated (manual water exempt), so the retained count equals the metadata mapper's `sum(is_rewarded)`. Free water fires at the go cue and the trial then continues normally, so `is_rewarded` describes the animal's own choice, not the free water — meaning free water on an unrewarded trial is real water the animal received and is excluded from this series. The trials table's `auto_water*` / `anti_bias_*` columns remain ungated and still record it. |
+| 2026-08-17 | `auto_waterL` / `auto_waterR` now read `trial.metadata.extra.is_autowater` rather than the `is_auto_reward_right` channel, making them **scheduled autowater only** and mutually exclusive with `anti_bias_left_water` / `anti_bias_right_water`. `is_auto_reward_right` says free water fired and on which side but not what kind; the mechanism is in the metadata. Neither column is gated on `is_rewarded`, since both record what the task did. This is narrower than the legacy `dynamic-foraging-task` column of the same name, which was the ungated channel and predates anti-bias water. |
+| 2026-08-17 | The reward-delivery labels stay `earned` / `auto` / `manual`: free water is `auto` whatever mechanism produced it, so the series does not split scheduled autowater from anti-bias water. That split lives in the trials table. Consequence: the series' `auto` count tracks the channel while `auto_waterL` / `auto_waterR` track `is_autowater`, so the two are not expected to be equal. |

src/dynamic_foraging_processing/nwb/acquisition/acquisition_builder.py

@@ -11,7 +11,7 @@
 )
 from dynamic_foraging_processing.nwb.utils import clean_for_nwb
 from dynamic_foraging_processing.raw_data_loader import RawDataLoader
-from dynamic_foraging_processing.utils.rewards import get_annotated_rewards
+from dynamic_foraging_processing.utils.rewards import get_reward_deliveries
 
 
 class LickSource(t.NamedTuple):
@@ -49,8 +49,8 @@ def __init__(self, loader: RawDataLoader):
         """
         self.loader = loader
 
-    def get_reward_delivery(self) -> pd.DataFrame:
-        """Get the reward delivery stream from the dataset.
+    def get_valve_writes(self) -> pd.DataFrame:
+        """Get the raw valve command stream.
 
         Returns
         -------
@@ -76,6 +76,26 @@ def get_trial_outcomes(self) -> pd.DataFrame:
             self.loader.dataset.at("Behavior").at("SoftwareEvents").at("TrialOutcome").load().data
         )
 
+    def get_response_times(self) -> np.ndarray:
+        """Get the per-trial ``Response`` software-event timestamps.
+
+        The event fires when the animal's choice is registered, within
+        milliseconds of the valve opening, so it anchors a reward delivery to
+        its trial. Only the event timestamp is used; the payload's ``Item1``
+        field nominally carries a response time but is unreliable (it can lag
+        the event by thousands of seconds), so it is ignored.
+
+        Returns
+        -------
+        numpy.ndarray
+            The ``Response`` event timestamps, positionally aligned with the
+            ``TrialOutcome`` stream.
+        """
+        responses = (
+            self.loader.dataset.at("Behavior").at("SoftwareEvents").at("Response").load().data
+        )
+        return responses.index.to_numpy()
+
     def get_manual_water_times(self) -> pd.DataFrame:
         """Get the manual-water software-event stream.
 
@@ -172,6 +192,7 @@ def _reward_delivery_series(
         writes: pd.DataFrame,
         trial_outcomes: pd.DataFrame,
         manual_water: pd.DataFrame,
+        response_times: np.ndarray,
         *,
         port_column: str,
         is_right: bool,
@@ -181,8 +202,10 @@ def _reward_delivery_series(
         """Build one lick port's reward-delivery series with reward annotations.
 
         Only valve-open events (``port_column`` is truthy) are reward
-        deliveries; the ``data`` field annotates each as earned, manual, or
-        auto via :func:`get_annotated_rewards`.
+        deliveries; the ``data`` field annotates each as earned, manual, or auto
+        via :func:`get_reward_deliveries`, which also drops deliveries on trials
+        that did not pay out, so the series reports reward rather than every
+        valve opening.
 
         Parameters
         ----------
@@ -193,6 +216,9 @@ def _reward_delivery_series(
         manual_water : pandas.DataFrame
             The ``GiveManualWaterRight`` stream; the ``data`` column selects the
             side (``True`` right, ``False`` left).
+        response_times : numpy.ndarray
+            ``Response`` event timestamps, one per trial, used to match each
+            delivery to its trial.
         port_column : str
             Supply-port column for this side (``"SupplyPort0"`` left,
             ``"SupplyPort1"`` right).
@@ -211,10 +237,11 @@ def _reward_delivery_series(
         open_writes = writes[writes[port_column].fillna(False).astype(bool)]
         delivery_times = open_writes.index.to_numpy()
         manual_water_times = manual_water.index[manual_water["data"] == is_right].to_numpy()
-        annotations = get_annotated_rewards(
+        delivery_times, annotations = get_reward_deliveries(
             delivery_times,
             trial_outcomes,
             manual_water_times,
+            response_times,
         )
         return AcquisitionSeries(
             name=name,
@@ -248,9 +275,10 @@ def build_acquisition(
         list of AcquisitionSeries or AcquisitionTable
             Acquisition entries to write to the NWB acquisition module.
         """
-        rewards = self.get_reward_delivery()
+        rewards = self.get_valve_writes()
         trial_outcomes = self.get_trial_outcomes()
         manual_water = self.get_manual_water_times()
+        response_times = self.get_response_times()
 
         acquisition_streams = self.loader.get_all_raw_data()
         acqusition_streams_descriptions = self.loader.raw_data_stream_descriptions
@@ -274,6 +302,7 @@ def build_acquisition(
                 rewards,
                 trial_outcomes,
                 manual_water,
+                response_times,
                 port_column="SupplyPort0",
                 is_right=False,
                 name="left_reward_delivery_time",
@@ -285,6 +314,7 @@ def build_acquisition(
                 rewards,
                 trial_outcomes,
                 manual_water,
+                response_times,
                 port_column="SupplyPort1",
                 is_right=True,
                 name="right_reward_delivery_time",