Skip to content

Add training-fitted normalization to time-series processors - #1231

Closed
LIU-kaiyu wants to merge 1 commit into
sunlabuiuc:masterfrom
LIU-kaiyu:liu-kaiyu/491-timeseries-normalization
Closed

LIU-kaiyu wants to merge 1 commit into
sunlabuiuc:masterfrom
LIU-kaiyu:liu-kaiyu/491-timeseries-normalization

Conversation

@LIU-kaiyu

Copy link
Copy Markdown

Fixes #491.

Adds optional per-feature z-score normalization to TimeseriesProcessor and TemporalTimeseriesProcessor through normalize_strategy="standard". Normalization remains disabled by default.

The processors incrementally fit the training mean and population standard deviation after resampling and imputation, then apply those fixed statistics to subsequent samples. Each resulting timestep contributes equally; constant features use a scale of 1. Temporal timestamps remain unchanged.

The change also:

  • Validates normalization inputs and requires successful fitting before processing.
  • Clears previous statistics when refitting, including after a failed refit.
  • Preserves fitted statistics through existing serialization and cache fingerprints.
  • Adds documentation and a synthetic example showing patient-level splitting before fitting and explicit reuse of training processors for validation/test data.

Validation: 39 focused tests passed in Pixi, including 20 normalization tests covering defaults, imputation, invalid inputs, refitting, serialization, cache fingerprints, and processor transfer. The synthetic example and local contribution checks passed.

A broader run encountered three dataset-backed schema-test failures during synthetic MIMIC loading because of Windows URL/path handling; those are outside this change.

@Logiquo

Logiquo commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

there is a conflict, could you resolve the merge conflict first? thank you.

@Logiquo Logiquo added the status: conflict There is a merge conflict label Oct 26, 2026
@solarsys

Copy link
Copy Markdown
Collaborator

Closing as superseded by #1337, which added training-fitted zscore and minmax normalization to TimeseriesProcessor and TensorProcessor. TemporalTimeseriesProcessor still has no normalization, and a small follow-up PR adding it the same way would be welcome.

@solarsys solarsys closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: conflict There is a merge conflict

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add normalization support in pyhealth/processors/timeseries_processor.py

3 participants