v0.1.0: monorepo merge — unified training + inference, canonical model, archive-free inference, vendored baselines

[xuefei-wang]

Summary

Merges the separate training repository (deepcelltypes-cell-type-assignment-pytorch)
into this repo and replaces the legacy CellTypeCLIPModel inference path with the
current canonical model. This is the v0.1.0 release cut.

Before this PR, vanvalenlab/deepcell-types was inference-only — it shipped
CellTypeCLIPModel, the dct_kit/ helpers, and a top-level __init__ that
exported just predict. After it, a single package covers training and
inference: inference stays a plain pip install deepcell-types, the full
training pipeline lives behind a [train] extra, and the four paper comparison
baselines are vendored behind per-baseline extras.

⚠️ Breaking changes — see below.

Canonical model

model.py is rewritten around CellTypeAnnotator; CellTypeCLIPModel /
CellTypeDataEncoder are removed. Canonical training defaults (scripts/train.py,
click-based CLI): --resnet_channels 48, --domain_weight 0.1,
--best_metric macro_f1.

• Mean-intensity injection — per-cell mean marker intensity is scattered
  into a marker-position vector and injected as a CLS residual. The output
  projection is zero-init, so warm-starting from a checkpoint preserves
  predictions at step 0.
• DANN domain adaptation via a gradient-reversal head, on by default
  (--domain_weight 0.1; 0 disables it).
• Adapter-style fine-tuning: --freeze_backbone trains only the
  mean-intensity branches on top of an existing checkpoint; --unfreeze_ct_head
  additionally co-adapts the CT head / CLS token / final norm without unfreezing
  the transformer backbone.
• Padding-channel positions are explicitly zeroed (masked_fill) through the
  channel encoder, fusion, and mean-intensity paths so masked tokens contribute
  exactly zero rather than leaking bias/spatial_feat into the transformer.
• Self-describing checkpoints: scripts/train.py bundles ct2idx, n_heads,
  and compat_marker0_zero into the checkpoint, and inference asserts the
  vocabulary ordering matches (a permuted vocabulary previously passed the
  count-only check and silently mislabeled cells).

Canonical-only inference

• Archive-free by default: the marker / cell-type registry ships as a small
  packaged vocab.json snapshot, so pip install deepcell-types +
  download_model() is enough to run predict() — the multi-GB TissueNet zarr
  archive is no longer required (pass zarr_path= / set
  DEEPCELL_TYPES_ZARR_PATH only if you need it). Verified identical
  predictions with vs. without the archive on the paper checkpoint.
• Post-hoc abstention on by default (ct_abstention_k=0.2), bucketed
  per-FOV everywhere (CLI, Python API, library): cells below an IQR fence on
  the FOV confidence distribution are relabeled to the "Unknown" sentinel
  (skipped when k is disabled or the FOV has <4 cells).
• Custom preprocessing hook: predict(..., preprocess=...) overrides the
  per-FOV normalization without retraining, backed by a bounded op library
  (apply_config, make_preprocessor, DEFAULT_CONFIG) and a
  composition-guided adaptation loop (skills/preproc-adapt/).
• The bright-spot clip percentile (DCTConfig.PERCENTILE_THRESHOLD) is now
  99.9, matching the recipe the training archive was built with (was 99.0,
  a carryover from the original packaging).
• predict(return_probabilities=True) returns a PredictionResult dataclass
  with the full per-cell softmax matrix, cell indices, and the pre-abstention
  argmax labels (cell_types_raw).
• _torch_load_weights loads with weights_only=True and emits a loud warning
  if it has to fall back to unsafe pickle on an older torch; a missing
  checkpoint raises a clear FileNotFoundError pointing at download_model().

New public API

• predict, DCTConfig, PredictionResult, preprocess_fov, apply_config,
  make_preprocessor, and DEFAULT_CONFIG are importable from deepcell_types
  directly. preprocess_fov(raw, mask, native_mpp, channel_names) → PreprocessedFov is the standalone preprocessing entry point.

Monorepo: training pipeline

• deepcell_types.training ships from this repo behind pip install "deepcell-types[train]": config.py, dataset.py, archive.py,
  annotations.py, baseline_features.py, gold_metadata.py, losses.py,
  metrics.py, patch.py, utils.py, abstention.py.
• Scripts under scripts/: train.py, pretrain.py, predict.py,
  generate_openai_embeddings.py, generate_splits.py, split_val_for_test.py,
  plus the release-archive gate (validate_archive_contract.py,
  check_release_archive.sh).
• Canonical split manifests committed under splits/
  (fov_split{,_valsubset,_test}.json + README), so the published
  train/val/test partition is reproducible from the repo.
• Experiment logging is plain Python logging — no Weights & Biases dependency
  anywhere (--enable_wandb is gone; confusion matrices save locally as PNGs).
• zarr>=3.1 pulls the Python floor up to 3.11 for the train extra.

Baselines

• Four paper comparison baselines vendored under deepcell_types/baselines/
  (cellsighter, maps, nimbus, xgb), invoked through the unified runner
  python -m deepcell_types.baselines <name>, each with a self-contained
  install extra (baseline-cellsighter, baseline-maps, baseline-nimbus,
  baseline-xgboost).
• Each baseline ships a README documenting every deviation from its upstream
  source; third-party licenses are tracked in deepcell_types/baselines/NOTICE.
• extract_features_from_zarr(missing_value=...) lets each baseline choose its
  absent-marker sentinel: MAPS / CellSighter keep 0.0; XGBoost can pass
  np.nan so absent markers route through XGBoost's learned missing direction
  instead of being conflated with "present, intensity 0.0". The feature matrix
  records a present_markers mask and the cache stays missing-value-agnostic.

Breaking changes

• CellTypeCLIPModel removed. No shim — use from deepcell_types import predict, DCTConfig.
• All predict() arguments after mpp are keyword-only, preventing
  accidental transposition of the adjacent string arguments. device= is the
  preferred spelling (device_num= remains a deprecated alias).
• predict(num_workers=...) default is now 0 (was 24) — 24 workers
  OOM'd machines with <64 GB RAM.
• Abstention on by default changes returned labels vs. the unfiltered argmax
  of prior releases; pass ct_abstention_k=0 to recover raw argmax.
• Clip percentile 99.0 → 99.9 shifts ~5% of predicted labels; on a
  held-out test-split sample it reproduces the canonical predictions slightly
  better (92.5% vs 91.9% argmax agreement).

Packaging / infra

• Package data now ships vocab.json, channel_mapping.yaml, and
  training/config/*.yaml (incl. combined_celltypes.yaml), which were
  previously outside the package tree and absent after pip install.
• tifffile declared in the [train] extra.
• CI workflow added (.github/workflows/ci.yml); inference vs. [train] test
  boundary enforced.
• LICENSE text matches the OSI Apache 2.0 text exactly (LIC: Revert licence text to exactly match OSI Apache 2 #42); NOTICE
  aligned to the vanvalenlab convention.

Tests

35 test modules under tests/ (plus tests/baselines/) covering canonical
inference, abstention CLI, checkpoint round-trip, dataset/split/sampler
behavior, preprocessing + the preprocess hook, losses, hierarchical eval,
archive-contract validation, baseline feature splits, and vendored-baseline
equivalence against upstream.

See CHANGELOG.md
for the full 0.1.0 entry and migration notes.