[parashardhapola]

Summary

Results of one single-cell study often do not replicate in the next, and studies are
compared or combined only once their samples, genes and cell types are aligned. Aligned
by hand, they go wrong in two ways: donor ids repeat across studies, and genes that a
study did not measure are read as zeros. CubeCollection aligns the cubes of several studies,
each kept as it was given, with what each study measured and has, and gives each
study's cube on the collection's axes for analyses study by study, their results lined
up gene by gene, and one stacked cube for a joint model. Combined p-values, replication
metrics and SumRank, the next pull requests, build on it.

Changes

• cytepool.cube.CubeCollection(cubes, *, align_groups_by=None, group_mapping=None, within_study=()):
  study name to cube, aligned. Properties studies, n_studies, n_samples,
  n_groups, n_features, cubes (as given), samples (sample_id written
  study::sample_id, study, every study's columns), groups (aligned ids and the
  columns the studies agree on), features (feature_id, symbol), measured
  (studies x features booleans), group_members (each study's groups under their
  aligned ids) and provenance.
• CubeCollection.aligned_cube(study): the study's cube with its groups under their
  aligned ids (merged where several share one) and the features it measured, in the
  collection's order, with its own samples; derived with Cube.select and
  Cube.regroup, so lazy for cube files and recorded in its provenance.
• CubeCollection.select(studies, groups, features): a collection of some studies,
  groups and features, as Cube.select reads selections.
• CubeCollection.stack(*, groups, features, layers): one cube of every study's
  samples, with the groups every study has and the features every study measured by
  default; anything a study lacks is refused, never filled with zeros.
• cytepool.meta_analysis.align_study_results(collection, results, group, contrast):
  the results of every study for one cell type and contrast, a row per study and gene,
  with status (tested, not_tested, not_measured, not_analysed), reason and
  every statistic of the studies' tables (NaN where not tested).
• describe_provenance describes a stacked cube's construction; the provenance
  documentation lists its keys.
• cube._axes.selected_positions (private): Cube.select's reading of ids, masks and
  positions, moved out of cube.py to be shared; behaviour unchanged.
• validation/cube_collection.py and fixtures in tests/data/cube_collection/
  (67 KB).
• Documentation: a "Several studies" section on the cube page, a paragraph on lining up
  results on the meta-analysis page, the API pages' introductions, a capability matrix
  row (and an item under "Not implemented yet"), the Validation page (summary row,
  totals, date, section, command), the Cell Ontology paper (Diehl et al. 2016, checked
  against Crossref) in references.bib and on the Bibliography page, select under
  [planned] for the combining-studies guide, the provenance keys page, the overview's
  two questions on combining studies, and the README status table.

Design notes

• What this pull request includes, and what the next one does. Here: the
  alignment, selection, each study's aligned cube (any per-group analysis then runs
  study by study with its own design), results lined up gene by gene, and stacking.
  Combined p-values and replication metrics are statistics over aligned results, so
  they come next, reading align_study_results and measured. Both of the data's
  parts belong here because they decide what a missing value means (a gene not
  measured is not a gene measured and filtered out, and neither is a zero), which the
  validation checks against tools that know nothing of the statistics; a pull request
  that added the statistics on its own alignment would have validated both at once.
  There is no runner that analyses every study: designs differ by study, and the loop
  is one line, {s: limma_voom(c.aligned_cube(s), designs[s]) for s in c.studies},
  whose result differential_expression_meta_analysis reads unchanged.
• Each cube kept as it is. The collection reads no layer: it aligns ids and tables.
  An aligned cube is derived when asked for (a copy for a cube in memory, lazy for a
  cube file, the cube itself when nothing changes), and its provenance chains to the
  cube as given by fingerprint.
• Missing, not zero. A study measured a feature when its cube has it and the cube's
  existing measured column does not say otherwise; the collection's features are
  those some study measured. stack takes only the features every study measured and
  the groups every study has, by default or as given: a cube holds no missing values,
  and a study without a group did not resolve that cell type, so cells counted as zero
  would mislead a composition test. Stacking needs cell numbers in every study or in
  none, and normalized layers only from studies normalized alike.
• Groups. Aligned by the values of one text column for the whole collection:
  cell_ontology_id when every cube has it, group_id otherwise, or a column named.
  group_mapping gives some of a study's groups their aligned ids instead (None leaves
  a group out). Keys are equal values only, never mixed per group, which would not be
  transitive. Groups of one study with the same key are merged with Cube.regroup,
  exactly; aligned groups are named by their key, so ontology ids by default, and a
  mapping gives readable names. A group without a key is an error that says how to map
  it, never dropped silently. Ontology-aware roll-up to a common ancestor needs the
  Cell Ontology as a resource and is left for later; the mapping covers it meanwhile.
• Samples and donors. Sample ids are always written study::sample_id; which other
  columns hold ids cannot be guessed, so within_study names them (donors), and the
  validation's control shows what goes wrong without it. Sample columns combine as
  pandas combines them (union in order of appearance, integers and floats to float64,
  missing where a study lacks a column), except that booleans that need a missing value
  become float64 with NaN, as in Table, and a column of two kinds of values is refused
  with its studies and types. A per-study aligned cube keeps its own sample ids, so
  designs built from its table apply.
• Orders. Studies as given; features and groups in the order the studies first list
  them, as the meta-analysis orders features; a selection sets its own order.
  symbol is the first one a study gives. Feature ids are compared as they are: two
  studies that share no feature are refused, with a hint when the ids differ only by an
  Ensembl version.
• Lining up results. align_study_results refuses results that did not come from
  the studies' aligned cubes (unaligned group ids, a tested feature the study did not
  measure), and gives a complete table, study by study and gene by gene, so that
  column.reshape(n_studies, n_features) gives a matrix for the statistics to come.
• Left out: NaN-filled views of a layer across studies (for heatmaps), lazy
  stacking, Ensembl version and ortholog mapping, and the guide.

Validation

validation/cube_collection.py: three simulated studies pooled with from_cells
(8, 10 and 6 samples; 700, 750 and 950 genes, 40 of them zero-filled and marked not
measured; cell types aligned by ontology id, two of them merged, one study by a mapping
that leaves one out; sample and donor ids that repeat across studies). anndata and
pandas join each study's to_anndata export themselves (measured genes only, rows of a
sample and aligned cell type summed); R runs limma-voom on each study's own counts and
on the stacked counts of two studies.

Check family	Reference	Largest difference	Tolerance	Checks
Sample table: keyed ids, donors, union of columns, types, missing values	pandas.concat	0 mismatches	0	1 pass
Genes (set and order) and which study measured which	anndata.concat outer join (NaN fill), pandas.unique	0	0	2 pass
Every layer of every aligned cube	outer join values	0	1e-10	1 pass
stack(): axes, every layer, cells per entry and an entry without cells	anndata.concat inner join	0	0, 1e-10	3 pass
limma_voom on each aligned cube via align_study_results, 12 cell types	limma, study by study	3.8e-13	1e-10	3 pass
Status of every gene of every study	limma's tables, measured genes	0 mismatches	0	3 pass
Joint model ~ donor + condition on the stack, donors study::donor	limma on stacked counts	1.4e-13	1e-10	1 pass
Control: the same with the studies' own donor ids	the same	1.9 (logFC)	must exceed 1e-10	1 control

15 checks: 14 pass, 1 control. limma ran on the counts with the zero-filled genes and
its tables equal those of the aligned cubes, which drop them: marking a gene missing
changes no other gene's result. References: anndata 0.13.4, pandas 3.1.0, numpy 2.5.4,
Python 3.13.12; R 4.4.3, limma 3.62.1, edgeR 4.4.0. 26.7 s, 495 MiB peak.

REFERENCE_PYTHON=$REF/python-scanpy/bin/python RSCRIPT=$REF/r/bin/Rscript \
    PYTHONPATH=. $REF/python/bin/python validation/cube_collection.py [--write-fixtures]

Tests

tests/test_cube_collection.py, 46 tests, 0.9 s warm: against the fixtures (samples as
pandas joins them; features, order and measured as anndata's outer join, and aligned
counts as its values; stack as its inner join; each study's limma-voom lined up and
the joint model as limma, to 1e-10, with every gene's status); a two-study example
worked out by hand (merged cell types, a zero-filled gene, repeated donor ids, symbols,
the stacked layers and cells); mappings that rename, merge and leave out, a column the
studies agree or disagree on, ontology ids given by a mapping; invariants (the order of
studies and of genes within a cube, select then stack against stack then select, a
single study against its own cube, cube files against cubes in memory); layer types;
normalizations; provenance of the collection, of aligned and stacked cubes and their
history; pickling, copying and immutability; and every message. The masked-input
inventory gains the constructor, select, stack, aligned_cube and
align_study_results (16 cases).

Checks

• uv run pytest -q --durations=15: 4,306 passed, 18 skipped (27 s warm, 155 s cold).
• The touched test files with OPENBLAS_CORETYPE=Sandybridge OPENBLAS_NUM_THREADS=4 NUMBA_NUM_THREADS=4 NUMBA_CPU_NAME=generic and a fresh cache: 1,489 passed.
• ruff check and ruff format --check on cytepool, tests, validation and docs; mypy cytepool: clean.
• make -C docs api (warnings as errors) and make -C docs html (Node.js 24): clean;
  tests/test_documentation.py: 74 passed.