[parashardhapola]

Summary

Which transcription factors and pathways are active, in each sample or in a contrast, is
the question decoupler answers for most single-cell users. ulm, mlm and
weighted_aggregate now compute decoupler's activities from an expression matrix, a
cell type of a cube or the statistics of a differential-expression result, and return a
record that the linear models and the heatmap take as they take pathway scores. The
networks CollecTRI and PROGENy are downloaded from their releases on first use, pinned
by size and SHA-256, cached, and returned with their licence, source and citation;
CollecTRI comes by default at a licence level usable by a company. The results agree
with decoupler 2.2.0 and progeny 1.28.0 to rounding (89 checks, none failed).

Changes

• cytepool.activity.ulm(data, network, *, gene_ids, sample_ids, group, contrast, min_cells, expression_filter, id_column, statistic_column, min_targets=5, n_threads): one linear model per source of the profile's genes on the source's weights; t-statistics, slopes, p-values.
• cytepool.activity.mlm(...) (same arguments, without n_threads): one model of all the sources with an intercept; refuses, by name, sources it cannot tell apart.
• cytepool.activity.weighted_aggregate(..., statistic="mean" | "sum", n_permutations=1000, rng=None): the weighted mean or sum of the targets' values, normalized by permutations of the genes, with permutation p-values.
• cytepool.activity.Activities: profiles x sources scores, estimates, p_values, adjusted_p_values (Benjamini-Hochberg over the sources of each profile), n_targets, profiles (a Table with profile_id), network, method, settings; sources; table().
• cytepool.networks.Network(edges, *, name, version, url, licence, citation, file, sha256): a read-only mapping of source to {target: weight}, from a table or DataFrame of source, target, weight (extra columns kept), a mapping of columns, or a mapping of source to targets (a GeneSets included); edges, sources, select().
• cytepool.networks.collectri(*, licence="commercial", cache_dir=None) and progeny(*, organism="human", top=None, max_p_value=0.05, cache_dir=None).
• cytepool.plotting.plot_activities(activities, *, sources, top=25, color_by, standardize, ax), a heatmap drawn by the pathway scores' heatmap, and plot_top_activities(activities, profile, *, top=25, ax), decoupler's pl.barplot.
• cytepool/_expression_values.py: the reading of an array or a cube's group that pathway_scores used, moved so that both modules share it (behaviour and messages of the pathway scores unchanged but for naming what ids are matched to).
• validation/activity.py; tests/test_activity.py, tests/test_networks.py, fixtures in tests/data/activity; entries in tests/test_masked_inputs.py.
• Documentation: a section "Activities of regulators and pathways" in How the statistics work, Gene-set tests; API pages for cytepool.activity and cytepool.networks; a row in the capability matrix (and its line under not implemented removed); the networks and their licences on the Datasets page; the Validation page (row, section, totals, command, R environment); five works in references.bib and the Bibliography page; Choosing a method, How to cite, Related software, Glossary, the Overview and the README (status row, planned list, known limits). THIRD_PARTY_NOTICES.md records what is adapted from decoupler, with its BSD 3-Clause licence, and the two networks.

Design notes

• Profiles are the rows: samples, or one per group and contrast of a result (F-tests left out). NaN marks a missing value, so each profile of a result uses the genes its group tested, and MLM fits each set of profiles that observe the same genes with its own design. As in decoupler, genes zero in every profile are left out and a source with fewer than min_targets targets among a profile's genes has no score there; sources keep the network's order.
• ULM is computed from sums over each source's targets, O(profiles x (genes + edges)), with the exact t-statistic. MLM uses a Householder QR (LAPACK); a source is aliased when the diagonal of R is at most 1e-7 of its column's norm, the rank rule of the linear models.
• The weighted aggregate draws its permutations as decoupler's _ridx does (a tiled range shuffled row by row by numpy's generator), so that decoupler's seed and gene order give decoupler's permutations; the mean and standard deviation of the permuted scores are accumulated in one pass (Welford 1962).
• Downloads use the standard library: a temporary file in the cache, checked for size and SHA-256 before an atomic rename, and every load checks the cached file again. Bioconductor refuses clients that do not name themselves, so requests name CytePool as their client.
• PROGENy's weights are read from progeny's R data files by a reader of R's serialization format limited to what data frames of numbers, text and factors use. Its top keeps ties at the cutoff, as progeny's getModel (dplyr's top_n) does.
• The fixtures hold no third-party data: the tests' network, the rows in CollecTRI's format and the package in progeny's layout are simulated; the validation reads the real files.

Networks and their licences

Network	Source	File	Licence
CollecTRI, 1,186 human transcription factors, 43,175 signed interactions	Zenodo record 8192729, version 2.0 (doi:10.5281/zenodo.8192729), as decoupler 2.2.0 reads it	CollecTRI_regulons.csv, 4,345,649 bytes, SHA-256 4473c9189dd53dacc80297709ad1452dda1086a1cc2185f9a56146c261668701 (Zenodo's MD5 cee4a3943c059e6dd8796ced6dec44f6)	CC BY 4.0 for the record; each resource keeps its terms
PROGENy, 14 pathways, 22,479 human and 17,426 mouse genes	the R package progeny 1.28.0, Bioconductor 3.20	progeny_1.28.0.tar.gz, 10,642,873 bytes, SHA-256 2c41531ef6d088047e2830ccc5741aa0f28f48fb51d4c164cc8331e19b925d7b (Bioconductor's MD5 98f76e162cce19fd43ab13251969b117)	Apache License 2.0

CollecTRI's licence levels follow the licence purposes that OmniPath's resources service
lists for its twelve resources. Every resource but GEREDB is listed for commercial use
(CC BY 4.0 for ExTRI, NTNU Curated, TFactS, SIGNOR, IntAct, CytReg, GOA and
Pavlidis2021; CC BY-SA 4.0 for TRRUST; LGPL 3.0 for HTRIdb; an informal licence for
DoRothEA's level A). licence="commercial", the default, keeps the 39,896 interactions
that one of them supports and leaves out the 3,279 that GEREDB alone supports;
licence="academic" keeps all 43,175. OmniPath lists GEREDB's licence, the Academic
Free License 3.0, for academic use, although that OSI-approved licence does not itself
forbid commercial use, so the commercial level errs on the safe side. decoupler 2.2.0
accepts a licence for CollecTRI but returns the whole file at every level. PROGENy's
genes keep the symbols of its release; decoupler's op.progeny reads PROGENy through
OmniPath, which renames some genes to current symbols. VIPER is not included: the
licence of its reference implementation allows only non-profit academic use.

Deliberate differences from decoupler

• MLM's p-values: decoupler reports them unadjusted as its adjusted p-values; here
  p_values are the raw ones and adjusted_p_values are adjusted as for ULM.
  decoupler also keeps adjusted p-values in single precision and computes MLM's
  p-values as 1 - cdf (0 below about 1e-16); here both are double precision, from the
  upper tail.
• The weighted aggregate's p-values: decoupler counts permutations whose absolute
  score exceeds the absolute observed score, then folds that share as for a one-sided
  count, so a score near zero gets about 2 / n; here the p-value doubles the smaller
  one-sided p-value, (1 + b) / (1 + n). A normalized score with no spread among the
  permutations is NaN here, infinite or 0 in decoupler.
• A sample whose values are all zero: decoupler leaves it out of its result; here its
  scores are NaN. Missing values, which decoupler refuses, are left out per profile.
• Two sources with the same targets in MLM: decoupler returns NaN t-statistics for them
  from a singular inverse; here the call is refused, naming the source.
• CollecTRI's 185 interactions with an empty field of references: decoupler's loader
  drops them (pandas' dropna); here they are kept, as the file and the paper count
  them.
• ULM's t-statistic is exact; decoupler adds 2.2e-16 to 1 - r and 1 + r, which matters
  only for |r| within about 1e-6 of 1.

Validation

RSCRIPT=$REF/r-progeny/bin/Rscript PYTHONPATH=. $REF/python/bin/python validation/activity.py
(decoupler 2.2.0 and scipy 1.18.1 in $REF/python; progeny 1.28.0 on R 4.4.3 in
$REF/r-progeny, run under ulimit -v 8000000). 89 checks: 82 pass, 5 deliberate
differences, 2 information; 2 min 39 s, 987 MiB.

Check family	Reference	Largest difference	Tolerance	Outcome
ULM's t-statistics and slopes, 10 cases (CollecTRI and PROGENy; samples, t-statistics, missing values, a sample of zeros, a cube's cell type, limma_voom's t of two cell types)	mt.ulm	2.3e-13	1e-10	20 pass
ULM's p-values and their Benjamini-Hochberg adjustment	mt.ulm; scipy	4.4e-11	1e-10	20 pass
ULM's adjusted p-values in single precision	mt.ulm	0	1.2e-7	10 pass
MLM's t-statistics and coefficients, 6 cases	mt.mlm	3.2e-14	1e-10	12 pass
MLM's p-values	scipy's t.sf of mt.mlm's t	3.3e-12	1e-10	6 pass
Weighted means and sums; normalized scores given decoupler's permutations (100 and 1,000)	mt.waggr	1.1e-14	1e-10	4 pass
Weighted sums of PROGENy's top 100 genes	progeny's progeny(scale = FALSE)	6.5e-16	1e-10	1 pass
plot_top_activities: sources and lengths of 15 bars	pl.barplot	4.8e-14	1e-10	1 pass
CollecTRI academic and commercial levels; PROGENy's weights and p-values (human, mouse) and top genes	op.collectri, OmniPath's resources, progeny's models and getModel	0	0	6 pass
The downloaded files: sizes and MD5 digests	Zenodo, Bioconductor	0	0	2 pass
op.collectri at its levels; PROGENy through OmniPath	decoupler	n/a	n/a	2 information
The deliberate differences above	decoupler	0.97	n/a	5 deliberate

Scores and estimates are compared relative to the largest absolute value of a case,
p-values relative to each value.

Tests

tests/test_activity.py (34) and tests/test_networks.py (15), 1.9 s warm and 37 s
cold (compiling), and 96 new checks of the masked-input inventory. They compare with
decoupler's outputs written by the validation (its permutations replayed through a
numpy.random.Generator subclass, so the check does not depend on numpy's streams);
with values derived by hand (a source of two targets on three samples, t = -13 /
sqrt(108) and p = 1 - 13 / sqrt(385); MLM of one source equal to ULM; the weighted mean
and sum; weights of one sign as Student's pooled two-sample t-test); with the exact
null distribution over all 120 permutations of five genes; and with invariants (orders
of genes, sources and samples; negated weights negate scores; scaling and shifting a
profile scales ULM's slopes and keeps its t; zero weights, zero genes and genes
without ids change nothing; a profile with missing values equals that profile alone
over its observed genes). Networks: decoupler's parse of rows in CollecTRI's format,
the commercial level, R's values of R data files (factor and text genes) and dplyr's
top_n with a tie; refusal of a file that is not the one pinned, a refused download
that leaves nothing behind, offline use from the cache, XDG_CACHE_HOME, the
name of the client sent; the tests never connect.

Checks

• The full suite, on top of the combined p-values pull request: 4,716 passed, 18
  skipped; the same with bounds checks, and with the rounding changed
  (OPENBLAS_CORETYPE=Sandybridge, 4 threads, NUMBA_CPU_NAME=generic, a fresh
  cache); Python 3.14: 4,737 passed; Python 3.15: 4,631 passed, 107 skipped.
• uv run ruff check cytepool tests validation docs and ruff format --check: clean.
• uv run mypy cytepool: no issues in 76 files.
• make -C docs api: no warning; make -C docs html (Node.js 24): built;
  tests/test_documentation.py: 74 passed.