Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Docs

on:
push:
branches: [main]

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- name: Install dependencies
run: uv sync --group docs
- name: Build docs
run: uv run mkdocs build --strict
- name: Setup Pages
uses: actions/configure-pages@v5
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: site

deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
3 changes: 3 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ on:
push:
pull_request:

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
Expand Down
111 changes: 111 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# bootstraptools

> The population is to the sample as the sample is to the bootstrap samples

Local storage, query, and uncertainty-quantification infrastructure for
**bootstrap experiments** that evaluate regression and classification model
performance. A bootstrap runner writes one run (B replicates) to a local
store, and downstream processing scripts query it back and compute point
estimates and confidence intervals **on demand**.

`bootstraptools` is **model-agnostic**: it sees a model as a generic `fit` /
`predict_proba` object and stores whatever scalars and named arrays a runner
hands it. It is shaped around plain sklearn architectures, so other model
types may need gentle massaging (a `fit(X, y)` / `predict_proba(X) -> (n,
2)` shim and a way to slice examples) to plug in.

```bash
uv sync # uv manages deps/venv
uv run pytest -q # full test suite
uv run python examples/reference_runner.py # end-to-end demo, prints CIs
```

## Data model

| Concept | Meaning | On disk |
|---|---|---|
| **Store** | A root directory holding many runs. | `<store>/runs/` |
| **Run** | One bootstrap experiment: B replicates for a `(dataset, model, procedure)` config, plus tags/config/summary. | `<store>/runs/<run_id>/` |
| **Replicate** | One refit + evaluation: scalar metrics, per-sample predictions, resample membership, seeds, optional rich arrays / model state. | rows/files within a run |

Per-run layout:

```
<store>/runs/<run_id>/
meta.json # tags, config, procedure, dataset, model_label, seeds,
# universe_size, status, n_replicates, summary
replicates.parquet # 1 row / replicate: seeds, fit metadata, scalar metrics
predictions.parquet # 1 row / (replicate, sample): sample_idx, y_true, p, split
membership.parquet # 1 row / (replicate, sample, count>0)
apparent.parquet # opt-in: the full-data model's predictions on all n rows
arrays/replicate_XXXX.npz # opt-in rich per-replicate arrays
model_state/replicate_XXXX.npz # opt-in full fitted params (store_model_state=True)
```

Tables are [polars](https://github.com/pola-rs/polars)/parquet, arrays are
npz. `arrays/` and `model_state/` are written **immediately** per replicate
(durable if a long run crashes).

## Writing a run

`init` -> per-replicate `log_replicate` -> `finish`. The runner owns the
fit/predict loop.

```python
import bootstraptools as bs

seeds = bs.derive_seeds(RANDOM_SEED, ["dataset", "model", "bootstrap"])
plans = bs.train_resample_holdout(
train_idx, val_idx, universe_size=n, n_replicates=100,
bootstrap_seed=seeds["bootstrap"],
)
fit_seeds = bs.replicate_seeds(seeds["model"], len(plans))

with bs.init(store, procedure="train_resample_holdout", dataset="bal",
model_label="ULTRA",
config={"rank": 4, "B": 100, "bootstrap_seed": seeds["bootstrap"]},
tags=["fig3", "pooling"]) as run:
for plan in plans:
model.fit(bs.select(X, plan.fit_indices), y[plan.fit_indices])
p = model.predict_proba(bs.select(X, plan.eval_indices))[:, 1]
run.log_replicate(
plan,
metrics={"brier": ..., "auroc": ...},
y_true=y[plan.eval_indices], p=p, # -> predictions.parquet
arrays={"attn_entropy": ...}, # opt-in rich artifacts
model_fit_seed=fit_seeds[plan.replicate_idx],
)
```

`log_replicate` pulls the replicate index / seed / membership from the
`ResamplePlan`. There is no fixed metric schema; column names are whatever
you pass. `bs.select(X, idx)` slices ndarrays (`X[idx]`) or lists of bags
(`[X[i] for i in idx]`). See the [procedures reference](reference/procedures.md)
for which procedure to pick, and the [resample reference](reference/resample.md)
for the variable-size draw primitives beneath them.

## Querying results

```python
runs = bs.query_runs(store, {"tags": "fig3", "dataset": "bal"}) # 1 row / run
tbl = bs.query_run_table(store, run_id, table="replicates") # one run's table
all_ = bs.load_runs_table(store, {"tags": "fig3"}, table="replicates") # concat + run_id col
membership = bs.membership_matrix(store, run_id) # dense (B, n) counts (for BCa)
```

`query_runs` filters (AND-ed) on `procedure`/`dataset`/`model_label`/`status`/
`run_id`, on `tags`, and on any `config.<key>`. config/summary flatten into
`config.*` / `summary.*` columns.

## Comparing across runs (paired analyses)

Represent any "same resample, different fit" axis as **separate runs
sharing the `bootstrap_seed`**: identical seeds give identical per-replicate
`fit_indices`, so the runs' replicates are row-joinable on `replicate_idx`
for paired comparisons.

## Where to go next

Browse the [API reference](reference/resample.md) for details on every
public function and class, or read `DOCS.md` in the repository for a deeper
walkthrough of the bootstrap procedures and reproducibility model.
3 changes: 3 additions & 0 deletions docs/reference/optimism.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.optimism`

::: bootstraptools.optimism
3 changes: 3 additions & 0 deletions docs/reference/procedures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.procedures`

::: bootstraptools.procedures
3 changes: 3 additions & 0 deletions docs/reference/query.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.query`

::: bootstraptools.query
3 changes: 3 additions & 0 deletions docs/reference/resample.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.resample`

::: bootstraptools.resample
3 changes: 3 additions & 0 deletions docs/reference/seeds.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.seeds`

::: bootstraptools.seeds
3 changes: 3 additions & 0 deletions docs/reference/store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.store`

::: bootstraptools.store
3 changes: 3 additions & 0 deletions docs/reference/uq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# `bootstraptools.uq`

::: bootstraptools.uq
59 changes: 59 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
site_name: bootstraptools
site_description: Local storage, query, and uncertainty-quantification infrastructure for bootstrap experiments
repo_url: https://github.com/meyer-lab/bootstraptools
repo_name: meyer-lab/bootstraptools
edit_uri: edit/main/docs/

theme:
name: material
palette:
- media: "(prefers-color-scheme: light)"
scheme: default
primary: indigo
toggle:
icon: material/brightness-7
name: Switch to dark mode
- media: "(prefers-color-scheme: dark)"
scheme: slate
primary: indigo
toggle:
icon: material/brightness-4
name: Switch to light mode
features:
- navigation.sections
- navigation.top
- content.code.copy
- content.action.edit

nav:
- Home: index.md
- API reference:
- Resampling: reference/resample.md
- Procedures: reference/procedures.md
- Seeds: reference/seeds.md
- Store: reference/store.md
- Query: reference/query.md
- Uncertainty quantification: reference/uq.md
- Optimism correction: reference/optimism.md

markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.superfences
- pymdownx.highlight
- tables
- toc:
permalink: true

plugins:
- search
- mkdocstrings:
handlers:
python:
options:
docstring_style: google
show_source: true
show_root_heading: true
merge_init_into_class: true
show_signature_annotations: true
separate_signature: true
6 changes: 5 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ dependencies = [
]

[dependency-groups]
docs = [
"mkdocs>=1.6.1",
"mkdocs-material>=9.7.7",
"mkdocstrings[python]>=1.0.6",
]
dev = [
"pytest-cov>=6.0,<7.0",
"ruff>=0.15",
Expand All @@ -28,6 +33,5 @@ target-version = "py313"
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B"]
ignore = ["E501"]

[tool.ty.environment]
python-version = "3.13"
Loading
Loading