Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 12 additions & 43 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,53 +4,27 @@ These instructions apply to the whole repository.

## Start here

- Edit the published SDK in `cytebase/`, the Modal build pipeline in `cytebase/pipeline/`, tests in
`tests/`, documentation in `docs/`, and the agent skill in `skills/cytebase/`.
- Before analysing data with CyteBase, read `skills/cytebase/SKILL.md`. Analysis steps on an opened
or mounted DataStore follow CyteArc's `cytearc-single-cell` skill. When a change alters a public
API that a skill recipe uses, update the recipe in the same change.
- Edit the published SDK in `cytebase/`, the Modal build pipeline in `cytebase/pipeline/`, tests in `tests/`, documentation in `docs/`, and the agent skill in `skills/cytebase/`.
- Before analysing data with CyteBase, read `skills/cytebase/SKILL.md`. Analysis steps on an opened or mounted DataStore follow CyteArc's `cytearc-single-cell` skill. When a change alters a public API that a skill recipe uses, update the recipe in the same change.
- Read `docs/AGENTS.md` before executing or publishing documentation.

## CyteBase developer mental model

- `Catalog` downloads one verified DuckDB catalog per release bucket and answers searches locally.
`DatasetEntry` describes a catalog row without opening its store. `Catalog.open_datastore` and
`Catalog.mount_datastore` return CyteArc `DataStore` objects after checking the dataset's published
record and build receipt against the opened arrays.
- Releases are public Hugging Face buckets named `Nygen/cytebase-YYMM` or `Nygen/cytebase-YYMM.N`,
listed in `cytebase/_releases.py`. A published release never changes in place.
- The pipeline inspects CELLxGENE H5AD sources, converts them with CyteArc, verifies the stores
against their sources, and publishes records and the catalog. Its outputs are persisted
contracts: `dataset.json` records, build receipts, `cytearc_ingest.json`, the catalog schema, and
the `.cytebase.json` mount receipt. Change their shapes only with an explicit compatibility
decision; do not add silent migrations.
- `Catalog` downloads one verified DuckDB catalog per release bucket and answers searches locally. `DatasetEntry` describes a catalog row without opening its store. `Catalog.open_datastore` and `Catalog.mount_datastore` return CyteArc `DataStore` objects after checking the dataset's published record and build receipt against the opened arrays.
- Releases are public Hugging Face buckets named `Nygen/cytebase-YYMM` or `Nygen/cytebase-YYMM.N`, listed in `cytebase/_releases.py`. A published release never changes in place.
- The pipeline inspects CELLxGENE H5AD sources, converts them with CyteArc, verifies the stores against their sources, and publishes records and the catalog. Its outputs are persisted contracts: `dataset.json` records, build receipts, `cytearc_ingest.json`, the catalog schema, and the `.cytebase.json` mount receipt. Change their shapes only with an explicit compatibility decision; do not add silent migrations.

## CyteArc boundary

- The SDK depends on `cytearc>=1.1.0,<2` and must keep working across that range. CyteArc does
not depend on CyteBase, and it opens CyteBase datasets by their `hf://` URIs. Import CyteArc
through its public facades. `tests/test_cytearc_boundary.py` lists the few private CyteArc
names the SDK uses, with the reason for each, and fails when CyteArc stops providing one. Prefer
a public CyteArc API over a new entry.
- CyteArc 1.1 ships an earlier copy of this SDK as `cytearc.cytebase`; later CyteArc releases keep
only their example-data downloads under that name. Never import `cytearc.cytebase` here.
- The pipeline is not published and may use CyteArc internals. It runs against the CyteArc version
that `uv.lock` pins, and its build receipts record `cytearcVersion`. After upgrading CyteArc in
the lock, run the complete suite, including the pipeline tests.
- CyteArc's own analysis features, such as composite stores, belong in CyteArc. CyteBase adds what
needs the catalog: discovery, provenance, and choosing and assembling datasets across studies.
- The SDK depends on `cytearc>=1.1.0,<2` and must keep working across that range. CyteArc does not depend on CyteBase, and it opens CyteBase datasets by their `hf://` URIs. Import CyteArc through its public facades. `tests/test_cytearc_boundary.py` lists the few private CyteArc names the SDK uses, with the reason for each, and fails when CyteArc stops providing one. Prefer a public CyteArc API over a new entry.
- CyteArc 1.1 ships an earlier copy of this SDK as `cytearc.cytebase`; later CyteArc releases keep only their example-data downloads under that name. Never import `cytearc.cytebase` here.
- The pipeline is not published and may use CyteArc internals. It runs against the CyteArc version that `uv.lock` pins, and its build receipts record `cytearcVersion`. After upgrading CyteArc in the lock, run the complete suite, including the pipeline tests.
- CyteArc's own analysis features, such as composite stores, belong in CyteArc. CyteBase adds what needs the catalog: discovery, provenance, and choosing and assembling datasets across studies.

## Source and conventions

- Python 3.13 or newer is required. Use `uv` for every Python command and dependency change.
- Preserve unrelated working-tree changes. Trace callers and tests before changing a public
function or a persisted record.
- Use built-in generics and `X | None`. Do not add `typing.List`, `typing.Optional`, or
`from __future__ import annotations`.
- Python 3.13 or newer is required.
- Pydantic model attributes use camel case.
- Do not add implicit compatibility branches, ad hoc schema versions, or silent migrations.
- Do not commit credentials, Modal secrets or tokens, private bucket names, or local endpoints.
- Avoid em dashes in public prose, prompts, docstrings, and comments.

## Environment and checks

Expand All @@ -62,8 +36,7 @@ uv run ruff format --check cytebase tests docs
uv run mypy cytebase
```

Run focused tests without pytest-xdist: `uv run pytest -n 0 tests/test_file.py::test_name`. After a
packaging change, build and check the wheel:
Run focused tests without pytest-xdist: `uv run pytest -n 0 tests/test_file.py::test_name`. After a packaging change, build and check the wheel:

```bash
uv build
Expand All @@ -72,8 +45,4 @@ uv run --no-project python tests/smoke_wheel.py dist/*.whl

## Universal safety

- Never deploy or run the Modal pipeline, and never write to a CyteBase bucket. Deployment
(`uv run python -m cytebase.pipeline.deploy`) and publication are user actions. Ask before any
paid or long-running cloud job.
- Do not publish datasets, catalogs, or releases, or use write credentials, unless the user
explicitly requests it.
- Never run the Modal pipeline, and never write to a CyteBase bucket. Deployment (`uv run python -m cytebase.pipeline.deploy`) is a user action.
62 changes: 14 additions & 48 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,12 @@

These instructions cover documentation sources, notebook execution, and site builds.

- Author executable pages as tracked `.ipynb` files in `docs/source/`. Each notebook is the
page's sole source and contains its prose, code, outputs, and execution metadata.
- Author executable pages as tracked `.ipynb` files in `docs/source/`. Each notebook is the page's sole source and contains its prose, code, outputs, and execution metadata.
- Keep prose guides and API references in Markdown.
- Explain each logical step in notebook code with a comment. Put comments on their own lines,
with a blank line before each comment unless it starts the code cell.
- MyST builds guides and tutorials. Sphinx builds only `docs/source/reference/api/`.
- Import the SDK as `import cytebase` or `from cytebase import ...`, and use `cytebase.*`
autodoc targets. Never use `cytearc.cytebase`: CyteArc 1.1 ships an earlier copy of the SDK
under that name, so code using it silently runs different code.
- Link CyteArc pages with absolute `https://docs.nygen.io/CyteArc/...` URLs. Relative links
must resolve to a page in `docs/source/`.
- `developers/build_pipeline.md` documents the operator-only `cytebase.pipeline`. Keep it in
step with the module docstrings and command-line help in `cytebase/pipeline/`.
- Import the SDK as `import cytebase` or `from cytebase import ...`, and use `cytebase.*` autodoc targets. Never use `cytearc.cytebase`: CyteArc 1.1 ships an earlier copy of the SDK under that name, so code using it silently runs different code.
- Link CyteArc pages with absolute `https://docs.nygen.io/CyteArc/...` URLs. Relative links must resolve to a page in `docs/source/`.
- `developers/build_pipeline.md` documents the operator-only `cytebase.pipeline`. Keep it in step with the module docstrings and command-line help in `cytebase/pipeline/`.

## Setup and rendering

Expand All @@ -25,49 +18,25 @@ uv sync --group docs --group docs-modal
make -C docs html
```

The site is written to ignored `docs/build/site/_build/html/`. Rendering does not execute code.
Every executable notebook needs a completed execution timestamp, a matching code fingerprint,
executed code cells, and no errors. Prose edits can reuse its outputs. Changed code requires
re-execution and the updated notebook in the same change. For a lightweight source preview,
use `make -C docs preview`; unexecuted pages display a warning that figures and results are
unavailable. Preview builds still reject changed code and incomplete or failed executions.

Pages excluded by `project.exclude` in `docs/source/myst.yml` remain source-only and need no saved
execution. Builds clear the generated site and API output so excluded pages leave no old routes.
Package, dependency, and catalog release changes require judgment about which pages to refresh;
there is no dependency fingerprint or cache invalidation service.

Pull requests, release checks, and GitHub Pages validate and render the notebooks from the
checkout with `make -C docs html`. CI does not execute them. Missing, failed, or stale executions
fail the build. GitHub Pages publishes from `master` at `https://docs.nygen.io/CyteBase/`. The
workflow builds with `BASE_URL=/CyteBase` and uploads the generated site as the Pages artifact;
the organization's Pages site serves it under `/CyteBase/` and owns the root `robots.txt`. Keep
rendered site files outside Git.
The site is written to ignored `docs/build/site/_build/html/`. Rendering does not execute code. Every executable notebook needs a completed execution timestamp, a matching code fingerprint, executed code cells, and no errors. Prose edits can reuse its outputs. Changed code requires re-execution and the updated notebook in the same change. For a lightweight source preview, use `make -C docs preview`; unexecuted pages display a warning that figures and results are unavailable. Preview builds still reject changed code and incomplete or failed executions.

Pages excluded by `project.exclude` in `docs/source/myst.yml` remain source-only and need no saved execution. Builds clear the generated site and API output so excluded pages leave no old routes. Package, dependency, and catalog release changes require judgment about which pages to refresh; there is no dependency fingerprint or cache invalidation service.

Pull requests, release checks, and GitHub Pages validate and render the notebooks from the checkout with `make -C docs html`. CI does not execute them. Missing, failed, or stale executions fail the build. GitHub Pages publishes from `master` at `https://docs.nygen.io/CyteBase/`. The workflow builds with `BASE_URL=/CyteBase` and uploads the generated site as the Pages artifact; the organization's Pages site serves it under `/CyteBase/` and owns the root `robots.txt`. Keep rendered site files outside Git.

## Execute affected pages on Modal

Ask before paid or long-running cloud work unless the current request already authorizes it.
Use an ephemeral app; never deploy. Select the environment explicitly:
Select the environment explicitly:

```bash
make -C docs execute-docs-modal PAGES="cytebase cytebase_covid19" MODAL_ENVIRONMENT=your_environment
```

One invocation submits all selected source notebooks concurrently, one container per page.
Each successful execution atomically replaces the same `.ipynb` file in `docs/source/` and
records `metadata.executed_at` and `metadata.executed_code_sha256`. The latter identifies the
executed code; do not edit this fingerprint by hand. Failed pages raise an error and leave
their previous notebook intact. Rerun the failed pages directly.
One invocation submits all selected source notebooks concurrently, one container per page. Each successful execution atomically replaces the same `.ipynb` file in `docs/source/` and records `metadata.executed_at` and `metadata.executed_code_sha256`. The latter identifies the executed code; do not edit this fingerprint by hand. Failed pages raise an error and leave their previous notebook intact. Rerun the failed pages directly.

The runner automatically records each code cell's UTC start time and elapsed wall time and
displays them below its output. CPU time is not measured. Keep `%%time` out of notebook cells.
The runner automatically records each code cell's UTC start time and elapsed wall time and displays them below its output. CPU time is not measured. Keep `%%time` out of notebook cells.

Do not start another invocation writing the same notebooks, or render while notebooks are being
replaced. The container runs the checkout's `cytebase` source with the dependencies that
`uv.lock` pins, including CyteArc. Notebook pages pin the catalog release they read, such as
`Catalog(version="2610")`, and read public buckets anonymously; this runner injects no secrets.
Catalog builds and release publication are separate operator actions requiring explicit
authorization.
Do not start another invocation writing the same notebooks, or render while notebooks are being replaced. The container runs the checkout's `cytebase` source with the dependencies that `uv.lock` pins, including CyteArc. Notebook pages pin the catalog release they read, such as `Catalog(version="2610")`, and read public buckets anonymously; this runner injects no secrets. Catalog builds and release publication are separate operator actions requiring explicit authorization.

## Checks

Expand All @@ -77,7 +46,4 @@ uv run pytest -n 0 tests/test_docs_structure.py tests/test_docs_render.py \
make -C docs check-reference
```

The API check is a nitpicky, warnings-as-errors Sphinx build. Its inventory must document every
`cytebase.__all__` export and `cytebase.display.CatalogResults` under their public names, and
every public `Catalog`, `DatasetEntry`, `Release`, and `CatalogResults` member exactly once.
Inspect the rendered figures and downloaded notebook after changing execution or presentation.
The API check is a nitpicky, warnings-as-errors Sphinx build. Its inventory must document every `cytebase.__all__` export and `cytebase.display.CatalogResults` under their public names, and every public `Catalog`, `DatasetEntry`, `Release`, and `CatalogResults` member exactly once. Inspect the rendered figures and downloaded notebook after changing execution or presentation.