Repository navigation
docs: add guide on in-tree tests for packages with compiled code #821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
InessaPawson
wants to merge
2
commits into
pyOpenSci:main
Choose a base branch
from
InessaPawson:343-treetests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,171 @@ | ||||||||||
| (in-tree-tests)= | ||||||||||
| # Shipping tests with your package (in-tree tests) | ||||||||||
|
|
||||||||||
| Throughout this guide we recommend keeping your `tests/` directory **outside** | ||||||||||
| of your package directory (an "out-of-tree" layout). This keeps your wheel | ||||||||||
| small and, combined with the [src layout](src-layout-test), helps ensure | ||||||||||
| that your tests run against the *installed* package rather than the files in | ||||||||||
| your working directory. | ||||||||||
|
|
||||||||||
| Some packages, however, benefit from shipping their tests inside the package | ||||||||||
| ("in-tree" tests). This is common in the scientific Python ecosystem: NumPy, | ||||||||||
| SciPy, and scikit-image all do it. | ||||||||||
|
|
||||||||||
| ## When in-tree tests make sense | ||||||||||
| Consider in-tree tests if your package: | ||||||||||
| * **Contains compiled code** (C, C++, Cython, Rust, or Fortran). A wheel built | ||||||||||
| on one platform may import fine but produce wrong results on another. | ||||||||||
| Letting users run the test suite against the wheel they installed | ||||||||||
| is the most reliable way to find out. | ||||||||||
| * **Is built by downstream packagers** such as conda-forge, Debian, or Spack. | ||||||||||
| Downstream packagers building from an sdist can run tests included in that | ||||||||||
| source distribution, even when tests are excluded from wheels. If supporting | ||||||||||
| these builds is the only goal, including tests in the sdist may be sufficient | ||||||||||
| (see below). | ||||||||||
| * Has users who need to **verify an installation** on an HPC cluster or | ||||||||||
| another locked-down environment where they can't clone the repository. | ||||||||||
|
|
||||||||||
| If your package is pure Python with a small test suite, out-of-tree tests are | ||||||||||
| simpler, and we recommend sticking with them. | ||||||||||
|
|
||||||||||
| ## Alternative: include tests in the sdist only | ||||||||||
| Tests do not have to be installed with a wheel to be available to downstream | ||||||||||
| packagers. A project can keep its `tests/` directory outside the package and | ||||||||||
| include those files in its source distribution (sdist) without including them | ||||||||||
| in the wheel. Downstream packagers building from the sdist can then run the | ||||||||||
| tests without increasing the size of wheels distributed to end users. | ||||||||||
| This supports downstream validation using files from the sdist, but users | ||||||||||
| cannot run those tests from an installed wheel alone. | ||||||||||
|
|
||||||||||
| For example, with setuptools, an out-of-tree `tests/` directory can be | ||||||||||
| included in the sdist using a `MANIFEST.in` entry: | ||||||||||
|
|
||||||||||
| ```text | ||||||||||
| graft tests | ||||||||||
| ``` | ||||||||||
| This does not itself control wheel contents: ensure `tests/` is not selected | ||||||||||
| as an installed package or included by other wheel configuration. Different | ||||||||||
| build backends provide different controls for source and wheel contents. | ||||||||||
| Check both built artifacts to confirm which files are included. For example, | ||||||||||
| build into a fresh output directory containing one sdist and one wheel: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| python -m build --outdir dist-check | ||||||||||
| tar -tf dist-check/*.tar.gz | grep '/tests/' | ||||||||||
| unzip -Z -1 dist-check/*.whl | grep -E '(^|/)tests/' | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| For the sdist-only approach, the first command listing files should show the | ||||||||||
| tests, while the wheel listing should have no matching files. The final `grep` | ||||||||||
| will exit with status 1 when no tests are found; that is expected here. | ||||||||||
| See the [setuptools guidance on wheel and sdist contents](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#wheel-vs-sdist). | ||||||||||
|
|
||||||||||
| If users need to run tests against an installed wheel, including tests in the | ||||||||||
| importable package as described below may be more appropriate. | ||||||||||
|
|
||||||||||
| ## Layout | ||||||||||
| Move `tests/` inside your package directory and give it an `__init__.py` so | ||||||||||
| that it is discovered and installed like any other subpackage: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| src/ | ||||||||||
| mypackage/ | ||||||||||
| __init__.py | ||||||||||
| core.py | ||||||||||
| tests/ | ||||||||||
| __init__.py | ||||||||||
| test_core.py | ||||||||||
| docs/ | ||||||||||
| pyproject.toml | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| :::{admonition} Make sure the tests ship | ||||||||||
| :class: tip | ||||||||||
| Check your build backend's package-discovery settings. If you list packages | ||||||||||
| explicitly, add `mypackage.tests`. After building, inspect the wheel | ||||||||||
| (`unzip -l dist/*.whl`) to confirm that the test files are included. | ||||||||||
| ::: | ||||||||||
|
|
||||||||||
| ## Run tests against the installed package | ||||||||||
| With tests inside the package, run them using pytest's `--pyargs` flag. This | ||||||||||
| tells pytest to import `mypackage` from the active environment, locate its | ||||||||||
| `tests` subpackage, and run it, regardless of your current working directory: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| pytest --pyargs mypackage | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| Many scientific packages also expose a convenience function so that users | ||||||||||
| don't need to remember the flag: | ||||||||||
| ```python | ||||||||||
| # src/mypackage/__init__.py | ||||||||||
| def test(*args): | ||||||||||
| """Run the mypackage test suite.""" | ||||||||||
| import pytest | ||||||||||
|
|
||||||||||
| return pytest.main(["--pyargs", "mypackage", *args]) | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ```python | ||||||||||
| >>> import mypackage | ||||||||||
| >>> mypackage.test() | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| Because `pytest` is imported only when the function is called, it is not | ||||||||||
| required for ordinary package use. If the distributed tests import pytest | ||||||||||
| or the public `test()` helper shown above uses pytest, declare it as an | ||||||||||
| optional dependency so users can install it when they need to run tests: | ||||||||||
|
|
||||||||||
| ```toml | ||||||||||
| [project.optional-dependencies] | ||||||||||
| test = ["pytest"] | ||||||||||
| ``` | ||||||||||
| If distributed tests use `unittest` and do not require pytest, there is no | ||||||||||
| need to expose pytest as an optional dependency solely because developers | ||||||||||
| use it as a test runner. Instead, projects can declare pytest in a | ||||||||||
| development dependency group or manage it through their development environment. | ||||||||||
|
|
||||||||||
| For packages that ship `unittest.TestCase` tests inside the importable | ||||||||||
| package, users can run an installed test module with Python's built-in | ||||||||||
| `unittest` runner without installing pytest: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| python -m unittest mypackage.tests.test_core | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| Replace `mypackage.tests.test_core` with the fully qualified name of an | ||||||||||
| installed test module. To discover all compatible tests in the installed | ||||||||||
| tests package, use: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| python -m unittest discover -s mypackage.tests | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| The `unittest` runner does not collect standalone pytest-style test functions | ||||||||||
| and does not provide pytest fixtures. Tests that depend on pytest-specific | ||||||||||
| features may require pytest to run successfully. | ||||||||||
|
|
||||||||||
| If distributed tests or a public test helper require third-party dependencies, | ||||||||||
| projects may declare those dependencies in a `test` extra to support running | ||||||||||
|
Comment on lines
+148
to
+149
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
| the tests after installation. | ||||||||||
|
|
||||||||||
| ## Pitfalls | ||||||||||
| **Running tests on the source tree by accident.** This is the main reason we | ||||||||||
| recommend out-of-tree tests for beginners. If you run | ||||||||||
| `pytest src/mypackage/tests` from the repository root, Python may import your | ||||||||||
| *un-built* source files instead of the installed package. With compiled code, | ||||||||||
| this surfaces as confusing `ImportError`s about missing extension modules. | ||||||||||
| Always use `pytest --pyargs mypackage` (or a `nox` or `tox` session that | ||||||||||
| installs the package first) rather than pointing pytest at a path. | ||||||||||
|
|
||||||||||
| **Test data.** Don't ship large datasets inside your wheel. See | ||||||||||
| [don't include test suite datasets in your package](src-layout-test) | ||||||||||
| for alternatives such as Pooch. | ||||||||||
|
|
||||||||||
| **Wheel size.** Every test file is downloaded by every user. Keep the in-tree | ||||||||||
| suite focused. Very slow or data-heavy tests can stay out of tree and run only | ||||||||||
| in CI. | ||||||||||
|
|
||||||||||
| For a real-world example, see how | ||||||||||
| [NumPy](https://github.com/numpy/numpy/tree/main/numpy) organizes | ||||||||||
| `numpy/tests` and `numpy/<submodule>/tests`, and exposes `numpy.test()`. | ||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need to ask a packaging learner to ensure that a sibling
testsisn't installed by a wheel build? That's pretty technical, and if they are following all our guidance, even these optional bits, it won't get included.