diff --git a/guide/environments.md b/guide/environments.md index 623e39e..4d6e9cd 100644 --- a/guide/environments.md +++ b/guide/environments.md @@ -9,8 +9,9 @@ The one-sentence version: **a test item runs in the same kind of sandbox that `P | Where the test file lives | What the test item runs in | | --- | --- | | Inside a package that has a `test/Project.toml` | A sandbox built from `test/Project.toml`, with the package itself added; versions pinned by the package's `Manifest.toml` if there is one | -| Inside a package that lists test dependencies in `[extras]` / `[targets]` | A sandbox generated from those sections, exactly as `Pkg.test` would; versions pinned by the package's `Manifest.toml` if there is one | +| Inside a package that lists test dependencies in `[extras]` / `[targets]` | A sandbox generated from those sections *plus the package's own `[deps]`*, exactly as `Pkg.test` would; versions pinned by the package's `Manifest.toml` if there is one | | Inside a package that is `dev`ed by an enclosing project (or by the active environment) | The same sandbox, but versions come from *that project's* `Manifest.toml` instead of the package's own | +| Inside a package, under a nested project that `dev`s it (`test/special/`, say) | The same sandbox again, with versions from *that* nested `Manifest.toml`. It pins versions; it does not add dependencies — see [below](#a-nested-project-pins-versions-it-does-not-add-dependencies) | | Outside any package | Nothing — there is no package to build a test environment for, and the test item cannot run. See [Troubleshooting](#troubleshooting). | The rest of this page explains how those rows are arrived at, and what happens in the corners. @@ -32,7 +33,7 @@ Belonging to a package says *which* package to test. It does not by itself say w 1. Take the **innermost enclosing project** of the test file (a folder with both a project file and a manifest). For an ordinary package with a checked-in `Manifest.toml`, that is the package folder itself. 2. If there is no enclosing project, take the **active environment**, if the surface you are using has one and it is a real project — that is, it has a manifest. An active environment without a manifest is ignored. Which surfaces have an active environment, and where it comes from, is described in the [next section](#the-active-environment-as-fallback). 3. Whatever project step 1 or 2 produced is kept only if it is **the package folder itself**, or a project whose manifest **`dev`s the package** (has it as a `path`-tracked dependency). A project that merely *contains* the package on disk, or that has the package as a regular registered dependency, does not count. -4. If nothing survives, the test item runs with the package folder alone as its source. Since a package folder with its own manifest would already have been picked up in step 1, this means a package without a manifest, and its dependencies are resolved fresh (see [Step 3](#step-3-with-or-without-a-manifest)). +4. If nothing survives, the test item runs with the package folder alone as its source — and the package folder's own manifest, if it has one, is used (see [Step 3](#step-3-with-or-without-a-manifest)). Note that step 1 searches the ancestors of the *file*, not of the package, so this is not only the manifest-less case: a `test/` folder that happens to hold both a `Project.toml` and a `Manifest.toml` is the innermost project for every file under it, is then discarded by rule 3 because it does not `dev` the package, and the package folder takes over. The point of rule 3 is that the manifest actually has to describe the code you are testing. If your workspace environment `dev`s `MyPackage` at `~/code/MyPackage`, its manifest is a faithful description of the dependency graph the package sees, and using it means your tests see the same versions your REPL does. If it does not `dev` the package, using its manifest would pin versions for a graph the package is not part of, so it is left out. @@ -90,12 +91,28 @@ Two version notes for older Julia releases: on Julia < 1.11 there is no `[source With the mirror active, the runner does what `Pkg.test` does — through the same code, in fact: it uses a vendored copy of [TestEnv.jl](https://github.com/JuliaTesting/TestEnv.jl), which is `Pkg.test`'s sandbox logic extracted into a package. For the package that owns the file: - **If `test/Project.toml` exists**, that project is the test environment. The package itself is added to it, and versions of anything shared with the main environment are carried over from the manifest chosen in Step 3. -- **Otherwise**, a test project is generated from the `[extras]` and `[targets]` sections of the package's own `Project.toml` (`test = ["Test", "..."]`), again with the package added and manifest versions carried over. +- **Otherwise**, a test project is generated from the package's own `[deps]` plus the names its `[targets]` `test` list names (`test = ["Test", "..."]`), each resolved through `[extras]` or `[weakdeps]` — again with the package added and manifest versions carried over. Note that the package's own dependencies come along here, which is why they are importable in a test item without being repeated anywhere. Either way the result is written to a temporary directory, resolved, and precompiled. If the versions pinned by the manifest cannot all be kept together with the test dependencies, the environment is re-resolved with a warning — `Could not use exact versions of packages in manifest, re-resolving` — which is the same warning `Pkg.test` prints in that situation. `[extras]` and `[targets]` are always read from the *real* package `Project.toml`, so a package that keeps its test dependencies there does not need to do anything special. +### A nested project pins versions, it does not add dependencies + +Steps 2 and 4 pull in opposite directions, and it is worth being explicit about where they meet. A `Project.toml` + `Manifest.toml` pair placed *inside* a package — `test/special/`, say — is chosen by Step 2 like any other enclosing project, provided its manifest `dev`s the package. What that changes is **Step 3**: test items under `test/special/` take their versions from that manifest, while test items elsewhere in the package take theirs from whatever Step 2 picks for them. Each group gets its own test process. That is a supported way to run different groups of test items against different pinned versions — a JET or GPU suite held at a specific version, for instance. + +What it does **not** change is Step 4. The test target is always read from the package, so a package listed only in `test/special/Project.toml` is not a dependency of the environment the test items run in, and `using` it fails: + +``` +ArgumentError: Package Aqua not found in current path. +``` + +A package has exactly one test target — the same one `Pkg.test` uses — and Julia has no notion of per-directory test dependencies. So the two halves are used together: declare the dependency once in the package's test target, and let the nested manifest decide which version of it that group of test items resolves to. A version pinned in the nested manifest only takes effect for packages the test target actually reaches; anything else in that manifest is pruned away with the rest of the dependency graph the tests do not use. + +::: tip +This is the nested counterpart of the monorepo case above, and the rule is the same in both: a project supplies the manifest, the package supplies the test dependencies. +::: + ## What the test process sees A few concrete facts, for when a test item asks questions about its own surroundings: @@ -112,7 +129,7 @@ A few concrete facts, for when a test item asks questions about its own surround An environment is materialized once per test process, when the process starts. Test processes are pooled and reused across runs (see [Test Processes](./test-processes)), and the pool is keyed by everything above — package, project, Julia binary and flags, environment variables, coverage mode, bounds checking. A run that asks for the same combination reuses a process that already has the environment loaded, which is why the second run is fast. -If the content of the chosen project's `Project.toml` or `Manifest.toml` changes between runs, the pooled process is not reused: its environment was resolved against the old content, and it is restarted rather than patched. Edits to your source code do not have that effect — those are picked up by Revise. +If the content of any file the environment was built from changes between runs, the pooled process is not reused. That is the chosen project's `Project.toml` and `Manifest.toml`, the package's own `Project.toml` and manifest, and the package's `test/Project.toml` and `test/Manifest.toml` — so editing your test dependencies restarts the process, as it must: its environment was resolved against the old content, and it is restarted rather than patched. Edits to your source code do not have that effect — those are picked up by Revise. When several processes start into the same cold environment, one of them precompiles and the others wait for it, so a cold start costs one precompilation, not one per process. @@ -133,6 +150,8 @@ If you use both — `juliati` day to day and `Pkg.test` for registries and downs **Tests see different versions than my REPL** — your REPL's environment is not being used as the fallback project. Check that it has a `Manifest.toml`, that it `dev`s the package (not `add`s it), and that the surface you are on has an active environment at all — `juliati` never does. Alternatively check in a manifest in the package folder itself. +**"Package X not found in current path"** — `X` is not in the package's test target. Test dependencies come from the package's `test/Project.toml`, or from its `[extras]`/`[targets]`; a `Project.toml` nested deeper in the tree supplies version pins only. See [Step 4](#step-4-the-test-target). + **"Could not use exact versions of packages in manifest, re-resolving"** — the manifest and the test dependencies could not be satisfied together, so the test environment was resolved fresh. Usually a `[compat]` entry in `test/Project.toml` or `[extras]` conflicts with what the manifest pins. **A whole process fails before any test item runs** — this is nearly always precompilation of the freshly built environment failing. `test log ` in the [REPL](./repl#managing-test-processes) or the Julia Workspace panel in VS Code shows the process's raw output with the actual error. diff --git a/guide/pkg-test.md b/guide/pkg-test.md index 02d3bad..6bd73cc 100644 --- a/guide/pkg-test.md +++ b/guide/pkg-test.md @@ -20,6 +20,8 @@ The two approaches are not exclusive. The same `@testitem` blocks work under bot Pkg.add("TestItemRunner") ``` + Note that this writes a `test/Manifest.toml` as well. That makes `test/` a project in the sense [environment discovery](./environments#step-2-which-project-supplies-the-manifest) uses, and since it does not `dev` the package it is then discarded — so test items under `test/` fall back to the package folder rather than to an enclosing project or your active environment. Delete it, or `dev` the package into it, if you were relying on those for your versions. + 2. Create or update `test/runtests.jl`: ```julia using TestItemRunner diff --git a/guide/test-processes.md b/guide/test-processes.md index 419e545..8612b0d 100644 --- a/guide/test-processes.md +++ b/guide/test-processes.md @@ -14,7 +14,7 @@ The pool is keyed by the *environment* a test item needs — its package and pro Two things invalidate a pooled process rather than reusing it: -- **The environment changed.** If the content of the [chosen project's](./environments#step-2-which-project-supplies-the-manifest) `Project.toml` or `Manifest.toml` changed since the process was started, its loaded packages are stale in a way no amount of reloading can fix, and it is restarted. +- **The environment changed.** If any file the environment was built from changed since the process was started — the [chosen project's](./environments#step-2-which-project-supplies-the-manifest) `Project.toml` and `Manifest.toml`, the package's own pair, or the package's `test/Project.toml` and `test/Manifest.toml` — its loaded packages are stale in a way no amount of reloading can fix, and it is restarted. - **You terminate it.** VS Code offers **Stop Test Process** in the Julia Workspace panel, DevREPL has [`test kill`](./repl#managing-test-processes). Both are the way out if a process gets into a state you do not trust. ### Revise-based hot reload diff --git a/integrating/jsonrpc.md b/integrating/jsonrpc.md index ffb7a93..d363475 100644 --- a/integrating/jsonrpc.md +++ b/integrating/jsonrpc.md @@ -63,7 +63,7 @@ Response: | `mode` | string | `"Normal"`, `"Coverage"` or `"Debug"`. | | `packageName`, `packageUri` | string | The package under test. | | `projectUri` | string, optional | The project supplying the manifest ([Environments](../guide/environments)). | -| `envContentHash` | string, optional | Changes when Project/Manifest change; a pooled process is revised when it matches and restarted when it differs. | +| `envContentHash` | string, optional | Changes when any file the environment is built from changes — the chosen project's Project/Manifest, the package's own pair, and the package's `test/Project.toml`/`test/Manifest.toml`. A pooled process is revised when it matches and restarted when it differs. | | `checkBounds` | string, optional | `"auto"` (default) or `"yes"`; see [`juliati --check-bounds`](../guide/cli#bounds-checking). | These are exactly the values the language server returns from [`julia/getTestEnv`](./language-server#julia-gettestenv-client-→-server). diff --git a/integrating/julia-apis.md b/integrating/julia-apis.md index acd0494..422c2e6 100644 --- a/integrating/julia-apis.md +++ b/integrating/julia-apis.md @@ -23,7 +23,7 @@ add_folder_from_disc!(jw, "path/to/MyPackage") add_file_from_disc!(jw, "path/to/MyPackage/test/new_tests.jl") # a file appeared update_file_from_disc!(jw, "path/to/MyPackage/test/parsing_tests.jl") # a file changed remove_file!(jw, JuliaWorkspaces.filepath2uri("path/to/MyPackage/test/old_tests.jl")) # a file went away -set_active_project!(jw, JuliaWorkspaces.filepath2uri("path/to/some/env")) # fallback environment for files outside any project +set_active_project!(jw, JuliaWorkspaces.filepath2uri("path/to/some/env")) # fallback env; as a test project only if it has a manifest that devs the package ``` `workspace_from_folders(folders; store_path, …)` is what every one-shot tool uses; a long-lived tool (an editor, an MCP server) keeps one workspace and feeds it file changes. The workspace is **not thread-safe**: hold your own lock around every call if a file watcher and a request handler both touch it. @@ -57,7 +57,7 @@ Notes: - `option_skip` is either a `Bool` or the *source text* of the `skip=` expression, which the test process evaluates just before running the item. - Test items outside a Julia package produce only a `TestErrorDetail` ("Test items must be defined inside a Julia package"). - `id` is package-scoped (`MyPkg@a1b2c3d4/test/file.jl::name`); identify items by `(id, package_uri)`. See [the shared concepts](./overview#concepts-every-layer-shares). -- `get_test_env` implements the environment rules of [Environments](../guide/environments): the file's owning package, the project that supplies the manifest (or the active project when the file's own project has no manifest), and a content hash that changes when Project/Manifest change. +- `get_test_env` implements the environment rules of [Environments](../guide/environments): the file's owning package, the project that supplies the manifest (or the active project when no folder above the file has both a project file and a manifest — and either way only if it is the package folder or `dev`s the package), and a content hash that changes when any file the environment is built from changes. - Discovery honors [`JuliaTestItems.toml`](../guide/configuration). ## Execution with TestItemControllers diff --git a/integrating/language-server.md b/integrating/language-server.md index 705c1fd..b50bc85 100644 --- a/integrating/language-server.md +++ b/integrating/language-server.md @@ -112,13 +112,13 @@ Result: | `packageName` | string, optional | The package the file belongs to. | | `packageUri` | string, optional | URI of the package root folder. | | `projectUri` | string, optional | URI of the project that supplies the manifest — see [Environments](../guide/environments). | -| `envContentHash` | string, optional | Changes when the environment's Project/Manifest content changes. | +| `envContentHash` | string, optional | Changes when any file the environment is built from changes: the chosen project's Project/Manifest, the package's own pair, and the package's `test/Project.toml`/`test/Manifest.toml`. | Each field is absent when unknown (a file outside any package has none). The four values are exactly the `packageName`, `packageUri`, `projectUri` and `envContentHash` fields of a TestItemControllers [`TestEnvironment`](./jsonrpc#testenvironment). ## `julia/setEnvironmentPath` (client → server) -`{ "envPath": "/path/to/environment" }` — tells the server which Julia environment the editor considers active. It becomes the fallback environment for files outside any project and the fallback test project when a package has no manifest of its own ([Environments](../guide/environments) explains the rules). +`{ "envPath": "/path/to/environment" }` — tells the server which Julia environment the editor considers active. It becomes the fallback environment for files outside any project, and the fallback test project when no folder above the file is a project — that is, when none has both a project file and a manifest. As a test project it is subject to the same rule as any other: it is used only if it is the package folder itself or its manifest `dev`s the package ([Environments](../guide/environments) explains the rules). ## From publish to run diff --git a/integrating/testitemruns.md b/integrating/testitemruns.md index 7815beb..3550d39 100644 --- a/integrating/testitemruns.md +++ b/integrating/testitemruns.md @@ -60,7 +60,7 @@ exit(ok ? 0 : 1) | `on_event` | `nothing` | Receives a `DiscoveryFinished` event and then every [run event](#events). | | `log_min_level` | `Logging.Warn` | Minimum level for log records emitted while the run is active (the controller logs its lifecycle at info level). `nothing` leaves the current logger in place. | | `store_path` | `nothing` | JuliaWorkspaces on-disk store. | -| `active_project` | `nothing` | Project folder or file used as the fallback environment for files outside any project — see [Environments](../guide/environments). | +| `active_project` | `nothing` | Project folder or file used as the fallback environment for files outside any project. As a test project it is used only if it has a manifest and that manifest `dev`s the package — see [Environments](../guide/environments). | Discovery honors [`JuliaTestItems.toml`](../guide/configuration) exactly like every other surface. Test processes never inherit `JULIA_LOAD_PATH`, `JULIA_PROJECT` or `JULIA_DEPOT_PATH` from the host, so a tool installed as a Pkg app can run tests for arbitrary packages.