feat: run system-tests locally instead of via Farm#10253
Open
basvandijk wants to merge 109 commits into
Open
Conversation
…89d85fa0818479c81c1025049a7ff ic-build: sha256:4bd0bda360187ee57fb9923c788a61c51675b8c4d430695892e65b57333d5795 ic-dev: sha256:c6da125dd03ebc4b55cf357e6ecb528e5a9154ce38ce1b877d81ca4cb9c0c5c9
Contributor
|
Run URL: https://github.com/dfinity/ic/actions/runs/26099531371 New container images with tag: |
…12d9b1189a814656d579c04095e25 ic-build: sha256:151d1ebf531a9d83baeb566a0ac5dcc91983aee94cf01b76c2b88722dada037f ic-dev: sha256:a4ad94ed89fc266191fb01e168332f2d73c808cf110eb71f0b7a956b895e4c58
Contributor
|
Run URL: https://github.com/dfinity/ic/actions/runs/26635030301 New container images with tag: |
…4690bb71e96a05635f67ab9ed912b ic-build: sha256:dcb838604bc2a7643b677965d1b59e1e7b1e9830106b528219a47ac754731d6e ic-dev: sha256:e4969a5a299c7505c4934edf718dc3c9d78518ce8dce9f57b70588478241ffb5
Contributor
|
Run URL: https://github.com/dfinity/ic/actions/runs/26645828461 New container images with tag: |
…12d9b1189a814656d579c04095e25 ic-build: sha256:a9d38170c5bb83a91d916d0aa6d70270f675e2f8f3dea09a1dad8fdfcc08c9d5 ic-dev: sha256:5f4557cc25d33aaf034f0053f5f33e89d9681cf8eb1af9b91616e61e51d9a514
Commit 0ebf1d0 added a fallback to `get_local_ipv6()` that enumerates interface addresses when the route-based `local_ipv6()` probe fails. It was needed because the local system-test backend advertised SLAAC with a router lifetime of 0, so guests configured a global address but installed no default IPv6 route, making the route-based lookup fail with ENETUNREACH. Commit d3dad5b later changed that Router Advertisement to a non-zero router lifetime (1800s), installing the host as the guests' default router. The route-based `local_ipv6()` lookup therefore succeeds again and the fallback is no longer necessary. Revert the nss_icos parts of 0ebf1d0: the `get_local_ipv6()` fallback and its `first_global_unicast_ipv6`/`is_global_unicast_ipv6` helpers and unit tests, the BUILD.bazel `rust_test` target, and the README paragraph. The bootstrap.rs firewall whitelist from that commit is a separate concern and is kept. Verified //rs/tests/networking:canister_http_correctness_test_local still passes.
Systems now always set kernel.apparmor_restrict_unprivileged_userns=0, which was this script's only remaining job, so it is no longer needed. Drop the script and clean up all references: - container-run.sh: remove the BOOTSTRAP wrapper; exec runs the command directly. - devcontainer.json: remove the postStartCommand hook and the stale groupmod comment. - Dockerfile and packages.common: reword comments that referenced the script. KVM access (--group-add) and the baked-in ic-net-admin launcher are unaffected.
…23cbbf99c347f6eedbeb169a37a92 ic-build: sha256:c3219dc2255d9359dd798913375f559e17f95772373b74c1db6e7221f50af78d ic-dev: sha256:7c945d1374d709ffdcd1ebc4e651a47dde7fb4811158143994132aba2d8e6b04
Contributor
|
Run URL: https://github.com/dfinity/ic/actions/runs/27171059138 New container images with tag: |
On the Local backend the test driver sources all host->node traffic from a single per-group management IPv6 address. The journald log-stream task held a persistent connection to each node's systemd-journal-gatewayd (port 19531) from that same address, consuming one of the node firewall's per-source-IP connection slots (max_simultaneous_connections_per_ip_address = 1000, counted per source IP across all ports). The test opens exactly 1000 connections from the management address and expects all to be accepted, so the 1000th raced the log stream for the last slot and was persistently dropped, failing after retries. Give the IC-node journald stream its own dedicated per-group source address (group_logs_ipv6, <prefix>:2::1, assigned to lo alongside the management address) and bind the streaming socket to it, so it no longer competes with any test's per-source connection budget. The stream source is resolved per backend: Local binds to the dedicated address, Farm lets the kernel pick (no per-source firewall budget to protect there), so enabling IC-node log streaming on Farm later works too. Verified with 3 runs (--runs_per_test=3 --cache_test_results=no): all pass.
The Local backend's per-group file server (serve_files_task) listened on the management IPv6 address, which is also the source the test driver originates all its host->node traffic from. Give the file server its own dedicated per-group address (group_files_ipv6, <prefix>:3::1, assigned to lo) and point the node image-download URLs at it, mirroring the journald log-stream address (group_logs_ipv6, <prefix>:2::1). This reserves the management address exclusively for the test driver's own host->node traffic, keeping the node firewall's per-source connection budget easy to reason about (it matters for tests that deliberately saturate it, e.g. the firewall connection_count_test). Also remove the logs and files addresses from lo in delete_group alongside the management address. Verified: firewall_max_connections_test_local passes and the file server now binds to [<prefix>:3::1]:8080.
libvirt's QEMU driver defaults `stdio_handler` to "logd", which makes the per-test libvirtd spawn a `virtlogd` daemon to manage VM console logs. Its only added value is rolling log files over at a size limit, which bounded test runs do not need, and it is one more double-forked daemon the teardown reaper must track. Set `stdio_handler = "file"` in the per-test `qemu.conf` so QEMU writes the console log directly to `console.log` and no `virtlogd` is started. The domain XML's `append='on'` is honoured natively by QEMU's file chardev, so console output is still preserved across domain restarts (guest reboots and `vm().kill()` + `vm().start()`). Also fix now-stale `virtlogd` mentions in the local-backend and descendant-reaper comments. Verified uncached on the local backend: //rs/tests/idx:basic_health_test_local and //rs/tests/node:kill_start_short_test_local both PASS with no virtlogd process spawned (the latter exercises the VM kill+restart console-append path).
The `_local` system-tests use too many resources, causing the `bazel-test-all` job to run for over 3 hours. Move them to a dedicated `local-system-tests` job in schedule-daily.yml by excluding the `local_system_test` tag from the targets inferred for `bazel-test-all`. To keep smoke-testing the local-backend machinery on PRs, the `//rs/tests/idx:basic_health_test_local` test is re-added to `bazel-test-all`, but only when its Farm sibling `//rs/tests/idx:basic_health_test` is already being tested (so it respects the diff-only target selection). Also revert the temporary `bazel-test-all` timeout bump (720 -> 150).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a “Local” system-test backend that runs IC system-tests fully inside the Bazel sandbox (no sudo, no requires-network) by provisioning libvirt/QEMU resources locally, while keeping the existing Farm backend for large/unsupported tests. It also updates CI/container tooling and test targets so _local variants can run in dedicated nightly jobs and a small smoke test can run on PR CI.
Changes:
- Extend
system_testrules to generate_localvariants, support backend selection (farm/local), and allow CPU reservation tagging for local runs. - Update the Rust system-test driver to support
SystemTestBackend::Localfor group creation, VM lifecycle operations, disk image handling, logging/streaming, and backend-aware IC-OS image URLs. - Update CI/devcontainer/container images and workflows to include libvirt/QEMU dependencies and
/dev/kvmgroup access, plus add a nightly job to run_localtests.
Reviewed changes
Copilot reviewed 99 out of 102 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| rs/tests/testnets/BUILD.bazel | Add backend/cpus annotations to size/route testnets appropriately for local vs Farm execution. |
| rs/tests/system_tests.bzl | Add backend + cpus parameters; generate _local sh_test variants without requires-network and with local-only runtime deps. |
| rs/tests/sdk/src/asset.rs | Make gateway asset fetching work on Local backend via DNS resolve override + self-signed TLS acceptance. |
| rs/tests/sdk/BUILD.bazel | Add local CPU reservations for SDK system-tests. |
| rs/tests/run_systest.sh | Make IC-OS image handling backend-aware: compute hashes locally vs upload+URL on Farm. |
| rs/tests/query_stats/BUILD.bazel | Add local CPU reservations for query-stats system-tests. |
| rs/tests/node/BUILD.bazel | Add CPU reservations; mark known-incompatible tests as backend = "farm" with rationale. |
| rs/tests/nns/sns/BUILD.bazel | Add local CPU reservations for SNS/NNS-related system-tests. |
| rs/tests/nns/nns_dapp_test.rs | Make NNS dapp HTTP fetching work on Local backend via resolve override + self-signed TLS acceptance. |
| rs/tests/nns/BUILD.bazel | Add CPU reservations and backend constraints for NNS tests. |
| rs/tests/networking/nns_delegation_test.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/networking/firewall/firewall_priority_test.rs | Adjust deny-rule prefixes on Local backend to include driver’s fd00::/8 source. |
| rs/tests/networking/firewall/BUILD.bazel | Add local CPU reservations; mark API-BN-playnet-dependent tests as Farm-only. |
| rs/tests/networking/canister_http/canister_http.rs | Simplify UVM IPv4 retrieval (no longer conditional on Farm-only infra provider). |
| rs/tests/nested/sev_recovery.rs | Switch to backend-aware tagged GuestOS disk image selection. |
| rs/tests/nested/nns_recovery/common.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/nested/nns_recovery/BUILD.bazel | Add CPU reservations and Farm-only markers for large nested recovery tests. |
| rs/tests/nested/hostos_upgrade.rs | Use backend-aware HostOS update image URL retrieval. |
| rs/tests/nested/guestos_upgrade.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/nested/BUILD.bazel | Add CPU reservations; mark nested-VM-dependent tests as Farm-only (Local nested VMs not implemented yet). |
| rs/tests/message_routing/xnet/xnet_compatibility.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/message_routing/xnet/BUILD.bazel | Add CPU reservations; mark large xnet SLO tests as Farm-only. |
| rs/tests/message_routing/BUILD.bazel | Add CPU reservations; mark external-download-dependent tests as Farm-only. |
| rs/tests/idx/BUILD.bazel | Add CPU reservations and explain which tests deploy no VMs on local backend. |
| rs/tests/financial_integrations/rosetta/BUILD.bazel | Add local CPU reservations for Rosetta tests (including extra UVMs). |
| rs/tests/financial_integrations/BUILD.bazel | Add local CPU reservations for financial integration tests. |
| rs/tests/execution/BUILD.bazel | Add local CPU reservations for execution tests; note unknown CPU needs for perf test setup. |
| rs/tests/driver/templates/guestos_vm_template.xml | Add libvirt domain template for Local backend VM creation (TAP devices, deterministic PCIe, console append). |
| rs/tests/driver/src/driver/uvms_logs_stream_task.rs | Remove old UVM journald streaming task (replaced by new logging/streaming approach). |
| rs/tests/driver/src/driver/universal_vm.rs | Make Universal VM provisioning work on both Farm and Local backends (upload vs local attach). |
| rs/tests/driver/src/driver/test_setup.rs | Rename InfraProvider → SystemTestBackend and add Local variant. |
| rs/tests/driver/src/driver/test_env_api.rs | Add Local backend group creation, VM control dispatch, and make DNS/cert APIs no-op/unsupported on Local. |
| rs/tests/driver/src/driver/subprocess_task.rs | Forward new --stream-ic-node-logs / --stream-console-logs options to subprocesses. |
| rs/tests/driver/src/driver/prometheus_vm.rs | Use new DiskImage::Url form; add Local backend URL handling (raw IPv6) for Prometheus/Grafana. |
| rs/tests/driver/src/driver/nested.rs | Route resource allocation through backend-aware allocator. |
| rs/tests/driver/src/driver/mod.rs | Register new Local backend + logging + file-serving modules. |
| rs/tests/driver/src/driver/ic.rs | Skip API BN playnet setup on Local backend (no DNS/TLS). |
| rs/tests/driver/src/driver/context.rs | Extend group context with log/console streaming toggles. |
| rs/tests/driver/src/driver/bootstrap.rs | Make image URLs backend-aware; add Local firewall whitelist for fd00::/8; attach/start VMs via Local backend when selected. |
| rs/tests/driver/Cargo.toml | Add dependencies needed for Local backend (askama/axum/virt/rcgen/inotify/tokio-util/etc.). |
| rs/tests/driver/BUILD.bazel | Add new template to compile_data; add new Rust deps and libvirt linkage. |
| rs/tests/dre/guest_os_qualification.rs | Use backend-aware GuestOS image URL retrieval. |
| rs/tests/dre/BUILD.bazel | Add CPU reservations; mark as Farm-only due to size. |
| rs/tests/configure_icos.bzl | Track local_only_icos_images to serve mainnet update images via local file server instead of Farm/CDN. |
| rs/tests/common.bzl | Define MIN_LOCAL_CPUS and DEFAULT_VCPUS_PER_VM for consistent CPU tagging heuristics. |
| rs/tests/ckbtc/BUILD.bazel | Add local CPU reservations for ckBTC/ckDOGE tests (node + bitcoind UVM). |
| rs/tests/BUILD.bazel | Add libvirt runtime package to test container package set. |
| rs/tests/boundary_nodes/BUILD.bazel | Add CPU reservations; mark large perf tests as Farm-only; document “no VMs deployed” cases. |
| rs/tests/consensus/vetkd/BUILD.bazel | Add local CPU reservations for vetKD tests. |
| rs/tests/consensus/upgrade/upgrade_with_alternative_urls.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/consensus/upgrade/common.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/consensus/upgrade/BUILD.bazel | Add CPU reservations; mark known-flaky/slow-on-local upgrade tests as Farm-only with rationale. |
| rs/tests/consensus/subnet_recovery/common.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/consensus/subnet_recovery/BUILD.bazel | Add CPU reservations and mark large recovery tests as Farm-only. |
| rs/tests/consensus/orchestrator/unstuck_subnet_test.rs | Use backend-aware initial/update GuestOS image URL retrieval. |
| rs/tests/consensus/orchestrator/BUILD.bazel | Add CPU reservations; mark external-download-dependent tests as Farm-only. |
| rs/tests/consensus/BUILD.bazel | Add CPU reservations broadly; mark large/perf tests as Farm-only where needed. |
| rs/tests/consensus/backup/common.rs | Use backend-aware GuestOS update image URL retrieval. |
| rs/tests/consensus/backup/BUILD.bazel | Add CPU reservations; mark upgrade-dependent tests as Farm-only due to local-upgrade limitations. |
| rs/tests/crypto/BUILD.bazel | Add CPU reservations; mark API-BN-playnet-dependent test as Farm-only. |
| rs/tests/cross_chain/BUILD.bazel | Add CPU reservations; mark external-download-dependent test as Farm-only with rationale. |
| MODULE.bazel | Vendor Universal/Prometheus VM images via http_file for Local backend use. |
| ci/scripts/targets.py | Exclude local_system_test from PR target inference (run in nightly instead). |
| ci/container/TAG | Update CI container image tag. |
| ci/container/files/packages.common | Install libvirt/qemu/dnsmasq/ovmf/libcap tools needed for Local backend. |
| ci/container/Dockerfile | Bake ic-net-admin capability launcher into the image with narrow networking caps. |
| ci/container/container-run.sh | Add host /dev/kvm group GID as supplemental group at container creation time. |
| Cargo.toml | Add inotify workspace dependency config. |
| Cargo.lock | Lockfile updates for new deps (inotify, windows-sys bumps, etc.). |
| Cargo.Bazel.toml.lock | Bazel cargo lock updates for new deps. |
| Cargo.Bazel.json.lock | Bazel cargo JSON lock updates for new deps. |
| bazel/rust.MODULE.bazel | Add inotify crate spec for Bazel rust toolchain. |
| bazel/resolute.yaml | Add libvirt0 runtime package (for binaries linking libvirt). |
| bazel/mainnet-icos-images.bzl | Download/export GuestOS update image tar for Local backend file-serving. |
| .github/workflows/update-mainnet-canister-revisions.yaml | Bump CI container image digest. |
| .github/workflows/system-tests-benchmarks-nightly.yml | Bump CI container image digest. |
| .github/workflows/schedule-rust-bench.yml | Bump CI container image digest. |
| .github/workflows/schedule-daily.yml | Add local-system-tests job; bump container digest; configure /dev/kvm group access. |
| .github/workflows/salt-sharing-canister-release.yml | Bump CI container image digest. |
| .github/workflows/rosetta-release.yml | Bump CI container image digest. |
| .github/workflows/release-testing.yml | Bump CI container image digest. |
| .github/workflows/rate-limits-backend-release.yml | Bump CI container image digest. |
| .github/workflows/pocket-ic-tests-windows.yml | Bump CI container image digest. |
| .github/workflows/container-scan-nightly.yml | Bump CI container image digest. |
| .github/workflows/container-api-bn-recovery.yml | Bump CI container image digest. |
| .github/workflows/ci-pr-only.yml | Bump CI container image digest. |
| .github/workflows/ci-main.yml | Ensure container has /dev/kvm group access; re-add one _local test as PR smoke test; bump image digest. |
| .github/workflows/api-bn-recovery-test.yml | Bump CI container image digest. |
| .devcontainer/devcontainer.json | Add /dev/kvm group access (numeric GID) for local backend QEMU/KVM usage. |
| .claude/skills/fix-flaky-tests/SKILL.md | Document how to query flaky logs for _local tests (no ES/Farm log downloads). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):
-
Update
unreleased_changelog.md(if there are behavior changes, even if they are
non-breaking). -
Are there BREAKING changes?
-
Is a data migration needed?
-
Security review?
How to Satisfy This Automatic Review
-
Go to the bottom of the pull request page.
-
Look for where it says this bot is requesting changes.
-
Click the three dots to the right.
-
Select "Dismiss review".
-
In the text entry box, respond to each of the numbered items in the previous
section, declare one of the following:
-
Done.
-
$REASON_WHY_NO_NEED. E.g. for
unreleased_changelog.md, "No
canister behavior changes.", or for item 2, "Existing APIs
behave as before.".
Brief Guide to "Externally Visible" Changes
"Externally visible behavior change" is very often due to some NEW canister API.
Changes to EXISTING APIs are more likely to be "breaking".
If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.
If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.
Reference(s)
For a more comprehensive checklist, see here.
GOVERNANCE_CHECKLIST_REMINDER_DEDUP
adamspofford-dfinity
approved these changes
Jun 9, 2026
basvandijk
commented
Jun 10, 2026
tmu0
approved these changes
Jun 10, 2026
Comment on lines
+214
to
+217
| # TODO: figure out the number of vCPUs used by this performance test's setup | ||
| # (Jaeger UVM + an Application subnet of `subnet_size` nodes, all overridden | ||
| # to 16 vCPUs). It is tagged `manual` and not meant to run on the local backend. | ||
| cpus = MIN_LOCAL_CPUS, |
Contributor
There was a problem hiding this comment.
Maybe start off with a reasonable default (13 * 6? 13 * 16 + epsilon?). MIN_LOCAL_CPUS is likely wrong.
Also, shouldn't this run on Farm?
Collaborator
Author
There was a problem hiding this comment.
Yes, will go for:
cpus = MIN_LOCAL_CPUS + 16 + 13 * 16, # 16 vCPUs for Jaeger UVM + 13 IC Node VMs * 16 vCPUs.
…orkload and force the test to be farm-only
mbjorkqvist
approved these changes
Jun 10, 2026
Comment on lines
+39
to
+47
| # TODO: support this test on the local backend (drop `backend = "farm"`). | ||
| # The test currently fails on the local backend because it tries to download from an external URL | ||
| # and we don't allow network access for local tests: | ||
| # | ||
| # thread 'main' (727) panicked at rs/tests/cross_chain/ic_xc_cketh_test.rs:829:58: | ||
| # called `Result::unwrap()` on an `Err` value: block_on_bash_script: exit_status = 1. | ||
| # Output: Err: Error: error sending request for url (https://binaries.soliditylang.org/linux-amd64/list.json) | ||
| # | ||
| # Maybe that artifact can be provided via a bazel dependency. |
Contributor
There was a problem hiding this comment.
Maybe @gregorydemay knows more about this? Doesn't look like it's blocking this PR though.
pierugo-dfinity
approved these changes
Jun 10, 2026
pierugo-dfinity
left a comment
pierugo-dfinity
left a comment
Contributor
There was a problem hiding this comment.
This is great, thanks Bas!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Provide a "local" variant for each system-test which deploys testnets inside the local bazel sandbox (without
sudoand withoutrequires-network) instead of using the external, proprietary and VPN-only Farm web-service which deploys testnets to DFINITY's DCs.To run a system-test locally suffix its name with
_localfor example:The
_localsystem-tests currently don't run as part of thebazel-test-alljob since they reserve a high number of CPUs from bazel causing them to spend most of their time acquiring resources from the single 64-CPU GitHub runner. In total it takes over 3 hours to run them which is obviously too long for PRs. The plan is to run all tests on a Bazel Remote Execution cluster such that they get parallelised over many 64-CPU machines. This is WIP. For the moment we just run the single//rs/tests/idx:basic_health_test_localinbazel-test-allsuch that thelocal_backendcode gets exercised. The remaining jobs run in thelocal-system-testsjob of the nightlyschedule-dailyworkflow.Why?
Status?
143 non farm-only tests are passing with the local backed!
62 tests are configured as farm-only because they're considered too big to fit on the local backend. We could decide to overload runners a bit (like we do with Farm hosts) so that we can run more big tests on the local backend.
18 tests still need to be supported on the local backend.
How?
Architecture
Currently a system-test using the
Farmbackend works like visualised in the following diagram:flowchart TD subgraph k8s_cluster["zh1-idx1"] subgraph runner["github-runner / devenv"] subgraph devc["dev-container"] subgraph bazel["bazel test //rs/tests/..."] subgraph sandbox["linux-sandbox (×N parallel)"] subgraph driver["ic-system-test-driver"] end end end end end bazel_cache["artifacts.zh1-idx1.dfinity.network"] end subgraph aws["AWS"] farm["farm.dfinity.systems"] end subgraph hosts["Farm hosts (dfinity.network)"] subgraph fr1["fr1"] subgraph fr1dll01["fr1-dll01"] libvirtd Q["qemu-system-x86_64 (×N)"] end fr1dll02["fr1-dll02"] end subgraph zh1["zh1"] zh1b["zh1-dll02"] zh1a["zh1-dll01"] end subgraph dm1["dm1"] dm1b["dm1-dll02"] dm1a["dm1-dll01"] end end driver -->|"POST /group/{group-name}/vm/{vm-name}"| farm farm -->|"virsh --connect qemu+ssh://fr1-dll01.fr1.dfinity.network define ..."| fr1dll01 fr1dll01 -->|GET /cas/hash_of_image|bazel_cache Q -->|GET /cas/hash_of_image|bazel_cacheSo the test will make an HTTP requests to the DFINITY-VPN-only Farm web-service in AWS to create a VM. Farm will then pick a suitable host to deploy the VM to and instruct the host to download the image from the remote bazel cache. Farm will then call
virshwhich connects over SSH to thelibvirtdrunning on the host to create the VM.For some tests the IC node running in the VM will do an upgrade and will download another image from the remote bazel cache. It's actually a bit more involved since there's an additional caching layer in front of the bazel cache called the
dc_http_proxywhich is left out of this diagram for simplicity.This PR introduces an additional
Localbackend which moves thelibvirtdrunning on a Farm host to run locally under the test itself. Since images are now locally available they don't need to be downloaded anymore from the bazel cache. IC nodes running in the VM will still need to download images from a web-server during upgrades. For that reason the test runs an internalserve_files_taskweb-server serving the local images:flowchart TD subgraph k8s_cluster["zh1-idx1 / Namespace.so"] subgraph runner["github-runner / devenv"] subgraph devc["dev-container"] subgraph bazel["bazel test //rs/tests/..."] subgraph sandbox["linux-sandbox (×N parallel)"] subgraph driver["ic-system-test-driver"] libvirtd dnsmasq Q["qemu-system-x86_64 (×N)"] serve_files_task end end end end end end classDef green stroke:#008a0e,color:#008a0e,fill:#fff; class libvirtd,dnsmasq,Q,serve_files_task green; Q -->|GET /hash_of_image|serve_files_taskNetworking
Motivation
The Farm backend provisions VMs on a multi-tenant cluster with managed DNS, TLS, and system-mode libvirt networking. The local backend (
local_backend.rs) reproduces just enough of that on a single developer/CI host fully unprivileged — nosudo, no root-ownedlibvirtd, no system-mode libvirt networks.libvirtdruns as the current user in session mode (qemu:///session) and QEMU opens pre-created, user-owned TAPs directly.The only privileged primitive is a narrow capability launcher,
ic-net-admin, asetcap'd copy ofcapshgranting exactlycap_net_admin,cap_net_raw,cap_net_bind_service(never root). It raises those caps into the ambient set andexecs a short shell script, so theip/dnsmasqcommands it runs inherit them across theexec. Everything else is unprivileged.Topology (per test group)
flowchart LR subgraph Host drv["ic-system-test-driver<br/>(sources from mgmt addr)"] lo["lo (all /128)<br/>:1::1 mgmt · :2::1 logs · :3::1 files"] fs["serve_files_task<br/>files:8080"] dq["dnsmasq<br/>RA + DHCPv4 + default route"] br["Linux bridge vbr-xxxx<br/>fd00:AABB:CCDD::1/64 · 10.X.Y.1/24"] drv --- lo fs --- lo dq --- br lo -. "route src override, metric 256" .- br end br === tap1["tap-aaaa"] === vm1["VM A<br/>enp1s0: SLAAC /64<br/>enp2s0: DHCPv4"] br === tap2["tap-bbbb"] === vm2["VM B<br/>enp1s0: SLAAC /64"]1. Per-group Linux bridge and deterministic addressing
Each test group gets its own Linux bridge (
vbr-<hash>, kept within the 15-charIFNAMSIZlimit), created and torn down via the capability launcher increate_group/delete_group. All addresses are derived from a SHA-256 of the group name, so concurrently-running groups get distinct bridges, prefixes, and subnets with no central allocator:/64fd00:AABB:CCDD::/64(ULA)fd00:AABB:CCDD::1/64(the host is the guests' default router via RA, §2)fd00:AABB:CCDD:1::1lofd00:AABB:CCDD:2::1lofd00:AABB:CCDD:3::1lo/2410.X.Y.0/24(RFC 1918)2. SLAAC + Router Advertisements via dnsmasq
The IC GuestOS does not statically configure its global IPv6 address — it brings up link-local only, then derives its deterministic global address via SLAAC from a Router Advertisement:
global = group /64 prefix + EUI-64 of the deterministic MAC. Because the MAC is itself a hash of(group, vm), the driver can compute each VM's IPv6 up front.A minimal per-group
dnsmasq(run through the launcher) supplies this instart_ra_daemon:--dhcp-range=<prefix>,ra-onlyadvertises the on-link, autonomous prefix so guests SLAAC (no stateful IPv6 leases).--ra-param=<bridge>,10,1800sets a non-zero router lifetime, installing the bridge (i.e. the host) as the guests' default router.--dhcp-range=10.X.Y.2,…,254serves stateful DHCPv4 for the optional second NIC.--port=0disables DNS. We might later enable name serving to simulate DNS in tests.3. Per-VM TAP devices
In
start_vm, each VM's TAP (tap-<hash>) is created via the launcher, taggeduser <current-user>so the unprivileged QEMU can open it, and enslaved to the group bridge. The libvirt domain XML references it withmanaged='no'(guestos_vm_template.xml) so libvirt uses the existing device rather than trying to create one (which would need root). MACs and the IPv6 are deterministic, so addressing survives re-runs and reboots.4. Off-
/64management address (the firewall-testing trick)This is the subtlest part. The GuestOS firewall has a built-in accept for the node's own
/64. If the driver reached nodes from an address inside that prefix, the registry-derived deny rules under test would be shadowed and never exercised. So the driver sources host→node traffic from a second/64— the management addressfd00:AABB:CCDD:1::1assigned tolo— which lies outside every node/64but still inside thefd00::/8range the GuestOS bootstrap whitelists.Making this work requires two things set up in
create_group:ip -6 route replace <prefix>/64 dev <bridge> proto kernel metric 256 src <mgmt>, which rewrites the kernel route thataddr addauto-creates so host-originated traffic is sourced from the management address rather than the on-bridge gateway.No IP forwarding or NAT is involved — the management address is on
lo, so traffic to it terminates on the host and is delivered locally.Why three off-
/64addresses, not one? The GuestOS firewall also caps the number of simultaneous connections per source address (max_simultaneous_connections_per_ip_address). The management address is therefore kept exclusively for the test's own host→node traffic: the driver's long-lived auxiliary connections — the per-node journald streams (fd00:AABB:CCDD:2::1; see the Logging section below) and the file server (fd00:AABB:CCDD:3::1, §6) — each bind to their own siblingloaddress so they don't consume a slot in the management address' budget. This matters forfirewall_max_connections_test, which deliberately saturates that budget from the management address and would otherwise race the journald stream for the last connection slot.5. Optional IPv4 second NIC
VMs that request IPv4 get a second TAP on the same bridge (the guest's
enp2s0), pinned to PCIe bus 2 in the domain XML for deterministic interface naming (bus 1 →enp1s0primary, bus 2 →enp2s0). It obtains an address purely via DHCPv4; since the driver always reaches VMs over IPv6, this subnet needs no routing or NAT.6. Per-group file server
There is no external network, so ICOS images that nodes fetch over HTTP (e.g. GuestOS/HostOS update images for upgrade tests) are served by a small web server spawned from the driver (
serve_files_task.rs), listening on the group's dedicated file-server addressfd00:AABB:CCDD:3::1(onlo; see §1 and §4) at fixed portFILE_SERVER_PORT = 8080. Serving from an off-/64address mirrors production, where the web server hosting GuestOS/HostOS images is not on the IC nodes'/64; nodes still reach it via their RA-installed default route (§2), and their download replies are accepted by the firewall's statefulestablished,relatedrule. The port can be fixed because each group has its own (per-group-unique) file-server address, so there's no cross-group contention.7. Isolation and teardown
Per-group hashing plus per-group network namespaces keep concurrent groups from colliding. Teardown in
delete_groupstops the RA daemon, destroys all of the group's domains (releasing their TAPs), deletes every TAP still enslaved to the bridge, removes the bridge, and removes the management, log-streaming and file-server addresses fromlo.What has no local equivalent: managed playnet DNS, TLS issuance, and multi-tenant scheduling — these log a warning and return dummy values.
Logging
Since VMs running via the local system-test backend don't have access to the external network their journald logs can't be streamed to DFINITY's ElasticSearch cluster. Logs of their consoles are also local and not available via Farm's console feature.
However we need access to both to effectively debug tests. For this reason both journald and console logs of each VM are streamed to the the test log which will look like:
Process Management
The main test process is turned into a child-subreaper (think systemd) such that all daemons like
libvirtd,dnsmasq,virtlogdand theqemu-system-x86_64per VM get reparented to the main test process. This allows the main process to kill any remaining childs at finalisation. If we didn't do this these daemons would become children of bazel's child-subreaper (process-wrapper/linux-sandbox) causing the test to never finish since these daemons would keep on running.The process tree of a
bazel test //rs/tests/idx:basic_health_test_localinvocation will look like: