Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions errors/caching-artifacts/cache-ghes-tls-certificate-verify-failed.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
id: caching-artifacts-149
title: 'actions/cache fails on GHES with TLS certificate verification error — unable to verify the first certificate'
category: caching-artifacts
severity: error
tags:
- ghes
- tls
- ssl
- certificate
- self-signed
- enterprise
- node-tls
- cache-restore
patterns:
- regex: 'unable to verify the first certificate'
flags: 'i'
- regex: 'downloadCache failed.*unable to verify'
flags: 'i'
- regex: 'Failed to restore.*certificate.*verify|Failed to save.*certificate.*verify'
flags: 'i'
- regex: 'Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE'
flags: 'i'
- regex: 'RequestError.*certificate.*has expired|RequestError.*self.signed certificate'
flags: 'i'
error_messages:
- 'Warning: Failed to restore: downloadCache failed: unable to verify the first certificate'
- 'Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE'
- 'RequestError: unable to verify the first certificate'
- 'Error: self signed certificate in certificate chain'
root_cause: |
`actions/cache` uses Node.js HTTP clients to communicate with the GitHub Actions cache
service endpoint. On GitHub Enterprise Server (GHES) deployments that use self-signed
TLS certificates or certificates issued by a private Certificate Authority (CA), Node.js
cannot verify the TLS certificate chain by default.

Node.js does not use the system's native OS certificate store on Linux/macOS — it ships
with its own bundled CA bundle. If the GHES instance's TLS certificate is not signed by
a CA in Node.js's bundle (e.g., it's self-signed, or signed by an internal CA not in
the Mozilla bundle), all HTTP requests from the runner's Node.js process will fail with
"unable to verify the first certificate" or "UNABLE_TO_VERIFY_LEAF_SIGNATURE".

This affects:
- `actions/cache@v3`, `actions/cache@v4`, `actions/cache@v5` on GHES
- Any JavaScript action that makes HTTPS requests to the GHES instance
- Occurs on GHES versions that use self-signed certificates or internal CAs
fix: |
Set the `NODE_EXTRA_CA_CERTS` environment variable to point to the CA bundle that includes
the private CA certificate chain. Node.js reads this env var to extend its trusted CA list.

1. Export the GHES TLS certificate chain (PEM format) from your GHES admin console or
via `openssl s_client -showcerts -connect your-ghes-host:443 < /dev/null`

2. Copy the PEM file to the self-hosted runner's filesystem (e.g., `/etc/ssl/ghes-ca.pem`)

3. Set NODE_EXTRA_CA_CERTS in the runner's environment before the cache step runs.

Alternatively, use the `self-signed-certificate` runner configuration option in
actions-runner or set NODE_TLS_REJECT_UNAUTHORIZED=0 (NOT recommended for production
as it disables all TLS verification).
fix_code:
- language: yaml
label: 'Set NODE_EXTRA_CA_CERTS workflow-level to trust GHES self-signed certificate'
code: |
# Configure the custom CA cert at workflow level
env:
# Path to the PEM file containing your GHES private CA certificate
# The file must exist on the self-hosted runner filesystem
NODE_EXTRA_CA_CERTS: /etc/ssl/certs/ghes-ca.pem

jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4

- uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}

- name: Install dependencies
run: npm ci
- language: yaml
label: 'Set NODE_EXTRA_CA_CERTS at the runner level via .env file'
code: |
# In the runner's .env file (on the self-hosted runner host):
# NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ghes-ca.pem
#
# Or set it in the runner service environment. On Linux with systemd:
# sudo systemctl edit actions.runner.<org>.<name>.service
# [Service]
# Environment="NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ghes-ca.pem"
#
# This avoids needing to set it in every workflow file.

jobs:
build:
runs-on: self-hosted
steps:
# NODE_EXTRA_CA_CERTS is set at the runner level — no workflow change needed
- uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
prevention:
- 'Set NODE_EXTRA_CA_CERTS globally in the runner service environment so all JavaScript actions (not just cache) trust the GHES private CA'
- 'Use a publicly trusted TLS certificate (e.g., from Let''s Encrypt) on your GHES instance to avoid Node.js TLS verification issues entirely'
- 'Test connectivity from the runner by running `node -e "const https = require(''https''); https.get(''https://your-ghes-host/'', r => console.log(r.statusCode)).on(''error'', e => console.error(e.message))"` to confirm the cert issue before debugging the workflow'
- 'Document the required NODE_EXTRA_CA_CERTS path in your team''s runner setup guide so new runner deployments are pre-configured'
docs:
- url: 'https://github.com/actions/cache/issues/1286'
label: 'actions/cache#1286 — downloadCache failed: unable to verify the first certificate (GHES)'
- url: 'https://nodejs.org/api/cli.html#node_extra_ca_certsfile'
label: 'Node.js CLI: NODE_EXTRA_CA_CERTS documentation'
- url: 'https://docs.github.com/en/enterprise-server@latest/admin/overview/about-github-enterprise-server'
label: 'GitHub Enterprise Server documentation'
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
id: ku-131
title: 'setup-node mirror input is only a download fallback — node pulled from github.com when version is found in manifest'
category: known-unsolved
severity: limitation
tags:
- setup-node
- mirror
- artifactory
- nexus
- self-hosted
- known-limitation
- node-distribution
- air-gapped
patterns:
- regex: 'Downloading node from.*github\.com|node-versions.*releases.*download'
flags: 'i'
- regex: 'Found in cache.*node|Resolved node version.*from manifest'
flags: 'i'
error_messages:
- '(no error — node is silently downloaded from github.com instead of the configured mirror URL)'
- 'Downloading node from https://github.com/actions/node-versions/releases/download/...'
root_cause: |
The `mirror` and `mirror-token` inputs in `actions/setup-node` are intended to
support authenticated artifact mirrors (e.g., JFrog Artifactory, Sonatype Nexus).
However, the action's resolution logic always consults the GitHub-hosted versions
manifest FIRST. When the requested version is found in the manifest, node is
downloaded directly from `github.com/actions/node-versions` — the mirror URL is
never used.

The mirror is only consulted in these cases:
1. The requested version is NOT present in the GitHub versions manifest (e.g., nightly,
RC versions, custom versions).
2. The primary download from GitHub fails (network error, timeout).

This means that for all stable Node.js releases (which are all in the manifest),
setup-node bypasses the configured mirror entirely. Organizations using mirrors
for security compliance, bandwidth optimization, or air-gapped environments cannot
force setup-node to use their mirror as the primary download source.

This is by design per setup-node maintainers (confirmed in issue #1412): "The mirror
input is intended as an alternative source and is used only as a fallback when the
required version is not present in the manifest or cannot be downloaded."
fix: |
There is no supported way to force setup-node to always download from a mirror for
versions present in the GitHub manifest. The following workarounds exist:

Option A — Block github.com at the network level (forces mirror fallback):
When the primary download fails due to network policy, setup-node falls back to
the mirror automatically. Requires network-level controls outside of GitHub Actions.

Option B — Pre-install node in a custom runner image:
Bundle the required Node.js version into your self-hosted runner or ARC container
image; setup-node will detect the cached version and skip the download entirely.

Option C — Use runner cache with a pre-populated tool cache:
Place node binaries in the runner's RUNNER_TOOL_CACHE directory; setup-node skips
download when the exact version is found in the tool cache.

Track upstream: actions/setup-node#1412 — there is no current ETA for mirror-first support.
fix_code:
- language: yaml
label: 'Current behavior — mirror is configured but node still downloads from github.com'
code: |
- uses: actions/setup-node@v4
with:
node-version: '22.20.0'
mirror: 'https://artifactory.example.com/nodejs-remote'
mirror-token: ${{ secrets.MIRROR_TOKEN }}
# NOTE: For stable versions in the manifest, this still downloads from
# github.com/actions/node-versions — the mirror is NOT used as primary source.
- language: yaml
label: 'Workaround — pre-populate runner tool cache to skip download entirely'
code: |
# On your self-hosted runner or ARC image, pre-download node binaries to:
# $RUNNER_TOOL_CACHE/node/<version>/<arch>/
#
# setup-node checks this directory first and skips network download
# if the exact version+arch is found, regardless of mirror configuration.
prevention:
- 'Do not rely on the mirror input for compliance enforcement — it is a fallback only'
- 'For true mirror-first enforcement, block outbound access to github.com at the network level'
- 'Use pre-built runner images with node bundled in RUNNER_TOOL_CACHE for air-gapped environments'
- 'Track actions/setup-node#1412 for upstream progress on mirror-as-primary support'
docs:
- url: 'https://github.com/actions/setup-node/issues/1412'
label: 'setup-node#1412 — Node distributions not pulled from authenticated mirrors'
- url: 'https://github.com/actions/setup-node#usage'
label: 'setup-node usage documentation — mirror input'
125 changes: 125 additions & 0 deletions errors/known-unsolved/ubuntu-arm64-chrome-edge-not-preinstalled.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
id: known-unsolved-141
title: 'Chrome and Edge are not preinstalled on Ubuntu ARM64 runners (ubuntu-22.04-arm, ubuntu-24.04-arm, ubuntu-26.04-arm)'
category: known-unsolved
severity: limitation
tags:
- arm64
- chrome
- chromium
- edge
- selenium
- playwright
- ubuntu-arm
- browser-testing
patterns:
- regex: 'cannot find Chrome binary'
flags: 'i'
- regex: 'CHROMEWEBDRIVER.*?is not set|CHROMEWEBDRIVER.*?empty'
flags: 'i'
- regex: 'google-chrome.*?command not found|google-chrome-stable.*?not found'
flags: 'i'
- regex: 'chromedriver.*?not found|no such file.*?chromedriver'
flags: 'i'
- regex: 'ubuntu-(22|24|26)\.04-arm'
flags: 'i'
error_messages:
- 'cannot find Chrome binary'
- 'google-chrome: command not found'
- 'google-chrome-stable: not found'
- 'Error: Failed to launch the browser process! /usr/bin/google-chrome-stable'
- 'WebDriverException: Message: ''chromedriver'' executable needs to be in PATH'
- 'SessionNotCreatedException: Message: chrome not reachable'
root_cause: |
Ubuntu ARM64 GitHub Actions hosted runners (ubuntu-22.04-arm, ubuntu-24.04-arm,
ubuntu-26.04-arm) do NOT include Google Chrome, Chromium, Microsoft Edge, or their
respective WebDrivers (ChromeDriver, EdgeDriver) as preinstalled software.

On x86-64 Ubuntu runners (ubuntu-22.04, ubuntu-24.04, ubuntu-26.04), Chrome, ChromeDriver,
Edge, and EdgeDriver are all preinstalled and the environment variables
CHROMEWEBDRIVER and EDGEWEBDRIVER point to valid paths.

On ARM64 Ubuntu runners, the corresponding environment variables are set to empty strings:
CHROMEWEBDRIVER = ""
EDGEWEBDRIVER = ""

Only Mozilla Firefox (with Geckodriver) is preinstalled on Ubuntu ARM64 runners.

This is a known, intentional limitation of the Ubuntu ARM64 runner images as of
June 2026. Google does not publish official Chrome or ChromeDriver builds for
Linux ARM64 via their standard Debian package repository. Chromium for Linux ARM64
is available but is not included in the runner image.

Workflows that migrate from ubuntu-24.04 (x86) to ubuntu-24.04-arm or ubuntu-26.04-arm
and rely on Chrome-based browser testing will fail silently or with the above errors.
fix: |
There is no fully equivalent fix because Google does not officially support Chrome
for Linux ARM64 via the standard deb package. Available mitigations:

1. Use Firefox instead of Chrome for browser tests on ARM64 runners. Firefox is
preinstalled. Update your test framework to target Firefox when running on ARM64.

2. Install Chromium (unofficial ARM64 build from apt) and use chromium-driver:
sudo apt-get install -y chromium-browser chromium-driver
Note: Chromium from apt on Ubuntu may be a snap package that does not work in
headless CI environments. Test carefully.

3. Use Playwright with its bundled browser download:
npx playwright install chromium
Playwright distributes its own Chromium builds for Linux ARM64.

4. Run browser tests only on x86-64 runners and use ARM64 runners for non-browser workloads.
Use a matrix strategy to select the appropriate runner per test type.

5. Use a container image that pre-bundles Chrome for ARM64 (e.g., some Selenium
Docker images have ARM64 variants with Chromium).
fix_code:
- language: yaml
label: 'Use matrix to run browser tests on x86 and unit tests on ARM64'
code: |
jobs:
test:
strategy:
matrix:
include:
# Browser tests: x86 only (Chrome available)
- runner: ubuntu-24.04
test-type: browser
# Unit tests: ARM64 (faster, no Chrome needed)
- runner: ubuntu-24.04-arm
test-type: unit
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- if: matrix.test-type == 'browser'
name: Run browser tests (Chrome available on x86)
run: npm run test:e2e
- if: matrix.test-type == 'unit'
name: Run unit tests
run: npm run test:unit
- language: yaml
label: 'Install Chromium via Playwright on ARM64 runners'
code: |
jobs:
test:
runs-on: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
- run: npm ci
# Playwright manages its own Chromium builds including Linux ARM64
- run: npx playwright install chromium --with-deps
- run: npx playwright test
prevention:
- 'Check that your target ARM64 runner image supports the browser you need before migrating; run `echo $CHROMEWEBDRIVER` in a test step to verify'
- 'Use Playwright instead of Selenium/WebDriver for cross-architecture browser tests — Playwright distributes bundled Chromium for Linux ARM64'
- 'Separate browser tests from unit tests in your CI matrix so ARM64 runners handle non-browser workloads only'
- 'Monitor actions/runner-images for announcements about Chrome ARM64 support on GitHub-hosted runners'
docs:
- url: 'https://github.com/actions/runner-images/releases/tag/ubuntu24-arm64%2F20260607.22'
label: 'ubuntu-24.04-arm64 runner image release notes — Browsers and Drivers section'
- url: 'https://github.com/actions/runner-images/releases/tag/ubuntu26-arm64%2F20260604.22'
label: 'ubuntu-26.04-arm64 runner image release notes — CHROMEWEBDRIVER is empty'
- url: 'https://playwright.dev/docs/ci'
label: 'Playwright CI documentation — supports Linux ARM64 via bundled browser download'
Loading
Loading