Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
f47cdb4
Configure webhook delivery options
niemyjski Oct 13, 2026
537354e
Preserve webhook destinations and exception diagnostics
niemyjski Oct 13, 2026
08c6d4f
Simplify webhook delivery test repository mocks
niemyjski Oct 13, 2026
ae5012b
Preserve safe webhook diagnostics and pooled client lifetimes
niemyjski Oct 14, 2026
86a229f
Remove webhook delivery doc and use JSON pattern matching
niemyjski Oct 14, 2026
0a1fe69
Prevent OTel context propagation to webhook receivers
niemyjski Oct 14, 2026
54fd0c0
Simplify webhook disposal and reuse shared test logging
niemyjski Oct 14, 2026
47f8904
Configure trusted forwarded header peers
niemyjski Oct 13, 2026
9679428
Clarify forwarded header compatibility and align code style
niemyjski Oct 13, 2026
830924d
Support disabling forwarded headers and cover proxy boundaries
niemyjski Oct 13, 2026
3759784
Merge main after webhook delivery merge into forwarded headers
niemyjski Oct 15, 2026
5a72699
Validate forwarded peer settings and clarify hosting migration
niemyjski Oct 15, 2026
2b5137c
Discover gateway trust during dev and prod deployment
niemyjski Oct 15, 2026
a8368d3
Remove CI tooling tests and use JavaScript modules by default
niemyjski Oct 16, 2026
4c00198
Merge remote-tracking branch 'origin/main' into feature/update-stack-…
niemyjski Oct 16, 2026
370d284
Harden and clarify gateway trust discovery
niemyjski Oct 16, 2026
6a79c55
Harden forwarded-header discovery output
niemyjski Oct 16, 2026
d48b97a
Enforce script braces and reject unrestricted forwarding
niemyjski Oct 16, 2026
7408a77
Use JavaScript module extensions and shared script linting
niemyjski Oct 16, 2026
0402e2c
Merge local main lint cleanup and reuse shared ESLint configuration
niemyjski Oct 16, 2026
8c88687
Sync amended local main and remove root package manifest
niemyjski Oct 16, 2026
711caec
Limit pull request to forwarded header security changes
niemyjski Oct 16, 2026
8addb15
Group forwarded header settings and document framework keys
niemyjski Oct 16, 2026
0e7e925
Clarify explicit forwarded header configuration binding
niemyjski Oct 16, 2026
6c3d290
Separate trusted proxy options from middleware configuration
niemyjski Oct 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,8 @@ jobs:
if: ${{ needs.version.outputs.is_dev_deploy == 'true' }}
run: |
sed -i "s/^appVersion:.*$/appVersion: '${VERSION}'/" ./k8s/exceptionless/Chart.yaml
helm upgrade --set "version=${VERSION}" --reuse-values --values ./k8s/ex-dev-values.yaml ex-dev --namespace $DEV_NAMESPACE ./k8s/exceptionless
node k8s/scripts/discover-forwarded-headers.js ex-dev k8s/ex-dev-values.yaml "$RUNNER_TEMP/forwarded-headers.json"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pin Node 24 before running the discovery script

The deploy job has no actions/setup-node step, unlike the other jobs that explicitly select Node 24. The new .js script uses top-level ESM imports, but the repository has no ancestor package.json declaring "type": "module"; Node's package documentation notes that syntax detection for ambiguous .js files became enabled by default only in Node 22.7. On a runner exposing Node 20 or older, both this development invocation and the production invocation fail with Cannot use import statement outside a module before Helm runs, so the deploy job should select Node 24 explicitly.

Useful? React with 👍 / 👎.

helm upgrade --set "version=${VERSION}" --reuse-values --values ./k8s/ex-dev-values.yaml --values "$RUNNER_TEMP/forwarded-headers.json" ex-dev --namespace $DEV_NAMESPACE ./k8s/exceptionless

- name: Run Development Migrations
if: ${{ needs.version.outputs.is_dev_deploy == 'true' }}
Expand Down Expand Up @@ -842,7 +843,8 @@ jobs:
if: ${{ needs.version.outputs.is_prod_deploy == 'true' }}
run: |
sed -i "s/^appVersion:.*$/appVersion: '${VERSION}'/" ./k8s/exceptionless/Chart.yaml
helm upgrade --set "version=${VERSION}" --reuse-values --values ./k8s/ex-prod-values.yaml ex-prod --namespace ex-prod ./k8s/exceptionless
node k8s/scripts/discover-forwarded-headers.js ex-prod k8s/ex-prod-values.yaml "$RUNNER_TEMP/forwarded-headers.json"
helm upgrade --set "version=${VERSION}" --reuse-values --values ./k8s/ex-prod-values.yaml --values "$RUNNER_TEMP/forwarded-headers.json" ex-prod --namespace ex-prod ./k8s/exceptionless

- name: Summary
if: ${{ always() }}
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/self-hosting/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ title: "Docker"

# Docker

If you place a reverse proxy in front of the container, configure its [trusted forwarding peers](/docs/self-hosting/#reverse-proxies-and-https) before upgrading. The direct-port examples below do not require proxy trust settings.

If you would like to test Exceptionless locally, please follow this section.

## Requirements
Expand Down
40 changes: 40 additions & 0 deletions docs/docs/self-hosting/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,46 @@ You can also use Kubernetes while self-hosting Exceptionless. We'll cover both t
* [Kubernetes](/docs/self-hosting/kubernetes)
* [Upgrading](/docs/self-hosting/upgrading-self-hosted-instance)

## Reverse proxies and HTTPS

Direct HTTP/HTTPS hosting needs no forwarded-header configuration. The API accepts `X-Forwarded-For` and `X-Forwarded-Proto` only from loopback or explicitly trusted proxies. A known proxy is the **source IP of the connection reaching the API**, not its hostname, public frontend IP, Kubernetes Service IP, or proxy identifier. For Docker bridges or NAT, verify the address seen inside the API container.

Before upgrading from unrestricted forwarding, configure every non-loopback proxy that must supply client addresses or HTTPS schemes. Otherwise requests use the proxy IP and backend scheme: unrelated users can share IP-based limits and HTTPS detection can be incorrect. Set the public website `EX_BaseURL` and API `EX_ApiUrl` separately; forwarded headers do not configure public origins.

Use this section in the web application's `appsettings.Local.yml`, replacing the example address with the verified proxy source IP:

```yaml
ForwardedHeaders:
Enabled: true
ForwardLimit: 1
KnownProxies:
- 192.0.2.10
```

The equivalent Docker Compose `environment` entries, or Helm values under the chart's existing `config` map, are:

```yaml
EX_ForwardedHeaders__Enabled: "true"
EX_ForwardedHeaders__ForwardLimit: "1"
EX_ForwardedHeaders__KnownProxies__0: "192.0.2.10"
```

For changing proxy addresses, use `KnownNetworks` **instead of** `KnownProxies` with a verified, dedicated proxy CIDR, for example `EX_ForwardedHeaders__KnownNetworks__0: "192.0.2.0/24"`. Do not trust the whole cluster, Docker bridge, or private address space when other workloads can connect from it. Use network access controls to restrict backend access. IPv4 and IPv6 addresses/CIDRs are supported; IPv4 entries also match IPv4-mapped IPv6 peers. Use canonical network CIDRs; hostnames, ports, comma-separated scalar lists, malformed entries, and `/0` networks are rejected at startup.

All supported Exceptionless settings belong under `ForwardedHeaders`. `Enabled` defaults to `true`; set `ForwardedHeaders:Enabled` to `false` (environment variable `EX_ForwardedHeaders__Enabled=false`) to leave client addresses and schemes unchanged. Exceptionless reads and validates this section once at startup into `TrustedForwardedHeadersOptions`, then configures ASP.NET Core's `ForwardedHeadersOptions` from those typed values through [.NET's options registration](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/options?view=aspnetcore-10.0). ASP.NET Core does not automatically bind this section. The group and `Enabled` switch are Exceptionless settings.

`ForwardLimit` and `KnownProxies` match [ASP.NET Core's forwarded-header options](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-10.0#forwarded-headers-middleware-options). The `KnownNetworks` configuration key retains the older framework name; Exceptionless parses its CIDR strings into .NET 10's [`KnownIPNetworks` property](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.builder.forwardedheadersoptions.knownipnetworks?view=aspnetcore-10.0). Both address lists are parsed and validated explicitly while preserving the framework's loopback defaults. Boolean settings use .NET's typed `GetValue` conversion; malformed booleans fail startup.

`ForwardLimit` defaults to one entry, processed from right to left. Multiple proxies require a finite limit matching the intended chain and trust for each traversed peer. Processing stops at an unknown peer even with unused hop budget. Zero also disables processing for compatibility; negative or non-integer limits are invalid.

Do not set the separate framework variable `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true`. [Microsoft documents that this shortcut clears the trusted-peer lists](https://learn.microsoft.com/en-us/dotnet/core/compatibility/aspnet-core/8.0/forwarded-headers-unknown-proxies). ASP.NET Core exposes it internally as [`ForwardedHeaders_Enabled`](https://github.com/dotnet/aspnetcore/blob/release/10.0/src/DefaultBuilder/src/ForwardedHeadersOptionsSetup.cs#L21); Exceptionless rejects the active shortcut at startup. This framework key is not part of the Exceptionless configuration group and cannot replace `ForwardedHeaders:Enabled`.

The public edge proxy must derive the scheme from its connection and replace untrusted forwarding headers, or append the observed client address so the configured hop limit cannot reach user-supplied entries. Configure Nginx, Caddy, Traefik, or other proxies accordingly. Loopback proxies and IIS out-of-process integration normally need no additional trust entries. A remote IIS/ARR proxy needs the same explicit peer configuration as other remote proxies.

Environment variables override YAML by key; command-line values override both. Use double underscores for nesting and numbered entries for lists. Overriding index `0` does not remove other entries inherited from YAML; remove obsolete trust entries from their source and restart after changes. Avoid defining the same key with both prefixed and unprefixed environment names.

The API processes IP/scheme headers before redirect, authentication, and throttling middleware. It does not consume `X-Forwarded-Host` or `X-Forwarded-Prefix`; preserve the public Host and root paths. Verify HTTPS, client IPs, redirects, and IP-based limits through the actual proxy chain before rollout. Enable `EX_Serilog__MinimumLevel__Override__Microsoft.AspNetCore.HttpOverrides: "Debug"` temporarily to diagnose unknown peers. See [Microsoft's proxy guidance](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-10.0).

---

[Next > Docker](/docs/self-hosting/docker)
32 changes: 31 additions & 1 deletion docs/docs/self-hosting/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,37 @@ The following section will cover how to configure exceptionless inside of a Kube

All configuration options and settings can be found in the various option classes located [here](https://github.com/exceptionless/Exceptionless/tree/master/src/Exceptionless.Core/Configuration).

_Please note that if you are specifying configuration via `docker-compose`, then you will need to drop the `EX_` and `EX_ConnectionStrings__` prefixes._
The same `EX_` environment keys work in Docker Compose. Nested keys use double underscores.

### Trusted ingress peers

Before deploying behind ingress, configure [trusted forwarding peers](/docs/self-hosting/#reverse-proxies-and-https) through the Helm chart's existing `config` map. For example, after replacing the documentation address with a verified proxy source address:

```yaml
config:
EX_ForwardedHeaders__KnownProxies__0: "192.0.2.10"
EX_ForwardedHeaders__Enabled: "true"
EX_ForwardedHeaders__ForwardLimit: "1"
```

The repository's ingress templates use Azure Application Gateway for Containers, terminate TLS at ingress, and route `/` to the API Service on port 80 and container port 8080. The hosted development and production deployment workflow discovers the gateway's association subnet before each Helm upgrade, so subnet addresses do not need to be duplicated in environment values files.

The shared `k8s/scripts/discover-forwarded-headers.js` script renders the environment's API ingress locally, reads the selected `ApplicationLoadBalancer` through Kubernetes, and resolves its association subnet IDs through Azure. It requires the existing deployment identity to have Kubernetes read access to that resource and Azure `Microsoft.Network/virtualNetworks/subnets/read` access to its subnets. Missing associations, failed reads, missing `Microsoft.ServiceNetworking/trafficControllers` delegation, invalid prefixes, and unrestricted `/0` prefixes stop deployment.

Discovery writes a `forwardedHeaders.knownNetworks` Helm list. The chart converts it into indexed `EX_ForwardedHeaders__KnownNetworks__*` keys, enables forwarding, and sets a forward limit of one for both application hosts. Replacing the list removes obsolete subnet entries even with `--reuse-values`. Combining discovery with manual `ForwardedHeaders` entries in `config` is rejected; remove those old entries from saved release values before using discovery.

For manual upgrades, run discovery from the repository root after selecting the intended Kubernetes context and Azure subscription, then pass the generated file **last** to Helm. For example, the development commands are:

```sh
node k8s/scripts/discover-forwarded-headers.js ex-dev k8s/ex-dev-values.yaml /tmp/ex-dev-forwarded-headers.json
# Run only if discovery succeeded:
helm upgrade ex-dev k8s/exceptionless --namespace ex-dev --reuse-values \
--values k8s/ex-dev-values.yaml --values /tmp/ex-dev-forwarded-headers.json
```

For production, use `ex-prod`, `k8s/ex-prod-values.yaml`, and a separate output file. The script only reads infrastructure and replaces the local values file; Helm performs the upgrade. It removes any previous output before discovery so a failed refresh cannot leave stale trust available for reuse. Automated workflows stop on discovery failure; manual operators must also stop.

This assumes the gateway connects directly to the application. Verify the API-visible peer and any intermediate proxies before the first rollout; discovering Azure configuration does not prove the live traffic path. The ALB controller pod and Service address are not substitutes for the association subnet. Other ingress topologies should use explicit trusted peers in `config` as shown above. See Microsoft's [association subnet documentation](https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/application-gateway-for-containers-components).

## ConnectionStrings

Expand Down
4 changes: 4 additions & 0 deletions docs/docs/self-hosting/upgrading-self-hosted-instance.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ title: "Upgrading"

# Upgrading

## Trusted proxy migration

Upgrading from unrestricted forwarding requires [explicit trust for non-loopback reverse proxies](/docs/self-hosting/#reverse-proxies-and-https). Configure and verify the actual ingress peers before rollout, or client-IP limits and HTTPS detection can change. Direct hosting needs no new settings.

**Please ensure that you have created backups before upgrading!**

**If you are upgrading from v1 or [v2](https://github.com/exceptionless/Exceptionless/releases/tag/v2.0.0) you will need to upgrade to [v3.0](https://github.com/exceptionless/Exceptionless/releases/tag/v3.0.0) before upgrading to the latest release.**
Expand Down
13 changes: 13 additions & 0 deletions k8s/exceptionless/templates/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@ data:
{{- end }}
EX_DisabledPlugins: GeoPlugin,LocationPlugin
EX_EnableSsl: "true"
{{- $knownNetworks := dig "forwardedHeaders" "knownNetworks" (list) .Values.AsMap }}
{{- if $knownNetworks }}
{{- range $key, $value := .Values.config }}
{{- if hasPrefix "forwardedheaders:" (lower (replace "__" ":" (trimPrefix "EX_" $key))) }}
{{- fail "Do not combine discovered forwardedHeaders.knownNetworks with ForwardedHeaders config overrides; remove the old overrides" }}
{{- end }}
{{- end }}
EX_ForwardedHeaders__Enabled: "true"
EX_ForwardedHeaders__ForwardLimit: "1"
{{- range $index, $network := $knownNetworks }}
EX_ForwardedHeaders__KnownNetworks__{{ $index }}: {{ $network | quote }}
{{- end }}
{{- end }}
EX_ConnectionStrings__Elasticsearch: {{ default "server=http://elasticsearch:9200" .Values.elasticsearch.connectionString }}
{{- if (empty .Values.redis.connectionString) }}
EX_ConnectionStrings__Redis: {{ template "exceptionless.fullname" . }}-redis-master,abortConnect=false
Expand Down
5 changes: 5 additions & 0 deletions k8s/exceptionless/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ jobs:
config: {}
# key: value

# Populated by k8s/scripts/discover-forwarded-headers.js for hosted deployments.
# A replacement list prevents stale subnet entries surviving --reuse-values.
forwardedHeaders:
knownNetworks: []

ingress:
enabled: false
clusterIssuer: cert-manager-webhook-dnsimple-production
Expand Down
Loading
Loading