Add self-hosted Grafana by theseanything · Pull Request #2287 · govuk-forms/forms-deploy

theseanything · 2026-09-24T06:16:51Z

This PR adds Grafana to give GOV.UK Forms a central tool for visualising data.

We'll use it to show our key performance indicators (KPIs) and to centralise our observability stack, so there are fewer places to look during an incident.

As agreed in the ADR: govuk-forms/forms#264

Self-hosted rather than Amazon Managed Grafana (AMG)

AMG only supports the previous major version of Grafana.
It requires SSO through IAM Identity Center, which is managed by EE and isn't ready for us yet.
Most other teams self-host Grafana and use GitHub sign-in, so we're following the same pattern.

At our team size, the costs are roughly the same. We can switch to AMG once Identity Center is set up and AWS supports Grafana v13.

What this adds

Grafana OSS on Fargate in the forms-<env> cluster, behind its own public ALB at grafana.<root_domain>. HTTP redirects to HTTPS.
An Aurora Serverless v2 PostgreSQL database that can scale to zero, reachable only from the Grafana task.
GitHub sign-in, limited to govuk-forms. govuk-forms-infrastructure members are admins and govuk-forms-devs members are editors. We'll add the rest of the team as viewers later.
CloudWatch, X-Ray and CloudWatch PromQL data sources, which authenticate with the task role.
Deployer permissions and new outputs from the environment root.

This has been tested and verified in dev.

This only switches Grafana on in dev. A follow-up PR will enable it in staging and production.

Deploying to dev

Apply the roots in order:

make dev deployer_role forms/account apply
make dev deployer_role forms/environment apply
make dev deployer_role forms/health apply
make dev deployer_role forms/dns apply

Create a GitHub OAuth App in govuk-forms with these URLs:
- homepage: https://grafana.dev.forms.service.gov.uk
- callback: https://grafana.dev.forms.service.gov.uk/login/github

Put its credentials into SSM, then restart the service:

aws ssm put-parameter --region eu-west-2 --overwrite --type SecureString \
  --name /grafana/github/client-id --value '<client id>'
aws ssm put-parameter --region eu-west-2 --overwrite --type SecureString \
  --name /grafana/github/client-secret --value '<client secret>'
aws ecs update-service --region eu-west-2 \
  --cluster forms-dev --service grafana --force-new-deployment

The break-glass admin password is in /grafana/admin-password. Its login form is at /login?disableAutoLogin=true.

Copilot

Copilot review overview

🔵 Needs a closer look

Four moderate issues affect deployment reliability, configuration, permissions, and safe teardown.

Review effort: Balanced
Findings: None

What changed in this PR

Adds self-hosted Grafana on ECS/Fargate with Aurora PostgreSQL, GitHub authentication, AWS observability data sources, and environment/DNS integration.

Changes:

Adds Grafana infrastructure, networking, IAM, logging, secrets, ALB, and database resources.
Adds deployment permissions, DNS, environment outputs, and ACM validation support.
Enables Grafana in development while leaving staging and production disabled.

Required changes:

infra/deployments/forms/health/grafana.tf:2 — Moderate (1 vote): Document or redesign the one-way feature flag because protected resources prevent disabling it.
infra/modules/grafana/ecs.tf:135 — Moderate (1 vote): Add an explicit HTTPS listener dependency to avoid first-deployment ordering failures.
infra/modules/grafana/ecs.tf:31 — Moderate (1 vote): Join allowed_organizations with commas rather than spaces.
infra/modules/grafana/iam.tf:64 — Moderate (1 vote): Grant logs:DescribeLogStreams for CloudWatch log browsing and queries.

File	Description
`infra/modules/grafana/variables.tf`	Defines Grafana module inputs.
`infra/modules/grafana/security-groups.tf`	Configures ALB, ECS, and database network access.
`infra/modules/grafana/README.md`	Documents setup and operations.
`infra/modules/grafana/rds.tf`	Provisions Aurora PostgreSQL.
`infra/modules/grafana/providers.tf`	Declares provider requirements.
`infra/modules/grafana/parameters.tf`	Configures secrets and credentials.
`infra/modules/grafana/outputs.tf`	Exposes ALB details.
`infra/modules/grafana/main.tf`	Defines shared module values.
`infra/modules/grafana/logging.tf`	Configures application logging.
`infra/modules/grafana/iam.tf`	Defines task and data-source permissions.
`infra/modules/grafana/ecs.tf`	Defines the Grafana task and ECS service.
`infra/modules/grafana/datasources.tf`	Provisions AWS data sources.
`infra/modules/grafana/alb.tf`	Creates the public HTTPS ALB and certificate.
`infra/modules/environment/outputs.tf`	Exposes networking and logging outputs.
`infra/modules/environment/alb.tf`	Permits Grafana ALB log delivery.
`infra/modules/deployer-access/policy.tf`	Grants Grafana deployment permissions.
`infra/modules/acm-cert-with-dns-validation/variables.tf`	Adds configurable validation-zone input.
`infra/modules/acm-cert-with-dns-validation/main.tf`	Uses the configured certificate validation zone.
`infra/deployments/forms/tfvars/staging.tfvars`	Keeps Grafana disabled in staging.
`infra/deployments/forms/tfvars/production.tfvars`	Keeps Grafana disabled in production.
`infra/deployments/forms/tfvars/dev.tfvars`	Enables Grafana in development.
`infra/deployments/forms/inputs.tf`	Defines deployment-level Grafana settings.
`infra/deployments/forms/health/outputs.tf`	Exposes Grafana ALB outputs.
`infra/deployments/forms/health/grafana.tf`	Conditionally instantiates Grafana.
`infra/deployments/forms/health/dependencies.tf`	Reads required account state.
`infra/deployments/forms/environment/outputs.tf`	Forwards environment outputs.
`infra/deployments/forms/dns/grafana.tf`	Creates the Grafana DNS alias.
`infra/deployments/forms/dns/dependencies.tf`	Reads health state for DNS.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

whi-tw

Other than a few nitpicks, lgtm

The Grafana hostname has no hosted zone of its own.

Grafana's load balancer in the health root needs both.

Copilot

Copilot review overview

🟡 Changes recommended

ECS service creation needs an explicit dependency on the ALB listener to avoid nondeterministic fresh-deployment failures.

Review effort: Balanced
Findings: 1

Open (1)

Add dependency on HTTPS listener before creating ECS service · New

The other ECS services name their task and execution roles and policies `<environment>-<application>-...`. Do the same for Grafana so the deployer policy is consistent.

ECS rejects a service whose target group is not attached to a load balancer. The service only referenced the target group, so in a new environment Terraform could create it before the HTTPS listener had made that attachment.

Copilot

Copilot review overview

🔵 Needs a closer look

It introduces a public authentication-enabled service and coordinated IAM, database, networking, DNS, and cross-root Terraform changes.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Add dependency on HTTPS listener before creating ECS service

whi-tw

LGTM

theseanything marked this pull request as ready for review September 24, 2026 06:39

theseanything requested review from a team and a balanced review from Copilot September 24, 2026 06:39

Copilot started reviewing on behalf of theseanything September 24, 2026 06:40 View session

Copilot AI reviewed Sep 24, 2026

View reviewed changes

whi-tw requested changes Oct 6, 2026

View reviewed changes

Comment thread infra/modules/grafana/datasources.tf

Comment thread infra/modules/deployer-access/policy.tf Outdated

Comment thread infra/modules/deployer-access/policy.tf Outdated

Comment thread infra/modules/grafana/alb.tf

theseanything added 10 commits October 6, 2026 12:35

Allow certificate module to use a parent zone

3329307

The Grafana hostname has no hosted zone of its own.

Expose public subnets and ALB logs bucket

21e5118

Grafana's load balancer in the health root needs both.

Allow deployer role to manage Grafana resources

c797c67

Add Grafana database and secrets

c6fcb7a

Add Grafana IAM roles

c48eab8

Add Grafana load balancer, networking and logging

1fcfb4c

Add Grafana ECS service

3aacf8e

Add Grafana settings for each environment

30f9f76

Deploy Grafana from the health root

58b6b59

Add Grafana DNS record

f05f0ad

Copilot AI balanced review requested due to automatic review settings October 6, 2026 11:44

theseanything force-pushed the grafana-deploy branch from eef0ea0 to f05f0ad Compare October 6, 2026 11:44

Copilot started reviewing on behalf of theseanything October 6, 2026 11:45 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread infra/modules/grafana/ecs.tf

theseanything added 2 commits October 6, 2026 13:01

Prefix Grafana IAM names with the environment

face9f6

The other ECS services name their task and execution roles and policies `<environment>-<application>-...`. Do the same for Grafana so the deployer policy is consistent.

Create the Grafana service after its listener

2c371f2

ECS rejects a service whose target group is not attached to a load balancer. The service only referenced the target group, so in a new environment Terraform could create it before the HTTPS listener had made that attachment.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 12:02

Copilot started reviewing on behalf of theseanything October 6, 2026 12:02 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

whi-tw approved these changes Oct 6, 2026

View reviewed changes

theseanything added this pull request to the merge queue Oct 6, 2026

Merged via the queue into main with commit 19af157 Oct 6, 2026
24 checks passed

theseanything deleted the grafana-deploy branch October 6, 2026 12:14

theseanything mentioned this pull request Oct 6, 2026

Enable Grafana in staging and production #2306

Merged

Navigation Menu

Repository navigation

Add self-hosted Grafana - #2287

Add self-hosted Grafana#2287
theseanything merged 12 commits into
mainfrom
grafana-deploy

theseanything commented Sep 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

whi-tw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

whi-tw left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

theseanything commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Self-hosted rather than Amazon Managed Grafana (AMG)

What this adds

Deploying to dev

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

whi-tw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

whi-tw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theseanything commented Sep 24, 2026 •

edited

Loading