Terraform to spin up a PDS

.github/workflows/terraform-checks.yml

@@ -0,0 +1,102 @@
+name: Terraform Checks
+
+# fmt/validate for every root in the matrix below on every PR; a dev plan (job
+# summary) for infra/gcp on same-repo PRs. NO workflow applies terraform —
+# applies are human-run (each root's README.md).
+on:
+  pull_request:
+    paths:
+      - infra/gcp/**
+      - .github/workflows/terraform-checks.yml
+      - .github/actions/setup-terraform/**
+  push:
+    branches:
+      - main
+    paths:
+      - infra/gcp/**
+      - .github/workflows/terraform-checks.yml
+      - .github/actions/setup-terraform/**
+
+env:
+  # Keep in lockstep with required_version in each matrix root's versions.tf.
+  TF_VERSION: 1.16.0
+
+jobs:
+  fmt_and_validate:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        root:
+          - infra/gcp
+          - infra/gcp/dns
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup-terraform
+        with:
+          terraform_version: ${{ env.TF_VERSION }}
+      - name: Check formatting
+        run: terraform -chdir=${{ matrix.root }} fmt -check -recursive
+      - name: Validate
+        run: |
+          terraform -chdir=${{ matrix.root }} init -backend=false -input=false
+          terraform -chdir=${{ matrix.root }} validate
+
+  plan_dev:
+    needs: fmt_and_validate
+    # Upstream, same-repo PRs only: the terraform-plan environment holding the
+    # planner's GCP_SERVICE_ACCOUNT_KEY lives in codeforboston/maple, and fork
+    # PRs get no secrets either way. (Not `dev`: that environment's deployment
+    # branch policy allows only main, so a PR job bound to it can never start.)
+    if: github.repository_owner == 'codeforboston' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    environment: terraform-plan
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/actions/setup-terraform
+        with:
+          terraform_version: ${{ env.TF_VERSION }}
+          # No wrapper: we redirect plan stdout to a file ourselves.
+          terraform_wrapper: "false"
+      - uses: google-github-actions/auth@v3
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+      - uses: google-github-actions/setup-gcloud@v3
+      - name: Check for the state bucket
+        id: bootstrap
+        # Gate on the actual precondition rather than swallowing init errors:
+        # only a missing bucket (not yet bootstrapped, the runbook in
+        # infra/gcp/README.md) skips the plan. A 403 means the planner lacks
+        # its grant (ci_planner in envs/dev.tfvars, iam.tf) and is red, not
+        # "skipped" forever; so is any other failure.
+        run: |
+          bucket=$(sed -n 's/^bucket *= *"\(.*\)".*/\1/p' infra/gcp/envs/dev.gcs.tfbackend)
+          if err=$(gcloud storage ls "gs://$bucket" 2>&1 >/dev/null); then
+            echo "ok=true" >> "$GITHUB_OUTPUT"
+          elif grep -q "404\|NotFound\|not found" <<< "$err"; then
+            echo "ok=false" >> "$GITHUB_OUTPUT"
+            echo "state bucket gs://$bucket not bootstrapped yet (see infra/gcp/README.md); plan skipped" >> "$GITHUB_STEP_SUMMARY"
+          else
+            echo "cannot read gs://$bucket: $err" >&2
+            exit 1
+          fi
+      - name: Init
+        if: steps.bootstrap.outputs.ok == 'true'
+        run: terraform -chdir=infra/gcp init -backend-config=envs/dev.gcs.tfbackend -input=false
+      - name: Plan (dev)
+        if: steps.bootstrap.outputs.ok == 'true'
+        # -lock=false: an advisory CI plan must never leave a stale lock that
+        # blocks a human apply. -refresh=false: the plan is config-vs-state, so
+        # it needs no read access beyond the state bucket and the parent DNS
+        # zone (the data source in dns.tf), which are the planner's two grants.
+        run: |
+          terraform -chdir=infra/gcp plan -var-file=envs/dev.tfvars -input=false -no-color -lock=false -refresh=false > "$RUNNER_TEMP/plan.txt"
+          {
+            echo "### terraform plan — infra/gcp (dev)"
+            echo '<details><summary>plan output</summary>'
+            echo
+            echo '```'
+            head -c 60000 "$RUNNER_TEMP/plan.txt"
+            echo '```'
+            echo '</details>'
+          } >> "$GITHUB_STEP_SUMMARY"

docs/adr/0001-atproto-infra.md

@@ -0,0 +1,49 @@
+# ADR 0001: atproto infrastructure on GCP
+
+- **Status:** Accepted
+- **Date:** 2026-08-25
+
+## Context
+
+atproto phase 1 puts MAPLE's legislative data on a PDS: a long-lived, stateful service
+(SQLite on local disk, plus a blobstore). MAPLE's application state and backend compute
+live in the Firebase projects `digital-testimony-dev` and `digital-testimony-prod`. The
+only other infrastructure is Typesense on AWS, managed from a repo this team cannot write
+to, deployed to dev and prod together with no gate, unchanged since June 2023.
+
+We hold `roles/editor` on dev and nothing on prod; prod waits on a handoff to the upstream
+maintainers.
+
+## Decision
+
+New atproto infrastructure runs in the existing GCP projects, one environment per project,
+defined as Terraform in this repo under `infra/gcp`. Terraform owns only the atproto
+resources; Firestore, Cloud Functions and Firebase extensions stay with the `firebase`
+CLI. Secret values are never Terraform inputs: state stores them in plaintext, so they are
+added out of band.
+
+The PDS is a Compute Engine VM running the reference `bluesky-social/pds` image, its data
+on a persistent disk, blobs in a Cloud Storage bucket. The PDS expects a POSIX filesystem
+for SQLite, and Bluesky's installer targets exactly this shape.
+
+## Consequences
+
+- Production is a reviewable `terraform apply` run by an owner, not a request for owner
+  access for an outside contributor.
+- Dev and prod are the same module with different variables.
+- We patch a VM. A managed PDS would absorb that at the cost of custody of the data and
+  keys.
+- Terraform is a second IaC tool beside the AWS CDK; that stack is AWS-only and
+  unmaintained.
+
+## Alternatives
+
+- **The AWS ECS cluster.** Not writable by this team, no dev/prod gate, non-durable
+  storage, unmaintained.
+- **Cloud Run for the PDS.** No block storage; SQLite over GCS FUSE or Filestore is
+  unsafe.
+- **GKE.** More moving parts than one small service justifies.
+- **A managed PDS.** Gives up custody of data and signing keys for a project whose point
+  is custody of public records.
+- **Hand-run `gcloud` commands.** Makes production an access request rather than a
+  reviewable plan, and drifts from what is deployed.

infra/gcp/.gitignore

@@ -0,0 +1 @@
+**/.terraform/

infra/gcp/CI.md

@@ -0,0 +1,40 @@
+# Terraform CI
+
+`.github/workflows/terraform-checks.yml` runs `fmt -check` and `validate` on both roots (this one
+and `dns/`) for every PR, with no setup. Its `plan_dev` job also posts an advisory dev plan to the
+job summary, on same-repo PRs in codeforboston/maple only (fork PRs get no secrets and show it
+skipped; the comment on that job has the details). Turning the plan on is one-time work for an
+owner of `digital-testimony-dev` with admin on codeforboston/maple:
+
+1. **The planner identity.** A service account with no roles of its own:
+
+   ```sh
+   gcloud iam service-accounts create atproto-ci-planner --project=digital-testimony-dev \
+     --display-name="terraform plan from GitHub Actions"
+   ```
+
+2. **Its two grants.** In `envs/dev.tfvars` set
+   `ci_planner = "serviceAccount:atproto-ci-planner@digital-testimony-dev.iam.gserviceaccount.com"`,
+   merge it like any other change, and apply as an owner of both projects (see Permissions; the
+   grants are the `ci_planner_*` members in `iam.tf`: read-only on the state bucket and on the
+   parent DNS zone's project).
+3. **A key.** The one step outside Terraform, so the key material never enters state. Outside the
+   working tree too — a service-account private key sitting in the repo is one `git add -A` away
+   from being committed:
+
+   ```sh
+   keydir=$(mktemp -d)
+   gcloud iam service-accounts keys create "$keydir/planner.json" \
+     --iam-account=atproto-ci-planner@digital-testimony-dev.iam.gserviceaccount.com
+   ```
+
+4. **The GitHub environment.** codeforboston/maple → Settings → Environments → New environment,
+   name `terraform-plan`, deployment branches and tags **No restriction**, no required reviewers.
+   Environment secrets → Add → `GCP_SERVICE_ACCOUNT_KEY`, value: the contents of
+   `$keydir/planner.json`. Then `rm -rf "$keydir"`.
+5. **Check.** A PR from a branch in codeforboston/maple that touches `infra/gcp/**` shows the plan
+   in the `plan_dev` job summary. A 404 in "Check for the state bucket" means
+   `infra/gcp/scripts/bootstrap.sh dev` has not run; a 403 means step 2 has not landed.
+
+Rotate or revoke with `gcloud iam service-accounts keys list|delete --iam-account=…`, then redo
+step 4.

infra/gcp/README.md

@@ -0,0 +1,68 @@
+# atproto PDS on GCP
+
+One Terraform root, one state per environment (`envs/<env>.*`). Applies are human-run; CI only
+plans. Design: [ADR 0001](../../docs/adr/0001-atproto-infra.md).
+
+## Permissions
+
+- **The environment's project**: `roles/editor` for every step, plus `storage.hmacKeys.create` for
+  step 3. `roles/owner` for the grants in `iam.tf`: an editor's apply ends red on the grants only,
+  and an owner's apply afterwards plans exactly those.
+- **`digital-testimony-prod`**: `roles/dns.admin` for the NS record in the parent zone. Every plan
+  reads that zone, so without at least `roles/dns.reader` nothing plans.
+
+## Apply
+
+`infra/gcp/dns` is applied first (this root looks its zone up). The hostname is apply-once
+(it lands in the DID document of every account the PDS creates).
+
+```sh
+infra/gcp/scripts/bootstrap.sh dev                              # 1. APIs and the state bucket
+terraform -chdir=infra/gcp init -backend-config=envs/dev.gcs.tfbackend
+terraform -chdir=infra/gcp apply -var-file=envs/dev.tfvars       # 2. everything below
+infra/gcp/scripts/secrets.sh dev                                 # 3. the five secret versions; never rotates
+curl https://pds-dev.mapletestimony.org/xrpc/_health             # the VM starts the PDS within 3 min
+gcloud storage ls gs://digital-testimony-dev-atproto-pds-blobs/   # done once one uploadBlob lands here
+gcloud compute instances get-serial-port-output atproto-pds --zone=us-central1-a | grep pds-startup   # if not
+```
+
+Prod: the same with `prod`.
+
+## What gets applied
+
+A static IP and firewall (80/443 open, 22 via IAP); the `atproto-pds` VM with a 20 GB data disk
+snapshotted daily for 14 days; the delegated zone, its A record and the parent NS record; five
+Secret Manager secrets without versions; the blob bucket; the VM's service account and its
+grants. Knobs: `envs/<env>.tfvars`. Not here: secret versions and the HMAC key (`secrets.sh`),
+the state bucket (`bootstrap.sh`).
+
+## Rollback
+
+- **Config**: revert and apply. A startup-script change lands on the next boot:
+  `gcloud compute instances reset atproto-pds --zone=us-central1-a` applies it now.
+- **A secret**: `gcloud secrets versions add <id> --data-file=-`, reset the VM (it re-reads every
+  secret on boot, nothing on disk), disable the old version.
+- **Data**: create a disk from a snapshot and attach it as `pds-data`. Blobs are in the bucket.
+- **State**: the bucket is versioned; restore the earlier object.
+- **Teardown**: `destroy` refuses by design (`prevent_destroy` on disk, bucket and zone; the VM
+  is deletion-protected). Lifting those is its own reviewed change.
+
+## Monitoring
+
+Three alerts, all to `alert_channels` in `envs/<env>.tfvars` (dev: one email; prod: the pager, a
+channel-type swap there changes no policy). Subjects start with `[<env>]`, and each page carries
+its own first step; this section is what a page cannot.
+
+- **PDS down** (critical): `https://<pds_hostname>/xrpc/_health` failing from two regions for 5 min.
+  One check covers VM, docker, caddy, cert expiry and DNS, from where the relay stands. Expect one
+  during bring-up: that page is the channel test.
+- **Disk ≥ 80%** on any of the VM's disks, and **memory ≥ 90%** for 10 min, via the Ops Agent the
+  startup script installs (`pds-startup.sh.tftpl`).
+
+Prove it once per environment: `gcloud compute ssh atproto-pds --tunnel-through-iap --zone=us-central1-a`,
+`sudo systemctl stop pds.service`, wait for the page (≤ 6 min), `start` it.
+
+## CI
+
+`.github/workflows/terraform-checks.yml`: `fmt`, `validate` and an advisory dev plan on PRs. What
+runs, and the one-time setup the plan needs: [CI.md](CI.md).

infra/gcp/accounts.tf

@@ -0,0 +1,4 @@
+resource "google_service_account" "pds" {
+  account_id   = "atproto-pds"
+  display_name = "atproto PDS VM"
+}

infra/gcp/backend.tf

@@ -0,0 +1,6 @@
+# Partial backend configuration — the bucket/prefix come from
+# envs/<env>.gcs.tfbackend at init time:
+#   terraform init -backend-config=envs/dev.gcs.tfbackend
+terraform {
+  backend "gcs" {}
+}

infra/gcp/blobs.tf

@@ -0,0 +1,26 @@
+# Blob storage for the PDS: uploaded media, content-addressed by CID. The PDS
+# talks to it through its S3 blobstore over Cloud Storage's S3-compatible XML
+# API (storage.googleapis.com, HMAC credentials), so the data disk holds only
+# SQLite and the actor store and never grows with uploads.
+#
+# The HMAC key is NOT a Terraform resource: google_storage_hmac_key stores the
+# secret in state (ADR 0001). README.md step 3 creates it
+# with gcloud for the PDS service account and adds both halves to Secret
+# Manager; the key inherits the account's IAM, granted in iam.tf.
+resource "google_storage_bucket" "pds_blobs" {
+  name     = local.blob_bucket
+  location = var.region
+  labels   = local.labels
+
+  uniform_bucket_level_access = true
+  public_access_prevention    = "enforced"
+
+  # Blobs are immutable and content-addressed; object versioning would only
+  # double the bill. The default 7-day soft delete covers accidental deletes.
+
+  # Deliberate destruction means removing this block first, in its own
+  # reviewed change — same rule as the data disk.
+  lifecycle {
+    prevent_destroy = true
+  }
+}