Skip to content

Terraform to spin up a PDS - #2236

Draft
fastfadingviolets wants to merge 4 commits into
codeforboston:mainfrom
hyphacoop:atproto-pds-terraform
Draft

Terraform to spin up a PDS#2236
fastfadingviolets wants to merge 4 commits into
codeforboston:mainfrom
hyphacoop:atproto-pds-terraform

Conversation

@fastfadingviolets

@fastfadingviolets fastfadingviolets commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a short summary of the changes, and a reference to the original issue using # and the issue number, like #1

Checklist

  • On the frontend, I've made my strings translate-able.
  • If I've added shared components, I've added a storybook story.
  • I've made pages responsive and look good on mobile.
  • If I've added new Firestore queries, I've added any new required indexes to firestore.indexes.json (Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)

Screenshots

Add some screenshots highlighting your changes.

Known issues

If you've run against limitations or caveats, include them here. Include follow-up issues as well.

Steps to test/reproduce

For each feature or bug fix, create a step by step list for how a reviewer can test it out. E.g.:

  1. Go to the home page
  2. Click on a testimony
  3. See that it's loaded with a loading spinner

@vercel

vercel Bot commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
maple-dev Ready Ready Preview Sep 14, 2026 7:48pm UTC

Request Review

fastfadingviolets added a commit to hyphacoop/maple that referenced this pull request Sep 9, 2026
@fastfadingviolets @claude
PDS startup: retry timer instead of reboots, safer disk guard, GCS ch…
6dd346b 
…ecksums off (codeforboston#174)

Review of codeforboston#2236 found the boot script fragile in five ways; all fixed here.

- Secrets: fetch_secret had no retry and swallowed curl's stderr, so one
  Secret Manager blip left the PDS down until a human rebooted, and 403/404/
  network all printed the same line. Now: --retry 5 --retry-all-errors, the
  HTTP status in the log (404 no version, 403 grant unapplied, 000 network),
  and a pds-startup-retry.timer that re-runs the script every 3 minutes until
  /run/pds/pds.env exists. secrets.sh no longer needs to reset the VM. Exit
  status is 0 only when every gap is "no version yet".
- A flock around the whole run so the timer cannot race a slow first boot.
- apt: DPkg::Lock::Timeout=600 — apt-daily/unattended-upgrades hold the lock
  on a fresh Debian 12 image and, under set -e, ended the run before the disk
  was mounted or the unit installed.
- Data disk: format only on blkid exit 2 ("no filesystem"); exit 0 with an
  empty type or any other failure refuses. The old empty-string test could
  not tell "no filesystem" from "probe failed".
- Blobs: AWS SDK v3 >= 3.729 (bundled in pds 0.4.5027) sends CRC32 flexible
  checksums by default and GCS's XML API rejects the signature, so uploadBlob
  4xx'd while _health stayed green. AWS_REQUEST_CHECKSUM_CALCULATION and
  AWS_RESPONSE_CHECKSUM_VALIDATION=when_required turn that off.
- Container logs capped (10m x 3, as upstream's installer does) on the 10 GB
  boot disk; LOG_ENABLED=true so the PDS logs at all.
- Comment correction: dockerd persists the container env on the boot disk;
  "never on the persistent disk" now says "never on the data disk".

Rendered with dev values: bash -n and shellcheck clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BuwSnBqJkHM4qWekjr6TjD
fastfadingviolets and others added 4 commits September 14, 2026 15:40
@fastfadingviolets @claude
ADR 0001: atproto infrastructure on GCP
18fb047 
New atproto infrastructure runs in the existing Firebase projects as
Terraform in this repo, and the PDS is a Compute Engine VM with a
persistent data disk and blobs in Cloud Storage. Alternatives recorded:
the AWS ECS cluster, Cloud Run, GKE, a managed PDS, hand-run gcloud.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWpp9AEzuwEsXy1tdxiNjs
@fastfadingviolets @claude
Terraform root for the atproto PDS
351d286 
One root, one state per environment (envs/<env>.*), applied by people;
no workflow applies it. Delivers the dev PDS at pds-dev.mapletestimony.org
with pds.mapletestimony.org reserved for prod.

- PDS on an e2-small VM with a separate data disk and snapshot policy;
  the startup script retries on a timer instead of rebooting, guards the
  disk before mounting, and renders pds.env into /run (tmpfs) so secrets
  never touch the data disk or its snapshots.
- Blobs in a GCS bucket through the PDS's S3 blobstore; the HMAC key is
  minted out of band by scripts/secrets.sh so it never enters state.
- Secret Manager holds the PDS's five boot secrets as resources only;
  secrets.sh adds the versions, and the VM picks them up within minutes
  without a reboot.
- IAM lives in iam.tf with *_iam_member only, written so an editor's
  apply creates everything and fails only the grants; an owner's apply
  then plans exactly the grants.
- The PDS hostname is apply-once (it lands in the DID document) and the
  delegated zone carries prevent_destroy to enforce it.
- The delegation is an NS record the root writes into the parent zone
  infra/gcp/dns owns in digital-testimony-prod. A data source reads that
  zone at plan time and a precondition refuses a hostname outside it, so
  a missing parent fails the plan rather than one resource at apply.
- scripts/bootstrap.sh enables the APIs and creates the state bucket,
  idempotently; the same bucket the dns root uses in prod, under a
  different prefix.
- ci_planner names the service account a CI plan may run as; it gets
  read-only on the state bucket and the parent zone, nothing else.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWpp9AEzuwEsXy1tdxiNjs
@fastfadingviolets @claude
CI: Terraform checks for both roots; dev plan for the PDS root
4aff3d9 
One workflow, terraform-checks.yml. fmt and validate run as a matrix over
infra/gcp and infra/gcp/dns on every PR that touches infra/gcp, so
dns-zone-checks.yml goes away along with the path negation that kept the
two apart. A DNS-only PR now also validates the PDS root, which costs
seconds.

plan_dev posts a dev `plan -refresh=false -lock=false` to the job summary
for same-repo PRs, run in a terraform-plan GitHub environment whose key
belongs to the service account named by ci_planner in envs/dev.tfvars.
The plan is config-versus-state plus one read of the parent zone. A
missing bucket skips the plan; a 403 fails it. Fork PRs get no secrets
and skip it. No workflow applies.

The PDS README gains a CI section, and the one-time planner setup
(service account, grants, key, GitHub environment) lives in CI.md so the
runbook stays short.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWpp9AEzuwEsXy1tdxiNjs
@fastfadingviolets @claude
Tier 0 monitoring for the PDS: alert channel per env, uptime check, d…
36c95be 
…isk and memory alerts

Nothing watched the PDS: no monitoring resource existed, and the default
GCE metrics have neither memory nor filesystem usage, so disk full on
/pds (the PDS's characteristic death) was invisible.

monitoring.tf is the shared foundation: one notification channel per
entry in the new alert_channels variable, and local.alert_channel_ids
that every policy in this root notifies. Channels are per environment by
design (dev pages one email; prod's pager is prod's own entry), and any
channel type whose labels are not secrets drops in through tfvars alone.
Token-bearing types are refused by validation and the write-only path
for them is described on the variable.

monitoring-pds.tf: an HTTPS uptime check on /xrpc/_health with a
CRITICAL policy (two regions failing for five minutes); WARNING policies
on the Ops Agent's disk (any real device over 80%) and memory (over 90%
for ten minutes) metrics for the VM. Each page's documentation names the
first command and the runbook section.

The startup script installs the Ops Agent after the PDS start,
non-fatal: nothing the PDS needs depends on it, so a failed install is
logged and left for the next boot. bootstrap.sh enables the Monitoring
API. The README gains a Monitoring section: where alerts go, the three
signals, the expected bring-up page, and how to prove the path once per
environment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YWpp9AEzuwEsXy1tdxiNjs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sashamaryl sashamaryl Awaiting requested review from sashamaryl sashamaryl will be requested when the pull request is marked ready for review sashamaryl is a code owner
@mertbagt mertbagt Awaiting requested review from mertbagt mertbagt will be requested when the pull request is marked ready for review mertbagt is a code owner
@nesanders nesanders Awaiting requested review from nesanders nesanders will be requested when the pull request is marked ready for review nesanders is a code owner
@mvictor55 mvictor55 Awaiting requested review from mvictor55 mvictor55 will be requested when the pull request is marked ready for review mvictor55 is a code owner
@timblais timblais Awaiting requested review from timblais timblais will be requested when the pull request is marked ready for review timblais is a code owner
@alexjball alexjball Awaiting requested review from alexjball alexjball will be requested when the pull request is marked ready for review alexjball is a code owner
@Mephistic Mephistic Awaiting requested review from Mephistic Mephistic will be requested when the pull request is marked ready for review Mephistic is a code owner
@kiminkim724 kiminkim724 Awaiting requested review from kiminkim724 kiminkim724 will be requested when the pull request is marked ready for review kiminkim724 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fastfadingviolets