Add Terraform infrastructure for managed OpenSearch web analytics pipeline - #77
Open
jordanpadams wants to merge 103 commits into
Open
Add Terraform infrastructure for managed OpenSearch web analytics pipeline#77jordanpadams wants to merge 103 commits into
jordanpadams wants to merge 103 commits into
Conversation
Deploys the full web analytics infrastructure: managed OpenSearch domain (VPC-only, FGAC via IAM, OpenSearch 2.19), Logstash EC2, S3 log bucket, and IAM policies. Splits admin-only operations (IAM policy creation, EC2 launch) into dedicated tasks. Removes hardcoded sensitive values (SSO role ARN, instance profile name) in favor of variables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Rename pds_resource_prefix -> s3_bucket_prefix (S3 only) and ec2_name_prefix/policy_name_prefix -> resource_prefix (all other resources). Removes gh01dc from IAM policy and EC2 resource names. Add dev.tfvars.example files for all modules. Fix YAML colon quoting in Taskfile aliases and deps. Run terraform fmt across all modules. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Automates first-boot setup: clones repo, deploys Logstash pipeline config, applies OpenSearch ECS index template, and starts Logstash. Can also be run manually via SSM on existing instances. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes stale Chunk 1/2/3 references, adds logstash-init.sh step, clarifies permission requirements per step, updates teardown and architecture notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FGAC (fine-grained access control) required mapping IAM roles to backend roles via the OpenSearch security API after every domain deployment. Since access is controlled entirely by the IAM resource-based policy (which allows the EC2 instance role and mcp-tenantOperator), FGAC provides no additional security benefit and only adds operational friction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tenantOperator roles are no longer applicable. EC2 instance role is the only principal that needs access to S3 (for log ingestion) and OpenSearch (for writing indexed data). Also collapses the two-statement OpenSearch policy into a single AllowEC2Role statement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…example
The {venue}-en-platform-engineer role only exists in test/prod, not dev.
AWS rejects resource-based policies that reference non-existent principals,
so the AllowPlatformEngineer statement must only be included when the role
exists in the target account.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces tfvars/common.tfvars (aws_region, partition, tenant, component, cicd, managedby) shared by all three Terraform root modules. Each module's venue-specific tfvars now only contains venue + module-specific values. Taskfile updated to pass -var-file=.../tfvars/common.tfvars before each module's own tfvars. Adds missing prod.tfvars.example files for root and IAM modules. Removes opensearch_master_user_arn from OpenSearch tfvars now that FGAC is disabled. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clearly marks Steps 3 (IAM) and 4 (EC2) as admin-only with required permissions. Adds unset AWS_PROFILE note for Terraform S3 backend compatibility. Updates teardown section with admin markers. Documents OpenSearch access policy and platform-engineer role pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenSearch module does not declare a partition variable, so passing it via common.tfvars produced a warning. partition is now declared only in the root and IAM module tfvars where it is actually used. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ss root and IAM modules
partition, s3_bucket_prefix, resource_prefix, ec2_role_name, and opensearch_domain_name
were duplicated between terraform/tfvars/ and terraform/iam/policies/tfvars/. These values
are shared between root and IAM modules but differ per venue, so they can't go in the global
common.tfvars (which would cause undeclared-variable warnings in opensearch_managed).
Solution: common-infra-<venue>.tfvars, passed to root and IAM tasks via {{.VENUE}} in
Taskfile. Module-specific venue files now only contain what's unique to that module.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…+ IAM
Replaces the two-file pattern (common.tfvars + common-infra-{venue}.tfvars) with
a single common-{venue}.tfvars containing both global and infra-specific values.
Opensearch tasks remain unaffected — they use only their own per-venue tfvars.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace concrete identifiers (bucket names, role names, VPC IDs, SG names, domain names) with empty strings or generic descriptions. Examples should show structure, not real values. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
opensearch_managed uses only its own per-venue tfvars — common-{venue}.tfvars
is for root and IAM modules only (opensearch doesn't declare those variables).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Smoke test script checks S3 access, OpenSearch reachability, and OpenSearch SigV4 auth from the EC2 — all via SSM parameter lookups, no hardcoded values - Publish S3 bucket name to SSM at /pds/web-analytics/s3/bucket_name so the smoke test and other tooling can look it up without hardcoding - Add s3_bucket_name SSM parameter target to s3:plan/s3:deploy Taskfile tasks Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…bility sudo pip installs to root's home by default on AL2023; --break-system-packages installs to the system site-packages so all users can import boto3. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- enforce *tfplan* gitignore pattern (was comment-only) - un-ignore .terraform.lock.hcl to pin provider versions - commit lock files for root, logstash, and iam/policies modules - rename opensearch_managed → opensearch in SSM parameter paths (logstash/main.tf, logstash-init.sh, smoke-test.sh, common.tfvars.example) - update terraform/README.md diagram labels and SSM path references Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EC2 Name tag was pds-dev-observability, which was wrong on two counts: it named a Logstash instance after the OpenSearch component, and baked in the venue when MCP uses separate accounts per environment. Now derived from the existing component variable so the name is account-agnostic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… common tfvars - IAM policy now reads the OpenSearch ARN from SSM (/pds/observability/opensearch/opensearch_arn) instead of constructing it from a variable — removes cross-repo dependency on knowing the domain name - remove opensearch_domain_name from common.tfvars.example, iam/policies/variables.tf, iam/policies/web-analytics/variables.tf, and the module call in iam/policies/main.tf - move s3_bucket_prefix from common.tfvars.example to tfvars/dev.tfvars.example (root S3 module only; not needed by logstash or IAM) - fix iam/policies/tfvars/dev|prod.tfvars.example to include logs_s3_bucket_name (required variable that was missing from examples) - strip stale logstash variables from tfvars/dev.tfvars.example (vpc_id, ec2_security_group_name, logstash_instance_type, logstash_version belong in logstash/tfvars/dev.tfvars, not the root module) - update README variables table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ta source - Replace pdsoperator@jpl.nasa.gov and your-team-distro@jpl.nasa.gov with <your-email> placeholder in prod.tfvars.example files - Remove default = "pdsoperator@jpl.nasa.gov" from iam/policies/variables.tf (managedby is required; caller must supply via tfvars) - Remove stale logstash vars (vpc_id, ec2_security_group_name, etc.) from root prod.tfvars.example — those live in logstash/tfvars/ not root - Drop unused data "aws_caller_identity" "current" from iam/policies/main.tf (leftover after opensearch ARN was moved to SSM lookup) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ent IPs - Rename test data files: E1QK9ERQ1P4H6H → EXAMPLE1234567 (distribution ID) - Replace d1vk9jvdr9m2gz.cloudfront.net → example1234567.cloudfront.net - Replace pds.mcp.nasa.gov → pds.example.nasa.gov - Replace real client IPs with RFC 5737 documentation addresses (198.51.100.x, 203.0.113.x) in both .conf test inputs and .gz test data files - Regenerate .gz files from anonymized content Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All root .tf files and backend configs are now in terraform/s3/, matching the layout of the logstash/ and iam/policies/ modules. Taskfile tasks renamed init/validate → s3:init/s3:validate; all s3 task dirs and var-file paths updated accordingly. Backend state key updated from web-analytics/terraform.tfstate → web-analytics/s3.tfstate. Module-specific tfvars examples moved to s3/tfvars/. Drops deprecated overwrite = true from aws_ssm_parameter.s3_bucket_name. State migration required after applying this commit: cd terraform/s3 terraform init -backend-config=backend-<venue>.hcl -migrate-state Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
State migration from a directory with no .terraform/ cache does not carry the old key — existing state would be orphaned. Keep the original key so the module finds its state immediately after init. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
M2 rename s3/web-analytics.tf -> s3/main.tf; add per-module README.md
to s3/, logstash/, and iam/policies/ (inputs, outputs, deploy steps)
M7 add use_lockfile = true to all backend-*.hcl files (native S3
locking, Terraform >= 1.10 — no DynamoDB table required)
M9 tighten required_version from >= 1.0 to >= 1.10.0 across all
three modules (prerequisite for native S3 locking)
M12 replace hardcoded MCP AMI owner account ID (794625662971) in
logstash/main.tf with var.mcp_ami_owner_id (default preserved)
M16 pin pds-tf-modules source to ?ref=v1.1.0-dev (only published tag)
Also picks up s3/backend.tf key update to web-analytics/s3.tfstate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Account IDs must not be hardcoded in committed .tf files (M12). The MCP AMI owner ID is now required via tfvars — added placeholder to dev.tfvars.example and prod.tfvars.example. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add lifecycle ignore_changes to aws_instance.logstash so MCP-stamped tags (Audit, CreatedBy, CreationTime) and instance-level user_data are not stripped on apply. Also updates pds-tf-modules s3/bucket ref to v0.1.0. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
s3:deploy/logstash:deploy/iam:deploy depended on validate (which doesn't touch the backend), and their :destroy counterparts had no deps at all, so switching VENUE between runs without an intervening `task init` would apply/destroy using stale-venue backend state while reading the new venue's tfvars. All six now depend on their module's init task, matching plan/refresh, which always re-run `terraform init -reconfigure` for the venue actually requested. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tfvars/common-<venue>.tfvars, s3/tfvars/<venue>.tfvars, logstash/tfvars/<venue>.tfvars,
and iam/policies/tfvars/<venue>.tfvars are gitignored here, so the real values had no
shared source of truth. They're now tracked in cds-infra-deploy at
venues/<venue>/web-analytics/{common,s3,logstash,iam-policies}.tfvars; point
CDS_INFRA_DEPLOY_DIR at a checkout of that repo, or pass LOCAL=1 to a task to keep
using this repo's own local tfvars for personal iteration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous commit linked to a github.com/NASA-PDS/cds-infra-deploy URL that does not exist; cds-infra-deploy is a private GitLab repo. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…po branch var - Fix sincedb directory path (plugins/inputs/s3/ not sincedb/) - Fix CloudFront W3C grok to match tab between date and time fields - Update logstash-integration-aws plugin on init to fix PageableResponse error - Add repo_branch terraform variable (default main, dev=develop) so EC2 clones correct branch - Pass S3_BUCKET_NAME and S3_CF_BUCKET_NAME explicitly from userdata to init script - Remove default="" from s3_cf_bucket_name variable so missing value fails at plan time - Fix all bash <(curl ...) patterns (not supported on MCP Amazon Linux) - Update READMEs with correct operational runbooks including required env vars Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e DNS Applies the tld plugin to source.domain after reverse DNS resolution, populating ECS 8.17 fields already present in the index template. Enables grouping traffic by organization (e.g. bbtec.net) rather than individual hostnames or raw IPs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The S3 input uses the default plain codec, so each CloudFront Standard
Logging v2 line arrives in [message] as a raw JSON string. The filter
cf_format==json branch tried to rename fields (c-ip, date, time, etc.)
that were never decoded — causing _missing_source_address and
_datetimeparsefailure on every JSON CF event.
Add json { source => message } as the first step in the CF JSON
branch so fields are decoded into the event before any renaming occurs.
message is preserved so the downstream fingerprint filter still works.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…SSM), add missing S3_CF_BUCKET_NAME S3_BUCKET_NAME and OPENSEARCH_ENDPOINT are auto-fetched from SSM by logstash-init.sh — only S3_CF_BUCKET_NAME must be passed explicitly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The CloudFront W3C grok pattern uses [ \t] inside the named capture group for [event][start], so tab-delimited files capture date<TAB>time. The date filter pattern yyyy-MM-dd HH:mm:ss expects a space, causing _datetimeparsefailure on every W3C log line. Add a gsub to replace tabs with spaces in [event][start] before the date filter runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
path_depth is an array of all cumulative path prefixes, e.g. for /data/mgs/vol1 it produces ["/data", "/data/mgs", "/data/mgs/vol1"]. Use this with a Terms aggregation + Sum of http.response.body.bytes to aggregate egress at any directory depth without needing separate depth-1/2/3 fields. url.path_parts (individual segments) is unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Explicitly maps the new url.path_depth array field (cumulative path prefixes for hierarchical egress aggregation) as keyword type, consistent with url.path_parts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jordanpadams
marked this pull request as ready for review
August 11, 2026 17:39
Other node pipelines keep the global defaults (1 worker/100 batch) so they lazy-load in the background while en catches up first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
number_of_shards/number_of_replicas were unset, silently defaulting to OpenSearch's 1 shard / 1 replica -- undersized once 5 years of backfill across 9 mission sources lands in monthly indices. 3 shards matches the 3-node prod data cluster. index_patterns was pds-weblogs-*, which never matched any venue's real INDEX_PREFIX (pds-dev, pds-test, pds-prod per common-<venue>.tfvars) -- this template has likely never actually applied to a real index. Fixed to match all three explicitly (not a broad pds-* wildcard, to avoid also matching the unrelated pds-cloudfront-realtime-index-* Firehose index). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the previous AOSS-based Terraform with a complete managed OpenSearch infrastructure for the PDS Web Analytics pipeline.
What this deploys (this repo):
Key design decisions:
iam:deploy,logstash:deploy) split into dedicated tasks so Power-User can deploy S3 independently.tffiles — all environment-specific values in gitignored.tfvarsDeployment order:
task iam:deploy VENUE=dev— IAM policy (admin:iam:CreatePolicy,iam:AttachRolePolicy)task s3:deploy VENUE=dev— S3 log bucket (Power-User)task logstash:deploy VENUE=dev— Logstash EC2 (admin:iam:PassRole)🤖 Generated with Claude Code
Test plan
/pds/observability/opensearch/opensearch_endpoint,/pds/observability/opensearch/opensearch_arntask iam:plan VENUE=devreviewed and appliedtask s3:plan VENUE=devreviewed and appliedtask logstash:plan VENUE=devreviewed — confirms EC2 replace (-/+) from existing Docker-based instancetask logstash:deploy VENUE=devapplied — new EC2 created with RPM Logstash + systemdpds-dev-observability(new domain)pds-dev-web-analyticsOpenSearch domain after new EC2 confirmed healthyTesting Completed
I can confirm logstash is able to run in Dev and I have dashboards working with some test logs.
