diff --git a/.agents/references/diagrams.md b/.agents/references/diagrams.md index 87abb5d1..37237fae 100644 --- a/.agents/references/diagrams.md +++ b/.agents/references/diagrams.md @@ -93,6 +93,10 @@ in one diagram to a box in another without reading the label. | πŸ”Œ | network, subnet | | ☸️ | Kubernetes cluster | | πŸ—„οΈ | container registry | +| πŸ‘€ | end user, application team acting as a user | +| πŸ’¬ | user-facing chat interface | +| πŸšͺ | gateway, abstraction layer | +| πŸ“ˆ | observability, tracing, evaluation | | βš™οΈ | CI/CD wiring | | πŸ”€ | git repository | | 🧠 | model serving / AI API | @@ -105,7 +109,7 @@ a concept already listed. Copy the emoji straight from this table β€” several of them carry a **U+FE0F variation selector** that is invisible in the source but decides whether the glyph renders in colour. `☸ βš™ πŸ—„ πŸ—‚ πŸ›°` default to -*text* presentation and come out as small monochrome symbols without it; `🏒 πŸ“ πŸ”‘ 🌐 πŸ”Œ πŸ”€ 🧠 πŸ›¬ πŸ“¦` +*text* presentation and come out as small monochrome symbols without it; `🏒 πŸ“ πŸ”‘ 🌐 πŸ”Œ πŸ”€ 🧠 πŸ›¬ πŸ“¦ πŸ‘€ πŸ’¬ πŸšͺ πŸ“ˆ` are colour by default. If an icon renders monochrome in the SVG, it is missing the selector. The **same emoji with a different fill** is deliberate and useful: a green `πŸ”€` repo inside the diff --git a/AGENTS.md b/AGENTS.md index 5c36f59d..07ac7d25 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -338,7 +338,9 @@ getting-started steps, and shared responsibility matrix. ### Conventions - Folder name: `-` (e.g. `azure-kubernetes`, `stackit-kubernetes`), with the - architecture itself in `README.md`. + architecture itself in `README.md`. Use `` alone (e.g. `ai-platform`) when the + architecture is genuinely multi-cloud β€” it lists several `cloudProviders` and its components are + cloud-agnostic, with each provider contributing only a small provider-specific module. - Logo: colocate as `logo.png` (or `logo.svg`) in the architecture folder, the same convention `buildingblock/logo.png` uses. The website generator copies it to `website/public/assets/reference-architecture-logos/.png` β€” never add files there by hand, diff --git a/modules/ai/README.md b/modules/ai/README.md new file mode 100644 index 00000000..afd94a80 --- /dev/null +++ b/modules/ai/README.md @@ -0,0 +1,10 @@ +--- +name: AI +description: meshStack integration with AI platforms puts a gateway in front of your model providers, so application teams call one OpenAI-compatible endpoint with a virtual key while the platform team keeps the provider credentials, the budgets and the spend records. +category: ai +benefits: + - Model Gateway + - Virtual Key Issuing + - Budget and Spend Tracking +official: true +--- diff --git a/modules/ai/azure-openai/buildingblock/README.md b/modules/ai/azure-openai/buildingblock/README.md new file mode 100644 index 00000000..f51c3a7b --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/README.md @@ -0,0 +1,78 @@ +--- +name: Azure OpenAI Model Backend +supportedPlatforms: + - ai +description: Registers an Azure OpenAI deployment as a model backend on the LiteLLM gateway, scoped to the tenant's team. +# The module talks only to the LiteLLM admin API. It creates nothing in Azure and receives the +# Azure OpenAI endpoint, key and deployment name as static inputs, so there is no cloud-side setup +# to perform ahead of time. +requiresBackplane: false +--- + +# Azure OpenAI Model Backend Building Block + +This building block registers an existing Azure OpenAI deployment as a model entry on the LiteLLM +gateway. It is the Azure entry in the model layer of the AI platform architecture, next to +`stackit/model-serving` for STACKIT, and it is what makes the architecture's claim of two cloud +providers true. + +The module is thin on purpose. It creates one `litellm_model` resource and no Azure resources at +all, which is why it declares `requiresBackplane: false` and needs no Azure identity. The platform +team creates the Azure OpenAI resource and its deployment once, outside this module, and passes the +endpoint, the key and the deployment name in as static inputs. + +The `team_id` input scopes the entry to one LiteLLM team, so the gateway offers the model to that +tenant alone and attributes its spend to that team. In the AI landing zone the team ID comes from +the platform tenant ID that `ai/model-access` produces, so the application team fills in nothing. + +Azure OpenAI addresses a **deployment**, not a model. The provider joins `custom_llm_provider` and +`base_model` into `azure/`, so `azure_deployment_name` must carry the Azure deployment +name rather than the underlying model name. + +Application teams keep calling the LiteLLM gateway. The Azure credential stays on the gateway, so +the platform team can rotate it without a change on the application side. + +The `ncecere/litellm` provider is pinned to exactly `2.0.1`. This is a deliberate exception to the +hub rule that provider constraints use `>=`, and `versions.tf` explains why. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [litellm](#requirement\_litellm) | = 2.0.1 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [litellm_model.this](https://registry.terraform.io/providers/ncecere/litellm/2.0.1/docs/resources/model) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [azure\_deployment\_name](#input\_azure\_deployment\_name) | Name of the model deployment in the Azure OpenAI resource, for example 'gpt-4o'. Azure routes on the deployment name, not on the model name. | `string` | n/a | yes | +| [azure\_openai\_api\_key](#input\_azure\_openai\_api\_key) | Key of the Azure OpenAI resource. LiteLLM stores it and sends it upstream in the 'api-key' header. | `string` | n/a | yes | +| [azure\_openai\_api\_version](#input\_azure\_openai\_api\_version) | Azure OpenAI data plane API version. '2024-10-21' is the latest dated GA version of the inference API. | `string` | `"2024-10-21"` | no | +| [azure\_openai\_endpoint](#input\_azure\_openai\_endpoint) | Endpoint of the Azure OpenAI resource, for example 'https://my-aoai.openai.azure.com'. LiteLLM calls the deployment under this host. | `string` | n/a | yes | +| [litellm\_api\_base](#input\_litellm\_api\_base) | Base URL of the LiteLLM gateway, for example 'https://litellm.example.com'. The provider talks to the admin API under this URL. | `string` | n/a | yes | +| [litellm\_api\_key](#input\_litellm\_api\_key) | LiteLLM admin key the provider authenticates with. It needs permission to register models. | `string` | n/a | yes | +| [mode](#input\_mode) | What the deployment is used for. LiteLLM accepts 'chat', 'completion', 'embedding', 'audio\_speech', 'audio\_transcription', 'image\_generation', 'video\_generation', 'batch' and 'rerank'. | `string` | `"chat"` | no | +| [model\_name](#input\_model\_name) | Name the model is offered under on the LiteLLM gateway. Application teams pass this name in the 'model' field of their requests. | `string` | n/a | yes | +| [team\_id](#input\_team\_id) | ID of the LiteLLM team the model is registered for. Only that team can call the model. Leave unset to register the model for the whole gateway. | `string` | `null` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [api\_base](#output\_api\_base) | OpenAI-compatible base URL of the LiteLLM gateway, including the '/v1' suffix. Calls to this model go here, not to the Azure endpoint. | +| [model\_id](#output\_model\_id) | ID LiteLLM gave the model entry. | +| [model\_name](#output\_model\_name) | Name to pass in the 'model' field of a request to the gateway. | +| [summary](#output\_summary) | Summary with the model name and the endpoint to call it on. | + diff --git a/modules/ai/azure-openai/buildingblock/SUMMARY.md.tftpl b/modules/ai/azure-openai/buildingblock/SUMMARY.md.tftpl new file mode 100644 index 00000000..ce6ca454 --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/SUMMARY.md.tftpl @@ -0,0 +1,27 @@ +# Azure OpenAI Model: **${model_name}** + +## Details + +| Property | Value | +|----------|-------| +| **API Base URL** | `${api_base}` | +| **Model name** | `${model_name}` | +| **Azure deployment** | `${deployment_name}` | +| **Azure endpoint** | `${azure_endpoint}` | +| **Azure API version** | `${azure_api_version}` | +| **Model ID** | `${model_id}` | + +## Calling the model + +Call the LiteLLM gateway, not the Azure endpoint. Send your own virtual key as a bearer token and +pass the model name in the `model` field: + +```sh +curl "${api_base}/chat/completions" \ + -H "Authorization: Bearer $VIRTUAL_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "${model_name}", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +The Azure credential stays on the gateway. Your application never sees it and never talks to Azure +directly, so the platform team can rotate the credential without a change on your side. diff --git a/modules/ai/azure-openai/buildingblock/logo.png b/modules/ai/azure-openai/buildingblock/logo.png new file mode 100644 index 00000000..45d81b5e Binary files /dev/null and b/modules/ai/azure-openai/buildingblock/logo.png differ diff --git a/modules/ai/azure-openai/buildingblock/main.tf b/modules/ai/azure-openai/buildingblock/main.tf new file mode 100644 index 00000000..260ff48f --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/main.tf @@ -0,0 +1,24 @@ +locals { + # LiteLLM's OpenAI-compatible routes live under '/v1'. The gateway also answers without the + # prefix, but OpenAI client libraries expect it, so the output carries it. + api_base = "${trimsuffix(var.litellm_api_base, "/")}/v1" +} + +resource "litellm_model" "this" { + model_name = var.model_name + custom_llm_provider = "azure" + + # The provider joins these two into '/' and sends the result to + # LiteLLM. Azure OpenAI addresses a deployment, not a model, so base_model carries the Azure + # deployment name and the request goes to 'azure/'. + base_model = var.azure_deployment_name + + model_api_base = var.azure_openai_endpoint + model_api_key = var.azure_openai_api_key + api_version = var.azure_openai_api_version + mode = var.mode + + # When a team ID is given, LiteLLM offers the model to that team alone. The gateway keeps one + # entry per tenant, and the spend of each entry is attributed to its team. + team_id = var.team_id +} diff --git a/modules/ai/azure-openai/buildingblock/outputs.tf b/modules/ai/azure-openai/buildingblock/outputs.tf new file mode 100644 index 00000000..ea500b10 --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/outputs.tf @@ -0,0 +1,26 @@ +output "model_name" { + value = litellm_model.this.model_name + description = "Name to pass in the 'model' field of a request to the gateway." +} + +output "model_id" { + value = litellm_model.this.id + description = "ID LiteLLM gave the model entry." +} + +output "api_base" { + value = local.api_base + description = "OpenAI-compatible base URL of the LiteLLM gateway, including the '/v1' suffix. Calls to this model go here, not to the Azure endpoint." +} + +output "summary" { + description = "Summary with the model name and the endpoint to call it on." + value = templatefile("${path.module}/SUMMARY.md.tftpl", { + model_name = litellm_model.this.model_name + model_id = litellm_model.this.id + api_base = local.api_base + deployment_name = var.azure_deployment_name + azure_endpoint = var.azure_openai_endpoint + azure_api_version = var.azure_openai_api_version + }) +} diff --git a/modules/ai/azure-openai/buildingblock/provider.tf b/modules/ai/azure-openai/buildingblock/provider.tf new file mode 100644 index 00000000..7f3b4858 --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/provider.tf @@ -0,0 +1,4 @@ +provider "litellm" { + api_base = var.litellm_api_base + api_key = var.litellm_api_key +} diff --git a/modules/ai/azure-openai/buildingblock/variables.tf b/modules/ai/azure-openai/buildingblock/variables.tf new file mode 100644 index 00000000..0aa6e8dd --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/variables.tf @@ -0,0 +1,49 @@ +variable "litellm_api_base" { + type = string + description = "Base URL of the LiteLLM gateway, for example 'https://litellm.example.com'. The provider talks to the admin API under this URL." +} + +variable "litellm_api_key" { + type = string + sensitive = true + description = "LiteLLM admin key the provider authenticates with. It needs permission to register models." +} + +variable "azure_openai_endpoint" { + type = string + description = "Endpoint of the Azure OpenAI resource, for example 'https://my-aoai.openai.azure.com'. LiteLLM calls the deployment under this host." +} + +variable "azure_openai_api_key" { + type = string + sensitive = true + description = "Key of the Azure OpenAI resource. LiteLLM stores it and sends it upstream in the 'api-key' header." +} + +variable "azure_openai_api_version" { + type = string + default = "2024-10-21" + description = "Azure OpenAI data plane API version. '2024-10-21' is the latest dated GA version of the inference API." +} + +variable "azure_deployment_name" { + type = string + description = "Name of the model deployment in the Azure OpenAI resource, for example 'gpt-4o'. Azure routes on the deployment name, not on the model name." +} + +variable "model_name" { + type = string + description = "Name the model is offered under on the LiteLLM gateway. Application teams pass this name in the 'model' field of their requests." +} + +variable "mode" { + type = string + default = "chat" + description = "What the deployment is used for. LiteLLM accepts 'chat', 'completion', 'embedding', 'audio_speech', 'audio_transcription', 'image_generation', 'video_generation', 'batch' and 'rerank'." +} + +variable "team_id" { + type = string + default = null + description = "ID of the LiteLLM team the model is registered for. Only that team can call the model. Leave unset to register the model for the whole gateway." +} diff --git a/modules/ai/azure-openai/buildingblock/versions.tf b/modules/ai/azure-openai/buildingblock/versions.tf new file mode 100644 index 00000000..d0740798 --- /dev/null +++ b/modules/ai/azure-openai/buildingblock/versions.tf @@ -0,0 +1,15 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + litellm = { + source = "ncecere/litellm" + # Exact pin, a deliberate exception to the hub rule that provider constraints use '>='. + # ncecere/litellm is a community provider with a single maintainer, and it has changed + # resource behaviour inside a minor release before: v1.2.0 replaced the id of litellm_key + # with a hash of the key. The same pin is used by modules/ai/model-access, so both modules + # move to a new provider release together, after a review of the changelog. + version = "= 2.0.1" + } + } +} diff --git a/modules/ai/azure-openai/meshstack_integration.tf b/modules/ai/azure-openai/meshstack_integration.tf new file mode 100644 index 00000000..a1171ff4 --- /dev/null +++ b/modules/ai/azure-openai/meshstack_integration.tf @@ -0,0 +1,291 @@ +variable "litellm_api_base" { + type = string + description = "Base URL of the LiteLLM gateway, for example 'https://litellm.example.com'. The model is registered on this gateway." +} + +variable "litellm_admin_api_key" { + type = string + sensitive = true + description = "LiteLLM admin key the building block authenticates with. It needs permission to register models." +} + +variable "litellm_platform_type_name" { + type = string + default = "LiteLLM" + description = "Name of the meshStack platform type the LiteLLM platform is registered under. It must match the platform type used by the `ai/litellm` module." +} + +variable "litellm_model_name" { + type = string + default = "azure-gpt-4o" + description = "Name the model is offered under on the gateway. Application teams pass this name in the 'model' field of their requests." +} + +variable "litellm_model_mode" { + type = string + default = "chat" + description = "What the deployment is used for. LiteLLM accepts 'chat', 'completion', 'embedding', 'audio_speech', 'audio_transcription', 'image_generation', 'video_generation', 'batch' and 'rerank'." +} + +variable "azure_openai_endpoint" { + type = string + description = "Endpoint of the Azure OpenAI resource, for example 'https://my-aoai.openai.azure.com'." +} + +variable "azure_openai_api_key" { + type = string + sensitive = true + description = "Key of the Azure OpenAI resource. LiteLLM stores it and sends it upstream, so no application team ever sees it." +} + +variable "azure_openai_api_version" { + type = string + default = "2024-10-21" + description = "Azure OpenAI data plane API version. '2024-10-21' is the latest dated GA version of the inference API." +} + +variable "azure_openai_deployment_name" { + type = string + default = "gpt-4o" + description = "Name of the model deployment in the Azure OpenAI resource. Azure routes on the deployment name, not on the model name." +} + +variable "meshstack" { + type = object({ + owning_workspace_identifier = string + tags = optional(map(list(string)), {}) + }) + description = "Shared meshStack context. Tags are optional and propagated to building block definition metadata." +} + +variable "hub" { + type = object({ + git_ref = optional(string, "main") + bbd_draft = optional(bool, true) + }) + const = true + default = { + git_ref = "main" + bbd_draft = true + } + description = <<-EOT + `git_ref`: Hub release reference. Set to a tag (e.g. 'v1.2.3') or branch or commit sha of meshcloud/meshstack-hub repo. + `bbd_draft`: If true, allows changing the building block definition for upgrading dependent building blocks. + EOT +} + +output "building_block_definition" { + description = "BBD is consumed in building block compositions, for example by the ai-platform reference architecture." + value = { + uuid = meshstack_building_block_definition.this.metadata.uuid + version_ref = var.hub.bbd_draft ? meshstack_building_block_definition.this.version_latest : meshstack_building_block_definition.this.version_latest_release + } +} + +resource "meshstack_building_block_definition" "this" { + metadata = { + owned_by_workspace = var.meshstack.owning_workspace_identifier + tags = var.meshstack.tags + } + + spec = { + display_name = "Azure OpenAI Model Backend" + symbol = "https://raw.githubusercontent.com/meshcloud/meshstack-hub/${var.hub.git_ref}/modules/ai/azure-openai/buildingblock/logo.png" + description = "Registers an Azure OpenAI deployment as a model backend on the LiteLLM gateway, offered to the ordering team alone." + support_url = "https://learn.microsoft.com/azure/ai-foundry/openai/" + target_type = "TENANT_LEVEL" + run_transparency = true + supported_platforms = [{ name = var.litellm_platform_type_name }] + + readme = chomp(<<-EOT + This building block adds an Azure OpenAI model to your LiteLLM team. The model is registered + for your team alone, so the gateway offers it to you and counts its spend against your budget. + You keep calling the same LiteLLM endpoint and only pass a different model name. + + ## 🎯 When to use it + + Use this building block when you: + - Need a model that Azure OpenAI serves, next to or instead of the sovereign models the platform offers by default. + - Want the Azure credential to stay on the gateway rather than in your application. + - Want the calls to appear in the same budget and the same usage reports as the rest of your model traffic. + + ## πŸ’‘ Usage examples + + **Example 1: Adding GPT-4o to an existing assistant** + A team already calls the gateway through its virtual key. It orders this building block, reads + the `model_name` output and changes the `model` field of its requests to that name. Nothing + else in the application changes. + + **Example 2: Comparing two models before choosing one** + A team wants to compare a sovereign model against Azure OpenAI on its own prompts. Both models + answer on the same endpoint and the same key, so the comparison is one field in the request. + + ## πŸ”‘ Calling the model + + Send your virtual key as a bearer token and pass the model name in the `model` field: + + ```sh + curl "$API_BASE/chat/completions" \ + -H "Authorization: Bearer $VIRTUAL_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "$MODEL_NAME", "messages": [{"role": "user", "content": "Hello"}]}' + ``` + + ## πŸ“Š Shared Responsibility + + | Responsibility | Platform Team | Application Team | + |---|:---:|:---:| + | Create the Azure OpenAI resource and its model deployment | βœ… | ❌ | + | Provide and rotate the Azure OpenAI credential on the gateway | βœ… | ❌ | + | Choose which Azure deployment is offered and under which model name | βœ… | ❌ | + | Watch the Azure quota the deployment draws from | βœ… | ❌ | + | Pass the model name in requests to the gateway | ❌ | βœ… | + | Stay within the granted budget | ❌ | βœ… | + | Build and operate the application that calls the model | ❌ | βœ… | + EOT + ) + } + + version_spec = { + draft = var.hub.bbd_draft + deletion_mode = "DELETE" + + # A second model entry with the same name for the same team would only duplicate the first one, + # so meshStack allows one per tenant. Register a further deployment with its own definition. + only_apply_once_per_tenant = true + + implementation = { + terraform = { + terraform_version = "1.12.2" + repository_url = "https://github.com/meshcloud/meshstack-hub.git" + repository_path = "modules/ai/azure-openai/buildingblock" + ref_name = var.hub.git_ref + async = false + use_mesh_http_backend_fallback = true + } + } + + inputs = { + litellm_api_base = { + display_name = "LiteLLM API Base URL" + description = "Base URL of the LiteLLM gateway the model is registered on." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_api_base) + } + + litellm_api_key = { + display_name = "LiteLLM Admin Key" + description = "Admin key the building block authenticates with against the LiteLLM API." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.litellm_admin_api_key + secret_version = nonsensitive(sha256(var.litellm_admin_api_key)) + } + } + } + + team_id = { + display_name = "LiteLLM Team ID" + description = "ID of the LiteLLM team the model is registered for. It is the platform tenant ID that the LiteLLM Team building block produces." + type = "STRING" + assignment_type = "PLATFORM_TENANT_ID" + } + + azure_openai_endpoint = { + display_name = "Azure OpenAI Endpoint" + description = "Endpoint of the Azure OpenAI resource the deployment lives in." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.azure_openai_endpoint) + } + + azure_openai_api_key = { + display_name = "Azure OpenAI Key" + description = "Key of the Azure OpenAI resource. LiteLLM stores it and sends it upstream." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.azure_openai_api_key + secret_version = nonsensitive(sha256(var.azure_openai_api_key)) + } + } + } + + azure_openai_api_version = { + display_name = "Azure OpenAI API Version" + description = "Data plane API version LiteLLM calls the deployment with." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.azure_openai_api_version) + } + + azure_deployment_name = { + display_name = "Azure Deployment Name" + description = "Name of the model deployment in the Azure OpenAI resource." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.azure_openai_deployment_name) + } + + model_name = { + display_name = "Model Name" + description = "Name the model is offered under on the gateway." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_model_name) + } + + mode = { + display_name = "Mode" + description = "What the deployment is used for, for example 'chat' or 'embedding'." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_model_mode) + } + } + + outputs = { + model_name = { + display_name = "Model Name" + description = "Name to pass in the 'model' field of a request to the gateway." + type = "STRING" + assignment_type = "NONE" + } + + model_id = { + display_name = "Model ID" + description = "ID LiteLLM gave the model entry." + type = "STRING" + assignment_type = "NONE" + } + + api_base = { + display_name = "API Base URL" + description = "OpenAI-compatible base URL of the gateway, including the '/v1' suffix." + type = "STRING" + assignment_type = "NONE" + } + + summary = { + display_name = "Summary" + type = "STRING" + assignment_type = "SUMMARY" + } + } + } +} + +terraform { + required_version = ">= 1.12.0" + + required_providers { + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.23.0" + } + } +} diff --git a/modules/ai/clickhouse/buildingblock/.helmignore b/modules/ai/clickhouse/buildingblock/.helmignore new file mode 100644 index 00000000..4758f993 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/.helmignore @@ -0,0 +1,25 @@ +# The module directory is the chart directory, so everything that is not part of the chart has +# to be excluded here. Helm stores the packaged chart in a Kubernetes Secret β€” keeping it small +# is critical. + +# Terraform files and state +*.tf +*.tfstate +*.tfstate.backup +*.tfplan +.terraform/ +.terraform.lock.hcl + +# Terraform test files +*.tftest.hcl + +# Kubernetes credentials β€” never bundle into the chart +kubeconfig.yaml +kubeconfig-mock.yaml + +# Misc +.DS_Store +*.png +*.svg +README.md +APP_TEAM_README.md diff --git a/modules/ai/clickhouse/buildingblock/APP_TEAM_README.md b/modules/ai/clickhouse/buildingblock/APP_TEAM_README.md new file mode 100644 index 00000000..1d03b4c6 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/APP_TEAM_README.md @@ -0,0 +1,29 @@ +The platform team runs one ClickHouse cluster for the whole AI platform, and your Langfuse instance stores its traces in it. You get a database and a user of your own inside that cluster, so no other team can read or change your data. You never connect to ClickHouse yourself: Langfuse does it for you. + +## 🎯 When to use it + +Use this building block when you: +- run the AI platform and need the column store that Langfuse writes traces, observations and scores to +- want one ClickHouse for every tenant instead of one cluster per team, because a cluster per team costs several gigabytes of memory each +- need a ClickHouse release that Langfuse v4 accepts, which the version bundled with the Langfuse chart is not + +## πŸ’‘ Usage examples + +**Example 1: A team gets a Langfuse instance** +Your team orders a Langfuse instance. The platform creates a database named after your team in the shared cluster and a user that can only touch that database. Your traces land there and stay separate from every other team's. + +**Example 2: A trace query stays fast as volume grows** +Your application sends a few million spans a month. ClickHouse is a column store, so the dashboard queries that scan those spans stay fast without you tuning anything, and the platform team grows the volume when it fills up. + +## πŸ“Š Shared Responsibility + +| Responsibility | Platform Team | Application Team | +|---|:---:|:---:| +| Run the ClickHouse operator and the cluster | βœ… | ❌ | +| Size the replicas, the memory and the volumes | βœ… | ❌ | +| Create a database and a scoped user per tenant | βœ… | ❌ | +| Hold the administrative credential | βœ… | ❌ | +| Upgrade ClickHouse and ClickHouse Keeper | βœ… | ❌ | +| Decide what the application sends to Langfuse | ❌ | βœ… | +| Keep the trace volume inside the agreed quota | ❌ | βœ… | +| Delete traces that must not be retained | ❌ | βœ… | diff --git a/modules/ai/clickhouse/buildingblock/Chart.yaml b/modules/ai/clickhouse/buildingblock/Chart.yaml new file mode 100644 index 00000000..c09f742f --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/Chart.yaml @@ -0,0 +1,5 @@ +apiVersion: v2 +name: meshstack-clickhouse-cluster +description: ClickHouseCluster and KeeperCluster custom resources for the meshStack shared ClickHouse building block. +type: application +version: 1.0.0 diff --git a/modules/ai/clickhouse/buildingblock/README.md b/modules/ai/clickhouse/buildingblock/README.md new file mode 100644 index 00000000..129e8c78 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/README.md @@ -0,0 +1,184 @@ +--- +name: Shared ClickHouse Cluster +supportedPlatforms: + - kubernetes +description: Installs the official ClickHouse Kubernetes operator and one operator-managed ClickHouse and ClickHouse Keeper cluster, sized for a demonstration and shared by every tenant of the AI platform. +# The cluster credentials arrive through the providers the caller configures and the administrative +# password arrives as an input, so there is nothing to set up on the cloud side. +requiresBackplane: false +--- + +# Shared ClickHouse Cluster Building Block + +The platform team installs one ClickHouse cluster into the AI platform's Kubernetes cluster with this module. Every tenant's Langfuse instance stores its traces, observations and scores in that one cluster, separated by a database and a user of its own. + +This documentation is intended as a reference for cloud foundation or platform engineers using this module. + +## Sourced, not ordered + +There is no `meshstack_integration.tf` and no `backplane/`. A tenant-facing building block sources `buildingblock/` and so does a foundation from a Terragrunt unit. Application teams never order this module: they order a Langfuse instance, and the composition behind it wires that instance to the cluster this module installs. + +## Deployment cardinality + +**This module is deployed once per Kubernetes cluster.** The operator installs cluster-scoped CustomResourceDefinitions and watches every namespace, and the ClickHouse cluster it manages is shared by all tenants. Instantiating the module a second time in the same Kubernetes cluster installs the operator twice. + +`modules/ai/langfuse` is the opposite: it is instantiated once per tenant against the cluster this module provides. + +## Prerequisites + +| Prerequisite | Where it comes from | +|---|---| +| cert-manager | `modules/kubernetes/ingress` installs it. | +| A default StorageClass, or an explicit one | The cluster. ClickHouse and Keeper both claim persistent volumes. | +| A configured `kubernetes` and `helm` provider | The caller. | + +The ClickHouse operator serves its admission webhook over TLS from a certificate cert-manager issues, so **cert-manager has to be running before this module applies**. This module does not install it, because `modules/kubernetes/ingress` already does and installing cert-manager twice in one cluster fails on the shared CRDs. + +## The operator, not the bundled subchart + +The Langfuse Helm chart bundles a Bitnami ClickHouse subchart. That subchart's image is older than the 25.12 floor Langfuse v4 requires, which is why the `langfuse-k8s` [v4 installation example](https://github.com/langfuse/langfuse-k8s/tree/main/examples/v4-installation) runs ClickHouse outside the chart under the [official ClickHouse Kubernetes operator](https://github.com/ClickHouse/clickhouse-operator). This module follows that example. + +| | | +|---|---| +| Operator chart | `clickhouse-operator-helm` | +| Reference | `oci://ghcr.io/clickhouse/clickhouse-operator-helm` | +| Pinned version | `0.0.5` (`var.operator_chart_version`) | +| Custom resources | `clickhouse.com/v1alpha1` `ClickHouseCluster` and `KeeperCluster` | +| ClickHouse image | `clickhouse/clickhouse-server:26.4` (`var.clickhouse_version`) | +| Keeper image | `clickhouse/clickhouse-keeper:26.4` (`var.keeper_version`) | + +The operator chart is published to GHCR as an OCI artifact only, so the tag on the registry is the single source of truth. The helm provider takes the registry and the path prefix as `repository` and the chart name on its own as `chart`. + +### Why the custom resources run through a local Helm chart + +A `kubernetes_manifest` resource looks the CRD schema up at plan time. The operator installs the `ClickHouseCluster` and `KeeperCluster` CRDs, so the schema does not exist during the first plan and the plan fails. + +This module renders both custom resources through a Helm chart that lives in the module directory itself (`chart = path.module`). Helm applies the manifests at apply time and never asks Terraform for a schema, so the operator and its custom resources fit into a single Terraform unit. That is why the module directory carries a `Chart.yaml`, a `templates/` directory and a `.helmignore` that keeps every Terraform artifact out of the packaged chart. `modules/kubernetes/ingress` uses the same pattern for its ClusterIssuer. + +### The readiness Job + +Helm's `--wait` watches the resources a release creates. A `ClickHouseCluster` is a custom resource, and Helm has no idea what "ready" means for it, so the release reports success while the operator is still creating the StatefulSets. A Langfuse instance that starts against a ClickHouse that is not up yet crash-loops through its migrations. + +The chart therefore ships a `post-install,post-upgrade` hook Job that runs `clickhouse-client --query "SELECT 1"` in a loop until the server answers. Helm waits for a hook Job to finish, so the Terraform apply only completes once ClickHouse accepts queries. The Job uses the same image the server runs, so no second image has to be pulled. `var.wait_for_ready` turns it off and `var.readiness_timeout` bounds it β€” keep the timeout below `var.helm_timeout`, otherwise the Helm release times out first and reports a less useful error. + +## No provider blocks + +This module carries no `provider` block. The caller configures the `kubernetes` and the `helm` provider and passes both down through the `providers` argument of the module call. + +This is not a style preference. **A module with its own provider configuration cannot be called with `count` or `for_each`.** `modules/kubernetes/ingress` had to delete its `provider.tf` for exactly that reason, and the same constraint applies here, because a composition that deploys several platforms wants the shared ClickHouse under a `count`. + +## Sizing + +Every figure is a variable and every default is sized for a demonstration. The `langfuse-k8s` example ships the production shape: 3 ClickHouse replicas, 3 Keeper replicas, a 100Gi data volume per ClickHouse replica and 10Gi per Keeper replica. + +| | Default here | `langfuse-k8s` example | Note | +|---|---|---|---| +| ClickHouse replicas | 1 | 3 | One replica gives no redundancy: every restart interrupts ingestion. | +| ClickHouse storage | `20Gi` | `100Gi` | Growing a volume later depends on CSI volume expansion. | +| ClickHouse memory | request `1Gi`, limit `2Gi` | Bitnami `2xlarge` preset | See the floor below. | +| Keeper replicas | 1 | 3 | Raft: the CustomResource only accepts 0, 1, 3, 5, 7, 9, 11, 13 or 15. | +| Keeper storage | `5Gi` | `10Gi` | | +| Keeper memory | request `256Mi`, limit `512Mi` | β€” | Grows with the number of tables, not with trace volume. | + +**`1Gi` of memory is a floor, not a target.** ClickHouse allocates its mark and uncompressed caches at startup and does not start reliably below roughly that figure, so a smaller limit produces a pod that is OOMKilled during boot rather than a slow server. `var.clickhouse_resources` validates the request and the limit against that floor and rejects anything below it at plan time. + +## Per-tenant separation, and who creates it + +A Langfuse tenant needs two things in ClickHouse: **a database of its own** and **a user scoped to that database**. Neither is created here. + +**The composition creates them** β€” the tenant-facing building block that sources both `modules/ai/clickhouse` and `modules/ai/langfuse`. Three reasons: + +1. **This module is deployed once and must not churn with tenants.** A map of tenants as an input would mean every tenant order re-plans the operator release, the ClickHouseCluster and every other tenant's grants. Shared infrastructure would then share the failure domain of tenant churn. +2. **`modules/ai/langfuse` must never see the administrative credential.** It runs once per tenant, so its state is per-tenant state. A credential in there can drop any other tenant's database. Langfuse takes a tenant username and password as inputs, which means something else already created that user. +3. **The composition already owns the same decision for Postgres.** Langfuse takes `postgresql.auth.database` and `postgresql.auth.username` as inputs too, so whoever creates the Postgres database and its owner also creates the ClickHouse database and its user. Splitting the two across modules would put one tenant's identity in two places. + +`modules/ai/clickhouse` supports the composition by exposing the host, the ports, the ON CLUSTER cluster name and the administrative credential. `modules/ai/langfuse` supports it by taking the tenant's database name, username and password as explicit inputs. + +### The statements a composition runs + +```sql +CREATE DATABASE IF NOT EXISTS tenant_x ON CLUSTER default; + +CREATE USER IF NOT EXISTS tenant_x ON CLUSTER default + IDENTIFIED WITH sha256_password BY '' + DEFAULT DATABASE tenant_x; + +GRANT ON CLUSTER default + SELECT, INSERT, CREATE, DROP TABLE, ALTER UPDATE, ALTER DELETE, ALTER DROP INDEX + ON tenant_x.* TO tenant_x; +``` + +Three points about those statements: + +- **The tenant user does not need `CREATE DATABASE`.** Langfuse runs its ClickHouse schema migrations with golang-migrate, and golang-migrate never creates a database β€” it only creates tables inside one that already exists. Granting `CREATE DATABASE` would let a tenant create databases outside its own scope. +- **`ON CLUSTER default` is required** whenever the ClickHouse cluster has more than one replica, and harmless with one. `default` is the cluster name the operator configures, exposed as the `ddl_cluster_name` output. +- **The clause does not sit in the same place in every statement.** `CREATE DATABASE` and `CREATE USER` take it after the object name, but `GRANT` and `REVOKE` take it directly after the keyword, before the privilege list β€” `GRANT ON CLUSTER default SELECT ... ON db.* TO user`. Putting it at the end of a `GRANT` is a syntax error. + +There is no first-class Terraform provider for DDL against a self-hosted ClickHouse. In practice a composition runs the statements from a Kubernetes Job in the ClickHouse namespace, using the `clickhouse/clickhouse-server` image and mounting the administrative password from the Secret named by the `admin_secret` output. + +## Upgrading ClickHouse + +`var.clickhouse_version` and `var.keeper_version` are separate variables, but they belong on the same release: the two speak one protocol. Raise Keeper first, then ClickHouse. The operator rolls the StatefulSets one replica at a time, so a single-replica demonstration cluster is unavailable for the duration of the upgrade. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [helm](#requirement\_helm) | >= 3.0.0 | +| [kubernetes](#requirement\_kubernetes) | >= 2.38 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [helm_release.clickhouse_operator](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.cluster](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [kubernetes_namespace_v1.clickhouse](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_namespace_v1.operator](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_secret_v1.admin](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [admin\_password](#input\_admin\_password) | Password of the administrative ClickHouse user. The caller that creates the per-tenant databases and users authenticates with it, so it must not be handed to a tenant. | `string` | n/a | yes | +| [admin\_username](#input\_admin\_username) | Name of the administrative ClickHouse user the operator creates. The operator only manages the 'default' user, so changing this does not create a different user. | `string` | `"default"` | no | +| [clickhouse\_replicas](#input\_clickhouse\_replicas) | Number of ClickHouse replicas. The default of 1 is sized for a demonstration cluster and gives no redundancy: every restart or node drain interrupts ingestion. Production wants 3. | `number` | `1` | no | +| [clickhouse\_resources](#input\_clickhouse\_resources) | Resource requests and limits of each ClickHouse server container. The default is sized for a
demonstration cluster and a production consumer has to raise it.

`1Gi` of memory is a floor, not a target. ClickHouse allocates mark and uncompressed caches at
startup and does not start reliably below roughly that figure, so a smaller request produces a
pod that is OOMKilled during boot rather than a slow server. The Langfuse chart's own bundled
ClickHouse asks for the Bitnami `2xlarge` preset. Production wants `2` to `4` CPUs and `8Gi` to
`16Gi` of memory per replica. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "2",
    "memory": "2Gi"
  },
  "requests": {
    "cpu": "500m",
    "memory": "1Gi"
  }
}
| no | +| [clickhouse\_storage](#input\_clickhouse\_storage) | Size of the data volume of each ClickHouse replica. The default of 20Gi is sized for a demonstration cluster. Production wants 100Gi or more, because trace data grows quickly and a later resize depends on CSI volume expansion. | `string` | `"20Gi"` | no | +| [clickhouse\_storage\_class\_name](#input\_clickhouse\_storage\_class\_name) | StorageClass of the ClickHouse data volumes. Null uses the default StorageClass of the cluster. | `string` | `null` | no | +| [clickhouse\_version](#input\_clickhouse\_version) | Tag of the clickhouse/clickhouse-server image. Langfuse v4 needs 25.12 or newer, so do not lower this below that floor. | `string` | `"26.4"` | no | +| [cluster\_name](#input\_cluster\_name) | Name of the ClickHouseCluster and KeeperCluster resources. The operator names the headless Service '-clickhouse-headless'. | `string` | `"clickhouse"` | no | +| [helm\_timeout](#input\_helm\_timeout) | Seconds to wait for each Helm release of this module. The operator install and the custom resources share the same budget per release. | `number` | `900` | no | +| [keeper\_replicas](#input\_keeper\_replicas) | Number of ClickHouse Keeper replicas. The CustomResource accepts only 0, 1, 3, 5, 7, 9, 11, 13 or 15. The default of 1 is sized for a demonstration cluster and gives no redundancy. Production wants 3. | `number` | `1` | no | +| [keeper\_resources](#input\_keeper\_resources) | Resource requests and limits of each ClickHouse Keeper container. The default is sized for a
demonstration cluster and a production consumer has to raise it.

Keeper holds the coordination state β€” replication queues, the distributed DDL queue and part
metadata β€” in memory, so its footprint grows with the number of tables rather than with the
volume of trace data. Production wants `500m` CPU and `1Gi` to `2Gi` of memory per replica. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "500m",
    "memory": "512Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "256Mi"
  }
}
| no | +| [keeper\_storage](#input\_keeper\_storage) | Size of the data volume of each Keeper replica. The default of 5Gi is sized for a demonstration cluster. The langfuse-k8s example ships 10Gi, which is the production target. | `string` | `"5Gi"` | no | +| [keeper\_storage\_class\_name](#input\_keeper\_storage\_class\_name) | StorageClass of the Keeper data volumes. Null uses the default StorageClass of the cluster. | `string` | `null` | no | +| [keeper\_version](#input\_keeper\_version) | Tag of the clickhouse/clickhouse-keeper image. Keep it on the same release as clickhouse\_version, because the two speak one protocol. | `string` | `"26.4"` | no | +| [namespace](#input\_namespace) | Namespace the ClickHouse cluster and the ClickHouse Keeper cluster run in. The module creates it. Every tenant's Langfuse instance connects across namespaces to the Service in here. | `string` | `"clickhouse"` | no | +| [operator\_chart\_version](#input\_operator\_chart\_version) | Version of the clickhouse-operator-helm chart. See https://github.com/ClickHouse/clickhouse-operator/pkgs/container/clickhouse-operator-helm. | `string` | `"0.0.5"` | no | +| [operator\_namespace](#input\_operator\_namespace) | Namespace the ClickHouse operator runs in. The module creates it. The operator watches every namespace, so one installation serves the whole cluster. | `string` | `"clickhouse-operator"` | no | +| [operator\_resources](#input\_operator\_resources) | Resource requests and limits of the ClickHouse operator controller manager. The values match the
chart's own defaults, which are already small, so a production consumer rarely has to change
them. The controller reconciles custom resources and holds no data. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "500m",
    "memory": "128Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "64Mi"
  }
}
| no | +| [readiness\_timeout](#input\_readiness\_timeout) | Seconds the readiness Job waits for ClickHouse to answer a query before it fails. Keep it below helm\_timeout, otherwise the Helm release times out first and reports a less useful error. | `number` | `900` | no | +| [wait\_for\_ready](#input\_wait\_for\_ready) | Run a Helm hook Job after the custom resources are applied that blocks until ClickHouse answers a query. Turn it off only when the caller waits for readiness itself. | `bool` | `true` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [admin\_password](#output\_admin\_password) | Password of the administrative ClickHouse user. It grants full access to every tenant's database, so never pass it to a tenant. | +| [admin\_secret](#output\_admin\_secret) | Name and key of the Kubernetes Secret in the ClickHouse namespace that holds the administrative password. A Job that creates per-tenant databases can mount it instead of taking the value through Terraform. | +| [admin\_username](#output\_admin\_username) | Name of the administrative ClickHouse user. Use it to create the per-tenant databases and users. | +| [cluster\_name](#output\_cluster\_name) | Name of the ClickHouseCluster and KeeperCluster resources. | +| [ddl\_cluster\_name](#output\_ddl\_cluster\_name) | Name of the ClickHouse cluster as the server knows it. Every ON CLUSTER statement β€” the Langfuse migrations included β€” names it. | +| [host](#output\_host) | Fully qualified in-cluster hostname of the ClickHouse headless Service. Pass it to the Langfuse chart as clickhouse.host. | +| [http\_port](#output\_http\_port) | HTTP port of ClickHouse. Langfuse reads and writes trace data over it and the chart takes it as clickhouse.httpPort. | +| [namespace](#output\_namespace) | Namespace the ClickHouse cluster runs in. | +| [native\_port](#output\_native\_port) | Native protocol port of ClickHouse. The golang-migrate migrations and clickhouse-client use it, and the Langfuse chart takes it as clickhouse.nativePort. | + diff --git a/modules/ai/clickhouse/buildingblock/logo.png b/modules/ai/clickhouse/buildingblock/logo.png new file mode 100644 index 00000000..bfed8248 Binary files /dev/null and b/modules/ai/clickhouse/buildingblock/logo.png differ diff --git a/modules/ai/clickhouse/buildingblock/main.tf b/modules/ai/clickhouse/buildingblock/main.tf new file mode 100644 index 00000000..c7f06d9d --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/main.tf @@ -0,0 +1,152 @@ +locals { + # The operator derives the name of the public headless Service from the CustomResource: + # '-clickhouse' plus the 'headless' suffix. Every client outside the namespace needs the + # fully qualified name. + headless_service_name = "${var.cluster_name}-clickhouse-headless" + host = "${local.headless_service_name}.${var.namespace}.svc.cluster.local" + + # Fixed by the operator rather than configurable, so pinning them here keeps the outputs and the + # readiness Job in step with the ports the Service actually publishes. + http_port = 8123 + native_port = 9000 + + # The operator always names the ClickHouse cluster 'default'. Langfuse writes its ON CLUSTER DDL + # against that name, so a consumer that overrides it breaks the migrations. + ddl_cluster_name = "default" + + admin_secret_key = "password" + + # Helm renders these values into the pod spec as YAML, and the API server rejects a resource + # quantity that is null, so drop every field the caller left unset. + resources = { + for name, spec in { + operator = var.operator_resources + clickhouse = var.clickhouse_resources + keeper = var.keeper_resources + } : name => { + requests = { for key, value in spec.requests : key => value if value != null } + limits = { for key, value in spec.limits : key => value if value != null } + } + } +} + +resource "kubernetes_namespace_v1" "operator" { + metadata { + name = var.operator_namespace + } +} + +# The operator is cluster-scoped: it installs the ClickHouseCluster and KeeperCluster CRDs and +# watches every namespace. Install it once per Kubernetes cluster. +# +# Its admission webhook serves TLS from a certificate cert-manager issues, so cert-manager has to +# be present before this release. In this hub, modules/kubernetes/ingress installs it. +resource "helm_release" "clickhouse_operator" { + name = "clickhouse-operator" + namespace = kubernetes_namespace_v1.operator.metadata[0].name + repository = "oci://ghcr.io/clickhouse" + chart = "clickhouse-operator-helm" + version = var.operator_chart_version + + create_namespace = false + atomic = true + wait = true + timeout = var.helm_timeout + + values = [ + yamlencode({ + manager = { + resources = local.resources.operator + } + crd = { + enabled = true + # Keeping the CRDs on uninstall preserves every ClickHouseCluster and KeeperCluster object + # in the cluster. Deleting a CRD deletes its custom resources, and the operator would then + # tear down the StatefulSets and their volumes. + keep = true + } + certManager = { + enable = true + } + }) + ] +} + +resource "kubernetes_namespace_v1" "clickhouse" { + metadata { + name = var.namespace + } +} + +# The operator reads the password of the 'default' user from this secret, and the caller that +# creates the per-tenant databases and users authenticates with the same value. +resource "kubernetes_secret_v1" "admin" { + metadata { + name = "${var.cluster_name}-admin" + namespace = kubernetes_namespace_v1.clickhouse.metadata[0].name + } + + data = { + (local.admin_secret_key) = var.admin_password + } +} + +# ClickHouseCluster and KeeperCluster are custom resources whose CRDs only exist once the operator +# is installed. A kubernetes_manifest resource looks the schema up at plan time, so the first plan +# of this module would fail. Helm renders and applies the manifests at apply time and never asks +# Terraform for a schema, which is why the module directory itself is a chart with a Chart.yaml, a +# templates/ directory and a .helmignore that keeps every Terraform artifact out of the package. +resource "helm_release" "cluster" { + name = var.cluster_name + namespace = kubernetes_namespace_v1.clickhouse.metadata[0].name + chart = path.module + + create_namespace = false + atomic = true + wait = true + timeout = var.helm_timeout + + values = [ + yamlencode({ + cluster = { + name = var.cluster_name + } + + auth = { + username = var.admin_username + secretName = kubernetes_secret_v1.admin.metadata[0].name + secretKey = local.admin_secret_key + } + + clickhouse = { + replicas = var.clickhouse_replicas + image = { + repository = "clickhouse/clickhouse-server" + tag = var.clickhouse_version + } + storage = var.clickhouse_storage + storageClassName = var.clickhouse_storage_class_name + resources = local.resources.clickhouse + nativePort = local.native_port + } + + keeper = { + replicas = var.keeper_replicas + image = { + repository = "clickhouse/clickhouse-keeper" + tag = var.keeper_version + } + storage = var.keeper_storage + storageClassName = var.keeper_storage_class_name + resources = local.resources.keeper + } + + readiness = { + enabled = var.wait_for_ready + timeoutSeconds = var.readiness_timeout + } + }) + ] + + depends_on = [helm_release.clickhouse_operator] +} diff --git a/modules/ai/clickhouse/buildingblock/outputs.tf b/modules/ai/clickhouse/buildingblock/outputs.tf new file mode 100644 index 00000000..ca9c9bb7 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/outputs.tf @@ -0,0 +1,51 @@ +output "namespace" { + description = "Namespace the ClickHouse cluster runs in." + value = kubernetes_namespace_v1.clickhouse.metadata[0].name +} + +output "cluster_name" { + description = "Name of the ClickHouseCluster and KeeperCluster resources." + value = var.cluster_name +} + +output "host" { + description = "Fully qualified in-cluster hostname of the ClickHouse headless Service. Pass it to the Langfuse chart as clickhouse.host." + value = local.host + + depends_on = [helm_release.cluster] +} + +output "http_port" { + description = "HTTP port of ClickHouse. Langfuse reads and writes trace data over it and the chart takes it as clickhouse.httpPort." + value = local.http_port +} + +output "native_port" { + description = "Native protocol port of ClickHouse. The golang-migrate migrations and clickhouse-client use it, and the Langfuse chart takes it as clickhouse.nativePort." + value = local.native_port +} + +output "ddl_cluster_name" { + description = "Name of the ClickHouse cluster as the server knows it. Every ON CLUSTER statement β€” the Langfuse migrations included β€” names it." + value = local.ddl_cluster_name +} + +output "admin_username" { + description = "Name of the administrative ClickHouse user. Use it to create the per-tenant databases and users." + value = var.admin_username +} + +output "admin_password" { + description = "Password of the administrative ClickHouse user. It grants full access to every tenant's database, so never pass it to a tenant." + value = var.admin_password + sensitive = true +} + +output "admin_secret" { + description = "Name and key of the Kubernetes Secret in the ClickHouse namespace that holds the administrative password. A Job that creates per-tenant databases can mount it instead of taking the value through Terraform." + value = { + name = kubernetes_secret_v1.admin.metadata[0].name + namespace = kubernetes_namespace_v1.clickhouse.metadata[0].name + key = local.admin_secret_key + } +} diff --git a/modules/ai/clickhouse/buildingblock/templates/clickhouse-cluster.yaml b/modules/ai/clickhouse/buildingblock/templates/clickhouse-cluster.yaml new file mode 100644 index 00000000..97c1cebd --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/templates/clickhouse-cluster.yaml @@ -0,0 +1,35 @@ +# The operator turns this resource into a StatefulSet per replica, a public headless Service named +# '-clickhouse-headless' and a ClickHouse cluster named 'default'. Langfuse writes its +# ON CLUSTER DDL against that cluster name. +apiVersion: clickhouse.com/v1alpha1 +kind: ClickHouseCluster +metadata: + name: {{ .Values.cluster.name }} + namespace: {{ .Release.Namespace }} +spec: + replicas: {{ .Values.clickhouse.replicas }} + # Fixed at 1 β€” Langfuse does not support a sharded ClickHouse, and its own Helm chart fails the + # install with any other value. + shards: 1 + keeperClusterRef: + name: {{ .Values.cluster.name }} + containerTemplate: + image: + repository: {{ .Values.clickhouse.image.repository | quote }} + tag: {{ .Values.clickhouse.image.tag | quote }} + resources: + {{- toYaml .Values.clickhouse.resources | nindent 6 }} + dataVolumeClaimSpec: + {{- with .Values.clickhouse.storageClassName }} + storageClassName: {{ . | quote }} + {{- end }} + resources: + requests: + storage: {{ .Values.clickhouse.storage | quote }} + settings: + # The operator creates the 'default' user with this password. Terraform owns the secret, so the + # same value reaches the caller that creates the per-tenant databases and users. + defaultUserPassword: + secret: + name: {{ .Values.auth.secretName }} + key: {{ .Values.auth.secretKey }} diff --git a/modules/ai/clickhouse/buildingblock/templates/keeper-cluster.yaml b/modules/ai/clickhouse/buildingblock/templates/keeper-cluster.yaml new file mode 100644 index 00000000..1b3996c6 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/templates/keeper-cluster.yaml @@ -0,0 +1,24 @@ +# Keeper replaces ZooKeeper as the coordination service of the cluster. It holds the replication +# queues and the distributed DDL queue, so the ClickHouseCluster below refers to it and cannot +# start without it. +apiVersion: clickhouse.com/v1alpha1 +kind: KeeperCluster +metadata: + name: {{ .Values.cluster.name }} + namespace: {{ .Release.Namespace }} +spec: + # Raft needs an odd number of replicas to form a quorum. + replicas: {{ .Values.keeper.replicas }} + containerTemplate: + image: + repository: {{ .Values.keeper.image.repository | quote }} + tag: {{ .Values.keeper.image.tag | quote }} + resources: + {{- toYaml .Values.keeper.resources | nindent 6 }} + dataVolumeClaimSpec: + {{- with .Values.keeper.storageClassName }} + storageClassName: {{ . | quote }} + {{- end }} + resources: + requests: + storage: {{ .Values.keeper.storage | quote }} diff --git a/modules/ai/clickhouse/buildingblock/templates/readiness-job.yaml b/modules/ai/clickhouse/buildingblock/templates/readiness-job.yaml new file mode 100644 index 00000000..5eddbc04 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/templates/readiness-job.yaml @@ -0,0 +1,59 @@ +{{- if .Values.readiness.enabled }} +# Helm's --wait watches the resources the release creates. A ClickHouseCluster is a custom +# resource, and Helm has no idea what "ready" means for it, so the release would report success +# while the operator is still creating the StatefulSets. This Job runs as a post-install and +# post-upgrade hook, and Helm waits for a hook Job to finish, so the release only completes once +# ClickHouse answers a query. +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Values.cluster.name }}-clickhouse-readiness + namespace: {{ .Release.Namespace }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + backoffLimit: 0 + # The deadline turns an endless wait into a failed release with a readable log, which is easier + # to diagnose than a Terraform timeout. + activeDeadlineSeconds: {{ .Values.readiness.timeoutSeconds }} + template: + metadata: + name: {{ .Values.cluster.name }}-clickhouse-readiness + spec: + restartPolicy: Never + containers: + - name: wait + # The same image the server runs, so the node has it in its cache already and no second + # image has to be pulled. + image: "{{ .Values.clickhouse.image.repository }}:{{ .Values.clickhouse.image.tag }}" + env: + - name: CLICKHOUSE_PASSWORD + valueFrom: + secretKeyRef: + name: {{ .Values.auth.secretName }} + key: {{ .Values.auth.secretKey }} + command: + - /bin/bash + - -c + - | + set -u + host="{{ .Values.cluster.name }}-clickhouse-headless.{{ .Release.Namespace }}.svc.cluster.local" + until clickhouse-client --host "$host" --port {{ .Values.clickhouse.nativePort }} \ + --user "{{ .Values.auth.username }}" --password "$CLICKHOUSE_PASSWORD" \ + --query "SELECT 1" > /dev/null 2>&1; do + echo "waiting for ClickHouse at $host:{{ .Values.clickhouse.nativePort }}" + sleep 5 + done + echo "ClickHouse is accepting queries" + # The pod runs a sleep loop and one query, so it needs almost nothing. A small request + # also keeps it schedulable on a node the ClickHouse replicas have already filled. + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi +{{- end }} diff --git a/modules/ai/clickhouse/buildingblock/variables.tf b/modules/ai/clickhouse/buildingblock/variables.tf new file mode 100644 index 00000000..f00245cc --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/variables.tf @@ -0,0 +1,217 @@ +variable "operator_namespace" { + type = string + default = "clickhouse-operator" + description = "Namespace the ClickHouse operator runs in. The module creates it. The operator watches every namespace, so one installation serves the whole cluster." +} + +variable "operator_chart_version" { + type = string + default = "0.0.5" + # The chart is published to GHCR as an OCI artifact only, so the tag on the registry is the + # single source of truth. The `langfuse-k8s` v4 installation example pins the same version. + description = "Version of the clickhouse-operator-helm chart. See https://github.com/ClickHouse/clickhouse-operator/pkgs/container/clickhouse-operator-helm." +} + +variable "operator_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "64Mi" } + limits = { cpu = "500m", memory = "128Mi" } + } + description = <<-EOT + Resource requests and limits of the ClickHouse operator controller manager. The values match the + chart's own defaults, which are already small, so a production consumer rarely has to change + them. The controller reconciles custom resources and holds no data. + EOT +} + +variable "namespace" { + type = string + default = "clickhouse" + description = "Namespace the ClickHouse cluster and the ClickHouse Keeper cluster run in. The module creates it. Every tenant's Langfuse instance connects across namespaces to the Service in here." +} + +variable "cluster_name" { + type = string + default = "clickhouse" + # The operator derives every owned resource name from this: the headless Service is + # '-clickhouse-headless' and the StatefulSets carry the same prefix. + description = "Name of the ClickHouseCluster and KeeperCluster resources. The operator names the headless Service '-clickhouse-headless'." +} + +variable "clickhouse_version" { + type = string + default = "26.4" + # Langfuse v4 requires ClickHouse 25.12 or newer. The ClickHouse version the Langfuse chart + # bundles is older than that, which is the reason the operator runs the server instead. + description = "Tag of the clickhouse/clickhouse-server image. Langfuse v4 needs 25.12 or newer, so do not lower this below that floor." +} + +variable "clickhouse_replicas" { + type = number + default = 1 + # The langfuse-k8s example ships 3, which is the production target: three replicas survive the + # loss of one node and let a rolling upgrade proceed without downtime. + description = "Number of ClickHouse replicas. The default of 1 is sized for a demonstration cluster and gives no redundancy: every restart or node drain interrupts ingestion. Production wants 3." + + validation { + condition = var.clickhouse_replicas >= 1 + error_message = "clickhouse_replicas must be at least 1." + } +} + +variable "clickhouse_storage" { + type = string + default = "20Gi" + # The langfuse-k8s example ships 100Gi and recommends starting large, because growing a volume + # afterwards depends on the CSI driver supporting expansion. + description = "Size of the data volume of each ClickHouse replica. The default of 20Gi is sized for a demonstration cluster. Production wants 100Gi or more, because trace data grows quickly and a later resize depends on CSI volume expansion." +} + +variable "clickhouse_storage_class_name" { + type = string + default = null + description = "StorageClass of the ClickHouse data volumes. Null uses the default StorageClass of the cluster." +} + +variable "clickhouse_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "500m", memory = "1Gi" } + limits = { cpu = "2", memory = "2Gi" } + } + description = <<-EOT + Resource requests and limits of each ClickHouse server container. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + `1Gi` of memory is a floor, not a target. ClickHouse allocates mark and uncompressed caches at + startup and does not start reliably below roughly that figure, so a smaller request produces a + pod that is OOMKilled during boot rather than a slow server. The Langfuse chart's own bundled + ClickHouse asks for the Bitnami `2xlarge` preset. Production wants `2` to `4` CPUs and `8Gi` to + `16Gi` of memory per replica. + EOT + + validation { + condition = var.clickhouse_resources.limits.memory == null ? true : ( + can(regex("^[0-9]+Gi$", var.clickhouse_resources.limits.memory)) + ? tonumber(trimsuffix(var.clickhouse_resources.limits.memory, "Gi")) >= 1 + : can(regex("^[0-9]+Mi$", var.clickhouse_resources.limits.memory)) + ? tonumber(trimsuffix(var.clickhouse_resources.limits.memory, "Mi")) >= 1024 + : true + ) + error_message = "clickhouse_resources.limits.memory must be at least 1Gi. ClickHouse does not start reliably below that." + } + + validation { + condition = var.clickhouse_resources.requests.memory == null ? true : ( + can(regex("^[0-9]+Gi$", var.clickhouse_resources.requests.memory)) + ? tonumber(trimsuffix(var.clickhouse_resources.requests.memory, "Gi")) >= 1 + : can(regex("^[0-9]+Mi$", var.clickhouse_resources.requests.memory)) + ? tonumber(trimsuffix(var.clickhouse_resources.requests.memory, "Mi")) >= 1024 + : true + ) + error_message = "clickhouse_resources.requests.memory must be at least 1Gi. ClickHouse does not start reliably below that." + } +} + +variable "keeper_version" { + type = string + default = "26.4" + description = "Tag of the clickhouse/clickhouse-keeper image. Keep it on the same release as clickhouse_version, because the two speak one protocol." +} + +variable "keeper_replicas" { + type = number + default = 1 + # Keeper runs Raft, so a quorum needs an odd number and tolerates (n-1)/2 failures. One replica + # is a quorum of one: correct, but it stops the whole cluster whenever that pod restarts. + description = "Number of ClickHouse Keeper replicas. The CustomResource accepts only 0, 1, 3, 5, 7, 9, 11, 13 or 15. The default of 1 is sized for a demonstration cluster and gives no redundancy. Production wants 3." + + validation { + # The CRD declares an enum on this field, so the API server rejects anything else with a + # message that names the whole list. Catching it here fails the plan instead of the apply. + condition = contains([1, 3, 5, 7, 9, 11, 13, 15], var.keeper_replicas) + error_message = "keeper_replicas must be one of 1, 3, 5, 7, 9, 11, 13 or 15, because Keeper needs a Raft quorum and the CustomResource restricts the field to those values." + } +} + +variable "keeper_storage" { + type = string + default = "5Gi" + description = "Size of the data volume of each Keeper replica. The default of 5Gi is sized for a demonstration cluster. The langfuse-k8s example ships 10Gi, which is the production target." +} + +variable "keeper_storage_class_name" { + type = string + default = null + description = "StorageClass of the Keeper data volumes. Null uses the default StorageClass of the cluster." +} + +variable "keeper_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "100m", memory = "256Mi" } + limits = { cpu = "500m", memory = "512Mi" } + } + description = <<-EOT + Resource requests and limits of each ClickHouse Keeper container. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + Keeper holds the coordination state β€” replication queues, the distributed DDL queue and part + metadata β€” in memory, so its footprint grows with the number of tables rather than with the + volume of trace data. Production wants `500m` CPU and `1Gi` to `2Gi` of memory per replica. + EOT +} + +variable "admin_username" { + type = string + default = "default" + # The operator creates exactly one user from the CustomResource, and that user is named + # 'default'. Every other user is created with SQL against this one. + description = "Name of the administrative ClickHouse user the operator creates. The operator only manages the 'default' user, so changing this does not create a different user." +} + +variable "admin_password" { + type = string + sensitive = true + # Handed to the operator through a Kubernetes Secret and used by whoever creates the per-tenant + # databases and users. Keep it out of any per-tenant state. + description = "Password of the administrative ClickHouse user. The caller that creates the per-tenant databases and users authenticates with it, so it must not be handed to a tenant." + + validation { + condition = length(var.admin_password) >= 16 + error_message = "The admin password must be at least 16 characters long." + } +} + +variable "helm_timeout" { + type = number + default = 900 + description = "Seconds to wait for each Helm release of this module. The operator install and the custom resources share the same budget per release." +} + +variable "wait_for_ready" { + type = bool + default = true + # Helm's own --wait never looks at custom resources, so without this Job the module reports + # success while the operator is still creating the StatefulSets. + description = "Run a Helm hook Job after the custom resources are applied that blocks until ClickHouse answers a query. Turn it off only when the caller waits for readiness itself." +} + +variable "readiness_timeout" { + type = number + default = 900 + description = "Seconds the readiness Job waits for ClickHouse to answer a query before it fails. Keep it below helm_timeout, otherwise the Helm release times out first and reports a less useful error." +} diff --git a/modules/ai/clickhouse/buildingblock/versions.tf b/modules/ai/clickhouse/buildingblock/versions.tf new file mode 100644 index 00000000..cc6c09d8 --- /dev/null +++ b/modules/ai/clickhouse/buildingblock/versions.tf @@ -0,0 +1,16 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + helm = { + source = "hashicorp/helm" + # The helm provider takes its cluster credentials as the `kubernetes = {}` attribute + # starting with 3.0.0. Earlier versions expect a `kubernetes {}` block instead. + version = ">= 3.0.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38" + } + } +} diff --git a/modules/ai/langfuse/buildingblock/APP_TEAM_README.md b/modules/ai/langfuse/buildingblock/APP_TEAM_README.md new file mode 100644 index 00000000..c895c4a5 --- /dev/null +++ b/modules/ai/langfuse/buildingblock/APP_TEAM_README.md @@ -0,0 +1,50 @@ +Your team gets a Langfuse instance of its own: a URL you log in to with your normal company account, a project that is already set up, and an API keypair you point a tracing client at. Everything your applications send β€” prompts, responses, latencies, token counts and cost β€” lands there and stays separate from every other team's data. + +## 🎯 When to use it + +Use this building block when you: +- want to see what your application actually sends to a language model and what comes back +- need to find out why an answer was wrong, slow or expensive, on the level of a single request +- want to compare prompt versions or models against each other with real traffic +- have to show what a feature costs per user, per request or per month + +## πŸ’‘ Usage examples + +**Example 1: Find out why one answer was wrong** +A user reports a bad answer. You open your Langfuse instance, search for the trace by its id and see the exact prompt, the retrieved documents and the model's reply. You change the prompt and compare the two versions on the same input. + +**Example 2: Trace everything through the AI gateway** +Your applications already call models through the platform's LiteLLM gateway. The platform team points that gateway at your Langfuse instance, so every call shows up as a trace without you adding a single line of code. + +## πŸ”§ How to use it + +Point the Langfuse SDK at your instance with the keypair you were given: + +```python +from langfuse import Langfuse + +langfuse = Langfuse( + host="https://langfuse-my-team.example.com", + public_key="pk-lf-...", + secret_key="sk-lf-...", +) +``` + +Two things go wrong often enough to name them: + +- **The secret key is a credential.** It grants full read and write access to your project's traces. Keep it in your secret store and rotate it in the Langfuse UI when it leaks. +- **Traces keep whatever you put in them.** Prompts and responses are stored as they were sent, so a prompt that carries personal data ends up in the trace. Decide what your application sends before you turn tracing on. + +## πŸ“Š Shared Responsibility + +| Responsibility | Platform Team | Application Team | +|---|:---:|:---:| +| Run the Langfuse instance and its four backends | βœ… | ❌ | +| Keep one team's data separate from another's | βœ… | ❌ | +| Provide the URL, the login and the first API keypair | βœ… | ❌ | +| Upgrade Langfuse and run its migrations | βœ… | ❌ | +| Back up the trace data | βœ… | ❌ | +| Decide what the application sends into a trace | ❌ | βœ… | +| Keep the secret key safe and rotate it when it leaks | ❌ | βœ… | +| Name traces, sessions and users so they can be found | ❌ | βœ… | +| Delete traces that must not be retained | ❌ | βœ… | diff --git a/modules/ai/langfuse/buildingblock/README.md b/modules/ai/langfuse/buildingblock/README.md new file mode 100644 index 00000000..58fb556c --- /dev/null +++ b/modules/ai/langfuse/buildingblock/README.md @@ -0,0 +1,308 @@ +--- +name: Langfuse (per tenant) +supportedPlatforms: + - kubernetes +description: Installs one Langfuse v4 instance per tenant into its own Kubernetes namespace, against a shared Postgres, ClickHouse, Valkey and object storage, bootstrapped with an organisation, a project and an API keypair. +# Every backend connection, every credential and the identity provider arrive as inputs, so there +# is nothing to set up on the cloud side before this module runs. +requiresBackplane: false +--- + +# Langfuse Building Block + +The platform team installs one Langfuse instance per tenant with this module. Each instance has its own namespace, its own hostname, its own secrets and its own slice of four shared backends, and it comes up with an organisation, a project and an API keypair already in place, so a tracing client can be pointed at it in the same Terraform run. + +This documentation is intended as a reference for cloud foundation or platform engineers using this module. + +## Sourced, not ordered + +There is no `meshstack_integration.tf` and no `backplane/`. A tenant-facing building block sources `buildingblock/`, derives every per-tenant name, and passes them in. Application teams order that composition, not this module. + +## Deployment cardinality + +**This module is instantiated once per tenant.** Every value that identifies a tenant is an explicit input rather than something the module derives: the namespace, the release name, the hostname, the Postgres database, the ClickHouse database, the Valkey key prefix and database index, the bucket, the three secrets and the `LANGFUSE_INIT_*` values. The caller derives them from the tenant, this module consumes them. + +`modules/ai/clickhouse` is the opposite: it is deployed once per Kubernetes cluster and shared by every instance this module creates. + +## The chart, and why the image tag is overridden + +| | | +|---|---| +| Chart | `langfuse` | +| Repository | `https://langfuse.github.io/langfuse-k8s` β€” a classic Helm repository, not OCI | +| Pinned chart version | `1.5.41` (`var.chart_version`) | +| Chart `appVersion` | `3.224.1` | +| Image tag | `4.10.0` (`var.image_tag`) | + +The chart's `appVersion` is still a v3 release, so the chart on its own installs Langfuse v3. Overriding `langfuse.image.tag` is what selects v4, exactly as [`examples/v4-installation`](https://github.com/langfuse/langfuse-k8s/tree/main/examples/v4-installation) in `langfuse-k8s` does. One consequence: the `app.kubernetes.io/version` label on every rendered object reads `3.224.1` while the containers run v4. Read the image, not the label. + +The example uses the floating tag `"4"`. **Do not.** It moves whenever Langfuse publishes a v4 release, so two pods of one Deployment can land on two different builds, a `helm upgrade` that changes nothing still rolls the Deployment, and a regression cannot be undone by reverting the Terraform change. `var.image_tag` validates that the value is a full `4.x.y` version. + +## No provider blocks + +This module carries no `provider` block. The caller configures the `kubernetes` and the `helm` provider and passes both down through the `providers` argument of the module call. + +**A module with its own provider configuration cannot be called with `count` or `for_each`.** A composition that instantiates this module once per tenant needs exactly that, so the provider configuration has to stay outside. `modules/kubernetes/ingress` had to delete its `provider.tf` for the same reason. + +## All four backends are external and mandatory + +Every `*.deploy` flag in the chart is a **placement switch, not a feature switch**. Setting `clickhouse.deploy: false` does not make Langfuse run without ClickHouse β€” it says the ClickHouse lives outside the chart. All four backends are required, and the chart's own bundled subcharts are Bitnami images that no longer receive updates. + +| Backend | Separation per tenant | Chart keys this module sets | +|---|---|---| +| Postgres | a database and an owner user | `postgresql.deploy`, `.host`, `.args`, `.auth.username`, `.auth.database`, `.auth.existingSecret`, `.auth.secretKeys`, `.migration.autoMigrate` | +| ClickHouse | a database and a user scoped to `.*` | `clickhouse.deploy`, `.host`, `.httpPort`, `.nativePort`, `.database`, `.auth.username`, `.auth.existingSecret`, `.auth.existingSecretKey`, `.clusterEnabled`, `.migration.autoMigrate` | +| Valkey | a database index **and** a key prefix | `redis.deploy`, `.host`, `.port`, `.auth.username`, `.auth.database`, `.auth.existingSecret`, `.auth.existingSecretPasswordKey` | +| S3 | one bucket | `s3.deploy`, `.bucket`, `.region`, `.endpoint`, `.forcePathStyle`, `.accessKeyId`, `.secretAccessKey`, and `eventUpload`, `batchExport`, `mediaUpload` with the same keys plus `.prefix` | + +The databases, users and buckets are created by the composition, not here. See the [per-tenant separation section](../../clickhouse/buildingblock/README.md#per-tenant-separation-and-who-creates-it) in the ClickHouse module for the reasoning and for the exact ClickHouse statements. + +Every credential reaches the pods through a `secretKeyRef` into one Kubernetes Secret this module creates, so no secret ever appears in the Helm values or in a Terraform plan. + +### Postgres: the port is folded into the host + +**Langfuse never reads `DATABASE_PORT`.** Both container entrypoints build the connection URL as `postgresql://$DATABASE_USERNAME:$DATABASE_PASSWORD@$DATABASE_HOST/$DATABASE_NAME`, with `?$DATABASE_ARGS` appended. The chart does set `DATABASE_PORT` from `postgresql.port`, and nothing reads it, so a non-default port set that way is silently lost and the connection goes to 5432. + +This module therefore passes `postgresql.host = ":"` and leaves `postgresql.port` unset. `var.postgres_host` and `var.postgres_port` stay separate inputs. + +Two more consequences of that string substitution: + +- **The Postgres password has to be safe in a URL.** Nothing percent-encodes it, and Prisma answers `P1013` on a password containing `:`, `@`, `/`, `?`, `#` or `%`. `var.postgres_password` rejects those characters. +- **`postgresql.directUrl` would carry the password in plain text** into the pod spec and into Helm's release Secret. The module leaves that chart key unset and injects `DIRECT_URL` through `langfuse.additionalEnv` with a `secretKeyRef` instead, so `var.postgres_direct_url` is handled like every other credential. + +The ClickHouse password has the same problem in a different place: the migration script puts it into a query string, so `&`, `=`, `#`, `?`, `%`, `+`, `@` and spaces break the migration URL. `var.clickhouse_password` rejects those too. + +### Postgres: the connection pool is pinned, and the budget is shared + +**Prisma sizes its connection pool as `physical cores Γ— 2 + 1` when the connection URL names no `connection_limit`**, and it counts the physical cores of the *node*, not the pod's CPU limit. [prisma-engines#4341](https://github.com/prisma/prisma-engines/issues/4341) records that as an oversight and was closed without a fix. A pod with a `100m` CPU limit therefore takes 33 connections on a 16-core node, 40 pods take 1320, and the number changes silently when a pod is rescheduled onto a node with a different core count. + +STACKIT PostgreSQL Flex fixes `max_connections` by the flavour, reserves 15 of them for its own processes, and exposes no parameter group in the API, the SDK or the Terraform provider. The limit follows RAM rather than vCPUs, so 4 vCPU / 32 GiB and 16 vCPU / 32 GiB both give 785. There is no server-side lever, so every pod pins its own pool. + +| Variable | Default | Applies to | +|---|---|---| +| `postgres_connection_limit` | `5` | `DATABASE_URL`, so every web and worker pod. Appended to `postgres_args`. | +| `postgres_direct_url_connection_limit` | `2` | `DIRECT_URL`, the migration connection. Appended to `postgres_direct_url`. | + +This module is instantiated once per tenant against a shared instance, so the budget is a sum over tenants: + +``` +tenants Γ— pods per tenant Γ— connection_limit + 15 reserved ≀ max_connections +``` + +Four pods per tenant at `connection_limit = 5` come to 20 connections per tenant, so a 4 vCPU / 32 GiB instance carries roughly 34 tenants. Unpinned, the same instance runs out at five. Leave around 10% free for the migrations, for a `psql` session and for a rolling deploy in which the old and the new pods overlap. `modules/stackit/postgresflex` carries the [per-flavour table and the full budget](../../../stackit/postgresflex/buildingblock/README.md#how-many-connections-one-instance-carries). + +**The migration connection gets a limit of its own, and a much smaller one.** `DIRECT_URL` is a full URL the caller hands in, so it never passes through `postgres_args` and would otherwise carry Prisma's node-sized default. The web entrypoint runs `prisma db execute` and then `prisma migrate deploy` one after the other, and each opens a single connection, so two is a ceiling rather than a target. Leave `postgres_direct_url` unset and the entrypoint reuses `DATABASE_URL`, which already carries the pinned pool. + +Three details of how the parameters are written: + +- The module appends `connection_limit` to `postgres_args`, and `var.postgres_args` rejects a `connection_limit` the caller put there, because two occurrences in one query string leave the effective pool size to the parser. +- On `postgres_direct_url` the module appends `&connection_limit=…` when the URL already carries a query string and `?connection_limit=…` when it does not. `var.postgres_direct_url` rejects a URL that already names the parameter. +- **A query parameter does not weaken the password rule.** The password sits in the userinfo part of the URL, before the `?`, so `&` and `=` in the query string cannot reach it. The characters `var.postgres_password` rejects β€” `:`, `@`, `/`, `?`, `#` and `%` β€” are rejected for the same reason as before: nothing percent-encodes the value, and Prisma answers `P1013`. + +`pool_timeout` is the companion parameter: a request that finds no free connection waits that long and then fails with `P2024`. Prisma defaults to 10 seconds. Put `pool_timeout=…` into `var.postgres_args` when a pinned pool of 5 turns out to be tight under a burst, before raising the limit itself. + +### Valkey: set both the index and the key prefix + +`var.valkey_database` and `var.valkey_key_prefix` are both required, and they do different jobs. + +- The **database index** becomes the path of `REDIS_CONNECTION_STRING`. A connection that selected index *n* cannot reach a key in another index at all, so it is a hard namespace that no application bug can cross. +- The **key prefix** survives a later move to Redis Cluster, where indices vanish because a cluster only has database 0. + +**Sharing one Valkey database without a key prefix is a data-crossing bug, not a performance one.** BullMQ queue names are hardcoded in the application, so two tenants on one index with no prefix share `ingestion-queue` β€” and tenant A's worker consumes tenant B's ingestion jobs, writing B's traces into A's ClickHouse database. Do not simplify this away. + +**`REDIS_KEY_PREFIX` is not exposed by the chart.** It appears nowhere in `_helpers.tpl`, so the module injects it through `langfuse.additionalEnv`. It also has an application floor of **v3.157.0**: the variable existed before, but BullMQ ignored it until [langfuse/langfuse#11898](https://github.com/langfuse/langfuse/pull/11898) merged on 2026-03-06, and v3.157.0 is the first release that contains the fix. Any v4 release is far above that floor. + +### Three secrets that must differ per tenant + +| Variable | Environment variable | What it protects | +|---|---|---| +| `salt` | `SALT` | Hashes this tenant's API keys. Langfuse also mixes it into the fast hash of every key, so changing it invalidates every key the tenant holds. | +| `encryption_key` | `ENCRYPTION_KEY` | Encrypts secrets at rest in this tenant's Postgres database. 256 bits, 64 hex characters. | +| `nextauth_secret` | `NEXTAUTH_SECRET` | Signs the NextAuth JWT. Two tenants sharing it accept each other's session tokens. | + +All three are validated for length and shape and reach the pods through a `secretKeyRef`. + +`langfuse.nextauth.url` defaults to `http://localhost:3000` in the chart, which breaks every OAuth callback and every link in an invitation mail. The module sets it from `var.hostname`, or from `var.public_url` when the instance is reached on another name. + +## Authentication + +### Sign-up stays enabled, and that is deliberate + +`langfuse.features.signUpDisabled` is hardcoded to `false` in this module. It looks like the hardening switch and it is not. + +`AUTH_DISABLE_SIGNUP` is checked inside the NextAuth adapter's `createUser`, and an SSO login by a user who is not in the database yet goes through exactly that path. On a freshly provisioned per-tenant instance that is every user, so turning sign-up off blocks the first login of everybody. + +The control that belongs to the operator is `langfuse.auth.disableUsernamePassword`, which this module sets from `var.disable_username_password` and turns on automatically whenever `var.oidc` is set. Who may log in is decided at the identity provider. + +### SSO is free, and it removes member synchronisation + +Self-hosted SSO carries no entitlement in Langfuse β€” it is absent from the exhaustive entitlement list, and the `AUTH_*` variables are read with no entitlement check anywhere in provider construction. It works without an Enterprise licence. + +`var.oidc` configures Langfuse's generic `custom` OIDC provider, which the chart turns into `AUTH_CUSTOM_ISSUER`, `AUTH_CUSTOM_CLIENT_ID`, `AUTH_CUSTOM_CLIENT_SECRET`, `AUTH_CUSTOM_NAME`, `AUTH_CUSTOM_SCOPE` and `AUTH_CUSTOM_ALLOW_ACCOUNT_LINKING`. Langfuse discovers the endpoints from `/.well-known/openid-configuration`. Register the callback URL the `oidc_callback_url` output prints β€” `/api/auth/callback/custom` β€” at the provider. Keycloak, Entra ID, Okta and Auth0 all fit this shape unchanged. + +The identity provider is an input, never an assumption. STACKIT publishes OIDC discovery at `https://accounts.stackit.cloud`, but client registration is undocumented and unsupported for customer applications and its discovery document carries no `registration_endpoint`, so STACKIT is not the identity provider here. + +**There is no member synchronisation to build.** Langfuse upserts an organisation membership for every user who logs in, with the role in `LANGFUSE_DEFAULT_ORG_ROLE`, and there is no entitlement check on that path. The `rbac-project-roles` check next to it guards only *project*-level memberships, and without that entitlement users inherit their organisation role for every project. With one organisation and one project per tenant, the organisation role **is** the access grant. It fires on first login through `createUser` and on every later login through `linkAccount`, the upsert never overwrites a role a user already has, and it is idempotent. The Enterprise-gated memberships API and SCIM are irrelevant here. + +This module sets `LANGFUSE_DEFAULT_ORG_ID` to `var.init_org_id`, so the organisation the bootstrap creates is the one users join. + +### Access control: what the operator owns + +With sign-up enabled and `LANGFUSE_DEFAULT_ORG_ID` set, **everyone the identity provider will authenticate becomes a member of this tenant's organisation.** Nothing sits between "the provider said yes" and "you are in this tenant's Langfuse". + +The mitigation is **one OIDC client per tenant instance, with only that tenant's members assigned to it at the provider.** That is the operator's responsibility and this module cannot enforce it. `var.default_org_role = "NONE"` hands out a membership that grants nothing, which turns auto-join off while leaving users who were added by hand able to work. + +No `oauth2-proxy`. Langfuse speaks OIDC natively, so a proxy in front is redundant; the HAProxy ingress controller in `modules/kubernetes/ingress` has no forward-auth annotation at all, only `basic-auth` and mTLS; and a session cookie shared across `*.` would be presented to every tenant's instance, which defeats the isolation this module builds. + +## Bootstrap without an Enterprise licence + +Langfuse's project-management API is gated behind the `admin-api` entitlement, which needs self-hosted Enterprise. **`LANGFUSE_INIT_*` is not gated.** `web/src/initialize.ts` upserts an organisation, a project, an API keypair and, optionally, a user with `OWNER` membership, with no entitlement check on any of them, and it accepts a **predefined keypair** β€” so Terraform generates the keys and hands them to the caller instead of a human reading them out of a UI. + +`LANGFUSE_INIT_ORG_ID` is the trigger. With it unset, Langfuse logs a warning naming every other init variable and ignores all of them. The whole thing is `upsert`-based and therefore idempotent: a restart changes nothing. + +The module delivers the values through `langfuse.additionalEnv`, with the project secret key and the user password as `secretKeyRef` entries. + +**Use the init user for seeding only, and usually not at all.** `LANGFUSE_INIT_USER_EMAIL` and `LANGFUSE_INIT_USER_PASSWORD` create a *password* user, which cannot log in once `disable_username_password` is on. With an identity provider configured, leave both unset and let the human owner arrive through SSO and `LANGFUSE_DEFAULT_ORG_ROLE`. The variables are optional, and `var.oidc` validates that at least one login path exists. + +### Two v4 additions + +`initialize.ts` in v4 also reads `LANGFUSE_INIT_ORG_CLOUD_PLAN` and `LANGFUSE_INIT_PROJECT_RETENTION`. The retention one is entitlement-gated behind `data-retention` and is silently dropped without it. Neither affects the four objects this module relies on, and the module sets neither. + +## Migrations run on every pod start + +Both entrypoints run `prisma migrate deploy` and then the ClickHouse `migrate up` before the process starts, on **every** pod start of **every** replica. The ClickHouse migrations are `ON CLUSTER` DDL, which goes through one distributed DDL queue for the whole cluster. + +With a handful of tenants this is fine. Above that, several tenants restarting or upgrading at once contend for that single queue, and pods time out waiting for a distributed DDL that another tenant's pod is holding. Turn `var.clickhouse_auto_migrate` and `var.postgres_auto_migrate` off then, and run the migrations once out of band before rolling the instances. + +## Sizing + +The chart sets `resources: {}` for both the web and the worker deployment, so without these values the pods run unbounded. + +| | Request | Limit | Production target | +|---|---|---|---| +| `langfuse.web` | `100m` / `512Mi` | `500m` / `1Gi` | `500m` / `2Gi` | +| `langfuse.worker` | `100m` / `384Mi` | `500m` / `768Mi` | `500m` / `2Gi` | + +The module derives `NODE_OPTIONS=--max-old-space-size` from each memory limit at 75% β€” `768` for the web pod and `576` for the worker with the defaults above. Node sizes its old space from the host's memory rather than from the cgroup limit, so without that flag the heap grows past the container limit and the kernel kills the pod instead of the garbage collector running. Change a memory limit and the flag follows. + +## Wiring a LiteLLM gateway to this instance + +The `base_url`, `project_public_key` and `project_secret_key` outputs are everything a LiteLLM gateway needs. Set them on the gateway as: + +``` +LANGFUSE_OTEL_HOST = +LANGFUSE_PUBLIC_KEY = +LANGFUSE_SECRET_KEY = +``` + +**Set `LANGFUSE_OTEL_HOST`, not only `LANGFUSE_HOST`.** LiteLLM's `langfuse_otel` callback preset resolves the host as `LANGFUSE_OTEL_HOST` first and `LANGFUSE_HOST` second, then appends `/api/public/otel`. Two things follow: + +- With **neither** set, LiteLLM silently exports to `https://us.cloud.langfuse.com/api/public/otel` β€” the public cloud endpoint. Traces leave the cluster and no error is raised. +- The `LANGFUSE_HOST` fallback is recent. LiteLLM releases before roughly v1.85.0 read `LANGFUSE_OTEL_HOST` only, so `LANGFUSE_HOST` on its own also lands on the cloud endpoint there. `modules/ai/litellm` pins chart 1.96.2, which does have the fallback, but `LANGFUSE_OTEL_HOST` is the name that works on every version. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [helm](#requirement\_helm) | >= 3.0.0 | +| [kubernetes](#requirement\_kubernetes) | >= 2.38 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [helm_release.langfuse](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [kubernetes_namespace_v1.this](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_secret_v1.this](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [chart\_version](#input\_chart\_version) | Version of the langfuse chart from https://langfuse.github.io/langfuse-k8s. This is a classic Helm repository, not an OCI registry. | `string` | `"1.5.41"` | no | +| [clickhouse\_auto\_migrate](#input\_clickhouse\_auto\_migrate) | Run the ClickHouse `migrate up` on every web pod start. Turn it off above a handful of tenants and run the migrations once out of band: the statements are ON CLUSTER DDL, and several tenants upgrading at once contend for one distributed DDL queue. | `bool` | `true` | no | +| [clickhouse\_cluster\_enabled](#input\_clickhouse\_cluster\_enabled) | Run the ClickHouse DDL with ON CLUSTER. Keep it on for the operator-managed cluster, which is a real cluster named 'default' even with a single replica. Turn it off only for a standalone ClickHouse with no Keeper. | `bool` | `true` | no | +| [clickhouse\_database](#input\_clickhouse\_database) | Name of this tenant's ClickHouse database. It has to exist before the first apply: golang-migrate creates the tables inside it and never creates a database. | `string` | n/a | yes | +| [clickhouse\_host](#input\_clickhouse\_host) | Fully qualified hostname of the shared ClickHouse cluster, without a scheme. Take it from the `host` output of modules/ai/clickhouse. | `string` | n/a | yes | +| [clickhouse\_http\_port](#input\_clickhouse\_http\_port) | HTTP port of ClickHouse. Langfuse reads and writes trace data over it. | `number` | `8123` | no | +| [clickhouse\_native\_port](#input\_clickhouse\_native\_port) | Native protocol port of ClickHouse. The golang-migrate schema migrations use it. | `number` | `9000` | no | +| [clickhouse\_password](#input\_clickhouse\_password) | Password of the ClickHouse user. Avoid '&', '=', '#', '?', '%', '+', '@' and spaces: the migration script puts the value into a query string without encoding it. | `string` | n/a | yes | +| [clickhouse\_username](#input\_clickhouse\_username) | ClickHouse user Langfuse connects as. It needs SELECT, INSERT, CREATE, DROP TABLE, ALTER UPDATE, ALTER DELETE and ALTER DROP INDEX on its own database, and nothing beyond it. | `string` | n/a | yes | +| [default\_org\_role](#input\_default\_org\_role) | Role every user who logs in receives in this tenant's organisation. Langfuse upserts the
membership on first login and on every later login, and the upsert never overwrites a role a
user already has.

This removes member synchronisation from the picture: no group mapping and no SCIM. Set `NONE`
to hand out a membership that grants nothing, which is the way to turn auto-join off while
keeping the instance usable for users who were added by hand. | `string` | `"MEMBER"` | no | +| [disable\_username\_password](#input\_disable\_username\_password) | Turn off username and password login, so only the OIDC provider remains. Null turns it on whenever var.oidc is set. Never set it to true without an identity provider: nobody could log in. | `bool` | `null` | no | +| [encryption\_key](#input\_encryption\_key) | Key Langfuse encrypts secrets at rest with, in this tenant's Postgres database. Must be 256 bits, which is 64 hex characters. Generate one with `openssl rand -hex 32`. It must be unique per tenant, because it is the only thing separating one tenant's stored credentials from another's. | `string` | n/a | yes | +| [helm\_timeout](#input\_helm\_timeout) | Seconds to wait for the Helm release to become ready. Every web pod runs the Postgres and the ClickHouse migrations before it answers its readiness probe, so the first install takes part of this budget. | `number` | `900` | no | +| [hostname](#input\_hostname) | Canonical hostname of this tenant's Langfuse instance, without a scheme. The module derives NEXTAUTH\_URL and the Ingress rule from it. | `string` | n/a | yes | +| [image\_tag](#input\_image\_tag) | Tag of the langfuse/langfuse and langfuse/langfuse-worker images. It has to name a concrete v4
release, because the chart's own appVersion still points at v3.

Do not use the floating `"4"` tag. It moves whenever Langfuse publishes a v4 release, so two
pods of the same Deployment can end up on two different builds, a `helm upgrade` that changes
nothing still rolls the Deployment, and a regression cannot be rolled back by reverting the
Terraform change. Pin a full version and raise it deliberately. | `string` | `"4.10.0"` | no | +| [ingress\_annotations](#input\_ingress\_annotations) | Annotations on the Ingress. Set `cert-manager.io/cluster-issuer` here when the instance needs a certificate of its own instead of the controller's wildcard certificate. | `map(string)` | `{}` | no | +| [ingress\_class\_name](#input\_ingress\_class\_name) | Name of the IngressClass that serves this instance. It has to match the controller modules/kubernetes/ingress installs. | `string` | `"haproxy"` | no | +| [ingress\_enabled](#input\_ingress\_enabled) | Create an Ingress for var.hostname. Turn it off when the instance is reached in-cluster only, and set var.public\_url accordingly. | `bool` | `true` | no | +| [ingress\_tls\_secret\_name](#input\_ingress\_tls\_secret\_name) | Name of the secret holding the TLS certificate for var.hostname. Null leaves the Ingress without a tls block, which is correct when the ingress controller serves a wildcard certificate as its default. | `string` | `null` | no | +| [init\_org\_id](#input\_init\_org\_id) | Identifier of the organisation Langfuse creates on startup. It is the trigger of the whole bootstrap: with it unset, every other init value is silently ignored. | `string` | n/a | yes | +| [init\_org\_name](#input\_init\_org\_name) | Display name of the organisation. | `string` | n/a | yes | +| [init\_project\_id](#input\_init\_project\_id) | Identifier of the project Langfuse creates inside the organisation. Traces belong to a project, and the API keypair below is scoped to it. | `string` | n/a | yes | +| [init\_project\_name](#input\_init\_project\_name) | Display name of the project. | `string` | n/a | yes | +| [init\_project\_public\_key](#input\_init\_project\_public\_key) | Public key of the API keypair Langfuse creates for the project. Langfuse's own generator produces 'pk-lf-', and clients expect that shape. | `string` | n/a | yes | +| [init\_project\_secret\_key](#input\_init\_project\_secret\_key) | Secret key of the API keypair Langfuse creates for the project. Langfuse's own generator produces 'sk-lf-', and clients expect that shape. | `string` | n/a | yes | +| [init\_user\_email](#input\_init\_user\_email) | Email address of a first user with a password, given OWNER membership in the organisation. Leave it null when var.oidc is set: a password user cannot log in once username and password login is off. | `string` | `null` | no | +| [init\_user\_name](#input\_init\_user\_name) | Display name of the first user. Null lets Langfuse name it 'Provisioned User'. Only used when init\_user\_email and init\_user\_password are both set. | `string` | `null` | no | +| [init\_user\_password](#input\_init\_user\_password) | Password of the first user. Langfuse only sets it while creating the user, so changing this value later does not reset the password. Both init\_user\_email and init\_user\_password have to be set, or neither. | `string` | `null` | no | +| [namespace](#input\_namespace) | Namespace this tenant's Langfuse instance runs in. The module creates it. Give every tenant a namespace of its own. | `string` | `"langfuse"` | no | +| [nextauth\_secret](#input\_nextauth\_secret) | Secret NextAuth signs its JWTs and hashes email verification tokens with. It must be unique per tenant, otherwise a session token minted for one tenant is accepted by another. Generate one with `openssl rand -base64 32`. | `string` | n/a | yes | +| [oidc](#input\_oidc) | OIDC identity provider this tenant's users log in through. Null leaves the instance on username
and password login, which then needs `init_user_email` and `init_user_password`.

Self-hosted SSO is free in Langfuse. It carries no entitlement, so it works without an
Enterprise licence.

- `issuer_url`: discovery base URL of the provider, for example
`https://idp.example.com/realms/ai`. Langfuse reads `/.well-known/openid-configuration`.
- `client_id` and `client_secret`: credentials of the OIDC client.
- `display_name`: label on the login button. The provider is not registered without it.
- `scopes`: space-separated scope list. The default covers what Langfuse needs.
- `allow_account_linking`: link an OIDC login to an existing user with the same email address.
Turn it on when a user already exists from a password login or from another provider.

Register the callback URL `/api/auth/callback/custom` at the provider.

Give every tenant an OIDC client of its own and assign only that tenant's members to it. See the
access control note in the module README: with auto-join on, everyone the provider authenticates
becomes a member of this tenant's organisation. | 
object({
    issuer_url            = string
    client_id             = string
    client_secret         = string
    display_name          = optional(string, "Single Sign-On")
    scopes                = optional(string, "openid email profile")
    allow_account_linking = optional(bool, true)
  })
| `null` | no | +| [postgres\_args](#input\_postgres\_args) | Query string appended to the Postgres connection URL, without the leading '?'. Managed Postgres offerings terminate TLS, so requiring it is the safe default. A server without TLS needs 'sslmode=prefer' or 'sslmode=disable'. The module appends `connection_limit` from postgres\_connection\_limit, and `pool_timeout` belongs here when the default of 10 seconds is too short for the pinned pool. | `string` | `"sslmode=require"` | no | +| [postgres\_auto\_migrate](#input\_postgres\_auto\_migrate) | Run `prisma migrate deploy` on every web pod start. Turn it off above a handful of tenants and run the migrations once out of band, because every pod of every tenant runs them on every restart. | `bool` | `true` | no | +| [postgres\_connection\_limit](#input\_postgres\_connection\_limit) | Maximum number of Postgres connections one Langfuse pod opens. The module appends it to the
connection URL as `connection_limit`, next to `postgres_args`.

Without the parameter Prisma sizes the pool as `physical cores Γ— 2 + 1` read from the node, not
from the pod's CPU limit, so a pod takes 33 connections on a 16-core node and a different number
after it is rescheduled onto a node with more cores.

This module runs once per tenant against a shared instance, so the platform's budget is
`tenants Γ— pods per tenant Γ— connection_limit + 15 ≀ max_connections`. STACKIT PostgreSQL Flex
fixes `max_connections` per flavour and reserves 15 of them, so the default of 5 is what keeps a
4 vCPU / 32 GiB instance at roughly 34 tenants with four pods each. | `number` | `5` | no | +| [postgres\_database](#input\_postgres\_database) | Name of this tenant's Postgres database. It has to exist before the first apply, together with its owner user β€” Prisma creates the tables inside it, not the database itself. | `string` | n/a | yes | +| [postgres\_direct\_url](#input\_postgres\_direct\_url) | Full connection URL the schema migrations run against, used when the normal connection goes through a pooler or when migrations need a user with longer timeouts. The module appends `connection_limit` from postgres\_direct\_url\_connection\_limit to it. Null makes the migrations reuse the normal connection, which already carries the pinned pool. | `string` | `null` | no | +| [postgres\_direct\_url\_connection\_limit](#input\_postgres\_direct\_url\_connection\_limit) | Maximum number of Postgres connections the migration connection opens, appended to postgres\_direct\_url as `connection_limit`. Two covers the `prisma db execute` and the `prisma migrate deploy` the web entrypoint runs one after the other, each of which opens a single connection. Only used when postgres\_direct\_url is set. | `number` | `2` | no | +| [postgres\_host](#input\_postgres\_host) | Hostname of the shared Postgres server. Langfuse keeps its relational data β€” organisations, projects, users, prompts and encrypted secrets β€” here. | `string` | n/a | yes | +| [postgres\_password](#input\_postgres\_password) | Password of the Postgres user. Use only characters that are safe in a URL: Langfuse builds its connection URL by string substitution and does not percent-encode the value. | `string` | n/a | yes | +| [postgres\_port](#input\_postgres\_port) | Port of the Postgres server. The module appends it to the host, because Langfuse builds its connection URL from the host only and ignores the port variable the chart sets. | `number` | `5432` | no | +| [postgres\_username](#input\_postgres\_username) | User Langfuse connects as. It has to own this tenant's database, because `prisma migrate deploy` creates and alters tables under it on every web pod start. | `string` | n/a | yes | +| [public\_url](#input\_public\_url) | Full canonical URL of this tenant's Langfuse instance, used as NEXTAUTH\_URL. Null derives 'https://'. | `string` | `null` | no | +| [release\_name](#input\_release\_name) | Helm release name of this tenant's Langfuse instance. | `string` | `"langfuse"` | no | +| [s3\_access\_key\_id](#input\_s3\_access\_key\_id) | Access key id of the credential scoped to this tenant's bucket. | `string` | n/a | yes | +| [s3\_batch\_export\_prefix](#input\_s3\_batch\_export\_prefix) | Prefix inside the bucket for batch exports. Keep the trailing slash. | `string` | `"exports/"` | no | +| [s3\_bucket](#input\_s3\_bucket) | Bucket this tenant's event uploads, batch exports and media uploads go to. Give every tenant a bucket of its own. | `string` | n/a | yes | +| [s3\_endpoint](#input\_s3\_endpoint) | Endpoint URL of the object storage, including the scheme. | `string` | n/a | yes | +| [s3\_event\_upload\_prefix](#input\_s3\_event\_upload\_prefix) | Prefix inside the bucket for raw ingestion events. Keep the trailing slash. | `string` | `"events/"` | no | +| [s3\_force\_path\_style](#input\_s3\_force\_path\_style) | Address the bucket as a path on the endpoint instead of as a subdomain. Required for MinIO and for most S3-compatible object storage. | `bool` | `true` | no | +| [s3\_media\_upload\_prefix](#input\_s3\_media\_upload\_prefix) | Prefix inside the bucket for media uploads. Keep the trailing slash. | `string` | `"media/"` | no | +| [s3\_region](#input\_s3\_region) | Region of the bucket. 'auto' works for S3-compatible object storage that has no region concept. | `string` | `"auto"` | no | +| [s3\_secret\_access\_key](#input\_s3\_secret\_access\_key) | Secret access key of the credential scoped to this tenant's bucket. | `string` | n/a | yes | +| [salt](#input\_salt) | Salt Langfuse hashes API keys with. It must be unique per tenant, because two tenants sharing a salt share the hash space of their API keys. Changing it later invalidates every API key of the tenant. | `string` | n/a | yes | +| [telemetry\_enabled](#input\_telemetry\_enabled) | Report basic usage statistics to Langfuse. The chart turns this on by default; a self-hosted tenant instance usually should not phone home. | `bool` | `false` | no | +| [valkey\_database](#input\_valkey\_database) | Valkey database index of this tenant. It is a hard namespace that no application bug can cross, so give every tenant an index of its own. A stock Valkey serves indices 0 to 15. | `number` | n/a | yes | +| [valkey\_host](#input\_valkey\_host) | Hostname of the shared Valkey or Redis instance. Langfuse uses it as the BullMQ queue backend, the cache and the rate limit store. | `string` | n/a | yes | +| [valkey\_key\_prefix](#input\_valkey\_key\_prefix) | Prefix Langfuse puts in front of every Valkey key and every BullMQ queue name for this tenant,
for example `tenant-a:`. Give every tenant a prefix of its own.

Set it together with `valkey_database`, not instead of it. The index is a hard namespace that no
application bug can cross; the prefix survives a later move to Redis Cluster, where indices
vanish because a cluster only has database 0.

Langfuse needs at least app version 3.157.0 for this to work. The variable existed before, but
BullMQ ignored it until langfuse/langfuse#11898 merged on 2026-03-06. | `string` | n/a | yes | +| [valkey\_password](#input\_valkey\_password) | Password of the Valkey instance. Use only characters that are safe in a URL, because the chart substitutes the value into the connection URL without encoding it. | `string` | n/a | yes | +| [valkey\_port](#input\_valkey\_port) | Port of the Valkey instance. | `number` | `6379` | no | +| [valkey\_username](#input\_valkey\_username) | Username for Valkey authentication. Null omits the username from the connection string, which is what a Valkey without ACLs expects. | `string` | `"default"` | no | +| [web\_replicas](#input\_web\_replicas) | Number of Langfuse web pods. The default of 1 is sized for a demonstration and gives no redundancy: every restart interrupts the UI and the ingestion API. | `number` | `1` | no | +| [web\_resources](#input\_web\_resources) | Resource requests and limits of the Langfuse web pods. The default is sized for a demonstration
and a production consumer has to raise it.

The chart sets `resources: {}` for the web deployment, so the pods run unbounded without these
values. The web pod serves the UI and the ingestion API and runs both migrations on start, which
is the peak of its memory use. Production wants `500m` CPU and `2Gi` of memory.

The module derives `NODE_OPTIONS=--max-old-space-size` from the memory limit at roughly 75%, so
Node runs a garbage collection before the cgroup limit is reached instead of being OOMKilled. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "500m",
    "memory": "1Gi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "512Mi"
  }
}
| no | +| [worker\_replicas](#input\_worker\_replicas) | Number of Langfuse worker pods. The default of 1 is sized for a demonstration. Raise it when the ingestion queue backs up. | `number` | `1` | no | +| [worker\_resources](#input\_worker\_resources) | Resource requests and limits of the Langfuse worker pods. The default is sized for a
demonstration and a production consumer has to raise it.

The chart sets `resources: {}` for the worker deployment as well. The worker drains the BullMQ
queues and batches writes into ClickHouse, so its memory grows with the batch size rather than
with the number of users. Production wants `500m` CPU and `2Gi` of memory.

The module derives `NODE_OPTIONS=--max-old-space-size` from the memory limit at roughly 75%. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "500m",
    "memory": "768Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "384Mi"
  }
}
| no | + +## Outputs + +| Name | Description | +|------|-------------| +| [base\_url](#output\_base\_url) | In-cluster base URL of this Langfuse instance, with the scheme and the port. Set it as LANGFUSE\_OTEL\_HOST on a LiteLLM gateway that traces into this instance. | +| [host](#output\_host) | Fully qualified in-cluster hostname of the Langfuse web Service. | +| [namespace](#output\_namespace) | Namespace this tenant's Langfuse instance runs in. | +| [oidc\_callback\_url](#output\_oidc\_callback\_url) | Callback URL to register at the OIDC provider for this tenant's client. Null when var.oidc is not set. | +| [port](#output\_port) | Port the Langfuse web Service listens on. | +| [project\_id](#output\_project\_id) | Identifier of the project Langfuse created. Traces belong to it and the API keypair is scoped to it. | +| [project\_public\_key](#output\_project\_public\_key) | Public key of the project API keypair. A tracing client sends it together with the secret key as basic auth, for example as LANGFUSE\_PUBLIC\_KEY on a LiteLLM gateway. | +| [project\_secret\_key](#output\_project\_secret\_key) | Secret key of the project API keypair. A tracing client sends it together with the public key as basic auth, for example as LANGFUSE\_SECRET\_KEY on a LiteLLM gateway. | +| [public\_url](#output\_public\_url) | Canonical external URL of this Langfuse instance. It is what NEXTAUTH\_URL is set to and what a browser opens. | +| [release\_name](#output\_release\_name) | Helm release name of this tenant's Langfuse instance. | +| [web\_service\_name](#output\_web\_service\_name) | Name of the Service in front of the Langfuse web pods. | + diff --git a/modules/ai/langfuse/buildingblock/langfuse.tftest.hcl b/modules/ai/langfuse/buildingblock/langfuse.tftest.hcl new file mode 100644 index 00000000..0fd73a95 --- /dev/null +++ b/modules/ai/langfuse/buildingblock/langfuse.tftest.hcl @@ -0,0 +1,168 @@ +variables { + namespace = "langfuse-acme" + release_name = "langfuse-acme" + hostname = "langfuse-acme.example.com" + + salt = "0123456789abcdef0123456789abcdef" + encryption_key = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + nextauth_secret = "0123456789abcdef0123456789abcdef" + + postgres_host = "shared.postgresflex.eu01.onstackit.cloud" + postgres_database = "langfuse_acme" + postgres_username = "langfuse_acme" + postgres_password = "langfuse-password" + + clickhouse_host = "clickhouse-headless.clickhouse.svc.cluster.local" + clickhouse_database = "langfuse_acme" + clickhouse_username = "langfuse_acme" + clickhouse_password = "clickhouse-password" + + valkey_host = "valkey.valkey.svc.cluster.local" + valkey_password = "valkey-password" + valkey_database = 3 + valkey_key_prefix = "acme:" + + s3_bucket = "langfuse-acme" + s3_endpoint = "https://object.storage.eu01.onstackit.cloud" + s3_access_key_id = "AKIAMOCKACCESSKEY" + s3_secret_access_key = "mock-secret-access-key" + + init_org_id = "acme" + init_org_name = "ACME" + init_project_id = "acme-default" + init_project_name = "Default" + init_project_public_key = "pk-lf-11111111-2222-3333-4444-555555555555" + init_project_secret_key = "sk-lf-66666666-7777-8888-9999-000000000000" + init_user_email = "owner@example.com" + init_user_password = "owner-password" +} + +mock_provider "kubernetes" {} +mock_provider "helm" {} + +run "pins_the_prisma_pool_in_the_connection_arguments" { + command = plan + + assert { + # The chart turns postgresql.args into DATABASE_ARGS and the entrypoint appends it to the + # connection URL after a '?', so the whole string is asserted rather than only the parameter. + condition = yamldecode(helm_release.langfuse.values[0]).postgresql.args == "sslmode=require&connection_limit=5" + error_message = "the rendered postgresql.args must carry sslmode and the pinned connection_limit" + } + + assert { + condition = yamldecode(helm_release.langfuse.values[0]).postgresql.host == "shared.postgresflex.eu01.onstackit.cloud:5432" + error_message = "the port must stay folded into the host, because Langfuse never reads DATABASE_PORT" + } +} + +run "a_lowered_limit_reaches_the_connection_arguments" { + command = plan + + variables { + postgres_connection_limit = 3 + postgres_args = "sslmode=verify-full&pool_timeout=20" + } + + assert { + condition = yamldecode(helm_release.langfuse.values[0]).postgresql.args == "sslmode=verify-full&pool_timeout=20&connection_limit=3" + error_message = "the module must append its connection_limit behind the arguments the caller set" + } +} + +run "an_empty_argument_string_leaves_no_stray_separator" { + command = plan + + variables { + postgres_args = "" + } + + assert { + condition = yamldecode(helm_release.langfuse.values[0]).postgresql.args == "connection_limit=5" + error_message = "an empty postgres_args must not produce a query string that starts with '&'" + } +} + +run "the_migration_connection_gets_a_limit_of_its_own" { + command = plan + + variables { + postgres_direct_url = "postgresql://langfuse_acme:pa%2Fss@shared.postgresflex.eu01.onstackit.cloud:5432/langfuse_acme?sslmode=require" + } + + assert { + # nonsensitive() because the secret carries the password. The URL is asserted in full: the + # parameter has to land behind the existing query string with '&', and nothing about the + # percent-encoded password may change. + condition = nonsensitive(kubernetes_secret_v1.this.data["postgres-direct-url"]) == "postgresql://langfuse_acme:pa%2Fss@shared.postgresflex.eu01.onstackit.cloud:5432/langfuse_acme?sslmode=require&connection_limit=2" + error_message = "DIRECT_URL must keep its query string and gain the smaller migration limit" + } + + assert { + # The URL stays in the secret and reaches the pods through a secretKeyRef, never through + # postgresql.directUrl, which the chart would render into the pod spec in plain text. + condition = anytrue([ + for env in yamldecode(helm_release.langfuse.values[0]).langfuse.additionalEnv : + env.name == "DIRECT_URL" && try(env.valueFrom.secretKeyRef.key, null) == "postgres-direct-url" + ]) + error_message = "DIRECT_URL must reach the pods through a secretKeyRef" + } + + assert { + condition = !can(yamldecode(helm_release.langfuse.values[0]).postgresql.directUrl) + error_message = "postgresql.directUrl must stay unset, otherwise the password lands in the pod spec" + } +} + +run "a_migration_url_without_a_query_string_gets_a_question_mark" { + command = plan + + variables { + postgres_direct_url = "postgresql://langfuse_acme:secret@shared.postgresflex.eu01.onstackit.cloud:5432/langfuse_acme" + postgres_direct_url_connection_limit = 1 + } + + assert { + condition = nonsensitive(kubernetes_secret_v1.this.data["postgres-direct-url"]) == "postgresql://langfuse_acme:secret@shared.postgresflex.eu01.onstackit.cloud:5432/langfuse_acme?connection_limit=1" + error_message = "a migration URL without a query string must gain one with '?', not with '&'" + } +} + +run "no_migration_url_leaves_the_secret_key_out" { + command = plan + + assert { + condition = !contains(keys(nonsensitive(kubernetes_secret_v1.this.data)), "postgres-direct-url") + error_message = "without postgres_direct_url the secret must carry no migration URL, so the migrations reuse DATABASE_URL and its pinned pool" + } +} + +run "rejects_a_connection_limit_in_the_argument_string" { + command = plan + + variables { + postgres_args = "sslmode=require&connection_limit=20" + } + + expect_failures = [var.postgres_args] +} + +run "rejects_a_connection_limit_on_the_migration_url" { + command = plan + + variables { + postgres_direct_url = "postgresql://langfuse_acme:secret@shared.postgresflex.eu01.onstackit.cloud:5432/langfuse_acme?connection_limit=20" + } + + expect_failures = [var.postgres_direct_url] +} + +run "rejects_a_connection_limit_below_one" { + command = plan + + variables { + postgres_connection_limit = 0 + } + + expect_failures = [var.postgres_connection_limit] +} diff --git a/modules/ai/langfuse/buildingblock/logo.png b/modules/ai/langfuse/buildingblock/logo.png new file mode 100644 index 00000000..14c164ce Binary files /dev/null and b/modules/ai/langfuse/buildingblock/logo.png differ diff --git a/modules/ai/langfuse/buildingblock/main.tf b/modules/ai/langfuse/buildingblock/main.tf new file mode 100644 index 00000000..36d20cc9 --- /dev/null +++ b/modules/ai/langfuse/buildingblock/main.tf @@ -0,0 +1,500 @@ +locals { + # The chart names its resources '-langfuse-…', or '-…' when the release name + # already contains the chart's name, which is 'langfuse'. + fullname = strcontains(var.release_name, "langfuse") ? var.release_name : "${var.release_name}-langfuse" + web_service_name = "${local.fullname}-web" + + # Pinned here rather than left to the chart default, so the outputs cannot drift away from the + # port the Service actually listens on. + web_service_port = 3000 + + web_service_host = "${local.web_service_name}.${var.namespace}.svc.cluster.local" + base_url = "http://${local.web_service_host}:${local.web_service_port}" + + public_url = coalesce(var.public_url, "https://${var.hostname}") + + # Langfuse never reads DATABASE_PORT. Both entrypoints build the connection URL as + # postgresql://:@$DATABASE_HOST/, so a port that is not folded into the + # host is simply lost and the connection goes to 5432. + postgres_host_with_port = "${var.postgres_host}:${var.postgres_port}" + + # Prisma sizes its connection pool as `num_physical_cpus * 2 + 1` when the connection URL carries + # no connection_limit, and it counts the physical cores of the node instead of the pod's CPU + # limit. prisma-engines#4341 records that as an oversight and was closed without a fix, so a pod + # with a 100m limit takes 33 connections on a 16-core node and a different number after it is + # rescheduled. STACKIT PostgreSQL Flex caps max_connections by the flavour and exposes no + # parameter group, so every pod of every tenant has to pin its pool. compact() drops the empty + # string a caller who cleared postgres_args leaves behind, so the query string never starts or + # ends with a stray '&'. + postgres_args = join("&", compact([ + var.postgres_args, + "connection_limit=${var.postgres_connection_limit}", + ])) + + secret_name = "${var.release_name}-langfuse" + + secret_keys = { + salt = "salt" + encryption_key = "encryption-key" + nextauth_secret = "nextauth-secret" + postgres_password = "postgres-password" + clickhouse_password = "clickhouse-password" + valkey_password = "valkey-password" + s3_access_key_id = "s3-access-key-id" + s3_secret_access_key = "s3-secret-access-key" + postgres_direct_url = "postgres-direct-url" + init_project_secret_key = "init-project-secret-key" + init_user_password = "init-user-password" + oidc_client_secret = "oidc-client-secret" + } + + # var.oidc is sensitive as a whole, so every expression derived from it carries the sensitivity + # mark, and a values map that carries the mark hides the whole Helm release from every plan. + # Unmark the plain facts β€” is SSO on, which issuer, which client β€” while the client secret keeps + # its mark and reaches the pods through a secretKeyRef. nonsensitive() rejects an argument that + # carries no mark, so try() falls back to the bare value. + oidc_enabled = try(nonsensitive(var.oidc != null), var.oidc != null) + oidc = local.oidc_enabled ? { + issuer_url = try(nonsensitive(var.oidc.issuer_url), var.oidc.issuer_url) + client_id = try(nonsensitive(var.oidc.client_id), var.oidc.client_id) + display_name = try(nonsensitive(var.oidc.display_name), var.oidc.display_name) + scopes = try(nonsensitive(var.oidc.scopes), var.oidc.scopes) + allow_account_linking = try(nonsensitive(var.oidc.allow_account_linking), var.oidc.allow_account_linking) + } : null + + disable_username_password = coalesce(var.disable_username_password, local.oidc_enabled) + + init_user_enabled = var.init_user_email != null + + # Whether a direct URL was given is a plain fact, while the URL itself is a secret. The fact has + # to stay unmarked, because everything derived from a marked value carries the mark and the + # whole chart values map would then be hidden from every plan. + postgres_direct_url_set = try(nonsensitive(var.postgres_direct_url != null), var.postgres_direct_url != null) + + # DIRECT_URL is a full URL the caller hands in, so it never passes through postgres_args and + # carries whatever pool the caller left on it β€” Prisma's node-sized default, in practice. The + # migrations get a limit of their own, and a much smaller one: the entrypoint runs + # `prisma db execute` and `prisma migrate deploy` one after the other, and each opens a single + # connection. The '?' test looks at the whole URL rather than at its query string alone, which is + # correct here because a password carrying a raw '?' already breaks Prisma with P1013. + postgres_direct_url_given = var.postgres_direct_url == null ? "" : var.postgres_direct_url + postgres_direct_url = local.postgres_direct_url_set ? join("", [ + local.postgres_direct_url_given, + strcontains(local.postgres_direct_url_given, "?") ? "&" : "?", + "connection_limit=${var.postgres_direct_url_connection_limit}", + ]) : null + + # Helm renders these values into the pod spec as YAML, and the API server rejects a resource + # quantity that is null, so drop every field the caller left unset. + resources = { + for name, spec in { + web = var.web_resources + worker = var.worker_resources + } : name => { + requests = { for key, value in spec.requests : key => value if value != null } + limits = { for key, value in spec.limits : key => value if value != null } + } + } + + # Node sizes its old space from the host's memory, not from the cgroup limit, so without this + # flag the heap grows past the container limit and the kernel kills the pod instead of the + # garbage collector running. 75% of the limit leaves room for the rest of the process. + node_heap_mib = { + for name, limit in { + web = var.web_resources.limits.memory + worker = var.worker_resources.limits.memory + } : name => ( + limit == null ? null : + can(regex("^[0-9]+Gi$", limit)) ? floor(tonumber(trimsuffix(limit, "Gi")) * 1024 * 0.75) : + can(regex("^[0-9]+Mi$", limit)) ? floor(tonumber(trimsuffix(limit, "Mi")) * 0.75) : + null + ) + } + + node_options_env = { + for name, heap in local.node_heap_mib : + name => heap == null ? [] : [{ name = "NODE_OPTIONS", value = "--max-old-space-size=${heap}" }] + } + + # REDIS_KEY_PREFIX is not exposed by the chart β€” it appears nowhere in _helpers.tpl β€” so it has + # to travel through additionalEnv. Without it every tenant's worker reads the same BullMQ queue + # names, because those names are hardcoded in the application. + valkey_env = [ + { name = "REDIS_KEY_PREFIX", value = var.valkey_key_prefix } + ] + + # The chart writes postgresql.directUrl into the pod spec as a plain value, and a connection URL + # carries the password. Every other credential in this module travels through a secretKeyRef, so + # this one does too: the chart value stays unset and the environment variable comes from the + # tenant secret instead. + postgres_direct_url_env = local.postgres_direct_url_set ? [ + { + name = "DIRECT_URL" + valueFrom = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.postgres_direct_url + } + } + }, + ] : [] + + # LANGFUSE_INIT_* is not gated behind an entitlement. Langfuse upserts the organisation, the + # project, the API keypair and, optionally, a user with OWNER membership, and it accepts a + # predefined keypair, so Terraform generates the keys instead of a human reading them out of a + # UI. LANGFUSE_INIT_ORG_ID is the trigger: with it unset the rest is silently ignored. + # + # Each conditional below carries exactly one element, because a conditional whose branches are + # tuples of different length cannot be type-checked unless both sides unify to the same list + # type β€” and an entry with `value` and an entry with `valueFrom` are different object types. + init_env = concat( + [ + { name = "LANGFUSE_INIT_ORG_ID", value = var.init_org_id }, + { name = "LANGFUSE_INIT_ORG_NAME", value = var.init_org_name }, + { name = "LANGFUSE_INIT_PROJECT_ID", value = var.init_project_id }, + { name = "LANGFUSE_INIT_PROJECT_NAME", value = var.init_project_name }, + { name = "LANGFUSE_INIT_PROJECT_PUBLIC_KEY", value = var.init_project_public_key }, + { + name = "LANGFUSE_INIT_PROJECT_SECRET_KEY" + valueFrom = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.init_project_secret_key + } + } + }, + ], + local.init_user_enabled ? [ + { name = "LANGFUSE_INIT_USER_EMAIL", value = var.init_user_email }, + ] : [], + local.init_user_enabled ? [ + { + name = "LANGFUSE_INIT_USER_PASSWORD" + valueFrom = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.init_user_password + } + } + }, + ] : [], + local.init_user_enabled && var.init_user_name != null ? [ + { name = "LANGFUSE_INIT_USER_NAME", value = var.init_user_name }, + ] : [] + ) + + # Langfuse upserts an organisation membership for every user who logs in, with this role, and + # the upsert never overwrites a role the user already has. There is no entitlement check on that + # path, so this replaces member synchronisation entirely: no group mapping, no SCIM. + default_org_env = [ + { name = "LANGFUSE_DEFAULT_ORG_ID", value = var.init_org_id }, + { name = "LANGFUSE_DEFAULT_ORG_ROLE", value = var.default_org_role }, + ] + + chart_values = { + langfuse = { + # The chart's appVersion is still a v3 release, so the tag override is what selects v4. + image = { + tag = var.image_tag + } + + replicas = var.web_replicas + + features = { + telemetryEnabled = var.telemetry_enabled + + # This stays false on purpose. AUTH_DISABLE_SIGNUP is checked inside the NextAuth + # adapter's createUser, and an SSO login of a user who is not in the database yet goes + # through exactly that path. Turning sign-up off therefore blocks the first login of every + # SSO user on a freshly provisioned instance, which is all of them. + # + # Who may log in is decided at the identity provider, by assigning only this tenant's + # members to this tenant's OIDC client. The control inside Langfuse is + # auth.disableUsernamePassword below, not this flag. Do not "harden" it back to true. + signUpDisabled = false + + experimentalFeaturesEnabled = false + } + + auth = merge( + { + disableUsernamePassword = local.disable_username_password + }, + # The chart turns every key under a provider into AUTH__, and 'custom' is + # the generic OIDC provider Langfuse builds from AUTH_CUSTOM_ISSUER by discovery. Its + # callback path is /api/auth/callback/custom. + local.oidc_enabled ? { + providers = { + custom = { + issuer = local.oidc.issuer_url + clientId = local.oidc.client_id + clientSecret = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.oidc_client_secret + } + } + name = local.oidc.display_name + scope = local.oidc.scopes + allowAccountLinking = local.oidc.allow_account_linking + } + } + } : {} + ) + + salt = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.salt + } + } + + encryptionKey = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.encryption_key + } + } + + nextauth = { + # The chart default is http://localhost:3000, which every OAuth callback and every link in + # an invitation mail would then point at. + url = local.public_url + secret = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.nextauth_secret + } + } + } + + ingress = { + enabled = var.ingress_enabled + className = var.ingress_class_name + annotations = var.ingress_annotations + hosts = [ + { + host = var.hostname + paths = [{ path = "/", pathType = "Prefix" }] + } + ] + tls = { + # Without a secret of its own the Ingress carries no tls block, and the ingress + # controller serves its default wildcard certificate for this host. + enabled = var.ingress_tls_secret_name != null + secretName = var.ingress_tls_secret_name + } + } + + additionalEnv = concat( + local.valkey_env, + local.postgres_direct_url_env, + local.init_env, + local.default_org_env + ) + + web = { + replicas = var.web_replicas + resources = local.resources.web + pod = { + additionalEnv = local.node_options_env.web + } + } + + worker = { + replicas = var.worker_replicas + resources = local.resources.worker + pod = { + additionalEnv = local.node_options_env.worker + } + } + } + + # Every 'deploy' flag below is a placement switch, not a feature switch. Turning one off does + # not make Langfuse run without that backend β€” it only says the backend lives outside the + # chart. All four are mandatory. + postgresql = { + deploy = false + + # The port is part of the host, see the comment on local.postgres_host_with_port. The chart + # turns args into DATABASE_ARGS and the entrypoint appends it to the URL after a '?', so the + # pinned connection_limit rides along in local.postgres_args. + host = local.postgres_host_with_port + args = local.postgres_args + + auth = { + username = var.postgres_username + database = var.postgres_database + existingSecret = local.secret_name + secretKeys = { + userPasswordKey = local.secret_keys.postgres_password + adminPasswordKey = local.secret_keys.postgres_password + } + } + + # directUrl stays unset here β€” the module injects DIRECT_URL through additionalEnv with a + # secretKeyRef instead, see local.postgres_direct_url_env. + + migration = { + autoMigrate = var.postgres_auto_migrate + } + } + + redis = { + deploy = false + + host = var.valkey_host + port = var.valkey_port + + auth = { + username = var.valkey_username + # The DB index becomes the path of the connection URL. It is a hard namespace: a + # connection that selected index n cannot reach a key in another index at all. + database = var.valkey_database + existingSecret = local.secret_name + existingSecretPasswordKey = local.secret_keys.valkey_password + } + } + + clickhouse = { + deploy = false + + host = var.clickhouse_host + httpPort = var.clickhouse_http_port + nativePort = var.clickhouse_native_port + database = var.clickhouse_database + + auth = { + username = var.clickhouse_username + existingSecret = local.secret_name + existingSecretKey = local.secret_keys.clickhouse_password + } + + # The operator-managed cluster is a real cluster named 'default' even with one replica, so + # the ON CLUSTER statements of the migrations work. + clusterEnabled = var.clickhouse_cluster_enabled + + migration = { + autoMigrate = var.clickhouse_auto_migrate + } + } + + s3 = { + deploy = false + + storageProvider = "s3" + + bucket = var.s3_bucket + region = var.s3_region + endpoint = var.s3_endpoint + forcePathStyle = var.s3_force_path_style + + accessKeyId = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.s3_access_key_id + } + } + secretAccessKey = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.s3_secret_access_key + } + } + + # The three upload kinds share the tenant's bucket and separate by prefix. Each one is set + # out in full rather than left to the shared values above, so a caller can move a single + # kind to another bucket without rewriting the module. + eventUpload = local.s3_use.eventUpload + batchExport = merge({ enabled = true }, local.s3_use.batchExport) + mediaUpload = merge({ enabled = true }, local.s3_use.mediaUpload) + } + } + + s3_credentials = { + accessKeyId = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.s3_access_key_id + } + } + secretAccessKey = { + secretKeyRef = { + name = local.secret_name + key = local.secret_keys.s3_secret_access_key + } + } + } + + s3_use = { + for name, prefix in { + eventUpload = var.s3_event_upload_prefix + batchExport = var.s3_batch_export_prefix + mediaUpload = var.s3_media_upload_prefix + } : name => merge( + { + bucket = var.s3_bucket + prefix = prefix + region = var.s3_region + endpoint = var.s3_endpoint + forcePathStyle = var.s3_force_path_style + }, + local.s3_credentials + ) + } +} + +resource "kubernetes_namespace_v1" "this" { + metadata { + name = var.namespace + } +} + +# One secret per tenant holds every credential the chart references. Nothing sensitive reaches the +# Helm values, so the rendered release stays readable in a plan. +resource "kubernetes_secret_v1" "this" { + metadata { + name = local.secret_name + namespace = kubernetes_namespace_v1.this.metadata[0].name + } + + data = merge( + { + (local.secret_keys.salt) = var.salt + (local.secret_keys.encryption_key) = var.encryption_key + (local.secret_keys.nextauth_secret) = var.nextauth_secret + (local.secret_keys.postgres_password) = var.postgres_password + (local.secret_keys.clickhouse_password) = var.clickhouse_password + (local.secret_keys.valkey_password) = var.valkey_password + (local.secret_keys.s3_access_key_id) = var.s3_access_key_id + (local.secret_keys.s3_secret_access_key) = var.s3_secret_access_key + (local.secret_keys.init_project_secret_key) = var.init_project_secret_key + }, + local.postgres_direct_url_set ? { + (local.secret_keys.postgres_direct_url) = local.postgres_direct_url + } : {}, + local.init_user_enabled ? { + (local.secret_keys.init_user_password) = var.init_user_password + } : {}, + local.oidc_enabled ? { + (local.secret_keys.oidc_client_secret) = var.oidc.client_secret + } : {} + ) +} + +# The chart is published to a classic Helm repository, not to an OCI registry, so the repository +# URL and the chart name are given separately. +resource "helm_release" "langfuse" { + name = var.release_name + namespace = kubernetes_namespace_v1.this.metadata[0].name + repository = "https://langfuse.github.io/langfuse-k8s" + chart = "langfuse" + version = var.chart_version + + create_namespace = false + atomic = true + wait = true + timeout = var.helm_timeout + + values = [yamlencode(local.chart_values)] +} diff --git a/modules/ai/langfuse/buildingblock/outputs.tf b/modules/ai/langfuse/buildingblock/outputs.tf new file mode 100644 index 00000000..bed165ba --- /dev/null +++ b/modules/ai/langfuse/buildingblock/outputs.tf @@ -0,0 +1,70 @@ +output "namespace" { + description = "Namespace this tenant's Langfuse instance runs in." + value = kubernetes_namespace_v1.this.metadata[0].name +} + +output "release_name" { + description = "Helm release name of this tenant's Langfuse instance." + value = var.release_name +} + +output "web_service_name" { + description = "Name of the Service in front of the Langfuse web pods." + value = local.web_service_name + + depends_on = [helm_release.langfuse] +} + +output "host" { + description = "Fully qualified in-cluster hostname of the Langfuse web Service." + value = local.web_service_host + + depends_on = [helm_release.langfuse] +} + +output "port" { + description = "Port the Langfuse web Service listens on." + value = local.web_service_port +} + +output "base_url" { + # LiteLLM's langfuse_otel callback appends '/api/public/otel' to the host it is given, so this + # value goes into LANGFUSE_OTEL_HOST unchanged. + description = "In-cluster base URL of this Langfuse instance, with the scheme and the port. Set it as LANGFUSE_OTEL_HOST on a LiteLLM gateway that traces into this instance." + value = local.base_url + + depends_on = [helm_release.langfuse] +} + +output "public_url" { + description = "Canonical external URL of this Langfuse instance. It is what NEXTAUTH_URL is set to and what a browser opens." + value = local.public_url +} + +output "oidc_callback_url" { + description = "Callback URL to register at the OIDC provider for this tenant's client. Null when var.oidc is not set." + value = local.oidc_enabled ? "${local.public_url}/api/auth/callback/custom" : null +} + +output "project_id" { + description = "Identifier of the project Langfuse created. Traces belong to it and the API keypair is scoped to it." + value = var.init_project_id +} + +output "project_public_key" { + # Not a secret on its own, but it is half of a credential and a caller passes it around with the + # secret key, so it is marked as well. + description = "Public key of the project API keypair. A tracing client sends it together with the secret key as basic auth, for example as LANGFUSE_PUBLIC_KEY on a LiteLLM gateway." + value = var.init_project_public_key + sensitive = true + + depends_on = [helm_release.langfuse] +} + +output "project_secret_key" { + description = "Secret key of the project API keypair. A tracing client sends it together with the public key as basic auth, for example as LANGFUSE_SECRET_KEY on a LiteLLM gateway." + value = var.init_project_secret_key + sensitive = true + + depends_on = [helm_release.langfuse] +} diff --git a/modules/ai/langfuse/buildingblock/variables.tf b/modules/ai/langfuse/buildingblock/variables.tf new file mode 100644 index 00000000..b7bff917 --- /dev/null +++ b/modules/ai/langfuse/buildingblock/variables.tf @@ -0,0 +1,656 @@ +variable "namespace" { + type = string + default = "langfuse" + # One tenant per namespace. The caller derives the name, because this module is instantiated + # once per tenant and two tenants in one namespace would share Service names and Secrets. + description = "Namespace this tenant's Langfuse instance runs in. The module creates it. Give every tenant a namespace of its own." +} + +variable "release_name" { + type = string + default = "langfuse" + # The chart derives the Service name from the release name: '-langfuse-web', or just + # '-web' when the release name already contains 'langfuse'. + description = "Helm release name of this tenant's Langfuse instance." +} + +variable "chart_version" { + type = string + default = "1.5.41" + description = "Version of the langfuse chart from https://langfuse.github.io/langfuse-k8s. This is a classic Helm repository, not an OCI registry." +} + +variable "image_tag" { + type = string + default = "4.10.0" + # The chart's appVersion is 3.224.1, so the chart alone installs Langfuse v3. Overriding the tag + # is what selects v4, exactly as examples/v4-installation in langfuse-k8s does. + description = <<-EOT + Tag of the langfuse/langfuse and langfuse/langfuse-worker images. It has to name a concrete v4 + release, because the chart's own appVersion still points at v3. + + Do not use the floating `"4"` tag. It moves whenever Langfuse publishes a v4 release, so two + pods of the same Deployment can end up on two different builds, a `helm upgrade` that changes + nothing still rolls the Deployment, and a regression cannot be rolled back by reverting the + Terraform change. Pin a full version and raise it deliberately. + EOT + + validation { + condition = can(regex("^4\\.[0-9]+\\.[0-9]+$", var.image_tag)) + error_message = "image_tag must be a concrete Langfuse v4 release such as '4.10.0'. The floating tags '4' and '4.10' move under the running Deployment." + } +} + +variable "hostname" { + type = string + # NEXTAUTH_URL is baked into every OAuth callback and every link in an invitation mail. The + # chart's default is http://localhost:3000, which breaks both. + description = "Canonical hostname of this tenant's Langfuse instance, without a scheme. The module derives NEXTAUTH_URL and the Ingress rule from it." + + validation { + condition = !strcontains(var.hostname, "://") + error_message = "hostname must be a bare hostname without a scheme, for example 'langfuse-team-a.example.com'." + } +} + +variable "public_url" { + type = string + default = null + # Only needed when the instance is not reached over HTTPS on var.hostname, for example behind a + # proxy that terminates on a different name. + description = "Full canonical URL of this tenant's Langfuse instance, used as NEXTAUTH_URL. Null derives 'https://'." +} + +variable "helm_timeout" { + type = number + default = 900 + description = "Seconds to wait for the Helm release to become ready. Every web pod runs the Postgres and the ClickHouse migrations before it answers its readiness probe, so the first install takes part of this budget." +} + +# --- Secrets that must differ per tenant ------------------------------------------------------- + +variable "salt" { + type = string + sensitive = true + # SALT hashes the API keys of this tenant. Langfuse also mixes it into the fast hash of every + # key, so changing it invalidates every key the tenant holds. + description = "Salt Langfuse hashes API keys with. It must be unique per tenant, because two tenants sharing a salt share the hash space of their API keys. Changing it later invalidates every API key of the tenant." + + validation { + condition = length(var.salt) >= 32 + error_message = "The salt must be at least 32 characters long. Generate one with `openssl rand -base64 32`." + } +} + +variable "encryption_key" { + type = string + sensitive = true + # ENCRYPTION_KEY encrypts secrets at rest in this tenant's Postgres β€” LLM API keys, integration + # credentials and the like. + description = "Key Langfuse encrypts secrets at rest with, in this tenant's Postgres database. Must be 256 bits, which is 64 hex characters. Generate one with `openssl rand -hex 32`. It must be unique per tenant, because it is the only thing separating one tenant's stored credentials from another's." + + validation { + condition = can(regex("^[0-9a-fA-F]{64}$", var.encryption_key)) + error_message = "The encryption key must be exactly 64 hexadecimal characters, which is 256 bits." + } +} + +variable "nextauth_secret" { + type = string + sensitive = true + description = "Secret NextAuth signs its JWTs and hashes email verification tokens with. It must be unique per tenant, otherwise a session token minted for one tenant is accepted by another. Generate one with `openssl rand -base64 32`." + + validation { + condition = length(var.nextauth_secret) >= 32 + error_message = "The NextAuth secret must be at least 32 characters long." + } +} + +# --- Postgres --------------------------------------------------------------------------------- + +variable "postgres_host" { + type = string + description = "Hostname of the shared Postgres server. Langfuse keeps its relational data β€” organisations, projects, users, prompts and encrypted secrets β€” here." +} + +variable "postgres_port" { + type = number + default = 5432 + # Langfuse never reads DATABASE_PORT. Its entrypoint builds DATABASE_URL from DATABASE_HOST + # alone, so the module folds the port into the host it hands the chart. + description = "Port of the Postgres server. The module appends it to the host, because Langfuse builds its connection URL from the host only and ignores the port variable the chart sets." +} + +variable "postgres_database" { + type = string + # Per-tenant separation in Postgres is a database plus an owner user. The caller creates both; + # this module only connects. + description = "Name of this tenant's Postgres database. It has to exist before the first apply, together with its owner user β€” Prisma creates the tables inside it, not the database itself." +} + +variable "postgres_username" { + type = string + description = "User Langfuse connects as. It has to own this tenant's database, because `prisma migrate deploy` creates and alters tables under it on every web pod start." +} + +variable "postgres_password" { + type = string + sensitive = true + # The entrypoint substitutes the value into postgresql://:@/ verbatim + # and does not percent-encode it. Prisma then answers P1013 on a password with a reserved + # character in it. + description = "Password of the Postgres user. Use only characters that are safe in a URL: Langfuse builds its connection URL by string substitution and does not percent-encode the value." + + validation { + condition = !can(regex("[:@/?#%]", var.postgres_password)) + error_message = "The Postgres password must not contain ':', '@', '/', '?', '#' or '%'. Langfuse substitutes it into the connection URL without percent-encoding, and Prisma then fails with P1013." + } +} + +variable "postgres_args" { + type = string + default = "sslmode=require" + # Appended to the connection URL after a '?', so no leading question mark here. + description = "Query string appended to the Postgres connection URL, without the leading '?'. Managed Postgres offerings terminate TLS, so requiring it is the safe default. A server without TLS needs 'sslmode=prefer' or 'sslmode=disable'. The module appends `connection_limit` from postgres_connection_limit, and `pool_timeout` belongs here when the default of 10 seconds is too short for the pinned pool." + + validation { + condition = !can(regex("connection_limit", var.postgres_args)) + error_message = "postgres_args must not carry connection_limit. Set postgres_connection_limit instead: the module appends the parameter itself, and two occurrences in one query string leave the effective pool size to the parser." + } +} + +variable "postgres_connection_limit" { + type = number + default = 5 + # See the connection budget section in the module README for the arithmetic. + description = <<-EOT + Maximum number of Postgres connections one Langfuse pod opens. The module appends it to the + connection URL as `connection_limit`, next to `postgres_args`. + + Without the parameter Prisma sizes the pool as `physical cores Γ— 2 + 1` read from the node, not + from the pod's CPU limit, so a pod takes 33 connections on a 16-core node and a different number + after it is rescheduled onto a node with more cores. + + This module runs once per tenant against a shared instance, so the platform's budget is + `tenants Γ— pods per tenant Γ— connection_limit + 15 ≀ max_connections`. STACKIT PostgreSQL Flex + fixes `max_connections` per flavour and reserves 15 of them, so the default of 5 is what keeps a + 4 vCPU / 32 GiB instance at roughly 34 tenants with four pods each. + EOT + + validation { + condition = var.postgres_connection_limit >= 1 + error_message = "postgres_connection_limit must be at least 1." + } +} + +variable "postgres_direct_url" { + type = string + sensitive = true + default = null + # DIRECT_URL is what the migrations run against. Without it the entrypoint reuses DATABASE_URL. + description = "Full connection URL the schema migrations run against, used when the normal connection goes through a pooler or when migrations need a user with longer timeouts. The module appends `connection_limit` from postgres_direct_url_connection_limit to it. Null makes the migrations reuse the normal connection, which already carries the pinned pool." + + validation { + condition = var.postgres_direct_url == null ? true : !can(regex("connection_limit", var.postgres_direct_url)) + error_message = "postgres_direct_url must not carry connection_limit. Set postgres_direct_url_connection_limit instead: the module appends the parameter itself." + } +} + +variable "postgres_direct_url_connection_limit" { + type = number + default = 2 + # A migration is not a pool. The web entrypoint runs `prisma db execute` and then + # `prisma migrate deploy`, one after the other, and each opens a single connection. + description = "Maximum number of Postgres connections the migration connection opens, appended to postgres_direct_url as `connection_limit`. Two covers the `prisma db execute` and the `prisma migrate deploy` the web entrypoint runs one after the other, each of which opens a single connection. Only used when postgres_direct_url is set." + + validation { + condition = var.postgres_direct_url_connection_limit >= 1 + error_message = "postgres_direct_url_connection_limit must be at least 1." + } +} + +variable "postgres_auto_migrate" { + type = bool + default = true + # See the ClickHouse counterpart: every web pod runs the migrations on every start. + description = "Run `prisma migrate deploy` on every web pod start. Turn it off above a handful of tenants and run the migrations once out of band, because every pod of every tenant runs them on every restart." +} + +# --- ClickHouse ------------------------------------------------------------------------------- + +variable "clickhouse_host" { + type = string + description = "Fully qualified hostname of the shared ClickHouse cluster, without a scheme. Take it from the `host` output of modules/ai/clickhouse." +} + +variable "clickhouse_http_port" { + type = number + default = 8123 + description = "HTTP port of ClickHouse. Langfuse reads and writes trace data over it." +} + +variable "clickhouse_native_port" { + type = number + default = 9000 + description = "Native protocol port of ClickHouse. The golang-migrate schema migrations use it." +} + +variable "clickhouse_database" { + type = string + # Per-tenant separation in ClickHouse is a database plus a user scoped to '.*'. The + # caller creates both. + description = "Name of this tenant's ClickHouse database. It has to exist before the first apply: golang-migrate creates the tables inside it and never creates a database." +} + +variable "clickhouse_username" { + type = string + description = "ClickHouse user Langfuse connects as. It needs SELECT, INSERT, CREATE, DROP TABLE, ALTER UPDATE, ALTER DELETE and ALTER DROP INDEX on its own database, and nothing beyond it." +} + +variable "clickhouse_password" { + type = string + sensitive = true + # The migration script interpolates the password into a query string, so a reserved character + # in it breaks the migration URL rather than the connection. + description = "Password of the ClickHouse user. Avoid '&', '=', '#', '?', '%', '+', '@' and spaces: the migration script puts the value into a query string without encoding it." + + validation { + condition = !can(regex("[&=#?%+@ ]", var.clickhouse_password)) + error_message = "The ClickHouse password must not contain '&', '=', '#', '?', '%', '+', '@' or a space. The Langfuse migration script interpolates it into a query string without encoding." + } +} + +variable "clickhouse_cluster_enabled" { + type = bool + default = true + # The operator names its ClickHouse cluster 'default', which is what Langfuse writes its + # ON CLUSTER statements against. + description = "Run the ClickHouse DDL with ON CLUSTER. Keep it on for the operator-managed cluster, which is a real cluster named 'default' even with a single replica. Turn it off only for a standalone ClickHouse with no Keeper." +} + +variable "clickhouse_auto_migrate" { + type = bool + default = true + description = "Run the ClickHouse `migrate up` on every web pod start. Turn it off above a handful of tenants and run the migrations once out of band: the statements are ON CLUSTER DDL, and several tenants upgrading at once contend for one distributed DDL queue." +} + +# --- Valkey ----------------------------------------------------------------------------------- + +variable "valkey_host" { + type = string + description = "Hostname of the shared Valkey or Redis instance. Langfuse uses it as the BullMQ queue backend, the cache and the rate limit store." +} + +variable "valkey_port" { + type = number + default = 6379 + description = "Port of the Valkey instance." +} + +variable "valkey_username" { + type = string + default = "default" + description = "Username for Valkey authentication. Null omits the username from the connection string, which is what a Valkey without ACLs expects." +} + +variable "valkey_password" { + type = string + sensitive = true + # The chart puts the value into the connection URL as $(REDIS_PASSWORD), which Kubernetes + # substitutes verbatim. + description = "Password of the Valkey instance. Use only characters that are safe in a URL, because the chart substitutes the value into the connection URL without encoding it." +} + +variable "valkey_database" { + type = number + # The DB index is a hard namespace: SELECT n puts the connection in a keyspace no command can + # reach out of, so no application bug can cross it. + description = "Valkey database index of this tenant. It is a hard namespace that no application bug can cross, so give every tenant an index of its own. A stock Valkey serves indices 0 to 15." + + validation { + condition = var.valkey_database >= 0 + error_message = "valkey_database must be zero or greater." + } +} + +variable "valkey_key_prefix" { + type = string + # REDIS_KEY_PREFIX is not exposed by the chart, so the module injects it through + # langfuse.additionalEnv. It survives a later move to Redis Cluster, where DB indices vanish. + description = <<-EOT + Prefix Langfuse puts in front of every Valkey key and every BullMQ queue name for this tenant, + for example `tenant-a:`. Give every tenant a prefix of its own. + + Set it together with `valkey_database`, not instead of it. The index is a hard namespace that no + application bug can cross; the prefix survives a later move to Redis Cluster, where indices + vanish because a cluster only has database 0. + + Langfuse needs at least app version 3.157.0 for this to work. The variable existed before, but + BullMQ ignored it until langfuse/langfuse#11898 merged on 2026-03-06. + EOT + + validation { + condition = length(var.valkey_key_prefix) > 0 + error_message = "valkey_key_prefix must not be empty. Sharing one Valkey database without a key prefix lets one tenant's worker consume another tenant's ingestion jobs." + } +} + +# --- S3 --------------------------------------------------------------------------------------- + +variable "s3_bucket" { + type = string + # One bucket per tenant. Prefixes inside it only separate the three kinds of upload. + description = "Bucket this tenant's event uploads, batch exports and media uploads go to. Give every tenant a bucket of its own." +} + +variable "s3_region" { + type = string + default = "auto" + description = "Region of the bucket. 'auto' works for S3-compatible object storage that has no region concept." +} + +variable "s3_endpoint" { + type = string + description = "Endpoint URL of the object storage, including the scheme." +} + +variable "s3_force_path_style" { + type = bool + default = true + # Virtual-hosted style needs a wildcard DNS record per bucket, which S3-compatible storage + # outside AWS rarely provides. + description = "Address the bucket as a path on the endpoint instead of as a subdomain. Required for MinIO and for most S3-compatible object storage." +} + +variable "s3_access_key_id" { + type = string + sensitive = true + description = "Access key id of the credential scoped to this tenant's bucket." +} + +variable "s3_secret_access_key" { + type = string + sensitive = true + description = "Secret access key of the credential scoped to this tenant's bucket." +} + +variable "s3_event_upload_prefix" { + type = string + default = "events/" + description = "Prefix inside the bucket for raw ingestion events. Keep the trailing slash." +} + +variable "s3_batch_export_prefix" { + type = string + default = "exports/" + description = "Prefix inside the bucket for batch exports. Keep the trailing slash." +} + +variable "s3_media_upload_prefix" { + type = string + default = "media/" + description = "Prefix inside the bucket for media uploads. Keep the trailing slash." +} + +# --- Bootstrap -------------------------------------------------------------------------------- + +variable "init_org_id" { + type = string + # LANGFUSE_INIT_ORG_ID is the trigger of the whole bootstrap. Without it Langfuse logs a warning + # and silently ignores every other LANGFUSE_INIT_* variable. + description = "Identifier of the organisation Langfuse creates on startup. It is the trigger of the whole bootstrap: with it unset, every other init value is silently ignored." + + validation { + condition = length(var.init_org_id) > 0 + error_message = "init_org_id must not be empty." + } +} + +variable "init_org_name" { + type = string + description = "Display name of the organisation." +} + +variable "init_project_id" { + type = string + description = "Identifier of the project Langfuse creates inside the organisation. Traces belong to a project, and the API keypair below is scoped to it." +} + +variable "init_project_name" { + type = string + description = "Display name of the project." +} + +variable "init_project_public_key" { + type = string + # Langfuse accepts a predefined keypair, so Terraform generates the keys instead of reading them + # back out of a UI. Nothing about this is gated behind an Enterprise entitlement. + description = "Public key of the API keypair Langfuse creates for the project. Langfuse's own generator produces 'pk-lf-', and clients expect that shape." + + validation { + condition = startswith(var.init_project_public_key, "pk-lf-") + error_message = "init_project_public_key must start with 'pk-lf-'." + } +} + +variable "init_project_secret_key" { + type = string + sensitive = true + description = "Secret key of the API keypair Langfuse creates for the project. Langfuse's own generator produces 'sk-lf-', and clients expect that shape." + + validation { + condition = startswith(var.init_project_secret_key, "sk-lf-") + error_message = "init_project_secret_key must start with 'sk-lf-'." + } +} + +variable "init_user_email" { + type = string + default = null + # This creates a password user. It contradicts disable_username_password, so leave it unset + # whenever an identity provider is configured and let the owner arrive through SSO instead. + description = "Email address of a first user with a password, given OWNER membership in the organisation. Leave it null when var.oidc is set: a password user cannot log in once username and password login is off." +} + +variable "init_user_name" { + type = string + default = null + description = "Display name of the first user. Null lets Langfuse name it 'Provisioned User'. Only used when init_user_email and init_user_password are both set." +} + +variable "init_user_password" { + type = string + sensitive = true + default = null + # Langfuse sets it while creating the user and never again, so a later change to this value + # does not reset the password. + description = "Password of the first user. Langfuse only sets it while creating the user, so changing this value later does not reset the password. Both init_user_email and init_user_password have to be set, or neither." + + validation { + condition = var.init_user_password == null || length(coalesce(var.init_user_password, "")) >= 12 + error_message = "The initial user password must be at least 12 characters long." + } + + validation { + condition = (var.init_user_email == null) == (var.init_user_password == null) + error_message = "Set both init_user_email and init_user_password, or neither. Langfuse logs a warning and creates no user when only one of them is present." + } +} + +# --- Authentication --------------------------------------------------------------------------- + +variable "oidc" { + description = <<-EOT + OIDC identity provider this tenant's users log in through. Null leaves the instance on username + and password login, which then needs `init_user_email` and `init_user_password`. + + Self-hosted SSO is free in Langfuse. It carries no entitlement, so it works without an + Enterprise licence. + + - `issuer_url`: discovery base URL of the provider, for example + `https://idp.example.com/realms/ai`. Langfuse reads `/.well-known/openid-configuration`. + - `client_id` and `client_secret`: credentials of the OIDC client. + - `display_name`: label on the login button. The provider is not registered without it. + - `scopes`: space-separated scope list. The default covers what Langfuse needs. + - `allow_account_linking`: link an OIDC login to an existing user with the same email address. + Turn it on when a user already exists from a password login or from another provider. + + Register the callback URL `/api/auth/callback/custom` at the provider. + + Give every tenant an OIDC client of its own and assign only that tenant's members to it. See the + access control note in the module README: with auto-join on, everyone the provider authenticates + becomes a member of this tenant's organisation. + EOT + + type = object({ + issuer_url = string + client_id = string + client_secret = string + display_name = optional(string, "Single Sign-On") + scopes = optional(string, "openid email profile") + allow_account_linking = optional(bool, true) + }) + + default = null + sensitive = true + + validation { + condition = var.oidc != null || var.init_user_email != null + error_message = "Set var.oidc, or set init_user_email and init_user_password. Without one of the two, nobody can log in to the instance." + } + + # The message carries no interpolation, because var.oidc is sensitive and Terraform refuses to + # print a sensitive value in an error message. + validation { + condition = var.oidc == null || startswith(var.oidc.issuer_url, "https://") + error_message = "oidc.issuer_url must be an https URL. It is the discovery base URL, not the authorization endpoint." + } +} + +variable "disable_username_password" { + type = bool + default = null + # Null derives it from var.oidc: with an identity provider configured, username and password + # login is the thing to turn off, not sign-up. + description = "Turn off username and password login, so only the OIDC provider remains. Null turns it on whenever var.oidc is set. Never set it to true without an identity provider: nobody could log in." + + validation { + condition = var.disable_username_password != true || var.oidc != null + error_message = "disable_username_password cannot be true without var.oidc, because that would leave no way to log in." + } +} + +variable "default_org_role" { + type = string + default = "MEMBER" + # Langfuse upserts an organisation membership with this role for every user who logs in, and + # there is no entitlement check on that path. With one organisation and one project per tenant, + # the organisation role is the whole access grant. + description = <<-EOT + Role every user who logs in receives in this tenant's organisation. Langfuse upserts the + membership on first login and on every later login, and the upsert never overwrites a role a + user already has. + + This removes member synchronisation from the picture: no group mapping and no SCIM. Set `NONE` + to hand out a membership that grants nothing, which is the way to turn auto-join off while + keeping the instance usable for users who were added by hand. + EOT + + validation { + condition = contains(["OWNER", "ADMIN", "MEMBER", "VIEWER", "NONE"], var.default_org_role) + error_message = "default_org_role must be one of OWNER, ADMIN, MEMBER, VIEWER or NONE." + } +} + +# --- Ingress ---------------------------------------------------------------------------------- + +variable "ingress_enabled" { + type = bool + default = true + description = "Create an Ingress for var.hostname. Turn it off when the instance is reached in-cluster only, and set var.public_url accordingly." +} + +variable "ingress_class_name" { + type = string + default = "haproxy" + description = "Name of the IngressClass that serves this instance. It has to match the controller modules/kubernetes/ingress installs." +} + +variable "ingress_annotations" { + type = map(string) + default = {} + description = "Annotations on the Ingress. Set `cert-manager.io/cluster-issuer` here when the instance needs a certificate of its own instead of the controller's wildcard certificate." +} + +variable "ingress_tls_secret_name" { + type = string + default = null + # With the wildcard certificate from modules/kubernetes/ingress the controller already serves + # HTTPS for every host it does not have a certificate for, so the Ingress needs no tls block. + description = "Name of the secret holding the TLS certificate for var.hostname. Null leaves the Ingress without a tls block, which is correct when the ingress controller serves a wildcard certificate as its default." +} + +# --- Sizing ----------------------------------------------------------------------------------- + +variable "web_replicas" { + type = number + default = 1 + description = "Number of Langfuse web pods. The default of 1 is sized for a demonstration and gives no redundancy: every restart interrupts the UI and the ingestion API." +} + +variable "worker_replicas" { + type = number + default = 1 + description = "Number of Langfuse worker pods. The default of 1 is sized for a demonstration. Raise it when the ingestion queue backs up." +} + +variable "web_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "100m", memory = "512Mi" } + limits = { cpu = "500m", memory = "1Gi" } + } + description = <<-EOT + Resource requests and limits of the Langfuse web pods. The default is sized for a demonstration + and a production consumer has to raise it. + + The chart sets `resources: {}` for the web deployment, so the pods run unbounded without these + values. The web pod serves the UI and the ingestion API and runs both migrations on start, which + is the peak of its memory use. Production wants `500m` CPU and `2Gi` of memory. + + The module derives `NODE_OPTIONS=--max-old-space-size` from the memory limit at roughly 75%, so + Node runs a garbage collection before the cgroup limit is reached instead of being OOMKilled. + EOT +} + +variable "worker_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "100m", memory = "384Mi" } + limits = { cpu = "500m", memory = "768Mi" } + } + description = <<-EOT + Resource requests and limits of the Langfuse worker pods. The default is sized for a + demonstration and a production consumer has to raise it. + + The chart sets `resources: {}` for the worker deployment as well. The worker drains the BullMQ + queues and batches writes into ClickHouse, so its memory grows with the batch size rather than + with the number of users. Production wants `500m` CPU and `2Gi` of memory. + + The module derives `NODE_OPTIONS=--max-old-space-size` from the memory limit at roughly 75%. + EOT +} + +variable "telemetry_enabled" { + type = bool + default = false + description = "Report basic usage statistics to Langfuse. The chart turns this on by default; a self-hosted tenant instance usually should not phone home." +} diff --git a/modules/ai/langfuse/buildingblock/versions.tf b/modules/ai/langfuse/buildingblock/versions.tf new file mode 100644 index 00000000..cc6c09d8 --- /dev/null +++ b/modules/ai/langfuse/buildingblock/versions.tf @@ -0,0 +1,16 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + helm = { + source = "hashicorp/helm" + # The helm provider takes its cluster credentials as the `kubernetes = {}` attribute + # starting with 3.0.0. Earlier versions expect a `kubernetes {}` block instead. + version = ">= 3.0.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38" + } + } +} diff --git a/modules/ai/litellm/buildingblock/APP_TEAM_README.md b/modules/ai/litellm/buildingblock/APP_TEAM_README.md new file mode 100644 index 00000000..b6683e42 --- /dev/null +++ b/modules/ai/litellm/buildingblock/APP_TEAM_README.md @@ -0,0 +1,84 @@ +Your application reaches the large language models the platform team offers through one OpenAI-compatible endpoint. You get a virtual key and a model alias, and you point any OpenAI client at the gateway. The credentials of the model provider stay with the platform team, and your spend and rate limits are tracked against your own key. + +## 🎯 When to use it + +Use this building block when you: +- want to call a language model from your application without asking a provider for your own account and credential +- need to switch between models, or between providers, without changing your application code +- have to keep a budget per team or per application and see what has been spent +- want the same endpoint in every environment, so a change of model stays a configuration change + +## πŸ’‘ Usage examples + +**Example 1: Add a summarization feature** +Your service summarizes support tickets. You point the OpenAI SDK at the gateway URL, set your virtual key as the API key and ask for the model alias the platform team published. Nothing else in your code changes when the platform team moves that alias to another model. + +**Example 2: Keep an experiment inside a budget** +Your team tries a retrieval feature and does not want to spend more than the budget it was given. The gateway counts every call against your virtual key and refuses further requests once the budget is used up, so an experiment cannot run away with cost. + +## πŸ”§ How to use it + +The gateway speaks the OpenAI API, so every OpenAI client works. Set the base URL to the gateway, including the `/v1` suffix, and use your virtual key as the API key. + +```python +from openai import OpenAI + +client = OpenAI( + base_url="http://litellm.litellm.svc.cluster.local:4000/v1", + api_key="sk-your-virtual-key", +) + +response = client.chat.completions.create( + model="chat-large", + messages=[{"role": "user", "content": "Summarize this ticket."}], +) +``` + +The same call with curl: + +```bash +curl http://litellm.litellm.svc.cluster.local:4000/v1/chat/completions \ + -H "Authorization: Bearer sk-your-virtual-key" \ + -H "Content-Type: application/json" \ + -d '{"model": "chat-large", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +Two things go wrong often enough to name them: + +- **Keep the `/v1` in the base URL.** Without it every call answers "Not Found". +- **Use the alias, not the name of the model at the provider.** The gateway resolves the alias to a model and an endpoint, and the alias is the only name it accepts. + +## πŸ” The admin console holds five users, and none of them is you + +The gateway has a web console. It belongs to the platform team, and there is one console for the whole platform, because there is one gateway. You do not get a login to it, and this section explains why that is a rule and not an oversight. + +**The console holds at most five users, platform-wide.** "User" here means one row in the gateway's user table. It is not your virtual key and it is not your team: + +- Your virtual key costs **no** seat. The gateway creates no user when it issues a key. +- Your team on the gateway costs **no** seat either, with the setting this platform runs. +- A human who logs in to the console costs **one** seat, and there are five. + +**The failure mode is worse than a refusal.** The sixth person to log in gets in and writes the sixth row. From that point every login attempt is refused, for all six of them, including the people who were working in the console the day before. Sessions already open keep working until they expire, and that is the whole grace period. Getting out of it means the platform team deleting a user from the gateway by hand, which is an incident rather than a ticket. + +So there are two requests you should not make, however reasonable they look: + +- **Do not ask to be added as a member of your team on the gateway.** That writes one row per member and takes one of the five seats. Your virtual key already carries your budget and your rate limit; a team membership adds nothing you need. +- **Do not ask for a user account on the gateway.** Same cost, same reason. + +If you need to see what your key has spent, ask the platform team. A console login is not the way to get that number. + +An Enterprise licence would lift the limit. This platform does not buy one, so the limit of five is a fixed property of the gateway you are using. + +## πŸ“Š Shared Responsibility + +| Responsibility | Platform Team | Application Team | +|---|:---:|:---:| +| Run the gateway and its database | βœ… | ❌ | +| Hold the credentials of the model providers | βœ… | ❌ | +| Decide which models are available and under which alias | βœ… | ❌ | +| Issue virtual keys and set budgets and rate limits | βœ… | ❌ | +| Log in to the gateway's admin console | βœ… | ❌ | +| Keep the virtual key secret and rotate it when it leaks | ❌ | βœ… | +| Pick a model alias that fits the task and its cost | ❌ | βœ… | +| Handle rate limit and budget errors in the application | ❌ | βœ… | +| Decide what data the application sends to a model | ❌ | βœ… | diff --git a/modules/ai/litellm/buildingblock/README.md b/modules/ai/litellm/buildingblock/README.md new file mode 100644 index 00000000..30cd1a41 --- /dev/null +++ b/modules/ai/litellm/buildingblock/README.md @@ -0,0 +1,380 @@ +--- +name: LiteLLM AI Gateway +supportedPlatforms: + - kubernetes +description: Installs the LiteLLM gateway into a Kubernetes namespace and registers OpenAI-compatible model endpoints behind one API, with virtual keys, teams, budgets and spend tracking backed by Postgres, and optional OIDC single sign-on for the admin console. +# The cluster credentials, the database connection and the upstream model credentials all arrive +# as inputs, so there is nothing to set up on the cloud side before this module runs. +requiresBackplane: false +--- + +# LiteLLM AI Gateway Building Block + +The platform team installs the LiteLLM gateway into a Kubernetes namespace with this module. The gateway puts one OpenAI-compatible API in front of the model endpoints the platform team registers, and it issues virtual keys with their own budgets and rate limits, so an application team never sees the credential of an upstream provider. + +This documentation is intended as a reference for cloud foundation or platform engineers using this module. + +## Sourced, not ordered + +There is no `meshstack_integration.tf` and no `backplane/`. The `ai-platform` reference architecture sources `buildingblock/` from its own building block, and a foundation can source it from a Terragrunt unit. Application teams do not order this module; they order a virtual key from a separate building block that talks to the gateway this module installs. + +## The chart + +| | | +|---|---| +| Chart | `litellm-helm` | +| Reference | `oci://ghcr.io/berriai/litellm-helm` | +| Pinned version | `1.96.2` (`var.chart_version`) | +| Source | [`helm/litellm-helm/`](https://github.com/BerriAI/litellm/tree/main/helm/litellm-helm) in `BerriAI/litellm` | +| Prerequisites | Kubernetes 1.21+, Helm 3.8.0+ | + +Three details about this chart cost time when you meet them for the first time: + +- **The chart is published to GHCR as an OCI artifact only.** There is no classic Helm repository to add. The helm provider takes the registry and the path prefix as `repository` and the chart name on its own as `chart`, which is why `main.tf` reads `repository = "oci://ghcr.io/berriai"` and `chart = "litellm-helm"`. +- **`Chart.yaml` on the repository's main branch lies about the version.** It reads a much lower number, because the release pipeline overwrites the field while publishing. Pick the version from the [GHCR tag list](https://github.com/BerriAI/litellm/pkgs/container/litellm-helm) instead. +- **A second, newer chart lives at `helm/litellm/`.** That one splits the proxy into microservices. This module uses the monolithic `litellm-helm` chart. The old `deploy/charts/` path from older documentation no longer exists. + +## Postgres is required + +Virtual keys, teams, budgets and spend tracking all live in Postgres. Without a database the gateway is a stateless proxy and none of those endpoints work, which is the whole reason to run it here. The module therefore takes the connection as mandatory inputs: `postgres_host`, `postgres_port`, `postgres_database`, `postgres_username` and `postgres_password`. Any Postgres works. On STACKIT a PostgreSQL Flex instance fits; in another foundation an operator-managed cluster in the same Kubernetes cluster fits just as well. + +The chart bundles a Bitnami postgresql subchart under `db.deployStandalone`, and that switch defaults to **true**. This module sets it to `false` and sets `db.useExisting = true` instead. The chart's own README recommends the same: those images no longer receive updates and the subchart is pinned to `bitnamilegacy/postgresql`. + +Four points to keep in mind: + +- **The database has to exist before the first apply.** The migration Job creates the tables inside it, not the database itself. +- **The user needs rights to create and alter tables**, because the migration Job runs the schema migrations under it. +- **The password has to be safe in a URL.** The chart builds the connection URL from `$(DATABASE_USERNAME)` and `$(DATABASE_PASSWORD)`, which Kubernetes substitutes verbatim and does not URL-encode, so a password containing `:`, `@`, `/` or `?` breaks the URL. +- **`postgres_ssl_mode` defaults to `require`.** A managed Postgres terminates TLS and accepts this. A server without TLS needs `prefer` or `disable`. + +The chart's default connection URL carries no port, so the module writes its own `db.url` with `postgres_port`, the sslmode parameter and the pinned `connection_limit` in it. The credentials stay in a Kubernetes Secret and never appear in the pod spec, because the URL keeps the `$(…)` references that Kubernetes resolves from the environment. + +### The Prisma migration Job + +The chart runs the schema migrations in a Prisma Job (`migrationJob.enabled`, default true). Its default annotations address ArgoCD, which means nothing to a Terraform-driven install: the Job would be applied together with the Deployment, and the pods would restart until the schema exists, because the proxy itself runs with `DISABLE_SCHEMA_UPDATE=true` whenever the Job is enabled. + +This module therefore turns the Helm hook on and the ArgoCD annotations off. The Job runs as a `pre-install,pre-upgrade` hook, Helm waits for it to finish, and only then creates the Deployment. Two consequences follow: + +- A failing migration fails the whole `helm_release`, which is what you want β€” the alternative is a pod crash loop that Terraform reports as a timeout. +- The Job needs its share of `helm_timeout`. The default of 600 seconds covers a migration and a rollout together. + +### The connection pool is pinned, and by two settings + +**Prisma sizes its connection pool as `physical cores Γ— 2 + 1` when the connection URL names no `connection_limit`**, and it counts the physical cores of the *node*, not the pod's CPU limit. [prisma-engines#4341](https://github.com/prisma/prisma-engines/issues/4341) records that as an oversight and was closed without a fix. A pod with a `100m` CPU limit therefore takes 33 connections on a 16-core node, and a different number after it is rescheduled onto a node with a different core count. + +A managed Postgres caps `max_connections` by the shape you bought. STACKIT PostgreSQL Flex fixes it per flavour β€” 195 on 4 vCPU / 8 GiB, 785 on 4 vCPU / 32 GiB β€” reserves 15 for its own processes and exposes no parameter group in the API, the SDK or the Terraform provider. The pool has to be pinned on the client side, because there is no server-side lever. + +`var.postgres_connection_limit` pins it, and the module writes it in two places: + +| Where | What it binds | +|---|---| +| `connection_limit` on `db.url` | The Prisma migration Job, and anything else that reads `DATABASE_URL` as it stands. | +| `general_settings.database_connection_pool_limit` | The running proxy. | + +Both are needed. **The proxy rewrites `DATABASE_URL` on startup** and replaces `connection_limit` with `general_settings.database_connection_pool_limit`, so the URL parameter alone does not bind the pods. The module writes the same number into both, and the pool is then the same whichever path sets it. + +The default is `10`, which is LiteLLM's own default, so pinning the value changes nothing at runtime. The gateway is deployed once for the whole platform, so it costs `replica_count Γ— postgres_connection_limit` connections in total β€” 10 at the defaults. Budget it against the instance like this: + +``` +pods per tenant Γ— connection_limit Γ— tenants + 15 reserved ≀ max_connections +``` + +The gateway is one tenant of that sum. Everything else on the instance, a Langfuse instance per tenant above all, competes for the same ceiling. `modules/stackit/postgresflex` carries the [per-flavour table and the budget](../../../stackit/postgresflex/buildingblock/README.md#how-many-connections-one-instance-carries). + +`pool_timeout` is the companion parameter: a request that finds no free connection waits that long and then fails with `P2024`. LiteLLM sets it from `general_settings.database_connection_pool_timeout` and defaults to 60 seconds, which is generous, so this module leaves it alone. + +## Redis + +`redis.enabled` defaults to false in the chart and the module keeps the bundled Redis subchart off, for the same reason as the bundled Postgres. Redis is the coordination store of the gateway: cross-pod rate limits, spend tracking and the pod lock manager. + +| Deployment | Redis | +|---|---| +| `replica_count = 1` | Not needed. One pod counts everything in its own memory and the counts are correct. | +| `replica_count > 1` | Required as soon as you enforce budgets or rate limits. Without it every pod counts on its own, so a team with three pods in front of it can spend up to three times its budget before anything is refused. | + +Set `redis_host`, and optionally `redis_port` and `redis_password`, to point at an existing Redis. The module puts those values into the same secret as the model credentials and writes a `general_settings.coordination_redis` block into the proxy config that reads them from the environment. + +## Registering model backends + +`var.model_backends` is a map keyed by the alias a caller puts in the `model` field of a request. Each entry carries the name of the model at the upstream provider and the base URL of its OpenAI-compatible endpoint. The credentials live in `var.model_backend_api_keys`, keyed the same way and marked sensitive, so a plan stays readable and only the credentials are hidden. + +The module renders this into the chart's `proxy_config.model_list`: + +```yaml +model_list: + - model_name: chat-large + litellm_params: + model: openai/neuralmagic/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic + api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1 + api_key: os.environ/LITELLM_API_KEY_CHAT_LARGE +``` + +Two traps are worth naming, because both produce errors that point somewhere else: + +- **The `openai/` prefix on `model` selects the OpenAI-compatible driver.** Without it LiteLLM tries to guess the provider from the model name and reaches for a different driver. The module adds the prefix, so `var.model_backends` carries the bare upstream model name. +- **`api_base` has to end with `/v1`.** LiteLLM appends the route to this URL, so an endpoint without the suffix answers "Not Found" on every call. The module rejects an `api_base` without it at plan time. + +Each alias becomes an environment variable named `LITELLM_API_KEY_`, with every character outside `[A-Za-z0-9]` replaced by an underscore. The module writes those variables into a Kubernetes Secret and lists it under `environmentSecrets`, which the chart exports into the pods with `envFrom`. That is what the `os.environ/…` references in the proxy config resolve against. Two aliases that would collapse to the same variable name are rejected at plan time, because they would otherwise share one credential. + +Several aliases can share one upstream endpoint and one credential. Repeat the value in `model_backend_api_keys` for each alias. + +## The master key + +`var.master_key` is the root credential of the gateway. It authenticates every call to the `/key` and `/team` endpoints and works as a virtual key itself. LiteLLM rejects a key that does not start with `sk-`, so the module validates the prefix. + +The module writes the key into a Kubernetes Secret and points the chart at it with `masterkeySecretName`. Without that the chart generates a key of its own on every install, which no caller knows and which changes whenever the secret is recreated. + +## The admin console and single sign-on + +The console at `/ui` belongs to the platform team. Tenants never open it: they hold a virtual key and call the API. This module deploys **one** gateway for the whole platform, so there is one console for everybody who administers it. + +`var.oidc` turns on the proxy's native generic OIDC provider. Two things this module deliberately does not offer: + +- **No static `UI_USERNAME` and `UI_PASSWORD` fallback.** A shared password on a console that hands out the powers of the master key is not access control. Either an identity provider decides who gets in, or nobody logs in. +- **No console at all when `var.oidc` is null.** The gateway still works in full β€” every tenant reaches it with a virtual key β€” so a foundation without an identity provider gets a working gateway and no login page. + +Native SSO is free in the open-source proxy up to five users. The five are the subject of [the next section](#the-console-holds-five-users-platform-wide), and you have to read it before you hand the console to a sixth person. + +### What the module writes onto the pods + +`var.oidc` and `var.public_url` become plain environment variables on the proxy container, apart from the client secret, which travels in a Kubernetes Secret listed under `environmentSecrets` β€” the same path the model credentials take. + +| Environment variable | Set from | Default in the proxy at v1.96.2 | +|---|---|---| +| `GENERIC_CLIENT_ID` | `oidc.client_id` | none; its presence is what selects the generic provider | +| `GENERIC_CLIENT_SECRET` | `oidc.client_secret`, through the secret | none; login fails without it | +| `GENERIC_AUTHORIZATION_ENDPOINT` | discovery, or `oidc.authorization_endpoint` | none; login fails without it | +| `GENERIC_TOKEN_ENDPOINT` | discovery, or `oidc.token_endpoint` | none; login fails without it | +| `GENERIC_USERINFO_ENDPOINT` | discovery, or `oidc.userinfo_endpoint` | none; login fails without it | +| `GENERIC_SCOPE` | `oidc.scopes` | `openid email profile` | +| `GENERIC_USER_ID_ATTRIBUTE` | `oidc.user_id_attribute`, `sub` by default | `preferred_username` | +| `GENERIC_USER_EMAIL_ATTRIBUTE` | `oidc.user_email_attribute` | `email` | +| `GENERIC_USER_DISPLAY_NAME_ATTRIBUTE` | `oidc.user_display_name_attribute` | `sub` | +| `GENERIC_USER_ROLE_ATTRIBUTE` | `oidc.user_role_attribute` | `role` | +| `PROXY_BASE_URL` | `var.public_url` | the base URL of the incoming request | +| `PROXY_LOGOUT_URL` | `oidc.logout_url` | none | +| `PROXY_ADMIN_ID` | `oidc.proxy_admin_id` | none | +| `ALLOWED_EMAIL_DOMAINS` | `oidc.allowed_email_domains`, joined with commas | none, so every authenticated user may log in | +| `AUTO_REDIRECT_UI_LOGIN_TO_SSO` | `oidc.auto_redirect_to_sso` | `false` | + +A setting the caller leaves null is not written at all, so the proxy stays on its own default instead of receiving an empty string. + +Three details are worth carrying in your head, because the documentation and the code disagree: + +- **`GENERIC_USER_ID_ATTRIBUTE` defaults to `preferred_username` in the code**, while the documentation table says `sub`. This module pins `sub`. `preferred_username` is reassignable at most providers, and a reassignment produces a second row in the user table for the same person β€” which costs one of the five seats. +- **`GENERIC_USER_DISPLAY_NAME_ATTRIBUTE` defaults to `sub` in the code**, not to `display_name` as the documentation says. Set it to whichever claim carries a readable name, usually `name`, or the console shows opaque subject identifiers. +- **`ALLOWED_EMAIL_DOMAINS` matches the part after the `@` exactly.** There is no wildcard and no subdomain match, so `example.com` does not admit `mail.example.com`. + +### One issuer in, three endpoints out + +`modules/ai/langfuse` takes an issuer and reads the discovery document itself. The LiteLLM proxy does not: it reads the authorization, token and userinfo endpoints as three separate environment variables and fails the login when one of them is missing. This module closes that gap so both modules present the same `oidc` input β€” it reads `/.well-known/openid-configuration` through the `http` data source and takes the three endpoints from there. + +That request runs on **every plan**, which has a cost worth stating plainly: the identity provider has to answer the machine that runs Terraform, not only the pods in the cluster. An outage at the provider, or a provider reachable from inside the cluster only, fails a plan that has nothing to do with SSO β€” adding a model backend, for instance. + +The way out is in `var.oidc`: set `authorization_endpoint`, `token_endpoint` and `userinfo_endpoint`, and the module creates no data source at all. Set one or two of them and discovery still runs for the rest, with the override winning. Prefer the three explicit endpoints in a foundation where the Terraform runner cannot reach the provider. + +### The callback URL + +Register `/sso/callback` at the provider. + +The proxy builds that URL from `PROXY_BASE_URL` joined with `SERVER_ROOT_PATH` and then `/sso/callback`. This module sets no `SERVER_ROOT_PATH`, so the short form holds. + +`PROXY_BASE_URL` is not required by the code β€” it falls back to the base URL of the incoming request β€” but that fallback is the internal `http://` address of the pod behind a TLS-terminating Ingress, and the provider then rejects the redirect URI it receives. `var.public_url` is therefore mandatory whenever `var.oidc` is set, and the module rejects the pair at plan time. + +### There is no claim discovery on the free tier + +`/sso/debug/login`, the route the documentation offers for reading back the claims a provider returns, raises 403 without an Enterprise licence as soon as any SSO client id is configured. Take the claim names from the provider's own documentation or from its discovery document instead. + +## The console holds five users, platform-wide + +Read this section before you give a sixth person access to the console. Recovery from the failure it describes means a manual change in Postgres. + +### A user is a row in the user table + +Five is a limit on rows in `LiteLLM_UserTable`. It is **not** a limit on virtual keys, on teams, or on tenants. A platform with two hundred tenants and two thousand virtual keys can sit at zero users. + +`_raise_if_sso_exceeds_free_user_limit` in `litellm/proxy/management_endpoints/ui_sso.py` counts them through `UserRepository.count_billable_users()`, which is every row in `LiteLLM_UserTable` minus the rows whose metadata marks them SCIM-inactive. The comparison is strictly "greater than five", so the table may hold five rows and not six. + +Where rows come from, verified in the v1.96.2 source: + +| Action | Rows it writes to `LiteLLM_UserTable` | +|---|---| +| `/key/generate`, so every virtual key a tenant receives | **none.** There is an explicit guard against creating a user | +| `/team/new` with `disable_auto_add_proxy_admin_to_teams: true` | **none** | +| `/team/new` without that setting | **one, ever.** Every caller that authenticates with the master key is identified as the same constant user id, `default_user_id`, so the row is written on the first team and reused afterwards | +| `/team/member_add` | one per member | +| `/user/new` | one per user | +| An SSO login | one per human | + +With this module's defaults the only rows are the humans who log in to the console. `var.disable_auto_add_proxy_admin_to_teams` defaults to `true` and is what keeps `/team/new` at zero rows. + +### The failure mode: the sixth login locks out everybody + +The check runs at login **initiation** β€” `/sso/key/generate`, the route behind the login button β€” and not at the callback. Two consequences follow, and the second one is the expensive one: + +- The refusal arrives as a 403 before the browser is ever sent to the identity provider, so the person sees an error on the gateway and not at their provider. +- The check looks at the table, not at the person. The sixth human's own login still passes it, because five rows are not more than five, and it writes the sixth row. **From that moment every login initiation fails, for all six of them.** Nobody who was working in the console yesterday can start a new session today. + +Sessions already open keep working until they expire. That is the whole grace period. + +The error message names the cause: + +> You must be a LiteLLM Enterprise user to use SSO for more than 5 users. If you have a license please set `LITELLM_LICENSE` in your env. […] You are seeing this error message because You configured SSO […] in your env. Please unset it + +### Operations that are forbidden here + +Two API calls write one row per member into `LiteLLM_UserTable`, and each is one line of Terraform away: + +- **`/team/member_add`**, which the `ncecere/litellm` provider exposes as `resource "litellm_team_member"` and `resource "litellm_team_member_add"`. +- **`/user/new`**, for which that provider has no resource at version 2.0.1, so it takes a direct API call or a newer provider version. + +Neither belongs in this architecture, and no module in this repository creates either. A tenant needs no team membership and no user of its own: `modules/ai/model-access` creates a team and a virtual key, and the key alone carries the budget, the rate limit and the model allowance. A membership adds nothing a tenant can use and costs one of five seats. + +**The provider makes the mistake easy, which is exactly why the rule needs writing down.** `litellm_team_member` sits next to `litellm_team` in the provider documentation and reads like the natural next resource to add. It is not. Review any change that introduces it, and reject it unless somebody has bought an Enterprise licence first. + +### An Enterprise licence is out of scope + +`LITELLM_LICENSE` lifts the limit β€” the check returns immediately for a premium user β€” and this architecture does not buy one. Five console users is therefore a fixed property of the platform and not a temporary state to grow out of. Plan the platform team's console access around it. A sixth administrator is a licensing decision, not a Terraform change. + +## Notes for platform engineers + +- **Providers.** `kubernetes`, `helm` and `http`. No cloud provider enters this module, so it runs on SKE, AKS and anything else that speaks the Kubernetes API. The `http` provider only reads the OIDC discovery document, and only when `var.oidc` is set without all three endpoint overrides. +- **Permissions.** The token in `var.token` needs to create a namespace, secrets and the workloads of the release in that namespace. Cluster-admin is not required; the chart installs no CRDs and no cluster-scoped RBAC. +- **The namespace belongs to the module.** It creates `var.namespace` and destroys it again, together with the secrets it wrote there. +- **Reachability.** The Service is a ClusterIP, so the gateway answers inside the cluster at the `api_base` output. Put an Ingress in front of it when callers live outside the cluster, and in any case when you turn on console SSO: the identity provider redirects a browser to `var.public_url`, which therefore has to resolve from outside. + +## Usage + +```hcl +module "litellm" { + source = "github.com/meshcloud/meshstack-hub//modules/ai/litellm/buildingblock?ref=main" + + cluster_endpoint = var.cluster_endpoint + cluster_ca_certificate = var.cluster_ca_certificate + token = var.token + + master_key = var.litellm_master_key + + postgres_host = var.postgres_host + postgres_database = "litellm" + postgres_username = var.postgres_username + postgres_password = var.postgres_password + + model_backends = { + "chat-large" = { + model = "neuralmagic/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic" + api_base = "https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1" + } + "embed" = { + model = "intfloat/e5-mistral-7b-instruct" + api_base = "https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1" + } + } + + model_backend_api_keys = { + "chat-large" = var.stackit_model_serving_token + "embed" = var.stackit_model_serving_token + } + + # Leave both out for a gateway without an admin console. Everything else keeps working. + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "https://idp.example.com/realms/platform" + client_id = "litellm-console" + client_secret = var.litellm_oidc_client_secret + + user_display_name_attribute = "name" + allowed_email_domains = ["example.com"] + } +} +``` + +## Follow-up + +**A locked console has no tested recovery path yet.** Once a sixth row exists in +`LiteLLM_UserTable`, every login initiation fails and nobody can start a new session. The defaults +here make that unlikely β€” `var.disable_auto_add_proxy_admin_to_teams` keeps Terraform from writing +any row, and `user_id_attribute` is pinned to `sub` so one person cannot become two rows β€” but the +risk does not go away, because each console user is one of five. + +Removing a row through the API is the likely way out, and the master key does keep working while the +console is locked, because the seat check sits on the two SSO login routes and not in the general +authentication path. What is **not** tested is the side effects, so no procedure is documented here. +Two things need establishing on a throwaway database first: whether deleting a user also deletes +virtual keys that carry the same `user_id`, and whether the `scim_active` flag the counting function +honours can be set without an Enterprise licence. + +Until then, treat five console users as a hard limit to plan around rather than a threshold to +recover from. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [helm](#requirement\_helm) | >= 3.0.0 | +| [http](#requirement\_http) | >= 3.4 | +| [kubernetes](#requirement\_kubernetes) | >= 2.38 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [helm_release.litellm](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [kubernetes_namespace_v1.this](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_secret_v1.master_key](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [kubernetes_secret_v1.model_credentials](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [kubernetes_secret_v1.oidc](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [kubernetes_secret_v1.postgres](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [http_http.oidc_discovery](https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [chart\_version](#input\_chart\_version) | Version of the litellm-helm chart. See https://github.com/BerriAI/litellm/pkgs/container/litellm-helm. | `string` | `"1.96.2"` | no | +| [client\_certificate](#input\_client\_certificate) | PEM-encoded client certificate this module authenticates with, as an alternative to `token`. Pass the decoded certificate, not the base64 blob a kubeconfig carries. | `string` | `null` | no | +| [client\_key](#input\_client\_key) | PEM-encoded private key belonging to `client_certificate`. Pass the decoded key, not the base64 blob a kubeconfig carries. | `string` | `null` | no | +| [cluster\_ca\_certificate](#input\_cluster\_ca\_certificate) | Cluster CA certificate, base64 encoded. | `string` | n/a | yes | +| [cluster\_endpoint](#input\_cluster\_endpoint) | IP address or hostname of the cluster control plane, without the https:// scheme. | `string` | n/a | yes | +| [disable\_auto\_add\_proxy\_admin\_to\_teams](#input\_disable\_auto\_add\_proxy\_admin\_to\_teams) | Write `general_settings.disable_auto_add_proxy_admin_to_teams: true` into the proxy config, so
the proxy adds no admin member to a team it creates.

Leave it at `true`. With it `false`, the first call to `/team/new` writes one row to
`LiteLLM_UserTable` and that row consumes one of the five console seats the free open-source
proxy allows. The row is written once and not once per team, because every caller that
authenticates with the master key is identified as the same constant user id, but it still costs
one of the five seats. | `bool` | `true` | no | +| [helm\_timeout](#input\_helm\_timeout) | Seconds to wait for the Helm release to become ready. The Prisma migration Job runs first and takes part of this budget. | `number` | `600` | no | +| [master\_key](#input\_master\_key) | Master key of the gateway. It must start with 'sk-', because LiteLLM rejects a key without that prefix. | `string` | n/a | yes | +| [model\_backend\_api\_keys](#input\_model\_backend\_api\_keys) | API key per model alias, keyed exactly like model\_backends. Several aliases that share one upstream endpoint repeat the same value. | `map(string)` | n/a | yes | +| [model\_backends](#input\_model\_backends) | Models the gateway exposes, keyed by the alias callers ask for in the `model` field of a request.

- `model`: name of the model at the upstream provider. The module prefixes it with `openai/`,
which is what selects the OpenAI-compatible driver.
- `api_base`: base URL of the upstream OpenAI-compatible endpoint, including the `/v1` suffix.

Pass the credential for each alias in `model_backend_api_keys` under the same key. | 
map(object({
    model    = string
    api_base = string
  }))
| n/a | yes | +| [namespace](#input\_namespace) | Namespace the gateway runs in. The module creates it. | `string` | `"litellm"` | no | +| [oidc](#input\_oidc) | OIDC identity provider the platform engineers log in to the admin console through. Null leaves
the console without a login path, which is the correct setting for a gateway nobody administers
through the browser.

Native SSO is free in the open-source proxy for up to five users, and it needs no Enterprise
licence below that.

- `issuer_url`: discovery base URL of the provider, for example
`https://idp.example.com/realms/ai`. The module reads
`/.well-known/openid-configuration` and takes the three endpoints from it, because
the proxy wants them spelled out and does no discovery of its own.
- `client_id` and `client_secret`: credentials of the OIDC client.
- `scopes`: space-separated scope list.
- `authorization_endpoint`, `token_endpoint`, `userinfo_endpoint`: override one endpoint each and
skip discovery for it. Set all three when the provider is unreachable from the Terraform
runner, and the module then creates no discovery request at all.
- `user_id_attribute`: claim the proxy stores as the user id. It defaults to `sub` here, not to
the proxy's own default of `preferred_username`, because `preferred_username` is reassignable
at most providers and a reassignment produces a second row in the user table for the same
person. Every row counts against the limit of five.
- `user_email_attribute`, `user_display_name_attribute`, `user_role_attribute`: the rest of the
claim mapping. Null leaves the proxy on its own defaults, which are `email`, `sub` and `role`.
- `allowed_email_domains`: only users whose email address carries one of these domains may log
in. The proxy compares the part after the `@` exactly, so there is no wildcard and no
subdomain match. Null lets every user the provider authenticates log in.
- `proxy_admin_id`: user id that is set to the `proxy_admin` role on every login. It is compared
against the value of the `user_id_attribute` claim, so it is that claim's value and not an
email address unless the claim carries one.
- `logout_url`: URL the console sends the browser to after a logout.
- `auto_redirect_to_sso`: send the login page straight to the provider instead of showing a
button.

Register the callback URL `/sso/callback` at the provider. The module sets no
`SERVER_ROOT_PATH`, which is the only setting that would move the callback to another path.

**The console holds at most five users.** Read the free user limit section of the module README
before you hand the console to a sixth person: the sixth login locks out everyone. | 
object({
    issuer_url    = string
    client_id     = string
    client_secret = string
    scopes        = optional(string, "openid email profile")

    authorization_endpoint = optional(string)
    token_endpoint         = optional(string)
    userinfo_endpoint      = optional(string)

    user_id_attribute           = optional(string, "sub")
    user_email_attribute        = optional(string)
    user_display_name_attribute = optional(string)
    user_role_attribute         = optional(string)

    allowed_email_domains = optional(list(string))
    proxy_admin_id        = optional(string)
    logout_url            = optional(string)
    auto_redirect_to_sso  = optional(bool, false)
  })
| `null` | no | +| [postgres\_connection\_limit](#input\_postgres\_connection\_limit) | Maximum number of Postgres connections one gateway pod opens. The module writes it as
`connection_limit` on the connection URL and as `general_settings.database_connection_pool_limit`
in the proxy config, because the proxy rewrites the URL on startup from that setting.

Without the parameter Prisma sizes the pool as `physical cores Γ— 2 + 1` read from the node, not
from the pod's CPU limit, so a pod takes 33 connections on a 16-core node and a different number
after it is rescheduled. The gateway is deployed once for the platform, so it costs
`replica_count Γ— postgres_connection_limit` connections in total.

The default of 10 is LiteLLM's own default, so pinning the value changes nothing at runtime and
only bounds what the URL asks for. | `number` | `10` | no | +| [postgres\_database](#input\_postgres\_database) | Name of the database on the Postgres server. It has to exist before the first apply; the Prisma migration Job creates the tables inside it, not the database itself. | `string` | `"litellm"` | no | +| [postgres\_host](#input\_postgres\_host) | Hostname of the Postgres server that holds virtual keys, teams, budgets and spend records. | `string` | n/a | yes | +| [postgres\_password](#input\_postgres\_password) | Password of the Postgres user. Use only characters that are safe in a URL, because the chart substitutes the value into the connection URL without encoding it. | `string` | n/a | yes | +| [postgres\_port](#input\_postgres\_port) | Port of the Postgres server. | `number` | `5432` | no | +| [postgres\_ssl\_mode](#input\_postgres\_ssl\_mode) | Value of the sslmode parameter on the Postgres connection URL. One of 'disable', 'prefer', 'require', 'verify-ca' or 'verify-full'. | `string` | `"require"` | no | +| [postgres\_username](#input\_postgres\_username) | User the gateway connects as. It needs rights to create and alter tables, because the Prisma migration Job runs the schema migrations under this user. | `string` | n/a | yes | +| [public\_url](#input\_public\_url) | Canonical URL the gateway is reached at from outside the cluster, for example
`https://litellm.example.com`, without a trailing slash. The module writes it as
`PROXY_BASE_URL`, and the proxy builds the SSO callback as `/sso/callback`.

Required when `var.oidc` is set. The proxy falls back to the base URL of the incoming request,
which behind a TLS-terminating Ingress is the internal `http://` address of the pod, and the
provider then rejects the redirect URI. The module creates no Ingress, so this is the URL of
whatever Ingress or load balancer stands in front of the Service. | `string` | `null` | no | +| [redis\_host](#input\_redis\_host) | Hostname of an existing Redis instance the gateway coordinates through. Leave it null to run without Redis, which is only correct with a single replica. | `string` | `null` | no | +| [redis\_password](#input\_redis\_password) | Password of the Redis instance. Leave it null for a Redis without authentication. Only used when redis\_host is set. | `string` | `null` | no | +| [redis\_port](#input\_redis\_port) | Port of the Redis instance. Only used when redis\_host is set. | `number` | `6379` | no | +| [release\_name](#input\_release\_name) | Helm release name of the gateway. | `string` | `"litellm"` | no | +| [replica\_count](#input\_replica\_count) | Number of gateway pods. Set redis\_host as well when this is greater than 1. | `number` | `1` | no | +| [token](#input\_token) | Token of the service account this module runs as. It needs permission to create a namespace, secrets and the workloads of the Helm release. Leave it null and set `client_certificate` and `client_key` instead when the cluster hands out a certificate pair. | `string` | `null` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [api\_base](#output\_api\_base) | In-cluster base URL of the gateway, including the '/v1' suffix. Callers send the master key or a virtual key as a bearer token. | +| [console\_url](#output\_console\_url) | URL of the admin console. Null when var.public\_url is not set, because the console is then reachable in-cluster only. | +| [master\_key\_secret\_name](#output\_master\_key\_secret\_name) | Name of the secret in the gateway namespace that holds the master key under the 'masterkey' key. | +| [model\_aliases](#output\_model\_aliases) | Model aliases the gateway exposes. A caller puts one of them in the 'model' field of a request. | +| [namespace](#output\_namespace) | Namespace the gateway runs in. | +| [oidc\_callback\_url](#output\_oidc\_callback\_url) | Callback URL to register at the identity provider. Null when var.oidc is not set. | +| [service\_name](#output\_service\_name) | Name of the Service in front of the gateway pods. | +| [service\_port](#output\_service\_port) | Port the Service in front of the gateway pods listens on. An Ingress backend needs it. | + diff --git a/modules/ai/litellm/buildingblock/litellm.tftest.hcl b/modules/ai/litellm/buildingblock/litellm.tftest.hcl new file mode 100644 index 00000000..f3722177 --- /dev/null +++ b/modules/ai/litellm/buildingblock/litellm.tftest.hcl @@ -0,0 +1,346 @@ +variables { + cluster_endpoint = "api.cluster.example.com" + cluster_ca_certificate = "dGVzdC1jYQ==" + token = "test-token" + + master_key = "sk-test-master-key" + + postgres_host = "shared.postgresflex.eu01.onstackit.cloud" + postgres_username = "litellm" + postgres_password = "litellm-password" + + model_backends = { + "chat-large" = { + model = "neuralmagic/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic" + api_base = "https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1" + } + } + + model_backend_api_keys = { + "chat-large" = "upstream-token" + } +} + +mock_provider "kubernetes" {} +mock_provider "helm" {} + +run "pins_the_prisma_pool_on_the_connection_url" { + command = plan + + assert { + condition = yamldecode(helm_release.litellm.values[0]).db.url == "postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST):5432/$(DATABASE_NAME)?sslmode=require&connection_limit=10" + # The whole URL is asserted rather than only the parameter, because the credentials have to + # stay '$(…)' references and the port has to stay in the host. + error_message = "the rendered db.url must carry the credential references, the port, sslmode and the pinned connection_limit" + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).proxy_config.general_settings.database_connection_pool_limit == 10 + error_message = "the proxy config must pin the same pool size, because the proxy rewrites connection_limit on the URL from this setting" + } +} + +run "a_lowered_limit_reaches_both_places" { + command = plan + + variables { + postgres_connection_limit = 3 + } + + assert { + condition = strcontains(yamldecode(helm_release.litellm.values[0]).db.url, "?sslmode=require&connection_limit=3") + error_message = "postgres_connection_limit must reach the connection URL" + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).proxy_config.general_settings.database_connection_pool_limit == 3 + error_message = "postgres_connection_limit must reach general_settings.database_connection_pool_limit" + } +} + +run "the_ssl_mode_stays_the_first_parameter" { + command = plan + + variables { + postgres_ssl_mode = "disable" + postgres_port = 15432 + } + + assert { + condition = strcontains(yamldecode(helm_release.litellm.values[0]).db.url, ":15432/$(DATABASE_NAME)?sslmode=disable&connection_limit=10") + error_message = "the port, the sslmode and the connection limit must all reach the URL in that order" + } +} + +run "rejects_a_connection_limit_below_one" { + command = plan + + variables { + postgres_connection_limit = 0 + } + + expect_failures = [var.postgres_connection_limit] +} + +run "keeps_the_user_table_empty_by_default" { + command = plan + + assert { + # This is the switch that holds the console inside the free limit of five users. Without it the + # first /team/new call writes one row to LiteLLM_UserTable and takes one of the five seats. + condition = yamldecode(helm_release.litellm.values[0]).proxy_config.general_settings.disable_auto_add_proxy_admin_to_teams == true + error_message = "the proxy config must carry general_settings.disable_auto_add_proxy_admin_to_teams: true" + } +} + +run "the_user_table_switch_can_be_turned_off" { + command = plan + + variables { + disable_auto_add_proxy_admin_to_teams = false + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).proxy_config.general_settings.disable_auto_add_proxy_admin_to_teams == false + error_message = "the variable must reach general_settings.disable_auto_add_proxy_admin_to_teams unchanged" + } +} + +run "renders_no_sso_environment_without_an_identity_provider" { + command = plan + + assert { + condition = yamldecode(helm_release.litellm.values[0]).envVars == {} + error_message = "a gateway without var.oidc must carry no SSO environment variables at all" + } + + assert { + # The client secret secret is the only one added for SSO, so its absence is what proves the + # module created nothing for a caller who has no identity provider. + condition = length(yamldecode(helm_release.litellm.values[0]).environmentSecrets) == 1 + error_message = "without var.oidc the only environment secret must be the model credentials one" + } +} + +run "renders_the_sso_environment_from_explicit_endpoints" { + command = plan + + variables { + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + + authorization_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/auth" + token_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/token" + userinfo_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo" + + user_display_name_attribute = "name" + user_role_attribute = "litellm_role" + allowed_email_domains = ["example.com", "example.org"] + proxy_admin_id = "platform-team-lead" + logout_url = "https://litellm.example.com/ui" + auto_redirect_to_sso = true + } + } + + assert { + # All three endpoints given, so the module must create no discovery request. + condition = length(data.http.oidc_discovery) == 0 + error_message = "three explicit endpoints must skip OIDC discovery entirely" + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).envVars == { + GENERIC_CLIENT_ID = "litellm-console" + GENERIC_AUTHORIZATION_ENDPOINT = "https://idp.example.com/realms/ai/protocol/openid-connect/auth" + GENERIC_TOKEN_ENDPOINT = "https://idp.example.com/realms/ai/protocol/openid-connect/token" + GENERIC_USERINFO_ENDPOINT = "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo" + GENERIC_SCOPE = "openid email profile" + + GENERIC_USER_ID_ATTRIBUTE = "sub" + GENERIC_USER_DISPLAY_NAME_ATTRIBUTE = "name" + GENERIC_USER_ROLE_ATTRIBUTE = "litellm_role" + + PROXY_BASE_URL = "https://litellm.example.com" + PROXY_LOGOUT_URL = "https://litellm.example.com/ui" + PROXY_ADMIN_ID = "platform-team-lead" + ALLOWED_EMAIL_DOMAINS = "example.com,example.org" + AUTO_REDIRECT_UI_LOGIN_TO_SSO = "true" + } + # The whole map is asserted rather than single keys, so a variable the module renders by + # accident β€” the client secret above all β€” fails the test. + error_message = "the rendered envVars must carry exactly the SSO environment variables the caller configured" + } + + assert { + condition = contains(yamldecode(helm_release.litellm.values[0]).environmentSecrets, "litellm-oidc") + error_message = "the client secret must reach the pods through an environmentSecrets entry" + } +} + +run "pins_the_user_id_claim_and_leaves_the_rest_to_the_proxy" { + command = plan + + variables { + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + + authorization_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/auth" + token_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/token" + userinfo_endpoint = "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo" + } + } + + assert { + # The proxy's own default is 'preferred_username', which is reassignable and would produce a + # second user row for the same person. The module pins 'sub' instead. + condition = yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_USER_ID_ATTRIBUTE == "sub" + error_message = "the module must pin GENERIC_USER_ID_ATTRIBUTE to 'sub' rather than leave the proxy on 'preferred_username'" + } + + assert { + # The proxy defaults the email claim to 'email', so an unset attribute has to stay out of the + # pod spec instead of arriving as an empty string. + condition = length(setintersection(keys(yamldecode(helm_release.litellm.values[0]).envVars), [ + "GENERIC_USER_EMAIL_ATTRIBUTE", + "GENERIC_USER_DISPLAY_NAME_ATTRIBUTE", + "GENERIC_USER_ROLE_ATTRIBUTE", + "PROXY_LOGOUT_URL", + "PROXY_ADMIN_ID", + "ALLOWED_EMAIL_DOMAINS", + "AUTO_REDIRECT_UI_LOGIN_TO_SSO", + ])) == 0 + error_message = "an unset optional SSO setting must not be rendered at all" + } +} + +run "derives_the_endpoints_from_the_discovery_document" { + command = plan + + variables { + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + } + } + + # The values have to be literals, so the discovery document is written out as JSON text. + override_data { + target = data.http.oidc_discovery + values = { + status_code = 200 + response_body = <<-EOT + { + "issuer": "https://idp.example.com/realms/ai", + "authorization_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/auth", + "token_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/token", + "userinfo_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo" + } + EOT + } + } + + assert { + condition = data.http.oidc_discovery[0].url == "https://idp.example.com/realms/ai/.well-known/openid-configuration" + error_message = "the module must read the discovery document at '/.well-known/openid-configuration'" + } + + assert { + condition = alltrue([ + yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_AUTHORIZATION_ENDPOINT == "https://idp.example.com/realms/ai/protocol/openid-connect/auth", + yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_TOKEN_ENDPOINT == "https://idp.example.com/realms/ai/protocol/openid-connect/token", + yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_USERINFO_ENDPOINT == "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo", + ]) + error_message = "the three endpoints must come out of the discovery document" + } +} + +run "an_endpoint_override_wins_over_the_discovery_document" { + command = plan + + variables { + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + + token_endpoint = "https://token.example.com/oauth2/token" + } + } + + override_data { + target = data.http.oidc_discovery + values = { + status_code = 200 + response_body = <<-EOT + { + "authorization_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/auth", + "token_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/token", + "userinfo_endpoint": "https://idp.example.com/realms/ai/protocol/openid-connect/userinfo" + } + EOT + } + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_TOKEN_ENDPOINT == "https://token.example.com/oauth2/token" + error_message = "an endpoint set in var.oidc must win over the one in the discovery document" + } + + assert { + condition = yamldecode(helm_release.litellm.values[0]).envVars.GENERIC_AUTHORIZATION_ENDPOINT == "https://idp.example.com/realms/ai/protocol/openid-connect/auth" + error_message = "the endpoints the caller left unset must still come out of the discovery document" + } +} + +run "rejects_an_identity_provider_without_a_public_url" { + command = plan + + variables { + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + } + } + + expect_failures = [var.oidc] +} + +run "rejects_an_issuer_that_is_not_an_https_url" { + command = plan + + variables { + public_url = "https://litellm.example.com" + + oidc = { + issuer_url = "idp.example.com/realms/ai" + client_id = "litellm-console" + client_secret = "oidc-client-secret" + } + } + + expect_failures = [var.oidc] +} + +run "rejects_a_public_url_with_a_trailing_slash" { + command = plan + + variables { + public_url = "https://litellm.example.com/" + } + + expect_failures = [var.public_url] +} diff --git a/modules/ai/litellm/buildingblock/logo.png b/modules/ai/litellm/buildingblock/logo.png new file mode 100644 index 00000000..5b8b3f90 Binary files /dev/null and b/modules/ai/litellm/buildingblock/logo.png differ diff --git a/modules/ai/litellm/buildingblock/main.tf b/modules/ai/litellm/buildingblock/main.tf new file mode 100644 index 00000000..130a6cf6 --- /dev/null +++ b/modules/ai/litellm/buildingblock/main.tf @@ -0,0 +1,327 @@ +locals { + # The chart names the Service after the release: '-litellm', or just '' when + # the release name already contains the chart's nameOverride, which is 'litellm'. + service_name = strcontains(var.release_name, "litellm") ? var.release_name : "${var.release_name}-litellm" + + # Pinned here rather than left to the chart default, so the api_base output cannot drift away + # from the port the Service actually listens on. + service_port = 4000 + + # Every credential reaches the pods as an environment variable, and the proxy config refers to it + # as os.environ/. Characters an environment variable name cannot carry collapse into an + # underscore; variables.tf rejects two aliases that would collapse to the same name. + api_key_env_names = { + for alias in keys(var.model_backends) : + alias => "LITELLM_API_KEY_${upper(replace(alias, "/[^a-zA-Z0-9]/", "_"))}" + } + + # Prisma sizes its connection pool as `num_physical_cpus * 2 + 1` when the connection URL carries + # no connection_limit, and it counts the physical cores of the node instead of the pod's CPU + # limit. prisma-engines#4341 records that as an oversight and was closed without a fix, so a pod + # with a 100m limit takes 33 connections on a 16-core node and a different number after it is + # rescheduled. A managed Postgres caps max_connections by its flavour, so the pool is pinned here. + postgres_url_query = join("&", [ + "sslmode=${var.postgres_ssl_mode}", + "connection_limit=${var.postgres_connection_limit}", + ]) + + redis_enabled = var.redis_host != null + + # Whether a Redis password was given is a plain fact, while the password itself is a secret. The + # fact has to stay unmarked, because everything derived from a marked value carries the mark, and + # the chart values would then be hidden in full from every plan. nonsensitive() rejects an + # argument that carries no mark, so try() falls back to the bare comparison. + redis_password_set = local.redis_enabled ? try(nonsensitive(var.redis_password != null), var.redis_password != null) : false + + redis_env = local.redis_enabled ? merge( + { + REDIS_HOST = var.redis_host + REDIS_PORT = tostring(var.redis_port) + }, + local.redis_password_set ? { REDIS_PASSWORD = var.redis_password } : {} + ) : {} + + # The proxy reads its coordination store from general_settings.coordination_redis. The chart only + # renders that block for its own bundled Redis, so an external Redis is written out here. A block + # supplied in proxy_config always wins over the chart's own. + coordination_redis = local.redis_enabled ? { + coordination_redis = merge( + { + host = "os.environ/REDIS_HOST" + port = "os.environ/REDIS_PORT" + }, + local.redis_password_set ? { password = "os.environ/REDIS_PASSWORD" } : {} + ) + } : {} + + # var.oidc is sensitive as a whole, so every expression derived from it carries the sensitivity + # mark, and a values map that carries the mark hides the whole Helm release from every plan. + # Unmark the plain facts β€” is SSO on, which issuer, which client, which endpoints β€” while the + # client secret keeps its mark and reaches the pods through a secret. nonsensitive() rejects an + # argument that carries no mark, so try() falls back to the bare value. + oidc_enabled = try(nonsensitive(var.oidc != null), var.oidc != null) + oidc = local.oidc_enabled ? { + issuer_url = try(nonsensitive(var.oidc.issuer_url), var.oidc.issuer_url) + client_id = try(nonsensitive(var.oidc.client_id), var.oidc.client_id) + scopes = try(nonsensitive(var.oidc.scopes), var.oidc.scopes) + authorization_endpoint = try(nonsensitive(var.oidc.authorization_endpoint), var.oidc.authorization_endpoint) + token_endpoint = try(nonsensitive(var.oidc.token_endpoint), var.oidc.token_endpoint) + userinfo_endpoint = try(nonsensitive(var.oidc.userinfo_endpoint), var.oidc.userinfo_endpoint) + user_id_attribute = try(nonsensitive(var.oidc.user_id_attribute), var.oidc.user_id_attribute) + user_email_attribute = try(nonsensitive(var.oidc.user_email_attribute), var.oidc.user_email_attribute) + user_display_name_attribute = try(nonsensitive(var.oidc.user_display_name_attribute), var.oidc.user_display_name_attribute) + user_role_attribute = try(nonsensitive(var.oidc.user_role_attribute), var.oidc.user_role_attribute) + allowed_email_domains = try(nonsensitive(var.oidc.allowed_email_domains), var.oidc.allowed_email_domains) + proxy_admin_id = try(nonsensitive(var.oidc.proxy_admin_id), var.oidc.proxy_admin_id) + logout_url = try(nonsensitive(var.oidc.logout_url), var.oidc.logout_url) + auto_redirect_to_sso = try(nonsensitive(var.oidc.auto_redirect_to_sso), var.oidc.auto_redirect_to_sso) + } : null + + # The proxy reads the three endpoints as separate environment variables and performs no discovery + # of its own, while the caller supplies an issuer. The gap is closed here, with the discovery + # document read at plan time. A caller who overrides all three endpoints skips the request + # entirely, which is the way out when the provider is unreachable from the Terraform runner. + oidc_endpoint_overrides = local.oidc_enabled ? { + authorization_endpoint = local.oidc.authorization_endpoint + token_endpoint = local.oidc.token_endpoint + userinfo_endpoint = local.oidc.userinfo_endpoint + } : {} + + oidc_discovery_needed = local.oidc_enabled && anytrue([ + for endpoint in values(local.oidc_endpoint_overrides) : endpoint == null + ]) + + oidc_discovery = local.oidc_discovery_needed ? jsondecode(data.http.oidc_discovery[0].response_body) : {} + + oidc_endpoints = { + for name, override in local.oidc_endpoint_overrides : + name => coalesce(override, try(local.oidc_discovery[name], null)) + } + + # Every value here is a plain string in the pod spec, so the client secret is not among them. + # A null entry is dropped, which leaves the proxy on its own default for that setting. + oidc_env = local.oidc_enabled ? { + for name, value in { + GENERIC_CLIENT_ID = local.oidc.client_id + GENERIC_AUTHORIZATION_ENDPOINT = local.oidc_endpoints.authorization_endpoint + GENERIC_TOKEN_ENDPOINT = local.oidc_endpoints.token_endpoint + GENERIC_USERINFO_ENDPOINT = local.oidc_endpoints.userinfo_endpoint + GENERIC_SCOPE = local.oidc.scopes + + GENERIC_USER_ID_ATTRIBUTE = local.oidc.user_id_attribute + GENERIC_USER_EMAIL_ATTRIBUTE = local.oidc.user_email_attribute + GENERIC_USER_DISPLAY_NAME_ATTRIBUTE = local.oidc.user_display_name_attribute + GENERIC_USER_ROLE_ATTRIBUTE = local.oidc.user_role_attribute + + PROXY_BASE_URL = var.public_url + PROXY_LOGOUT_URL = local.oidc.logout_url + PROXY_ADMIN_ID = local.oidc.proxy_admin_id + ALLOWED_EMAIL_DOMAINS = local.oidc.allowed_email_domains == null ? null : join(",", local.oidc.allowed_email_domains) + AUTO_REDIRECT_UI_LOGIN_TO_SSO = local.oidc.auto_redirect_to_sso ? "true" : null + } : name => value if value != null + } : {} + + chart_values = { + replicaCount = var.replica_count + + service = { + type = "ClusterIP" + port = local.service_port + } + + # Without this the chart generates a master key of its own on every install, which no caller + # knows and which changes whenever the secret is recreated. + masterkeySecretName = kubernetes_secret_v1.master_key.metadata[0].name + masterkeySecretKey = "masterkey" + + # Exports the secrets into the pods as environment variables, which is what the + # os.environ/ references in proxy_config resolve against. + environmentSecrets = concat( + [kubernetes_secret_v1.model_credentials.metadata[0].name], + [for secret in kubernetes_secret_v1.oidc : secret.metadata[0].name], + ) + + # The chart renders these as plain `env` entries on the container. The proxy reads its SSO + # settings from the environment and not from the proxy config, so they belong here rather than + # under proxy_config. + envVars = local.oidc_env + + db = { + # The chart bundles a Bitnami postgresql subchart and turns it on by default. Those images + # no longer receive updates and the subchart pins bitnamilegacy/postgresql, so the database + # comes from outside the chart. + deployStandalone = false + useExisting = true + + endpoint = var.postgres_host + database = var.postgres_database + + # Kubernetes substitutes $(VAR) from the environment variables declared before this one in + # the same container, so the credentials stay in the secret and never appear in the pod spec. + # The chart's default URL carries no port, hence the override. + url = "postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST):${var.postgres_port}/$(DATABASE_NAME)?${local.postgres_url_query}" + + secret = { + name = kubernetes_secret_v1.postgres.metadata[0].name + usernameKey = "username" + passwordKey = "password" + } + } + + # The bundled Redis subchart carries the same retired Bitnami images as the Postgres one, so it + # stays off. An external Redis arrives through REDIS_HOST, REDIS_PORT and REDIS_PASSWORD. + redis = { + enabled = false + } + + migrationJob = { + enabled = true + hooks = { + # As a Helm pre-install and pre-upgrade hook the Job runs to completion before the + # Deployment is created, so the pods never start against a database without the schema. + # The chart's default instead annotates the Job for ArgoCD, which means nothing here. + helm = { enabled = true } + argocd = { enabled = false } + } + } + + proxy_config = { + model_list = [ + for alias, backend in var.model_backends : { + model_name = alias + litellm_params = { + # The 'openai/' prefix selects the OpenAI-compatible driver. + model = "openai/${backend.model}" + # api_base carries the '/v1' suffix; variables.tf rejects an endpoint without it. + api_base = backend.api_base + api_key = "os.environ/${local.api_key_env_names[alias]}" + } + } + ] + + general_settings = merge( + { + master_key = "os.environ/PROXY_MASTER_KEY" + + # The proxy rewrites DATABASE_URL on startup and replaces connection_limit with this + # setting, so the parameter on db.url alone does not bind the running pods. Both carry + # the same number, and the pool is then the same on whichever path sets it. + database_connection_pool_limit = var.postgres_connection_limit + + # Keeps LiteLLM_UserTable empty, and the console therefore inside the free limit of five + # users. Without it the first /team/new call writes one row and takes one of the five. + disable_auto_add_proxy_admin_to_teams = var.disable_auto_add_proxy_admin_to_teams + }, + local.coordination_redis + ) + } + } +} + +# The identity provider publishes its authorization, token and userinfo endpoints in this document, +# and the proxy wants all three named individually. Reading it here keeps var.oidc down to an +# issuer and holds the interface identical to modules/ai/langfuse, which discovers the same +# document itself. The request runs on every plan, so the provider has to answer the Terraform +# runner; the three endpoint overrides in var.oidc are the way around that. +data "http" "oidc_discovery" { + count = local.oidc_discovery_needed ? 1 : 0 + + url = "${trimsuffix(local.oidc.issuer_url, "/")}/.well-known/openid-configuration" + + request_headers = { + Accept = "application/json" + } + + lifecycle { + postcondition { + condition = self.status_code == 200 + error_message = "The OIDC discovery document did not answer with 200. Check oidc.issuer_url, or set authorization_endpoint, token_endpoint and userinfo_endpoint in var.oidc to skip discovery." + } + + postcondition { + condition = alltrue([ + for field in ["authorization_endpoint", "token_endpoint", "userinfo_endpoint"] : + try(jsondecode(self.response_body)[field], null) != null + ]) + error_message = "The OIDC discovery document names no authorization_endpoint, token_endpoint or userinfo_endpoint. Set the missing ones in var.oidc." + } + } +} + +resource "kubernetes_namespace_v1" "this" { + metadata { + name = var.namespace + } +} + +resource "kubernetes_secret_v1" "master_key" { + metadata { + name = "${var.release_name}-master-key" + namespace = kubernetes_namespace_v1.this.metadata[0].name + } + + data = { + masterkey = var.master_key + } +} + +resource "kubernetes_secret_v1" "postgres" { + metadata { + name = "${var.release_name}-postgres" + namespace = kubernetes_namespace_v1.this.metadata[0].name + } + + data = { + username = var.postgres_username + password = var.postgres_password + } +} + +resource "kubernetes_secret_v1" "model_credentials" { + metadata { + name = "${var.release_name}-model-credentials" + namespace = kubernetes_namespace_v1.this.metadata[0].name + } + + data = merge( + { + for alias, env_name in local.api_key_env_names : + env_name => var.model_backend_api_keys[alias] + }, + local.redis_env + ) +} + +# The client secret is the only sensitive part of the SSO configuration, so it travels the same way +# as the model credentials: a secret listed under environmentSecrets, which the chart exports into +# the pods with envFrom. Everything else about the provider stays a plain value in the pod spec. +resource "kubernetes_secret_v1" "oidc" { + count = local.oidc_enabled ? 1 : 0 + + metadata { + name = "${var.release_name}-oidc" + namespace = kubernetes_namespace_v1.this.metadata[0].name + } + + data = { + GENERIC_CLIENT_SECRET = var.oidc.client_secret + } +} + +# The chart is published to GHCR as an OCI artifact and has no classic Helm repository. The helm +# provider takes the registry and the path prefix as the repository, and the chart name on its own +# as the chart, so the reference resolves to oci://ghcr.io/berriai/litellm-helm. +resource "helm_release" "litellm" { + name = var.release_name + namespace = kubernetes_namespace_v1.this.metadata[0].name + repository = "oci://ghcr.io/berriai" + chart = "litellm-helm" + version = var.chart_version + + create_namespace = false + atomic = true + wait = true + timeout = var.helm_timeout + + values = [yamlencode(local.chart_values)] +} diff --git a/modules/ai/litellm/buildingblock/outputs.tf b/modules/ai/litellm/buildingblock/outputs.tf new file mode 100644 index 00000000..079e7fed --- /dev/null +++ b/modules/ai/litellm/buildingblock/outputs.tf @@ -0,0 +1,47 @@ +output "namespace" { + description = "Namespace the gateway runs in." + value = kubernetes_namespace_v1.this.metadata[0].name +} + +output "service_name" { + description = "Name of the Service in front of the gateway pods." + value = local.service_name + + depends_on = [helm_release.litellm] +} + +output "api_base" { + # The '/v1' suffix belongs to the base URL. A client that drops it gets a 'Not Found' error, so + # this output carries the suffix instead of leaving it to the caller. + description = "In-cluster base URL of the gateway, including the '/v1' suffix. Callers send the master key or a virtual key as a bearer token." + value = "http://${local.service_name}.${kubernetes_namespace_v1.this.metadata[0].name}.svc.cluster.local:${local.service_port}/v1" + + depends_on = [helm_release.litellm] +} + +output "model_aliases" { + description = "Model aliases the gateway exposes. A caller puts one of them in the 'model' field of a request." + value = sort(keys(var.model_backends)) +} + +output "console_url" { + description = "URL of the admin console. Null when var.public_url is not set, because the console is then reachable in-cluster only." + value = var.public_url == null ? null : "${var.public_url}/ui" +} + +output "oidc_callback_url" { + description = "Callback URL to register at the identity provider. Null when var.oidc is not set." + value = local.oidc_enabled ? "${var.public_url}/sso/callback" : null +} + +output "master_key_secret_name" { + description = "Name of the secret in the gateway namespace that holds the master key under the 'masterkey' key." + value = kubernetes_secret_v1.master_key.metadata[0].name +} + +output "service_port" { + # The chart fixes the port and the module does not take it as an input, so a caller that puts an + # Ingress in front of the Service reads it here instead of repeating a number it does not own. + description = "Port the Service in front of the gateway pods listens on. An Ingress backend needs it." + value = local.service_port +} diff --git a/modules/ai/litellm/buildingblock/provider.tf b/modules/ai/litellm/buildingblock/provider.tf new file mode 100644 index 00000000..0836010b --- /dev/null +++ b/modules/ai/litellm/buildingblock/provider.tf @@ -0,0 +1,24 @@ +provider "kubernetes" { + host = "https://${var.cluster_endpoint}" + cluster_ca_certificate = base64decode(var.cluster_ca_certificate) + + # A service account token is the usual credential. The certificate pair is accepted as well, so a + # kubeconfig produced by a managed cluster's own credential API also works. Exactly one of the two + # is set; the unused attributes stay null and the provider ignores them. + token = var.token + client_certificate = var.client_certificate + client_key = var.client_key +} + +# The helm provider talks to the same control plane with the same credentials, so the namespace, +# the secrets and the Helm release all land in one cluster without extra wiring. +provider "helm" { + kubernetes = { + host = "https://${var.cluster_endpoint}" + cluster_ca_certificate = base64decode(var.cluster_ca_certificate) + + token = var.token + client_certificate = var.client_certificate + client_key = var.client_key + } +} diff --git a/modules/ai/litellm/buildingblock/variables.tf b/modules/ai/litellm/buildingblock/variables.tf new file mode 100644 index 00000000..dc892a92 --- /dev/null +++ b/modules/ai/litellm/buildingblock/variables.tf @@ -0,0 +1,365 @@ +variable "cluster_endpoint" { + type = string + description = "IP address or hostname of the cluster control plane, without the https:// scheme." +} + +variable "cluster_ca_certificate" { + type = string + description = "Cluster CA certificate, base64 encoded." +} + +# The cluster credential is either a service account token or a client certificate pair. A +# composition that installs the gateway on a cluster it just created usually holds the certificate +# pair, because that is what a managed cluster's credential API returns β€” STACKIT SKE among them. +# `modules/ai/model-access` accepts both for the same reason. +variable "token" { + type = string + sensitive = true + default = null + description = "Token of the service account this module runs as. It needs permission to create a namespace, secrets and the workloads of the Helm release. Leave it null and set `client_certificate` and `client_key` instead when the cluster hands out a certificate pair." + + # The message carries no interpolation, because both values are sensitive and Terraform refuses + # to print a sensitive value in an error message. + validation { + condition = (var.token != null) != (var.client_certificate != null) + error_message = "Set either token or client_certificate, not both and not neither. Without a credential every call to the API server is anonymous and the install fails on the first namespace." + } +} + +variable "client_certificate" { + type = string + sensitive = true + default = null + description = "PEM-encoded client certificate this module authenticates with, as an alternative to `token`. Pass the decoded certificate, not the base64 blob a kubeconfig carries." +} + +variable "client_key" { + type = string + sensitive = true + default = null + description = "PEM-encoded private key belonging to `client_certificate`. Pass the decoded key, not the base64 blob a kubeconfig carries." + + validation { + condition = (var.client_certificate == null) == (var.client_key == null) + error_message = "client_certificate and client_key belong together. Set both or neither." + } +} + +variable "namespace" { + type = string + default = "litellm" + description = "Namespace the gateway runs in. The module creates it." +} + +variable "release_name" { + type = string + default = "litellm" + # The chart derives the Service name from the release name: '-litellm', or just + # '' when the release name already contains 'litellm'. + description = "Helm release name of the gateway." +} + +variable "chart_version" { + type = string + default = "1.96.2" + # The chart is published to GHCR as an OCI artifact only, so the tag on the registry is the + # single source of truth. Chart.yaml on the repository's main branch reads a different, much + # lower version, because the release pipeline overwrites it while publishing. + description = "Version of the litellm-helm chart. See https://github.com/BerriAI/litellm/pkgs/container/litellm-helm." +} + +variable "replica_count" { + type = number + default = 1 + # Anything above 1 needs Redis, otherwise each pod counts rate limits and spend on its own. + description = "Number of gateway pods. Set redis_host as well when this is greater than 1." +} + +variable "helm_timeout" { + type = number + default = 600 + description = "Seconds to wait for the Helm release to become ready. The Prisma migration Job runs first and takes part of this budget." +} + +# --- Admin console ---------------------------------------------------------------------------- + +variable "public_url" { + type = string + default = null + # The proxy reads it as PROXY_BASE_URL and builds the SSO callback from it. Its fallback is the + # base URL of the incoming request, which behind a TLS-terminating Ingress is the internal + # http:// address, so the redirect URI never matches what the provider has registered. + description = <<-EOT + Canonical URL the gateway is reached at from outside the cluster, for example + `https://litellm.example.com`, without a trailing slash. The module writes it as + `PROXY_BASE_URL`, and the proxy builds the SSO callback as `/sso/callback`. + + Required when `var.oidc` is set. The proxy falls back to the base URL of the incoming request, + which behind a TLS-terminating Ingress is the internal `http://` address of the pod, and the + provider then rejects the redirect URI. The module creates no Ingress, so this is the URL of + whatever Ingress or load balancer stands in front of the Service. + EOT + + validation { + condition = var.public_url == null || startswith(coalesce(var.public_url, ""), "https://") + error_message = "public_url must be an https URL, because the identity provider redirects the browser back to it." + } + + validation { + condition = var.public_url == null || !endswith(coalesce(var.public_url, ""), "/") + error_message = "public_url must not end with a slash. The proxy appends '/sso/callback' to it." + } +} + +variable "oidc" { + description = <<-EOT + OIDC identity provider the platform engineers log in to the admin console through. Null leaves + the console without a login path, which is the correct setting for a gateway nobody administers + through the browser. + + Native SSO is free in the open-source proxy for up to five users, and it needs no Enterprise + licence below that. + + - `issuer_url`: discovery base URL of the provider, for example + `https://idp.example.com/realms/ai`. The module reads + `/.well-known/openid-configuration` and takes the three endpoints from it, because + the proxy wants them spelled out and does no discovery of its own. + - `client_id` and `client_secret`: credentials of the OIDC client. + - `scopes`: space-separated scope list. + - `authorization_endpoint`, `token_endpoint`, `userinfo_endpoint`: override one endpoint each and + skip discovery for it. Set all three when the provider is unreachable from the Terraform + runner, and the module then creates no discovery request at all. + - `user_id_attribute`: claim the proxy stores as the user id. It defaults to `sub` here, not to + the proxy's own default of `preferred_username`, because `preferred_username` is reassignable + at most providers and a reassignment produces a second row in the user table for the same + person. Every row counts against the limit of five. + - `user_email_attribute`, `user_display_name_attribute`, `user_role_attribute`: the rest of the + claim mapping. Null leaves the proxy on its own defaults, which are `email`, `sub` and `role`. + - `allowed_email_domains`: only users whose email address carries one of these domains may log + in. The proxy compares the part after the `@` exactly, so there is no wildcard and no + subdomain match. Null lets every user the provider authenticates log in. + - `proxy_admin_id`: user id that is set to the `proxy_admin` role on every login. It is compared + against the value of the `user_id_attribute` claim, so it is that claim's value and not an + email address unless the claim carries one. + - `logout_url`: URL the console sends the browser to after a logout. + - `auto_redirect_to_sso`: send the login page straight to the provider instead of showing a + button. + + Register the callback URL `/sso/callback` at the provider. The module sets no + `SERVER_ROOT_PATH`, which is the only setting that would move the callback to another path. + + **The console holds at most five users.** Read the free user limit section of the module README + before you hand the console to a sixth person: the sixth login locks out everyone. + EOT + + type = object({ + issuer_url = string + client_id = string + client_secret = string + scopes = optional(string, "openid email profile") + + authorization_endpoint = optional(string) + token_endpoint = optional(string) + userinfo_endpoint = optional(string) + + user_id_attribute = optional(string, "sub") + user_email_attribute = optional(string) + user_display_name_attribute = optional(string) + user_role_attribute = optional(string) + + allowed_email_domains = optional(list(string)) + proxy_admin_id = optional(string) + logout_url = optional(string) + auto_redirect_to_sso = optional(bool, false) + }) + + default = null + sensitive = true + + # The messages carry no interpolation, because var.oidc is sensitive and Terraform refuses to + # print a sensitive value in an error message. + validation { + condition = var.oidc == null || startswith(var.oidc.issuer_url, "https://") + error_message = "oidc.issuer_url must be an https URL. It is the discovery base URL, not the authorization endpoint." + } + + validation { + condition = var.oidc == null || var.public_url != null + error_message = "Set var.public_url together with var.oidc. The proxy builds the SSO callback URL from PROXY_BASE_URL, and login fails without it." + } +} + +variable "disable_auto_add_proxy_admin_to_teams" { + type = bool + default = true + # This is the switch that keeps LiteLLM_UserTable empty, which is what keeps the console inside + # the free limit of five users. See the free user limit section of the module README. + description = <<-EOT + Write `general_settings.disable_auto_add_proxy_admin_to_teams: true` into the proxy config, so + the proxy adds no admin member to a team it creates. + + Leave it at `true`. With it `false`, the first call to `/team/new` writes one row to + `LiteLLM_UserTable` and that row consumes one of the five console seats the free open-source + proxy allows. The row is written once and not once per team, because every caller that + authenticates with the master key is identified as the same constant user id, but it still costs + one of the five seats. + EOT +} + +variable "master_key" { + type = string + sensitive = true + # The proxy reads it as PROXY_MASTER_KEY and treats it as the root credential of the gateway: + # it authenticates every call to the /key and /team endpoints and works as a virtual key itself. + description = "Master key of the gateway. It must start with 'sk-', because LiteLLM rejects a key without that prefix." + + validation { + condition = startswith(var.master_key, "sk-") + error_message = "The master key must start with 'sk-'." + } +} + +variable "model_backends" { + type = map(object({ + model = string + api_base = string + })) + description = <<-EOT + Models the gateway exposes, keyed by the alias callers ask for in the `model` field of a request. + + - `model`: name of the model at the upstream provider. The module prefixes it with `openai/`, + which is what selects the OpenAI-compatible driver. + - `api_base`: base URL of the upstream OpenAI-compatible endpoint, including the `/v1` suffix. + + Pass the credential for each alias in `model_backend_api_keys` under the same key. + EOT + + validation { + condition = length(var.model_backends) > 0 + error_message = "Register at least one model backend. The gateway refuses to start with an empty model list." + } + + validation { + condition = alltrue([for backend in var.model_backends : endswith(backend.api_base, "/v1")]) + error_message = "Every api_base must end with '/v1'. LiteLLM appends the route to this URL, so an endpoint without the suffix answers 'Not Found'." + } + + validation { + # Each alias becomes an environment variable name, and everything outside [A-Za-z0-9] collapses + # into an underscore on the way. Two aliases that collapse to the same name would share one + # credential, so reject that pair here instead of routing a request with the wrong key. + condition = length(distinct([ + for alias in keys(var.model_backends) : upper(replace(alias, "/[^a-zA-Z0-9]/", "_")) + ])) == length(var.model_backends) + error_message = "Two aliases differ only in characters outside [A-Za-z0-9] and would map to the same environment variable name. Rename one of them." + } +} + +variable "model_backend_api_keys" { + type = map(string) + sensitive = true + # Kept out of var.model_backends so the backend map stays readable in plan output and in the + # meshStack UI, and so only the credentials carry the sensitivity mark. + description = "API key per model alias, keyed exactly like model_backends. Several aliases that share one upstream endpoint repeat the same value." + + validation { + condition = alltrue([ + for alias in keys(var.model_backends) : + contains(try(nonsensitive(keys(var.model_backend_api_keys)), keys(var.model_backend_api_keys)), alias) + ]) + error_message = "Every key in model_backends needs an entry with the same key in model_backend_api_keys." + } +} + +variable "postgres_host" { + type = string + # Virtual keys, teams, budgets and spend tracking all live in Postgres. Without a database the + # gateway is a stateless proxy and none of those endpoints work, so the database is required. + description = "Hostname of the Postgres server that holds virtual keys, teams, budgets and spend records." +} + +variable "postgres_port" { + type = number + default = 5432 + description = "Port of the Postgres server." +} + +variable "postgres_database" { + type = string + default = "litellm" + description = "Name of the database on the Postgres server. It has to exist before the first apply; the Prisma migration Job creates the tables inside it, not the database itself." +} + +variable "postgres_username" { + type = string + description = "User the gateway connects as. It needs rights to create and alter tables, because the Prisma migration Job runs the schema migrations under this user." +} + +variable "postgres_password" { + type = string + sensitive = true + # The chart builds the connection URL from $(DATABASE_USERNAME) and $(DATABASE_PASSWORD), which + # Kubernetes substitutes verbatim and does not URL-encode. A password containing ':', '@', '/' + # or '?' therefore breaks the URL. + description = "Password of the Postgres user. Use only characters that are safe in a URL, because the chart substitutes the value into the connection URL without encoding it." +} + +variable "postgres_ssl_mode" { + type = string + default = "require" + # Managed Postgres offerings terminate TLS, so requiring it is the safe default. A server + # without TLS needs 'prefer' or 'disable'. + description = "Value of the sslmode parameter on the Postgres connection URL. One of 'disable', 'prefer', 'require', 'verify-ca' or 'verify-full'." + + validation { + condition = contains(["disable", "prefer", "require", "verify-ca", "verify-full"], var.postgres_ssl_mode) + error_message = "postgres_ssl_mode must be one of 'disable', 'prefer', 'require', 'verify-ca' or 'verify-full'." + } +} + +variable "postgres_connection_limit" { + type = number + default = 10 + # Both the gateway and its migration Job talk to Postgres through Prisma, which sizes its pool + # from the node's physical cores when the connection URL names no limit. See the connection + # budget section in the module README. + description = <<-EOT + Maximum number of Postgres connections one gateway pod opens. The module writes it as + `connection_limit` on the connection URL and as `general_settings.database_connection_pool_limit` + in the proxy config, because the proxy rewrites the URL on startup from that setting. + + Without the parameter Prisma sizes the pool as `physical cores Γ— 2 + 1` read from the node, not + from the pod's CPU limit, so a pod takes 33 connections on a 16-core node and a different number + after it is rescheduled. The gateway is deployed once for the platform, so it costs + `replica_count Γ— postgres_connection_limit` connections in total. + + The default of 10 is LiteLLM's own default, so pinning the value changes nothing at runtime and + only bounds what the URL asks for. + EOT + + validation { + condition = var.postgres_connection_limit >= 1 + error_message = "postgres_connection_limit must be at least 1." + } +} + +variable "redis_host" { + type = string + default = null + # Redis is the coordination store of the gateway: cross-pod rate limits, spend tracking and the + # pod lock manager. One pod needs none of that, several pods do. + description = "Hostname of an existing Redis instance the gateway coordinates through. Leave it null to run without Redis, which is only correct with a single replica." +} + +variable "redis_port" { + type = number + default = 6379 + description = "Port of the Redis instance. Only used when redis_host is set." +} + +variable "redis_password" { + type = string + sensitive = true + default = null + description = "Password of the Redis instance. Leave it null for a Redis without authentication. Only used when redis_host is set." +} diff --git a/modules/ai/litellm/buildingblock/versions.tf b/modules/ai/litellm/buildingblock/versions.tf new file mode 100644 index 00000000..c1807e4b --- /dev/null +++ b/modules/ai/litellm/buildingblock/versions.tf @@ -0,0 +1,22 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + helm = { + source = "hashicorp/helm" + # The helm provider takes its cluster credentials as the `kubernetes = {}` attribute + # starting with 3.0.0. Earlier versions expect a `kubernetes {}` block instead. + version = ">= 3.0.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38" + } + # Only used to read the OIDC discovery document of the identity provider. The module creates + # no data source from it when var.oidc is null or when all three endpoints are given. + http = { + source = "hashicorp/http" + version = ">= 3.4" + } + } +} diff --git a/modules/ai/logo.svg b/modules/ai/logo.svg new file mode 100644 index 00000000..40a4e9e7 --- /dev/null +++ b/modules/ai/logo.svg @@ -0,0 +1 @@ + diff --git a/modules/ai/model-access/buildingblock/README.md b/modules/ai/model-access/buildingblock/README.md new file mode 100644 index 00000000..e0edbc14 --- /dev/null +++ b/modules/ai/model-access/buildingblock/README.md @@ -0,0 +1,406 @@ +--- +name: AI Model Access +supportedPlatforms: + - ai +description: Gives a project a governed OpenAI-compatible model endpoint with a budget, delivers the credential as a Kubernetes Secret in the namespace of the project, and deploys a tracing instance of its own for it together with the per-tenant database, bucket and ClickHouse database that instance needs. +# The module creates resources on STACKIT, but it receives every credential it needs for them as a +# static input: the two cluster kubeconfigs, the gateway admin key, the STACKIT service account key +# and the administrative object storage credential. Nothing has to be provisioned per tenant ahead of +# time, so there is no backplane. See "The STACKIT credential" below for what the platform team +# creates by hand instead, and why workload identity federation would need a backplane. +requiresBackplane: false +--- + +# AI Model Access Building Block + +This is the one building block an application team gets when its project lands in the AI landing +zone. The AI landing zone lists the building block definition in +`spec.mandatory_building_block_refs`, so meshStack provisions it when the tenant is created and the +application team fills in nothing. + +The display name is **AI Model Access**, named for the capability. LiteLLM and Langfuse stay behind +that name, so the platform team can replace either without renaming what the application team +ordered. + +## What one run does + +1. Creates the tenant's team on the shared LiteLLM gateway, with the budget and the model allow-list + of the landing zone, and one virtual key scoped to that team. +2. Creates the three per-tenant backend resources the tenant's Langfuse instance needs: a database + and its owner user on the shared Postgres instance, a database and a scoped user in the shared + ClickHouse cluster, and a bucket with a credential of its own. +3. Deploys the tenant's own Langfuse instance into the AI platform cluster, by sourcing + `modules/ai/langfuse/buildingblock`, wired to the three resources of step 2. +4. Looks up the sibling tenant of the same meshProject on the demo application cluster, to learn its + namespace. +5. Writes the virtual key and the endpoint into a Kubernetes Secret in that namespace, in the demo + application cluster. + +## One root module, and why that is the whole design + +A `buildingblock` directory is always a meshStack root module: the meshStack Terraform runner checks +it out and runs `tofu plan` and `tofu apply` inside it. Provider configuration therefore belongs +here, and this module configures seven providers β€” `litellm`, two `kubernetes` configurations with +aliases, `helm`, `meshstack`, `stackit` and `aws`. + +The four steps above are not split into separate building blocks, and they are not split into +submodules either. **A building block output cannot be sensitive.** `version_spec.outputs` of +`meshstack_building_block_definition` has no `sensitive` block, unlike `version_spec.inputs`, and an +output's `assignment_type` is limited to `NONE`, `PLATFORM_TENANT_ID`, `SIGN_IN_URL`, `RESOURCE_URL` +and `SUMMARY`. So a building block output is stored and displayed in cleartext in meshPanel. LiteLLM +returns a virtual key once, at creation, and never again. Keeping the mint and the delivery in one +Terraform run is what keeps the key inside that run: it is never an output of this module, and it +never crosses a building block boundary. + +## Two clusters, two provider aliases + +The tenant's Langfuse instance and the tenant's workload live in different clusters, so there are two +`kubernetes` provider configurations: + +| Alias | Cluster | What it does there | +|---|---|---| +| `kubernetes.ai_platform`, `helm.ai_platform` | AI platform cluster | Creates the namespace, the secret and the Helm release of the tenant's Langfuse instance. | +| `kubernetes.demo_app` | Demo application cluster | Writes one Secret into the namespace of the application team. | + +Both kubeconfigs arrive as `STATIC` encrypted inputs, in the `sensitive = { argument = { +secret_value, secret_version } }` shape. + +## The namespace comes from the sibling tenant + +The block runs on the AI model tenant, on the LiteLLM platform. The workload runs on a second tenant +of the same meshProject, a namespace on the demo application cluster. meshStack does not hand the +sibling tenant to a building block run, so the run asks for it: + +```hcl +data "meshstack_tenants" "sibling" { + workspace = var.workspace_identifier + project = var.project_identifier + platform = var.demo_app_platform_identifier +} +``` + +`spec.platform_tenant_id` of a Kubernetes tenant **is** the namespace name. The filter is on +`platform`, the full `.` identifier, and never on `platform_type`: two Kubernetes +clusters are two platforms of one type, so a type filter would match a tenant on the wrong cluster. + +The lookup needs an ephemeral API token, which `version_spec.permissions = ["TENANT_LIST"]` grants. +`modules/aks/github-connector` proves that a `TENANT_LEVEL` block can hold those permissions. + +Three preconditions on the Secret cover the three ways the lookup can go wrong: no match, more than +one match, and a match whose `platform_tenant_id` is still null because meshStack has not replicated +the tenant yet. The third one is the dangerous case. The Kubernetes provider falls back to the +namespace of the kubeconfig's current context when `metadata.namespace` is null, so a null id would +put the credential in a namespace that belongs to somebody else. + +`one()` is deliberately not used to read the single match. It raises an error of its own on a +collection with more than one element, and that error would replace the precondition message, which +names the platform and the project. + +## Residual risk of the demo cluster credential + +The credential for the demo application cluster is deliberately narrow: a ClusterRole permitting +`get`, `create`, `update`, `patch` and `delete` on `secrets`, and nothing else. `get` cannot be +dropped β€” without it the Terraform provider can neither plan nor detect drift. + +**Be honest about what remains.** A ClusterRole is not namespaced, so the credential can write a +Secret into **any** namespace of that cluster. The boundary is enforced by this module's code rather +than by the permissions, and `create` on secrets is a known privilege-escalation path: a Secret can +carry a service account token, and a controller that consumes Secrets can be steered by one. + +What mitigates it is that no tenant can influence which namespace is chosen. Every input that reaches +the namespace decision is either injected by meshStack from the tenant β€” `WORKSPACE_IDENTIFIER` and +`PROJECT_IDENTIFIER` β€” or a `STATIC` value the platform team fixed in the building block definition: +the platform identifier, the Secret name and both kubeconfigs. **No `USER_INPUT` reaches the +namespace decision, and none may be added.** A test in `../definition.tftest.hcl` asserts that every +input of the definition carries one of those four assignment types. + +### Why `delete` is in the list + +`version_spec.deletion_mode` is `DELETE`, so meshStack destroys everything the run created when the +building block is deleted. Destroying the Secret needs `delete` on `secrets`, and without the verb the +destroy fails with a 403 on that one resource: the deletion stops half-way and the credential in the +namespace outlives the building block it belonged to. + +`delete` grants little beyond what the credential already has. `update` can replace the content of any +Secret in the cluster, so a credential that holds `update` can already make any Secret useless to the +workload reading it; `delete` removes the object instead of emptying it. `create` remains the verb that +carries the real risk, for the reason above. + +## Per-tenant names + +`naming.tf` derives every per-tenant name from `.`, the +pair being unique in meshStack. Each name carries the first eight characters of the SHA256 hash of +the untruncated pair, because Postgres identifiers, DNS labels, Kubernetes namespaces and bucket +names all stop at 63 characters and two long identifier pairs that share a prefix would otherwise +collide after truncation. + +The module creates the resources carrying these names, so a name that changes takes its resource with +it: Terraform replaces the database, the user or the bucket, and the data in it is gone. Every +expression in `naming.tf` is part of the module's contract. + +| What | Shape | Example for `acme.payments` | +|---|---|---| +| Langfuse namespace and hostname label | `langfuse--` | `langfuse-acme-payments-817e29e9` | +| Postgres database, and the user owning it | `langfuse__` | `langfuse_acme_payments_817e29e9` | +| ClickHouse database, and the user scoped to it | `langfuse__` | `langfuse_acme_payments_817e29e9` | +| Bucket | `langfuse--` | `langfuse-acme-payments-817e29e9` | +| ClickHouse DDL release, and the objects of it | `langfuse--` | `langfuse-acme-payments-817e29e9` | +| Valkey key prefix | `-:` | `acme-payments-817e29e9:` | +| Valkey database index | hash modulo the index count | | +| Langfuse organisation and project | the plain meshStack identifiers | `acme`, `payments` | + +A database and a user carry the same name on purpose, in Postgres as in ClickHouse: an operator reads +one name in two places. Both users have to be per-tenant, because a Postgres role and a ClickHouse user +are objects of the whole server, and two tenants sharing a user share every grant that user holds. + +The DDL release gets a shorter slug than everything else. Helm stops a release name at 53 characters, +which is the tightest limit of the family, and the Jobs of the release derive their names from the +release name plus a suffix β€” a Job name in turn ends up in the `job-name` label of its pods, and a label +value stops at 63 characters. + +The Valkey index and the key prefix carry different guarantees, and both are set. The prefix is +unique per tenant and is what actually separates two tenants: Langfuse's BullMQ queue names are +hardcoded, so two tenants sharing a keyspace without a prefix would have one tenant's worker consume +the other tenant's ingestion jobs. The index is a hard namespace that no application bug can cross, +but an instance serves only so many indices, so it is derived by hash and repeats once there are more +tenants than indices. It is defence in depth behind the prefix, never the separation itself. + +## The three backend resources this building block creates + +Langfuse creates tables inside a database and objects inside a bucket, never the database or the bucket +itself. The Postgres database, the ClickHouse database and the bucket therefore have to exist before its +first pod starts, and tenants arrive one at a time, so nothing applied once at the level of the platform +can pre-create them. **This building block creates them**, in the same run that deploys the instance +using them. + +That makes the block STACKIT-bound. It is a deliberate choice: `modules/ai/` groups a capability rather +than promising cloud-agnosticism, and `modules/ai/azure-openai` sits in the same directory. An Azure +twin of this block would keep the shape and swap the two submodules. + +| Backend | How it is created | What is destroyed with the block | +|---|---|---| +| Postgres | `modules/stackit/postgresflex/buildingblock/database` in database-only mode | the database and its owner user | +| Object storage | `modules/stackit/storage-bucket/buildingblock/bucket` | the credentials group, the credential and the bucket, if the bucket is empty | +| ClickHouse | a Helm release of the chart in `clickhouse-ddl/`, two hook Jobs | the database and the user, dropped by the `pre-delete` hook | + +### Postgres, through the provider-free submodule + +`modules/stackit/postgresflex/buildingblock/database` declares no provider of its own, which is what +makes it usable from here: a module carrying its own provider configuration is a legacy module, and +OpenTofu rejects `count`, `for_each` and `depends_on` on every call to it. Passing +`existing_instance_id` selects the submodule's database-only mode, in which it creates +`stackit_postgresflex_database` and `stackit_postgresflex_user` against the shared instance and touches +nothing about the instance itself. Tenant churn therefore never re-plans the shared server. + +The owner user is created with the role `login` and nothing else. `prisma migrate deploy` creates and +alters tables inside a database the user owns and never creates a database, so `createdb` would only let +a tenant create databases outside its own scope. + +STACKIT generates the password of the user, so it is not an input either. One consequence is worth +knowing: `modules/ai/langfuse` rejects a Postgres password containing `:`, `@`, `/`, `?`, `#` or `%`, +because Langfuse substitutes the value into its connection URL without percent-encoding it and Prisma +then answers `P1013`. Should STACKIT ever generate such a password, the apply stops with that message +and the way out is to have STACKIT generate a new one for the user. + +`modules/ai/langfuse` receives the host, the port, the database, the user and the password from that +call, and `DIRECT_URL` from its `direct_connection_string` output. **Both connection limits stay +pinned.** The pool of a pod is capped at 5 connections and the pool of a migration at 2, which are the +defaults of `modules/ai/langfuse`: the shared instance has a fixed `max_connections`, of which STACKIT +reserves 15, and every pod of every tenant draws from the rest. The value handed to `DIRECT_URL` carries +no `connection_limit` of its own β€” the submodule puts `sslmode=require` on it and nothing else β€” and +`modules/ai/langfuse` appends the smaller limit itself, because two occurrences in one query string +leave the effective pool size to the parser. + +### The bucket, created with the `aws` provider + +`modules/stackit/storage-bucket/buildingblock/bucket` is provider-free for the same reason, and it needs +two providers: `stackit` for the credentials group and the credential, and **`aws` for the bucket and its +policy**. The `aws` provider is configured here as a generic S3 client against the STACKIT Object Storage +endpoint, because the stackit provider has no permission to create a bucket. `provider.tf` mirrors +`modules/stackit/storage-bucket/buildingblock/provider.tf`, region included: the region is fixed at +`eu01` and is not an input, because the endpoint URL carries it and the two cannot be set apart. + +The credential the tenant's instance uses comes out of that call rather than in as an input. The +submodule creates a credentials group per bucket and writes a bucket policy that denies every principal +outside that group and the administrative group, so a tenant's credential reaches its own bucket and no +other tenant's. + +### ClickHouse: a Helm release with two hook Jobs + +ClickHouse has no Terraform resource for a database and a user, and **it cannot have one here**. The +shared cluster answers on an in-cluster hostname such as +`clickhouse-clickhouse-headless.clickhouse.svc.cluster.local`, which the meshStack Terraform runner can +neither resolve nor reach. Any provider speaking the ClickHouse protocol would need a route from the +runner into the cluster network, so the DDL has to execute inside the cluster whatever drives it. This is +the decisive difference to Postgres, where the STACKIT API is a public endpoint and a provider does the +work. + +What was weighed: + +| Option | Verdict | +|---|---| +| The official `ClickHouse/clickhouse` provider | Not applicable. It manages ClickHouse Cloud services and has no resource for DDL against a self-managed server. | +| A community ClickHouse provider | Rejected. Each is a single-maintainer project, and none of them changes the reachability problem above: the runner still cannot open a connection to a Service inside the cluster. | +| A bare `kubernetes_job_v1` | Rejected. It creates the database, but Terraform destroys resources and has no way to run a Job on the way out, so the tenant's data would outlive the tenant. | +| A Helm release with hook Jobs | **Chosen.** | + +The chart lives in `clickhouse-ddl/` inside this module, the pattern `modules/ai/clickhouse` and +`modules/kubernetes/ingress` already use for manifests Terraform cannot express. It carries two Jobs: + +- a `post-install,post-upgrade` hook that creates the database, the user and the grants; +- a `pre-delete` hook that drops the database and the user again. + +Both run in the namespace of the shared cluster, not in the tenant's own namespace. The administrative +password is a Secret in that namespace, and the tenant's namespace does not exist yet at that point, +because `modules/ai/langfuse` creates it and has to run afterwards. + +This answers the three awkward parts of running DDL from a Job: + +- **Ordering.** The Langfuse module call carries `depends_on` on the release, so the instance is deployed + after the DDL has run. Inside the release, Helm applies the hook before it reports the release ready. +- **Failure.** `clickhouse-client` exits non-zero on a failed statement, `set -e` fails the container, + `backoffLimit: 0` fails the Job, Helm fails the release and Terraform fails the apply. A failed + statement cannot pass silently. `activeDeadlineSeconds` bounds a cluster that never answers, so the + result is a failed Job with a log rather than a Terraform timeout without one. +- **Deletion.** `helm uninstall` runs the `pre-delete` hook and waits for it before it removes anything + else, and destroying the release is what the runner does when the building block is deleted. + +Every apply of the building block runs the create Job again, so every statement in it is idempotent: +`CREATE DATABASE IF NOT EXISTS`, `CREATE USER IF NOT EXISTS` followed by an `ALTER USER … IDENTIFIED +WITH`, and a `GRANT … WITH REPLACE OPTION`. The `ALTER USER` is there because `CREATE USER IF NOT EXISTS` +leaves the password of an existing user untouched, and Terraform holds the value it also hands to +Langfuse. `WITH REPLACE OPTION` revokes what the user held before, so the grants converge on the list +below instead of only ever growing. + +The grants are exactly what Langfuse uses on its own database: + +```sql +GRANT ON CLUSTER default + SELECT, INSERT, CREATE, DROP TABLE, ALTER UPDATE, ALTER DELETE, ALTER DROP INDEX + ON langfuse_acme_payments_817e29e9.* TO langfuse_acme_payments_817e29e9 + WITH REPLACE OPTION; +``` + +`CREATE DATABASE` is deliberately absent. Langfuse runs its ClickHouse migrations with golang-migrate, +which creates tables inside a database that already exists and never creates one, so the right would only +let a tenant create databases outside its own scope. + +Two details worth knowing when comparing this with the DDL written out in +`modules/ai/clickhouse/buildingblock/README.md`. First, the position of `ON CLUSTER`: the documented +ClickHouse grammar puts it **directly after `GRANT`**, unlike in `CREATE DATABASE` and `CREATE USER`, +where it follows the object the statement names, and that readme places it at the end of the `GRANT` +instead. Second, the password of the tenant's user is generated in this run rather than taken as an +input, for the same reason the three Langfuse secrets are: a `STATIC` input is the same value for every +tenant, and a shared password would let one tenant connect as another. + +### What deletion actually does, and where it stops + +`deletion_mode` is `DELETE`, and each of the three backends behaves differently: + +| Resource | On deletion | +|---|---| +| Postgres database and owner user | Destroyed by the STACKIT API. The data is gone. | +| ClickHouse database and user | Dropped by the `pre-delete` hook Job, with `SYNC`, so the tables are gone before the Job finishes. | +| Bucket, credentials group and credential | Destroyed β€” **but only when the bucket is empty.** | + +**A bucket that still holds objects fails the destroy.** The S3 API refuses to delete a non-empty bucket, +and the submodule sets no `force_destroy`. The submodule destroys the credential only after the bucket, so +a destroy that stops on a non-empty bucket leaves the credential in place: an operator can empty the +bucket with it and run the deletion again. Emptying it stays a deliberate step, because those objects are +the tenant's trace payloads. + +Two more failure modes to know about: + +- If the ClickHouse cluster is unreachable when the building block is deleted, the `pre-delete` hook Job + never succeeds, the uninstall fails and the destroy stops. Removing the release from the state with + `tofu state rm` and cleaning up by hand is the way out β€” the same command is also the way to keep a + tenant's trace data on purpose. +- A first install that fails is removed again, because the release is `atomic`, and that removal runs the + same `pre-delete` hook. On a fresh tenant that is what should happen: no half-created database is left + behind. A failed *upgrade* rolls back and runs no hook at all. + +## The STACKIT credential + +The module authenticates against STACKIT with a service account key, a `STATIC` sensitive input. The hub +convention for modules under `modules/stackit/` is workload identity federation instead, and it is the +better pattern: no long-lived secret to rotate. It is not available here without more work, because a +federated identity provider has to assert the subject of this building block definition, and creating one +needs a backplane this module does not have. `modules/kubernetes/ingress` takes a STACKIT service account +key as an input for the same reason. + +The platform team therefore creates three things by hand, or from its own foundation code, and passes them +in: the service account and its key, the administrative object storage credentials group, and the shared +PostgreSQL Flex instance itself. A backplane that creates the first two and moves the block to workload +identity federation is follow-up work. + +The administrative ClickHouse password is **not** an input. The DDL Job mounts the Secret the shared +cluster already holds it in, named by the `admin_secret` output of `modules/ai/clickhouse`. + +## Access control of the tracing instances + +Langfuse upserts an organisation membership with `langfuse_default_org_role` for every user who logs +in, which is what removes member synchronisation from the picture: no group mapping, no SCIM. + +With one OIDC client shared across tenants, that has a consequence worth stating plainly: every user +the identity provider authenticates reaches every tenant's instance with that role. The three ways +out are a client per tenant, which contradicts the zero-input rule of a mandatory block; restricting +who is assigned to the shared client at the identity provider; or `langfuse_default_org_role = "NONE"` +plus memberships added by hand. Pick one deliberately. + +Every tenant's instance has a callback URL of its own, reported by the `langfuse_oidc_callback_url` +output, and the shared client needs each of them as an allowed redirect URI. Where the identity +provider supports a wildcard redirect URI, one entry covers every tenant. + +## Generated secrets + +Four secrets have to differ per tenant, or the isolation between the instances is defeated: the salt +that hashes the tenant's API keys, the encryption key for its stored credentials, the NextAuth secret +that signs its session tokens, and the password of its ClickHouse user. They are generated in the run, +because a `STATIC` input is the same value for every tenant. `random_bytes` is used instead of +`random_id` wherever the value is a secret: both produce hex, but only `random_bytes` marks it +sensitive, so the value stays out of the plan. + +The password of the Postgres user is generated by STACKIT, and the credential of the bucket by the +Object Storage API. Neither is an input either. + +## Provider constraints + +Two constraints in `versions.tf` are not the `>=` the hub rule asks for, and both are deliberate: + +- `ncecere/litellm` is pinned to exactly `2.0.1`, because it is a community provider that has changed + resource behaviour inside a minor release and a recreated `litellm_key` takes the credential away + from a running application. `versions.tf` carries the full reasoning. +- `hashicorp/aws` carries an upper bound of `< 5.0`. That is not a choice made here: + `modules/stackit/storage-bucket/buildingblock/bucket` carries the same constraint, because v5 of the + provider always sends `LocationConstraint` in `CreateBucket` while STACKIT's StorageGRID only accepts + a request without it. A wider constraint here would not intersect with the submodule's and `init` + would fail. + +## Validating locally + +The module sources three modules with `?ref=${var.hub.git_ref}` β€” `modules/ai/langfuse/buildingblock` +and the two STACKIT submodules β€” and `var.hub` is `const`, so its value has to come from an environment +variable rather than from `-var`: + +```sh +export TF_VAR_hub='{"git_ref":""}' +tofu init && tofu validate && tofu test +``` + +The chart in `clickhouse-ddl/` renders on its own, which is the quickest way to read the statements the +Jobs run: + +```sh +helm template t ./clickhouse-ddl \ + --set clickhouse.host=clickhouse-clickhouse-headless.clickhouse.svc.cluster.local \ + --set clickhouse.image=clickhouse/clickhouse-server:26.4 \ + --set admin.secretName=clickhouse-admin \ + --set tenant.database=langfuse_acme_payments_817e29e9 \ + --set tenant.username=langfuse_acme_payments_817e29e9 \ + --set tenant.secretName=langfuse-acme-payments-817e29e9-user \ + --set 'tenant.grants={SELECT,INSERT}' +``` + +The same interpolation stops `terraform-docs` from reading this directory, which is why there is no +generated input and output table below. `reference-architectures/stackit-landingzone` and +`reference-architectures/stackit-kubernetes` fail the same way, for the same reason. diff --git a/modules/ai/model-access/buildingblock/SUMMARY.md.tftpl b/modules/ai/model-access/buildingblock/SUMMARY.md.tftpl new file mode 100644 index 00000000..69e29d43 --- /dev/null +++ b/modules/ai/model-access/buildingblock/SUMMARY.md.tftpl @@ -0,0 +1,54 @@ +# AI Model Access + +Your project has a governed, OpenAI-compatible model endpoint and a tracing instance of its own. + +## Where the credential is + +The credential is a bearer token, so it is not shown here. The platform wrote it into a Kubernetes +Secret in your namespace: + +| Property | Value | +|----------|-------| +| **Namespace** | `${secret_namespace}` | +| **Secret** | `${secret_name}` | +| **Key with the credential** | `${secret_key_env}` | +| **Key with the endpoint** | `${base_url_env}` | + +The two keys carry the names the OpenAI client libraries read from the environment, so a Deployment +can mount the whole Secret and the client needs no configuration: + +```yaml +envFrom: + - secretRef: + name: ${secret_name} +``` + +## Endpoint + +| Property | Value | +|----------|-------| +| **Base URL** | `${api_base}` | +| **Team** | `${team_alias}` | +| **Team ID** | `${team_id}` | +| **Key ID** | `${key_id}` | +| **Budget** | ${max_budget} per ${budget_duration} | +| **Allowed models** | ${length(models) > 0 ? join(", ", models) : "set by the platform team on the gateway"} | + +The key ID above is the SHA256 hash of the credential, not the credential itself. Use it to identify +the credential in support requests. + +The base URL ends in `/v1`, so an OpenAI client library works unchanged: + +```sh +curl "${api_base}/models" \ + -H "Authorization: Bearer $OPENAI_API_KEY" +``` + +The endpoint stops answering once your project reaches its budget for the current period, and the +spend counter resets at the end of every period. + +## Tracing + +Your project has its own tracing instance at ${langfuse_url}. Log in with your usual account. Every +call your application sends through the endpoint above can be traced there, and the traces, the +evaluations and the stored credentials of this instance belong to your project alone. diff --git a/modules/ai/model-access/buildingblock/bucket.tf b/modules/ai/model-access/buildingblock/bucket.tf new file mode 100644 index 00000000..63142475 --- /dev/null +++ b/modules/ai/model-access/buildingblock/bucket.tf @@ -0,0 +1,21 @@ +# The tenant's bucket, its own Object Storage credentials group and the credential inside that group. +# `modules/stackit/storage-bucket/buildingblock/bucket` is provider-free for the same reason the +# Postgres submodule is, and it needs both providers this module configures: the `stackit` provider +# for the credentials group and the credential, and the `aws` provider for the bucket and its policy. +# +# The bucket is created with the `aws` provider used as a generic S3 client, because the stackit +# provider has no permission to create one. That is a property of the submodule, not a choice made +# here, and it is the reason this module configures an `aws` provider at all. +# +# The credential the tenant's Langfuse instance uses comes out of this call rather than in as an +# input. The bucket policy the submodule writes denies every principal outside the tenant's own +# credentials group and the administrative group, so the credential Langfuse receives reaches this +# bucket and no other tenant's. +module "bucket" { + source = "github.com/meshcloud/meshstack-hub//modules/stackit/storage-bucket/buildingblock/bucket?ref=${var.hub.git_ref}" + + project_id = var.stackit_project_id + bucket_name = local.langfuse_bucket + + admin_credentials_group_urn = var.stackit_s3_admin_credentials_group_urn +} diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/Chart.yaml b/modules/ai/model-access/buildingblock/clickhouse-ddl/Chart.yaml new file mode 100644 index 00000000..ff6976c3 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/Chart.yaml @@ -0,0 +1,5 @@ +apiVersion: v2 +name: meshstack-langfuse-clickhouse-ddl +description: Per-tenant ClickHouse database and scoped user for a Langfuse instance, created by a post-install hook Job and dropped again by a pre-delete hook Job. +type: application +version: 1.0.0 diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/_helpers.tpl b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/_helpers.tpl new file mode 100644 index 00000000..101b71e3 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/_helpers.tpl @@ -0,0 +1,84 @@ +{{/* +The preamble both Jobs share. It defines a shell function that sends one statement to the shared +cluster as the administrative user, and it waits until the cluster answers at all. + +The wait is what makes the Job usable right after the cluster was installed or restarted: a +`clickhouse-client` that cannot connect exits non-zero, and with `set -e` the Job would fail on a +server that is only a few seconds from being ready. A command in the condition of `until` does not +trip `set -e`, so the loop is the one place a failure is tolerated. Past the deadline the Job fails +with a message that names the address it tried. +*/}} +{{- define "clickhouse-ddl.preamble" -}} +set -eu + +host="{{ required "clickhouse.host is required" .Values.clickhouse.host }}" +port="{{ .Values.clickhouse.nativePort }}" + +run() { + clickhouse-client --host "$host" --port "$port" \ + --user "{{ .Values.admin.username }}" --password "$ADMIN_PASSWORD" \ + --query "$1" +} + +deadline=$(( $(date +%s) + {{ .Values.job.timeoutSeconds }} )) +until run "SELECT 1" > /dev/null 2>&1; do + if [ "$(date +%s)" -ge "$deadline" ]; then + echo "ClickHouse at $host:$port did not answer within {{ .Values.job.timeoutSeconds }} seconds" + exit 1 + fi + echo "waiting for ClickHouse at $host:$port" + sleep 5 +done +{{- end -}} + +{{/* +The statements that create the tenant's database and its user. Every one of them is idempotent, +because every apply of the building block runs this Job again. + +The password reaches the statement as a shell variable from a secretKeyRef and is put into a quoted +SQL literal. ClickHouse replaces it with '[HIDDEN]' in its query log. +*/}} +{{- define "clickhouse-ddl.script.create" -}} +{{ include "clickhouse-ddl.preamble" . }} + +database="{{ required "tenant.database is required" .Values.tenant.database }}" +username="{{ required "tenant.username is required" .Values.tenant.username }}" +cluster="{{ .Values.clickhouse.ddlCluster }}" + +run "CREATE DATABASE IF NOT EXISTS $database ON CLUSTER $cluster" + +run "CREATE USER IF NOT EXISTS $username ON CLUSTER $cluster IDENTIFIED WITH sha256_password BY '$TENANT_PASSWORD' DEFAULT DATABASE $database" + +# CREATE USER IF NOT EXISTS leaves the password of a user that already exists as it is, so the +# password is set again here. Terraform holds the value and hands the same one to Langfuse, and +# without this statement the two would drift apart the moment the value in state is replaced. +run "ALTER USER $username ON CLUSTER $cluster IDENTIFIED WITH sha256_password BY '$TENANT_PASSWORD'" + +# ON CLUSTER belongs directly after GRANT here, unlike in the statements above, where it follows the +# object the statement names. WITH REPLACE OPTION revokes everything the user held before, so the +# grants converge on this list instead of only ever growing. +run "GRANT ON CLUSTER $cluster {{ join ", " (required "tenant.grants is required" .Values.tenant.grants) }} ON $database.* TO $username WITH REPLACE OPTION" + +echo "database $database and user $username are in place" +{{- end -}} + +{{/* +The statements that remove the tenant again. They run as a pre-delete hook, so the database and the +user go when the building block is deleted. + +SYNC makes the server wait for the tables to be gone before it answers, rather than dropping them in +the background. Without it the Job could finish while the replicated tables of the database still hold +their entries in Keeper, and a tenant recreated under the same name would meet them. +*/}} +{{- define "clickhouse-ddl.script.drop" -}} +{{ include "clickhouse-ddl.preamble" . }} + +database="{{ required "tenant.database is required" .Values.tenant.database }}" +username="{{ required "tenant.username is required" .Values.tenant.username }}" +cluster="{{ .Values.clickhouse.ddlCluster }}" + +run "DROP DATABASE IF EXISTS $database ON CLUSTER $cluster SYNC" +run "DROP USER IF EXISTS $username ON CLUSTER $cluster" + +echo "database $database and user $username are gone" +{{- end -}} diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/create-job.yaml b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/create-job.yaml new file mode 100644 index 00000000..9036dc84 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/create-job.yaml @@ -0,0 +1,64 @@ +# The Job that creates the tenant's database and its scoped user. +# +# It is a post-install and post-upgrade hook, so Helm runs it on the first apply and on every later +# one, and Helm waits for a hook Job to finish. A statement that fails takes the Job with it, the Job +# takes the release, and the release takes the Terraform apply β€” which is the whole point of running +# the DDL from here instead of from a script somebody remembers to run. +# +# The delete policy keeps a failed Job for its log and removes a successful one right away, so the +# shared ClickHouse namespace does not collect one finished Job per tenant per apply. +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Release.Name }}-create + namespace: {{ .Release.Namespace }} + labels: + app.kubernetes.io/managed-by: meshstack + app.kubernetes.io/part-of: ai-model-access + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + # One attempt. The script waits for the cluster itself, so a failure past that point is a failure of + # a statement, and repeating it would only hide the error behind a retry. + backoffLimit: 0 + activeDeadlineSeconds: {{ .Values.job.timeoutSeconds }} + template: + metadata: + name: {{ .Release.Name }}-create + labels: + app.kubernetes.io/managed-by: meshstack + app.kubernetes.io/part-of: ai-model-access + spec: + restartPolicy: Never + # The Job talks to ClickHouse and never to the Kubernetes API. + automountServiceAccountToken: false + containers: + - name: ddl + image: {{ required "clickhouse.image is required" .Values.clickhouse.image | quote }} + env: + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: {{ required "admin.secretName is required" .Values.admin.secretName }} + key: {{ .Values.admin.secretKey }} + - name: TENANT_PASSWORD + valueFrom: + secretKeyRef: + name: {{ required "tenant.secretName is required" .Values.tenant.secretName }} + key: {{ .Values.tenant.secretKey }} + command: + - /bin/bash + - -c + - | + {{- include "clickhouse-ddl.script.create" . | nindent 14 }} + # The pod sends a handful of statements and waits, so it needs almost nothing. A small + # request also keeps it schedulable on a node the ClickHouse replicas have already filled. + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/drop-job.yaml b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/drop-job.yaml new file mode 100644 index 00000000..410eb730 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/drop-job.yaml @@ -0,0 +1,53 @@ +# The Job that removes the tenant's database and its user again. +# +# It is a pre-delete hook. `helm uninstall` runs it and waits for it before it removes anything else, +# and destroying this Helm release is what the meshStack runner does when the building block is +# deleted, so the tenant's trace data does not outlive the tenant. +# +# It reads no tenant password, because dropping a user needs none. The Secret holding that password is +# a Terraform resource and is destroyed with the rest of the run. +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Release.Name }}-drop + namespace: {{ .Release.Namespace }} + labels: + app.kubernetes.io/managed-by: meshstack + app.kubernetes.io/part-of: ai-model-access + annotations: + helm.sh/hook: pre-delete + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + backoffLimit: 0 + activeDeadlineSeconds: {{ .Values.job.timeoutSeconds }} + template: + metadata: + name: {{ .Release.Name }}-drop + labels: + app.kubernetes.io/managed-by: meshstack + app.kubernetes.io/part-of: ai-model-access + spec: + restartPolicy: Never + automountServiceAccountToken: false + containers: + - name: ddl + image: {{ required "clickhouse.image is required" .Values.clickhouse.image | quote }} + env: + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: {{ required "admin.secretName is required" .Values.admin.secretName }} + key: {{ .Values.admin.secretKey }} + command: + - /bin/bash + - -c + - | + {{- include "clickhouse-ddl.script.drop" . | nindent 14 }} + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/tenant-configmap.yaml b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/tenant-configmap.yaml new file mode 100644 index 00000000..f909c683 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/templates/tenant-configmap.yaml @@ -0,0 +1,16 @@ +# Both Jobs of this chart are Helm hooks, and a hook is not part of the manifest of the release. This +# ConfigMap is, so the release owns one object: an operator sees which tenant a release belongs to and +# what it granted, and `helm uninstall` has something to remove after the pre-delete hook has run. +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-ddl + namespace: {{ .Release.Namespace }} + labels: + app.kubernetes.io/managed-by: meshstack + app.kubernetes.io/part-of: ai-model-access +data: + database: {{ required "tenant.database is required" .Values.tenant.database | quote }} + username: {{ required "tenant.username is required" .Values.tenant.username | quote }} + cluster: {{ .Values.clickhouse.ddlCluster | quote }} + grants: {{ join ", " .Values.tenant.grants | quote }} diff --git a/modules/ai/model-access/buildingblock/clickhouse-ddl/values.yaml b/modules/ai/model-access/buildingblock/clickhouse-ddl/values.yaml new file mode 100644 index 00000000..7cbb5a4a --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse-ddl/values.yaml @@ -0,0 +1,37 @@ +# Every value is set by the Terraform module that installs this chart. The keys are listed here so the +# shape of the values is documented in one place. The templates call `required` on every value that +# would otherwise render a statement with a hole in it, so a chart rendered by hand fails instead. + +clickhouse: + # Fully qualified in-cluster hostname of the shared ClickHouse cluster. + host: "" + # Native protocol port. clickhouse-client speaks it. + nativePort: 9000 + # Name of the cluster as the server knows it. Every statement runs ON CLUSTER with it. + ddlCluster: default + # Image the Jobs run clickhouse-client from. The same image the servers run, so the node has it + # cached already. + image: "" + +# The administrative user. It creates the tenant's database and user, so it must never reach a tenant. +admin: + username: default + # Secret in the release namespace that holds the administrative password, and the key inside it. + secretName: "" + secretKey: password + +# The tenant. Every name here is derived from the meshStack workspace and project by the calling +# module, so two tenants never collide. +tenant: + database: "" + username: "" + # The rights the user holds on its own database, and nothing beyond them. + grants: [] + # Secret in the release namespace that holds the password of the tenant's user. + secretName: "" + secretKey: password + +job: + # Seconds a Job may run before Kubernetes fails it. It covers the wait for the cluster to answer a + # query and the statements themselves. + timeoutSeconds: 600 diff --git a/modules/ai/model-access/buildingblock/clickhouse.tf b/modules/ai/model-access/buildingblock/clickhouse.tf new file mode 100644 index 00000000..7348f5f2 --- /dev/null +++ b/modules/ai/model-access/buildingblock/clickhouse.tf @@ -0,0 +1,127 @@ +# The tenant's ClickHouse database and the user scoped to it. +# +# ClickHouse has no Terraform resource for this, and it cannot have one here: the shared cluster +# answers on an in-cluster hostname such as +# 'clickhouse-clickhouse-headless.clickhouse.svc.cluster.local', which the meshStack Terraform runner +# cannot resolve, let alone reach. Any provider that spoke the ClickHouse protocol would need a route +# from the runner into the cluster network. The DDL therefore runs inside the cluster, as a Job, with +# the `kubernetes` and `helm` providers this module already configures for the AI platform cluster. +# See the ClickHouse section of the module README for the alternatives that were weighed. +# +# The Job runs in the ClickHouse namespace and not in the tenant's own namespace, for two reasons: the +# administrative password is a Secret in that namespace, and the tenant's namespace does not exist +# yet, because `modules/ai/langfuse` creates it and has to run after this. + +locals { + # Exactly the rights Langfuse uses on its own database, and nothing beyond them. `CREATE DATABASE` + # is deliberately absent: Langfuse runs its ClickHouse migrations with golang-migrate, which creates + # tables inside a database that already exists and never creates one, so the grant would only let a + # tenant create databases outside its own scope. + langfuse_clickhouse_grants = [ + "SELECT", + "INSERT", + "CREATE", + "DROP TABLE", + "ALTER UPDATE", + "ALTER DELETE", + "ALTER DROP INDEX", + ] + + clickhouse_ddl_secret_key = "password" +} + +# The password of the tenant's ClickHouse user is generated here rather than taken as an input, for +# the same reason the three Langfuse secrets are: a STATIC input is the same value for every tenant, +# and a shared password would let one tenant connect as another. +# +# No special characters. The Langfuse migration script puts the value into a query string without +# encoding it, and the DDL below puts it into a quoted SQL literal. +resource "random_password" "langfuse_clickhouse" { + length = 32 + special = false +} + +# The DDL Job reads the password from this Secret. It does not travel through the Helm values, because +# the values of a release are stored as they are given and a sensitive value in them would also hide +# the whole rendered release from every plan. +resource "kubernetes_secret_v1" "langfuse_clickhouse_user" { + provider = kubernetes.ai_platform + + metadata { + name = "${local.clickhouse_ddl_name}-user" + namespace = var.langfuse_clickhouse_namespace + + labels = { + "app.kubernetes.io/managed-by" = "meshstack" + "app.kubernetes.io/part-of" = "ai-model-access" + } + } + + type = "Opaque" + + data = { + (local.clickhouse_ddl_secret_key) = random_password.langfuse_clickhouse.result + } +} + +# Two Jobs in one Helm release, and the release is what makes deletion work. A Helm chart can carry a +# `pre-delete` hook, and `helm uninstall` runs it and waits for it before it removes anything else, so +# the tenant's database and user are dropped when the building block is deleted. Terraform on its own +# cannot do that: it destroys resources, and it has no way to run a Job on the way out. +# +# The chart lives in a directory of this module. `modules/ai/clickhouse` and `modules/kubernetes/ingress` +# use the same pattern for manifests Terraform cannot express. +resource "helm_release" "clickhouse_ddl" { + provider = helm.ai_platform + + name = local.clickhouse_ddl_name + namespace = var.langfuse_clickhouse_namespace + chart = "${path.module}/clickhouse-ddl" + + create_namespace = false + + # Helm waits for a hook Job to finish and fails the release when the Job fails, so a failed + # statement fails the apply instead of passing silently. + wait = true + + # An install that fails is removed again, so the next apply installs cleanly instead of stopping on + # a release name that is already taken. It also means a failed first install leaves no half-created + # database behind: the removal runs the same `pre-delete` hook the deletion does. A failed upgrade + # rolls back and runs no hook at all. + atomic = true + + # Above the deadline of the Jobs, so a ClickHouse that never answers produces a failed Job with a + # log rather than a Helm timeout without one. + timeout = var.langfuse_clickhouse_ddl_timeout + 120 + + # `disable_webhooks` is Helm's `--no-hooks`. It has to stay off: both Jobs of this release are + # hooks, and with hooks disabled the release would create nothing and delete nothing. + disable_webhooks = false + + values = [yamlencode({ + clickhouse = { + host = var.langfuse_clickhouse_host + nativePort = var.langfuse_clickhouse_native_port + ddlCluster = var.langfuse_clickhouse_ddl_cluster_name + image = var.langfuse_clickhouse_client_image + } + + admin = { + username = var.langfuse_clickhouse_admin_username + secretName = var.langfuse_clickhouse_admin_secret_name + secretKey = var.langfuse_clickhouse_admin_secret_key + } + + tenant = { + database = local.langfuse_clickhouse_database + username = local.langfuse_clickhouse_username + grants = local.langfuse_clickhouse_grants + secretName = kubernetes_secret_v1.langfuse_clickhouse_user.metadata[0].name + secretKey = local.clickhouse_ddl_secret_key + } + + job = { + timeoutSeconds = var.langfuse_clickhouse_ddl_timeout + } + })] +} diff --git a/modules/ai/model-access/buildingblock/langfuse.tf b/modules/ai/model-access/buildingblock/langfuse.tf new file mode 100644 index 00000000..9f242fde --- /dev/null +++ b/modules/ai/model-access/buildingblock/langfuse.tf @@ -0,0 +1,130 @@ +# The tenant's own Langfuse instance. `modules/ai/langfuse` declares no providers of its own, so it +# is configured with the AI platform cluster's provider aliases from here. + +# Three secrets have to differ per tenant, or the isolation between the instances is defeated: the +# salt hashes the tenant's API keys, the encryption key encrypts its stored credentials at rest, and +# the NextAuth secret signs its session tokens. They are generated here rather than taken as inputs, +# because a STATIC input is the same value for every tenant. +resource "random_password" "langfuse_salt" { + length = 32 + # The value travels into a Kubernetes Secret and into an environment variable, so it is kept to + # letters and digits. + special = false +} + +resource "random_password" "langfuse_nextauth_secret" { + length = 32 + special = false +} + +# random_bytes rather than random_id: both produce hex, but only random_bytes marks it sensitive, so +# the key stays out of the plan. Langfuse wants 256 bits, which is 32 bytes and 64 hex characters. +resource "random_bytes" "langfuse_encryption_key" { + length = 32 +} + +# Langfuse accepts a predefined API keypair at bootstrap, so the keys are generated here instead of +# a human reading them out of a UI. +resource "random_uuid" "langfuse_project_public_key" {} + +# The secret half comes from random_bytes for the same reason as the encryption key: random_uuid is +# not marked sensitive. +resource "random_bytes" "langfuse_project_secret_key" { + length = 16 +} + +locals { + langfuse_project_public_key = "pk-lf-${random_uuid.langfuse_project_public_key.result}" + + # Langfuse's own generator produces 'sk-lf-', and the 16 random bytes are formatted into that + # shape so nothing downstream has to accept a second one. + langfuse_secret_key_hex = random_bytes.langfuse_project_secret_key.hex + langfuse_project_secret_key = format("sk-lf-%s-%s-%s-%s-%s", + substr(local.langfuse_secret_key_hex, 0, 8), + substr(local.langfuse_secret_key_hex, 8, 4), + substr(local.langfuse_secret_key_hex, 12, 4), + substr(local.langfuse_secret_key_hex, 16, 4), + substr(local.langfuse_secret_key_hex, 20, 12), + ) +} + +module "langfuse" { + source = "github.com/meshcloud/meshstack-hub//modules/ai/langfuse/buildingblock?ref=${var.hub.git_ref}" + + providers = { + kubernetes = kubernetes.ai_platform + helm = helm.ai_platform + } + + # Langfuse creates tables inside its databases and objects inside its bucket, never the databases or + # the bucket themselves, and every web pod runs both sets of migrations before it answers its + # readiness probe. The release is installed with `wait`, so an instance that starts before its + # backends exist does not merely warn: it crash-loops through its migrations and fails the apply. + # + # The Postgres and the bucket dependency are already carried by the values below. The ClickHouse + # dependency is not: the tenant's ClickHouse credentials are generated in this run rather than read + # out of the DDL release, so the ordering has to be stated. + depends_on = [helm_release.clickhouse_ddl] + + namespace = local.langfuse_namespace + hostname = local.langfuse_hostname + + ingress_class_name = var.langfuse_ingress_class_name + + # The chart version and the Langfuse image tag are not inputs. They come from the version of + # `modules/ai/langfuse` that `var.hub.git_ref` selects, so an upgrade is a hub reference bump and a + # new building block definition version, not a value somebody edits on a live definition. + + salt = random_password.langfuse_salt.result + encryption_key = random_bytes.langfuse_encryption_key.hex + nextauth_secret = random_password.langfuse_nextauth_secret.result + + # The database, the owner user and its password come out of the module that created them, so the name + # Langfuse connects to cannot drift away from the name that exists. The host and the port come from + # the same place, because the submodule reads them off the shared instance. + postgres_host = module.postgres.host + postgres_port = module.postgres.port + postgres_database = module.postgres.database_name + postgres_username = module.postgres.username + postgres_password = module.postgres.password + + # DIRECT_URL, the connection the Prisma migrations run over. `postgres_connection_limit` and + # `postgres_direct_url_connection_limit` are deliberately left at the defaults of + # `modules/ai/langfuse`, which pin the pool of a pod to 5 connections and the pool of a migration to + # 2: the shared instance has a fixed `max_connections` and every pod of every tenant draws from it. + # The value below carries no `connection_limit` of its own β€” the submodule puts `sslmode=require` on + # it and nothing else β€” and `modules/ai/langfuse` rejects one that does, because two occurrences in + # one query string leave the effective pool size to the parser. + postgres_direct_url = module.postgres.direct_connection_string + + clickhouse_host = var.langfuse_clickhouse_host + clickhouse_native_port = var.langfuse_clickhouse_native_port + clickhouse_database = local.langfuse_clickhouse_database + clickhouse_username = local.langfuse_clickhouse_username + clickhouse_password = random_password.langfuse_clickhouse.result + + valkey_host = var.langfuse_valkey_host + valkey_password = var.langfuse_valkey_password + valkey_database = local.langfuse_valkey_database + valkey_key_prefix = local.langfuse_valkey_key_prefix + + # The bucket, its endpoint and a credential the bucket policy scopes to this bucket alone, all from + # the module that created them. + s3_bucket = module.bucket.bucket_name + s3_region = module.bucket.region + s3_endpoint = module.bucket.endpoint + s3_access_key_id = module.bucket.s3_access_key + s3_secret_access_key = module.bucket.s3_secret_access_key + + init_org_id = local.langfuse_org_id + init_org_name = local.langfuse_org_name + init_project_id = local.langfuse_project_id + init_project_name = local.langfuse_project_name + init_project_public_key = local.langfuse_project_public_key + init_project_secret_key = local.langfuse_project_secret_key + + # No initial password user: var.oidc is required, and a user with a password nobody can read would + # be of no use anyway, because a building block output cannot carry a password. + oidc = var.oidc + default_org_role = var.langfuse_default_org_role +} diff --git a/modules/ai/model-access/buildingblock/logo.png b/modules/ai/model-access/buildingblock/logo.png new file mode 100644 index 00000000..5b8b3f90 Binary files /dev/null and b/modules/ai/model-access/buildingblock/logo.png differ diff --git a/modules/ai/model-access/buildingblock/main.tf b/modules/ai/model-access/buildingblock/main.tf new file mode 100644 index 00000000..a0799536 --- /dev/null +++ b/modules/ai/model-access/buildingblock/main.tf @@ -0,0 +1,118 @@ +locals { + # The team alias joins both identifiers for the same reason every derived name does: a project + # identifier alone can repeat across workspaces. It stays readable in the LiteLLM UI. + team_alias = local.tenant_key + key_alias = "${local.team_alias}-key" + + # LiteLLM's OpenAI-compatible routes live under '/v1'. The gateway also answers without the + # prefix, but OpenAI client libraries expect it, so the endpoint carries it. + api_base = "${trimsuffix(var.litellm_api_base, "/")}/v1" + + # The two keys of the Secret are the environment variable names the OpenAI client libraries read, + # so a workload can mount the Secret with `envFrom` and needs no code that knows about a gateway. + secret_keys = { + api_key = "OPENAI_API_KEY" + api_base = "OPENAI_BASE_URL" + } +} + +resource "litellm_team" "this" { + team_alias = local.team_alias + models = var.models + + # The budget belongs on the team rather than on the key. LiteLLM counts the spend of every key + # of a team against the team budget, so a single limit here covers the tenant even if the + # platform team later hands out a second key. + max_budget = var.max_budget + budget_duration = var.budget_duration + + metadata = { + meshstack_workspace_identifier = var.workspace_identifier + meshstack_project_identifier = var.project_identifier + meshstack_tenant_uuid = var.meshstack_tenant_uuid + } +} + +# The key deliberately omits `models`: the provider sends "all-team-models" when `team_id` is set +# and `models` is left out, which keeps the allow-list on the team alone. +resource "litellm_key" "this" { + key_alias = local.key_alias + team_id = litellm_team.this.id +} + +# --- The sibling tenant of the same meshProject ------------------------------------------------- +# +# This block runs on the AI model tenant. The workload runs on a second tenant of the same +# meshProject, a namespace on the demo application cluster, and that namespace is where the model +# credential has to land. meshStack does not hand the sibling to a building block run, so the run +# asks for it with the ephemeral API token that `version_spec.permissions` grants. +# +# The filter is on `platform`, the full '.' identifier, and never on +# `platform_type`: two Kubernetes clusters are two platforms of one type, and a type filter would +# match a tenant on the wrong cluster. +data "meshstack_tenants" "sibling" { + workspace = var.workspace_identifier + project = var.project_identifier + platform = var.demo_app_platform_identifier +} + +locals { + sibling_tenants = data.meshstack_tenants.sibling.tenants + sibling_tenant_count = length(local.sibling_tenants) + + # `one()` raises an error of its own on a collection with more than one element, which would + # replace the precondition below with a message that names neither the platform nor the project. + # The conditional keeps it out of the way until the count is known to be one. + # + # `platform_tenant_id` of a Kubernetes tenant is the namespace name itself. + sibling_namespace = local.sibling_tenant_count == 1 ? one(local.sibling_tenants).spec.platform_tenant_id : null +} + +# --- The model credential, in the namespace of the application team ----------------------------- + +resource "kubernetes_secret_v1" "model_access" { + provider = kubernetes.demo_app + + metadata { + name = var.secret_name + namespace = local.sibling_namespace + + labels = { + "app.kubernetes.io/managed-by" = "meshstack" + "app.kubernetes.io/part-of" = "ai-model-access" + } + } + + type = "Opaque" + + data = { + (local.secret_keys.api_key) = litellm_key.this.key + (local.secret_keys.api_base) = local.api_base + } + + lifecycle { + precondition { + condition = local.sibling_tenant_count > 0 + error_message = "No tenant of platform '${var.demo_app_platform_identifier}' exists in project '${var.project_identifier}' of workspace '${var.workspace_identifier}'. The project needs a tenant on that platform, because its namespace is where the model credential is delivered." + } + + precondition { + condition = local.sibling_tenant_count < 2 + error_message = "Project '${var.project_identifier}' of workspace '${var.workspace_identifier}' has ${local.sibling_tenant_count} tenants of platform '${var.demo_app_platform_identifier}'. The namespace the model credential is delivered to would be ambiguous, so the run stops instead of picking one." + } + + # A null platform tenant id means meshStack has not replicated the sibling tenant yet, so its + # namespace does not exist. This is the dangerous case: the Kubernetes provider falls back to + # the namespace of the kubeconfig's current context when `metadata.namespace` is null, and the + # Secret would silently land in a namespace that belongs to somebody else. The condition holds + # for every matched tenant, so it says nothing about the number of matches and leaves that to + # the two preconditions above. + precondition { + condition = alltrue([ + for tenant in local.sibling_tenants : + tenant.spec.platform_tenant_id != null && tenant.spec.platform_tenant_id != "" + ]) + error_message = "The tenant of platform '${var.demo_app_platform_identifier}' in project '${var.project_identifier}' of workspace '${var.workspace_identifier}' carries no platform tenant id, so meshStack has not replicated it yet and its namespace does not exist. Wait for the replication and apply this building block again." + } + } +} diff --git a/modules/ai/model-access/buildingblock/model_access.tftest.hcl b/modules/ai/model-access/buildingblock/model_access.tftest.hcl new file mode 100644 index 00000000..664a3697 --- /dev/null +++ b/modules/ai/model-access/buildingblock/model_access.tftest.hcl @@ -0,0 +1,577 @@ +variables { + workspace_identifier = "acme" + project_identifier = "payments" + + litellm_api_base = "https://litellm.example.com/" + litellm_api_key = "sk-admin-mock" + + ai_platform_cluster_kubeconfig = <<-EOT + current-context: ai-platform + clusters: + - name: ai-platform + cluster: + server: https://ai-platform.example.invalid + certificate-authority-data: "" + users: + - name: ai-platform + user: + token: mock-ai-platform-token + contexts: + - name: ai-platform + context: + cluster: ai-platform + user: ai-platform + EOT + + demo_app_cluster_kubeconfig = <<-EOT + current-context: demo-app + clusters: + - name: demo-app + cluster: + server: https://demo-app.example.invalid + certificate-authority-data: "" + users: + - name: demo-app + user: + token: mock-demo-app-token + contexts: + - name: demo-app + context: + cluster: demo-app + user: demo-app + EOT + + demo_app_platform_identifier = "kubernetes.eu01" + + langfuse_domain = "ai.example.com" + + # STACKIT, where the module creates the tenant's Postgres database, its owner user and its bucket. + stackit_project_id = "6e8c1f30-6c4d-4b1f-9f7a-2c9d8e5f1a2b" + stackit_service_account_key = "{\"id\":\"mock-key\"}" + stackit_s3_admin_access_key = "AKIAMOCKADMINKEY" + stackit_s3_admin_secret_access_key = "mock-admin-secret-access-key" + stackit_s3_admin_credentials_group_urn = "urn:sgws:identity::12345678901234567890:group/mock-admin-group" + + langfuse_postgres_instance_id = "3f2504e0-4f89-11d3-9a0c-0305e82c3301" + + langfuse_clickhouse_host = "clickhouse-clickhouse-headless.clickhouse.svc.cluster.local" + + langfuse_valkey_host = "valkey.valkey.svc.cluster.local" + langfuse_valkey_password = "valkey-password" + + oidc = { + issuer_url = "https://idp.example.com/realms/ai" + client_id = "langfuse" + client_secret = "mock-client-secret" + } +} + +# A distinctive value, so the assertions that check no output carries the credential cannot pass by +# accident. +mock_provider "litellm" { + mock_resource "litellm_key" { + defaults = { + key = "sk-mock-virtual-key-must-never-leave-the-run" + id = "mock-key-hash" + } + } +} + +# The generated Langfuse secrets are validated for their length and their alphabet by the module they +# are passed into, so the mocked values have to be shaped like the real ones. random_bytes is mocked +# once for two resources of different length, which is why the value is 32 bytes of hex: the shorter +# use only reads the first 32 characters of it. +mock_provider "random" { + mock_resource "random_password" { + defaults = { + result = "mockmockmockmockmockmockmockmock" + } + } + + mock_resource "random_bytes" { + defaults = { + hex = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + base64 = "ASNFZ4mrze8BI0VniavN7wEjRWeJq83vASNFZ4mrze8=" + } + } + + mock_resource "random_uuid" { + defaults = { + result = "44444444-4444-4444-4444-444444444444" + } + } +} + +mock_provider "helm" { alias = "ai_platform" } +mock_provider "kubernetes" { alias = "ai_platform" } +mock_provider "kubernetes" { alias = "demo_app" } + +# The two STACKIT submodules read the shared instance and generate credentials, and both feed values +# into `modules/ai/langfuse`, which validates several of them. Every value a generated mock would leave +# null or shape wrongly is therefore set here: the connection info the submodule reads the host and the +# port off, the ACL its summary joins into a string, and the two passwords Langfuse checks the alphabet +# of. +mock_provider "stackit" { + mock_data "stackit_postgresflex_instance" { + defaults = { + name = "shared-langfuse" + version = "17" + + connection_info = { + write = { + host = "shared.postgresflex.eu01.onstackit.cloud" + port = 5432 + } + } + + network = { + acl = ["45.129.40.0/21"] + access_scope = "PUBLIC" + instance_address = "10.0.0.10" + router_address = "10.0.0.1" + } + } + } + + mock_resource "stackit_postgresflex_user" { + defaults = { + password = "mockPostgresPassword" + } + } + + mock_resource "stackit_objectstorage_credentials_group" { + defaults = { + credentials_group_id = "44444444-4444-4444-4444-444444444444" + urn = "urn:sgws:identity::12345678901234567890:group/mock-tenant-group" + } + } + + mock_resource "stackit_objectstorage_credential" { + defaults = { + access_key = "AKIAMOCKTENANTKEY" + secret_access_key = "mock-tenant-secret-access-key" + } + } +} + +# The bucket itself is created with the `aws` provider used as a generic S3 client, so the mock covers +# the bucket and its policy. +mock_provider "aws" {} + +mock_provider "meshstack" { + mock_data "meshstack_tenants" { + defaults = { + tenants = [ + { + metadata = { owned_by_project = "payments", owned_by_workspace = "acme", uuid = "11111111-1111-1111-1111-111111111111" } + ref = { kind = "meshTenant", uuid = "11111111-1111-1111-1111-111111111111" } + spec = { + landing_zone_ref = { kind = "meshLandingZone", name = "namespace" } + platform_ref = { kind = "meshPlatform", uuid = "22222222-2222-2222-2222-222222222222" } + platform_tenant_id = "acme-payments" + quotas = [] + requested_quotas = null + } + status = { + applied_quotas = null + platform_type_identifier = "Kubernetes" + platform_workspace_id = "acme" + tags = {} + tenant_name = "acme.payments" + } + }, + ] + } + } +} + +run "derives_every_per_tenant_name_from_the_tenant_context" { + command = plan + + assert { + # 8 hexadecimal characters of sha256("acme.payments"), appended to every derived name so that + # two long identifier pairs sharing a prefix stay apart after truncation. + condition = output.langfuse_namespace == "langfuse-acme-payments-${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the Langfuse namespace must be derived from the workspace and the project, with the hash of the pair appended" + } + + assert { + condition = output.langfuse_url == "https://langfuse-acme-payments-${substr(sha256("acme.payments"), 0, 8)}.ai.example.com" + error_message = "the Langfuse URL must put the derived label in front of the domain the platform team configured" + } + + assert { + condition = output.langfuse_backend_names.postgres_database == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the Postgres database name must use underscores, because a Postgres identifier may not carry a dash without quoting" + } + + assert { + condition = output.langfuse_backend_names.clickhouse_database == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the ClickHouse database name must be derived the same way as the Postgres one" + } + + assert { + condition = output.langfuse_backend_names.bucket == "langfuse-acme-payments-${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the bucket name must use dashes, because a bucket name may not carry an underscore" + } + + assert { + condition = output.langfuse_backend_names.valkey_key_prefix == "acme-payments-${substr(sha256("acme.payments"), 0, 8)}:" + error_message = "the Valkey key prefix must be unique per tenant and end in a colon" + } + + assert { + # The index is the hash read as a hexadecimal number, modulo the number of indices the instance + # serves. It repeats across tenants, which is why the prefix above carries the separation. + condition = output.langfuse_backend_names.valkey_database == parseint(substr(sha256("acme.payments"), 0, 8), 16) % 16 + error_message = "the Valkey database index must be derived from the tenant hash, modulo the number of indices" + } + + assert { + condition = output.api_base == "https://litellm.example.com/v1" + error_message = "the endpoint must carry the '/v1' suffix exactly once, whether or not the configured base URL ends in a slash" + } +} + +run "two_projects_of_the_same_name_in_two_workspaces_get_different_names" { + command = plan + + variables { + workspace_identifier = "other" + project_identifier = "payments" + } + + assert { + condition = output.langfuse_namespace == "langfuse-other-payments-${substr(sha256("other.payments"), 0, 8)}" + error_message = "a project identifier alone can repeat across workspaces, so the workspace has to be part of every derived name" + } + + assert { + condition = output.langfuse_backend_names.postgres_database != "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "two projects of the same name in two workspaces must not share a Postgres database" + } +} + +run "long_identifiers_stay_inside_the_63_character_limit" { + command = plan + + variables { + # 40 characters each. The pair is 81 characters with the separator, so every derived name has to + # be truncated, and the two pairs below differ only after the truncation point. + workspace_identifier = "a-very-long-workspace-identifier-indeed1" + project_identifier = "a-very-long-project-identifier-for-tests" + } + + assert { + condition = length(output.langfuse_namespace) <= 63 + error_message = "the namespace has to stay inside the 63 character limit of a DNS label" + } + + assert { + condition = length(output.langfuse_backend_names.postgres_database) <= 63 + error_message = "the Postgres database name has to stay inside the 63 character limit of an identifier" + } + + assert { + condition = length(output.langfuse_backend_names.bucket) <= 63 + error_message = "the bucket name has to stay inside the 63 character limit" + } + + assert { + condition = can(regex("^[a-z0-9]([-a-z0-9]*[a-z0-9])?$", output.langfuse_namespace)) + error_message = "the namespace has to be a valid DNS label after truncation, so it may not end in a dash" + } + + assert { + # Truncation cuts the slug at the same place for both pairs, so only the hash keeps the two + # apart. Without it, two long identifier pairs would share one database. + condition = endswith(output.langfuse_backend_names.postgres_database, substr(sha256("a-very-long-workspace-identifier-indeed1.a-very-long-project-identifier-for-tests"), 0, 8)) + error_message = "a truncated name has to end in the hash of the untruncated identifier pair" + } +} + +run "identifiers_with_unexpected_characters_are_folded_into_valid_names" { + command = plan + + variables { + workspace_identifier = "ACME_Corp" + project_identifier = "Payments.EU" + } + + assert { + condition = output.langfuse_namespace == "langfuse-acme-corp-payments-eu-${substr(sha256("ACME_Corp.Payments.EU"), 0, 8)}" + error_message = "every run of characters that a DNS label does not allow has to collapse into a single dash, and the name has to be lowercased" + } + + assert { + condition = output.langfuse_backend_names.postgres_database == "langfuse_acme_corp_payments_eu_${substr(sha256("ACME_Corp.Payments.EU"), 0, 8)}" + error_message = "the SQL form has to use underscores for the same runs of characters" + } +} + +# --- The per-tenant backend resources ----------------------------------------------------------- + +run "the_backend_resources_carry_the_names_naming_tf_derives" { + command = plan + + assert { + # Read off the created database rather than off the local it came from: the module now owns the + # name, so a rename would replace the database and take the data in it with it. + condition = output.langfuse_backend_names.postgres_database == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the database created on the shared Postgres instance has to carry the name naming.tf derives" + } + + assert { + # A Postgres role is an object of the whole server, so the owner has to be per-tenant as well. + condition = output.langfuse_backend_names.postgres_username == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the owner of the tenant's Postgres database has to carry the name of the database it owns" + } + + assert { + condition = output.langfuse_backend_names.bucket == "langfuse-acme-payments-${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the created bucket has to carry the name naming.tf derives" + } + + assert { + # The DDL Job receives the two ClickHouse names as Helm values, so this is where they are checked. + condition = yamldecode(helm_release.clickhouse_ddl.values[0]).tenant.database == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the DDL Job has to create the ClickHouse database under the name naming.tf derives" + } + + assert { + condition = yamldecode(helm_release.clickhouse_ddl.values[0]).tenant.username == "langfuse_acme_payments_${substr(sha256("acme.payments"), 0, 8)}" + error_message = "the ClickHouse user has to carry the name of the database it is scoped to" + } + + assert { + # The DDL runs in the namespace of the shared cluster, because the administrative password is a + # Secret in it and the tenant's own namespace does not exist yet at that point. + condition = helm_release.clickhouse_ddl.namespace == "clickhouse" && kubernetes_secret_v1.langfuse_clickhouse_user.metadata[0].namespace == "clickhouse" + error_message = "the DDL release and the Secret holding the tenant's ClickHouse password have to live in the namespace of the shared cluster" + } +} + +run "the_clickhouse_user_is_granted_only_what_langfuse_uses" { + command = plan + + assert { + # The full list, in order, so adding or removing a right is a deliberate change this test has to be + # updated for. + condition = yamldecode(helm_release.clickhouse_ddl.values[0]).tenant.grants == [ + "SELECT", + "INSERT", + "CREATE", + "DROP TABLE", + "ALTER UPDATE", + "ALTER DELETE", + "ALTER DROP INDEX", + ] + error_message = "the grant list of the tenant's ClickHouse user changed; it has to be exactly what Langfuse uses on its own database" + } + + assert { + # golang-migrate creates tables inside a database that already exists and never creates one, so + # this right would only let a tenant create databases outside its own scope. + condition = !contains(yamldecode(helm_release.clickhouse_ddl.values[0]).tenant.grants, "CREATE DATABASE") + error_message = "the tenant's ClickHouse user must not be granted CREATE DATABASE" + } + + assert { + # Required as soon as the cluster has more than one replica, and harmless with one. + condition = yamldecode(helm_release.clickhouse_ddl.values[0]).clickhouse.ddlCluster == "default" + error_message = "the DDL has to name the cluster the shared ClickHouse answers as, because every statement runs ON CLUSTER" + } +} + +run "long_identifiers_keep_the_ddl_release_name_inside_the_helm_limit" { + command = plan + + variables { + workspace_identifier = "a-very-long-workspace-identifier-indeed1" + project_identifier = "a-very-long-project-identifier-for-tests" + } + + assert { + # Helm stops a release name at 53 characters. Every object of the release derives its name from the + # release name, so this limit is the tightest of the family and the one the budget is cut for. + condition = length(helm_release.clickhouse_ddl.name) <= 53 + error_message = "the name of the DDL release has to stay inside the 53 characters Helm allows" + } + + assert { + # The Job names are '-create' and '-drop', and a Job name ends up in a label + # value, which stops at 63 characters. + condition = length("${helm_release.clickhouse_ddl.name}-create") <= 63 + error_message = "the Job names derived from the release name have to stay inside the 63 characters a label value allows" + } + + assert { + condition = endswith(helm_release.clickhouse_ddl.name, substr(sha256("a-very-long-workspace-identifier-indeed1.a-very-long-project-identifier-for-tests"), 0, 8)) + error_message = "the truncated release name has to end in the hash of the untruncated identifier pair, or two long identifier pairs collide on one release" + } +} + +# --- The sibling tenant of the same meshProject ------------------------------------------------- + +run "the_namespace_of_the_secret_comes_from_the_platform_tenant_id" { + command = plan + + assert { + condition = kubernetes_secret_v1.model_access.metadata[0].namespace == "acme-payments" + error_message = "the Secret has to go into the namespace the sibling tenant reports as its platform tenant id" + } + + assert { + condition = output.secret_namespace == "acme-payments" + error_message = "the reported namespace has to be the one the Secret was written into" + } + + assert { + condition = kubernetes_secret_v1.model_access.metadata[0].name == "ai-model-access" + error_message = "the Secret has to carry the name the platform team configured" + } +} + +run "no_sibling_tenant_stops_the_run" { + command = plan + + override_data { + target = data.meshstack_tenants.sibling + values = { + tenants = [] + } + } + + expect_failures = [resource.kubernetes_secret_v1.model_access] +} + +run "two_sibling_tenants_stop_the_run" { + command = plan + + override_data { + target = data.meshstack_tenants.sibling + values = { + tenants = [ + { + metadata = { owned_by_project = "payments", owned_by_workspace = "acme", uuid = "11111111-1111-1111-1111-111111111111" } + ref = { kind = "meshTenant", uuid = "11111111-1111-1111-1111-111111111111" } + spec = { + landing_zone_ref = { kind = "meshLandingZone", name = "namespace" } + platform_ref = { kind = "meshPlatform", uuid = "22222222-2222-2222-2222-222222222222" } + platform_tenant_id = "acme-payments" + quotas = [] + requested_quotas = null + } + status = { + applied_quotas = null + platform_type_identifier = "Kubernetes" + platform_workspace_id = "acme" + tags = {} + tenant_name = "acme.payments" + } + }, + { + metadata = { owned_by_project = "payments", owned_by_workspace = "acme", uuid = "33333333-3333-3333-3333-333333333333" } + ref = { kind = "meshTenant", uuid = "33333333-3333-3333-3333-333333333333" } + spec = { + landing_zone_ref = { kind = "meshLandingZone", name = "namespace" } + platform_ref = { kind = "meshPlatform", uuid = "22222222-2222-2222-2222-222222222222" } + platform_tenant_id = "acme-payments-second" + quotas = [] + requested_quotas = null + } + status = { + applied_quotas = null + platform_type_identifier = "Kubernetes" + platform_workspace_id = "acme" + tags = {} + tenant_name = "acme.payments.second" + } + }, + ] + } + } + + expect_failures = [resource.kubernetes_secret_v1.model_access] +} + +run "a_sibling_tenant_without_a_platform_tenant_id_stops_the_run" { + command = plan + + # meshStack has accepted the tenant but has not replicated it yet, so its namespace does not exist. + # This is the dangerous case: a null namespace would make the Kubernetes provider fall back to the + # namespace of the kubeconfig's current context. + override_data { + target = data.meshstack_tenants.sibling + values = { + tenants = [ + { + metadata = { owned_by_project = "payments", owned_by_workspace = "acme", uuid = "11111111-1111-1111-1111-111111111111" } + ref = { kind = "meshTenant", uuid = "11111111-1111-1111-1111-111111111111" } + spec = { + landing_zone_ref = { kind = "meshLandingZone", name = "namespace" } + platform_ref = { kind = "meshPlatform", uuid = "22222222-2222-2222-2222-222222222222" } + platform_tenant_id = null + quotas = [] + requested_quotas = null + } + status = { + applied_quotas = null + platform_type_identifier = "Kubernetes" + platform_workspace_id = "acme" + tags = {} + tenant_name = "acme.payments" + } + }, + ] + } + } + + expect_failures = [resource.kubernetes_secret_v1.model_access] +} + +# --- The credential never leaves the run -------------------------------------------------------- + +run "no_output_and_no_summary_carries_the_credential" { + command = apply + + assert { + # The Secret is the one place the credential is written to, so the test would be meaningless if + # it were not there. + condition = nonsensitive(kubernetes_secret_v1.model_access.data["OPENAI_API_KEY"]) == nonsensitive(litellm_key.this.key) + error_message = "the Secret has to carry the credential, otherwise the workload has no way to reach the endpoint" + } + + assert { + condition = nonsensitive(kubernetes_secret_v1.model_access.data["OPENAI_BASE_URL"]) == "https://litellm.example.com/v1" + error_message = "the Secret has to carry the endpoint next to the credential" + } + + assert { + condition = !strcontains(output.summary, nonsensitive(litellm_key.this.key)) + error_message = "the summary is published in meshPanel, so it must not contain the credential" + } + + assert { + condition = alltrue([ + for value in [ + output.team_id, + output.team_alias, + output.key_id, + output.api_base, + output.secret_name, + output.secret_namespace, + output.langfuse_url, + output.langfuse_namespace, + output.langfuse_oidc_callback_url, + output.summary, + ] : !strcontains(value, nonsensitive(litellm_key.this.key)) + ]) + error_message = "no output of this module may contain the credential, because a building block output cannot be sensitive" + } + + assert { + condition = strcontains(output.summary, "ai-model-access") && strcontains(output.summary, "acme-payments") + error_message = "the summary has to tell the application team where the credential is: the Secret by name, in its namespace" + } +} diff --git a/modules/ai/model-access/buildingblock/naming.tf b/modules/ai/model-access/buildingblock/naming.tf new file mode 100644 index 00000000..c39f5090 --- /dev/null +++ b/modules/ai/model-access/buildingblock/naming.tf @@ -0,0 +1,92 @@ +# Every per-tenant name the tenant's Langfuse instance needs is derived here, from the tenant +# context alone. `modules/ai/langfuse` takes each of them as an explicit input, because it is +# instantiated once per tenant against shared backends and two tenants that collide on a name share +# their data. +# +# This module creates the Postgres database, the ClickHouse database and the bucket under these +# names, so a name that changes takes the resource with it: Terraform replaces the database, the user +# or the bucket, and the data in it is gone. Treat every expression here as part of the module's +# contract. + +locals { + # A project identifier alone can repeat across workspaces, so every derived name starts from the + # pair. The pair is unique in meshStack. + tenant_key = "${var.workspace_identifier}.${var.project_identifier}" + + # Truncation is unavoidable: Postgres and ClickHouse identifiers, DNS labels, Kubernetes + # namespaces and bucket names all stop at 63 characters, while the two meshStack identifiers + # together can be longer. A hash of the untruncated pair is appended to every name, so two long + # identifier pairs that share a prefix stay apart after truncation. + tenant_hash = substr(sha256(local.tenant_key), 0, 8) + + # DNS labels and bucket names allow lowercase letters, digits and dashes. Every run of other + # characters collapses into a single dash, and leading and trailing dashes are dropped. + tenant_slug_dns = replace(replace(lower(local.tenant_key), "/[^a-z0-9]+/", "-"), "/^-+|-+$/", "") + + # Postgres and ClickHouse identifiers are case-folded and may not start with a digit, so the SQL + # form uses underscores and every name built from it carries a fixed prefix. + tenant_slug_sql = replace(replace(lower(local.tenant_key), "/[^a-z0-9]+/", "_"), "/^_+|_+$/", "") + + # 63 characters, minus the longest prefix any derived name carries ('langfuse' plus a separator), + # minus the separator and the hash at the end. + slug_budget = 63 - length("langfuse") - 1 - 1 - length(local.tenant_hash) + + # substr fails when the requested length runs past the end of the string, so the length is capped + # first. Truncation can leave a trailing separator, which is dropped so the hash never follows a + # doubled separator. + tenant_slug_dns_short = replace(substr(local.tenant_slug_dns, 0, min(length(local.tenant_slug_dns), local.slug_budget)), "/-+$/", "") + tenant_slug_sql_short = replace(substr(local.tenant_slug_sql, 0, min(length(local.tenant_slug_sql), local.slug_budget)), "/_+$/", "") + + # One label, used both as the Kubernetes namespace in the AI platform cluster and as the first + # label of the hostname. Keeping them identical means an operator who has the hostname knows the + # namespace. + langfuse_label = "langfuse-${local.tenant_slug_dns_short}-${local.tenant_hash}" + langfuse_namespace = local.langfuse_label + langfuse_hostname = "${local.langfuse_label}.${var.langfuse_domain}" + langfuse_url = "https://${local.langfuse_hostname}" + + # Separation in Postgres and in ClickHouse is a database per tenant. Both carry the same name, so + # an operator reads one name in two places. + langfuse_postgres_database = "langfuse_${local.tenant_slug_sql_short}_${local.tenant_hash}" + langfuse_clickhouse_database = "langfuse_${local.tenant_slug_sql_short}_${local.tenant_hash}" + + # The owner of the Postgres database and the tenant's ClickHouse user carry the name of the database + # they are scoped to, which is what an owner is usually called in both systems. Both names have to + # be per-tenant: a Postgres role and a ClickHouse user are objects of the whole server, so two + # tenants sharing a user share every grant that user holds. + langfuse_postgres_username = local.langfuse_postgres_database + langfuse_clickhouse_username = local.langfuse_clickhouse_database + + # The Kubernetes objects that run the ClickHouse DDL live in the shared ClickHouse namespace, so + # each one carries the tenant in its name. Helm stops a release name at 53 characters, the tightest + # limit of the family, and every object of the release derives its name from the release name plus a + # suffix of at most seven characters, which keeps a Job name inside the 63 characters a label value + # may carry. + clickhouse_ddl_slug_budget = 53 - length("langfuse") - 1 - 1 - length(local.tenant_hash) + clickhouse_ddl_slug_short = replace(substr(local.tenant_slug_dns, 0, min(length(local.tenant_slug_dns), local.clickhouse_ddl_slug_budget)), "/-+$/", "") + clickhouse_ddl_name = "langfuse-${local.clickhouse_ddl_slug_short}-${local.tenant_hash}" + + # One bucket per tenant. The three kinds of upload separate by prefix inside it. + langfuse_bucket = "langfuse-${local.tenant_slug_dns_short}-${local.tenant_hash}" + + # Valkey needs both a key prefix and a database index, and the two carry different guarantees. + # + # The prefix is unique per tenant, because it is built from the same slug and hash as every other + # name here. It is what actually keeps two tenants apart: BullMQ queue names are hardcoded in + # Langfuse, so two tenants sharing a keyspace without a prefix would have one tenant's worker + # consume the other tenant's ingestion jobs. + # + # The index is a hard namespace that no application bug can cross, but there are only as many + # indices as the instance serves, so it is derived by hash and repeats once there are more tenants + # than indices. It is defence in depth behind the prefix, never the separation itself. + langfuse_valkey_key_prefix = "${local.tenant_slug_dns_short}-${local.tenant_hash}:" + langfuse_valkey_database = parseint(local.tenant_hash, 16) % var.langfuse_valkey_database_count + + # The instance holds exactly one organisation and one project, so the identifiers can be the plain + # meshStack identifiers: the workspace is the organisation, the project is the project. They are + # not truncated, because Langfuse puts no length limit on either. + langfuse_org_id = var.workspace_identifier + langfuse_org_name = var.workspace_identifier + langfuse_project_id = var.project_identifier + langfuse_project_name = var.project_identifier +} diff --git a/modules/ai/model-access/buildingblock/outputs.tf b/modules/ai/model-access/buildingblock/outputs.tf new file mode 100644 index 00000000..d428e898 --- /dev/null +++ b/modules/ai/model-access/buildingblock/outputs.tf @@ -0,0 +1,89 @@ +# The virtual key is deliberately absent from this file. It is created in this run, written into the +# Secret in this run, and never returned: a building block output cannot be sensitive, because +# `version_spec.outputs` of meshstack_building_block_definition has no `sensitive` block, unlike +# `version_spec.inputs`. Not returning it at all is what keeps it inside the run. + +output "team_id" { + value = litellm_team.this.id + description = "ID of the LiteLLM team. meshStack uses it as the platform tenant ID, so later building blocks can bind resources to this team." +} + +output "team_alias" { + value = litellm_team.this.team_alias + description = "Alias of the LiteLLM team, shown in the LiteLLM UI." +} + +output "key_id" { + value = litellm_key.this.id + description = "Hash of the virtual key. LiteLLM identifies the key by this value, and it is safe to show in logs." +} + +output "api_base" { + value = local.api_base + description = "OpenAI-compatible base URL of the gateway, including the '/v1' suffix. The same value is written into the Secret." +} + +output "secret_name" { + value = kubernetes_secret_v1.model_access.metadata[0].name + description = "Name of the Kubernetes Secret that carries the model credential and the endpoint." +} + +output "secret_namespace" { + value = kubernetes_secret_v1.model_access.metadata[0].namespace + description = "Namespace the Secret was written into. It is the platform tenant id of the sibling tenant of this meshProject, which for a Kubernetes tenant is the namespace name." +} + +output "langfuse_url" { + value = local.langfuse_url + description = "URL of the tenant's own Langfuse instance. meshStack publishes it as the sign-in URL of this building block." +} + +output "langfuse_namespace" { + value = module.langfuse.namespace + description = "Namespace the tenant's Langfuse instance runs in, in the AI platform cluster." +} + +output "langfuse_oidc_callback_url" { + value = module.langfuse.oidc_callback_url + description = "Callback URL to register at the identity provider for this tenant's Langfuse instance." +} + +# The names of the per-tenant resources in the four shared backends. The Postgres and the bucket names +# are read back off the resources that carry them rather than off the locals they were derived from, so +# the output describes what the run created. The ClickHouse pair comes from the derivation, because +# ClickHouse has no Terraform resource here and the names travel into the DDL Job as Helm values. +output "langfuse_backend_names" { + description = "The per-tenant names the tenant's Langfuse instance uses in the four shared backends. The module creates the Postgres database and its owner user, the ClickHouse database and its user, and the bucket; Valkey is separated by key prefix and database index alone." + value = { + postgres_database = module.postgres.database_name + postgres_username = module.postgres.username + clickhouse_database = local.langfuse_clickhouse_database + clickhouse_username = local.langfuse_clickhouse_username + bucket = module.bucket.bucket_name + valkey_key_prefix = local.langfuse_valkey_key_prefix + valkey_database = local.langfuse_valkey_database + } +} + +# meshStack publishes this value in meshPanel through the `SUMMARY` assignment type, so the template +# must not contain the virtual key. It carries the key ID instead, which is a hash of the key, and it +# names the Secret the key was delivered in. +output "summary" { + description = "Summary with the endpoint, the Secret the credential was delivered in, the Langfuse URL and the budget. It does not contain the virtual key." + value = templatefile("${path.module}/SUMMARY.md.tftpl", { + api_base = local.api_base + team_alias = litellm_team.this.team_alias + team_id = litellm_team.this.id + key_id = litellm_key.this.id + # Read from the Secret rather than from the inputs it was built from, so the summary can only + # describe a Secret that was actually written. + secret_name = kubernetes_secret_v1.model_access.metadata[0].name + secret_namespace = kubernetes_secret_v1.model_access.metadata[0].namespace + secret_key_env = local.secret_keys.api_key + base_url_env = local.secret_keys.api_base + langfuse_url = local.langfuse_url + max_budget = var.max_budget + budget_duration = var.budget_duration + models = var.models + }) +} diff --git a/modules/ai/model-access/buildingblock/postgres.tf b/modules/ai/model-access/buildingblock/postgres.tf new file mode 100644 index 00000000..8393f8f0 --- /dev/null +++ b/modules/ai/model-access/buildingblock/postgres.tf @@ -0,0 +1,26 @@ +# The tenant's Postgres database and its owner user, inside the PostgreSQL Flex instance every tenant +# shares. `modules/stackit/postgresflex/buildingblock/database` was split out of that building block +# provider-free for exactly this call: it declares no provider of its own, so it inherits the +# `stackit` provider configured in provider.tf, and a module that carries its own provider +# configuration could not be called with `count` or `for_each` by any composition at all. +# +# `existing_instance_id` selects the submodule's database-only mode. It then creates +# `stackit_postgresflex_database` and `stackit_postgresflex_user` against the instance and touches +# nothing about the instance itself, which is what keeps tenant churn out of the shared server: every +# input describing the instance shape has no effect in that mode. +module "postgres" { + source = "github.com/meshcloud/meshstack-hub//modules/stackit/postgresflex/buildingblock/database?ref=${var.hub.git_ref}" + + project_id = var.stackit_project_id + stackit_region = local.stackit_region + + existing_instance_id = var.langfuse_postgres_instance_id + + database_name = local.langfuse_postgres_database + database_username = local.langfuse_postgres_username + + # `login` and nothing else. Langfuse runs its Postgres migrations with `prisma migrate deploy`, + # which creates and alters tables inside a database the user owns and never creates a database, so + # `createdb` would let a tenant create databases outside its own scope. + database_user_roles = ["login"] +} diff --git a/modules/ai/model-access/buildingblock/provider.tf b/modules/ai/model-access/buildingblock/provider.tf new file mode 100644 index 00000000..ea413005 --- /dev/null +++ b/modules/ai/model-access/buildingblock/provider.tf @@ -0,0 +1,112 @@ +# A `buildingblock` directory is a meshStack root module: the meshStack Terraform runner checks it +# out and runs `tofu plan` and `tofu apply` inside it. Provider configuration therefore belongs +# here, and this module configures several providers because it touches several systems in one run. + +locals { + # STACKIT serves PostgreSQL Flex and Object Storage from one endpoint per region, and this module + # fixes the region at eu01 rather than taking it as an input. That mirrors + # `modules/stackit/storage-bucket/buildingblock/provider.tf`, which fixes the same value for the + # same reason: the S3 endpoint URL carries the region, so the two cannot be set apart. + stackit_region = "eu01" + stackit_s3_endpoint = "https://object.storage.${local.stackit_region}.onstackit.cloud" + + ai_platform_kubeconfig = yamldecode(var.ai_platform_cluster_kubeconfig) + ai_platform_kubeconfig_cluster = one(local.ai_platform_kubeconfig["clusters"])["cluster"] + ai_platform_kubeconfig_user = one(local.ai_platform_kubeconfig["users"])["user"] + + demo_app_kubeconfig = yamldecode(var.demo_app_cluster_kubeconfig) + demo_app_kubeconfig_cluster = one(local.demo_app_kubeconfig["clusters"])["cluster"] + demo_app_kubeconfig_user = one(local.demo_app_kubeconfig["users"])["user"] +} + +provider "litellm" { + api_base = var.litellm_api_base + api_key = var.litellm_api_key +} + +# STACKIT, where the tenant's Postgres database, its owner user and its bucket are created. Both +# submodules this module sources for that declare no provider of their own, so they inherit this one. +provider "stackit" { + # The Object Storage resources of the bucket submodule take the region as an optional attribute and + # fall back to this value, so it is what puts them in eu01. The PostgreSQL Flex resources take it + # explicitly, from `local.stackit_region` as well. + default_region = local.stackit_region + + # A key, not workload identity federation. See the STACKIT credential section of the module README + # for why, and for what it would take to move. + service_account_key = var.stackit_service_account_key +} + +# The `aws` provider is configured as a generic S3 client against the STACKIT Object Storage +# endpoint. The bucket is created with it and not with the stackit provider, because the stackit +# provider has no permission to create a bucket. This block mirrors +# `modules/stackit/storage-bucket/buildingblock/provider.tf`, including the region. +provider "aws" { + access_key = var.stackit_s3_admin_access_key + secret_key = var.stackit_s3_admin_secret_access_key + region = local.stackit_region + + endpoints { + s3 = local.stackit_s3_endpoint + } + + # STACKIT Object Storage is StorageGRID behind an S3 API, so every check the provider would run + # against AWS itself has to be skipped: there is no STS to validate the credential with, 'eu01' is + # not an AWS region name, there is no account id to request and there is no instance metadata + # service. Buckets are addressed as a path on the endpoint, because virtual-hosted style would + # need a DNS record per bucket. + skip_credentials_validation = true + skip_region_validation = true + skip_requesting_account_id = true + skip_metadata_api_check = true + s3_use_path_style = true +} + +# The AI platform cluster. The tenant's Langfuse instance is deployed here, next to the shared +# LiteLLM gateway and the shared ClickHouse. +provider "kubernetes" { + alias = "ai_platform" + + host = local.ai_platform_kubeconfig_cluster["server"] + cluster_ca_certificate = base64decode(local.ai_platform_kubeconfig_cluster["certificate-authority-data"]) + + # A service account token is what the platform team hands out. The certificate pair is accepted + # as well, so a kubeconfig produced by a cluster's own credential API also works. + token = try(local.ai_platform_kubeconfig_user["token"], null) + client_certificate = try(base64decode(local.ai_platform_kubeconfig_user["client-certificate-data"]), null) + client_key = try(base64decode(local.ai_platform_kubeconfig_user["client-key-data"]), null) +} + +# Helm talks to the same control plane with the same credentials, so the Langfuse namespace, its +# secret and its release all land in the AI platform cluster. +provider "helm" { + alias = "ai_platform" + + kubernetes = { + host = local.ai_platform_kubeconfig_cluster["server"] + cluster_ca_certificate = base64decode(local.ai_platform_kubeconfig_cluster["certificate-authority-data"]) + + token = try(local.ai_platform_kubeconfig_user["token"], null) + client_certificate = try(base64decode(local.ai_platform_kubeconfig_user["client-certificate-data"]), null) + client_key = try(base64decode(local.ai_platform_kubeconfig_user["client-key-data"]), null) + } +} + +# The demo application cluster, a different cluster with a different credential. Only the Secret +# carrying the model credential is written here. See the residual risk section of the module README +# for what this credential may do. +provider "kubernetes" { + alias = "demo_app" + + host = local.demo_app_kubeconfig_cluster["server"] + cluster_ca_certificate = base64decode(local.demo_app_kubeconfig_cluster["certificate-authority-data"]) + + token = try(local.demo_app_kubeconfig_user["token"], null) + client_certificate = try(base64decode(local.demo_app_kubeconfig_user["client-certificate-data"]), null) + client_key = try(base64decode(local.demo_app_kubeconfig_user["client-key-data"]), null) +} + +# meshStack injects the credentials of an ephemeral API token into the run environment, so the +# provider needs no configuration. The token carries the permissions listed in +# `version_spec.permissions` of the building block definition, which is `TENANT_LIST` here. +provider "meshstack" {} diff --git a/modules/ai/model-access/buildingblock/variables.tf b/modules/ai/model-access/buildingblock/variables.tf new file mode 100644 index 00000000..b6195edb --- /dev/null +++ b/modules/ai/model-access/buildingblock/variables.tf @@ -0,0 +1,311 @@ +# --- meshStack tenant context ------------------------------------------------------------------- +# +# Every variable in this section is filled by meshStack from the tenant the run belongs to, through +# an assignment type. None of them is a USER_INPUT, and none of them may become one: together with +# the STATIC values below they decide which namespace in which cluster receives the model +# credential. See the residual risk section of the module README. + +variable "workspace_identifier" { + type = string + description = "Identifier of the meshStack workspace the tenant belongs to. It becomes part of the LiteLLM team alias, part of every per-tenant name this module derives, and the identifier of the Langfuse organisation." +} + +variable "project_identifier" { + type = string + description = "Identifier of the meshStack project the tenant belongs to. It becomes part of the LiteLLM team alias, part of every per-tenant name this module derives, and the identifier of the Langfuse project." +} + +variable "meshstack_tenant_uuid" { + type = string + default = "" + description = "UUID of the meshStack tenant. It is written to the LiteLLM team metadata so an operator can trace a team back to its tenant." +} + +# --- The shared LiteLLM gateway ----------------------------------------------------------------- + +variable "litellm_api_base" { + type = string + description = "Base URL of the LiteLLM gateway, for example 'https://litellm.example.com'. The provider talks to the admin API under this URL." +} + +variable "litellm_api_key" { + type = string + sensitive = true + description = "LiteLLM admin key the provider authenticates with. It needs permission to create teams and keys." +} + +variable "models" { + type = list(string) + default = [] + description = "Names of the models on the LiteLLM gateway that the team may call. An empty list sends no allow-list to LiteLLM." +} + +variable "max_budget" { + type = number + default = 100 + description = "Spending limit of the team for one budget period, in the currency the gateway reports spend in. LiteLLM blocks the team once the limit is reached." +} + +variable "budget_duration" { + type = string + default = "30d" + description = "Length of one budget period, after which LiteLLM resets the spend counter. Written as a LiteLLM duration such as '30d', '7d' or '1h'." +} + +# --- The two clusters --------------------------------------------------------------------------- + +variable "ai_platform_cluster_kubeconfig" { + type = string + sensitive = true + description = "kubeconfig of the AI platform cluster, as YAML. The tenant's Langfuse instance is deployed here. The credential needs permission to create a namespace, a secret and a Helm release." +} + +variable "demo_app_cluster_kubeconfig" { + type = string + sensitive = true + # `delete` is in the list because `version_spec.deletion_mode` is `DELETE`: destroying the building + # block destroys the Secret, and without the verb that destroy fails with a 403. See the residual + # risk section of the module README. + description = "kubeconfig of the demo application cluster, as YAML. Only the Secret with the model credential is written here, so the credential needs no more than `get`, `create`, `update`, `patch` and `delete` on secrets." +} + +variable "demo_app_platform_identifier" { + type = string + description = "Full identifier of the meshStack platform the demo application cluster is registered as, in the form '.'. The module looks up the sibling tenant of this platform in the same meshProject to learn the namespace it writes the Secret into. This is the platform identifier, never the platform type: two clusters of the same type would both match a type filter." + + validation { + condition = length(split(".", var.demo_app_platform_identifier)) >= 2 + error_message = "demo_app_platform_identifier must be the full platform identifier '.', for example 'kubernetes.eu01'." + } +} + +variable "secret_name" { + type = string + default = "ai-model-access" + description = "Name of the Kubernetes Secret this module writes into the namespace of the application team. The workload reads the model credential and the endpoint from it." +} + +# --- The tenant's Langfuse instance ------------------------------------------------------------- + +variable "langfuse_domain" { + type = string + description = "Domain the tenant's Langfuse instance is published under. The module prepends the derived per-tenant label, so the instance answers on '.'. Give a bare domain without a scheme and without a leading dot." + + validation { + condition = !strcontains(var.langfuse_domain, "://") && !startswith(var.langfuse_domain, ".") + error_message = "langfuse_domain must be a bare domain without a scheme and without a leading dot, for example 'ai.example.com'." + } +} + +variable "langfuse_ingress_class_name" { + type = string + default = "haproxy" + description = "Name of the IngressClass that serves the tenants' Langfuse instances in the AI platform cluster. The default matches the controller `modules/kubernetes/ingress` installs. The Ingress carries no tls block, because that controller serves a wildcard certificate as its default." +} + +variable "langfuse_default_org_role" { + type = string + default = "MEMBER" + # Langfuse upserts an organisation membership with this role for every user who logs in, which is + # what removes member synchronisation from the picture. It also means that with one OIDC client + # shared across tenants, every user the identity provider authenticates reaches every tenant's + # instance with this role. See the access control section of the module README. + description = "Role every user who logs in through the identity provider receives in the tenant's Langfuse organisation. Set 'NONE' to hand out a membership that grants nothing, which is how to turn auto-join off when one OIDC client is shared across tenants." + + validation { + condition = contains(["OWNER", "ADMIN", "MEMBER", "VIEWER", "NONE"], var.langfuse_default_org_role) + error_message = "langfuse_default_org_role must be one of OWNER, ADMIN, MEMBER, VIEWER or NONE." + } +} + +variable "oidc" { + description = <<-EOT + OIDC identity provider the tenant's Langfuse instance trusts. It is required: this building block + creates no password user, because a password nobody can read is of no use, and a building block + output cannot carry one. + + Register the callback URL `https://./api/auth/callback/custom` at + the provider. The `langfuse_oidc_callback_url` output reports the exact URL of this tenant. + EOT + + type = object({ + issuer_url = string + client_id = string + client_secret = string + display_name = optional(string, "Single Sign-On") + scopes = optional(string, "openid email profile") + allow_account_linking = optional(bool, true) + }) + + sensitive = true +} + +# --- STACKIT, where the per-tenant backends are created ----------------------------------------- +# +# The module creates the tenant's Postgres database, its owner user and its bucket, so it is bound to +# STACKIT. That is deliberate: `modules/ai/` groups a capability rather than promising +# cloud-agnosticism, and `modules/ai/azure-openai` sits in the same directory. An Azure twin of this +# module would repeat the shape with Azure submodules. + +variable "stackit_project_id" { + type = string + description = "STACKIT project the shared PostgreSQL Flex instance and the tenants' buckets live in. The module creates one database and one bucket per tenant inside it." +} + +variable "stackit_service_account_key" { + type = string + sensitive = true + # The hub convention for modules under modules/stackit/ is workload identity federation, which + # needs a backplane that registers a federated identity provider for this definition. This module + # has no backplane, so it takes the key. See the STACKIT credential section of the module README. + description = "Service account key of the STACKIT service account, as the JSON STACKIT returns when the key is created, including the private key. The account needs permission to create databases and users on the PostgreSQL Flex instance and to create Object Storage credentials groups in the project." +} + +variable "stackit_s3_admin_access_key" { + type = string + sensitive = true + # The bucket itself is created with the `aws` provider used as a generic S3 client, because the + # stackit provider has no permission to create one. modules/stackit/storage-bucket does the same. + description = "Access key of the administrative Object Storage credential the bucket is created with. It has to belong to the credentials group named by `stackit_s3_admin_credentials_group_urn`, otherwise the bucket policy locks the run out of the bucket it just created." +} + +variable "stackit_s3_admin_secret_access_key" { + type = string + sensitive = true + description = "Secret access key of the administrative Object Storage credential." +} + +variable "stackit_s3_admin_credentials_group_urn" { + type = string + description = "URN of the administrative Object Storage credentials group, in the form 'urn:sgws:identity:::group/'. The bucket policy keeps access for this group, so the platform team can still reach a tenant's bucket." +} + +# --- The backends the Langfuse instances share -------------------------------------------------- +# +# Each of the four backends is shared across tenants and separated by name: a database, a bucket, a +# key prefix. This module derives those names from the tenant context. It creates the Postgres +# database, the ClickHouse database and the bucket; the shared servers themselves, and the credentials +# it creates those resources with, arrive as STATIC inputs. + +variable "langfuse_postgres_instance_id" { + type = string + description = "UUID of the shared STACKIT PostgreSQL Flex instance. The module creates the tenant's database and its owner user inside it and never creates or changes the instance itself." +} + +variable "langfuse_clickhouse_host" { + type = string + description = "Fully qualified hostname of the shared ClickHouse cluster, without a scheme. Take it from the `host` output of modules/ai/clickhouse." +} + +variable "langfuse_clickhouse_native_port" { + type = number + default = 9000 + # The `native_port` output of modules/ai/clickhouse. Langfuse's golang-migrate migrations and the + # clickhouse-client in the DDL Job both speak the native protocol. + description = "Native protocol port of the shared ClickHouse cluster. The DDL Job and the Langfuse migrations both connect over it." +} + +variable "langfuse_clickhouse_namespace" { + type = string + default = "clickhouse" + # The `namespace` output of modules/ai/clickhouse. The DDL Job runs here rather than in the tenant's + # own namespace, because it has to run before the tenant's Langfuse namespace exists. + description = "Namespace of the shared ClickHouse cluster in the AI platform cluster. The DDL Job of this tenant runs in it and mounts the administrative password from the Secret in it." +} + +variable "langfuse_clickhouse_ddl_cluster_name" { + type = string + default = "default" + # The `ddl_cluster_name` output of modules/ai/clickhouse. Every ON CLUSTER statement names it, the + # Langfuse migrations included. + description = "Name of the ClickHouse cluster as the server knows it. The DDL of this module runs ON CLUSTER with this name, which is required as soon as the cluster has more than one replica and harmless with one." +} + +variable "langfuse_clickhouse_admin_username" { + type = string + default = "default" + # The `admin_username` output of modules/ai/clickhouse. The operator only manages the 'default' + # user, so this is the name in practice. + description = "Name of the administrative ClickHouse user the DDL Job authenticates as. It creates the tenant's database and user, so it must never be handed to a tenant." +} + +variable "langfuse_clickhouse_admin_secret_name" { + type = string + default = "clickhouse-admin" + # The `admin_secret` output of modules/ai/clickhouse names it. Mounting the Secret keeps the + # administrative password out of the building block definition entirely. + description = "Name of the Kubernetes Secret in the ClickHouse namespace that holds the administrative password. The DDL Job reads the password from it instead of taking it as an input." +} + +variable "langfuse_clickhouse_admin_secret_key" { + type = string + default = "password" + description = "Key inside the administrative Secret that holds the password." +} + +variable "langfuse_clickhouse_client_image" { + type = string + default = "clickhouse/clickhouse-server:26.4" + # The same image the servers run, so the node has it in its cache and no second image is pulled. + # Keep the tag on the `clickhouse_version` of modules/ai/clickhouse. + description = "Image the DDL Job runs `clickhouse-client` from. Keep it on the same tag the shared cluster runs, so no second image has to be pulled." +} + +variable "langfuse_clickhouse_ddl_timeout" { + type = number + default = 600 + # Bounds both Jobs. Without a deadline a ClickHouse that never answers turns into a Terraform + # timeout with no log instead of a failed Job with one. + description = "Seconds each ClickHouse DDL Job may run before Kubernetes fails it. It covers waiting for the cluster to answer a query and running the statements." + + validation { + condition = var.langfuse_clickhouse_ddl_timeout >= 30 + error_message = "langfuse_clickhouse_ddl_timeout must be at least 30 seconds." + } +} + +variable "langfuse_valkey_host" { + type = string + description = "Hostname of the shared Valkey instance. Langfuse uses it as the queue backend, the cache and the rate limit store." +} + +variable "langfuse_valkey_password" { + type = string + sensitive = true + description = "Password of the Valkey instance. Use only characters that are safe in a URL, because the chart substitutes the value into the connection URL without encoding it." +} + +variable "langfuse_valkey_database_count" { + type = number + default = 16 + # The index is derived from a hash of the tenant, so it repeats once there are more tenants than + # indices. The key prefix is what keeps two tenants apart in that case, see naming.tf. + description = "Number of Valkey database indices the instance serves. The module derives the tenant's index from a hash of the tenant, modulo this number. A stock Valkey serves 16 indices, numbered 0 to 15." + + validation { + condition = var.langfuse_valkey_database_count >= 1 + error_message = "langfuse_valkey_database_count must be at least 1." + } +} + +# The endpoint of the object storage and the credential scoped to the tenant's bucket are not inputs. +# The bucket submodule creates a credentials group and a credential per bucket and reports both, so +# the tenant's Langfuse instance gets a credential that its own bucket policy scopes to its own +# bucket. + +# --- Hub reference ------------------------------------------------------------------------------ + +variable "hub" { + type = object({ + git_ref = optional(string, "main") + }) + const = true + default = { git_ref = "main" } + + description = <<-EOT + `git_ref`: meshstack-hub reference this module sources `modules/ai/langfuse/buildingblock` from. + `const` so it can be interpolated into the module source at init time. The building block + definition passes the same reference it checks this module out at, so the tenant's Langfuse comes + from the release this building block was published from. + EOT +} diff --git a/modules/ai/model-access/buildingblock/versions.tf b/modules/ai/model-access/buildingblock/versions.tf new file mode 100644 index 00000000..801183e5 --- /dev/null +++ b/modules/ai/model-access/buildingblock/versions.tf @@ -0,0 +1,57 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + litellm = { + source = "ncecere/litellm" + # Exact pin, a deliberate exception to the hub rule that provider constraints use '>='. + # ncecere/litellm is a community provider with a single maintainer, and it has changed + # resource behaviour inside a minor release before: v1.2.0 replaced the id of litellm_key + # with a hash of the key. LiteLLM returns a virtual key once and never again, so a provider + # change that recreates the key takes the credential away from a running application. The + # version is therefore pinned here and raised deliberately after a review of the changelog. + version = "= 2.0.1" + } + + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38" + } + + helm = { + source = "hashicorp/helm" + # The helm provider takes its cluster credentials as the `kubernetes = {}` attribute + # starting with 3.0.0. Earlier versions expect a `kubernetes {}` block instead. + version = ">= 3.0.0" + } + + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.24.0" + } + + stackit = { + source = "stackitcloud/stackit" + # The floor of modules/stackit/postgresflex/buildingblock/database, which this module sources. + # Earlier versions do not carry the v3 PostgreSQL Flex schema. + version = ">= 0.110.0" + } + + aws = { + source = "hashicorp/aws" + # An upper bound, and the second deliberate exception to the hub rule that provider constraints + # use '>='. It is not a choice made here: modules/stackit/storage-bucket/buildingblock/bucket + # carries the same constraint, because v5 of this provider always sends LocationConstraint in + # CreateBucket while STACKIT's StorageGRID only accepts a request without it. A wider + # constraint here would not intersect with the submodule's and init would fail. + version = ">= 4.0, < 5.0" + } + + random = { + source = "hashicorp/random" + # random_bytes arrived in 3.5.0. Its `hex` and `base64` attributes are marked sensitive, + # unlike those of random_id, which keeps the generated Langfuse secrets out of the plan. + version = ">= 3.5.0" + } + } +} diff --git a/modules/ai/model-access/definition.tftest.hcl b/modules/ai/model-access/definition.tftest.hcl new file mode 100644 index 00000000..bb747cdb --- /dev/null +++ b/modules/ai/model-access/definition.tftest.hcl @@ -0,0 +1,135 @@ +variables { + litellm_api_base = "https://litellm.example.com" + litellm_admin_api_key = "sk-admin-mock" + + ai_platform_cluster_kubeconfig = "current-context: ai-platform\n" + demo_app_cluster_kubeconfig = "current-context: demo-app\n" + demo_app_platform_identifier = "kubernetes.eu01" + + langfuse_domain = "ai.example.com" + + stackit_project_id = "6e8c1f30-6c4d-4b1f-9f7a-2c9d8e5f1a2b" + stackit_service_account_key = "{\"id\":\"mock-key\"}" + stackit_s3_admin_access_key = "AKIAMOCKADMINKEY" + stackit_s3_admin_secret_access_key = "mock-admin-secret-access-key" + stackit_s3_admin_credentials_group_urn = "urn:sgws:identity::12345678901234567890:group/mock-admin-group" + + langfuse_postgres_instance_id = "3f2504e0-4f89-11d3-9a0c-0305e82c3301" + + langfuse_clickhouse_host = "clickhouse-clickhouse-headless.clickhouse.svc.cluster.local" + + langfuse_valkey_host = "valkey.valkey.svc.cluster.local" + langfuse_valkey_password = "valkey-password" + + oidc_issuer_url = "https://idp.example.com/realms/ai" + oidc_client_id = "langfuse" + oidc_client_secret = "mock-client-secret" + + meshstack = { + owning_workspace_identifier = "platform-team" + } +} + +mock_provider "meshstack" {} + +run "the_definition_is_a_mandatory_landing_zone_block" { + command = plan + + assert { + # Named for the capability. The products behind it are not part of what the application team + # ordered, so neither may appear in the name. + condition = meshstack_building_block_definition.this.spec.display_name == "AI Model Access" + error_message = "the display name has to name the capability, not the products that deliver it" + } + + assert { + condition = meshstack_building_block_definition.this.spec.target_type == "TENANT_LEVEL" + error_message = "the block runs on the AI model tenant, so it has to be TENANT_LEVEL" + } + + assert { + condition = meshstack_building_block_definition.this.spec.use_in_landing_zones_only == true + error_message = "the application team never orders this block by hand, so it may only be used from a landing zone" + } + + assert { + condition = meshstack_building_block_definition.this.version_spec.only_apply_once_per_tenant == true + error_message = "a second apply in the same tenant would create a second team, a second credential and a second tracing instance" + } + + assert { + # The run looks up the sibling tenant of the same meshProject with an ephemeral API token. + condition = contains(meshstack_building_block_definition.this.version_spec.permissions, "TENANT_LIST") + error_message = "the run needs TENANT_LIST to look up the sibling tenant it delivers the credential to" + } + + assert { + condition = meshstack_building_block_definition.this.version_spec.implementation.terraform.repository_path == "modules/ai/model-access/buildingblock" + error_message = "the definition has to point at this module's buildingblock directory" + } +} + +run "no_input_asks_a_human_and_none_reaches_the_namespace_decision_from_the_tenant" { + command = plan + + assert { + # A mandatory block that stops to ask a human defeats its purpose, and API-driven tenant creation + # only succeeds when every input is defaulted or static. The four assignment types below are the + # only ones this definition may use: STATIC is fixed by the platform team, and the other three are + # injected by meshStack from the tenant, so the tenant cannot forge them. + condition = alltrue([ + for name, input in meshstack_building_block_definition.this.version_spec.inputs : + contains( + ["STATIC", "WORKSPACE_IDENTIFIER", "PROJECT_IDENTIFIER", "MESHSTACK_TENANT_UUID"], + input.assignment_type + ) + ]) + error_message = "every input has to be STATIC or assigned from tenant context, and no input may be a USER_INPUT" + } + + assert { + condition = alltrue([ + for name in ["workspace_identifier", "project_identifier", "demo_app_platform_identifier", "secret_name"] : + contains(keys(meshstack_building_block_definition.this.version_spec.inputs), name) + ]) + error_message = "the four inputs that decide which namespace receives the credential have to be declared" + } +} + +run "no_definition_output_can_carry_the_credential" { + command = plan + + assert { + # A building block output is always cleartext in meshPanel, because `version_spec.outputs` has no + # `sensitive` block. The set is asserted in full, so adding an output is a deliberate change that + # this test has to be updated for. + condition = toset(keys(meshstack_building_block_definition.this.version_spec.outputs)) == toset([ + "team_id", + "team_alias", + "key_id", + "api_base", + "secret_name", + "secret_namespace", + "langfuse_url", + "langfuse_namespace", + "langfuse_oidc_callback_url", + "summary", + ]) + error_message = "the set of definition outputs changed; every one of them is cleartext in meshPanel, so check that the new one carries no credential" + } + + assert { + condition = meshstack_building_block_definition.this.version_spec.outputs["team_id"].assignment_type == "PLATFORM_TENANT_ID" + error_message = "the team id is the platform tenant id, so building blocks ordered later can bind to it" + } + + assert { + condition = meshstack_building_block_definition.this.version_spec.outputs["langfuse_url"].assignment_type == "SIGN_IN_URL" + error_message = "the tracing URL is a login URL, which SIGN_IN_URL is the assignment type for" + } + + assert { + condition = meshstack_building_block_definition.this.version_spec.outputs["summary"].assignment_type == "SUMMARY" + error_message = "the summary has to be published as the summary of the building block" + } +} diff --git a/modules/ai/model-access/meshstack_integration.tf b/modules/ai/model-access/meshstack_integration.tf new file mode 100644 index 00000000..0a2dcd12 --- /dev/null +++ b/modules/ai/model-access/meshstack_integration.tf @@ -0,0 +1,811 @@ +variable "litellm_api_base" { + type = string + description = "Base URL of the LiteLLM gateway, for example 'https://litellm.example.com'. Every team and virtual key this definition creates lives on this gateway." +} + +variable "litellm_admin_api_key" { + type = string + sensitive = true + description = "LiteLLM admin key the building block authenticates with. It needs permission to create teams and keys." +} + +variable "litellm_platform_type_name" { + type = string + # The platform type is named for the capability, not for the product that delivers it, the same way + # this building block is "AI Model Access" rather than a LiteLLM block. A dash, not an underscore: + # meshstack_platform_type.metadata.name is validated against ^[A-Z0-9]+(-[A-Z0-9]+)*$, so both a + # mixed-case name and an underscore are rejected. + default = "AI-MODEL" + description = "Name of the meshStack platform type the LiteLLM gateway is registered under. It must match the platform type the AI platform reference architecture creates, because this building block runs on the AI model tenant." +} + +variable "litellm_team_models" { + type = list(string) + default = [] + description = "Names of the models on the gateway that every team created by this definition may call. An empty list sends no allow-list to LiteLLM. Create one definition per landing zone to grant different model sets." +} + +variable "litellm_team_max_budget" { + type = number + default = 100 + description = "Spending limit of a team for one budget period, in the currency the gateway reports spend in. LiteLLM blocks the team once the limit is reached." +} + +variable "litellm_team_budget_duration" { + type = string + default = "30d" + description = "Length of one budget period, after which LiteLLM resets the spend counter. Written as a LiteLLM duration such as '30d', '7d' or '1h'." +} + +variable "ai_platform_cluster_kubeconfig" { + type = string + sensitive = true + description = "kubeconfig of the AI platform cluster, as YAML. The tenant's tracing instance is deployed here. The credential needs permission to create a namespace, a secret and a Helm release." +} + +variable "demo_app_cluster_kubeconfig" { + type = string + sensitive = true + description = "kubeconfig of the demo application cluster, as YAML. Only the Secret with the model credential is written here, so the credential needs no more than `get`, `create`, `update`, `patch` and `delete` on secrets. `delete` is needed because this definition deletes what it created when the building block is deleted." +} + +variable "demo_app_platform_identifier" { + type = string + description = "Full identifier of the meshStack platform the demo application cluster is registered as, in the form '.'. The building block looks up the sibling tenant of this platform in the same meshProject to learn the namespace it writes the Secret into." +} + +variable "kubernetes_secret_name" { + type = string + default = "ai-model-access" + description = "Name of the Kubernetes Secret the building block writes into the namespace of the application team." +} + +variable "langfuse_domain" { + type = string + description = "Domain the tenants' tracing instances are published under. Each instance answers on '.', and the label is derived from the workspace and the project." +} + +variable "langfuse_ingress_class_name" { + type = string + default = "haproxy" + description = "Name of the IngressClass that serves the tenants' tracing instances. The default matches the controller the `kubernetes/ingress` module installs." +} + +variable "langfuse_default_org_role" { + type = string + default = "MEMBER" + description = "Role every user who logs in through the identity provider receives in a tenant's tracing organisation. Set 'NONE' when one OIDC client is shared across tenants and auto-join is not wanted." +} + +variable "stackit_project_id" { + type = string + description = "STACKIT project the shared PostgreSQL Flex instance and the tenants' buckets live in. The building block creates one database and one bucket per tenant inside it." +} + +variable "stackit_service_account_key" { + type = string + sensitive = true + description = "Service account key of the STACKIT service account, as the JSON STACKIT returns when the key is created. The account needs permission to create databases and users on the PostgreSQL Flex instance and to create Object Storage credentials groups in the project." +} + +variable "stackit_s3_admin_access_key" { + type = string + sensitive = true + description = "Access key of the administrative Object Storage credential the buckets are created with. It has to belong to the credentials group named by `stackit_s3_admin_credentials_group_urn`." +} + +variable "stackit_s3_admin_secret_access_key" { + type = string + sensitive = true + description = "Secret access key of the administrative Object Storage credential." +} + +variable "stackit_s3_admin_credentials_group_urn" { + type = string + description = "URN of the administrative Object Storage credentials group. Every tenant's bucket policy keeps access for this group, so the platform team can still reach a bucket." +} + +variable "langfuse_postgres_instance_id" { + type = string + description = "UUID of the shared STACKIT PostgreSQL Flex instance the tenants' databases are created in. Take it from the `instance_id` output of the `stackit/postgresflex` module." +} + +variable "langfuse_clickhouse_host" { + type = string + description = "Fully qualified hostname of the shared ClickHouse cluster, without a scheme. Take it from the `host` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_native_port" { + type = number + default = 9000 + description = "Native protocol port of the shared ClickHouse cluster. Take it from the `native_port` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_namespace" { + type = string + default = "clickhouse" + description = "Namespace of the shared ClickHouse cluster in the AI platform cluster. Each tenant's DDL Job runs in it. Take it from the `namespace` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_ddl_cluster_name" { + type = string + default = "default" + description = "Name of the ClickHouse cluster as the server knows it. Every statement the building block runs names it in an ON CLUSTER clause. Take it from the `ddl_cluster_name` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_admin_username" { + type = string + default = "default" + description = "Administrative ClickHouse user the DDL Job authenticates as to create a tenant's database and user. Take it from the `admin_username` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_admin_secret_name" { + type = string + default = "clickhouse-admin" + description = "Name of the Kubernetes Secret in the ClickHouse namespace that holds the administrative password. The DDL Job mounts it, so the password never becomes an input of this definition. Take it from the `admin_secret` output of the `ai/clickhouse` module." +} + +variable "langfuse_clickhouse_admin_secret_key" { + type = string + default = "password" + description = "Key inside the administrative Secret that holds the password." +} + +variable "langfuse_clickhouse_client_image" { + type = string + default = "clickhouse/clickhouse-server:26.4" + description = "Image the DDL Job runs `clickhouse-client` from. Keep the tag on the version the shared cluster runs, so no second image has to be pulled." +} + +variable "langfuse_clickhouse_ddl_timeout" { + type = number + default = 600 + description = "Seconds each ClickHouse DDL Job may run. It covers waiting for the cluster to answer a query and running the statements." +} + +variable "langfuse_valkey_host" { + type = string + description = "Hostname of the shared Valkey instance the tracing instances use as their queue backend, cache and rate limit store." +} + +variable "langfuse_valkey_password" { + type = string + sensitive = true + description = "Password of the Valkey instance. Use only characters that are safe in a URL." +} + +variable "langfuse_valkey_database_count" { + type = number + default = 16 + description = "Number of Valkey database indices the instance serves. A stock Valkey serves 16, numbered 0 to 15." +} + +# The identity provider is deliberately six flat variables rather than one object with optional +# attributes. Each field then carries its own explicit default, so no consumer of this file depends on +# Terraform's object-attribute defaulting to fill the three that have one. +variable "oidc_issuer_url" { + type = string + description = "Discovery base URL of the OIDC provider the tenants' tracing instances trust, for example 'https://idp.example.com/realms/ai'. The instances read '/.well-known/openid-configuration'. It is required, because the building block creates no password user." +} + +variable "oidc_client_id" { + type = string + description = "Client id of the OIDC client. Every tenant's tracing instance has a callback URL of its own, so this client needs each of them as an allowed redirect URI, or a wildcard where the provider supports one." +} + +variable "oidc_client_secret" { + type = string + sensitive = true + description = "Client secret of the OIDC client." +} + +variable "oidc_display_name" { + type = string + default = "Single Sign-On" + description = "Label on the login button of the tracing instances." +} + +variable "oidc_scopes" { + type = string + default = "openid email profile" + description = "Space-separated scope list the tracing instances request. The default covers what they need." +} + +variable "oidc_allow_account_linking" { + type = bool + default = true + description = "Link an OIDC login to an existing user with the same email address. Turn it on when a user can already exist from another login path." +} + +variable "meshstack" { + type = object({ + owning_workspace_identifier = string + tags = optional(map(list(string)), {}) + }) + description = "Shared meshStack context. Tags are optional and propagated to building block definition metadata." +} + +variable "hub" { + type = object({ + git_ref = optional(string, "main") + bbd_draft = optional(bool, true) + }) + const = true + default = { + git_ref = "main" + bbd_draft = true + } + description = <<-EOT + `git_ref`: Hub release reference. Set to a tag (e.g. 'v1.2.3') or branch or commit sha of meshcloud/meshstack-hub repo. + `bbd_draft`: If true, allows changing the building block definition for upgrading dependent building blocks. + EOT +} + +output "building_block_definition" { + description = "BBD is consumed in building block compositions. Add it to `spec.mandatory_building_block_refs` of the AI landing zone, so meshStack provisions model access when a tenant of that landing zone is created." + value = { + uuid = meshstack_building_block_definition.this.metadata.uuid + version_ref = var.hub.bbd_draft ? meshstack_building_block_definition.this.version_latest : meshstack_building_block_definition.this.version_latest_release + } +} + +locals { + # The building block takes the identity provider as one object, because Langfuse needs the whole set + # to register the provider. It is assembled here from the flat variables above. + oidc = { + issuer_url = var.oidc_issuer_url + client_id = var.oidc_client_id + client_secret = var.oidc_client_secret + display_name = var.oidc_display_name + scopes = var.oidc_scopes + allow_account_linking = var.oidc_allow_account_linking + } + + oidc_argument = jsonencode(local.oidc) +} + +resource "meshstack_building_block_definition" "this" { + metadata = { + owned_by_workspace = var.meshstack.owning_workspace_identifier + tags = var.meshstack.tags + } + + spec = { + # Named for the capability, not for the products that deliver it. LiteLLM and Langfuse stay + # behind this name, so the platform team can replace either without renaming what the + # application team ordered. + display_name = "AI Model Access" + symbol = "https://raw.githubusercontent.com/meshcloud/meshstack-hub/${var.hub.git_ref}/modules/ai/model-access/buildingblock/logo.png" + description = "Gives a project a governed OpenAI-compatible model endpoint with a budget, a credential delivered into its namespace, and a tracing instance of its own." + support_url = "https://docs.litellm.ai/docs/proxy/virtual_keys" + target_type = "TENANT_LEVEL" + run_transparency = true + supported_platforms = [{ name = var.litellm_platform_type_name }] + + # The application team never orders this block by hand. The AI landing zone lists it in + # `spec.mandatory_building_block_refs`, so meshStack provisions everything when the tenant is + # created. Every input below is STATIC or assigned from tenant context, which is what makes that + # work: a mandatory block that stops to ask a human defeats its own purpose, and API-driven + # tenant creation only succeeds when every input is defaulted or static. + use_in_landing_zones_only = true + + readme = chomp(<<-EOT + This building block gives your project a governed, OpenAI-compatible model endpoint with a + budget of its own, delivers the credential as a Kubernetes Secret in your namespace, and gives + you a tracing instance where you can see every call your application made. meshStack + provisions all of it when your tenant in the AI landing zone is created, so there is nothing + to order and nothing to fill in. + + ## 🎯 When to use it + + Use this building block when you: + - Want a governed model endpoint for your application instead of a credential shared across the whole platform. + - Need the spend of your project to be counted and capped on its own. + - Want to see the prompts, the answers, the latency and the cost of every call your application made. + - Want the platform team to decide which models you may call, through the landing zone you picked. + + ## πŸ’‘ Usage examples + + **Example 1: A chat assistant in a web application** + Your project lands in the AI landing zone, and the platform writes the credential into the + Secret `ai-model-access` in your namespace. Your Deployment mounts the whole Secret with + `envFrom`, and the OpenAI client library of your application picks up the endpoint and the + credential from the environment without a line of configuration. + + **Example 2: Finding out why an answer got worse** + A user reports a bad answer. You open your tracing instance from the sign-in link of this + building block, find the call, and see the prompt, the answer, the model that served it and + what it cost. Nobody outside your project can see any of it. + + ## πŸ”‘ Getting the credential + + The credential is a bearer token, so this building block never shows it in meshPanel. It is + delivered as a Kubernetes Secret in the namespace of your project, and your workload reads it + from there. The Secret carries two keys, named after the environment variables the OpenAI + client libraries read: + + | Key | Content | + |---|---| + | `OPENAI_API_KEY` | The credential, as a bearer token. | + | `OPENAI_BASE_URL` | The endpoint, already ending in `/v1`. | + + ```sh + curl "$OPENAI_BASE_URL/models" \ + -H "Authorization: Bearer $OPENAI_API_KEY" + ``` + + The endpoint stops answering once your project reaches its budget for the current period, and + the spend counter resets at the end of every period. + + ## πŸ“Š Shared Responsibility + + | Responsibility | Platform Team | Application Team | + |---|:---:|:---:| + | Operate the gateway, the model backends and the tracing backends | βœ… | ❌ | + | Set the budget, the budget period and the allowed models per landing zone | βœ… | ❌ | + | Create the endpoint credential and the tracing instance when the tenant is created | βœ… | ❌ | + | Create the database, the bucket and the trace storage the tracing instance of your project uses | βœ… | ❌ | + | Deliver the credential into the namespace of the project as a Kubernetes Secret | βœ… | ❌ | + | Upgrade the tracing instance | βœ… | ❌ | + | Mount the Secret into the workload and keep the credential out of source control | ❌ | βœ… | + | Stay within the granted budget and the allowed models | ❌ | βœ… | + | Review own traces and evaluations | ❌ | βœ… | + | Build and operate the application that calls the endpoint | ❌ | βœ… | + EOT + ) + } + + version_spec = { + draft = var.hub.bbd_draft + deletion_mode = "DELETE" + + # One tenant is one team, one credential and one tracing instance. Applying the block a second + # time in the same tenant would create a second set, so meshStack refuses it. + only_apply_once_per_tenant = true + + # The run looks up the sibling tenant of the same meshProject to learn the namespace the Secret + # goes into. meshStack grants an ephemeral API token with these permissions to the run. + # `modules/aks/github-connector` proves that a TENANT_LEVEL block can hold them. + permissions = ["TENANT_LIST"] + + implementation = { + terraform = { + terraform_version = "1.12.2" + repository_url = "https://github.com/meshcloud/meshstack-hub.git" + repository_path = "modules/ai/model-access/buildingblock" + ref_name = var.hub.git_ref + async = false + use_mesh_http_backend_fallback = true + } + } + + inputs = { + # ── The shared gateway ── + + litellm_api_base = { + display_name = "LiteLLM API Base URL" + description = "Base URL of the LiteLLM gateway the team is created on." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_api_base) + } + + litellm_api_key = { + display_name = "LiteLLM Admin Key" + description = "Admin key the building block authenticates with against the LiteLLM API." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.litellm_admin_api_key + secret_version = nonsensitive(sha256(var.litellm_admin_api_key)) + } + } + } + + models = { + display_name = "Allowed Models" + description = "Names of the models on the gateway that the team may call." + type = "CODE" + assignment_type = "STATIC" + # jsonencode twice is correct, see https://registry.terraform.io/providers/meshcloud/meshstack/latest/docs/resources/building_block_definition#argument-1 + argument = jsonencode(jsonencode(var.litellm_team_models)) + } + + max_budget = { + display_name = "Budget" + description = "Spending limit of the team for one budget period." + type = "INTEGER" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_team_max_budget) + } + + budget_duration = { + display_name = "Budget Duration" + description = "Length of one budget period, for example '30d'." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.litellm_team_budget_duration) + } + + # ── The tenant context every derived name comes from ── + # + # These three are assigned by meshStack and cannot be forged by the tenant. Together with the + # STATIC values above and below they decide which namespace in which cluster receives the + # credential, so none of them may ever become a USER_INPUT. + + workspace_identifier = { + display_name = "Workspace Identifier" + description = "Identifier of the meshStack workspace. It is part of the team alias and of every per-tenant name the building block derives." + type = "STRING" + assignment_type = "WORKSPACE_IDENTIFIER" + } + + project_identifier = { + display_name = "Project Identifier" + description = "Identifier of the meshStack project. It is part of the team alias and of every per-tenant name the building block derives." + type = "STRING" + assignment_type = "PROJECT_IDENTIFIER" + } + + meshstack_tenant_uuid = { + display_name = "Tenant UUID" + description = "UUID of the meshStack tenant, written to the team metadata so an operator can trace a team back to its tenant." + type = "STRING" + assignment_type = "MESHSTACK_TENANT_UUID" + } + + # ── The two clusters ── + + ai_platform_cluster_kubeconfig = { + display_name = "AI Platform Cluster kubeconfig" + description = "kubeconfig of the AI platform cluster, as YAML. The tenant's tracing instance is deployed here." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.ai_platform_cluster_kubeconfig + secret_version = nonsensitive(sha256(var.ai_platform_cluster_kubeconfig)) + } + } + } + + demo_app_cluster_kubeconfig = { + display_name = "Application Cluster kubeconfig" + description = "kubeconfig of the demo application cluster, as YAML. Only the Secret with the model credential is written here." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.demo_app_cluster_kubeconfig + secret_version = nonsensitive(sha256(var.demo_app_cluster_kubeconfig)) + } + } + } + + demo_app_platform_identifier = { + display_name = "Application Platform Identifier" + description = "Full identifier of the meshStack platform the demo application cluster is registered as, used to look up the sibling tenant of the same meshProject." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.demo_app_platform_identifier) + } + + secret_name = { + display_name = "Secret Name" + description = "Name of the Kubernetes Secret the credential is delivered in." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.kubernetes_secret_name) + } + + # ── The tenant's tracing instance ── + + langfuse_domain = { + display_name = "Tracing Domain" + description = "Domain the tenants' tracing instances are published under." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_domain) + } + + langfuse_ingress_class_name = { + display_name = "Ingress Class" + description = "Name of the IngressClass that serves the tenants' tracing instances." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_ingress_class_name) + } + + langfuse_default_org_role = { + display_name = "Tracing Default Role" + description = "Role a user who logs in receives in the tenant's tracing organisation." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_default_org_role) + } + + oidc = { + display_name = "Identity Provider" + description = "JSON object with the issuer URL, the client id and the client secret of the OIDC client the tracing instances trust." + type = "CODE" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = local.oidc_argument + secret_version = nonsensitive(sha256(local.oidc_argument)) + } + } + } + + # ── STACKIT, where the per-tenant backends are created ── + # + # The building block creates the tenant's Postgres database, its owner user and its bucket, so it + # needs a STACKIT credential. The user and the password of each backend are not inputs any more: + # the building block derives the names and reads the credentials off the resources it created. + + stackit_project_id = { + display_name = "STACKIT Project" + description = "STACKIT project the shared Postgres instance and the tenants' buckets live in." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.stackit_project_id) + } + + stackit_service_account_key = { + display_name = "STACKIT Service Account Key" + description = "Service account key the building block authenticates against STACKIT with, as JSON." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.stackit_service_account_key + secret_version = nonsensitive(sha256(var.stackit_service_account_key)) + } + } + } + + stackit_s3_admin_access_key = { + display_name = "Object Storage Admin Access Key" + description = "Access key of the administrative Object Storage credential the buckets are created with." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.stackit_s3_admin_access_key + secret_version = nonsensitive(sha256(var.stackit_s3_admin_access_key)) + } + } + } + + stackit_s3_admin_secret_access_key = { + display_name = "Object Storage Admin Secret Access Key" + description = "Secret access key of the administrative Object Storage credential." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.stackit_s3_admin_secret_access_key + secret_version = nonsensitive(sha256(var.stackit_s3_admin_secret_access_key)) + } + } + } + + stackit_s3_admin_credentials_group_urn = { + display_name = "Object Storage Admin Credentials Group" + description = "URN of the administrative Object Storage credentials group every tenant's bucket policy keeps access for." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.stackit_s3_admin_credentials_group_urn) + } + + langfuse_postgres_instance_id = { + display_name = "Postgres Instance" + description = "UUID of the shared PostgreSQL Flex instance the tenants' databases are created in." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_postgres_instance_id) + } + + # ── The shared ClickHouse cluster ── + # + # Every value here comes from an output of the `ai/clickhouse` module. The administrative password + # is deliberately absent: the DDL Job mounts the Secret the cluster already holds it in. + + langfuse_clickhouse_host = { + display_name = "ClickHouse Host" + description = "Fully qualified hostname of the shared ClickHouse cluster." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_host) + } + + langfuse_clickhouse_native_port = { + display_name = "ClickHouse Native Port" + description = "Native protocol port of the shared ClickHouse cluster." + type = "INTEGER" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_native_port) + } + + langfuse_clickhouse_namespace = { + display_name = "ClickHouse Namespace" + description = "Namespace of the shared ClickHouse cluster. Each tenant's DDL Job runs in it." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_namespace) + } + + langfuse_clickhouse_ddl_cluster_name = { + display_name = "ClickHouse Cluster Name" + description = "Name of the ClickHouse cluster as the server knows it. Every statement runs ON CLUSTER with it." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_ddl_cluster_name) + } + + langfuse_clickhouse_admin_username = { + display_name = "ClickHouse Admin User" + description = "Administrative ClickHouse user that creates a tenant's database and user." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_admin_username) + } + + langfuse_clickhouse_admin_secret_name = { + display_name = "ClickHouse Admin Secret" + description = "Name of the Kubernetes Secret in the ClickHouse namespace that holds the administrative password." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_admin_secret_name) + } + + langfuse_clickhouse_admin_secret_key = { + display_name = "ClickHouse Admin Secret Key" + description = "Key inside the administrative Secret that holds the password." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_admin_secret_key) + } + + langfuse_clickhouse_client_image = { + display_name = "ClickHouse Client Image" + description = "Image the DDL Job runs `clickhouse-client` from." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_client_image) + } + + langfuse_clickhouse_ddl_timeout = { + display_name = "ClickHouse DDL Timeout" + description = "Seconds each ClickHouse DDL Job may run." + type = "INTEGER" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_clickhouse_ddl_timeout) + } + + langfuse_valkey_host = { + display_name = "Valkey Host" + description = "Hostname of the shared Valkey instance." + type = "STRING" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_valkey_host) + } + + langfuse_valkey_password = { + display_name = "Valkey Password" + description = "Password of the Valkey instance." + type = "STRING" + assignment_type = "STATIC" + sensitive = { + argument = { + secret_value = var.langfuse_valkey_password + secret_version = nonsensitive(sha256(var.langfuse_valkey_password)) + } + } + } + + langfuse_valkey_database_count = { + display_name = "Valkey Database Count" + description = "Number of Valkey database indices the instance serves." + type = "INTEGER" + assignment_type = "STATIC" + argument = jsonencode(var.langfuse_valkey_database_count) + } + + hub = { + display_name = "Hub" + description = "JSON object with `git_ref`, the meshstack-hub reference the building block sources the tracing module from. It is the same reference the building block itself is checked out at." + type = "CODE" + assignment_type = "STATIC" + argument = jsonencode(jsonencode({ git_ref = var.hub.git_ref })) + } + } + + # No output below carries the virtual key, and none can: `version_spec.outputs` has no + # `sensitive` block, unlike `version_spec.inputs`, so every building block output is stored and + # displayed in cleartext in meshPanel. The key is created, written into the Kubernetes Secret and + # left inside the Terraform run. The `summary` output names the Secret it landed in. + outputs = { + team_id = { + display_name = "Team ID" + description = "ID of the LiteLLM team. It becomes the platform tenant ID, so building blocks ordered later can bind their resources to this team." + type = "STRING" + assignment_type = "PLATFORM_TENANT_ID" + } + + team_alias = { + display_name = "Team Alias" + description = "Alias of the team on the gateway." + type = "STRING" + assignment_type = "NONE" + } + + key_id = { + display_name = "Credential ID" + description = "Hash the gateway identifies the credential by. It is not the credential." + type = "STRING" + assignment_type = "NONE" + } + + api_base = { + display_name = "API Base URL" + description = "OpenAI-compatible base URL of the endpoint, including the '/v1' suffix." + type = "STRING" + assignment_type = "NONE" + } + + secret_name = { + display_name = "Secret Name" + description = "Name of the Kubernetes Secret the credential was delivered in." + type = "STRING" + assignment_type = "NONE" + } + + secret_namespace = { + display_name = "Secret Namespace" + description = "Namespace of the application team the Secret was written into." + type = "STRING" + assignment_type = "NONE" + } + + langfuse_url = { + display_name = "Tracing URL" + description = "URL of the tenant's own tracing instance." + type = "STRING" + assignment_type = "SIGN_IN_URL" + } + + langfuse_namespace = { + display_name = "Tracing Namespace" + description = "Namespace the tenant's tracing instance runs in, in the AI platform cluster." + type = "STRING" + assignment_type = "NONE" + } + + langfuse_oidc_callback_url = { + display_name = "Tracing Callback URL" + description = "Callback URL to register at the identity provider for this tenant's tracing instance." + type = "STRING" + assignment_type = "NONE" + } + + summary = { + display_name = "Summary" + type = "STRING" + assignment_type = "SUMMARY" + } + } + } +} + +terraform { + required_version = ">= 1.12.0" + + required_providers { + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.23.0" + } + } +} diff --git a/modules/kubernetes/ingress/buildingblock/.helmignore b/modules/kubernetes/ingress/buildingblock/.helmignore new file mode 100644 index 00000000..4758f993 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/.helmignore @@ -0,0 +1,25 @@ +# The module directory is the chart directory, so everything that is not part of the chart has +# to be excluded here. Helm stores the packaged chart in a Kubernetes Secret β€” keeping it small +# is critical. + +# Terraform files and state +*.tf +*.tfstate +*.tfstate.backup +*.tfplan +.terraform/ +.terraform.lock.hcl + +# Terraform test files +*.tftest.hcl + +# Kubernetes credentials β€” never bundle into the chart +kubeconfig.yaml +kubeconfig-mock.yaml + +# Misc +.DS_Store +*.png +*.svg +README.md +APP_TEAM_README.md diff --git a/modules/kubernetes/ingress/buildingblock/APP_TEAM_README.md b/modules/kubernetes/ingress/buildingblock/APP_TEAM_README.md new file mode 100644 index 00000000..dab476ab --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/APP_TEAM_README.md @@ -0,0 +1,95 @@ +Your services in this cluster get a public HTTPS URL with a certificate that browsers trust. The platform team runs an ingress controller and cert-manager for you, so all you add to your service is an Ingress object with a hostname. + +## 🎯 When to use it + +Use this building block when you: +- want to reach a service in the cluster from outside, over a real domain name instead of a port forward +- need TLS that browsers, mobile apps and API clients accept without a warning +- do not want to buy, renew or store certificates yourself + +## πŸ’‘ Usage examples + +**Example 1: Publish a web frontend** +You deploy a frontend Service in your namespace and add an Ingress for `shop.example.com` with the platform's ingress class. The controller starts routing traffic to your Service and the hostname answers over HTTPS right away. + +**Example 2: Expose an API for a partner** +Your team needs a stable HTTPS endpoint for a partner integration. You create an Ingress for `api.example.com`, hand the URL to the partner, and the certificate keeps renewing itself as long as the Ingress exists. + +## πŸ”§ How to use it + +Add an Ingress to your namespace and set the ingress class the platform team gave you: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: shop +spec: + ingressClassName: haproxy + rules: + - host: shop.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: shop + port: + number: 8080 +``` + +When the platform team runs a wildcard certificate for the cluster domain, that is all you need β€” the controller already serves a valid certificate for every hostname in that domain. + +For a hostname outside the cluster domain, ask cert-manager for its own certificate. Add the ClusterIssuer annotation and a `tls` section: + +```yaml +metadata: + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod +spec: + tls: + - hosts: + - shop.example.com + secretName: shop-tls +``` + +The hostname has to resolve to the ingress load balancer before Let's Encrypt can validate it, so create the DNS record first. + +## βš™οΈ What the platform team installs + +The platform team runs this module against your cluster. The module brings no cluster credentials +of its own: the caller configures the `kubernetes` and the `helm` provider and hands both to the +module. + +```hcl +module "ingress" { + source = "github.com/meshcloud/meshstack-hub//modules/kubernetes/ingress/buildingblock?ref=main" + + providers = { + kubernetes = kubernetes + helm = helm + } + + acme_email = "platform@example.com" +} +``` + +**The resource requests and limits of the ingress controller and of cert-manager default to +demonstration sizes.** HAProxy runs as a single replica with 100m CPU requested, which is enough +to show a service answering over HTTPS and not enough to carry production traffic. Ask your +platform team to raise `haproxy_replica_count` and the `*_resources` variables before you put a +real workload behind this ingress. + +## πŸ“Š Shared Responsibility + +| Responsibility | Platform Team | Application Team | +|---|:---:|:---:| +| Run the ingress controller and its load balancer | βœ… | ❌ | +| Run cert-manager and the Let's Encrypt ClusterIssuer | βœ… | ❌ | +| Renew the wildcard certificate for the cluster domain | βœ… | ❌ | +| Create DNS records for the cluster domain | βœ… | ❌ | +| Create the Ingress object and pick the hostname | ❌ | βœ… | +| Keep the backend Service healthy and reachable | ❌ | βœ… | +| Create DNS records for hostnames outside the cluster domain | ❌ | βœ… | +| Authentication and authorization inside the application | ❌ | βœ… | diff --git a/modules/kubernetes/ingress/buildingblock/Chart.yaml b/modules/kubernetes/ingress/buildingblock/Chart.yaml new file mode 100644 index 00000000..03f59feb --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/Chart.yaml @@ -0,0 +1,5 @@ +apiVersion: v2 +name: meshstack-ingress-issuer +description: ClusterIssuer and wildcard Certificate for the meshStack Kubernetes ingress building block. +type: application +version: 1.0.0 diff --git a/modules/kubernetes/ingress/buildingblock/README.md b/modules/kubernetes/ingress/buildingblock/README.md new file mode 100644 index 00000000..c9e26f60 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/README.md @@ -0,0 +1,237 @@ +--- +name: Kubernetes Ingress with TLS +supportedPlatforms: + - kubernetes +description: Installs cert-manager, the HAProxy ingress controller and a Let's Encrypt ClusterIssuer so every service in the cluster can get a public HTTPS URL with a valid certificate. +# The cluster credentials arrive through the providers the caller configures and the DNS provider +# credentials arrive as inputs, so there is nothing to set up on the cloud side. +requiresBackplane: false +--- + +# Kubernetes Ingress with TLS + +This module delivers one capability: **services in the cluster get a public HTTPS URL with a valid certificate.** It installs the three pieces that capability needs, in one Terraform run: + +1. **cert-manager** β€” requests and renews certificates from Let's Encrypt. +2. **HAProxy ingress controller** β€” terminates TLS and routes traffic to Services, behind a cloud load balancer. +3. **A Let's Encrypt ClusterIssuer** β€” the issuer application teams reference from their Ingress objects. + +This documentation is intended as a reference for cloud foundation or platform engineers using this module. + +## Why the ClusterIssuer runs through a local Helm chart + +A `kubernetes_manifest` resource looks the CRD schema up at plan time. cert-manager installs the `ClusterIssuer` CRD, so the schema does not exist yet during the first plan. Foundations worked around this by putting the ClusterIssuer in a second Terragrunt unit that runs after cert-manager. + +This module renders the ClusterIssuer through a Helm chart that lives in the module directory itself (`chart = path.module`). Helm applies the manifests at apply time and never asks Terraform for a schema, so cert-manager and the ClusterIssuer fit into a single unit. That is why the module directory carries a `Chart.yaml`, a `templates/` directory and a `.helmignore` that keeps every Terraform artifact out of the packaged chart. + +## Two ways to get certificates + +**HTTP-01, per hostname (default).** Leave `dns01` unset. cert-manager solves the ACME challenge over the ingress itself, so every hostname an application team asks for gets its own certificate. The hostname has to resolve to the load balancer before issuance can finish. + +**DNS-01, one wildcard certificate.** Set `dns01` with a `zone_name` and exactly one provider. The module then creates a single `Certificate` for `*.` in `haproxy_namespace` and points HAProxy's `controller.defaultTLSSecret.secret` at the resulting secret. HAProxy serves that certificate for every host that brings none of its own, so a new application needs no certificate request at all. The certificate lives in the long-lived HAProxy namespace, so tearing an application namespace down never takes it with it. The HTTP-01 solver stays in place next to the DNS-01 one for hostnames outside the zone. + +### The certificate may cover less than the zone + +`zone_name` is the zone the solver is authorised for, and `certificate_domain` is the domain the certificate covers. They are the same by default: a caller that sets no `certificate_domain` gets `*.`. + +Set `certificate_domain` to a name below the zone when several clusters share one zone. With the zone `likvid.stackit.run` and `certificate_domain = "cluster1.likvid.stackit.run"`, the module issues `*.cluster1.likvid.stackit.run`, while the ClusterIssuer keeps the zone itself in the `dnsZones` selector of the solver. That selector matches every name below the zone, so the solver still answers the challenge for the narrower certificate, and the credential it uses stays a zone-wide one. The `wildcard_certificate_domain` output reports the domain that was used. + +The module rejects a `certificate_domain` outside the zone, because the solver cannot answer for it. + +### DNS-01 providers + +| Provider | Extra chart | Status | +|---|---|---| +| `stackit` | `stackit-cert-manager-webhook` | Implemented and exercised. | +| `route53` | none β€” cert-manager solves Route53 natively | Implemented, **not exercised**. | + +STACKIT DNS has no built-in cert-manager solver, so the module installs the [stackit-cert-manager-webhook](https://github.com/stackitcloud/stackit-cert-manager-webhook) chart into `cert_manager_namespace` and stores the service account key in the secret the webhook mounts. The ClusterIssuer then calls it with `solverName: stackit` and `groupName: acme.stackit.de`. + +The `route53` branch is a documented shape rather than a tested path. AKS foundations still issue per-hostname HTTP-01 certificates today, so nobody has run this branch against a live hosted zone. Treat it as a starting point and verify it before you rely on it. + +## What this module replaces + +Five foundation units carried copies of the same `certmanager.tf`, `haproxy.tf` and `cluster_issuer.tf`. The copies had drifted: + +| | SKE | AKS | +|---|---|---| +| cert-manager version | `v1.20.0` | `v1.19.4` | +| `--certificate-request-minimum-backoff-duration=1m` | present | missing | +| Service annotations | none | Azure health probe path | +| ACME contact | `ske@meshcloud.io` | `platform@likvid-bank.com`, `devops-platform@meshcloud.io` | + +The module defaults match the SKE copy, which is the most current one. AKS callers set `haproxy_service_annotations` and get the newer cert-manager and the retry-backoff tuning along the way. + +## The caller configures the providers + +This module carries no `provider` block. The caller configures the `kubernetes` and the `helm` +provider and passes both down through the `providers` argument of the module call. Two things +follow from that. + +**Any credential works.** The module used to take a `cluster_endpoint`, a `cluster_ca_certificate` +and a bearer `token`, which ruled out every cluster that hands out a client certificate instead of +a token. STACKIT SKE is one of them: `stackit_ske_kubeconfig` issues a client certificate and a +client key, and nothing can mint a cluster-admin token before the cluster exists. A caller that +configures the provider itself picks whichever credential its cluster offers. + +**The module call accepts `count`.** A module with its own provider configuration is a legacy +module, and OpenTofu refuses `count`, `for_each` and `depends_on` on calls to it. The restriction +is transitive, so wrapping the module in a local module does not lift it. Compositions that make +ingress optional need `count`, which is only possible without a local provider configuration. + +## Notes for platform engineers + +- **Providers.** Only `kubernetes` and `helm`. No cloud provider ever enters this module, so it works on SKE, AKS and anything else that speaks the Kubernetes API. Cloud-specific behaviour arrives as strings, mainly through `haproxy_service_annotations`. +- **Sourced, not ordered.** There is no `meshstack_integration.tf` and no `backplane/`. Foundations source `buildingblock/` from a Terragrunt unit, and reference architectures source it from their own building block. +- **Permissions.** The credentials the caller puts into the two providers need cluster-admin rights, because cert-manager installs CRDs and cluster-scoped RBAC. +- **DNS records.** Point your DNS A record at the `haproxy_lb_ip` output. Nothing can be issued or served before that record resolves. +- **First apply.** HAProxy comes up before the wildcard certificate is issued. Until the secret exists, HAProxy serves its own self-signed certificate for unmatched hosts and picks the real one up as soon as cert-manager writes it. + +## Resource sizing + +Every workload this module installs gets an explicit resource request and limit, and every one of +those values is a variable. **The defaults are sized for a demonstration cluster and a production +consumer has to raise them.** The defaults keep the whole ingress stack under roughly 300m CPU and +600Mi of requested memory, so it fits next to an application on a two-node cluster. + +| Workload | Request | Limit | Variable | +|---|---|---|---| +| HAProxy controller | 100m / 256Mi | 500m / 768Mi | `haproxy_resources` | +| HAProxy CRD Job | 50m / 64Mi | 200m / 256Mi | `haproxy_crdjob_resources` | +| cert-manager controller | 10m / 64Mi | 200m / 256Mi | `cert_manager_resources` | +| cert-manager cainjector | 10m / 64Mi | 200m / 256Mi | `cert_manager_cainjector_resources` | +| cert-manager webhook | 10m / 32Mi | 100m / 128Mi | `cert_manager_webhook_resources` | +| cert-manager startupapicheck | 10m / 32Mi | 100m / 128Mi | `cert_manager_startupapicheck_resources` | +| STACKIT cert-manager webhook | 10m / 64Mi | 100m / 128Mi | `stackit_webhook_resources` | + +`haproxy_replica_count` defaults to 1 for the same reason. One replica gives no redundancy: every +restart and every node drain interrupts ingress traffic. + +The HAProxy limit is the one value that cannot go much lower. The pod runs HAProxy and the Go +controller side by side, and the container entrypoint hands HAProxy two thirds of the cgroup +memory limit. Users of chart 1.49.0 report that HAProxy reloads in a loop instead of serving +traffic when the memory limit stays below 500Mi, and a maintainer recommends at least 1Gi. + +For production, raise the HAProxy controller to 500m / 1Gi requested with a 2Gi limit, run at +least two replicas on separate nodes, and give the cert-manager controller and cainjector 100m +CPU and 512Mi memory each. + +## Usage + +The caller configures both providers and passes them into the module call. + +```hcl +provider "kubernetes" { + host = module.cluster.provider_config.host + cluster_ca_certificate = module.cluster.provider_config.cluster_ca_certificate + client_certificate = module.cluster.provider_config.client_certificate + client_key = module.cluster.provider_config.client_key +} + +provider "helm" { + kubernetes = { + host = module.cluster.provider_config.host + cluster_ca_certificate = module.cluster.provider_config.cluster_ca_certificate + client_certificate = module.cluster.provider_config.client_certificate + client_key = module.cluster.provider_config.client_key + } +} + +module "ingress" { + # The call may use count, because the module configures no provider of its own. + count = var.expose == "none" ? 0 : 1 + source = "github.com/meshcloud/meshstack-hub//modules/kubernetes/ingress/buildingblock?ref=main" + + providers = { + kubernetes = kubernetes + helm = helm + } + + acme_email = "ske@meshcloud.io" + + dns01 = { + zone_name = "likvid.stackit.run" + + # Optional. Without it the certificate covers `*.likvid.stackit.run`, the whole zone. + certificate_domain = "cluster1.likvid.stackit.run" + + stackit = { + project_id = var.stackit_project_id + service_account_key = var.stackit_service_account_key + } + } +} +``` + +A Terragrunt unit does the same through a `generate "provider"` block that writes both provider +configurations next to the module call. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [helm](#requirement\_helm) | >= 3.0.0 | +| [kubernetes](#requirement\_kubernetes) | >= 2.38 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [helm_release.cert_manager](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.haproxy](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.issuer](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [helm_release.stackit_cert_manager_webhook](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | +| [kubernetes_namespace_v1.cert_manager](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_namespace_v1.haproxy_ingress](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | +| [kubernetes_secret_v1.route53_dns01](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [kubernetes_secret_v1.stackit_dns01](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1) | resource | +| [kubernetes_service_v1.haproxy_controller](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/service_v1) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [acme\_email](#input\_acme\_email) | Contact address Let's Encrypt uses for expiry warnings and account recovery. | `string` | n/a | yes | +| [acme\_private\_key\_secret\_name](#input\_acme\_private\_key\_secret\_name) | Name of the secret in which cert-manager stores the ACME account private key. | `string` | `"letsencrypt-prod-account-key"` | no | +| [acme\_server](#input\_acme\_server) | ACME directory URL. Point this at https://acme-staging-v02.api.letsencrypt.org/directory while you test, because the production endpoint has strict rate limits. | `string` | `"https://acme-v02.api.letsencrypt.org/directory"` | no | +| [cert\_manager\_cainjector\_resources](#input\_cert\_manager\_cainjector\_resources) | Resource requests and limits of the cert-manager cainjector. The default is sized for a
demonstration cluster and a production consumer has to raise it.

The cainjector watches every Secret in the cluster, so its memory grows with the number of
Secrets. cert-manager issue #6217 reports it reaching gigabytes on large clusters. The limit
here is `256Mi` because a demonstration cluster holds few Secrets, and a production cluster
wants `512Mi` or more together with the `--namespace` flag that narrows the watch. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "200m",
    "memory": "256Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "64Mi"
  }
}
| no | +| [cert\_manager\_crds\_keep](#input\_cert\_manager\_crds\_keep) | Keep the cert-manager CRDs when the Helm release is destroyed. Keeping them preserves existing Certificate and ClusterIssuer objects across a reinstall. | `bool` | `false` | no | +| [cert\_manager\_extra\_args](#input\_cert\_manager\_extra\_args) | Extra command line arguments for the cert-manager controller. | `list(string)` | 
[
  "--certificate-request-minimum-backoff-duration=1m"
]
| no | +| [cert\_manager\_namespace](#input\_cert\_manager\_namespace) | Namespace for cert-manager and, when DNS-01 runs through STACKIT, for the STACKIT cert-manager webhook. The webhook chart expects both in the same namespace. | `string` | `"cert-manager"` | no | +| [cert\_manager\_resources](#input\_cert\_manager\_resources) | Resource requests and limits of the cert-manager controller. The default is sized for a
demonstration cluster and a production consumer has to raise it.

The cert-manager Helm chart sets no resources at all and documents `10m` CPU and `32Mi` memory
as its example request. The memory request here is `64Mi` instead, because the controller keeps
informer caches for Certificates, Secrets and Ingresses and a container that runs out of memory
is OOMKilled rather than slowed down. A cluster that issues certificates continuously wants
`100m` CPU and `512Mi` memory. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "200m",
    "memory": "256Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "64Mi"
  }
}
| no | +| [cert\_manager\_startupapicheck\_resources](#input\_cert\_manager\_startupapicheck\_resources) | Resource requests and limits of the cert-manager startupapicheck Job. The default is sized for
a demonstration cluster and a production consumer has to raise it.

The Job runs once per install, checks that the webhook answers and then exits, so it never
holds resources for long. Its request still has to fit on a node, which is why it is kept this
small. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "100m",
    "memory": "128Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "32Mi"
  }
}
| no | +| [cert\_manager\_version](#input\_cert\_manager\_version) | Version of the cert-manager Helm chart. See https://github.com/cert-manager/cert-manager/releases. | `string` | `"v1.20.0"` | no | +| [cert\_manager\_webhook\_resources](#input\_cert\_manager\_webhook\_resources) | Resource requests and limits of the cert-manager admission webhook. The default is sized for a
demonstration cluster and a production consumer has to raise it.

The webhook validates cert-manager objects and holds no cache, so it is the smallest of the
three cert-manager pods. Every apply that touches a Certificate or an Issuer goes through it,
so keep the limit above the request. A production cluster wants `100m` CPU and `256Mi` memory. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "100m",
    "memory": "128Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "32Mi"
  }
}
| no | +| [cluster\_issuer\_name](#input\_cluster\_issuer\_name) | Name of the ClusterIssuer. Application teams reference it from the cert-manager.io/cluster-issuer annotation on their Ingress. | `string` | `"letsencrypt-prod"` | no | +| [dns01](#input\_dns01) | Enables a wildcard certificate via DNS-01. Set exactly one provider. Null keeps HTTP-01
per-hostname issuance.

`zone_name` is the DNS zone the solver is authorised for, and the ClusterIssuer selects the
solver for every name inside it. `certificate_domain` is the domain the wildcard certificate
covers and defaults to `zone_name`, which gives `*.`. Set it to a name below the
zone, for example `cluster1.likvid.stackit.run` inside the zone `likvid.stackit.run`, to narrow
the certificate to that label while the solver keeps answering for the whole zone. | 
object({
    zone_name          = string
    certificate_domain = optional(string)
    stackit            = optional(object({ project_id = string, service_account_key = string }))
    route53            = optional(object({ hosted_zone_id = string, access_key_id = string, secret_access_key = string, region = optional(string, "eu-central-1") }))
  })
| `null` | no | +| [haproxy\_crdjob\_resources](#input\_haproxy\_crdjob\_resources) | Resource requests and limits of the Job the HAProxy chart runs to install its CRDs. The default
is sized for a demonstration cluster and a production consumer has to raise it.

The chart requests `250m` CPU and `400Mi` memory for this Job. The Job applies a handful of
CRDs and exits, so a much smaller request is enough, and a smaller request also means the Job
still schedules on a small node. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "200m",
    "memory": "256Mi"
  },
  "requests": {
    "cpu": "50m",
    "memory": "64Mi"
  }
}
| no | +| [haproxy\_namespace](#input\_haproxy\_namespace) | Namespace for the HAProxy ingress controller. The wildcard certificate is created here as well, so its secret survives the teardown of any application namespace. | `string` | `"haproxy-ingress"` | no | +| [haproxy\_release\_name](#input\_haproxy\_release\_name) | Helm release name of the HAProxy ingress controller. The chart names the controller Service '-kubernetes-ingress'. | `string` | `"haproxy"` | no | +| [haproxy\_replica\_count](#input\_haproxy\_replica\_count) | Number of HAProxy ingress controller replicas. The default of 1 is sized for a demonstration cluster and gives no redundancy: every restart or node drain interrupts ingress traffic. Production wants at least 2, spread over separate nodes. | `number` | `1` | no | +| [haproxy\_resources](#input\_haproxy\_resources) | Resource requests and limits of the HAProxy ingress controller. The default is sized for a
demonstration cluster and a production consumer has to raise it.

The chart requests `250m` CPU and `400Mi` memory and sets no limit. The pod runs two processes,
HAProxy itself and the Go controller, and the container entrypoint hands HAProxy two thirds of
the cgroup memory limit. Users of this chart version report that HAProxy reloads in a loop
instead of serving traffic when the memory limit stays below `500Mi`, and a maintainer
recommends at least `1Gi` (haproxytech/kubernetes-ingress issue #799). The `768Mi` limit here is
the smallest value that clears that threshold with headroom. Production wants `1Gi` to `2Gi`
and a CPU limit that matches the traffic the controller has to terminate. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "500m",
    "memory": "768Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "256Mi"
  }
}
| no | +| [haproxy\_service\_annotations](#input\_haproxy\_service\_annotations) | Annotations on the HAProxy controller Service. The cloud provider reads them to configure the
load balancer. Two values matter in practice:
- AKS needs `service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path = "/healthz"`.
- STACKIT uses `lb.stackit.cloud/internal-lb` to keep the load balancer off the public internet. | `map(string)` | `{}` | no | +| [haproxy\_service\_type](#input\_haproxy\_service\_type) | Service type of the HAProxy ingress controller. | `string` | `"LoadBalancer"` | no | +| [haproxy\_timeout](#input\_haproxy\_timeout) | Seconds to wait for the HAProxy Helm release to become ready. The default of 20 minutes covers the time a cloud provider takes to provision the load balancer. | `number` | `1200` | no | +| [haproxy\_version](#input\_haproxy\_version) | Version of the haproxytech/kubernetes-ingress Helm chart. See https://github.com/haproxytech/helm-charts/blob/main/kubernetes-ingress/Chart.yaml. | `string` | `"1.49.0"` | no | +| [ingress\_class\_name](#input\_ingress\_class\_name) | Name of the IngressClass the controller serves. The HTTP-01 solver of the ClusterIssuer uses the same name. | `string` | `"haproxy"` | no | +| [stackit\_webhook\_resources](#input\_stackit\_webhook\_resources) | Resource requests and limits of the STACKIT cert-manager webhook. Only used when dns01.stackit
is set. The default is sized for a demonstration cluster and a production consumer has to raise
it.

The chart sets no resources and its values file states that `100m` CPU and `128Mi` memory are
enough for the webhook, which is what the limit uses. The webhook answers one DNS-01 challenge
per certificate renewal, so the request stays well below that. Production wants the chart's own
figures as the request as well. | 
object({
    requests = optional(object({ cpu = optional(string), memory = optional(string) }), {})
    limits   = optional(object({ cpu = optional(string), memory = optional(string) }), {})
  })
| 
{
  "limits": {
    "cpu": "100m",
    "memory": "128Mi"
  },
  "requests": {
    "cpu": "10m",
    "memory": "64Mi"
  }
}
| no | +| [stackit\_webhook\_version](#input\_stackit\_webhook\_version) | Version of the stackit-cert-manager-webhook Helm chart. Must be a version served by the chart index at https://stackitcloud.github.io/stackit-cert-manager-webhook, which lags behind the GitHub release tags. Only used when dns01.stackit is set. | `string` | `"0.4.9"` | no | +| [wildcard\_certificate\_name](#input\_wildcard\_certificate\_name) | Name of the wildcard Certificate and of the secret it writes, both in haproxy\_namespace. Only used when dns01 is set. | `string` | `"wildcard-tls"` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [cluster\_issuer\_name](#output\_cluster\_issuer\_name) | Name of the ClusterIssuer an application references from the cert-manager.io/cluster-issuer annotation on its Ingress. | +| [haproxy\_lb\_ip](#output\_haproxy\_lb\_ip) | External IP of the HAProxy LoadBalancer service. Point your DNS A record here before TLS provisioning can complete. | +| [haproxy\_namespace](#output\_haproxy\_namespace) | Namespace of the HAProxy ingress controller and of the wildcard certificate secret. | +| [ingress\_class\_name](#output\_ingress\_class\_name) | Name of the IngressClass an application puts on its Ingress to be served by this controller. | +| [wildcard\_certificate\_domain](#output\_wildcard\_certificate\_domain) | Domain the wildcard certificate covers, so the certificate is issued for `*.`. Equals dns01.zone\_name when the caller set no dns01.certificate\_domain. Null when dns01 is not set. | +| [wildcard\_certificate\_secret\_name](#output\_wildcard\_certificate\_secret\_name) | Name of the secret in haproxy\_namespace holding the wildcard certificate. Null when dns01 is not set. | + diff --git a/modules/kubernetes/ingress/buildingblock/ingress.tftest.hcl b/modules/kubernetes/ingress/buildingblock/ingress.tftest.hcl new file mode 100644 index 00000000..71cbef29 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/ingress.tftest.hcl @@ -0,0 +1,109 @@ +# Unit tests for the DNS-01 wildcard certificate. Both providers are mocked, so these tests need +# no cluster, no DNS zone and no ACME account, and they run in seconds. +# +# The values the module hands to Helm carry a sensitivity mark as soon as dns01 is set, because +# var.dns01 is sensitive as a whole. try(nonsensitive(...), ...) unwraps them where the mark is +# present and passes them through where it is not, which is the idiom main.tf already uses. + +mock_provider "kubernetes" {} +mock_provider "helm" {} + +variables { + acme_email = "ske@meshcloud.io" +} + +run "http01_only_without_dns01" { + command = plan + + variables { + dns01 = null + } + + assert { + condition = output.wildcard_certificate_domain == null + error_message = "Without dns01 there is no wildcard certificate, so the domain has to be null." + } + + assert { + condition = yamldecode(try(nonsensitive(helm_release.issuer.values[0]), helm_release.issuer.values[0])).wildcardCertificate.enabled == false + error_message = "Without dns01 the chart must not render the wildcard Certificate." + } +} + +# The regression guard that matters most: a caller that sets no certificate_domain has to keep +# rendering *., exactly as before the attribute existed. +run "certificate_domain_defaults_to_the_zone" { + command = plan + + variables { + dns01 = { + zone_name = "likvid.stackit.run" + stackit = { + project_id = "11111111-2222-3333-4444-555555555555" + service_account_key = "{}" + } + } + } + + assert { + condition = output.wildcard_certificate_domain == "likvid.stackit.run" + error_message = "A caller that sets no certificate_domain has to get the zone name." + } + + assert { + condition = yamldecode(try(nonsensitive(helm_release.issuer.values[0]), helm_release.issuer.values[0])).wildcardCertificate.domain == "likvid.stackit.run" + error_message = "The chart has to receive the zone name as the certificate domain, which renders *.likvid.stackit.run." + } +} + +run "certificate_domain_narrows_the_certificate" { + command = plan + + variables { + dns01 = { + zone_name = "likvid.stackit.run" + certificate_domain = "cluster1.likvid.stackit.run" + stackit = { + project_id = "11111111-2222-3333-4444-555555555555" + service_account_key = "{}" + } + } + } + + assert { + condition = output.wildcard_certificate_domain == "cluster1.likvid.stackit.run" + error_message = "The certificate has to cover the domain the caller asked for." + } + + assert { + condition = yamldecode(try(nonsensitive(helm_release.issuer.values[0]), helm_release.issuer.values[0])).wildcardCertificate.domain == "cluster1.likvid.stackit.run" + error_message = "The chart has to receive the caller's certificate domain, which renders *.cluster1.likvid.stackit.run." + } + + # The solver still answers for the whole zone. cert-manager matches every name below a zone in + # the dnsZones selector, so the narrower certificate is issued through the same solver. + assert { + condition = yamldecode(try(nonsensitive(helm_release.issuer.values[0]), helm_release.issuer.values[0])).dns01.zoneName == "likvid.stackit.run" + error_message = "The ClusterIssuer has to keep the zone in the dnsZones selector of the solver." + } +} + +# The solver only answers for names inside its zone, so a certificate domain outside the zone can +# never be issued. This run also covers the error message itself: var.dns01 is sensitive, so the +# message must not interpolate any part of it. +run "certificate_domain_outside_the_zone_is_rejected" { + command = plan + + variables { + dns01 = { + zone_name = "likvid.stackit.run" + certificate_domain = "cluster1.example.com" + stackit = { + project_id = "11111111-2222-3333-4444-555555555555" + service_account_key = "{}" + } + } + } + + expect_failures = [var.dns01] +} diff --git a/modules/kubernetes/ingress/buildingblock/logo.png b/modules/kubernetes/ingress/buildingblock/logo.png new file mode 100644 index 00000000..ff97e71f Binary files /dev/null and b/modules/kubernetes/ingress/buildingblock/logo.png differ diff --git a/modules/kubernetes/ingress/buildingblock/main.tf b/modules/kubernetes/ingress/buildingblock/main.tf new file mode 100644 index 00000000..ec193279 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/main.tf @@ -0,0 +1,244 @@ +locals { + # var.dns01 is sensitive as a whole, so every expression derived from it carries the sensitivity + # mark. count arguments, resource names and outputs reject a marked value, so unmark the plain + # facts β€” is DNS-01 on, which provider, which zone β€” while the credentials keep their mark. + # nonsensitive() rejects an argument that carries no mark, so try() falls back to the value. + dns01_enabled = try(nonsensitive(var.dns01 != null), var.dns01 != null) + dns01_stackit_enabled = local.dns01_enabled ? try(nonsensitive(var.dns01.stackit != null), var.dns01.stackit != null) : false + dns01_route53_enabled = local.dns01_enabled ? try(nonsensitive(var.dns01.route53 != null), var.dns01.route53 != null) : false + dns01_zone_name = local.dns01_enabled ? try(nonsensitive(var.dns01.zone_name), var.dns01.zone_name) : null + + # The solver answers for the whole zone, while the certificate may cover a narrower domain + # inside it. A caller that sets no certificate_domain gets the zone itself, which is the + # wildcard `*.`. + dns01_certificate_domain = local.dns01_enabled ? coalesce( + try(nonsensitive(var.dns01.certificate_domain), var.dns01.certificate_domain), + local.dns01_zone_name + ) : null + + # The chart derives the controller Service name from the release name. + haproxy_service_name = "${var.haproxy_release_name}-kubernetes-ingress" + + # Helm renders these values into the pod spec as YAML, and the API server rejects a resource + # quantity that is null, so drop every field the caller left unset. + resources = { + for name, spec in { + cert_manager = var.cert_manager_resources + cert_manager_webhook = var.cert_manager_webhook_resources + cert_manager_cainjector = var.cert_manager_cainjector_resources + cert_manager_startupapicheck = var.cert_manager_startupapicheck_resources + stackit_webhook = var.stackit_webhook_resources + haproxy = var.haproxy_resources + haproxy_crdjob = var.haproxy_crdjob_resources + } : name => { + requests = { for key, value in spec.requests : key => value if value != null } + limits = { for key, value in spec.limits : key => value if value != null } + } + } +} + +resource "kubernetes_namespace_v1" "cert_manager" { + metadata { + name = var.cert_manager_namespace + } +} + +resource "helm_release" "cert_manager" { + name = "cert-manager" + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + repository = "https://charts.jetstack.io" + chart = "cert-manager" + version = var.cert_manager_version + + create_namespace = false + wait = true + timeout = 300 + + values = [ + yamlencode({ + crds = { + enabled = true + keep = var.cert_manager_crds_keep + } + extraArgs = var.cert_manager_extra_args + + # The chart ships no resources for any of its four workloads, so each of them would run + # unbounded without these values. + resources = local.resources.cert_manager + webhook = { resources = local.resources.cert_manager_webhook } + cainjector = { resources = local.resources.cert_manager_cainjector } + startupapicheck = { resources = local.resources.cert_manager_startupapicheck } + }) + ] +} + +# The webhook mounts the STACKIT service account key from a secret, so the secret has to sit in +# the same namespace as the webhook pod. +resource "kubernetes_secret_v1" "stackit_dns01" { + count = local.dns01_stackit_enabled ? 1 : 0 + + metadata { + name = "stackit-sa-authentication" + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + } + + data = { + "sa.json" = var.dns01.stackit.service_account_key + } +} + +# cert-manager has no built-in solver for STACKIT DNS. This chart registers one under the +# acme.stackit.de API group, which the ClusterIssuer then calls as solverName "stackit". +resource "helm_release" "stackit_cert_manager_webhook" { + count = local.dns01_stackit_enabled ? 1 : 0 + + name = "stackit-cert-manager-webhook" + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + repository = "https://stackitcloud.github.io/stackit-cert-manager-webhook" + chart = "stackit-cert-manager-webhook" + version = var.stackit_webhook_version + + create_namespace = false + wait = true + timeout = 300 + + values = [ + yamlencode({ + groupName = "acme.stackit.de" + certManager = { + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + serviceAccountName = "cert-manager" + } + stackitSaAuthentication = { + enabled = true + secretName = kubernetes_secret_v1.stackit_dns01[0].metadata[0].name + } + resources = local.resources.stackit_webhook + }) + ] + + depends_on = [helm_release.cert_manager] +} + +# The route53 solver is built into cert-manager, so it needs no extra chart. It only needs the +# secret access key handed to it through a secret reference. +resource "kubernetes_secret_v1" "route53_dns01" { + count = local.dns01_route53_enabled ? 1 : 0 + + metadata { + name = "route53-dns01-credentials" + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + } + + data = { + "secret-access-key" = var.dns01.route53.secret_access_key + } +} + +resource "kubernetes_namespace_v1" "haproxy_ingress" { + metadata { + name = var.haproxy_namespace + } +} + +# The ClusterIssuer and the wildcard Certificate are custom resources whose CRDs only exist once +# cert-manager is installed. Helm renders and applies them without a plan-time schema lookup, +# which is what kubernetes_manifest would need β€” that lookup is the reason foundations had to run +# the ClusterIssuer as a separate Terraform unit. +resource "helm_release" "issuer" { + name = "ingress-issuer" + namespace = kubernetes_namespace_v1.cert_manager.metadata[0].name + chart = path.module + + atomic = true + wait = true + timeout = 300 + + values = [ + yamlencode({ + clusterIssuer = { + name = var.cluster_issuer_name + email = var.acme_email + server = var.acme_server + privateKeySecretName = var.acme_private_key_secret_name + } + ingressClassName = var.ingress_class_name + dns01 = { + zoneName = local.dns01_zone_name + stackit = local.dns01_stackit_enabled ? { + projectId = var.dns01.stackit.project_id + } : null + route53 = local.dns01_route53_enabled ? { + region = var.dns01.route53.region + hostedZoneID = var.dns01.route53.hosted_zone_id + accessKeyID = var.dns01.route53.access_key_id + secretAccessKeySecretName = kubernetes_secret_v1.route53_dns01[0].metadata[0].name + secretAccessKeySecretKey = "secret-access-key" + } : null + } + wildcardCertificate = { + enabled = local.dns01_enabled + domain = local.dns01_certificate_domain + name = var.wildcard_certificate_name + namespace = kubernetes_namespace_v1.haproxy_ingress.metadata[0].name + secretName = var.wildcard_certificate_name + } + }) + ] + + depends_on = [ + helm_release.cert_manager, + helm_release.stackit_cert_manager_webhook + ] +} + +resource "helm_release" "haproxy" { + name = var.haproxy_release_name + namespace = kubernetes_namespace_v1.haproxy_ingress.metadata[0].name + repository = "https://haproxytech.github.io/helm-charts" + chart = "kubernetes-ingress" + version = var.haproxy_version + + create_namespace = false + timeout = var.haproxy_timeout + + values = [ + yamlencode({ + # The chart requests 250m CPU and 400Mi memory for the controller and for the CRD Job, and + # sets no limit on either. + crdjob = { resources = local.resources.haproxy_crdjob } + + controller = merge( + { + replicaCount = var.haproxy_replica_count + ingressClass = var.ingress_class_name + ingressClassResource = { name = var.ingress_class_name } + resources = local.resources.haproxy + service = { + type = var.haproxy_service_type + annotations = var.haproxy_service_annotations + } + }, + # HAProxy serves this certificate for every host that brings no certificate of its own. + # controller.defaultTLSSecret.secretNamespace defaults to the release namespace, which is + # where the wildcard Certificate writes its secret, so only the name has to be set. + # Without DNS-01 the chart default stays in place and HAProxy keeps its self-signed + # certificate for unmatched hosts. + local.dns01_enabled ? { defaultTLSSecret = { secret = var.wildcard_certificate_name } } : {} + ) + }) + ] + + depends_on = [helm_release.issuer] +} + +# The cloud provider assigns the load balancer address after HAProxy is up. Foundations point +# their DNS A records at it. +data "kubernetes_service_v1" "haproxy_controller" { + metadata { + name = local.haproxy_service_name + namespace = kubernetes_namespace_v1.haproxy_ingress.metadata[0].name + } + + depends_on = [helm_release.haproxy] +} diff --git a/modules/kubernetes/ingress/buildingblock/outputs.tf b/modules/kubernetes/ingress/buildingblock/outputs.tf new file mode 100644 index 00000000..9514e7ca --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/outputs.tf @@ -0,0 +1,37 @@ +output "haproxy_lb_ip" { + description = "External IP of the HAProxy LoadBalancer service. Point your DNS A record here before TLS provisioning can complete." + value = data.kubernetes_service_v1.haproxy_controller.status[0].load_balancer[0].ingress[0].ip +} + +output "ingress_class_name" { + description = "Name of the IngressClass an application puts on its Ingress to be served by this controller." + value = var.ingress_class_name + + depends_on = [helm_release.haproxy] +} + +output "cluster_issuer_name" { + description = "Name of the ClusterIssuer an application references from the cert-manager.io/cluster-issuer annotation on its Ingress." + value = var.cluster_issuer_name + + depends_on = [helm_release.issuer] +} + +output "haproxy_namespace" { + description = "Namespace of the HAProxy ingress controller and of the wildcard certificate secret." + value = kubernetes_namespace_v1.haproxy_ingress.metadata[0].name +} + +output "wildcard_certificate_domain" { + description = "Domain the wildcard certificate covers, so the certificate is issued for `*.`. Equals dns01.zone_name when the caller set no dns01.certificate_domain. Null when dns01 is not set." + value = local.dns01_certificate_domain + + depends_on = [helm_release.issuer] +} + +output "wildcard_certificate_secret_name" { + description = "Name of the secret in haproxy_namespace holding the wildcard certificate. Null when dns01 is not set." + value = local.dns01_enabled ? var.wildcard_certificate_name : null + + depends_on = [helm_release.issuer] +} diff --git a/modules/kubernetes/ingress/buildingblock/templates/cluster-issuer.yaml b/modules/kubernetes/ingress/buildingblock/templates/cluster-issuer.yaml new file mode 100644 index 00000000..a92a958e --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/templates/cluster-issuer.yaml @@ -0,0 +1,43 @@ +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: {{ .Values.clusterIssuer.name }} +spec: + acme: + email: {{ .Values.clusterIssuer.email | quote }} + server: {{ .Values.clusterIssuer.server | quote }} + privateKeySecretRef: + name: {{ .Values.clusterIssuer.privateKeySecretName }} + solvers: + {{- with .Values.dns01.stackit }} + # DNS-01 through the STACKIT webhook. It answers for the whole zone, which is the only way + # to get a wildcard certificate β€” HTTP-01 can never validate one. + - dns01: + webhook: + solverName: stackit + groupName: acme.stackit.de + config: + projectId: {{ .projectId | quote }} + selector: + dnsZones: + - {{ $.Values.dns01.zoneName | quote }} + {{- end }} + {{- with .Values.dns01.route53 }} + # cert-manager solves DNS-01 against Route53 itself, so this branch needs no extra chart. + - dns01: + route53: + region: {{ .region | quote }} + hostedZoneID: {{ .hostedZoneID | quote }} + accessKeyID: {{ .accessKeyID | quote }} + secretAccessKeySecretRef: + name: {{ .secretAccessKeySecretName }} + key: {{ .secretAccessKeySecretKey }} + selector: + dnsZones: + - {{ $.Values.dns01.zoneName | quote }} + {{- end }} + # HTTP-01 stays in place for every hostname outside the DNS-01 zone. cert-manager prefers + # the DNS-01 solver above, because its dnsZones selector is more specific than this one. + - http01: + ingress: + ingressClassName: {{ .Values.ingressClassName }} diff --git a/modules/kubernetes/ingress/buildingblock/templates/wildcard-certificate.yaml b/modules/kubernetes/ingress/buildingblock/templates/wildcard-certificate.yaml new file mode 100644 index 00000000..459fa15e --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/templates/wildcard-certificate.yaml @@ -0,0 +1,23 @@ +{{- if .Values.wildcardCertificate.enabled }} +{{- /* +wildcardCertificate.domain is the DNS-01 zone, unless the caller narrowed the certificate to a name +inside that zone. Helm drops this comment from the rendered manifest, so the manifest of a caller +that sets no certificate domain stays exactly as it was. +*/}} +# One certificate covers every hostname in the zone. It lives in the HAProxy namespace, which +# outlives any application namespace, so tearing an application down never takes the certificate +# with it. HAProxy serves the resulting secret as its default TLS certificate. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: {{ .Values.wildcardCertificate.name }} + namespace: {{ .Values.wildcardCertificate.namespace }} +spec: + secretName: {{ .Values.wildcardCertificate.secretName }} + dnsNames: + - {{ printf "*.%s" .Values.wildcardCertificate.domain | quote }} + issuerRef: + name: {{ .Values.clusterIssuer.name }} + kind: ClusterIssuer + group: cert-manager.io +{{- end }} diff --git a/modules/kubernetes/ingress/buildingblock/variables.tf b/modules/kubernetes/ingress/buildingblock/variables.tf new file mode 100644 index 00000000..8db76f60 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/variables.tf @@ -0,0 +1,307 @@ +variable "cert_manager_version" { + type = string + default = "v1.20.0" + description = "Version of the cert-manager Helm chart. See https://github.com/cert-manager/cert-manager/releases." +} + +variable "cert_manager_namespace" { + type = string + default = "cert-manager" + description = "Namespace for cert-manager and, when DNS-01 runs through STACKIT, for the STACKIT cert-manager webhook. The webhook chart expects both in the same namespace." +} + +variable "cert_manager_extra_args" { + type = list(string) + # Cuts the retry backoff for failed certificate requests from the default 1h down to 1m, so an + # ACME order recovers quickly after a transient DNS or ingress problem. + default = ["--certificate-request-minimum-backoff-duration=1m"] + description = "Extra command line arguments for the cert-manager controller." +} + +variable "cert_manager_crds_keep" { + type = bool + default = false + description = "Keep the cert-manager CRDs when the Helm release is destroyed. Keeping them preserves existing Certificate and ClusterIssuer objects across a reinstall." +} + +variable "cert_manager_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "64Mi" } + limits = { cpu = "200m", memory = "256Mi" } + } + description = <<-EOT + Resource requests and limits of the cert-manager controller. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + The cert-manager Helm chart sets no resources at all and documents `10m` CPU and `32Mi` memory + as its example request. The memory request here is `64Mi` instead, because the controller keeps + informer caches for Certificates, Secrets and Ingresses and a container that runs out of memory + is OOMKilled rather than slowed down. A cluster that issues certificates continuously wants + `100m` CPU and `512Mi` memory. + EOT +} + +variable "cert_manager_webhook_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "32Mi" } + limits = { cpu = "100m", memory = "128Mi" } + } + description = <<-EOT + Resource requests and limits of the cert-manager admission webhook. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + The webhook validates cert-manager objects and holds no cache, so it is the smallest of the + three cert-manager pods. Every apply that touches a Certificate or an Issuer goes through it, + so keep the limit above the request. A production cluster wants `100m` CPU and `256Mi` memory. + EOT +} + +variable "cert_manager_cainjector_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "64Mi" } + limits = { cpu = "200m", memory = "256Mi" } + } + description = <<-EOT + Resource requests and limits of the cert-manager cainjector. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + The cainjector watches every Secret in the cluster, so its memory grows with the number of + Secrets. cert-manager issue #6217 reports it reaching gigabytes on large clusters. The limit + here is `256Mi` because a demonstration cluster holds few Secrets, and a production cluster + wants `512Mi` or more together with the `--namespace` flag that narrows the watch. + EOT +} + +variable "cert_manager_startupapicheck_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "32Mi" } + limits = { cpu = "100m", memory = "128Mi" } + } + description = <<-EOT + Resource requests and limits of the cert-manager startupapicheck Job. The default is sized for + a demonstration cluster and a production consumer has to raise it. + + The Job runs once per install, checks that the webhook answers and then exits, so it never + holds resources for long. Its request still has to fit on a node, which is why it is kept this + small. + EOT +} + +variable "haproxy_version" { + type = string + default = "1.49.0" + description = "Version of the haproxytech/kubernetes-ingress Helm chart. See https://github.com/haproxytech/helm-charts/blob/main/kubernetes-ingress/Chart.yaml." +} + +variable "haproxy_namespace" { + type = string + default = "haproxy-ingress" + description = "Namespace for the HAProxy ingress controller. The wildcard certificate is created here as well, so its secret survives the teardown of any application namespace." +} + +variable "haproxy_release_name" { + type = string + default = "haproxy" + description = "Helm release name of the HAProxy ingress controller. The chart names the controller Service '-kubernetes-ingress'." +} + +variable "haproxy_replica_count" { + type = number + # The chart defaults to 2. One replica is enough to serve traffic and it halves what the + # controller costs on a demonstration cluster. + default = 1 + description = "Number of HAProxy ingress controller replicas. The default of 1 is sized for a demonstration cluster and gives no redundancy: every restart or node drain interrupts ingress traffic. Production wants at least 2, spread over separate nodes." +} + +variable "haproxy_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "100m", memory = "256Mi" } + limits = { cpu = "500m", memory = "768Mi" } + } + description = <<-EOT + Resource requests and limits of the HAProxy ingress controller. The default is sized for a + demonstration cluster and a production consumer has to raise it. + + The chart requests `250m` CPU and `400Mi` memory and sets no limit. The pod runs two processes, + HAProxy itself and the Go controller, and the container entrypoint hands HAProxy two thirds of + the cgroup memory limit. Users of this chart version report that HAProxy reloads in a loop + instead of serving traffic when the memory limit stays below `500Mi`, and a maintainer + recommends at least `1Gi` (haproxytech/kubernetes-ingress issue #799). The `768Mi` limit here is + the smallest value that clears that threshold with headroom. Production wants `1Gi` to `2Gi` + and a CPU limit that matches the traffic the controller has to terminate. + EOT +} + +variable "haproxy_crdjob_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "50m", memory = "64Mi" } + limits = { cpu = "200m", memory = "256Mi" } + } + description = <<-EOT + Resource requests and limits of the Job the HAProxy chart runs to install its CRDs. The default + is sized for a demonstration cluster and a production consumer has to raise it. + + The chart requests `250m` CPU and `400Mi` memory for this Job. The Job applies a handful of + CRDs and exits, so a much smaller request is enough, and a smaller request also means the Job + still schedules on a small node. + EOT +} + +variable "haproxy_service_type" { + type = string + # The chart defaults to NodePort, which gives no public address at all, so this value is + # load-bearing rather than cosmetic. + default = "LoadBalancer" + description = "Service type of the HAProxy ingress controller." +} + +variable "haproxy_service_annotations" { + type = map(string) + default = {} + description = <<-EOT + Annotations on the HAProxy controller Service. The cloud provider reads them to configure the + load balancer. Two values matter in practice: + - AKS needs `service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path = "/healthz"`. + - STACKIT uses `lb.stackit.cloud/internal-lb` to keep the load balancer off the public internet. + EOT +} + +variable "haproxy_timeout" { + type = number + default = 1200 + description = "Seconds to wait for the HAProxy Helm release to become ready. The default of 20 minutes covers the time a cloud provider takes to provision the load balancer." +} + +variable "ingress_class_name" { + type = string + default = "haproxy" + description = "Name of the IngressClass the controller serves. The HTTP-01 solver of the ClusterIssuer uses the same name." +} + +variable "acme_email" { + type = string + description = "Contact address Let's Encrypt uses for expiry warnings and account recovery." +} + +variable "acme_server" { + type = string + default = "https://acme-v02.api.letsencrypt.org/directory" + description = "ACME directory URL. Point this at https://acme-staging-v02.api.letsencrypt.org/directory while you test, because the production endpoint has strict rate limits." +} + +variable "cluster_issuer_name" { + type = string + default = "letsencrypt-prod" + description = "Name of the ClusterIssuer. Application teams reference it from the cert-manager.io/cluster-issuer annotation on their Ingress." +} + +variable "acme_private_key_secret_name" { + type = string + default = "letsencrypt-prod-account-key" + description = "Name of the secret in which cert-manager stores the ACME account private key." +} + +variable "wildcard_certificate_name" { + type = string + default = "wildcard-tls" + description = "Name of the wildcard Certificate and of the secret it writes, both in haproxy_namespace. Only used when dns01 is set." +} + +# The default follows the published chart index rather than the GitHub release tags, because the +# two diverged: the repository tagged a `stackit-cert-manager-webhook-0.4.10` release, but +# https://stackitcloud.github.io/stackit-cert-manager-webhook/index.yaml was never regenerated and +# still ends at 0.4.9. `helm_release` resolves the version through that index, so pinning 0.4.10 +# fails at apply with "no chart version found". Raise this default only after the index serves the +# newer version. +variable "stackit_webhook_version" { + type = string + default = "0.4.9" + description = "Version of the stackit-cert-manager-webhook Helm chart. Must be a version served by the chart index at https://stackitcloud.github.io/stackit-cert-manager-webhook, which lags behind the GitHub release tags. Only used when dns01.stackit is set." +} + +variable "stackit_webhook_resources" { + type = object({ + requests = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + limits = optional(object({ cpu = optional(string), memory = optional(string) }), {}) + }) + nullable = false + default = { + requests = { cpu = "10m", memory = "64Mi" } + limits = { cpu = "100m", memory = "128Mi" } + } + description = <<-EOT + Resource requests and limits of the STACKIT cert-manager webhook. Only used when dns01.stackit + is set. The default is sized for a demonstration cluster and a production consumer has to raise + it. + + The chart sets no resources and its values file states that `100m` CPU and `128Mi` memory are + enough for the webhook, which is what the limit uses. The webhook answers one DNS-01 challenge + per certificate renewal, so the request stays well below that. Production wants the chart's own + figures as the request as well. + EOT +} + +variable "dns01" { + description = <<-EOT + Enables a wildcard certificate via DNS-01. Set exactly one provider. Null keeps HTTP-01 + per-hostname issuance. + + `zone_name` is the DNS zone the solver is authorised for, and the ClusterIssuer selects the + solver for every name inside it. `certificate_domain` is the domain the wildcard certificate + covers and defaults to `zone_name`, which gives `*.`. Set it to a name below the + zone, for example `cluster1.likvid.stackit.run` inside the zone `likvid.stackit.run`, to narrow + the certificate to that label while the solver keeps answering for the whole zone. + EOT + type = object({ + zone_name = string + certificate_domain = optional(string) + stackit = optional(object({ project_id = string, service_account_key = string })) + route53 = optional(object({ hosted_zone_id = string, access_key_id = string, secret_access_key = string, region = optional(string, "eu-central-1") })) + }) + default = null + sensitive = true + + validation { + condition = var.dns01 == null || ( + (try(var.dns01.stackit, null) == null ? 0 : 1) + (try(var.dns01.route53, null) == null ? 0 : 1) == 1 + ) + error_message = "Set exactly one DNS-01 provider in var.dns01: either stackit or route53." + } + + # The message carries no interpolation, because var.dns01 is sensitive and Terraform refuses to + # print a sensitive value in an error message. + validation { + condition = var.dns01 == null || try(var.dns01.certificate_domain, null) == null || endswith(var.dns01.certificate_domain, var.dns01.zone_name) + error_message = "dns01.certificate_domain must be dns01.zone_name or a name below it, because the DNS-01 solver only answers for names inside the zone." + } +} diff --git a/modules/kubernetes/ingress/buildingblock/versions.tf b/modules/kubernetes/ingress/buildingblock/versions.tf new file mode 100644 index 00000000..cc6c09d8 --- /dev/null +++ b/modules/kubernetes/ingress/buildingblock/versions.tf @@ -0,0 +1,16 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + helm = { + source = "hashicorp/helm" + # The helm provider takes its cluster credentials as the `kubernetes = {}` attribute + # starting with 3.0.0. Earlier versions expect a `kubernetes {}` block instead. + version = ">= 3.0.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38" + } + } +} diff --git a/modules/kubernetes/platform/buildingblock/APP_TEAM_README.md b/modules/kubernetes/platform/buildingblock/APP_TEAM_README.md new file mode 100644 index 00000000..79914d8f --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/APP_TEAM_README.md @@ -0,0 +1,44 @@ +This building block connects a Kubernetes cluster to meshStack, so that teams can order a namespace +on it from the marketplace. It creates the service accounts meshStack authenticates with inside the +cluster, registers the cluster as a platform, and adds one landing zone per environment with its own +quotas. After it runs, meshStack replicates every tenant of the platform into a namespace on the +cluster and keeps the role bindings in sync with the project roles. + +## 🎯 When to use it + +Use this building block when you: + +- run a Kubernetes cluster and want to hand out namespaces on it through meshStack instead of by hand +- want project roles in meshStack to drive `admin`, `edit` and `view` access inside the namespace +- want meshStack to enforce CPU, memory and storage quotas per namespace +- want usage data from the cluster to reach meshMetering for chargeback + +Do not use it for AKS. meshStack models AKS namespace platforms with a different configuration and an +Entra service principal, so AKS needs its own registration module. + +## πŸ’‘ Usage examples + +**Example 1: Opening a new cluster to the marketplace** +Your team has just finished a STACKIT Kubernetes Engine cluster and wants application teams to be able +to order namespaces on it. You order this building block with the cluster's API server URL and its +credentials. Application teams then see a `dev` and a `prod` landing zone in the marketplace and can +order a namespace on either. + +**Example 2: Tightening the quotas on a small cluster** +Your cluster is smaller than the defaults assume, so you want half the CPU and memory per namespace. +You order the building block with your own `quota_definitions` and `landing_zones` values, and +meshStack rejects any tenant request that goes over the new limits. + +## πŸ“Š Shared Responsibility + +| Responsibility | Platform Team | Application Team | +|---|:---:|:---:| +| Provide and operate the Kubernetes cluster | βœ… | ❌ | +| Create the replicator and metering service accounts in the cluster | βœ… | ❌ | +| Register the platform and its landing zones in meshStack | βœ… | ❌ | +| Choose the quota limits and the auto-approval thresholds | βœ… | ❌ | +| Rotate the cluster credentials the registration uses | βœ… | ❌ | +| Order a namespace on one of the landing zones | ❌ | βœ… | +| Deploy and operate workloads inside the namespace | ❌ | βœ… | +| Request a quota increase when a workload outgrows the landing zone | ❌ | βœ… | +| Monitor application health and logs | ❌ | βœ… | diff --git a/modules/kubernetes/platform/buildingblock/README.md b/modules/kubernetes/platform/buildingblock/README.md new file mode 100644 index 00000000..8e5e8a41 --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/README.md @@ -0,0 +1,126 @@ +--- +name: Kubernetes Platform Registration +supportedPlatforms: + - kubernetes +description: Registers a Kubernetes cluster as a meshStack platform of type kubernetes, with its namespace landing zones and the in-cluster service accounts meshStack authenticates with. +# The module creates its identities inside the target cluster and receives the cluster credentials +# as inputs, so there is no cloud-side setup to perform ahead of time. +requiresBackplane: false +--- + +# Kubernetes Platform Registration Building Block + +This module registers a Kubernetes cluster as a meshStack platform whose tenants are namespaces. It +creates three things: + +1. The in-cluster identities meshStack authenticates with β€” a replicator service account that creates + namespaces, resource quotas and role bindings, and a metering service account that reads pods and + persistent volume claims. +2. A `meshstack_platform` of type `kubernetes` that points at the cluster's API server. +3. One `meshstack_landingzone` per environment, each with its own quotas and tags. + +Nothing in the module names a cloud provider. It serves STACKIT Kubernetes Engine today and any +conformant cluster later, as long as the cluster accepts service account tokens for authentication. + +## Why this module cannot serve AKS + +meshStack models AKS namespace platforms differently, and the difference is a modelling fact rather +than a design choice: + +- AKS uses `spec.config.aks`, not `spec.config.kubernetes`, and its landing zones use + `platform_properties.aks`. +- AKS authenticates with an Entra service principal through workload identity federation, read from + `data.meshstack_integrations`, instead of an in-cluster service account token. +- The AKS config carries fields that have no counterpart here: `group_name_pattern`, + `user_lookup_strategy`, `send_azure_invitation_mail`, `redirect_url`, `aks_subscription_id`, + `aks_cluster_name`, `aks_resource_group` and the Entra tenant. + +An AKS platform registration therefore needs its own module. + +## What the module replaces + +Five copies of this configuration existed before, three of them on the SKE side. This module is built +from those three and turns every value they disagreed on into a variable. The defaults are the +`likvid-cloudfoundation` values. + +## Where the credentials come from + +The module needs cluster-admin credentials to create the service accounts and the cluster roles. +Supply them through `kube_host`, `cluster_ca_certificate`, `client_certificate` and `client_key`, for +example from the `provider_config` output of `modules/stackit/ske`. Callers that generate their own +`provider "kubernetes"` block β€” Terragrunt does this β€” can leave the three credential variables unset +and pass only `kube_host`, which meshStack also stores as the platform endpoint. + +## Running more than one registration on one cluster + +The in-cluster resource names are fixed (`meshfed-service`, `meshfed-metering`) so that an existing +deployment of the `terraform-kubernetes-meshplatform` module can be moved into this one without +renaming anything. Set `resource_name_suffix` when a single cluster carries more than one meshStack +platform registration, so the service accounts and cluster roles do not collide. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.12.0 | +| [kubernetes](#requirement\_kubernetes) | >= 2.38.0 | +| [meshstack](#requirement\_meshstack) | >= 0.20.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [kubernetes_cluster_role.metering](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role) | resource | +| [kubernetes_cluster_role.replicator](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role) | resource | +| [kubernetes_cluster_role_binding.metering](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role_binding) | resource | +| [kubernetes_cluster_role_binding.replicator](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role_binding) | resource | +| [kubernetes_namespace.meshcloud](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource | +| [kubernetes_secret.metering](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret) | resource | +| [kubernetes_secret.replicator](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret) | resource | +| [kubernetes_service_account.metering](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account) | resource | +| [kubernetes_service_account.replicator](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account) | resource | +| [meshstack_landingzone.this](https://registry.terraform.io/providers/meshcloud/meshstack/latest/docs/resources/landingzone) | resource | +| [meshstack_platform.this](https://registry.terraform.io/providers/meshcloud/meshstack/latest/docs/resources/platform) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [client\_certificate](#input\_client\_certificate) | PEM encoded client certificate the module authenticates with while it creates the in-cluster service accounts. Leave unset when the caller supplies its own provider configuration. | `string` | `null` | no | +| [client\_key](#input\_client\_key) | PEM encoded client key that belongs to `client_certificate`. Leave unset when the caller supplies its own provider configuration. | `string` | `null` | no | +| [cluster\_ca\_certificate](#input\_cluster\_ca\_certificate) | PEM encoded CA certificate of the cluster. Leave unset when the caller supplies its own provider configuration. | `string` | `null` | no | +| [disable\_ssl\_validation](#input\_disable\_ssl\_validation) | Skip SSL validation when meshStack calls the Kubernetes API server. SKE clusters serve a certificate that meshStack does not trust by default, which is why this is on. | `bool` | `true` | no | +| [documentation\_url](#input\_documentation\_url) | Link to the platform documentation shown in meshPanel. | `string` | `""` | no | +| [kube\_host](#input\_kube\_host) | URL of the Kubernetes API server, for example `https://k8s.example.com:6443`. meshStack calls this URL to replicate tenants and to collect metering data. | `string` | n/a | yes | +| [landing\_zones](#input\_landing\_zones) | Landing zones to create for the platform, keyed by environment. The key becomes the suffix of the landing zone identifier, for example `dev` gives `ske-namespace-dev`. | 
map(object({
    display_name = string
    description  = string
    info_link    = optional(string, "")
    tags         = optional(map(list(string)), {})
    quotas       = optional(list(object({ key = string, value = number })), [])
  }))
| 
{
  "dev": {
    "description": "Landing zone for development workloads.",
    "display_name": "SKE Kubernetes Namespace – Development",
    "quotas": [
      {
        "key": "limits.cpu",
        "value": 500
      },
      {
        "key": "requests.cpu",
        "value": 250
      },
      {
        "key": "limits.memory",
        "value": 512
      },
      {
        "key": "requests.memory",
        "value": 256
      },
      {
        "key": "requests.storage",
        "value": 1
      },
      {
        "key": "persistentvolumeclaims",
        "value": 2
      }
    ],
    "tags": {
      "LandingZoneFamily": [
        "cloud-native"
      ],
      "confidentiality": [
        "internal"
      ],
      "environment": [
        "dev"
      ]
    }
  },
  "prod": {
    "description": "Landing zone for production workloads.",
    "display_name": "SKE Kubernetes Namespace – Production",
    "quotas": [
      {
        "key": "limits.cpu",
        "value": 1000
      },
      {
        "key": "requests.cpu",
        "value": 500
      },
      {
        "key": "limits.memory",
        "value": 1024
      },
      {
        "key": "requests.memory",
        "value": 512
      },
      {
        "key": "requests.storage",
        "value": 2
      },
      {
        "key": "persistentvolumeclaims",
        "value": 4
      }
    ],
    "tags": {
      "LandingZoneFamily": [
        "cloud-native"
      ],
      "confidentiality": [
        "public"
      ],
      "environment": [
        "prod"
      ]
    }
  }
}
| no | +| [location\_identifier](#input\_location\_identifier) | Identifier of the meshStack location the platform is registered in. | `string` | n/a | yes | +| [metering\_additional\_rules](#input\_metering\_additional\_rules) | Extra RBAC rules added to the metering cluster role. | 
list(object({
    api_groups        = list(string)
    resources         = list(string)
    verbs             = list(string)
    resource_names    = optional(list(string))
    non_resource_urls = optional(list(string))
  }))
| `[]` | no | +| [metering\_enabled](#input\_metering\_enabled) | Create the metering service account and register metering on the platform. Turn this off when meshStack should not collect usage data from the cluster. | `bool` | `true` | no | +| [metering\_processing](#input\_metering\_processing) | How long meshMetering keeps timelines and raw data. Only used when `metering_enabled` is true. | 
object({
    compact_timelines_after_days = optional(number, 30)
    delete_raw_data_after_days   = optional(number, 65)
  })
| `{}` | no | +| [namespace\_name\_pattern](#input\_namespace\_name\_pattern) | Pattern meshStack uses to name the namespace it creates for a tenant. | `string` | `"#{workspaceIdentifier}-#{projectIdentifier}"` | no | +| [owning\_workspace\_identifier](#input\_owning\_workspace\_identifier) | Identifier of the meshStack workspace that owns the platform and its landing zones. | `string` | n/a | yes | +| [platform\_description](#input\_platform\_description) | Description of the platform as users see it in meshPanel. | `string` | `"Provides a kubernetes namespace on STACKIT Kubernetes Engine (SKE)."` | no | +| [platform\_display\_name](#input\_platform\_display\_name) | Name of the platform as users see it in meshPanel. | `string` | `"Kubernetes namespace on SKE"` | no | +| [platform\_name](#input\_platform\_name) | meshStack platform identifier. The landing zones derive their names from it, for example `ske-namespace-dev`. | `string` | `"ske-namespace"` | no | +| [quota\_definitions](#input\_quota\_definitions) | Quota keys a tenant can request on this platform, with the upper bound and the threshold below which meshStack approves a request automatically. | 
list(object({
    quota_key               = string
    label                   = string
    description             = string
    unit                    = string
    min_value               = number
    max_value               = number
    auto_approval_threshold = number
  }))
| 
[
  {
    "auto_approval_threshold": 1000,
    "description": "The sum of CPU limits across all pods in a non-terminal state cannot exceed this value.",
    "label": "CPU limit",
    "max_value": 1000,
    "min_value": 0,
    "quota_key": "limits.cpu",
    "unit": "m"
  },
  {
    "auto_approval_threshold": 500,
    "description": "The sum of CPU requests across all pods in a non-terminal state cannot exceed this value.",
    "label": "CPU requests",
    "max_value": 1000,
    "min_value": 0,
    "quota_key": "requests.cpu",
    "unit": "m"
  },
  {
    "auto_approval_threshold": 1024,
    "description": "The sum of memory limits across all pods in a non-terminal state cannot exceed this value.",
    "label": "Memory limit",
    "max_value": 1024,
    "min_value": 0,
    "quota_key": "limits.memory",
    "unit": "Mi"
  },
  {
    "auto_approval_threshold": 512,
    "description": "The sum of memory requests across all pods in a non-terminal state cannot exceed this value.",
    "label": "Memory requests",
    "max_value": 1024,
    "min_value": 0,
    "quota_key": "requests.memory",
    "unit": "Mi"
  },
  {
    "auto_approval_threshold": 2,
    "description": "Across all persistent volume claims, the sum of storage requests cannot exceed this value.",
    "label": "Total Storage Requests",
    "max_value": 5,
    "min_value": 0,
    "quota_key": "requests.storage",
    "unit": "Gi"
  },
  {
    "auto_approval_threshold": 2,
    "description": "The total number of PersistentVolumeClaims that can exist in the namespace.",
    "label": "Persistent Volume Claims",
    "max_value": 4,
    "min_value": 0,
    "quota_key": "persistentvolumeclaims",
    "unit": ""
  }
]
| no | +| [replicator\_additional\_rules](#input\_replicator\_additional\_rules) | Extra RBAC rules added to the replicator cluster role. | 
list(object({
    api_groups        = list(string)
    resources         = list(string)
    verbs             = list(string)
    resource_names    = optional(list(string))
    non_resource_urls = optional(list(string))
  }))
| `[]` | no | +| [resource\_name\_suffix](#input\_resource\_name\_suffix) | Suffix appended to the in-cluster resource names. Set it when one cluster carries more than one meshStack platform registration, so the service accounts and cluster roles do not collide. | `string` | `""` | no | +| [service\_account\_namespace](#input\_service\_account\_namespace) | Namespace that holds the replicator and metering service accounts. | `string` | `"meshcloud"` | no | +| [support\_url](#input\_support\_url) | Link to the support channel shown in meshPanel. | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [landing\_zone\_identifiers](#output\_landing\_zone\_identifiers) | meshStack landing zone identifiers keyed by environment. | +| [landing\_zone\_refs](#output\_landing\_zone\_refs) | meshStack landing zone references keyed by environment, for use in building block compositions. | +| [metering\_token](#output\_metering\_token) | Access token of the metering service account, or null when metering is off. | +| [platform\_identifier](#output\_platform\_identifier) | Platform identifier in the `.` form meshStack uses to address a platform, for example `ske-namespace.eu-de-central`. | +| [platform\_ref](#output\_platform\_ref) | Reference to the platform, for use in building block compositions that create tenants on it. | +| [replicator\_token](#output\_replicator\_token) | Access token of the replicator service account. meshStack already holds this token, so you only need the output to debug the cluster connection. | + diff --git a/modules/kubernetes/platform/buildingblock/logo.png b/modules/kubernetes/platform/buildingblock/logo.png new file mode 100644 index 00000000..d6063251 Binary files /dev/null and b/modules/kubernetes/platform/buildingblock/logo.png differ diff --git a/modules/kubernetes/platform/buildingblock/main.tf b/modules/kubernetes/platform/buildingblock/main.tf new file mode 100644 index 00000000..02deb0e2 --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/main.tf @@ -0,0 +1,204 @@ +# In-cluster identities meshStack authenticates with. +# +# The replicator creates namespaces, resource quotas and role bindings for every tenant. The +# metering service account reads pods and persistent volume claims to collect usage data. Both are +# plain Kubernetes service accounts with a long-lived token, so no cloud identity provider is +# involved and the module works on any conformant cluster. + +resource "kubernetes_namespace" "meshcloud" { + metadata { + name = var.service_account_namespace + } +} + +# ── Replicator ────────────────────────────────────────────────────────────── + +resource "kubernetes_service_account" "replicator" { + metadata { + name = local.replicator_name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + annotations = { + "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" + } + } +} + +resource "kubernetes_secret" "replicator" { + metadata { + name = local.replicator_name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + annotations = { + "kubernetes.io/service-account.name" = kubernetes_service_account.replicator.metadata[0].name + } + } + + type = "kubernetes.io/service-account-token" + wait_for_service_account_token = true +} + +resource "kubernetes_cluster_role" "replicator" { + metadata { + name = local.replicator_name + annotations = { + "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" + } + } + + rule { + api_groups = [""] + resources = ["namespaces"] + verbs = ["get", "list", "watch", "create", "delete", "update"] + } + + rule { + api_groups = [""] + resources = ["resourcequotas", "resourcequotas/status"] + verbs = ["get", "list", "watch", "create", "delete", "deletecollection", "patch", "update"] + } + + rule { + api_groups = [""] + resources = ["appliedclusterresourcequotas", "clusterresourcequotas", "clusterresourcequotas/status"] + verbs = ["get", "list", "watch", "create", "delete", "deletecollection", "patch", "update"] + } + + rule { + api_groups = ["", "rbac.authorization.k8s.io"] + resources = ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] + verbs = ["get", "list", "watch"] + } + + rule { + api_groups = ["", "rbac.authorization.k8s.io"] + resources = ["rolebindings"] + verbs = ["create", "delete", "update"] + } + + rule { + api_groups = ["", "rbac.authorization.k8s.io"] + resources = ["clusterroles"] + verbs = ["bind"] + resource_names = ["admin", "edit", "view"] + } + + dynamic "rule" { + for_each = var.replicator_additional_rules + content { + api_groups = rule.value.api_groups + resources = rule.value.resources + verbs = rule.value.verbs + resource_names = rule.value.resource_names + non_resource_urls = rule.value.non_resource_urls + } + } +} + +resource "kubernetes_cluster_role_binding" "replicator" { + metadata { + name = local.replicator_name + annotations = { + "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" + } + } + + subject { + kind = "ServiceAccount" + name = kubernetes_service_account.replicator.metadata[0].name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + } + + role_ref { + api_group = "rbac.authorization.k8s.io" + kind = "ClusterRole" + name = kubernetes_cluster_role.replicator.metadata[0].name + } +} + +# ── Metering ──────────────────────────────────────────────────────────────── + +resource "kubernetes_service_account" "metering" { + count = var.metering_enabled ? 1 : 0 + + metadata { + name = local.metering_name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + annotations = { + "io.meshcloud/meshstack.metering-kubernetes.version" = "1.0" + } + } +} + +resource "kubernetes_secret" "metering" { + count = var.metering_enabled ? 1 : 0 + + metadata { + name = local.metering_name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + annotations = { + "kubernetes.io/service-account.name" = kubernetes_service_account.metering[0].metadata[0].name + } + } + + type = "kubernetes.io/service-account-token" + wait_for_service_account_token = true +} + +resource "kubernetes_cluster_role" "metering" { + count = var.metering_enabled ? 1 : 0 + + metadata { + name = local.metering_name + annotations = { + "io.meshcloud/meshstack.metering-kubernetes.version" = "1.0" + } + } + + rule { + api_groups = [""] + resources = ["pods", "persistentvolumeclaims"] + verbs = ["get", "list"] + } + + dynamic "rule" { + for_each = var.metering_additional_rules + content { + api_groups = rule.value.api_groups + resources = rule.value.resources + verbs = rule.value.verbs + resource_names = rule.value.resource_names + non_resource_urls = rule.value.non_resource_urls + } + } +} + +resource "kubernetes_cluster_role_binding" "metering" { + count = var.metering_enabled ? 1 : 0 + + metadata { + name = local.metering_name + annotations = { + "io.meshcloud/meshstack.metering-kubernetes.version" = "1.0" + } + } + + subject { + kind = "ServiceAccount" + name = kubernetes_service_account.metering[0].metadata[0].name + namespace = kubernetes_namespace.meshcloud.metadata[0].name + } + + role_ref { + api_group = "rbac.authorization.k8s.io" + kind = "ClusterRole" + name = kubernetes_cluster_role.metering[0].metadata[0].name + } +} + +locals { + name_suffix = var.resource_name_suffix == "" ? "" : "-${var.resource_name_suffix}" + replicator_name = "meshfed-service${local.name_suffix}" + metering_name = "meshfed-metering${local.name_suffix}" + + replicator_token = kubernetes_secret.replicator.data["token"] + metering_token = var.metering_enabled ? kubernetes_secret.metering[0].data["token"] : null +} diff --git a/modules/kubernetes/platform/buildingblock/meshstack_landingzone.tf b/modules/kubernetes/platform/buildingblock/meshstack_landingzone.tf new file mode 100644 index 00000000..1a2f028c --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/meshstack_landingzone.tf @@ -0,0 +1,42 @@ +resource "meshstack_landingzone" "this" { + for_each = var.landing_zones + + metadata = { + name = "${var.platform_name}-${each.key}" + owned_by_workspace = var.owning_workspace_identifier + tags = each.value.tags + } + + spec = { + display_name = each.value.display_name + description = each.value.description + automate_deletion_approval = true + automate_deletion_replication = true + info_link = each.value.info_link + + platform_ref = { + uuid = meshstack_platform.this.metadata.uuid + } + + platform_properties = { + kubernetes = { + kubernetes_role_mappings = [ + { + project_role_ref = { name = "admin" } + platform_roles = ["admin"] + }, + { + project_role_ref = { name = "user" } + platform_roles = ["edit"] + }, + { + project_role_ref = { name = "reader" } + platform_roles = ["view"] + }, + ] + } + } + + quotas = each.value.quotas + } +} diff --git a/modules/kubernetes/platform/buildingblock/meshstack_platform.tf b/modules/kubernetes/platform/buildingblock/meshstack_platform.tf new file mode 100644 index 00000000..49491324 --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/meshstack_platform.tf @@ -0,0 +1,62 @@ +resource "meshstack_platform" "this" { + metadata = { + name = var.platform_name + owned_by_workspace = var.owning_workspace_identifier + } + + spec = { + display_name = var.platform_display_name + description = var.platform_description + endpoint = var.kube_host + documentation_url = var.documentation_url + support_url = var.support_url + + location_ref = { + name = var.location_identifier + } + + # meshStack operators change publication state and access restriction after the first + # deployment, so the lifecycle block below keeps Terraform from resetting them. + availability = { + publication_state = "PUBLISHED" + restriction = "PUBLIC" + restricted_to_workspaces = [] + } + + contributing_workspaces = [] + + config = { + kubernetes = { + base_url = var.kube_host + disable_ssl_validation = var.disable_ssl_validation + + replication = { + client_config = { + access_token = { + secret_value = local.replicator_token + } + } + namespace_name_pattern = var.namespace_name_pattern + } + + metering = var.metering_enabled ? { + client_config = { + access_token = { + secret_value = local.metering_token + } + } + processing = { + compact_timelines_after_days = var.metering_processing.compact_timelines_after_days + delete_raw_data_after_days = var.metering_processing.delete_raw_data_after_days + } + } : null + } + } + + quota_definitions = var.quota_definitions + } + + lifecycle { + ignore_changes = [spec.availability] + } +} diff --git a/modules/kubernetes/platform/buildingblock/outputs.tf b/modules/kubernetes/platform/buildingblock/outputs.tf new file mode 100644 index 00000000..066ca79b --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/outputs.tf @@ -0,0 +1,34 @@ +output "platform_identifier" { + description = "Platform identifier in the `.` form meshStack uses to address a platform, for example `ske-namespace.eu-de-central`." + value = "${meshstack_platform.this.metadata.name}.${meshstack_platform.this.spec.location_ref.name}" +} + +output "platform_ref" { + description = "Reference to the platform, for use in building block compositions that create tenants on it." + value = { + uuid = meshstack_platform.this.metadata.uuid + kind = "meshPlatform" + } +} + +output "landing_zone_identifiers" { + description = "meshStack landing zone identifiers keyed by environment." + value = { for env, lz in meshstack_landingzone.this : env => lz.metadata.name } +} + +output "landing_zone_refs" { + description = "meshStack landing zone references keyed by environment, for use in building block compositions." + value = { for env, lz in meshstack_landingzone.this : env => lz.ref } +} + +output "replicator_token" { + description = "Access token of the replicator service account. meshStack already holds this token, so you only need the output to debug the cluster connection." + value = local.replicator_token + sensitive = true +} + +output "metering_token" { + description = "Access token of the metering service account, or null when metering is off." + value = local.metering_token + sensitive = true +} diff --git a/modules/kubernetes/platform/buildingblock/provider.tf b/modules/kubernetes/platform/buildingblock/provider.tf new file mode 100644 index 00000000..f3936b3c --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/provider.tf @@ -0,0 +1,9 @@ +# Callers that drive this module from Terragrunt usually replace this file with a generated +# `provider.tf` of their own. In that case the three credential variables stay unset and the +# generated block carries the credentials instead. +provider "kubernetes" { + host = var.kube_host + cluster_ca_certificate = var.cluster_ca_certificate + client_certificate = var.client_certificate + client_key = var.client_key +} diff --git a/modules/kubernetes/platform/buildingblock/variables.tf b/modules/kubernetes/platform/buildingblock/variables.tf new file mode 100644 index 00000000..19cda78b --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/variables.tf @@ -0,0 +1,269 @@ +variable "kube_host" { + type = string + nullable = false + description = "URL of the Kubernetes API server, for example `https://k8s.example.com:6443`. meshStack calls this URL to replicate tenants and to collect metering data." +} + +variable "cluster_ca_certificate" { + type = string + nullable = true + default = null + description = "PEM encoded CA certificate of the cluster. Leave unset when the caller supplies its own provider configuration." +} + +variable "client_certificate" { + type = string + nullable = true + default = null + sensitive = true + description = "PEM encoded client certificate the module authenticates with while it creates the in-cluster service accounts. Leave unset when the caller supplies its own provider configuration." +} + +variable "client_key" { + type = string + nullable = true + default = null + sensitive = true + description = "PEM encoded client key that belongs to `client_certificate`. Leave unset when the caller supplies its own provider configuration." +} + +variable "owning_workspace_identifier" { + type = string + nullable = false + description = "Identifier of the meshStack workspace that owns the platform and its landing zones." +} + +variable "location_identifier" { + type = string + nullable = false + description = "Identifier of the meshStack location the platform is registered in." +} + +variable "platform_name" { + type = string + nullable = false + default = "ske-namespace" + description = "meshStack platform identifier. The landing zones derive their names from it, for example `ske-namespace-dev`." +} + +variable "platform_display_name" { + type = string + nullable = false + default = "Kubernetes namespace on SKE" + description = "Name of the platform as users see it in meshPanel." +} + +variable "platform_description" { + type = string + nullable = false + default = "Provides a kubernetes namespace on STACKIT Kubernetes Engine (SKE)." + description = "Description of the platform as users see it in meshPanel." +} + +variable "documentation_url" { + type = string + nullable = false + default = "" + description = "Link to the platform documentation shown in meshPanel." +} + +variable "support_url" { + type = string + nullable = false + default = "" + description = "Link to the support channel shown in meshPanel." +} + +variable "namespace_name_pattern" { + type = string + nullable = false + default = "#{workspaceIdentifier}-#{projectIdentifier}" + description = "Pattern meshStack uses to name the namespace it creates for a tenant." +} + +variable "disable_ssl_validation" { + type = bool + nullable = false + default = true + description = "Skip SSL validation when meshStack calls the Kubernetes API server. SKE clusters serve a certificate that meshStack does not trust by default, which is why this is on." +} + +variable "service_account_namespace" { + type = string + nullable = false + default = "meshcloud" + description = "Namespace that holds the replicator and metering service accounts." +} + +variable "resource_name_suffix" { + type = string + nullable = false + default = "" + description = "Suffix appended to the in-cluster resource names. Set it when one cluster carries more than one meshStack platform registration, so the service accounts and cluster roles do not collide." +} + +variable "metering_enabled" { + type = bool + nullable = false + default = true + description = "Create the metering service account and register metering on the platform. Turn this off when meshStack should not collect usage data from the cluster." +} + +variable "metering_processing" { + type = object({ + compact_timelines_after_days = optional(number, 30) + delete_raw_data_after_days = optional(number, 65) + }) + nullable = false + default = {} + description = "How long meshMetering keeps timelines and raw data. Only used when `metering_enabled` is true." +} + +variable "replicator_additional_rules" { + type = list(object({ + api_groups = list(string) + resources = list(string) + verbs = list(string) + resource_names = optional(list(string)) + non_resource_urls = optional(list(string)) + })) + nullable = false + default = [] + description = "Extra RBAC rules added to the replicator cluster role." +} + +variable "metering_additional_rules" { + type = list(object({ + api_groups = list(string) + resources = list(string) + verbs = list(string) + resource_names = optional(list(string)) + non_resource_urls = optional(list(string)) + })) + nullable = false + default = [] + description = "Extra RBAC rules added to the metering cluster role." +} + +# Cluster sizing behind the defaults: 2 vCPU + 8 Gi RAM per node, 1 node default and 3 nodes max. +# Expected density is 20-30 namespaces across the cluster. CPU is given in millicores (m) and +# memory in mebibytes (Mi) so every value stays a whole integer. +variable "quota_definitions" { + type = list(object({ + quota_key = string + label = string + description = string + unit = string + min_value = number + max_value = number + auto_approval_threshold = number + })) + nullable = false + description = "Quota keys a tenant can request on this platform, with the upper bound and the threshold below which meshStack approves a request automatically." + + default = [ + { + quota_key = "limits.cpu" + label = "CPU limit" + description = "The sum of CPU limits across all pods in a non-terminal state cannot exceed this value." + unit = "m" + min_value = 0 + max_value = 1000 # 1 vCPU per namespace + auto_approval_threshold = 1000 + }, + { + quota_key = "requests.cpu" + label = "CPU requests" + description = "The sum of CPU requests across all pods in a non-terminal state cannot exceed this value." + unit = "m" + min_value = 0 + max_value = 1000 + auto_approval_threshold = 500 + }, + { + quota_key = "limits.memory" + label = "Memory limit" + description = "The sum of memory limits across all pods in a non-terminal state cannot exceed this value." + unit = "Mi" + min_value = 0 + max_value = 1024 # 1 Gi per namespace + auto_approval_threshold = 1024 + }, + { + quota_key = "requests.memory" + label = "Memory requests" + description = "The sum of memory requests across all pods in a non-terminal state cannot exceed this value." + unit = "Mi" + min_value = 0 + max_value = 1024 + auto_approval_threshold = 512 + }, + { + quota_key = "requests.storage" + label = "Total Storage Requests" + description = "Across all persistent volume claims, the sum of storage requests cannot exceed this value." + unit = "Gi" + min_value = 0 + max_value = 5 + auto_approval_threshold = 2 + }, + { + quota_key = "persistentvolumeclaims" + label = "Persistent Volume Claims" + description = "The total number of PersistentVolumeClaims that can exist in the namespace." + unit = "" + min_value = 0 + max_value = 4 + auto_approval_threshold = 2 + }, + ] +} + +variable "landing_zones" { + type = map(object({ + display_name = string + description = string + info_link = optional(string, "") + tags = optional(map(list(string)), {}) + quotas = optional(list(object({ key = string, value = number })), []) + })) + nullable = false + description = "Landing zones to create for the platform, keyed by environment. The key becomes the suffix of the landing zone identifier, for example `dev` gives `ske-namespace-dev`." + + default = { + dev = { + display_name = "SKE Kubernetes Namespace – Development" + description = "Landing zone for development workloads." + tags = { + "LandingZoneFamily" = ["cloud-native"] + "environment" = ["dev"] + "confidentiality" = ["internal"] + } + quotas = [ + { key = "limits.cpu", value = 500 }, + { key = "requests.cpu", value = 250 }, + { key = "limits.memory", value = 512 }, + { key = "requests.memory", value = 256 }, + { key = "requests.storage", value = 1 }, + { key = "persistentvolumeclaims", value = 2 }, + ] + } + prod = { + display_name = "SKE Kubernetes Namespace – Production" + description = "Landing zone for production workloads." + tags = { + "LandingZoneFamily" = ["cloud-native"] + "environment" = ["prod"] + "confidentiality" = ["public"] + } + quotas = [ + { key = "limits.cpu", value = 1000 }, + { key = "requests.cpu", value = 500 }, + { key = "limits.memory", value = 1024 }, + { key = "requests.memory", value = 512 }, + { key = "requests.storage", value = 2 }, + { key = "persistentvolumeclaims", value = 4 }, + ] + } + } +} diff --git a/modules/kubernetes/platform/buildingblock/versions.tf b/modules/kubernetes/platform/buildingblock/versions.tf new file mode 100644 index 00000000..a2689284 --- /dev/null +++ b/modules/kubernetes/platform/buildingblock/versions.tf @@ -0,0 +1,15 @@ +terraform { + required_version = ">= 1.12.0" + + required_providers { + kubernetes = { + source = "hashicorp/kubernetes" + version = ">= 2.38.0" + } + + meshstack = { + source = "meshcloud/meshstack" + version = ">= 0.20.0" + } + } +} diff --git a/modules/ske/ske-starterkit/e2e/main.tf b/modules/ske/ske-starterkit/e2e/main.tf index 4e730294..a1d84b87 100644 --- a/modules/ske/ske-starterkit/e2e/main.tf +++ b/modules/ske/ske-starterkit/e2e/main.tf @@ -60,12 +60,54 @@ resource "random_string" "suffix" { numeric = false } +# The platform registration used to be a local submodule that hand-rolled the service account, the +# secret and the cluster role. It is the hub module now, so the smoke test exercises the same code +# the foundations run. The module configures its own `kubernetes` provider, which is why the four +# credentials are passed as inputs rather than through the provider block in `provider.tf`. module "meshstack_kubernetes_platform" { - source = "./meshstack_kubernetes_platform" + source = "../../../kubernetes/platform/buildingblock" - kube_host = local.ske_kubeconfig["clusters"][0]["cluster"]["server"] - workspace = var.test_context.workspace - test_suffix = random_string.suffix.result + kube_host = local.ske_kubeconfig["clusters"][0]["cluster"]["server"] + cluster_ca_certificate = base64decode(local.ske_kubeconfig["clusters"][0]["cluster"]["certificate-authority-data"]) + client_certificate = base64decode(local.ske_kubeconfig["users"][0]["user"]["client-certificate-data"]) + client_key = base64decode(local.ske_kubeconfig["users"][0]["user"]["client-key-data"]) + + owning_workspace_identifier = var.test_context.workspace + location_identifier = "global" + + platform_name = "smoke-test-ske-platform-${random_string.suffix.result}" + platform_display_name = "Smoke Test ${random_string.suffix.result}" + platform_description = "Platform for Smoke Test ${random_string.suffix.result}" + documentation_url = "https://kubernetes.io" + + # Every run gets a namespace and a name suffix of its own, so two runs never fight over the same + # service account or cluster role. + service_account_namespace = "smoke-test-${random_string.suffix.result}" + resource_name_suffix = random_string.suffix.result + + # The starterkit building block requests no quotas, so the platform offers none. + quota_definitions = [] + + landing_zones = { + dev = { + display_name = "Smoke Test Landing Zone ${random_string.suffix.result}" + description = "Landing Zone for Smoke Test ${random_string.suffix.result}" + info_link = "https://dontcare.com" + tags = { + "confidentiality" = ["Internal"] + "environment" = ["dev"] + } + } + prod = { + display_name = "Smoke Test Landing Zone ${random_string.suffix.result}" + description = "Landing Zone for Smoke Test ${random_string.suffix.result}" + info_link = "https://dontcare.com" + tags = { + "confidentiality" = ["Internal"] + "environment" = ["prod"] + } + } + } } module "stackit_git_repository" { diff --git a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/main.tf b/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/main.tf deleted file mode 100644 index cdfa9018..00000000 --- a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/main.tf +++ /dev/null @@ -1,235 +0,0 @@ -# namespace for replication service account -resource "kubernetes_namespace" "meshcloud" { - metadata { - name = "smoke-test-${var.test_suffix}" - } -} - -# meshfed_service service account -resource "kubernetes_service_account" "meshfed_service" { - metadata { - name = "meshfed-service-${var.test_suffix}" - namespace = kubernetes_namespace.meshcloud.metadata[0].name - annotations = { - "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" - } - } -} - -# meshfed_service secret -resource "kubernetes_secret" "meshfed_service_secret" { - metadata { - name = "meshfed-service-${var.test_suffix}" - namespace = kubernetes_namespace.meshcloud.metadata[0].name - annotations = { - "kubernetes.io/service-account.name" = kubernetes_service_account.meshfed_service.metadata[0].name - } - } - - type = "kubernetes.io/service-account-token" -} - -resource "kubernetes_cluster_role" "meshfed-service" { - - metadata { - name = "meshfed-service-${var.test_suffix}" - annotations = { - "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" - } - } - - rule { - api_groups = [""] - resources = ["namespaces"] - verbs = ["get", "list", "watch", "create", "delete", "update"] - } - rule { - api_groups = [""] - resources = ["resourcequotas", "resourcequotas/status"] - verbs = ["get", "list", "watch", "create", "delete", "deletecollection", "patch", "update"] - } - rule { - api_groups = [""] - resources = ["appliedclusterresourcequotas", "clusterresourcequotas", "clusterresourcequotas/status"] - verbs = ["get", "list", "watch", "create", "delete", "deletecollection", "patch", "update"] - } - rule { - api_groups = ["", "rbac.authorization.k8s.io"] - resources = ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] - verbs = ["get", "list", "watch"] - } - rule { - api_groups = ["", "rbac.authorization.k8s.io"] - resources = ["rolebindings"] - verbs = ["create", "delete", "update"] - } - rule { - api_groups = ["", "rbac.authorization.k8s.io"] - resources = ["clusterroles"] - verbs = ["bind"] - resource_names = ["admin", "edit", "view"] - } -} - -# meshfed_service role binding (unique per instance) -resource "kubernetes_cluster_role_binding" "meshfed-service" { - subject { - kind = "ServiceAccount" - name = kubernetes_service_account.meshfed_service.metadata[0].name - namespace = kubernetes_namespace.meshcloud.metadata[0].name - } - role_ref { - api_group = "rbac.authorization.k8s.io" - kind = "ClusterRole" - name = kubernetes_cluster_role.meshfed-service.metadata[0].name - } - metadata { - name = "meshfed-service-${var.test_suffix}" - annotations = { - "io.meshcloud/meshstack.replicator-kubernetes.version" = "1.0" - } - } -} - -# meshPlatform - -resource "meshstack_platform" "this" { - metadata = { - name = "smoke-test-ske-platform-${var.test_suffix}" - owned_by_workspace = var.workspace - } - - spec = { - display_name = "Smoke Test ${var.test_suffix}" - description = "Platform for Smoke Test ${var.test_suffix}" - endpoint = var.kube_host - documentation_url = "https://kubernetes.io" - - location_ref = { name = "global" } - - availability = { - restriction = "PUBLIC" - publication_state = "PUBLISHED" - restricted_to_workspaces = [] - } - - quota_definitions = [] - - config = { - kubernetes = { - base_url = var.kube_host - disable_ssl_validation = true - - replication = { - client_config = { - access_token = { - secret_value = kubernetes_secret.meshfed_service_secret.data["token"] - } - } - - namespace_name_pattern = "#{workspaceIdentifier}-#{projectIdentifier}" - } - - metering = { - client_config = { - access_token = { - secret_value = "dont-care" - } - } - - processing = { - enabled = false - } - } - } - } - - contributing_workspaces = [] - } - - lifecycle { - ignore_changes = [spec.availability] - } -} - -# dev meshLandingZone - -resource "meshstack_landingzone" "dev" { - metadata = { - name = "smoketest-ske-dev-${var.test_suffix}" - owned_by_workspace = var.workspace - tags = { - "confidentiality" = ["Internal"], - "environment" = ["dev"], - } - } - spec = { - display_name = "Smoke Test Landing Zone ${var.test_suffix}" - description = "Landing Zone for Smoke Test ${var.test_suffix}" - automate_deletion_approval = true - automate_deletion_replication = true - info_link = "https://dontcare.com" - platform_ref = { - uuid = meshstack_platform.this.metadata.uuid - } - platform_properties = { - kubernetes = { - kubernetes_role_mappings = [ - { - project_role_ref = { name = "admin" } - platform_roles = ["admin"] - }, - { - project_role_ref = { name = "user" } - platform_roles = ["edit"] - }, - { - project_role_ref = { name = "reader" } - platform_roles = ["view"] - }, - ] - } - } - } -} - -# prod meshLandingZone - -resource "meshstack_landingzone" "prod" { - metadata = { - name = "smoketest-ske-prod-${var.test_suffix}" - owned_by_workspace = var.workspace - tags = { - "confidentiality" = ["Internal"], - "environment" = ["prod"], - } - } - spec = { - display_name = "Smoke Test Landing Zone ${var.test_suffix}" - description = "Landing Zone for Smoke Test ${var.test_suffix}" - automate_deletion_approval = true - automate_deletion_replication = true - info_link = "https://dontcare.com" - platform_ref = { - uuid = meshstack_platform.this.metadata.uuid - } - platform_properties = { - kubernetes = { - kubernetes_role_mappings = [ - { - project_role_ref = { name = "admin" } - platform_roles = ["admin"] - }, - { - project_role_ref = { name = "user" } - platform_roles = ["edit"] - }, - { - project_role_ref = { name = "reader" } - platform_roles = ["view"] - }, - ] - } - } - } -} diff --git a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/outputs.tf b/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/outputs.tf deleted file mode 100644 index d6e2e8a3..00000000 --- a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/outputs.tf +++ /dev/null @@ -1,29 +0,0 @@ -output "token_replicator" { - sensitive = true - value = kubernetes_secret.meshfed_service_secret.data["token"] -} - -output "full_platform_identifier" { - value = "${meshstack_platform.this.metadata.name}.${meshstack_platform.this.spec.location_ref.name}" -} - -output "platform_ref" { - value = { - uuid = meshstack_platform.this.metadata.uuid - kind = "meshPlatform" - } -} - -output "landing_zone_identifiers" { - value = { - dev = meshstack_landingzone.dev.metadata.name - prod = meshstack_landingzone.prod.metadata.name - } -} - -output "landing_zone_refs" { - value = { - dev = meshstack_landingzone.dev.ref - prod = meshstack_landingzone.prod.ref - } -} \ No newline at end of file diff --git a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/terraform.tf b/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/terraform.tf deleted file mode 100644 index 6b9b271a..00000000 --- a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/terraform.tf +++ /dev/null @@ -1,15 +0,0 @@ -terraform { - required_version = ">= 1.0" - - required_providers { - meshstack = { - source = "meshcloud/meshstack" - } - kubernetes = { - source = "hashicorp/kubernetes" - } - random = { - source = "hashicorp/random" - } - } -} diff --git a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/variables.tf b/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/variables.tf deleted file mode 100644 index bf80dc09..00000000 --- a/modules/ske/ske-starterkit/e2e/meshstack_kubernetes_platform/variables.tf +++ /dev/null @@ -1,15 +0,0 @@ -variable "kube_host" { - type = string - nullable = false - description = "The Kubernetes API server URL." -} - -variable "workspace" { - type = string - nullable = false - description = "The meshStack workspace identifier that will own the platform and landing zones." -} - -variable "test_suffix" { - type = string -} diff --git a/modules/ske/ske-starterkit/e2e/provider.tf b/modules/ske/ske-starterkit/e2e/provider.tf deleted file mode 100644 index 42541712..00000000 --- a/modules/ske/ske-starterkit/e2e/provider.tf +++ /dev/null @@ -1,6 +0,0 @@ -provider "kubernetes" { - host = yamldecode(var.ske_kubeconfig)["clusters"][0]["cluster"]["server"] - cluster_ca_certificate = base64decode(yamldecode(var.ske_kubeconfig)["clusters"][0]["cluster"]["certificate-authority-data"]) - client_certificate = base64decode(yamldecode(var.ske_kubeconfig)["users"][0]["user"]["client-certificate-data"]) - client_key = base64decode(yamldecode(var.ske_kubeconfig)["users"][0]["user"]["client-key-data"]) -} \ No newline at end of file diff --git a/modules/stackit/dns/backplane/README.md b/modules/stackit/dns/backplane/README.md new file mode 100644 index 00000000..860c30dd --- /dev/null +++ b/modules/stackit/dns/backplane/README.md @@ -0,0 +1,201 @@ +# STACKIT DNS – Backplane + +This module sets up the shared backplane configuration for the STACKIT DNS building block. It +creates a dedicated service account with a Workload Identity Federation (WIF) identity provider and +grants it the roles the building block needs: + +- **`dns.admin`**, on the projects named in `zone_project_ids` β€” create and delete DNS zones and + record sets in each of them. Or, when no project can be named, on the folder in `folder_id`, + which covers every project below it. +- **`iam.member-admin`**, on the organization β€” assign the `dns.admin` role to the DNS service + account the building block creates in the zone's project. + +Authentication uses WIF (OIDC token exchange) β€” no long-lived service account key is created or +stored for the building block identity itself. + +This backplane exists for the meshStack-ordered path, where meshStack federates a token into the run +and `../buildingblock` authenticates with it. A composition that brings its own STACKIT credentials +does not deploy this module at all: it configures the provider itself and sources +`../buildingblock/zone`, the submodule that declares none. Its own credential then needs the same +`dns.admin` reach described below, plus whatever role it uses to create service accounts. +`reference-architectures/stackit-landingzone` takes that route. + +## How one identity reaches one or two STACKIT projects + +On the usual path the building block writes into a single project: the zone, its records and the +DNS service account all live there. On the delegation path, which only works for a customer-owned +domain, it also writes the NS record into the platform team's own project, which owns the parent +zone. + +It reaches both with a single identity and a single provider configuration, because STACKIT +credentials are not bound to a project. A service account belongs to the project it was created in, +but every resource carries its own `project_id`, and access is decided by the role assignments the +account holds on the target resource. So there is no second provider, no second credential, and no +credential handed from one project to the other. + +What makes it work is the scope of the role assignment. + +## Project scope where the projects are known, folder scope where they are not + +Scope follows what the caller can name when the backplane is deployed. + +**`zone_project_ids` is the default path.** Name the projects that own the zones and the service +account is granted `dns.admin` on exactly those. Two cases make the projects knowable: + +- A composition fixes the zone's project as a static input. `reference-architectures/stackit-kubernetes` + does this β€” the platform team owns one zone in its own project and fills that project in when it + registers the building block definition. +- The delegation path. The parent zone's project is `delegation.parent_zone_project_id` on the + building block, so it is named there too. + +**`folder_id` is the fallback, for one case only.** The `TENANT_LEVEL` building block definition in +`../meshstack_integration.tf` takes its `project_id` from `PLATFORM_TENANT_ID`, so the zone lands in +whichever tenant project places the order, and the platform team registers the definition long +before it knows which those are. A folder covers every project below it, so it keeps the property +that makes a wide scope attractive: grant once, and every project a tenant later receives is +covered. **Every project involved then has to live under that folder** β€” a parent zone kept outside +it needs its project listed in `zone_project_ids` as well. + +Set at least one of the two. Setting both is fine and is what a mixed deployment needs. + +## Why not organization scope, in either case + +The sibling backplanes grant their role at organization scope, and `modules/stackit/network/backplane` +is right to do so, but that does not generalise. STACKIT's authorization API offers a different set +of roles per resource type, and **no dns role appears on an organization**. Assigning `dns.admin` +there fails. These three calls establish it, each returning `HTTP/1.1 200 OK`: + +```sh +stackit curl https://authorization.api.stackit.cloud/v2/organization//roles +# resourceType=organization, 76 roles, no dns role at all + +stackit curl https://authorization.api.stackit.cloud/v2/folder//roles +# resourceType=folder, 184 roles, including dns.admin and dns.reader + +stackit curl https://authorization.api.stackit.cloud/v2/project//roles +# resourceType=project, 182 roles, including dns.admin and dns.reader +``` + +The organization response is not filtered down to organization-management roles, so the absence is a +real one rather than an artifact of the call: `iaas.network.admin`, `organization.admin`, +`iam.member-admin` and `resource-manager.admin` are all present at organization scope. + +`iam.member-admin` is one of the 76, which is why this module still assigns that one on the +organization. When tenant projects are spread over several folders and none of them can be named, +deploy one backplane instance per folder and override `service_account_name`. + +## The one permission this module does not grant + +The building block also creates a service account and a service account key, which cert-manager and +ExternalDNS authenticate with. Creating a service account is an IAM operation, and this module does +not grant it, because the right role depends on how far you want the identity to reach. + +The role list above carries `iam.service-account-creator`, `iam.service-account-key-admin` and +`iam.service-account-admin`, and all three exist at organization, folder and project scope. Pick the +one your organization uses and pass it through `additional_organization_roles`. List the roles a +resource offers with the calls above. + +Two ways out if you would rather not widen the identity: + +- Set `dns_service_account_enabled = false` on the building block. It then creates the zone and its + records only, and the platform team supplies the DNS key by hand. +- Define a custom role that carries the service account permissions and nothing else, then list it + in `additional_organization_roles`. + +## What the DNS key can reach + +`dns.admin` is a project role. The key the building block hands to cert-manager and ExternalDNS can +therefore write every record in every zone of the project the zone lives in. Under a free STACKIT +subdomain that is unavoidable, because such a domain admits exactly one zone at exactly one label +and every cluster below it shares that zone. Give tenants a domain you own if you need a real +boundary between them. + +## Provider requirements + +`stackit_authorization_project_role_assignment`, which the building block uses, sits behind the +STACKIT provider's `iam` experiment. The building block's own `provider.tf` sets it. The root +configuration that applies **this** backplane has to set it as well, because a backplane module +carries no provider block: + +```hcl +provider "stackit" { + default_region = "eu01" + experiments = ["iam"] +} +``` + +## Prerequisites + +- A STACKIT project where the service account will be created. +- A STACKIT service account with permissions to manage service accounts, organization-level role + assignments, and project-level or folder-level role assignments depending on which scope you use. +- Either the STACKIT projects that own the zones, or the folder ID under which every project + involved lives. Projects placed directly under the organization are not covered by a folder. +- The STACKIT organization ID. +- meshStack WIF issuer and subject from `data.meshstack_integrations.integrations`. + +## Usage + +```hcl +module "dns_backplane" { + source = "./backplane" + + project_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" + organization_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" + + # Name the zones' projects where you can. Fall back to folder_id only when the zone lands in + # whichever tenant project places the order. + zone_project_ids = ["xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"] + folder_id = null + + # The role your organization uses to create service accounts in tenant projects. + additional_organization_roles = [] + + workload_identity_federation = { + issuer = data.meshstack_integrations.integrations.workload_identity_federation.replicator.issuer + subjects = [""] + } +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.11.0 | +| [stackit](#requirement\_stackit) | >= 0.110.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [stackit_authorization_folder_role_assignment.dns_admin](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/authorization_folder_role_assignment) | resource | +| [stackit_authorization_organization_role_assignment.additional](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/authorization_organization_role_assignment) | resource | +| [stackit_authorization_organization_role_assignment.member_admin](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/authorization_organization_role_assignment) | resource | +| [stackit_authorization_project_role_assignment.dns_admin](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/authorization_project_role_assignment) | resource | +| [stackit_service_account.building_block](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/service_account) | resource | +| [stackit_service_account_federated_identity_provider.building_block](https://registry.terraform.io/providers/stackitcloud/stackit/latest/docs/resources/service_account_federated_identity_provider) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [additional\_organization\_roles](#input\_additional\_organization\_roles) | Extra STACKIT roles granted to the service account at organization scope. Use this for the role your organization uses to create service accounts and service account keys in tenant projects, which the building block needs for the DNS credential. See backplane/README.md. | `list(string)` | `[]` | no | +| [folder\_id](#input\_folder\_id) | STACKIT folder ID under which the zones' projects live. The service account is granted `dns.admin`
on this folder, which covers every project below it.

This is the fallback for the one case where no project can be named: the `TENANT_LEVEL` building
block definition takes its `project_id` from `PLATFORM_TENANT_ID`, so the zone lands in whichever
tenant project places the order. Prefer `zone_project_ids` whenever the projects are known.
STACKIT offers no dns role at organization scope, so a folder is the widest scope available for it. | `string` | `null` | no | +| [organization\_id](#input\_organization\_id) | STACKIT organization ID the folder lives under. The service account is granted 'iam.member-admin' here, which lets the building block assign 'dns.admin' to the DNS service account it creates. | `string` | n/a | yes | +| [project\_id](#input\_project\_id) | STACKIT project ID where the service account will be created. | `string` | n/a | yes | +| [service\_account\_name](#input\_service\_account\_name) | Name of the service account created in the STACKIT project. Override when deploying multiple backplane instances in the same project. | `string` | `"mesh-dns"` | no | +| [workload\_identity\_federation](#input\_workload\_identity\_federation) | WIF issuer URL and subject list for the meshStack building block identity provider. | 
object({
    issuer   = string
    subjects = list(string)
  })
| n/a | yes | +| [zone\_project\_ids](#input\_zone\_project\_ids) | STACKIT projects that own the zones the building block writes into. The service account is granted
`dns.admin` on each of them, at **project** scope.

Use this wherever the projects are knowable when the backplane is deployed β€” a composition that
fixes the zone project as a static input, or the delegation path, where the parent zone's project
is named by `delegation.parent_zone_project_id`. Fall back to `folder_id` only when they are not. | `set(string)` | `[]` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [service\_account\_email](#output\_service\_account\_email) | Email of the STACKIT service account used by the buildingblock provider via WIF. | + \ No newline at end of file diff --git a/modules/stackit/dns/backplane/main.tf b/modules/stackit/dns/backplane/main.tf new file mode 100644 index 00000000..2d9e07c3 --- /dev/null +++ b/modules/stackit/dns/backplane/main.tf @@ -0,0 +1,110 @@ +# ───────────────────────────────────────────────────────────────────────────── +# One identity, one or two STACKIT projects +# +# On the usual path the building block writes into one project: it creates the zone, its records +# and the DNS service account there. On the delegation path, which only works for a customer-owned +# domain, it also writes the NS record into the platform team's own project, which owns the parent +# zone. +# +# It reaches both with a single identity and a single provider configuration, because STACKIT +# credentials are not bound to a project. A service account belongs to the project it was created +# in, but every resource carries its own `project_id` and access is decided by the role assignments +# the account holds on the target resource. +# +# So the backplane creates one service account and grants it the roles below on the projects it has +# to reach, or above them when it cannot name them. +# +# `iam.member-admin` is granted on the organization, because that is where STACKIT offers it. +# `dns.admin` is not among the 76 roles an organization offers, so it goes on projects or on a +# folder. See backplane/README.md for the calls that establish this. +# ───────────────────────────────────────────────────────────────────────────── + +resource "stackit_service_account" "building_block" { + project_id = var.project_id + name = var.service_account_name +} + +resource "stackit_service_account_federated_identity_provider" "building_block" { + for_each = { for i, s in var.workload_identity_federation.subjects : tostring(i) => s } + + project_id = var.project_id + service_account_email = stackit_service_account.building_block.email + name = "meshstack-${each.key}" + issuer = var.workload_identity_federation.issuer + + assertions = [ + { + item = "aud" + operator = "equals" + value = "api://AzureADTokenExchange" + }, + { + item = "sub" + operator = "equals" + value = each.value + } + ] +} + +# dns.admin allows creating and deleting zones and record sets. The building block needs it in the +# project the zone lives in, and on the delegation path also in the parent zone's project. +# +# Grant it on the projects, whenever the caller can name them. Scope follows what is knowable at +# grant time, and a project-scoped grant keeps the identity off every project that is not involved. +# The delegation path's parent zone project is always knowable β€” it is `delegation.parent_zone_project_id` +# on the building block β€” and so is the zone project of any composition that fixes it statically. +resource "stackit_authorization_project_role_assignment" "dns_admin" { + for_each = var.zone_project_ids + + resource_id = each.value + role = "dns.admin" + subject = stackit_service_account.building_block.email +} + +# The folder-scoped grant is the fallback for the one case where no project can be named: the +# `TENANT_LEVEL` building block definition in ../meshstack_integration.tf takes its `project_id` +# from `PLATFORM_TENANT_ID`, so the zone lands in whichever tenant project places the order, and the +# platform team registers the definition long before it knows which those are. A folder covers every +# project below it. +# +# Organization scope is not an option either way: STACKIT's authorization API offers a different set +# of roles per resource type, and no dns role appears on an organization. See backplane/README.md +# for the calls that establish this. +resource "stackit_authorization_folder_role_assignment" "dns_admin" { + count = var.folder_id == null ? 0 : 1 + + resource_id = var.folder_id + role = "dns.admin" + subject = stackit_service_account.building_block.email +} + +# The folder grant used to be unconditional, so an existing state carries it at the unindexed +# address. +moved { + from = stackit_authorization_folder_role_assignment.dns_admin + to = stackit_authorization_folder_role_assignment.dns_admin[0] +} + +# iam.member-admin allows assigning roles. The building block uses it to give the DNS service +# account it creates the `dns.admin` role on the zone's project and nothing beyond it. +# modules/stackit/project/backplane grants the same role for the same purpose. +resource "stackit_authorization_organization_role_assignment" "member_admin" { + resource_id = var.organization_id + role = "iam.member-admin" + subject = stackit_service_account.building_block.email +} + +# ── Creating service accounts in tenant projects ───────────────────────────── +# +# The building block also creates a service account and a service account key, which cert-manager +# and ExternalDNS authenticate with. STACKIT's predefined role for creating service accounts could +# not be established from public documentation, so this module does not name one β€” see +# backplane/README.md. Put whatever role your organization uses into +# `additional_organization_roles` rather than widening the two roles above. +resource "stackit_authorization_organization_role_assignment" "additional" { + for_each = toset(var.additional_organization_roles) + + resource_id = var.organization_id + role = each.value + subject = stackit_service_account.building_block.email +} diff --git a/modules/stackit/dns/backplane/outputs.tf b/modules/stackit/dns/backplane/outputs.tf new file mode 100644 index 00000000..3fde57dd --- /dev/null +++ b/modules/stackit/dns/backplane/outputs.tf @@ -0,0 +1,4 @@ +output "service_account_email" { + value = stackit_service_account.building_block.email + description = "Email of the STACKIT service account used by the buildingblock provider via WIF." +} diff --git a/modules/stackit/dns/backplane/variables.tf b/modules/stackit/dns/backplane/variables.tf new file mode 100644 index 00000000..d61564b3 --- /dev/null +++ b/modules/stackit/dns/backplane/variables.tf @@ -0,0 +1,70 @@ +variable "project_id" { + type = string + nullable = false + description = "STACKIT project ID where the service account will be created." +} + +variable "zone_project_ids" { + type = set(string) + nullable = false + default = [] + + description = <<-EOT + STACKIT projects that own the zones the building block writes into. The service account is granted + `dns.admin` on each of them, at **project** scope. + + Use this wherever the projects are knowable when the backplane is deployed β€” a composition that + fixes the zone project as a static input, or the delegation path, where the parent zone's project + is named by `delegation.parent_zone_project_id`. Fall back to `folder_id` only when they are not. + EOT +} + +variable "folder_id" { + type = string + nullable = true + default = null + + description = <<-EOT + STACKIT folder ID under which the zones' projects live. The service account is granted `dns.admin` + on this folder, which covers every project below it. + + This is the fallback for the one case where no project can be named: the `TENANT_LEVEL` building + block definition takes its `project_id` from `PLATFORM_TENANT_ID`, so the zone lands in whichever + tenant project places the order. Prefer `zone_project_ids` whenever the projects are known. + STACKIT offers no dns role at organization scope, so a folder is the widest scope available for it. + EOT + + validation { + condition = var.folder_id != null || length(var.zone_project_ids) > 0 + error_message = "The service account needs dns.admin somewhere. Name the zones' projects in zone_project_ids, or set folder_id when the target project is unknowable at grant time." + } +} + +variable "organization_id" { + type = string + nullable = false + description = "STACKIT organization ID the folder lives under. The service account is granted 'iam.member-admin' here, which lets the building block assign 'dns.admin' to the DNS service account it creates." +} + +variable "workload_identity_federation" { + type = object({ + issuer = string + subjects = list(string) + }) + nullable = false + description = "WIF issuer URL and subject list for the meshStack building block identity provider." +} + +variable "service_account_name" { + type = string + default = "mesh-dns" + nullable = false + description = "Name of the service account created in the STACKIT project. Override when deploying multiple backplane instances in the same project." +} + +variable "additional_organization_roles" { + type = list(string) + default = [] + nullable = false + description = "Extra STACKIT roles granted to the service account at organization scope. Use this for the role your organization uses to create service accounts and service account keys in tenant projects, which the building block needs for the DNS credential. See backplane/README.md." +} diff --git a/modules/stackit/dns/backplane/versions.tf b/modules/stackit/dns/backplane/versions.tf new file mode 100644 index 00000000..5c2057d3 --- /dev/null +++ b/modules/stackit/dns/backplane/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.11.0" + + required_providers { + stackit = { + source = "stackitcloud/stackit" + version = ">= 0.110.0" + } + } +} diff --git a/modules/stackit/dns/buildingblock/README.md b/modules/stackit/dns/buildingblock/README.md new file mode 100644 index 00000000..c7f92393 --- /dev/null +++ b/modules/stackit/dns/buildingblock/README.md @@ -0,0 +1,330 @@ +--- +name: STACKIT DNS Zone +supportedPlatforms: + - stackit +description: Creates a STACKIT DNS zone with its record sets and a service account key that lets cert-manager and ExternalDNS manage records at runtime. +--- + +# STACKIT DNS Zone Building Block + +This building block module creates one DNS zone in a STACKIT project, the record sets inside it, and +a service account key that lets a workload manage those records at runtime. The cert-manager DNS-01 +solver and ExternalDNS both take that key. With `create_zone = false` the module skips the zone and +writes its record sets into a zone that already exists. + +The module is used in two ways. A reference architecture sources it directly as a Terraform module, +which keeps STACKIT resources out of the architecture itself. A team orders it as a building block +through meshStack, wired up by the `meshstack_integration.tf` at the module root. + +## Two tiers: this root, and `zone/` + +Every resource lives in `zone/`, a submodule that declares **no provider configuration**. This root +is that submodule plus a `provider "stackit"` with `use_oidc = true`, which is what meshStack runs +when a team orders the building block. + +Source **this root** when meshStack drives the run and federates a workload identity token into it. +Source **`zone/`** from a composition that brings its own credentials, and there is no way around +it: a provider a child module configures itself cannot be overridden by its caller, so a caller +holding a STACKIT service account key must call a module that declares none. The same applies to +`count`, `for_each` and `depends_on`, which Terraform refuses on a module with its own providers. + +```hcl +provider "stackit" { + service_account_key = var.stackit_service_account_key + experiments = ["iam"] # gates stackit_authorization_project_role_assignment +} + +module "dns_zone" { + count = var.dns_enabled ? 1 : 0 + source = "github.com/meshcloud/meshstack-hub//modules/stackit/dns/buildingblock/zone?ref=main" + + project_id = var.platform_project_id + zone_name = "likvid.stackit.run" +} +``` + +`reference-architectures/stackit-landingzone` does exactly this to create the one zone its ordered +clusters share. `zone/` takes every input this root does except `service_account_email` and +`stackit_region`, which only ever configured the provider, and returns the same outputs. + +## One zone, one project, one credential + +STACKIT DNS is project-scoped end to end. `stackit_dns_record_set` carries its own `project_id` and +that project must own the `zone_id`, and the SKE `extensions.dns` block has no field for a foreign +project or a foreign credential. The zone, its records and the credential that writes them +therefore all belong to one project. + +## A free STACKIT subdomain admits exactly one label + +This is measured against the live API, not assumed. Creating a two-label zone under `stackit.run` +is rejected before any delegation or project logic is reached: + +```console +$ stackit dns zone create --project-id

--name aipoc-c6a37f-sub \ + --dns-name sub.aipoc-c6a37f.stackit.run +Error: create DNS zone: 400 Bad Request, status code 400, Body: +{"message":"zone dns name sub.aipoc-c6a37f.stackit.run has one error", + "error":"subdomain 'sub.aipoc-c6a37f' should only have one level"} +``` + +The error is byte-for-byte identical with a correct NS delegation already in place in the parent +zone, and identical again in a freshly created second project. The project makes no difference. +`var.zone_name` carries a validation that rejects this shape at plan time rather than at apply. + +The same name under a customer-owned domain fails differently β€” *"collides with a parent zone in a +different project and has no delegation"* β€” so STACKIT's cross-project delegation does exist and is +project-aware. It simply never gets reached for `stackit.run`, because the name check fires first. + +So `likvid.stackit.run` is a zone, and everything below it is a record set in that zone. A cluster +reachable at `cluster1.likvid.stackit.run` with a wildcard below it needs two entries: + +```hcl +records = { + "cluster1" = { type = "A", records = ["203.0.113.17"] } + "*.cluster1" = { type = "A", records = ["203.0.113.17"] } +} +``` + +## One zone, many clusters + +A zone is not a subzone, and that is what makes the shared zone work. STACKIT allows a record set +with a deeper name inside an existing zone, so a single platform-owned zone carries every cluster. +The platform team creates the zone once: + +```hcl +module "zone" { + source = "github.com/meshcloud/meshstack-hub//modules/stackit/dns/buildingblock?ref=main" + + project_id = var.platform_project_id + zone_name = "likvid.stackit.run" +} +``` + +Each cluster then runs the module again with `create_zone = false` and writes only its own records +into that zone: + +```hcl +module "cluster_dns" { + source = "github.com/meshcloud/meshstack-hub//modules/stackit/dns/buildingblock?ref=main" + + project_id = var.platform_project_id # the project that owns the zone + zone_name = "likvid.stackit.run" + create_zone = false + dns_service_account_enabled = false + + wildcard = { + label = "cluster1" + address = module.ingress.haproxy_lb_ip + } +} +``` + +That gives the record set `*.cluster1.likvid.stackit.run`, and `wildcard_domain` returns +`cluster1.likvid.stackit.run` for the certificate. `zone_id` is optional here β€” the module looks the +zone up by its name with the `stackit_dns_zone` data source, which needs read access on the zone's +project. + +## Apex or label, and why it matters + +`wildcard.label` decides where the wildcard sits: + +| `label` | Record set | Hostnames | Certificate | +|---|---|---|---| +| unset | `*.likvid.stackit.run` | `app.likvid.stackit.run` | `*.likvid.stackit.run` | +| `cluster1` | `*.cluster1.likvid.stackit.run` | `app.cluster1.likvid.stackit.run` | `*.cluster1.likvid.stackit.run` | + +**Only one cluster can hold the wildcard at the zone apex.** The apex record is a single record set +in the zone, so the second cluster that writes it collides with the first one. The apex shape is +therefore the shape of a zone with exactly one cluster in it, which is what the SKE foundations have +today. Every cluster beyond the first needs a label, and the label is also what keeps a cluster's +certificate down to its own names instead of the whole domain. + +Moving a cluster that already runs at the apex to a label renames every hostname it serves, from +`app.likvid.stackit.run` to `app.cluster1.likvid.stackit.run`. That is a breaking change for the +applications on it, so plan it as one. + +The module cannot detect the collision for you. A second cluster runs from its own state, and +STACKIT offers no data source that lists the record sets of a zone, so the first sign of the clash +is the error the API returns at apply time. + +## Moving an existing SKE foundation onto this module + +The three SKE foundations create their zone and one apex wildcard by hand: + +```hcl +resource "stackit_dns_zone" "this" { + name = "likvid-ske-starterkit" + dns_name = "likvid.stackit.run" +} + +resource "stackit_dns_record_set" "A" { + name = "*.likvid.stackit.run" + type = "A" + records = [var.haproxy_lb_ip] + comment = "Wildcard app routing to HAProxy ingress load balancer" +} +``` + +The module reproduces that record with `wildcard = { address = var.haproxy_lb_ip }` and no label: +the same fully qualified name, the same type, no TTL, and the same comment, which is the default of +`wildcard.comment`. Keep the zone's name with `zone_display_name = "likvid-ske-starterkit"`, because +STACKIT stores that name separately from the DNS name. Two `moved` blocks then carry the existing +state into the module and the plan stays empty: + +```hcl +moved { + from = stackit_dns_zone.this + to = module.dns.module.zone.stackit_dns_zone.this[0] +} + +moved { + from = stackit_dns_record_set.A + to = module.dns.module.zone.stackit_dns_record_set.wildcard[0] +} +``` + +The extra `module.zone` in the target is the submodule described above. Drop it when you source +`zone/` directly instead of this root. Callers that were already on this root before the split need +no `moved` block of their own β€” main.tf carries one per resource. + +## The permission trade-off you are accepting + +With a free STACKIT subdomain, every cluster and every application under `likvid.stackit.run` lives +in one zone, in one project, written with one credential. The key this module returns therefore +**can write any record in that zone, including over a record that belongs to another cluster.** +`dns.admin` is a project role β€” it cannot be narrowed to one zone, let alone to one name. + +A per-cluster label does not change that. `*.cluster1.likvid.stackit.run` is where the code writes, +not where the credential ends. **The label is a boundary this module draws, and the permission +system does not enforce it.** A cluster that gets the key can write outside its label, and nothing +in STACKIT stops it. + +That is a real trade-off, not a detail: + +- Hand the key only to workloads you would trust with the whole domain. +- Give each cluster its own label and treat the labels as a convention the code keeps, not as a + boundary an attacker respects. +- If you need a real boundary, use a domain you own and give each tenant a delegated zone of its + own in its own project. See the next section. + +## Delegation β€” for customer-owned domains only + +`var.delegation` writes the `NS` record that delegates this zone from a parent zone in another +STACKIT project. It is unset by default. Under a free STACKIT suffix the zone itself cannot exist, +so the module refuses that combination outright. + +```hcl +zone_name = "cluster1.example.com" + +delegation = { + parent_zone_project_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # the platform team's project + parent_zone_name = "example.com" +} +``` + +Two traps the code guards against, both seen in the proof-of-concept: + +- **Trailing dots are load-bearing.** STACKIT relativises a record value that does not end in a dot + against the zone, so `ns1.stackit.cloud` is stored as `ns1.stackit.cloud..`. The default + nameservers are `ns1.stackit.cloud.` and `ns2.stackit.zone.`, and a precondition rejects values + without the dot. The same applies to `CNAME`, `MX` and `NS` values in `records`. +- **An orphaned NS record fails silently.** If the delegation is published and the zone behind it + never appears, a direct query returns NOERROR with an empty answer and a referral, and a + recursive resolver returns SERVFAIL β€” with no error at apply time. The module therefore creates + the zone before the record, and a precondition refuses to delegate a name it is not creating. + +## The DNS credential + +`stackit_service_account_key.dns.json` is the raw service account key JSON that the +[STACKIT cert-manager webhook](https://github.com/stackitcloud/stackit-cert-manager-webhook) expects +as `sa.json`. Pass it, together with `zone_project_id`, to `modules/kubernetes/ingress`: + +```hcl +dns01 = { + zone_name = module.dns.zone_name + certificate_domain = module.dns.wildcard_domain + stackit = { + project_id = module.dns.zone_project_id + service_account_key = module.dns.dns_service_account_key + } +} +``` + +The two names differ once the zone is shared. `zone_name` is the zone the DNS-01 solver is +authorised for, `likvid.stackit.run`, and `certificate_domain` is the domain the wildcard +certificate covers, `cluster1.likvid.stackit.run`. Leave `certificate_domain` unset and the +certificate covers the whole zone, which is what a single cluster at the apex wants. + +Set `dns_service_account_enabled = false` when records are managed through Terraform only, or when +the backplane identity may not create service accounts. Both `dns_service_account_*` outputs are +`null` in that case. + +The account's name is derived as `mesh-dns--<4 hex digits of the full +zone name>`, for example `mesh-dns-likvid-9a3c`. **STACKIT caps a service account name at 20 +characters**, which the zone name pasted in whole overruns for every free STACKIT subdomain β€” +`mesh-dns-likvid-stackit-run` is 27 and the provider rejects it at plan time with *"Attribute name +string length must be at most 20"*. Override the name with `dns_service_account_name` if you want a +different one; it is validated against the same cap. + +`../backplane` cannot supply this credential. It federates an OIDC identity and issues no static +key, and a static key is what a controller running inside a cluster needs: the workload identity +token is federated into the Terraform run, and nothing in the cluster holds one afterwards. + +`dns_service_account_key_ttl_days` is unset by default, so the key stays valid until it is deleted. +A key that expires has to be rotated by re-applying the building block before a certificate can +renew. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.11.0 | +| [stackit](#requirement\_stackit) | >= 0.110.0 | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [zone](#module\_zone) | ./zone | n/a | + +## Resources + +No resources. + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [contact\_email](#input\_contact\_email) | Contact address stored on the zone. Leave empty to let STACKIT pick its own default. Only used when `create_zone` is `true`. | `string` | `""` | no | +| [create\_zone](#input\_create\_zone) | Create the zone. Leave this at `true` when the caller owns the zone.

Set it to `false` to write record sets into a zone that already exists and that another Terraform
configuration owns. The platform team creates `likvid.stackit.run` once, and every cluster then
adds its own record sets to that zone. STACKIT allows a record set with a deeper name inside an
existing zone, so `*.cluster1.likvid.stackit.run` is a record set in `likvid.stackit.run` and not
a zone of its own. See README.md. | `bool` | `true` | no | +| [delegation](#input\_delegation) | Write the `NS` record that delegates this zone from a parent zone in another STACKIT project.
Leave unset, which is the default and the usual case.

**This works only for a domain the customer owns.** Under a free STACKIT suffix such as
`stackit.run` the zone itself cannot be created, so the delegation has nothing to point at β€” see
the API error quoted in zone/main.tf. The module refuses that combination.

`nameservers` must carry trailing dots. STACKIT relativises a value without one against the zone,
so `ns1.stackit.cloud` is stored as `ns1.stackit.cloud..` and the delegation points nowhere. | 

object({
    parent_zone_project_id = string
    parent_zone_name       = string
    nameservers            = optional(list(string), ["ns1.stackit.cloud.", "ns2.stackit.zone."])
    ttl                    = optional(number, 3600)
  })
| `null` | no | +| [dns\_service\_account\_enabled](#input\_dns\_service\_account\_enabled) | Create a service account with `dns.admin` on the zone's project and a key for it. cert-manager's DNS-01 solver and ExternalDNS both authenticate with that key. Turn it off when the consumer manages records through Terraform only, or when the backplane identity may not create service accounts. | `bool` | `true` | no | +| [dns\_service\_account\_key\_ttl\_days](#input\_dns\_service\_account\_key\_ttl\_days) | Validity of the DNS service account key in days. Leave unset to create a key that stays valid until it is deleted. A key that expires has to be rotated by re-applying the building block before a certificate can renew. | `number` | `null` | no | +| [dns\_service\_account\_name](#input\_dns\_service\_account\_name) | Name of the DNS service account. Defaults to `mesh-dns--<4 hex digits of
the zone name>`, for example `mesh-dns-likvid-9a3c`, which keeps two zones in the same project
apart. **STACKIT caps the name at 20 characters**, which is why the zone name is not pasted in
whole β€” see zone/main.tf for the provider error. | `string` | `null` | no | +| [project\_id](#input\_project\_id) | STACKIT project ID that owns the zone. The record sets and the DNS service account are created in the same project, because STACKIT DNS is project-scoped. | `string` | n/a | yes | +| [records](#input\_records) | Record sets to create in the zone, keyed by the name relative to the zone. A key of `cluster1` in
the zone `likvid.stackit.run` gives `cluster1.likvid.stackit.run`, and `*.cluster1` gives the
wildcard below it.

A value that is itself a domain name β€” the target of a `CNAME`, `MX` or `NS` record β€” must end in
a dot. STACKIT relativises a value without one against the zone, so `example.com` is stored as
`example.com.likvid.stackit.run.`.
hcl
records = {
  "cluster1"   = { type = "A", records = ["203.0.113.17"] }
  "*.cluster1" = { type = "A", records = ["203.0.113.17"] }
}
| 
map(object({
    type    = string
    records = list(string)
    ttl     = optional(number)
    comment = optional(string)
  }))
| `{}` | no | +| [service\_account\_email](#input\_service\_account\_email) | Email of the STACKIT service account the provider authenticates as via workload identity federation. Leave unset when the caller supplies its own provider configuration. | `string` | `null` | no | +| [stackit\_region](#input\_stackit\_region) | STACKIT region used as the provider's default. STACKIT DNS itself is global. Ignored when the caller supplies its own provider configuration. | `string` | `"eu01"` | no | +| [wildcard](#input\_wildcard) | One wildcard `A` record that sends every hostname below a domain to the same address, usually the
ingress controller's load balancer. Leave it unset to create no wildcard.

`label` decides where the wildcard sits. Leave it unset and the record is `*.`, which
covers every hostname directly under the zone. Set it to the cluster's name and the record is
`*.