Skip to content

Migrate to the _v1 kubernetes resource names before provider v4 removes the old ones #253

Description

@grubmeshi

What

The hub uses the deprecated non-suffixed hashicorp/kubernetes resource types everywhere — 197
occurrences across the modules. Plans now print warnings like Deprecated; use kubernetes_service_account_v1, first noticed in the step logs of an ske/forgejo-connector
building block run.

Counts by type:

67  kubernetes_secret
40  kubernetes_service_account
32  kubernetes_cluster_role_binding
26  kubernetes_cluster_role
23  kubernetes_role_binding
 5  kubernetes_namespace
 4  kubernetes_default_service_account

Why this is not just noise

The provider documents that the plain names are removed in v4.0.0
(https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/versioned-resources).
v3.0.0 already deprecated all seven types above. Our building block modules pin the provider with an
open upper bound — three modules say >= 2.35.1 and one says >= 2.38 — so v4 will resolve by
itself on the day it ships. Every deployed building block instance then breaks at plan with Invalid resource type, with no warning phase and no rollback for instances that are already out there.

The rename is not a rename

A moved block cannot do it. All of these types are SDKv2-backed in hashicorp/kubernetes, and
SDKv2 refuses the MoveResourceState RPC unconditionally (helper/schema/grpc_provider.go, "Move
Resource State Not Supported"), so a cross-type moved block fails at plan on every Terraform and
OpenTofu version. This works for our own meshstack_tenant_v4 to meshstack_tenant moves only
because the meshstack provider is plugin-framework based and implements MoveState.

The only non-destructive path the provider documents is terraform state rm followed by
terraform import.

Changing the type names without that gives a destroy and create. For one existing forgejo connector
that means:

  • kubernetes_secret.forgejo_actions is a service account token, so the kubeconfig rotates. It feeds
    local.action_secrets, which is hashed into terraform_data.await_pipeline_workflow.triggers_replace,
    so the connector re-triggers the Forgejo pipeline and blocks up to 15 minutes waiting for
    build_image,deploy — and fails the building block run if that pipeline fails.
  • kubernetes_default_service_account.this inherits its delete from the plain service account
    resource, so it really deletes the namespace's default ServiceAccount. Kubernetes recreates a bare
    one and the create step re-patches imagePullSecrets. Pods started in that window get
    ImagePullBackOff, and the re-triggered deploy job runs in exactly that window.
  • The role binding and cluster role binding are destroyed before they are recreated, so there is a gap
    with no edit permission in the namespace.

The lever we do have: the pre-run script

implementation.terraform.pre_run_script on a building block definition version runs after the
"Initialize Terraform and select Workspace" step, so the backend is already configured and the state
is reachable from the runner. modules/stackit/project already uses one (prerun.sh exec'ing
prerun.py).

That makes the documented state rm plus import migration workable inside meshStack, which it is
not from an operator machine — the http backend gives nobody CLI access to that state. A migration
script would, per resource:

  1. Check whether the object is already tracked under the _v1 address, and do nothing if so, so the
    script is idempotent across re-runs.
  2. tofu state rm <old address>.
  3. tofu import <new address> <id> with the same id the resource already has.

It needs to be a no-op for building block instances created fresh off the migration ref, since they
have nothing to migrate. Once every instance has run it, a later ref drops the script.

Proposed steps

  1. Now, separately from any rename: pin the kubernetes provider to ~> 3.0 in every module that
    declares it. One line each, no plan change, and it turns "everything breaks the day v4 ships" into
    a decision we schedule.
  2. Write new modules with _v1 names from the start.
  3. Treat the 197 existing occurrences as a v4 upgrade project, module by module, using a pre-run
    migration script. ske/forgejo-connector is the natural first one, since it is where the warnings
    were noticed and it has eight resource blocks across six types.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions