Datastore crypto key is regenerated on every render when the chart is templated without cluster access (ArgoCD/tanka), silently breaking all encrypted datastore values

[LuK133]

Chart version

stackstorm-ha 1.1.0

What happens

templates/secrets_datastore_crypto_key.yaml preserves an auto-generated
datastore key by reading the existing Secret with lookup:

{{- $previous := lookup "v1" "Secret" .Release.Namespace $name }}
{{- if .Values.st2.datastore_crypto_key }}   ...use value
{{- else if $previous }}                      ...reuse existing
{{- else }}                                   ...generate (randBytes 32)

lookup returns an empty result whenever the chart is rendered without a live
cluster connection — which is exactly what helm template does, and therefore
what ArgoCD (client-side rendering) and tanka's helm.template() do. The
"preserved across upgrades" branch is never taken, so every render emits a new
random key.

Impact

Every value already encrypted in the ST2 datastore becomes undecryptable. Pack
config rendering then fails for any config item that references st2kv, so
every action fails before it runs:

Signature did not match digest.
  ...st2common/util/config_loader.py, _assign_dynamic_config_values
  ...st2common/services/config.py, deserialize_key_value
  ...st2common/util/crypto.py, cryptography_symmetric_decrypt

In our case this silently disabled an automation platform for three weeks: the
jobs and pods stayed Healthy, nothing was marked failed, and the only symptom
was that every execution died instantly.

Reproduce

helm template st ./stackstorm-ha | grep 'datastore_crypto_key:'
helm template st ./stackstorm-ha | grep 'datastore_crypto_key:'

Two different values.

Workaround

Set st2.existingDatastoreSecret and manage the Secret outside the chart (we
back it with Vault), or pin st2.datastore_crypto_key.

Suggested fix

• Document the caveat directly next to datastore_crypto_key in values.yaml:
  the "auto-generated and preserved" promise only holds for cluster-connected
  installs, not for helm template-based delivery.
• Optionally fail the render when no explicit key and no
  existingDatastoreSecret are set and lookup returns nothing — a loud error
  at deploy time beats undecryptable data discovered weeks later.