feat(data-app): let secrets-set write plain (unencrypted) env vars (#738) - #741
Draft
padak wants to merge 1 commit into
Draft
feat(data-app): let secrets-set write plain (unencrypted) env vars (#738)#741padak wants to merge 1 commit into
secrets-set write plain (unencrypted) env vars (#738)#741padak wants to merge 1 commit into
Conversation
…738) `parameters.dataApp.secrets` has always held two kinds of entry, and the Keboola UI offers exactly that choice ("optional encryption"): a `#`-prefixed key is encrypted under the project KMS, a bare key is a plain env var whose value stays readable. `secrets-list` / `secrets-get` / `secrets-remove` have handled both since 0.43.9 -- only WRITING a plain key was impossible, because `secrets-set` demanded the `#` and encrypted unconditionally. Storing a non-secret (a Storage Files tag, a feature flag, a log level) then had two bad options: encrypt it -- write-only, since the Encryption API has no decrypt endpoint, so nobody can ever read back what it was set to -- or `sync pull`, hand-edit `_config.yml`, `sync push`. - The `#` now DECIDES encryption instead of being mandatory: `#KEY=VALUE` is encrypted (unchanged), `KEY=VALUE` is written verbatim. Same rule for `--secrets-file`. - A payload with no `#` key never calls the Encryption API at all, so an unreachable API cannot fail a write that needs no encryption. - A `KBC::` value is still rejected under EITHER form: under `#` it would double-encrypt, under a bare key it would park a foreign project's ciphertext where nothing can decrypt it. - `--allow-plaintext-on-encrypt-failure` is unrelated and unchanged -- it covers a `#` key whose encryption FAILED, and `plaintext_written` stays distinct from the new `plaintext_keys`. - Response gains `encrypted_keys` / `plaintext_keys`; `secrets_set` still lists both, so existing parsers are unaffected. `secrets-list` entries gain `encrypted` and `value` (`null` for every encrypted entry) and human output prints plain values inline. The validation loop moved to `partition_secret_entries` in `services/_data_app_bodies.py` -- `data_app_service.py` is grandfathered by the file-size gate and may only shrink. Tested: 9 new unit tests (plain-only write skips encryption, mixed payload encrypts only the `#` half, existing entries of both kinds survive, `KBC::` refusal on a bare key, dry-run masking, list tagging, CLI plumbing) plus a live E2E case asserting the plain value lands verbatim in the config while its sibling lands as `KBC::` ciphertext. `make check` green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
data-app secrets-setnow accepts a bareKEY=VALUEalongside#KEY=VALUE. The#prefix decides encryption instead of being mandatory, mirroring the UI's "optional encryption" checkbox.Why
parameters.dataApp.secretshas always held both kinds of entry, andsecrets-list/secrets-get/secrets-removehave handled both since 0.43.9. Only writing a plain key was impossible —secrets-setdemanded the#and encrypted unconditionally.That left two bad options for a non-secret setting:
secrets-listandconfig detailboth show<encrypted>.sync pull, hand-edit_config.yml,sync push.Design notes
#key never calls the Encryption API. An unreachable Encryption API must not fail a write that needs no encryption.KBC::value is rejected under EITHER form. Under#it would double-encrypt; under a bare key it would park a foreign project's ciphertext in the config where nothing can ever decrypt it. (Previously this guard existed only for#keys, since bare keys could not be written.)--allow-plaintext-on-encrypt-failureis unrelated and unchanged. It covers a#key whose encryption failed, still warns loudly, and itsplaintext_writtenresponse key stays distinct from the newplaintext_keys(deliberate plain entries).secrets_setstill lists every written key, so existing parsers are unaffected;encrypted_keys/plaintext_keysare new.secrets-listentries gainencryptedandvalue—valueisnullfor every encrypted entry, preserving the never-decrypt boundary.--secrets-filefollows the same rule (a JSON object whose#-prefixed keys are encrypted and whose bare keys are written in clear).File placement
The key/value validation loop moved out of
set_data_app_secretsintopartition_secret_entriesinservices/_data_app_bodies.py:services/data_app_service.pyis grandfathered byscripts/check_file_size.pyand may only shrink, so the feature had to make room for itself (1641 → 1635 code lines).Behavior change
One existing test parametrization changed:
"BAD"(no#) is no longer a malformed key forsecrets-set— it is a valid plain env var._validate_secret_key(require_hash=True)still exists and still rejects it;secrets-setsimply no longer passes that mode. Covered by a renamed test that says so.How it was tested
#half, pre-existing entries of both kinds survive the read-modify-write,KBC::refusal on a bare key, dry-run shows the plain value while masking the encrypted one,secrets-listtagging, and CLI plumbing (bare key forwarded; an entry without=still exits 2).test_data_app_secrets_set_writes_plain_env_vars) that writes a mixed payload against a real project KMS and asserts viaconfig detailthat the plain value is stored verbatim while its sibling isKBC::ciphertext.make checkgreen (6576 tests).Docs updated:
CLAUDE.md,context.pyAGENT_CONTEXT,SKILL.md(regenerated),commands-reference.md,gotchas.md. Version-gated asvNEXTper the release process.Fixes #738