Skip to content

Truncate generated container names to the Kubernetes 63-character limit - #57

Draft
Mohit-Ak wants to merge 1 commit into
polyaxon:masterfrom
Mohit-Ak:fix/truncate-container-names
Draft

Truncate generated container names to the Kubernetes 63-character limit#57
Mohit-Ak wants to merge 1 commit into
polyaxon:masterfrom
Mohit-Ak:fix/truncate-container-names

Conversation

@Mohit-Ak

Copy link
Copy Markdown

Polyaxon builds init container names from the connection name so it's obvious what each container is for, but nothing caps the result. Kubernetes container names are DNS labels and top out at 63 characters, so a long connection name produces an invalid pod spec and the deployment is rejected outright:

Deployment.apps "plx-operation-UUID" is invalid:
spec.template.spec.initContainers[2].name: Invalid value:
"polyaxon-init-artifacts-VERY_LONG_NAME-39c73cfbbd": must be no more than 63 characters

generate_container_name just did "{}-{}".format(prefix, suffix) with no length awareness, and sanitize_container_name only handled underscores and casing. With a 53-character connection name the artifacts init container came out at 89 characters.

This isn't specific to the artifacts store from the report — every init container goes through the same helper, so the git, custom, tensorboard, dockerfile and file init containers all have the same failure mode. I checked each of those call sites in _k8s/converter/base/init.py and _docker/converter/base/init.py; they all pass a user-controlled connection name as the suffix.

What the fix does

generate_container_name now computes how much room the caller-supplied suffix can actually have — total budget minus the prefix, the joining dashes and the 10-character unique value — and trims only that part. The prefix stays intact so names remain readable and greppable, and the uuid fragment stays intact so distinct runs never collide. The unique=False path gets the same treatment against the prefix alone.

sanitize_container_name applies the same cap, which covers the _patch_container paths in the k8s and docker converters where a name arrives from elsewhere.

Trimming can leave a trailing dash mid-word, which would itself be an invalid DNS label, so _trim strips stray dashes off the ends.

Short names are unaffected — the budget only binds when the name would have been too long, so existing behaviour and existing test expectations don't move.

Testing

New cli/tests/test_containers/test_names.py covers the reported case and the siblings: the length cap across prefixes and both unique modes, uniqueness under truncation, DNS-label validity, and the sanitize path. There are also two cases asserting short names come through byte-for-byte unchanged, so the truncation can't silently start rewriting normal names.

To confirm the tests actually pin the bug rather than passing for free, I reverted names.py to its original contents and ran them with the limit inlined:

>   assert len(name) <= 63
E   AssertionError: assert 89 <= 63     # artifacts init container
E   AssertionError: assert 78 <= 63     # sanitize_container_name
2 failed, 5 passed

With the fix, all 7 pass. Full suite:

$ pytest -q
1489 passed        # master, before the change
1496 passed        # with the change (+7 new tests), 0 failures

Also spot-checked that truncation doesn't cost uniqueness: 500 generations from the same long connection name yielded 500 distinct results.

black and isort flag names.py, but they flag the pristine file on master identically (an import blank line I didn't touch, plus import ordering) — the same is true of several neighbouring modules, so that's pre-existing and I left it alone rather than adding unrelated churn.

Refs polyaxon/polyaxon#1529

Init container names are built from the connection name, so a long
connection produced a name longer than the 63 characters Kubernetes
allows for a DNS label and the pod was rejected as invalid.

Cap the generated name, trimming only the caller-supplied part so the
prefix and the unique suffix survive, and apply the same cap in
sanitize_container_name. Refs polyaxon/polyaxon#1529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant