Skip to content

🤖 fix: date the serving CA from the injected clock and backdate it - #223

Merged
ThomasK33 merged 1 commit into
mainfrom
fix/servingcert-ca-notbefore
Oct 12, 2026
Merged

ThomasK33 merged 1 commit into
mainfrom
fix/servingcert-ca-notbefore

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Refs #221. Stacked on #222: merge #222 first, then this PR retargets to main.

Summary

servingcert.Generate(namespace, now) dated the serving CA from the wall clock, not from now, and did not backdate it. Parse(..., now) therefore rejected a fresh CA as "not valid until" whenever the caller's now was even one second behind the clock that issued the CA. This PR passes NotBefore: now - clockSkewAllowance (1 h, the same margin the serving certificate already uses) to certutil.NewSelfSignedCACert.

Background

The merge queue rejected #222 (run 38277477505) on this flake in TestGenerateProducesVerifiableServingCert:

servingcert_test.go:77: generated bundle does not parse: ... the CA in data["ca.crt"] is not valid until 2026-10-12T16:28:48Z

Root cause:

  • internal/aggregated/servingcert/servingcert.go:116 called certutil.NewSelfSignedCACert without NotBefore. The vendored helper then uses its own time.Now() (vendor/k8s.io/client-go/util/cert/cert.go:61,68). x509 encodes the time to whole seconds.
  • servingcert.go:215 rejects the CA when now.Before(CACert.NotBefore).
  • The test takes now := time.Now() before Generate. When a second boundary falls between that call and certutil's time.Now(), the truncated CA NotBefore lands after now, and Parse fails.

This predates #222. vendor/k8s.io did not change in #222, and the original test also fails on main (go1.26.8, old deps).

Production impact (same cause): the serving certificate is backdated by 1 h for clock skew, but the CA was not. A replica, or a kube-apiserver, whose clock is behind the issuing pod's saw a CA from the future. A replica that adopts the Secret then got a CorruptSecretError ("delete it so coder-k8s generates a new CA"). I did not reproduce the kube-apiserver case on a cluster.

Implementation

  • Generate sets certutil.Config.NotBefore = now.Add(-clockSkewAllowance). The CA NotAfter is unchanged (certutil: wall clock + 10 years), and the serving certificate is still clamped to it.
  • New test TestGenerateDatesCAFromInjectedClock generates with now 30 min in the past. It asserts that the CA NotBefore is now - 1h (to the second), that the chain verifies at now, and that Parse(..., now) accepts the bundle.

Validation

Repro counts for the original test, go test -count=N -run '^TestGenerateProducesVerifiableServingCert$' ./internal/aggregated/servingcert/:

Tree Idle, 500 Idle, 20000 -race, 500 32 CPU burners, 20000 Burners + -race, 2000
main 55e7f34 1 FAIL 2 FAIL 0 1 FAIL not run
#222 dda8efa 0 0 0 1 FAIL 1 FAIL
this PR 0 0 0 0 0

The new test fails 100/100 without the fix (CA NotBefore = <wall clock>, want <now - 1h>) and passes 100/100 with it.

Risks

Low. Only newly generated CAs change: they are now valid from 1 h before issue. Existing Secrets are parsed as before.


Generated with mux • Model: anthropic:claude-opus-5-5 • Thinking: high

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T18:04:30.913848Z 5bf9622 Manual request
🔒 Security Review ✅ Completed 2026-10-12T18:03:57.907752Z 5bf9622 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1d3b44c9dc

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Base automatically changed from fix/govulncheck-go1.26.9 to main October 12, 2026 17:59
…efs #221)

Change-Id: I69ad7649e8cae216fff651ddd91ff6aff9f5833b
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33
ThomasK33 force-pushed the fix/servingcert-ca-notbefore branch from 1d3b44c to 5bf9622 Compare October 12, 2026 18:01
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5bf9622c8b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 5bf9622c8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit 1ba9960 Oct 12, 2026
10 checks passed
@ThomasK33
ThomasK33 deleted the fix/servingcert-ca-notbefore branch October 12, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant