Skip to content

fix: keep current key pair when key and certificate mismatch - #16

Merged
stokpop merged 10 commits into
cloudfoundry:mainfrom
stokpop:fix/key-manager-loose-ends
Nov 9, 2026
Merged

stokpop merged 10 commits into
cloudfoundry:mainfrom
stokpop:fix/key-manager-loose-ends

Conversation

@stokpop

@stokpop stokpop commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Ties up the loose ends of #8 after #12. Before replacing the KeyManager, the private key is checked against the leaf certificate (RSA, the type the Diego executor issues). On a reload with a mismatch or an empty certificate file the current pair is kept; on the first load an invalid pair is not installed, so no client certificate is presented but other TLS connections keep working. Both cases, and an expired chain, are logged as errors (JUL SEVERE). Reloads are serialized, each load logs the certificate serial, subject and chain expiry, and the commented-out key watcher is replaced by a note on why only the certificate is watched. Supersedes #9 and #10.

Note: a mismatch is not retried and does not fail fast; see the new "Certificate rotation" section in the README for alerting and recovery.

Draft: planned for 1.22.0, after a 1.21.1 release with the recent dependency updates, as these changes need more review and testing.

Note: if a reload happens between choosing a client alias and looking up its chain or key, the lookup returns null. This is a pre-existing, very small window. It is logged at SEVERE with the prefix Should not happen:, so we can see whether it ever occurs. If it does, a possible fix is to tag aliases with a generation number and keep the previous KeyManager for a short grace period. That is a heavier change, so it is left out of this PR.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive reload ordering needs deterministic concurrency coverage and final human validation.

1 open finding
What changed in this PR

Protects certificate rotation by retaining the current KeyManager when a replacement RSA key and certificate do not match.

Changes:

  • Serializes reloads and adds certificate identity and expiry logging.
  • Adds reload tests and documents warning recovery.
  • Advances the development version and adds locking annotations.
File Description
src/​test/​java/​org/​cloudfoundry/​security/​FileWatchingX509ExtendedKeyManagerTest.java Tests matching, mismatched, and empty certificate reloads and chain expiry.
src/​main/​java/​org/​cloudfoundry/​security/​FileWatchingX509ExtendedKeyManager.java Serializes reloads, checks replacement pairs, and logs certificate details.
README.md Documents rotation behavior, warnings, and recovery.
pom.xml Sets 1.22.0-SNAPSHOT and adds compile-time locking annotations.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- Check the private key matches the leaf certificate before replacing
  the KeyManager; keep the current one otherwise, also for an empty
  certificate file. Non-RSA keys are accepted with an info message.
- Serialize reloads so the shared KeyManagerFactory is never
  initialized concurrently.
- Log certificate serial, subject and chain expiry (UTC) on load;
  warn when the chain has expired.
- Remove the commented-out private key watcher and document why only
  the certificate file is watched.
- Document certificate rotation and mismatch recovery in the README.
- Bump revision to 1.22.0-SNAPSHOT.
- Warn when the first loaded pair mismatches; there is no current pair
  to keep, so it is installed as before
- Hint at the likely cause in the reload mismatch warning
- Link the Diego executor source for the key-before-certificate order
- Replace files atomically in the watcher test, fail within 30s
@stokpop
stokpop force-pushed the fix/key-manager-loose-ends branch from e2557a9 to dd59b75 Compare October 21, 2026 09:07

@jcvrabo jcvrabo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would argue that mismatched errors should be logged as errors instead of warnings (keeping a soon-to-be-expired certificate will keep valid for about 1 hour and assuming that the issue will remain, the app will never really have a valid certificate again).

I would also wonder if in case the app starts without a valid key/certificate pair there is any point in actually loading a (useless) certificate/key pair. Since that is an edge case which would come from a deeper problem with the diego cell, I would just log the error (as error, not a warning and not throwing exceptions as the app might not even need the certificate to operate) and leave the keymanager uninitialized.

- On first load, don't install a mismatched pair or an empty
  certificate file; delegate methods return null, so other TLS
  connections keep working and the next matching rotation installs it
- Log mismatches and expired chains as SEVERE (ERROR via SLF4J)
- README: describe the new first-load behaviour and error levels

Refs cloudfoundry#16 (review by jcvrabo)
@stokpop

stokpop commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

@jcvrabo thanks, both adopted in faa3593:

  • Mismatches (and expired chains) are now logged at SEVERE, shown as ERROR through SLF4J and Spring Boot.
  • On the first load an invalid pair, or an empty certificate file, is not installed. An error is logged and nothing is thrown. No client certificate is presented, other TLS connections keep working, and the next certificate change with a matching pair installs it.

Compared with 1.21.1 this is never worse:

  • A mismatch on first load used to be installed silently.
  • An empty file used to throw on every SSLContext.
  • A mismatch on reload used to replace the good pair.

An opt-in health signal (for example a JMX MBean that an app health check could read, triggering a restart) could be a follow-up issue.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An in-flight watcher log publication can race with iteration of LogRecorder’s backing ArrayList, causing intermittent ConcurrentModificationException failures in the test.

1 open issue
1 resolved since last review

Review effort: Balanced

@stokpop
stokpop marked this pull request as ready for review October 31, 2026 14:52
@stokpop
stokpop requested a balanced review from Copilot October 31, 2026 14:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the tests do not verify the expired-chain diagnostic or its severity, despite its documented role in operational alerting.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

The watcher thread can still publish a log record while the test reads
the recorded records, which could throw ConcurrentModificationException.
@stokpop
stokpop force-pushed the fix/key-manager-loose-ends branch from 74e3e2a to fabe771 Compare October 31, 2026 23:15
getClientAliases can return null or an empty array, so the helper no
longer indexes it blindly. It retries a bounded number of times, because
the watcher can swap the key manager between the alias and chain calls,
and then fails with a descriptive message.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Replacing the watched certificate can trigger an asynchronous reload after currentAlias is captured, causing intermittent assertion failures because the explicit update() does not wait for the watcher callback.

1 open issue

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the unchecked fallback can accept mismatched RSA/non-RSA key and certificate pairs, bypassing safeguards that preserve the current KeyManager.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@stokpop

stokpop commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

On the Copilot note about the unchecked non-RSA fallback in matches(): left as is on purpose. The Diego executor only generates RSA keys (credmanager.go uses *rsa.PrivateKey), and key and certificate are created together, so a mixed RSA/non-RSA pair would need an algorithm switch in Diego to coincide with the rename window during rotation. If Diego ever moves to EC or post-quantum keys, both change together and the existing INFO log ("not an RSA key ... the security provider probably needs an update") signals that a check for the new algorithm should be added.

@stokpop
stokpop merged commit 7c7b0e0 into cloudfoundry:main Nov 9, 2026
4 checks passed
@stokpop
stokpop deleted the fix/key-manager-loose-ends branch November 9, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants