Skip to content

feat: Add authenticated database readiness check based on driver session metadata - #2527

Open
erichare wants to merge 15 commits into
mainfrom
fix/2526-cassandra-readiness
Open

feat: Add authenticated database readiness check based on driver session metadata#2527
erichare wants to merge 15 commits into
mainfrom
fix/2526-cassandra-readiness

Conversation

@erichare

@erichare erichare commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

Adds GET /v1/health/ready, an authenticated readiness endpoint for both Astra and Cassandra.

  • Gets a session via the normal CQLSessionCache using the request's tenant, token, and User-Agent, and reports UP when the session's driver metadata has an UP node. No query is issued and no canary table is needed — a pod that cannot connect fails session creation and reports DOWN.
  • Requires the configured SLA User-Agent (stargate.jsonapi.operations.sla-user-agent) so probe sessions get the shorter SLA cache TTL. Fails closed if it is unset.
  • Responses: 200 {"status":"UP"}, 401 invalid auth, 403 wrong User-Agent, 503 {"status":"DOWN"}. Fully async with a 5s timeout. Readiness polling is excluded from tenant request metrics.

This also means a probe token that authenticates but cannot read the schema makes the endpoint report DOWN rather than an auth error.

Which issue(s) this PR fixes:

Fixes #2526

Validation:

  • Full unit suite green (3441 tests); formatting clean.
  • SessionEvictionIntegrationTest extended to assert readiness UP, DOWN, UP across a container outage.

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Unit Test Coverage Report

Overall Project 53.69% -0.08% 🍏
Files changed 75.99% 🍏

File Coverage
DatabaseReadinessResource.java 100% 🍏
DatabaseReadinessCheck.java 100% 🍏
CqlSessionCacheSupplier.java 96.12% 🍏
TenantRequestMetricsFilter.java 27.15% -56.29%

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (dse69-it)

Overall Project 71.5% -0.15% 🍏
Files changed 55.65%

File Coverage
CqlSessionCacheSupplier.java 99.06% -0.94% 🍏
DatabaseReadinessCheck.java 83.72% -16.28% 🍏
DatabaseReadinessResource.java 72.46% -27.54% 🍏
TenantRequestMetricsFilter.java 16.23% -67.53%

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (hcd-it)

Overall Project 72.83% -0.15% 🍏
Files changed 55.65%

File Coverage
CqlSessionCacheSupplier.java 99.06% -0.94% 🍏
DatabaseReadinessCheck.java 83.72% -16.28% 🍏
DatabaseReadinessResource.java 72.46% -27.54% 🍏
TenantRequestMetricsFilter.java 16.23% -67.53%

@erichare
erichare marked this pull request as ready for review July 27, 2026 17:19
@erichare
erichare requested a review from a team as a code owner July 27, 2026 17:19
@erichare
erichare requested review from amorton and clun July 27, 2026 17:19

@amorton amorton left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 this only works in non-astra, uses blocking IO, and reads from a non replicated table so why the C* node will need to be marked as UP to respond it will not verify that its able to communicate with enough nodes to achieve quourm

need to rethink what we are trying to do here

Comment thread src/main/java/io/stargate/sgv2/jsonapi/config/OperationsConfig.java Outdated
Comment thread src/test/java/io/stargate/sgv2/jsonapi/api/v1/SessionEvictionIntegrationTest.java Outdated
@amorton

amorton commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

this needs a rethink about what we are trying to do

@erichare

erichare commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

this needs a rethink about what we are trying to do

understood! i'll address the specific comments you made just for the practice, but yeah understood about rethinking the purpose

@erichare

erichare commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Okay, @amorton , so here's what i did...

First of all, renamed the endpoint to GET /v1/health/ready, no stargate in the path lol. Its an authenticated database readiness endpoint used for both Astra and Cassandra deployments. It gets a session through the normal session cache.

The endpoint executes SELECT * FROM datastax_sla.check LIMIT 1 with the table-read driver
profile. That profile uses LOCAL_QUORUM, so an UP response confirms that the coordinator can
complete a read from a replicated table at local quorum. It does not validate every tenant's
credentials, write availability, or cross-region availability.

Switched everything to async, response would look like {"status":"UP"}... and the readiness probe is pure java, no CDI.

I'll push shortly

@erichare
erichare requested a review from amorton August 4, 2026 03:23
@erichare erichare changed the title Add Cassandra-based readiness probe Add authenticated database readiness check for Astra and Cassandra Aug 4, 2026
@erichare erichare changed the title Add authenticated database readiness check for Astra and Cassandra Add authenticated database readiness check based on driver session metadata Aug 10, 2026
Add check when creating a new CqlSession that it has
Metadata so we know what keyspaces/tables in the DB.

See comments in CqlSessionFactory
@erichare
erichare force-pushed the fix/2526-cassandra-readiness branch from f9b504e to aa3031a Compare August 12, 2026 20:58
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

📈 Unit Test Coverage Delta vs Main Branch

Metric Value
Main Branch 53.39%
This PR 53.69%
Delta 🟢 +0.30%
✅ Coverage improved!

@erichare
erichare changed the base branch from main to ajm/fix-2539-session-metadata August 12, 2026 21:06
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

📈 Integration Test Coverage Delta vs Main Branch (dse69-it)

Metric Value
Main Branch 71.46%
This PR 71.50%
Delta 🟢 +0.04%
✅ Coverage improved!

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

📈 Integration Test Coverage Delta vs Main Branch (hcd-it)

Metric Value
Main Branch 72.78%
This PR 72.83%
Delta 🟢 +0.05%
✅ Coverage improved!

amorton and others added 10 commits August 13, 2026 11:33
Replace the datastax_sla.check canary query with a check of the driver
session metadata: the pod is ready when the session obtained through the
normal session cache reports at least one UP node. No query is issued and
no canary table needs to be provisioned.

Add stargate.jsonapi.operations.database-config.require-session-node-metadata
(default false): when enabled, CqlSessionFactory rejects a newly built
session whose metadata contains no nodes, closing it and failing creation
with the standard FAILED_TO_CONNECT_TO_DATABASE error so an unusable
session is never cached.
@erichare
erichare force-pushed the fix/2526-cassandra-readiness branch from 5be2ec1 to e84e3c7 Compare August 12, 2026 23:43
Base automatically changed from ajm/fix-2539-session-metadata to main August 13, 2026 00:08
@erichare erichare changed the title Add authenticated database readiness check based on driver session metadata feat: Add authenticated database readiness check based on driver session metadata Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Cassandra-Based Readiness Probe for Data API

2 participants