feat: Add api-schema-drift-sentinel kit - #341
Conversation
|
Warning Review limit reached
Next review available in: 59 minutes Limit details: You’ve used all 2 included reviews currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan includes up to 2 reviews per rolling hour; 1 remains after this review. WalkthroughThe kit adds deterministic OpenAPI schema comparison, normalized risk classification, Lamatic workflow analysis, a POST API endpoint, and a Next.js dashboard with setup and workflow test documentation. ChangesSchema Drift Sentinel
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
|
Failure recorded at 2026-08-11T10:56:37Z UTC. If this PR is not fixed within 4 weeks it will be automatically closed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 23
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/api-schema-drift-sentinel/.gitignore`:
- Around line 3-5: Update the environment ignore patterns in .gitignore to
ignore all .env.* files, including development, production, and test variants,
while explicitly re-including the tracked apps/.env.example file. Preserve the
existing .env and .env.local coverage.
In `@kits/api-schema-drift-sentinel/apps/actions/orchestrate.ts`:
- Around line 13-22: Update the server action around normalizeDiff and the
payload construction: pass oldSpecContent and newSpecContent to normalizeDiff so
parameter type changes are detected consistently with the analyze-drift route,
and set changesCount from normalizedChanges.facts.totalBreaking rather than
normalizedChanges.allChanges.length to preserve the breaking-change count
contract.
- Line 3: Add an app-local configuration module under the apps deployment root
defining the analyze-schema-drift step, using LAMATIC_DRIFT_FLOW_ID as its flow
identifier. Update orchestrate.ts and related app wiring to consume this local
configuration instead of importing the parent lamatic.config.ts, while
preserving the existing sentinel imports and behavior.
In `@kits/api-schema-drift-sentinel/apps/app/api/analyze-drift/route.ts`:
- Around line 6-17: Validate specA and specB in the route handler before calling
runOpenApiDiff: require each value to be a string and enforce the intended
maximum size for each specification, returning the existing 400 response for
invalid or oversized inputs. Keep valid payloads flowing unchanged into
runOpenApiDiff.
In `@kits/api-schema-drift-sentinel/apps/app/globals.css`:
- Line 1: Remove the unused Google Fonts `@import` from globals.css, or integrate
Inter through next/font/google in the root layout and apply its generated class
or CSS variable to the body alongside the existing classes. Ensure the font is
either actually used or no longer downloaded.
In `@kits/api-schema-drift-sentinel/apps/app/layout.tsx`:
- Line 15: Remove suppressHydrationWarning from the root body element in the app
layout, or move it to the smallest specific dynamic child that has the known
hydration mismatch; keep the body’s existing classes and children rendering
unchanged.
In `@kits/api-schema-drift-sentinel/apps/app/page.tsx`:
- Around line 209-238: Remove suppressHydrationWarning from the controls and
textareas in the component, including the elements associated with loadExample,
specA, specB, and the analysis workflow around the referenced locations. Do not
replace it with another suppression unless a confirmed browser-extension
mismatch exists; if one does, document that reason with a short comment.
- Around line 230-238: Add accessible names to both spec textareas in the page
component by associating visually hidden labels or applying descriptive
aria-label values to their existing ids. Mark the analyze button’s loading state
with aria-busy={loading}, and add role="alert" to the error panel so failures
are announced.
- Around line 154-160: Normalize the untrusted AI response fields before
rendering in the page component: make recommendation support string
executiveSummary values as well as executiveSummary.recommendation, and only
accept string values for recommendation, detailedImpact, and migrationGuide,
falling back to safe defaults otherwise. Update the existing derived fields
around risk, recommendation, detailedImpact, and migrationGuide without changing
the valid-object behavior, and use the response interface instead of
useState<any> if the route’s declared shape is available.
- Around line 163-196: Update the page root container to consume a font CSS
variable defined in globals.css instead of hardcoding the font stack in its
inline style. Replace every raw inline SVG icon in the page, including the
header hex and GitHub icons, with appropriate lucide-react components while
preserving their existing appearance, sizing, colors, and accessibility labels.
In `@kits/api-schema-drift-sentinel/apps/lib/sentinel.ts`:
- Around line 354-386: The polling loop in the function containing the status
check can exceed the serverless runtime limit before its timeout error is
returned. Either export an appropriate maxDuration from the calling route to
cover the full polling window, or reduce the attempts and delay; prefer bounded
exponential backoff if adjusting polling so early completions return sooner
while preserving successful and error status handling.
- Around line 372-374: Update the status-response parsing in the polling flow
around rawResult and parsedData so malformed string payloads cannot throw out of
the loop. Guard JSON.parse, preserve valid parsed data, and continue polling
when parsing fails instead of aborting the workflow.
- Around line 300-307: Update the REST request flow around the res.ok check and
catch block to log the failed response status before falling back to GraphQL,
and log the caught fetch error before fallback when fetch throws. Preserve the
existing successful response handling and GraphQL fallback behavior.
- Around line 329-344: Set an explicit Axios timeout on both outbound requests
in the execute call and the status-polling call, using the same bounded duration
for each. Update the Axios configuration near the visible POST request and its
corresponding status request without changing the polling or request behavior.
- Around line 169-184: Update the classification ladder in the change-mapping
logic to handle response.body.scope.remove and request.body.scope.add explicitly
before generic remove/add checks. Ensure response-side removals map to the
correct non-breaking or breaking classification based on isBreaking, and
breaking request-side scope additions map to REQUIRED_FIELD_ADDED with action
"add"; prevent the generic scope.add branch from overriding this behavior.
- Around line 76-84: Update the parameter comparison around v1Op.parameters and
v2Op.parameters to include each operation’s path-item parameters, merging
path-level and operation-level entries by name and location with operation-level
entries taking precedence. Use the merged parameter sets for the existing type
comparison so shared path-level parameter changes are detected.
In `@kits/api-schema-drift-sentinel/apps/package.json`:
- Around line 14-22: Update the dependency declarations in the apps package
manifest so react, react-dom, `@types/react`, and `@types/react-dom` use compatible
React 18 releases, while leaving the Next.js and unrelated dependencies
unchanged.
- Line 9: Update the lint script in the package scripts to stop invoking the
deprecated next lint wrapper. Add a project-owned ESLint or Biome dependency and
configuration, then invoke that tool directly; alternatively remove the lint
script if linting is not supported for this app.
In `@kits/api-schema-drift-sentinel/apps/test-orchestrate.js`:
- Around line 164-231: Update runMatrixTests to add deterministic assertions for
each normalized payload, validating the expected additive and breaking change
counts and normalized change contents. After each triggerWorkflowAndPoll call,
assert that the returned workflow result is present; throw or otherwise fail
explicitly when it is null or absent. Do not assert workflow-generated or LLM
prose.
- Around line 89-104: Update the request flow in test-orchestrate.js to create
one Axios client with a finite timeout, such as 15 seconds, and replace direct
axios calls for both the execution and status requests with that client.
Preserve the existing request methods, URLs, headers, and polling limit.
In
`@kits/api-schema-drift-sentinel/prompts/analyze-schema-drift_llm-node_system.md`:
- Around line 18-22: Update the “Risk classification” section in the
schema-drift analysis prompt to define only the deterministic HIGH/LOW mapping:
use HIGH when one or more breaking changes are present and LOW otherwise. Remove
the allowance for MEDIUM and CRITICAL so the prompt matches the HIGH/LOW values
produced through facts.calculatedRisk.
- Line 41: Update the prompt rule for breakingChangesCount to explicitly define
that facts with Severity: CRITICAL are breaking changes, and require the count
to equal the number of such supplied facts. Keep the existing fact-line format
and ensure the mapping is stated near the breakingChangesCount requirement.
In `@kits/api-schema-drift-sentinel/README.md`:
- Around line 186-194: Align the Test A scenario in the README with the harness
behavior in apps/test-orchestrate.js: either document the existing full_name
addition to GET /users/{id} or update the fixture to add POST /users. Ensure the
expected changesCount matches factsAdditive.totalBreaking, using 0 for this
additive case, while keeping breakingChangesCount and deploymentRisk consistent.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 053d2769-e2fc-44b9-8303-2f9495816410
⛔ Files ignored due to path filters (1)
kits/api-schema-drift-sentinel/apps/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (17)
kits/api-schema-drift-sentinel/.gitignorekits/api-schema-drift-sentinel/README.mdkits/api-schema-drift-sentinel/apps/.env.examplekits/api-schema-drift-sentinel/apps/actions/orchestrate.tskits/api-schema-drift-sentinel/apps/app/api/analyze-drift/route.tskits/api-schema-drift-sentinel/apps/app/globals.csskits/api-schema-drift-sentinel/apps/app/layout.tsxkits/api-schema-drift-sentinel/apps/app/page.tsxkits/api-schema-drift-sentinel/apps/lib/sentinel.tskits/api-schema-drift-sentinel/apps/next-env.d.tskits/api-schema-drift-sentinel/apps/package.jsonkits/api-schema-drift-sentinel/apps/postcss.config.jskits/api-schema-drift-sentinel/apps/tailwind.config.jskits/api-schema-drift-sentinel/apps/test-orchestrate.jskits/api-schema-drift-sentinel/apps/tsconfig.jsonkits/api-schema-drift-sentinel/lamatic.config.tskits/api-schema-drift-sentinel/prompts/analyze-schema-drift_llm-node_system.md
| @@ -0,0 +1,29 @@ | |||
| "use server"; | |||
|
|
|||
| import { runOpenApiDiff, normalizeDiff, triggerLamaticWorkflow } from '../lib/sentinel'; | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check the kit deploy root and the flow-ID env key alignment.
set -euo pipefail
fd -t f 'lamatic.config.ts' kits/api-schema-drift-sentinel --exec cat {}
fd -t f '.env.example' kits/api-schema-drift-sentinel --exec cat {}
rg -n 'LAMATIC_DRIFT_FLOW_ID|root-directory|links' kits/api-schema-drift-sentinelRepository: Lamatic/AgentKit
Length of output: 2352
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- orchestrate.ts ---'
cat -n kits/api-schema-drift-sentinel/apps/actions/orchestrate.ts
printf '%s\n' '--- app files related to configuration ---'
fd -t f . kits/api-schema-drift-sentinel/apps | sort | rg '(^|/)(lamatic|config|orchestrate)|package.json|next.config'
printf '%s\n' '--- configuration references ---'
rg -n 'lamatic\.config|LAMATIC_DRIFT_FLOW_ID|envKey|steps|workflowId' kits/api-schema-drift-sentinel/apps kits/api-schema-drift-sentinel/lamatic.config.tsRepository: Lamatic/AgentKit
Length of output: 2317
Add app-local step configuration.
The deployment root is kits/api-schema-drift-sentinel/apps. Do not import the parent lamatic.config.ts. The app currently has no local configuration module. Add one for the analyze-schema-drift step and use LAMATIC_DRIFT_FLOW_ID, which already matches sentinel.ts and the parent configuration.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/apps/actions/orchestrate.ts` at line 3, Add an
app-local configuration module under the apps deployment root defining the
analyze-schema-drift step, using LAMATIC_DRIFT_FLOW_ID as its flow identifier.
Update orchestrate.ts and related app wiring to consume this local configuration
instead of importing the parent lamatic.config.ts, while preserving the existing
sentinel imports and behavior.
Sources: Coding guidelines, Learnings
| const rawDiff = await runOpenApiDiff(oldSpecContent, newSpecContent); | ||
| const normalizedChanges = normalizeDiff(rawDiff); | ||
|
|
||
| const payload = { | ||
| apiName, | ||
| oldVersion, | ||
| newVersion, | ||
| changesCount: normalizedChanges.allChanges.length, | ||
| changes: normalizedChanges.allChanges | ||
| }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Two payload discrepancies against the rest of the mission.
-
Line 14 calls
normalizeDiff(rawDiff)without the specs.normalizeDiffrunsdetectParameterTypeChangesonly when bothspecAandspecBare present (kits/api-schema-drift-sentinel/apps/lib/sentinel.tsLine 246). This server action therefore reports zero parameter type changes, whilekits/api-schema-drift-sentinel/apps/app/api/analyze-drift/route.tsLine 21 passes the specs and reports them. The same input produces different results on the two entry points. -
Line 20 sets
changesCounttoallChanges.length.kits/api-schema-drift-sentinel/apps/test-orchestrate.jssets the same field tofacts.totalBreaking, and Rule 1 ofkits/api-schema-drift-sentinel/prompts/analyze-schema-drift_llm-node_system.mdrequiresbreakingChangesCountto equal the number of breaking changes supplied. Passing the total change count invites the model to report an inflated breaking-change count.
🎯 Proposed fix to align the payload with the route and the prompt contract
const rawDiff = await runOpenApiDiff(oldSpecContent, newSpecContent);
- const normalizedChanges = normalizeDiff(rawDiff);
+ const normalizedChanges = normalizeDiff(rawDiff, oldSpecContent, newSpecContent);
const payload = {
apiName,
oldVersion,
newVersion,
- changesCount: normalizedChanges.allChanges.length,
+ changesCount: normalizedChanges.totalBreaking,
changes: normalizedChanges.allChanges
};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const rawDiff = await runOpenApiDiff(oldSpecContent, newSpecContent); | |
| const normalizedChanges = normalizeDiff(rawDiff); | |
| const payload = { | |
| apiName, | |
| oldVersion, | |
| newVersion, | |
| changesCount: normalizedChanges.allChanges.length, | |
| changes: normalizedChanges.allChanges | |
| }; | |
| const rawDiff = await runOpenApiDiff(oldSpecContent, newSpecContent); | |
| const normalizedChanges = normalizeDiff(rawDiff, oldSpecContent, newSpecContent); | |
| const payload = { | |
| apiName, | |
| oldVersion, | |
| newVersion, | |
| changesCount: normalizedChanges.totalBreaking, | |
| changes: normalizedChanges.allChanges | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/apps/actions/orchestrate.ts` around lines 13 -
22, Update the server action around normalizeDiff and the payload construction:
pass oldSpecContent and newSpecContent to normalizeDiff so parameter type
changes are detected consistently with the analyze-drift route, and set
changesCount from normalizedChanges.facts.totalBreaking rather than
normalizedChanges.allChanges.length to preserve the breaking-change count
contract.
| async function runMatrixTests() { | ||
| console.log("=========================================="); | ||
| console.log("STEP 1: Verify Production Normalization via sentinel.ts"); | ||
| console.log("=========================================="); | ||
|
|
||
| const mockBreakingDiff = { | ||
| breakingDifferences: [ | ||
| { | ||
| code: "response.body.scope.add", | ||
| entity: "response.body.scope", | ||
| sourceSpecEntityDetails: [{ location: "paths./users/{id}.get" }], | ||
| details: { | ||
| differenceSchema: { | ||
| anyOf: [ | ||
| { required: ["name"] }, | ||
| { required: ["email"] } | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| }; | ||
|
|
||
| const normalizedMock = normalizeDiff(mockBreakingDiff); | ||
| console.log("Verified Mock Breaking Changes Normalization Output:"); | ||
| console.log(JSON.stringify(normalizedMock, null, 2)); | ||
|
|
||
| console.log("\n=========================================="); | ||
| console.log("STEP 2: Matrix Execution"); | ||
| console.log("=========================================="); | ||
|
|
||
| // Test Case A: Additive (Non-breaking Baseline) | ||
| console.log("\n--- TEST CASE A: Additive (Non-Breaking) ---"); | ||
| const diffAdditive = await runOpenApiDiff(v1, v2Additive); | ||
| const factsAdditive = normalizeDiff(diffAdditive, v1, v2Additive); | ||
| const payloadAdditive = { | ||
| apiName: "User Service API", | ||
| oldVersion: "1.0.0", | ||
| newVersion: "2.0.0", | ||
| changesCount: factsAdditive.totalBreaking, | ||
| changes: factsAdditive.allChanges | ||
| }; | ||
| console.log("Additive Normalized Payload:", JSON.stringify(payloadAdditive, null, 2)); | ||
| console.log("Triggering Lamatic Workflow for Additive Test Case..."); | ||
| const resultAdditive = await triggerWorkflowAndPoll(payloadAdditive); | ||
| console.log("Additive Test Result Output:", JSON.stringify(resultAdditive, null, 2)); | ||
|
|
||
| // Test Case B: Breaking Removal & Type Change | ||
| console.log("\n--- TEST CASE B: Breaking Removal & Type Change ---"); | ||
| const diffBreaking = await runOpenApiDiff(v1, v2Breaking); | ||
| const factsBreaking = normalizeDiff(diffBreaking, v1, v2Breaking); | ||
|
|
||
| console.log( | ||
| "FULL OPENAPI DIFF:", | ||
| JSON.stringify(diffBreaking, null, 2) | ||
| ); | ||
|
|
||
| const payloadBreaking = { | ||
| apiName: "User Service API", | ||
| oldVersion: "1.0.0", | ||
| newVersion: "2.0.0", | ||
| changesCount: factsBreaking.totalBreaking, | ||
| changes: factsBreaking.allChanges | ||
| }; | ||
| console.log("Breaking Normalized Payload:", JSON.stringify(payloadBreaking, null, 2)); | ||
| console.log("Triggering Lamatic Workflow for Breaking Test Case..."); | ||
| const resultBreaking = await triggerWorkflowAndPoll(payloadBreaking); | ||
| console.log("Breaking Test Result Output:", JSON.stringify(resultBreaking, null, 2)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Mission directive: Make the matrix fail on incorrect results.
runMatrixTests only writes results to stdout. Incorrect normalized changes and null workflow results still complete without an explicit failure.
Add deterministic assertions for each normalized payload. Fail when a workflow result is absent. Do not assert LLM prose because it is not deterministic.
Proposed fix
+const assert = require('node:assert/strict');
+
const factsAdditive = normalizeDiff(diffAdditive, v1, v2Additive);
+ assert.equal(factsAdditive.totalBreaking, 0);
+ assert.ok(
+ factsAdditive.nonBreakingChanges.some(
+ (change) => change.field === 'full_name' && change.action === 'add'
+ )
+ );
...
const resultAdditive = await triggerWorkflowAndPoll(payloadAdditive);
+ assert.ok(resultAdditive, 'Additive workflow returned no result');
...
const factsBreaking = normalizeDiff(diffBreaking, v1, v2Breaking);
+ assert.equal(factsBreaking.totalBreaking, 3);
...
const resultBreaking = await triggerWorkflowAndPoll(payloadBreaking);
+ assert.ok(resultBreaking, 'Breaking workflow returned no result');📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async function runMatrixTests() { | |
| console.log("=========================================="); | |
| console.log("STEP 1: Verify Production Normalization via sentinel.ts"); | |
| console.log("=========================================="); | |
| const mockBreakingDiff = { | |
| breakingDifferences: [ | |
| { | |
| code: "response.body.scope.add", | |
| entity: "response.body.scope", | |
| sourceSpecEntityDetails: [{ location: "paths./users/{id}.get" }], | |
| details: { | |
| differenceSchema: { | |
| anyOf: [ | |
| { required: ["name"] }, | |
| { required: ["email"] } | |
| ] | |
| } | |
| } | |
| } | |
| ] | |
| }; | |
| const normalizedMock = normalizeDiff(mockBreakingDiff); | |
| console.log("Verified Mock Breaking Changes Normalization Output:"); | |
| console.log(JSON.stringify(normalizedMock, null, 2)); | |
| console.log("\n=========================================="); | |
| console.log("STEP 2: Matrix Execution"); | |
| console.log("=========================================="); | |
| // Test Case A: Additive (Non-breaking Baseline) | |
| console.log("\n--- TEST CASE A: Additive (Non-Breaking) ---"); | |
| const diffAdditive = await runOpenApiDiff(v1, v2Additive); | |
| const factsAdditive = normalizeDiff(diffAdditive, v1, v2Additive); | |
| const payloadAdditive = { | |
| apiName: "User Service API", | |
| oldVersion: "1.0.0", | |
| newVersion: "2.0.0", | |
| changesCount: factsAdditive.totalBreaking, | |
| changes: factsAdditive.allChanges | |
| }; | |
| console.log("Additive Normalized Payload:", JSON.stringify(payloadAdditive, null, 2)); | |
| console.log("Triggering Lamatic Workflow for Additive Test Case..."); | |
| const resultAdditive = await triggerWorkflowAndPoll(payloadAdditive); | |
| console.log("Additive Test Result Output:", JSON.stringify(resultAdditive, null, 2)); | |
| // Test Case B: Breaking Removal & Type Change | |
| console.log("\n--- TEST CASE B: Breaking Removal & Type Change ---"); | |
| const diffBreaking = await runOpenApiDiff(v1, v2Breaking); | |
| const factsBreaking = normalizeDiff(diffBreaking, v1, v2Breaking); | |
| console.log( | |
| "FULL OPENAPI DIFF:", | |
| JSON.stringify(diffBreaking, null, 2) | |
| ); | |
| const payloadBreaking = { | |
| apiName: "User Service API", | |
| oldVersion: "1.0.0", | |
| newVersion: "2.0.0", | |
| changesCount: factsBreaking.totalBreaking, | |
| changes: factsBreaking.allChanges | |
| }; | |
| console.log("Breaking Normalized Payload:", JSON.stringify(payloadBreaking, null, 2)); | |
| console.log("Triggering Lamatic Workflow for Breaking Test Case..."); | |
| const resultBreaking = await triggerWorkflowAndPoll(payloadBreaking); | |
| console.log("Breaking Test Result Output:", JSON.stringify(resultBreaking, null, 2)); | |
| const assert = require('node:assert/strict'); | |
| async function runMatrixTests() { | |
| console.log("=========================================="); | |
| console.log("STEP 1: Verify Production Normalization via sentinel.ts"); | |
| console.log("=========================================="); | |
| const mockBreakingDiff = { | |
| breakingDifferences: [ | |
| { | |
| code: "response.body.scope.add", | |
| entity: "response.body.scope", | |
| sourceSpecEntityDetails: [{ location: "paths./users/{id}.get" }], | |
| details: { | |
| differenceSchema: { | |
| anyOf: [ | |
| { required: ["name"] }, | |
| { required: ["email"] } | |
| ] | |
| } | |
| } | |
| } | |
| ] | |
| }; | |
| const normalizedMock = normalizeDiff(mockBreakingDiff); | |
| console.log("Verified Mock Breaking Changes Normalization Output:"); | |
| console.log(JSON.stringify(normalizedMock, null, 2)); | |
| console.log("\n=========================================="); | |
| console.log("STEP 2: Matrix Execution"); | |
| console.log("=========================================="); | |
| // Test Case A: Additive (Non-breaking Baseline) | |
| console.log("\n--- TEST CASE A: Additive (Non-Breaking) ---"); | |
| const diffAdditive = await runOpenApiDiff(v1, v2Additive); | |
| const factsAdditive = normalizeDiff(diffAdditive, v1, v2Additive); | |
| assert.equal(factsAdditive.totalBreaking, 0); | |
| assert.ok( | |
| factsAdditive.nonBreakingChanges.some( | |
| (change) => change.field === 'full_name' && change.action === 'add' | |
| ) | |
| ); | |
| const payloadAdditive = { | |
| apiName: "User Service API", | |
| oldVersion: "1.0.0", | |
| newVersion: "2.0.0", | |
| changesCount: factsAdditive.totalBreaking, | |
| changes: factsAdditive.allChanges | |
| }; | |
| console.log("Additive Normalized Payload:", JSON.stringify(payloadAdditive, null, 2)); | |
| console.log("Triggering Lamatic Workflow for Additive Test Case..."); | |
| const resultAdditive = await triggerWorkflowAndPoll(payloadAdditive); | |
| assert.ok(resultAdditive, 'Additive workflow returned no result'); | |
| console.log("Additive Test Result Output:", JSON.stringify(resultAdditive, null, 2)); | |
| // Test Case B: Breaking Removal & Type Change | |
| console.log("\n--- TEST CASE B: Breaking Removal & Type Change ---"); | |
| const diffBreaking = await runOpenApiDiff(v1, v2Breaking); | |
| const factsBreaking = normalizeDiff(diffBreaking, v1, v2Breaking); | |
| assert.equal(factsBreaking.totalBreaking, 3); | |
| console.log( | |
| "FULL OPENAPI DIFF:", | |
| JSON.stringify(diffBreaking, null, 2) | |
| ); | |
| const payloadBreaking = { | |
| apiName: "User Service API", | |
| oldVersion: "1.0.0", | |
| newVersion: "2.0.0", | |
| changesCount: factsBreaking.totalBreaking, | |
| changes: factsBreaking.allChanges | |
| }; | |
| console.log("Breaking Normalized Payload:", JSON.stringify(payloadBreaking, null, 2)); | |
| console.log("Triggering Lamatic Workflow for Breaking Test Case..."); | |
| const resultBreaking = await triggerWorkflowAndPoll(payloadBreaking); | |
| assert.ok(resultBreaking, 'Breaking workflow returned no result'); | |
| console.log("Breaking Test Result Output:", JSON.stringify(resultBreaking, null, 2)); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/apps/test-orchestrate.js` around lines 164 -
231, Update runMatrixTests to add deterministic assertions for each normalized
payload, validating the expected additive and breaking change counts and
normalized change contents. After each triggerWorkflowAndPoll call, assert that
the returned workflow result is present; throw or otherwise fail explicitly when
it is null or absent. Do not assert workflow-generated or LLM prose.
| ### Test A — Additive (non-breaking) | ||
|
|
||
| **Input:** Base spec has `GET /users`, target spec adds `POST /users`. | ||
|
|
||
| **Expected result:** | ||
| - `changesCount: 1` | ||
| - `breakingChangesCount: 0` | ||
| - `deploymentRisk: LOW` | ||
| - One non-breaking change: `POST /users` added |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Mission directive: Align Test A with the harness.
The README says Test A adds POST /users. apps/test-orchestrate.js lines 27-45 instead add full_name to GET /users/{id}.
The documented changesCount: 1 also conflicts with the harness payload, which assigns changesCount from factsAdditive.totalBreaking. That value should be 0 for an additive case.
Choose one scenario. Update the fixture or this expected-result section consistently.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/README.md` around lines 186 - 194, Align the
Test A scenario in the README with the harness behavior in
apps/test-orchestrate.js: either document the existing full_name addition to GET
/users/{id} or update the fixture to add POST /users. Ensure the expected
changesCount matches factsAdditive.totalBreaking, using 0 for this additive
case, while keeping breakingChangesCount and deploymentRisk consistent.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/api-schema-drift-sentinel/.env.example`:
- Around line 2-4: Update the variable ordering in .env.example so
LAMATIC_API_URL and LAMATIC_DRIFT_FLOW_ID precede LAMATIC_PROJECT_ID, then
ensure the file ends with a final newline.
In `@kits/api-schema-drift-sentinel/agent.md`:
- Around line 3-10: Add blank lines before and after each of the four reported
Markdown headings in kits/api-schema-drift-sentinel/agent.md#L3-L10. Apply the
same heading-spacing correction in the template/source at
kits/api-schema-drift-sentinel/constitutions/default.md#L3-L15, then regenerate
that file so the generated output matches the source.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: be904b5d-5c86-4da1-8280-ef4cef986642
📒 Files selected for processing (4)
kits/api-schema-drift-sentinel/.env.examplekits/api-schema-drift-sentinel/agent.mdkits/api-schema-drift-sentinel/constitutions/default.mdkits/api-schema-drift-sentinel/flows/analyze-schema-drift.ts
| LAMATIC_PROJECT_ID=your_project_id_here | ||
| LAMATIC_API_URL=https://api.lamatic.ai | ||
| LAMATIC_DRIFT_FLOW_ID=your_id No newline at end of file |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Mission requirement: clear the environment-file lint findings.
dotenv-linter reports that LAMATIC_API_URL and LAMATIC_DRIFT_FLOW_ID must appear before LAMATIC_PROJECT_ID. Move LAMATIC_PROJECT_ID to the end and add the missing final newline.
Proposed ordering
LAMATIC_API_KEY=your_lamatic_api_key_here
-LAMATIC_PROJECT_ID=your_project_id_here
LAMATIC_API_URL=https://api.lamatic.ai
LAMATIC_DRIFT_FLOW_ID=your_id
+LAMATIC_PROJECT_ID=your_project_id_here📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| LAMATIC_PROJECT_ID=your_project_id_here | |
| LAMATIC_API_URL=https://api.lamatic.ai | |
| LAMATIC_DRIFT_FLOW_ID=your_id | |
| LAMATIC_API_KEY=your_lamatic_api_key_here | |
| LAMATIC_API_URL=https://api.lamatic.ai | |
| LAMATIC_DRIFT_FLOW_ID=your_id | |
| LAMATIC_PROJECT_ID=your_project_id_here |
🧰 Tools
🪛 dotenv-linter (4.0.0)
[warning] 3-3: [UnorderedKey] The LAMATIC_API_URL key should go before the LAMATIC_PROJECT_ID key
(UnorderedKey)
[warning] 4-4: [EndingBlankLine] No blank line at the end of the file
(EndingBlankLine)
[warning] 4-4: [UnorderedKey] The LAMATIC_DRIFT_FLOW_ID key should go before the LAMATIC_PROJECT_ID key
(UnorderedKey)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/.env.example` around lines 2 - 4, Update the
variable ordering in .env.example so LAMATIC_API_URL and LAMATIC_DRIFT_FLOW_ID
precede LAMATIC_PROJECT_ID, then ensure the file ends with a final newline.
Source: Linters/SAST tools
| ## Overview | ||
| API Schema Drift Sentinel detects breaking changes between OpenAPI specifications and produces grounded migration guidance. | ||
|
|
||
| ## Purpose | ||
| The goal of this kit is to prevent breaking API drift by combining deterministic AST diffing with an AI reasoning layer. | ||
|
|
||
| ## Flows | ||
| ### 1. Analyze Schema Drift |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Clear the shared Markdown heading-spacing findings at the correct source.
Both files place headings directly next to following content. Add blank lines around the headings in kits/api-schema-drift-sentinel/agent.md. Update the template/source for kits/api-schema-drift-sentinel/constitutions/default.md, then regenerate it.
kits/api-schema-drift-sentinel/agent.md#L3-L10: add blank lines around the four reported headings.kits/api-schema-drift-sentinel/constitutions/default.md#L3-L15: apply the spacing fix in the template/source and regenerate the file.
Based on learnings: kits/*/constitutions/default.md is a templated/auto-generated file.
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 3-3: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 6-6: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 9-9: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
[warning] 10-10: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Above
(MD022, blanks-around-headings)
[warning] 10-10: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below
(MD022, blanks-around-headings)
📍 Affects 2 files
kits/api-schema-drift-sentinel/agent.md#L3-L10(this comment)kits/api-schema-drift-sentinel/constitutions/default.md#L3-L15
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/api-schema-drift-sentinel/agent.md` around lines 3 - 10, Add blank lines
before and after each of the four reported Markdown headings in
kits/api-schema-drift-sentinel/agent.md#L3-L10. Apply the same heading-spacing
correction in the template/source at
kits/api-schema-drift-sentinel/constitutions/default.md#L3-L15, then regenerate
that file so the generated output matches the source.
Sources: Learnings, Linters/SAST tools
…, and test alignment
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
Studio Runtime Validation (Phase 2)✅ Studio validation passed. The kit loaded successfully in Lamatic Studio. This PR is ready for final review and merge. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/api-schema-drift-sentinel/apps/app/api/analyze-drift/route.ts`:
- Around line 26-35: Update the size check near MAX_SIZE to measure strA and
strB as UTF-8 byte lengths using TextEncoder rather than String.length. Compare
the encoded byte counts against the existing 2 MiB limit and preserve the
current 400 response for oversized payloads.
- Around line 19-24: Update the type validation in the analyze-drift route to
accept strings or only non-null, non-array objects for specA and specB. Reject
arrays and other invalid object-like values with the existing 400 response
before they enter the diff pipeline.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 217144e0-52a1-4d20-aa36-cc405809d37f
📒 Files selected for processing (6)
kits/api-schema-drift-sentinel/.gitignorekits/api-schema-drift-sentinel/apps/actions/orchestrate.tskits/api-schema-drift-sentinel/apps/app/api/analyze-drift/route.tskits/api-schema-drift-sentinel/apps/app/globals.csskits/api-schema-drift-sentinel/apps/test-orchestrate.jskits/api-schema-drift-sentinel/prompts/analyze-schema-drift_llm-node_system.md
💤 Files with no reviewable changes (1)
- kits/api-schema-drift-sentinel/apps/app/globals.css
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
@coderabbitai review |
|
|
@coderabbitai review |
|
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
@mohamad-shafeez phase 2 is failing also there are lots of coderabbit comments in PR |
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
@coderabbitai review |
|
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
|
@mohamad-shafeez there are some comments left by coderabbit please resolve them then we can merge |
2 participants
Overview
API Schema Drift Sentinel detects breaking changes between OpenAPI specifications and produces grounded migration guidance.
Problem
API schema changes can silently break downstream clients when response fields are removed, parameter types change, or other incompatible changes are introduced.
Architecture
The kit uses a two-layer pipeline:
openapi-difffor deterministic structural comparisonBreaking-change counts and deployment risk are derived from the deterministic change classification rather than the LLM output.
Verification
The application was verified with:
Secrets such as
.env.local,node_modules, and.nextare excluded from the repository.openapi-diff.POST /api/analyze-driftendpoint with input validation, size limits, workflow integration, response parsing, and deterministic fallback handling.analyzeSchemaDriftserver action.Analyze Schema Driftflow with trigger, dynamic LLM, and response nodes. The trigger passes confirmed schema-drift facts to the LLM node. The response node returns the structured analysis.