-
Notifications
You must be signed in to change notification settings - Fork 22
test: command hints, discover-and-add-mcps, sql.analyze success semantics #440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
anandgupta42
wants to merge
1
commit into
main
Choose a base branch
from
test/hourly-20260324-1500
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
84 changes: 84 additions & 0 deletions
84
packages/opencode/test/altimate/sql-analyze-format.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| import { describe, test, expect } from "bun:test" | ||
| import { Dispatcher } from "../../src/altimate/native" | ||
| import { registerAllSql } from "../../src/altimate/native/sql/register" | ||
| import type { SqlAnalyzeResult } from "../../src/altimate/native/types" | ||
|
|
||
| // Ensure sql.analyze is registered | ||
| registerAllSql() | ||
|
|
||
| // --------------------------------------------------------------------------- | ||
| // sql.analyze Dispatcher — success semantics and result shape | ||
| // | ||
| // The AI-5975 fix changed success semantics: finding issues IS a successful | ||
| // analysis (success:true). Previously it returned success:false when issues | ||
| // were found, causing ~4,000 false "unknown error" telemetry entries per day. | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| describe("sql.analyze: success semantics (AI-5975 regression)", () => { | ||
| test("query with lint issues still returns success:true", async () => { | ||
| // SELECT * is a known lint trigger — must still be a successful analysis | ||
| const result = await Dispatcher.call("sql.analyze", { | ||
| sql: "SELECT * FROM users", | ||
| dialect: "snowflake", | ||
| }) as SqlAnalyzeResult | ||
| // KEY INVARIANT: finding issues is a SUCCESSFUL analysis | ||
| expect(result.success).toBe(true) | ||
| // Verify issues were actually found (not a vacuous pass) | ||
| expect(result.issue_count).toBeGreaterThan(0) | ||
| expect(result.confidence).toBe("high") | ||
| }) | ||
|
|
||
| test("issue_count matches issues array length", async () => { | ||
| const result = await Dispatcher.call("sql.analyze", { | ||
| sql: "SELECT * FROM orders JOIN customers", | ||
| dialect: "snowflake", | ||
| }) as SqlAnalyzeResult | ||
| expect(result.issues.length).toBeGreaterThan(0) | ||
| expect(result.issue_count).toBe(result.issues.length) | ||
| }) | ||
| }) | ||
|
|
||
| describe("sql.analyze: issue structure", () => { | ||
| test("lint issues have required fields", async () => { | ||
| const result = await Dispatcher.call("sql.analyze", { | ||
| sql: "SELECT * FROM users", | ||
| dialect: "snowflake", | ||
| }) as SqlAnalyzeResult | ||
| const lintIssues = result.issues.filter((i) => i.type === "lint") | ||
| // Guard against vacuous pass — SELECT * must produce lint findings | ||
| expect(lintIssues.length).toBeGreaterThan(0) | ||
| for (const issue of lintIssues) { | ||
| expect(issue.severity).toBeDefined() | ||
| expect(issue.message).toBeDefined() | ||
| expect(typeof issue.recommendation).toBe("string") | ||
| expect(issue.confidence).toBe("high") | ||
| } | ||
| }) | ||
|
|
||
| test("issue types are limited to lint, semantic, safety", async () => { | ||
| const result = await Dispatcher.call("sql.analyze", { | ||
| sql: "SELECT * FROM users WHERE 1=1", | ||
| dialect: "snowflake", | ||
| }) as SqlAnalyzeResult | ||
| expect(result.issues.length).toBeGreaterThan(0) | ||
| const validTypes = ["lint", "semantic", "safety"] | ||
| for (const issue of result.issues) { | ||
| expect(validTypes).toContain(issue.type) | ||
| } | ||
| }) | ||
| }) | ||
|
|
||
| describe("sql.analyze: result shape", () => { | ||
| test("successful result has all required properties", async () => { | ||
| const result = await Dispatcher.call("sql.analyze", { | ||
| sql: "SELECT 1 LIMIT 1", | ||
| }) as SqlAnalyzeResult | ||
| expect(result).toHaveProperty("success") | ||
| expect(result).toHaveProperty("issues") | ||
| expect(result).toHaveProperty("issue_count") | ||
| expect(result).toHaveProperty("confidence") | ||
| expect(result).toHaveProperty("confidence_factors") | ||
| expect(Array.isArray(result.issues)).toBe(true) | ||
| expect(Array.isArray(result.confidence_factors)).toBe(true) | ||
| }) | ||
| }) | ||
94 changes: 94 additions & 0 deletions
94
packages/opencode/test/command/hints-discover-mcps.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| import { describe, test, expect } from "bun:test" | ||
| import { Command } from "../../src/command/index" | ||
| import { Instance } from "../../src/project/instance" | ||
| import { tmpdir } from "../fixture/fixture" | ||
|
|
||
| // --------------------------------------------------------------------------- | ||
| // Helpers | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| async function withInstance(fn: () => Promise<void>) { | ||
| await using tmp = await tmpdir({ git: true }) | ||
| await Instance.provide({ directory: tmp.path, fn }) | ||
| } | ||
|
|
||
| // --------------------------------------------------------------------------- | ||
| // Command.hints() — template hint extraction | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| describe("Command.hints()", () => { | ||
| test("extracts and sorts placeholders lexicographically", () => { | ||
| // Lexicographic sort means $10 sorts before $2 — this documents | ||
| // the actual contract (not numeric ordering) | ||
| const result = Command.hints("Use $10 then $2 and $1") | ||
| expect(result).toEqual(["$1", "$10", "$2"]) | ||
| }) | ||
|
|
||
| test("extracts $ARGUMENTS placeholder", () => { | ||
| const result = Command.hints("Run with $ARGUMENTS") | ||
| expect(result).toEqual(["$ARGUMENTS"]) | ||
| }) | ||
|
|
||
| test("extracts both numbered and $ARGUMENTS", () => { | ||
| const result = Command.hints("Use $1 and $ARGUMENTS") | ||
| expect(result).toEqual(["$1", "$ARGUMENTS"]) | ||
| }) | ||
|
|
||
| test("deduplicates repeated numbered placeholders", () => { | ||
| const result = Command.hints("$1 and $1 then $2") | ||
| expect(result).toEqual(["$1", "$2"]) | ||
| }) | ||
|
|
||
| test("returns empty array for template with no placeholders", () => { | ||
| const result = Command.hints("No placeholders here") | ||
| expect(result).toEqual([]) | ||
| }) | ||
|
|
||
| test("includes $0 as a valid placeholder", () => { | ||
| const result = Command.hints("$0 $1") | ||
| expect(result).toEqual(["$0", "$1"]) | ||
| }) | ||
| }) | ||
|
|
||
| // --------------------------------------------------------------------------- | ||
| // discover-and-add-mcps builtin command (#409) | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| describe("discover-and-add-mcps builtin command", () => { | ||
| test("is registered in Command.Default constants", () => { | ||
| expect(Command.Default.DISCOVER_MCPS).toBe("discover-and-add-mcps") | ||
| }) | ||
|
|
||
| test("is present in command list", async () => { | ||
| await withInstance(async () => { | ||
| const commands = await Command.list() | ||
| const names = commands.map((c) => c.name) | ||
| expect(names).toContain("discover-and-add-mcps") | ||
| }) | ||
| }) | ||
|
|
||
| test("has correct metadata", async () => { | ||
| await withInstance(async () => { | ||
| const cmd = await Command.get("discover-and-add-mcps") | ||
| expect(cmd).toBeDefined() | ||
| expect(cmd.name).toBe("discover-and-add-mcps") | ||
| expect(cmd.source).toBe("command") | ||
| expect(cmd.description).toContain("MCP") | ||
| }) | ||
| }) | ||
|
|
||
| test("template references mcp_discover tool", async () => { | ||
| await withInstance(async () => { | ||
| const cmd = await Command.get("discover-and-add-mcps") | ||
| const template = await cmd.template | ||
| expect(template).toContain("mcp_discover") | ||
| }) | ||
| }) | ||
|
|
||
| test("is not a subtask", async () => { | ||
| await withInstance(async () => { | ||
| const cmd = await Command.get("discover-and-add-mcps") | ||
| expect(cmd.subtask).toBeUndefined() | ||
| }) | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assert success-path explicitly in the “successful result” test.
The current assertions validate shape only; the failure path also has the same top-level shape. Add an explicit success assertion so this test enforces what its name promises.
✅ Suggested patch
test("successful result has all required properties", async () => { const result = await Dispatcher.call("sql.analyze", { sql: "SELECT 1 LIMIT 1", }) as SqlAnalyzeResult + expect(result.success).toBe(true) + expect(result.error).toBeUndefined() expect(result).toHaveProperty("success") expect(result).toHaveProperty("issues") expect(result).toHaveProperty("issue_count") expect(result).toHaveProperty("confidence") expect(result).toHaveProperty("confidence_factors") expect(Array.isArray(result.issues)).toBe(true) expect(Array.isArray(result.confidence_factors)).toBe(true) })📝 Committable suggestion
🤖 Prompt for AI Agents