-
Notifications
You must be signed in to change notification settings - Fork 24
feat: bump supercode-cli version to 0.1.86 and enhance hallucination guard: #228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,6 +69,9 @@ export async function runAgent( | |
| // Track tool call repetition — same tool + same args 3+ times signals a loop | ||
| const toolCallHistory: Array<{ toolName: string; argsKey: string }> = [] | ||
| let stopForRepetition = false | ||
| // Hallucination guard: detect when model claims actions but made zero tool calls | ||
| let lastStepHadToolCalls = false | ||
| let lastStepHadActionClaim = false | ||
|
|
||
| const messages = buildMessages(opts) | ||
|
|
||
|
|
@@ -99,6 +102,20 @@ export async function runAgent( | |
| }], | ||
| } | ||
| } | ||
| // Hallucination guard: previous step claimed actions but made no tool calls | ||
| if (lastStepHadActionClaim) { | ||
| lastStepHadActionClaim = false | ||
| return { | ||
| messages: [{ | ||
| role: "system" as const, | ||
| content: | ||
| "SYSTEM NOTICE: Your previous response claimed to have made changes " + | ||
| "(e.g., wrote, edited, refactored) but you did not call any tools. " + | ||
| "Do not describe changes — actually execute them by calling the appropriate " + | ||
| "tool (edit_file, write_file, run_command, etc.). Then summarize what you did.", | ||
| }], | ||
| } | ||
| } | ||
|
Comment on lines
+105
to
+118
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
rg -n '"ai"\s*:' apps/supercode-cli/server/package.json
find apps/supercode-cli/server -path '*/node_modules/ai/*' -type f \
\( -name '*.d.ts' -o -name '*.md' \) -print0 |
xargs -0 -r rg -n 'prepareStep|PrepareStepResult|messages'Repository: yashdev9274/supercli Length of output: 183 🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
printf '--- runner outline ---\n'
ast-grep outline apps/supercode-cli/server/src/agent/runner.ts || true
printf '\n--- runner relevant lines ---\n'
cat -n apps/supercode-cli/server/src/agent/runner.ts | sed -n '1,220p'
printf '\n--- lockfile ai package if present ---\n'
git ls-files | rg '(^|/)bun\.lockb$|(^|/)package-lock\.json$|(^|/)pnpm-lock\.yaml$|(^|/)yarn\.lock$' || true
if [ -f apps/supercode-cli/server/package-lock.json ]; then
jq -r '"ai"\? | select(.)' apps/supercode-cli/server/package-lock.json || true
fiRepository: yashdev9274/supercli Length of output: 249 🌐 Web query:
💡 Result: In AI SDK version 6.0.x (including 6.0.195), the Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "--- runner relevant lines ---"
cat -n apps/supercode-cli/server/src/agent/runner.ts | sed -n '1,220p'
echo
echo "--- package lock ai if present ---"
find apps/supercode-cli/server -maxdepth 3 \( -name 'package-lock.json' -o -name 'bun.lockb' -o -name 'pnpm-lock.yaml' -o -name 'yarn.lock' \) -print -exec sh -c 'echo "## $1"; case "$1" in *.json) jq -r ".packages.\"\".dependencies?.ai // \"\"" ;; *) rg -n "\"ai\"" "$1" || true ;; esac' sh {} \;Repository: yashdev9274/supercli Length of output: 10303 🌐 Web query:
💡 Result: In the Vercel AI SDK, the prepareStep callback is a function used to dynamically adjust settings such as the model, tool choices, active tools, instructions, and messages for each step in a multi-step generation loop [1][2][3]. The messages override logic in prepareStep has evolved in recent SDK versions (notably in AI SDK 7+). When you return a messages override from prepareStep, those messages now persist and serve as the base for all subsequent steps [4][2][3]. New assistant or tool response messages generated in the following steps are appended to this overridden message array [3]. The prepareStep function receives the following message-related parameters to help you manage state: - messages: Represents the current message state for the step. If you return an override here, this new array becomes the foundation for future steps [2][3]. - initialMessages: The original messages passed to the initial generateText or streamText call [2][5]. - responseMessages: The assistant and tool response messages accumulated from all previous steps [2][5]. Implementation Example: To maintain control over the message history (e.g., for compaction or cleaning), you can leverage these parameters [3]. If you want to replicate older behavior or ensure you are working from specific pieces of history, you can reconstruct the messages array manually inside your prepareStep function: const result = await streamText({ model: yourModel, tools: { weather }, prepareStep: ({ initialMessages, responseMessages }) => { // Example: Reconstruct messages to keep only specific parts, // or to perform compaction. return { messages: [...initialMessages,...responseMessages, // Add your logic to filter or modify messages here ], }; }, }); This pattern allows you to decide exactly what state is passed forward into the next iteration of the loop [3]. If you do not return a messages override, the SDK continues using the existing accumulated message state [2][3]. Citations:
🌐 Web query:
💡 Result: In the Vercel AI SDK, the prepareStep function allows you to customize the configuration of individual steps in a multi-step generation (e.g., when using maxSteps > 1) [1][2]. Regarding message overrides: 1. Current Behavior: If you return a messages override from prepareStep, those messages now carry forward as the base for subsequent steps [3][2]. This behavior was updated to be more intuitive, as previous versions of the SDK (such as AI SDK 6) only applied the override to the current step [3]. 2. Implementation Details: The prepareStep function receives the following relevant arguments: - initialMessages: The messages originally passed to streamText or generateText [4][2]. - responseMessages: The accumulated assistant and tool response messages so far [2]. - messages: The current full set of messages sent to the model for the current step [2]. 3. Maintaining Backward Compatibility (One-Step Overrides): If you need to maintain the older behavior where an override applies only to a single step, you can manually rebuild the messages within the prepareStep function by combining the initial and response messages, then applying your specific modification [3]. Example pattern for one-step-only modifications: prepareStep: ({ initialMessages, responseMessages }) => { return { messages: [...initialMessages,...responseMessages, // Add your one-step-only message changes here ], }; }, This approach ensures you have full control over the message history while leveraging the SDK's updated state handling [3][5]. Citations:
🌐 Web query:
💡 Result: In the context of the Vercel AI SDK (specifically the Citations:
🌐 Web query:
💡 Result: Version 6.0.195 of the ai npm package is a release of the Vercel AI SDK, an open-source TypeScript toolkit for building AI-powered applications [1][2]. This version was published on June 2, 2026 [1]. The source code for this version is hosted within the official Vercel AI SDK repository on GitHub (vercel/ai) [1][3]. The specific changes for version 6.0.195 involved a dependency update to Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
curl -fsSL -o "$tmp/ai-6.0.195.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz
tar -xzf "$tmp/ai-6.0.195.tgz" -C "$tmp"
echo "--- package version ---"
node -e 'const p=require("$tmp/package/package.json"); console.log(p.version)' sh "$tmp"
echo
echo "--- stream-text prepare step relevant snippets ---"
rg -n "prepareStep|stepMessages|messages.*override|initialMessages|responseMessages" "$tmp/package/src" "$tmp/package/dist"
echo
echo "--- stream-text.ts excerpt ---"
cat -n "$tmp/package/src/generate-text/stream-text.ts" | sed -n '1,260p'
echo
echo "--- generate-text.ts excerpt ---"
cat -n "$tmp/package/src/generate-text/generate-text.ts" | sed -n '1,260p'
echo
echo "--- prepare-step.ts excerpt ---"
cat -n "$tmp/package/src/generate-text/prepare-step.ts" | sed -n '1,260p'
echo
echo "--- build-messages excerpt if present ---"
find "$tmp/package/src" -name 'build-messages.ts' -o -name '*message*' | xargs -r rg -n "function|export|initialMessages|responseMessages" -C 3Repository: yashdev9274/supercli Length of output: 1052 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
curl -fsSL -o "$tmp/ai-6.0.195.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz
version="$(node -e "const p=require('./package/package.json'); console.log(p.version)" "$tmp")"
echo "--- package version: $version"
echo
echo "--- prepare step implementation ---"
cat -n "$tmp/package/src/generate-text/prepare-step.ts" | sed -n '1,280p'
echo
echo "--- stream text relevant message construction ---"
cat -n "$tmp/package/src/generate-text/stream-text.ts" | sed -n '1,340p'
echo
echo "--- generate text relevant message construction ---"
cat -n "$tmp/package/src/generate-text/generate-text.ts" | sed -n '1,340p'
echo
echo "--- exported types around PrepareStepResult if present ---"
rg -n "PrepareStepResult|MessagesPrepareStepResult|messages:.*override|stepMessages" "$tmp/package/src/generate-text" "$tmp/package/src/generate-text/prepare-step.ts" -C 2Repository: yashdev9274/supercli Length of output: 1025 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
pkg="$tmp/package"
mkdir -p "$pkg"
curl -fsSL -o "$pkg/ai-6.0.195.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz
tar -xzf "$pkg/ai-6.0.195.tgz" -C "$pkg"
echo "--- package version ---"
node -e 'const p=require(process.argv[1]); console.log(p.version)' "$pkg/package/package.json"
echo
echo "--- prepare-step.ts ---"
cat -n "$pkg/package/src/generate-text/prepare-step.ts" | sed -n '1,280p'
echo
echo "--- stream-text.ts ---"
cat -n "$pkg/package/src/generate-text/stream-text.ts" | sed -n '1,340p'
echo
echo "--- generate-text.ts ---"
cat -n "$pkg/package/src/generate-text/generate-text.ts" | sed -n '1,340p'
echo
echo "--- build-messages.ts candidates ---"
find "$pkg/package/src" -maxdepth 2 -type f | sort | xargs rg -n "build.*messages|initialMessages|responseMessages|Messages" -C 2 | head -n 220Repository: yashdev9274/supercli Length of output: 50376 🏁 Script executed: #!/bin/bash
set -euo pipefail
pkg="$(mktemp -d)"
tar -xzf <(curl -fsSL https://registry.npmjs.org/ai/-/ai-6.0.195.tgz) -C "$pkg"
echo "--- streamText loop/message construction snippets ---"
rg -n "prepareStep|stepMessages|messages =|responseMessages|previousStep|current messages|convertToLanguageModelPrompt|await .*prepareStep|const .*messages|response-messages" "$pkg/package/src/generate-text/stream-text.ts" -C 4
echo
echo "--- generated exact prepareStep call loop ---"
python3 - <<'PY'
from pathlib import Path
p = Path("$pkg/package/src/generate-text/stream-text.ts").with_name(Path("$pkg/package/src/generate-text/stream-text.ts").name)
# use shell var by reading literal file pattern passed via heredoc? not needed
PY
sed -n '480,720p' "$pkg/package/src/generate-text/stream-text.ts"Repository: yashdev9274/supercli Length of output: 345 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
pkg="$tmp/package"
curl -fsSL -o "$pkg/ai.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz
tar -xzf "$pkg/ai.tgz" -C "$pkg"
echo "--- streamText.js/TS exact message loop around step generation ---"
sed -n '560,740p' "$pkg/package/src/generate-text/stream-text.ts"
echo
echo "--- generateText.js exact message loop around step generation ---"
sed -n '520,720p' "$pkg/package/src/generate-text/generate-text.ts"Repository: yashdev9274/supercli Length of output: 209 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
curl -L -o "$tmp/ai.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz || { echo "curl failed"; exit 1; }
tar -xzf "$tmp/ai.tgz" -C "$tmp"
pkg="$tmp/package"
echo "--- streamText line count ---"
wc -l "$pkg/package/src/generate-text/stream-text.ts"
echo "--- streamText exact message loop snippets ---"
rg -n "prepareStep|messages|responseMessages|previousStep|asArray" "$pkg/package/src/generate-text/stream-text.ts" -C 5
echo
echo "--- surrounding streamText lines 560-710 ---"
cat -n "$pkg/package/src/generate-text/stream-text.ts" | sed -n '560,710p'
echo
echo "--- surrounding generateText lines 520-710 ---"
cat -n "$pkg/package/src/generate-text/generate-text.ts" | sed -n '520,710p'Repository: yashdev9274/supercli Length of output: 605 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
curl -L -o "$tmp/ai.tgz" https://registry.npmjs.org/ai/-/ai-6.0.195.tgz || { echo "curl failed"; exit 1; }
tar -xzf "$tmp/ai.tgz" -C "$tmp"
pkg="$tmp/package/package"
echo "--- top dir ---"
find "$pkg" -maxdepth 3 -type f | sort | head -n 40
echo
echo "--- streamText if present ---"
find "$pkg" -name 'stream-text.ts' -type f -print
fi
file="$(find "$pkg" -path '*generate-text/stream-text.ts' -type f | head -n 1)"
if [ -z "$file" ]; then
file="$(find "$pkg" -name 'stream-text.ts' -type f | head -n 1)"
fi
if [ -n "$file" ]; then
echo "--- streamText line count ---"
wc -l "$file"
echo
echo "--- streamText prepareStep and message construction ---"
rg -n "prepareStep|messages|responseMessages|previousStep|asArray|convertToLanguageModelPrompt" "$file" -C 6
echo
echo "--- streamText lines 560-710 ---"
cat -n "$file" | sed -n '560,710p'
fi
echo
echo "--- generateText if present ---"
file2="$(find "$pkg" -path '*generate-text/generate-text.ts' -type f | head -n 1)"
if [ -z "$file2" ]; then
file2="$(find "$pkg" -name 'generate-text.ts' -type f | head -n 1)"
fi
if [ -n "$file2" ]; then
echo "--- generateText lines 520-710 ---"
cat -n "$file2" | sed -n '520,710p'
fiRepository: yashdev9274/supercli Length of output: 562 🏁 Script executed: #!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
tar -xzf <(curl -L https://registry.npmjs.org/ai/-/ai-6.0.195.tgz) -C "$tmp"
pkg="$tmp/package"
echo "--- package files with generate ---"
find "$pkg" -name 'stream-text.ts' -o -name 'generate-text.ts' -o -name 'step-result.ts' | sort
echo
echo "--- stream-text.ts -- streamText -- prepareStep -- messages loop"
rg -n "prepareStep|messages|responseMessages|previousStep|asArray|stepMessages|responseMessagesForStep|current.*messages" "$pkg/src/generate-text/stream-text.ts" "$pkg/src/generate-text/generate-text.ts" -C 5
echo
echo "--- stream-text.ts lines 580-730 ---"
cat -n "$pkg/src/generate-text/stream-text.ts" | sed -n '580,730p'
echo
echo "--- generate-text.ts lines 540-720 ---"
cat -n "$pkg/src/generate-text/generate-text.ts" | sed -n '540,720p'Repository: yashdev9274/supercli Length of output: 345 🌐 Web query:
💡 Result: In the Vercel AI SDK, the Citations:
Preserve the conversation when injecting the notice.
🤖 Prompt for AI Agents |
||
| if (seenStepResults.length === 0) return undefined | ||
| const allEmpty = seenStepResults.every((r) => isEmptyToolResult(r.result)) | ||
| if (!allEmpty) return undefined | ||
|
|
@@ -187,6 +204,13 @@ export async function runAgent( | |
| toolCallHistory.splice(0, toolCallHistory.length - 12) | ||
| } | ||
| } | ||
| // Track whether this step had tool calls for the hallucination guard | ||
| lastStepHadToolCalls = (event.toolCalls?.length ?? 0) > 0 | ||
| lastStepHadActionClaim = false | ||
| if (!lastStepHadToolCalls && event.text) { | ||
| const actionClaimRe = /\b(wrote|updated|added|created|ran|executed|fixed|refactored|removed|deleted|installed|modified|edited|applied|saved|generated|wired|hooked)\b/i | ||
| lastStepHadActionClaim = actionClaimRe.test(event.text) | ||
| } | ||
|
Comment on lines
+207
to
+213
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Restrict action detection to affirmative completion claims. The verb-only regex also matches negated or hypothetical text such as “I could not run the tests” or “I did not modify the file,” causing a false corrective notice and an unnecessary extra model step. Match affirmative claim patterns and explicitly exclude negations. 🤖 Prompt for AI Agents |
||
| opts.onStepFinish?.(event) | ||
| }, | ||
| }) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Align both prompt templates around a complete async migration.
Both rules remove callbacks without explicitly requiring an
async/Promise contract or migrating callers, so agents can produce partial refactors that break existing call sites.apps/supercode-cli/server/src/agent/prompts/build.txt#L36-L37: requireasync/Promise returns and caller updates.apps/supercode-cli/server/src/agent/prompts/general.txt#L21-L22: apply the same complete migration guidance.📍 Affects 2 files
apps/supercode-cli/server/src/agent/prompts/build.txt#L36-L37(this comment)apps/supercode-cli/server/src/agent/prompts/general.txt#L21-L22🤖 Prompt for AI Agents