From 09112e7e870b26c47f94f287b712b15df5bca7df Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Mon, 1 Dec 2025 20:11:26 +0100 Subject: [PATCH 1/6] docs: add spec for suggestion hook --- .../design.md | 439 ++++++++++++++++++ .../proposal.md | 148 ++++++ .../specs/claude-agent-integration/spec.md | 196 ++++++++ .../specs/claude-plugin/spec.md | 76 +++ .../specs/cli-interface/spec.md | 134 ++++++ .../tasks.md | 414 +++++++++++++++++ 6 files changed, 1407 insertions(+) create mode 100644 openspec/changes/implement-intelligent-context-injection/design.md create mode 100644 openspec/changes/implement-intelligent-context-injection/proposal.md create mode 100644 openspec/changes/implement-intelligent-context-injection/specs/claude-agent-integration/spec.md create mode 100644 openspec/changes/implement-intelligent-context-injection/specs/claude-plugin/spec.md create mode 100644 openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md create mode 100644 openspec/changes/implement-intelligent-context-injection/tasks.md diff --git a/openspec/changes/implement-intelligent-context-injection/design.md b/openspec/changes/implement-intelligent-context-injection/design.md new file mode 100644 index 0000000..4e1ce55 --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/design.md @@ -0,0 +1,439 @@ +# Design: Intelligent Context Injection + +## Overview + +This document captures the architectural decisions and design rationale for replacing static context injection with an intelligent, LLM-powered system that dynamically selects relevant skills and tools based on user prompts. + +## Architecture + +### Component Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Claude Code Session │ +│ │ +│ User Prompt │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────────────────────┐ │ +│ │ UserPromptSubmit Hook (Bash) │ │ +│ │ - Reads user prompt from JSON │ │ +│ │ - Calls toolscript claude-suggest-context│ │ +│ │ - Parses JSON response │ │ +│ │ - Injects contextual suggestions │ │ +│ └──────────┬───────────────────────────┘ │ +│ │ │ +└─────────────┼─────────────────────────────────────────────┬──┘ + │ │ + ▼ │ + ┌───────────────────────────────────────┐ │ + │ toolscript context claude-usage- │ │ + │ suggestion (New CLI Command) │ │ + │ - Scans global/project/plugin skills │ │ + │ - Calls Claude SDK for selection │ │ + │ - Receives gateway URL via flag │ │ + │ - Searches MCP tools on gateway │ │ + │ - Formats context text │ │ + │ - Returns hook JSON response │ │ + └──────────┬────────────────────────────┘ │ + │ │ + ▼ │ + ┌────────────────────────────────┐ │ + │ Claude Agent SDK │ │ + │ - Model: "haiku" (user config)│ │ + │ - Config sources: user, project, local │ + │ - Tools: none │ │ + │ - Single turn execution │ │ + └────────────────────────────────┘ │ +``` + +### Data Flow + +#### 1. Session Start (Existing + Enhanced) + +``` +SessionStart Hook: +1. Start gateway on random port → background process +2. Save PID to ${TMPDIR}toolscript-gateway-${SESSION_ID}.pid +3. Save URL to ${TMPDIR}toolscript-gateway-${SESSION_ID}.url ← NEW +4. Export TOOLSCRIPT_GATEWAY_URL to CLAUDE_ENV_FILE +5. (Remove static context injection) ← MODIFIED +``` + +#### 2. User Prompt Submit (New) + +``` +UserPromptSubmit Hook: +1. Receive JSON: { prompt: "...", cwd: "...", session_id: "..." } +2. Read gateway URL from ${TMPDIR}toolscript-gateway-${SESSION_ID}.url +3. If gateway URL missing/empty: Exit successfully (no toolscript without gateway) +4. Call: toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL" +5. Receive hook JSON response from CLI +6. Output CLI response as-is (pass-through) +``` + +#### 3. CLI Command Execution + +``` +toolscript context claude-usage-suggestion: +1. Scan skills from all sources: + - Global: ~/.claude/skills/*/SKILL.md + - Project: .claude/skills/*/SKILL.md + - Plugins: Load installed_plugins.json and scan {installPath}/skills/*/SKILL.md + - Extract YAML frontmatter description from each + - Build unified skill list: [{name, description, source}, ...] +2. Create Claude Agent SDK query: + - Model: "haiku" (uses user's configured model) + - Prompt: Template with user prompt + skill list + - Options: + * settingSources: ["user", "project", "local"] + * systemPrompt: (custom, not Claude Code's) + * allowedTools: [] + * Single iteration (for await one response) +4. Parse LLM response (JSON with skills and toolQueries arrays) +5. If --gateway-url provided: + - For each toolQuery: + - Search gateway for matching tools + - Collect tool names and descriptions +6. Format context text: + - Skills: Instruct to use Skill(name) for each + - Tools: Instruct to use toolscript skill with tool names + - Combined: Unified message if both present +7. Wrap in hook JSON: + - If context: `{"hookSpecificOutput":{"additionalContext":"formatted text"}}` + - If no context: `{}` +8. Output JSON to stdout +9. Exit +``` + +## Key Design Decisions + +### Decision 1: UserPromptSubmit vs SessionStart + +**Choice**: Use UserPromptSubmit hook instead of enhancing SessionStart + +**Rationale**: +- SessionStart runs before any user prompt, so it can't be context-aware +- UserPromptSubmit receives the actual user prompt as input +- Allows adaptation to changing user needs within a session +- Can skip injection for irrelevant prompts (e.g., "hello", "thanks") + +**Trade-offs**: +- Adds latency to every user prompt (~200-500ms) +- Increases API costs (small: ~$0.0005 per prompt) +- More complex error handling required + +### Decision 2: CLI Command vs Inline Bash Logic + +**Choice**: Build `claude-suggest-context` as proper CLI command, not inline Bash + +**Rationale**: +- TypeScript provides better error handling and type safety +- Agent SDK is a TypeScript library (better integration) +- CLI command is testable independently +- Can reuse existing config loading, path utilities +- Follows project conventions (commands in src/cli/commands/) + +**Trade-offs**: +- More code to write and maintain +- Slightly slower startup than pure Bash +- Additional compilation during development + +### Decision 3: Claude Agent SDK vs Direct API Calls + +**Choice**: Use Claude Agent SDK wrapper, not direct fetch() calls + +**Rationale**: +- User explicitly requested "seamless SDK experience" +- SDK handles authentication, retry logic, error handling +- Benefit from SDK improvements over time +- Demonstrates SDK best practices for the project +- Supports all 3 config sources natively + +**Trade-offs**: +- Adds dependency (@anthropic-ai/claude-agent-sdk) +- Slightly larger bundle size +- Must follow SDK version updates + +### Decision 4: Gateway URL File Storage + +**Choice**: Store gateway URL in temp file, parallel to PID file + +**Rationale**: +- Hooks don't have access to environment variables set by other hooks +- Consistent with existing PID file pattern +- Atomic write, simple read +- Easy cleanup on session end + +**File Location**: `${TMPDIR}toolscript-gateway-${SESSION_ID}.url` + +**Trade-offs**: +- Another file to manage +- Potential race condition if hook runs before SessionStart completes (mitigated by SessionStart running first) + +### Decision 5: Model Selection (Haiku Category) + +**Choice**: Use "haiku" model category, letting users configure their preferred Haiku model + +**Rationale**: +- Task is simple classification/selection (Haiku is sufficient) +- Cost: $0.80/$4 per MTok (vs Sonnet $3/$15) +- Speed: Faster response times (~200ms vs ~500ms) +- Frequent usage pattern (every user prompt) +- Users can choose their preferred Haiku version via Claude Agent SDK config + +**Estimated Usage**: +- Input: ~400 tokens (prompt + skill list) +- Output: ~20 tokens (JSON response) +- Cost: ~$0.0004 per prompt +- For 1000 prompts: ~$0.40 + +**Trade-offs**: +- Slightly less accurate than Sonnet (acceptable for this task) + +### Decision 6: No Tools for Agent + +**Choice**: Agent runs with no tools available (allowedTools: []) + +**Rationale**: +- Task is pure text generation (select from list) +- No file reading, no bash execution needed +- Faster execution (no tool consideration overhead) +- More secure (can't execute unintended commands) + +**Trade-offs**: +- None - tools not needed for this task + +### Decision 7: Single Turn Execution + +**Choice**: Agent executes single turn, no interactive loop + +**Rationale**: +- Task has clear input/output contract (prompt → JSON) +- No clarification questions needed +- Faster execution +- Simpler error handling + +**Implementation**: +```typescript +const response = query({ prompt, options }); +for await (const message of response) { + if (message.type === 'assistant') { + return JSON.parse(message.content); + } +} +``` + +**Trade-offs**: +- Can't handle ambiguous prompts (acceptable - fallback is to suggest nothing) + +### Decision 8: Skill Discovery Method + +**Choice**: Scan skills from three locations: global, project, and plugins + +**Rationale**: +- `~/.claude/skills`: Global skills available to all sessions +- `.claude/skills`: Project-specific skills +- Plugin skills: installed_plugins.json provides plugin locations +- All use consistent SKILL.md format (YAML frontmatter) +- Progressive disclosure pattern requires descriptions < 300 chars +- Matches Claude Code's skill discovery pattern + +**Alternative Considered**: Only scan plugins (rejected - misses global/project skills) + +**Trade-offs**: +- More directories to scan (small performance impact) +- Must handle missing directories gracefully +- Must deduplicate if same skill name appears in multiple locations + +### Decision 9: Fallback Behavior + +**Choice**: Gracefully degrade when API unavailable, delegate auth to SDK + +**Scenarios**: +1. **SDK Auth Error**: Agent SDK handles authentication, return empty suggestions on failure +2. **API Error**: Return empty suggestions +3. **Timeout**: Abort after 5s, return empty suggestions +4. **Malformed Response**: Return empty suggestions + +**Rationale**: +- Session should never crash due to context injection +- Silent failure is acceptable (Claude still gets prompt) +- Claude Agent SDK handles all authentication concerns + +### Decision 10: CLI Command Owns Full Flow + +**Choice**: CLI command does all work including tool search and formatting + +**Rationale**: +- Single source of truth for context generation +- Hook becomes simple pass-through (no logic) +- CLI command testable end-to-end in isolation +- Easier to debug - all logic in one place +- CLI returns ready-to-inject text (no JSON parsing in bash) + +**Flow**: +```bash +# In UserPromptSubmit hook +GATEWAY_URL=$(cat "${TMPDIR}toolscript-gateway-${SESSION_ID}.url" 2>/dev/null || echo "") + +# Exit early if no gateway (toolscript requires gateway) +if [ -z "$GATEWAY_URL" ]; then + exit 0 +fi + +# Call CLI and output its JSON response directly +toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL" +``` + +## Implementation Phases + +### Phase 1: Foundation (MVP) + +- New CLI command `claude-suggest-context` +- Basic Claude Agent SDK integration +- Skill scanning from installed_plugins.json +- Simple prompt template +- JSON output format + +**Success Criteria**: Command returns valid JSON with skill names + +### Phase 2: Hook Integration + +- UserPromptSubmit hook in hooks.json +- Bash script that calls CLI command +- Gateway URL file persistence +- Basic context injection + +**Success Criteria**: Hook executes and injects skill suggestions + +### Phase 3: Tool Search Integration + +- Hook searches gateway for tools +- Combines skills + tools in context +- Handles empty results gracefully + +**Success Criteria**: Relevant tools appear in context when applicable + +### Phase 4: Polish & Optimization + +- Error handling and fallbacks +- Logging and debugging +- Performance optimization +- Documentation and examples + +**Success Criteria**: All error paths tested, docs complete + +## Testing Strategy + +### Unit Tests + +- `claude-suggest-context.test.ts`: CLI command logic + - Skill scanning from plugin directories + - YAML frontmatter parsing + - JSON output formatting + - Error handling + +### Integration Tests + +- `agent-integration.test.ts`: Agent SDK wrapper + - Model selection + - Config source loading + - Single-turn execution + - Response parsing + +### E2E Tests + +- `hook-e2e.test.ts`: Full hook flow + - SessionStart sets up gateway + URL file + - UserPromptSubmit calls CLI and injects context + - Verify context only appears when relevant + - Verify fallback behavior + +### Manual Testing + +- Install plugins with various skills +- Test prompts: + - Relevant: "search for code examples" + - Irrelevant: "hello" + - Ambiguous: "help me" +- Verify API key scenarios (present, missing, invalid) + +## Security Considerations + +### API Key Management + +- **Delegation**: Claude Agent SDK handles all authentication +- **Storage**: Managed by Agent SDK config sources (user, project, local) +- **Transmission**: HTTPS to api.anthropic.com (handled by SDK) +- **Error Messages**: SDK provides appropriate error messages + +### Input Validation + +- **User Prompts**: Passed to LLM (already trusted) +- **Plugin Paths**: Validate against installed_plugins.json +- **SKILL.md Files**: Parse carefully, handle malformed YAML + +### Resource Limits + +- **API Costs**: ~$0.0005 per prompt (acceptable) +- **Timeout**: 5 second limit on LLM calls +- **File Size**: SKILL.md files should be small (<5KB) + +### Prompt Injection + +- **Risk**: User could craft prompts to manipulate skill selection +- **Impact**: Low (worst case: irrelevant skills suggested) +- **Mitigation**: Structured output format (JSON), validate response + +## Performance Targets + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Hook Latency | < 500ms (p95) | Time from hook start to context injection | +| LLM Call | < 300ms (p95) | Claude API response time | +| Tool Search | < 100ms per query | toolscript search execution | +| Skill Scanning | < 50ms | Read all SKILL.md files | +| Total Cost | < $0.001 per prompt | API costs (input + output tokens) | + +## Open Architecture Questions + +1. **Caching**: Should we cache LLM responses for identical prompts? + - **Consideration**: Same prompt might have different context (files changed) + - **Proposal**: Short-lived cache (5 min TTL) with prompt hash as key + +2. **Rate Limiting**: Should we limit API calls? + - **Consideration**: Rapid-fire prompts could rack up costs + - **Proposal**: Max 1 call per 5 seconds, queue subsequent prompts + +3. **Plugin Updates**: How to handle plugin installations during session? + - **Current**: Skill list read once per prompt + - **Alternative**: Cache skill list, invalidate on plugin changes (complex) + +4. **Multi-Language Support**: Should descriptions support localization? + - **Current**: English only + - **Future**: Read locale from Claude Code settings, match descriptions + +5. **Tool Ranking**: How to prioritize tools when many match? + - **Current**: Return first N from search + - **Alternative**: Let LLM rank tools by relevance + +## Dependencies + +### New Dependencies + +- `@anthropic-ai/claude-agent-sdk` (TypeScript) + - Version: Latest stable + - Used for Claude API integration + - ~100KB bundle size + +### Environment Variables + +- Authentication managed by Claude Agent SDK configuration +- Existing: `TOOLSCRIPT_GATEWAY_URL` (still needed for tool search) + +### File System + +- Read: `~/.claude/plugins/installed_plugins.json` +- Read: `{plugin_path}/skills/*/SKILL.md` +- Write: `${TMPDIR}toolscript-gateway-${SESSION_ID}.url` diff --git a/openspec/changes/implement-intelligent-context-injection/proposal.md b/openspec/changes/implement-intelligent-context-injection/proposal.md new file mode 100644 index 0000000..df1be20 --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/proposal.md @@ -0,0 +1,148 @@ +# Implement Intelligent Context Injection + +**Change ID**: `implement-intelligent-context-injection` +**Status**: Proposed +**Created**: 2025-12-01 + +## Summary + +Replace the static context injection in SessionStart hook with an intelligent UserPromptSubmit hook that uses the Claude Agent SDK to dynamically select relevant skills and MCP tools based on the user's actual prompt. + +## Motivation + +### Current Problem + +The current SessionStart hook injects a hardcoded, aggressive message that tells Claude to ALWAYS use toolscript first for every task. This approach: + +1. **Lacks Context Awareness**: Prompts like "hello" or "what's the weather" trigger toolscript suggestions even when irrelevant +2. **Poor User Experience**: Aggressive all-caps messaging feels pushy and unnatural +3. **Inefficient**: Claude wastes time considering toolscript when it's not applicable +4. **Not Scalable**: As more skills are installed from various plugins, static injection becomes unmaintainable + +### Why Now + +- Reference implementation exists (svelte-claude-skills) demonstrating LLM-based skill evaluation +- Claude Agent SDK provides clean integration path with configuration flexibility +- Current architecture already supports hooks and has gateway infrastructure +- User feedback indicates desire for more intelligent, context-aware behavior + +## What Changes + +This proposal introduces an intelligent context injection system: + +1. **New CLI Command**: `toolscript context claude-usage-suggestion` that uses Claude Agent SDK to: + - Analyze user prompts with configured Haiku model + - Select relevant skills from global, project, and plugin skill directories + - Generate MCP tool search queries + - Search gateway for matching MCP tools (if gateway URL provided) + - Format context text and wrap in hook JSON response + - Return `{"hookSpecificOutput":{"additionalContext":"..."}}` or `{}` + +2. **UserPromptSubmit Hook**: New hook that: + - Reads gateway URL from session temp file + - Exits early if no gateway URL (no toolscript without gateway) + - Calls the CLI command with user prompt and gateway URL + - Outputs the CLI's JSON response directly (pass-through) + +3. **Gateway URL Persistence**: Store gateway URL in temp file (like PID) since hooks don't have access to environment variables + +4. **Agent SDK Integration**: Seamless Claude Agent SDK usage with: + - All 3 config sources (user, project, local) + - No Claude Code system prompt (custom prompt only) + - No tools available to agent (pure text generation) + - Single-turn execution (no interactive loop) + +**Breaking changes**: None - this is additive functionality + +## Impact + +### Affected Specs + +- **Modified capability**: `cli-interface` - New command for intelligent context suggestion +- **Modified capability**: `claude-plugin` - UserPromptSubmit hook, gateway URL persistence +- **New capability**: `claude-agent-integration` - Claude Agent SDK wrapper and configuration + +### Dependencies + +- **External**: `@anthropic-ai/claude-agent-sdk` TypeScript package +- **Configuration**: Managed by Claude Agent SDK (supports all 3 config sources) +- **Runtime**: Adds ~400 tokens input + ~20 tokens output per user prompt (~$0.0004 per prompt with Haiku) + +### User Impact + +**Positive**: +- More relevant, context-aware skill and tool suggestions +- Less noise and aggressive messaging +- Automatic discovery of tools from newly installed MCP servers +- Works across all Claude Code plugin skills, not just toolscript + +**Considerations**: +- Authentication managed by Claude Agent SDK configuration +- Graceful fallback to empty suggestions when API unavailable +- Small latency increase (<500ms) per prompt for LLM evaluation + +## Alternatives Considered + +### 1. Static Keyword Matching + +**Approach**: Pattern match user prompts against keyword lists + +**Pros**: +- No API costs +- Instant response +- No external dependencies + +**Cons**: +- Brittle and requires constant maintenance +- Poor handling of natural language variations +- Can't understand context or intent + +**Verdict**: Rejected - doesn't solve the core problem of context awareness + +### 2. Direct API Calls (No Agent SDK) + +**Approach**: Use fetch() to call Claude API directly, similar to svelte-claude-skills reference + +**Pros**: +- Simpler implementation +- Fewer dependencies +- More direct control + +**Cons**: +- Doesn't leverage Agent SDK configuration system +- No benefit from SDK improvements over time +- Misses opportunity to demonstrate SDK best practices +- User specifically requested Agent SDK usage + +**Verdict**: Rejected - user explicitly wants Agent SDK integration for seamless experience + +### 3. SessionStart with Complex Logic + +**Approach**: Keep SessionStart but add intelligence within that hook + +**Pros**: +- Single hook to maintain +- Runs only once per session + +**Cons**: +- Can't adapt to changing user needs within session +- Wrong lifecycle event (no user prompt context) +- Doesn't solve the core timing problem + +**Verdict**: Rejected - UserPromptSubmit is the correct hook for prompt-based decisions + +## Open Questions + +1. **Rate Limiting**: Should we implement rate limiting to prevent excessive API calls (e.g., max 1 call per 5 seconds)? +2. **Caching**: Should we cache results for identical/similar prompts? +3. **Tool Search Depth**: How many tools should we fetch from gateway per search query? +4. **Fallback Behavior**: What should happen when API call fails or times out? +5. **Cost Control**: Should there be a budget limit or opt-out mechanism? + +## Success Criteria + +1. **Relevance**: Context injection only appears when skills/tools are actually relevant +2. **Performance**: LLM evaluation completes within 500ms for 95% of prompts +3. **Accuracy**: Agent correctly identifies relevant skills at least 90% of the time +4. **Cost**: Average cost per prompt stays below $0.001 +5. **Reliability**: Graceful fallback when API unavailable, no session crashes diff --git a/openspec/changes/implement-intelligent-context-injection/specs/claude-agent-integration/spec.md b/openspec/changes/implement-intelligent-context-injection/specs/claude-agent-integration/spec.md new file mode 100644 index 0000000..4033bf6 --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/specs/claude-agent-integration/spec.md @@ -0,0 +1,196 @@ +# claude-agent-integration Specification + +## Purpose + +Provide a clean TypeScript wrapper around the Claude Agent SDK for intelligent skill and tool suggestion within the toolscript CLI, configured for single-turn, tool-free text generation optimized for classification tasks. + +## ADDED Requirements + +### Requirement: Agent SDK Dependency +The system SHALL use the official Claude Agent SDK for Claude API integration. + +#### Scenario: Deno package dependency +- **WHEN** project dependencies are installed +- **THEN** `@anthropic-ai/claude-agent-sdk` is included in deno.json imports + +#### Scenario: TypeScript SDK usage +- **WHEN** agent integration is implemented +- **THEN** TypeScript SDK is used (not Python SDK) + +#### Scenario: SDK version tracking +- **WHEN** SDK is updated +- **THEN** version constraints follow semantic versioning (e.g., `^1.0.0`) + +### Requirement: Query Function Wrapper +The system SHALL provide a TypeScript function that wraps the Agent SDK query function for skill/tool suggestion tasks. + +#### Scenario: Function signature +- **WHEN** suggestion function is called +- **THEN** function accepts parameters: `userPrompt: string`, `skills: Array<{name: string, description: string}>` + +#### Scenario: Return type +- **WHEN** suggestion function completes +- **THEN** function returns `Promise<{skills: string[], toolQueries: string[]}>` for internal processing + +#### Scenario: Model selection +- **WHEN** Agent SDK query is created +- **THEN** model is set to `"haiku"` to use the user's configured model for this category + +### Requirement: Configuration Sources +The agent wrapper SHALL load settings from all three Claude Agent SDK config sources. + +#### Scenario: User-level settings +- **WHEN** Agent SDK query is created +- **THEN** `settingSources` includes "user" for `~/.claude/settings.json` + +#### Scenario: Project-level settings +- **WHEN** Agent SDK query is created +- **THEN** `settingSources` includes "project" for `.claude/settings.json` + +#### Scenario: Local settings +- **WHEN** Agent SDK query is created +- **THEN** `settingSources` includes "local" for `.claude/local/*.md` + +### Requirement: Custom System Prompt +The agent wrapper SHALL use a custom system prompt specific to skill/tool suggestion, not Claude Code's default. + +#### Scenario: System prompt content +- **WHEN** Agent SDK query is created +- **THEN** `systemPrompt` instructs the model to act as a skill/tool classifier + +#### Scenario: JSON output instruction +- **WHEN** system prompt is set +- **THEN** prompt explicitly requests JSON output format with specific schema + +#### Scenario: No default prompt +- **WHEN** Agent SDK query is created +- **THEN** Claude Code's default system prompt is not used + +### Requirement: Tool-Free Execution +The agent SHALL execute without access to any tools for security and performance. + +#### Scenario: Empty tool list +- **WHEN** Agent SDK query is created +- **THEN** `allowedTools` option is set to empty array `[]` + +#### Scenario: No MCP servers +- **WHEN** Agent SDK query is created +- **THEN** `mcpServers` option is not provided or is empty object `{}` + +#### Scenario: Pure text generation +- **WHEN** agent executes +- **THEN** agent only returns text response without tool calls + +### Requirement: Single Turn Execution +The agent SHALL execute exactly one turn and return immediately. + +#### Scenario: Single iteration +- **WHEN** Agent SDK query stream is consumed +- **THEN** wrapper iterates once and returns first assistant message + +#### Scenario: No follow-up +- **WHEN** agent response is received +- **THEN** wrapper does not send additional messages or prompts + +#### Scenario: Stream completion +- **WHEN** agent response ends +- **THEN** wrapper exits iteration and returns result + +### Requirement: Timeout Protection +The agent call SHALL be protected by a timeout to prevent hanging. + +#### Scenario: Timeout duration +- **WHEN** agent query executes +- **THEN** wrapper aborts after 5 seconds using AbortSignal + +#### Scenario: Timeout error handling +- **WHEN** timeout is reached +- **THEN** wrapper throws timeout error that caller can catch + +#### Scenario: Cleanup on timeout +- **WHEN** timeout occurs +- **THEN** wrapper properly cleans up resources and connections + +### Requirement: Response Parsing +The wrapper SHALL parse and validate LLM responses to extract structured data. + +#### Scenario: JSON extraction +- **WHEN** LLM returns text response +- **THEN** wrapper extracts JSON from response (handling markdown code fences) + +#### Scenario: Schema validation +- **WHEN** JSON is parsed +- **THEN** wrapper validates required fields: `skills` (array), `toolQueries` (array) + +#### Scenario: Array type validation +- **WHEN** response fields are validated +- **THEN** `skills` and `toolQueries` are confirmed to be arrays of strings + +#### Scenario: Malformed response +- **WHEN** response is not valid JSON or missing required fields +- **THEN** wrapper throws descriptive error with parsing details + +#### Scenario: Empty arrays are valid +- **WHEN** LLM returns empty skills/queries +- **THEN** wrapper accepts `{"skills": [], "toolQueries": []}` + +### Requirement: Error Handling +The wrapper SHALL handle all error conditions gracefully with informative messages. + +#### Scenario: Network errors +- **WHEN** API request fails due to network issue +- **THEN** wrapper throws error indicating network problem + +#### Scenario: API errors +- **WHEN** Claude API returns error response +- **THEN** wrapper throws error with API error message and status code + +#### Scenario: Rate limiting +- **WHEN** API returns 429 status +- **THEN** wrapper throws error indicating rate limit exceeded + +#### Scenario: Authentication errors +- **WHEN** API returns 401 status +- **THEN** wrapper throws error indicating invalid API key + +#### Scenario: Error message formatting +- **WHEN** any error occurs +- **THEN** error message includes context about what operation failed + +### Requirement: Prompt Engineering +The wrapper SHALL construct effective prompts for skill/tool classification. + +#### Scenario: Prompt template +- **WHEN** wrapper builds LLM prompt +- **THEN** prompt includes user's original prompt, available skills list, and instructions + +#### Scenario: Skill list formatting +- **WHEN** skills are included in prompt +- **THEN** each skill is formatted as "- skill-name: description" + +#### Scenario: Output format instruction +- **WHEN** prompt is constructed +- **THEN** prompt explicitly requests JSON format with schema example + +#### Scenario: Zero-match handling +- **WHEN** prompt is constructed +- **THEN** prompt explicitly allows returning empty arrays when nothing is relevant + +### Requirement: Type Safety +The wrapper SHALL use TypeScript for compile-time type safety. + +#### Scenario: Input types +- **WHEN** wrapper function is called +- **THEN** TypeScript enforces types for userPrompt (string) and skills (array of objects) + +#### Scenario: Return type +- **WHEN** wrapper function completes +- **THEN** TypeScript enforces return type matching the expected schema + +#### Scenario: Error types +- **WHEN** wrapper throws errors +- **THEN** errors are typed (e.g., `ApiError`, `TimeoutError`, `ValidationError`) + +#### Scenario: Agent SDK types +- **WHEN** Agent SDK is used +- **THEN** wrapper imports and uses SDK's TypeScript type definitions diff --git a/openspec/changes/implement-intelligent-context-injection/specs/claude-plugin/spec.md b/openspec/changes/implement-intelligent-context-injection/specs/claude-plugin/spec.md new file mode 100644 index 0000000..24cb75b --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/specs/claude-plugin/spec.md @@ -0,0 +1,76 @@ +# claude-plugin Spec Delta + +## MODIFIED Requirements + +### Requirement: SessionStart Hook +The plugin SHALL provide a hook that starts gateway in background on session start and persists gateway URL for hook access. + +#### Scenario: Write gateway URL to file +- **WHEN** gateway starts successfully +- **THEN** hook writes gateway URL to `${TMPDIR}toolscript-gateway-${SESSION_ID}.url` for hooks without env var access + +#### Scenario: Gateway URL file format +- **WHEN** URL file is written +- **THEN** file contains only the URL (e.g., `http://localhost:12345`) with no additional text + +#### Scenario: Gateway URL file permissions +- **WHEN** URL file is created +- **THEN** file is created with read permissions for the current user + +## REMOVED Requirements + +### Requirement: SessionStart Context Injection +The SessionStart hook SHALL NOT inject static context encouraging toolscript usage. + +#### Scenario: No hardcoded context in SessionStart +- **WHEN** SessionStart hook executes +- **THEN** hook does not output `additionalContext` in JSON response + +## ADDED Requirements + +### Requirement: UserPromptSubmit Hook +The plugin SHALL provide a hook that intelligently suggests relevant skills and tools based on user prompts. + +#### Scenario: Hook registration +- **WHEN** plugin is loaded +- **THEN** UserPromptSubmit hook is registered in `plugins/toolscript/hooks/hooks.json` + +#### Scenario: Hook script location +- **WHEN** UserPromptSubmit hook executes +- **THEN** script is located at `plugins/toolscript/scripts/user-prompt-submit.sh` + +#### Scenario: Receive user prompt +- **WHEN** UserPromptSubmit hook executes +- **THEN** hook receives JSON input with fields: `prompt`, `cwd`, `session_id` + +#### Scenario: Read gateway URL from file +- **WHEN** hook executes +- **THEN** hook reads gateway URL from `${TMPDIR}toolscript-gateway-${SESSION_ID}.url` + +#### Scenario: Exit when no gateway URL +- **WHEN** gateway URL file doesn't exist or is empty +- **THEN** hook exits successfully without calling CLI command or injecting context + +#### Scenario: Call CLI suggestion command with gateway URL +- **WHEN** hook has user prompt and gateway URL exists +- **THEN** hook calls `toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL"` + +#### Scenario: Receive hook JSON response +- **WHEN** CLI command returns +- **THEN** hook receives valid JSON (either with additionalContext or empty object) + +#### Scenario: Output CLI response directly +- **WHEN** CLI command returns JSON +- **THEN** hook outputs the JSON response as-is (pass-through) + +#### Scenario: CLI command error +- **WHEN** `toolscript context claude-usage-suggestion` fails +- **THEN** hook logs error and exits successfully without context injection + +#### Scenario: Hook runs after SessionStart +- **WHEN** UserPromptSubmit hook executes +- **THEN** SessionStart hook has already created gateway URL file (enforced by lifecycle) + +#### Scenario: Multiple prompts in session +- **WHEN** user submits multiple prompts +- **THEN** hook evaluates each prompt independently and may suggest different skills/tools diff --git a/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md b/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md new file mode 100644 index 0000000..6cf03e6 --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md @@ -0,0 +1,134 @@ +# cli-interface Spec Delta + +## ADDED Requirements + +### Requirement: Claude Usage Suggestion Command +The CLI SHALL provide a command under the context group for LLM-based usage suggestion tailored for Claude Code hooks. + +#### Scenario: Command naming and grouping +- **WHEN** user executes context suggestion command +- **THEN** command is `toolscript context claude-usage-suggestion` under the context subcommand group + +#### Scenario: Accept user prompt +- **WHEN** `toolscript context claude-usage-suggestion` is invoked with `--prompt` flag +- **THEN** command receives the user's prompt text for analysis + +#### Scenario: Accept gateway URL +- **WHEN** command is invoked with `--gateway-url` flag +- **THEN** command receives the gateway URL for MCP tool search + +#### Scenario: Output hook JSON response +- **WHEN** command completes successfully +- **THEN** output is valid hook JSON response: `{"hookSpecificOutput":{"additionalContext":"..."}}` + +#### Scenario: Scan global skills directory +- **WHEN** command executes +- **THEN** command scans `~/.claude/skills/*/SKILL.md` for globally installed skills + +#### Scenario: Scan project skills directory +- **WHEN** command executes +- **THEN** command scans `.claude/skills/*/SKILL.md` for project-specific skills + +#### Scenario: Scan installed plugins +- **WHEN** command executes +- **THEN** command reads `~/.claude/plugins/installed_plugins.json` to discover all installed plugins + +#### Scenario: Discover skills from all plugins +- **WHEN** plugin paths are known +- **THEN** command scans `{plugin_path}/skills/*/SKILL.md` for each plugin to find available skills + +#### Scenario: Extract skill descriptions +- **WHEN** SKILL.md file is read +- **THEN** command parses YAML frontmatter to extract `description` field for each skill + +#### Scenario: Handle missing directories +- **WHEN** a skills directory doesn't exist +- **THEN** command continues scanning other locations without failing + +#### Scenario: Handle malformed SKILL.md +- **WHEN** SKILL.md has invalid YAML frontmatter +- **THEN** command logs warning and skips that skill, continues processing others + +#### Scenario: Use Claude Agent SDK +- **WHEN** command needs to evaluate prompt relevance +- **THEN** command uses `@anthropic-ai/claude-agent-sdk` query function to call Claude API + +#### Scenario: Configure Haiku model +- **WHEN** Agent SDK query is created +- **THEN** model is set to "haiku" to use user's configured model for this category + +#### Scenario: Load SDK config sources +- **WHEN** Agent SDK query is created +- **THEN** `settingSources` includes ["user", "project", "local"] for full config support + +#### Scenario: Disable tools for agent +- **WHEN** Agent SDK query is created +- **THEN** `allowedTools` is set to empty array (no tools needed for text generation) + +#### Scenario: Custom system prompt +- **WHEN** Agent SDK query is created +- **THEN** `systemPrompt` is provided with custom instructions (not Claude Code's default) + +#### Scenario: Single turn execution +- **WHEN** Agent SDK query executes +- **THEN** command iterates once through response and returns first assistant message + +#### Scenario: Timeout protection +- **WHEN** LLM call takes too long +- **THEN** command aborts after 5 seconds and returns empty suggestions + +#### Scenario: API error handling +- **WHEN** Claude API returns error response +- **THEN** command logs error and returns empty suggestions + +#### Scenario: Parse LLM JSON response +- **WHEN** LLM returns text response +- **THEN** command extracts JSON from response (handling markdown code fences) + +#### Scenario: Validate response format +- **WHEN** LLM response is parsed +- **THEN** command validates presence of required fields (skills, toolQueries arrays) + +#### Scenario: Empty results produce empty JSON +- **WHEN** no relevant skills or tools are found +- **THEN** command returns empty JSON object `{}` (hook will skip injection) + +#### Scenario: Use provided gateway URL +- **WHEN** command needs to search for MCP tools +- **THEN** command uses the gateway URL provided via `--gateway-url` flag + +#### Scenario: Search gateway for each tool query +- **WHEN** LLM suggests tool search queries +- **THEN** command calls gateway search endpoint for each query to find matching tools + +#### Scenario: Handle missing gateway URL +- **WHEN** gateway URL is not provided via flag +- **THEN** command skips tool search and only suggests skills (if any) + +#### Scenario: Handle tool search failures +- **WHEN** gateway search fails for a query +- **THEN** command continues with other queries and skill suggestions + +#### Scenario: Format skill suggestions +- **WHEN** relevant skills are identified +- **THEN** command formats each as instruction to use Skill(skill-name) tool + +#### Scenario: Format tool suggestions +- **WHEN** relevant MCP tools are found +- **THEN** command formats as instruction to use toolscript skill with specific tool names + +#### Scenario: Combined context formatting +- **WHEN** both skills and tools are relevant +- **THEN** command formats unified context mentioning both, instructing use of toolscript skill for MCP tools + +#### Scenario: Context tone +- **WHEN** command formats context +- **THEN** message is informative and helpful, not aggressive or demanding + +#### Scenario: Wrap context in hook JSON +- **WHEN** formatted context is ready +- **THEN** command wraps it in hook response format: `{"hookSpecificOutput":{"additionalContext":"formatted text here"}}` + +#### Scenario: Return empty JSON when no context +- **WHEN** no skills or tools are relevant +- **THEN** command returns `{}` to signal hook should not inject anything diff --git a/openspec/changes/implement-intelligent-context-injection/tasks.md b/openspec/changes/implement-intelligent-context-injection/tasks.md new file mode 100644 index 0000000..5f7cabf --- /dev/null +++ b/openspec/changes/implement-intelligent-context-injection/tasks.md @@ -0,0 +1,414 @@ +# Implementation Tasks + +## Overview + +This document outlines the implementation tasks for the intelligent context injection system. Tasks are ordered to deliver incremental, testable value and highlight dependencies. + +## Phase 1: Foundation & Dependencies + +### Task 1.1: Add Agent SDK Dependency +**Goal**: Install and configure Claude Agent SDK package + +- Add `@anthropic-ai/claude-agent-sdk` to deno.json imports +- Run `deno cache` to download dependencies +- Verify types are available in IDE + +**Deliverable**: SDK available for import +**Test**: Import SDK in test file, verify no errors + +--- + +### Task 1.2: Create Skill Discovery Utility +**Goal**: Implement skill scanning from all sources (global, project, plugins) + +- Create `src/utils/skill-discovery.ts` +- Implement `scanGlobalSkills()` to scan `~/.claude/skills/*/SKILL.md` +- Implement `scanProjectSkills()` to scan `.claude/skills/*/SKILL.md` +- Implement `loadInstalledPlugins()` to read `~/.claude/plugins/installed_plugins.json` +- Implement `scanPluginSkills(pluginPath)` to find plugin SKILL.md files +- Implement `parseSkillDescription(content)` to extract YAML frontmatter +- Implement `mergeSkills()` to combine all sources and deduplicate +- Handle missing directories and malformed YAML gracefully + +**Deliverable**: Utility that returns `Array<{name: string, description: string, source: string}>` +**Test**: Unit tests with mock skill directories and SKILL.md files + +--- + +### Task 1.3: Create Agent SDK Wrapper +**Goal**: Build TypeScript wrapper for Claude Agent SDK with configuration for suggestion tasks + +- Create `src/agent/suggestion.ts` +- Implement `suggestContext(userPrompt, skills)` function +- Configure Agent SDK query: + - Model: "haiku" (uses user's configured model) + - settingSources: ["user", "project", "local"] + - allowedTools: [] + - Custom systemPrompt for skill/tool classification +- Implement timeout protection (5s with AbortSignal) +- Implement response parsing and validation +- Handle all error cases (network errors, malformed response) +- Delegate authentication to Agent SDK + +**Deliverable**: Function that returns `Promise<{skills: string[], toolQueries: string[]}>` +**Test**: Unit tests with mocked Agent SDK responses + +--- + +## Phase 2: CLI Command Implementation + +### Task 2.1: Create CLI Command File +**Goal**: Scaffold the `context claude-usage-suggestion` command + +- Create `src/cli/commands/context.ts` for the context command group +- Create `src/cli/commands/context/claude-usage-suggestion.ts` for the subcommand +- Define command using @cliffy/command +- Add `--prompt` and `--gateway-url` options +- Wire up to main CLI in `src/cli/main.ts` + +**Deliverable**: Command appears in `toolscript context --help` +**Test**: Run `toolscript context claude-usage-suggestion --help` + +--- + +### Task 2.2: Implement Command Logic +**Goal**: Full end-to-end flow in CLI command + +- Import skill discovery utility +- Import agent wrapper +- Implement command action: + 1. Scan skills from all sources (global, project, plugins) + 2. Call agent wrapper with user prompt and skills + 3. Parse LLM response (skills and toolQueries) + 4. If --gateway-url provided: Search gateway for each tool query + 5. Format context text (skills + tools) + 6. Wrap in hook JSON response + 7. Output JSON to stdout +- Handle errors and return empty JSON `{}` on failures + +**Deliverable**: Working CLI command that outputs hook JSON response +**Test**: Manual test with sample prompts, verify JSON output + +**Dependencies**: Tasks 1.2, 1.3 + +--- + +### Task 2.3: Add Gateway Search Integration +**Goal**: Search gateway for MCP tools when URL provided + +- Import gateway search functionality (or use existing toolscript search) +- Check if --gateway-url was provided +- For each toolQuery from LLM, search gateway +- Collect tool names and descriptions +- Handle search failures gracefully (skip failed queries) + +**Deliverable**: CLI finds relevant MCP tools when gateway URL provided +**Test**: Test with and without --gateway-url, verify behavior + +**Dependencies**: Task 2.2 + +--- + +### Task 2.4: Implement Context Formatting +**Goal**: Format context text and wrap in hook JSON + +- Format skill suggestions: Instruct to use Skill(name) +- Format tool suggestions: Instruct to use toolscript skill with tool names +- Combine skills + tools into unified message +- Use informative, helpful tone (not aggressive) +- Wrap in hook JSON: `{"hookSpecificOutput":{"additionalContext":"..."}}` +- Return empty JSON `{}` when nothing relevant + +**Deliverable**: Well-formatted hook JSON responses +**Test**: Verify JSON structure and context quality + +**Dependencies**: Task 2.3 + +--- + +### Task 2.5: Add Fallback Behavior +**Goal**: Handle all error cases gracefully + +- Return empty JSON `{}` on SDK auth errors +- Return empty JSON `{}` on timeout/network errors +- Return empty JSON `{}` on malformed LLM responses +- Return empty JSON `{}` on gateway search failures + +**Deliverable**: Command never crashes, always returns valid JSON +**Test**: Test all error scenarios, verify valid JSON + +**Dependencies**: Task 2.4 + +--- + +## Phase 3: Gateway URL Persistence + +### Task 3.1: Modify SessionStart Hook Script +**Goal**: Save gateway URL to temp file for hook access + +- Edit `plugins/toolscript/scripts/session-start.sh` +- After starting gateway, write URL to `${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.url` +- Ensure file contains only URL (no extra text) +- Add logging of URL file path + +**Deliverable**: URL file created on session start +**Test**: Start session, verify file exists and contains URL + +--- + +### Task 3.2: Remove Static Context Injection +**Goal**: Remove hardcoded "MUST USE TOOLSCRIPT FIRST" message from SessionStart + +- Edit `plugins/toolscript/scripts/session-start.sh` +- Remove the `additionalContext` field from JSON output +- Keep other functionality intact (gateway start, PID file, env var) + +**Deliverable**: SessionStart no longer injects aggressive context +**Test**: Start session, verify no static context appears + +**Dependencies**: Task 3.1 + +--- + +## Phase 4: UserPromptSubmit Hook Implementation + +### Task 4.1: Create UserPromptSubmit Hook Script +**Goal**: Scaffold the hook script that will be called on each user prompt + +- Create `plugins/toolscript/scripts/user-prompt-submit.sh` +- Make executable (`chmod +x`) +- Implement JSON input reading (user prompt, cwd, session_id) +- Add basic logging to temp file + +**Deliverable**: Hook script that can be called manually +**Test**: Echo JSON to script, verify logging works + +--- + +### Task 4.2: Integrate CLI Command Call +**Goal**: Hook reads gateway URL and calls CLI command + +- In hook script, read gateway URL from ${TMPDIR}toolscript-gateway-${SESSION_ID}.url +- If URL missing/empty: Exit successfully without calling CLI +- If URL exists: call `toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL"` +- Output CLI response directly (no processing needed) +- Handle command failures gracefully (log and exit without output) + +**Deliverable**: Hook exits early without gateway, passes through CLI JSON when gateway exists +**Test**: Test with and without gateway URL file + +**Dependencies**: Task 2.5, 3.1, 4.1 + +--- + +### Task 4.3: Test Hook Behavior +**Goal**: Verify hook correctly handles all cases + +- Test with gateway URL present and relevant prompt (should output CLI JSON) +- Test with gateway URL missing (should exit without output) +- Test with CLI returning empty JSON (should pass through `{}`) +- Test with CLI errors (should exit without output) + +**Deliverable**: Hook works correctly in all scenarios +**Test**: Run hook with various inputs, verify correct behavior + +**Dependencies**: Task 4.2 + +--- + +### Task 4.4: Register UserPromptSubmit Hook +**Goal**: Add hook to hooks.json so Claude Code calls it + +- Edit `plugins/toolscript/hooks/hooks.json` +- Add UserPromptSubmit hook entry with matcher "prompt_input_submit" +- Point to `${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh` + +**Deliverable**: Hook is registered and called by Claude Code +**Test**: Submit prompt in Claude Code session, verify hook executes + +**Dependencies**: Task 4.3 + +--- + +## Phase 5: Testing & Validation + +### Task 5.1: Unit Tests for Agent Wrapper +**Goal**: Comprehensive unit tests for agent integration + +- Test timeout protection +- Test response parsing (valid JSON, malformed, empty) +- Test error handling (network, API errors, rate limits) +- Mock Agent SDK responses + +**Deliverable**: >90% coverage for agent wrapper +**Test**: Run `deno test src/agent/suggestion.test.ts` + +**Dependencies**: Task 1.3 + +--- + +### Task 5.2: Unit Tests for Skill Discovery +**Goal**: Test skill scanning and parsing + +- Test loading installed_plugins.json (valid, missing, malformed) +- Test scanning SKILL.md files (valid, missing, malformed YAML) +- Test description extraction +- Mock filesystem operations + +**Deliverable**: >90% coverage for skill discovery +**Test**: Run `deno test src/utils/skill-discovery.test.ts` + +**Dependencies**: Task 1.2 + +--- + +### Task 5.3: Integration Test for CLI Command +**Goal**: Test full CLI command flow including tool search + +- Test with mocked Agent SDK (no real API calls) +- Test with mocked gateway search +- Test with sample skills list +- Test various user prompts (relevant, irrelevant, ambiguous) +- Test error cases (API failure, timeout, gateway unavailable) +- Verify hook JSON output format + +**Deliverable**: CLI command tested end-to-end +**Test**: Run `deno test src/cli/commands/claude-suggest-context.test.ts` + +**Dependencies**: Task 2.5, 5.1, 5.2 + +--- + +### Task 5.4: E2E Test for Hook Flow +**Goal**: Test complete flow from prompt to context injection + +- Start toolscript gateway in test environment +- Trigger UserPromptSubmit hook with test JSON +- Verify CLI command is called with correct arguments +- Verify context is injected when relevant +- Verify no injection when irrelevant +- Test error scenarios (CLI failure, gateway down) + +**Deliverable**: Full hook flow validated +**Test**: Run `deno test tests/e2e/claude-hook-integration.test.ts` + +**Dependencies**: Task 4.4 + +--- + +### Task 5.5: Manual Testing with Real Plugins +**Goal**: Test with actual installed Claude Code plugins + +- Install multiple plugins with different skills +- Test prompts related to those skills +- Verify correct skills are suggested +- Verify MCP tools are found when relevant +- Test performance (latency should be <500ms p95) + +**Deliverable**: Verified behavior in real environment +**Test**: Manual checklist completed + +**Dependencies**: Task 4.6 + +--- + +## Phase 6: Documentation & Polish + +### Task 6.1: Update README +**Goal**: Document new intelligent context injection feature + +- Add section explaining UserPromptSubmit hook +- Document that authentication is handled by Claude Agent SDK +- Explain cost implications (~$0.0004 per prompt) +- Add troubleshooting section + +**Deliverable**: Updated README.md +**Test**: Review for clarity and completeness + +--- + +### Task 6.2: Update Architecture Docs +**Goal**: Document new components in architecture.md + +- Add diagram showing UserPromptSubmit flow +- Document agent wrapper module +- Document skill discovery utility +- Update data flow section + +**Deliverable**: Updated docs/architecture.md +**Test**: Review for accuracy + +--- + +### Task 6.3: Add CLAUDE.md Guidance +**Goal**: Help LLMs understand the new system + +- Document intelligent context injection in CLAUDE.md +- Explain when suggestions appear vs don't appear +- Note fallback behavior and SDK authentication + +**Deliverable**: Updated CLAUDE.md +**Test**: Review for LLM usability + +--- + +### Task 6.4: Error Handling Audit +**Goal**: Ensure all error paths are handled gracefully + +- Review all error cases in agent wrapper +- Review all error cases in CLI command +- Review all error cases in hook script +- Ensure no case causes session crash +- Ensure all errors are logged appropriately + +**Deliverable**: Robust error handling +**Test**: Test all identified error scenarios + +--- + +## Phase 7: Optimization (Future Work) + +### Task 7.1: Response Caching +**Goal**: Cache LLM responses to reduce costs for repeated prompts + +- Design cache key (hash of prompt + skills) +- Implement simple file-based cache +- Add TTL (5 minutes) +- Add cache hit/miss metrics + +**Deliverable**: Reduced API costs for repeated prompts +**Test**: Verify cache works, measure cost reduction + +**Note**: This is marked as future work, not required for initial release + +--- + +### Task 7.2: Rate Limiting +**Goal**: Prevent excessive API calls in rapid-fire scenarios + +- Track API calls per session +- Implement max 1 call per 5 seconds +- Queue subsequent prompts +- Add bypass flag for testing + +**Deliverable**: Cost protection for rapid prompts +**Test**: Verify rate limiting works, doesn't impact normal usage + +**Note**: This is marked as future work, not required for initial release + +--- + +## Summary + +**Total Tasks**: 24 (19 required + 5 validation/docs + 2 future work) + +**Critical Path**: +1.1 → 1.2 → 1.3 → 2.1 → 2.2 → 2.3 → 2.4 → 2.5 → (3.1) → 4.1 → 4.2 → 4.3 → 4.4 +Note: 3.1 needed before 4.2 for gateway URL lookup in hook + +**Parallelizable Work**: +- Task 3.1 (gateway URL) can happen early +- Task 5.x (tests) can be written alongside implementation +- Task 6.x (docs) can be drafted early + +**Estimated Effort**: ~2-3 days for full implementation + testing + docs (simplified from original estimate) From 5c6a8bf2c507d35ffa945ef307567f92856da067 Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Tue, 2 Dec 2025 09:57:47 +0100 Subject: [PATCH 2/6] feat: suggest skills and tools as prompt hook --- deno.json | 2 + deno.lock | 168 ++++++++-- .../tasks.md | 158 +++++----- plugins/toolscript/hooks/hooks.json | 11 + plugins/toolscript/scripts/session-end.sh | 6 +- plugins/toolscript/scripts/session-start.sh | 18 +- .../toolscript/scripts/user-prompt-submit.sh | 39 +++ plugins/toolscript/skills/toolscript/SKILL.md | 44 +-- .../skills/toolscript/references/examples.md | 39 ++- src/agent/suggestion.ts | 183 +++++++++++ src/cli/commands/context.ts | 13 + .../context/claude-usage-suggestion.ts | 138 ++++++++ src/cli/commands/search.ts | 28 +- src/cli/main.ts | 4 +- src/utils/skill-discovery.ts | 296 ++++++++++++++++++ 15 files changed, 976 insertions(+), 171 deletions(-) create mode 100755 plugins/toolscript/scripts/user-prompt-submit.sh create mode 100644 src/agent/suggestion.ts create mode 100644 src/cli/commands/context.ts create mode 100644 src/cli/commands/context/claude-usage-suggestion.ts create mode 100644 src/utils/skill-discovery.ts diff --git a/deno.json b/deno.json index 1eaa5c4..a50ee56 100644 --- a/deno.json +++ b/deno.json @@ -44,6 +44,7 @@ "include": ["src/**/*.test.ts", "tests/**/*.test.ts"] }, "imports": { + "@anthropic-ai/claude-agent-sdk": "npm:@anthropic-ai/claude-agent-sdk@^0.1.55", "@cliffy/command": "jsr:@cliffy/command@1.0.0-rc.8", "@cliffy/table": "jsr:@cliffy/table@1.0.0-rc.8", "@logtape/logtape": "jsr:@logtape/logtape@^0.8", @@ -53,6 +54,7 @@ "@std/path": "jsr:@std/path@^1.1.3", "json-schema-to-typescript": "npm:json-schema-to-typescript@15.0.4", "zod": "npm:zod@3.25.76", + "zod-to-json-schema": "npm:zod-to-json-schema@3.24.1", "hono": "jsr:@hono/hono@^4.6", "@hono/mcp": "jsr:@hono/mcp@0.1.4", "express": "npm:express@5.1.0", diff --git a/deno.lock b/deno.lock index 067e1d2..a71266d 100644 --- a/deno.lock +++ b/deno.lock @@ -18,6 +18,7 @@ "jsr:@std/regexp@^1.0.1": "1.0.1", "jsr:@std/text@1": "1.0.16", "jsr:@std/text@~1.0.7": "1.0.16", + "npm:@anthropic-ai/claude-agent-sdk@~0.1.55": "0.1.55_zod@3.25.76", "npm:@mkerix/transformers@3.8.0-patch.3": "3.8.0-patch.3", "npm:@modelcontextprotocol/sdk@1.23.0": "1.23.0_zod@3.25.76_ajv@8.17.1_express@5.1.0", "npm:@modelcontextprotocol/sdk@^1.17.4": "1.23.0_zod@3.25.76_ajv@8.17.1_express@5.1.0", @@ -27,6 +28,7 @@ "npm:fuse.js@7": "7.1.0", "npm:json-schema-to-typescript@15.0.4": "15.0.4", "npm:keytar@*": "7.9.0", + "npm:zod-to-json-schema@3.24.1": "3.24.1_zod@3.25.76", "npm:zod@3.25.76": "3.25.76" }, "jsr": { @@ -107,6 +109,22 @@ } }, "npm": { + "@anthropic-ai/claude-agent-sdk@0.1.55_zod@3.25.76": { + "integrity": "sha512-nwlxPjn/gc7I+iOGYY7AGtM2xcjzJFCxF9Bnr0xH1JNaNx+QXLM3h/wmzSvuEOKeJgPymf1GMBs4DZ3jyd/Z7Q==", + "dependencies": [ + "zod" + ], + "optionalDependencies": [ + "@img/sharp-darwin-arm64@0.33.5", + "@img/sharp-darwin-x64@0.33.5", + "@img/sharp-linux-arm@0.33.5", + "@img/sharp-linux-arm64@0.33.5", + "@img/sharp-linux-x64@0.33.5", + "@img/sharp-linuxmusl-arm64@0.33.5", + "@img/sharp-linuxmusl-x64@0.33.5", + "@img/sharp-win32-x64@0.33.5" + ] + }, "@apidevtools/json-schema-ref-parser@11.9.3": { "integrity": "sha512-60vepv88RwcJtSHrD6MjIL6Ta3SOYbgfnkHb+ppAVK+o9mXprRtulx7VlRl3lN3bbvysAfCS7WMVfhUYemB0IQ==", "dependencies": [ @@ -127,37 +145,73 @@ "@img/colour@1.0.0": { "integrity": "sha512-A5P/LfWGFSl6nsckYtjw9da+19jB8hkJ6ACTGcDfEJ0aE+l2n2El7dsVM7UVHZQ9s2lmYMWlrS21YLy2IR1LUw==" }, + "@img/sharp-darwin-arm64@0.33.5": { + "integrity": "sha512-UT4p+iz/2H4twwAoLCqfA9UH5pI6DggwKEGuaPy7nCVQ8ZsiY5PIcrRvD1DzuY3qYL07NtIQcWnBSY/heikIFQ==", + "optionalDependencies": [ + "@img/sharp-libvips-darwin-arm64@1.0.4" + ], + "os": ["darwin"], + "cpu": ["arm64"] + }, "@img/sharp-darwin-arm64@0.34.5": { "integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==", "optionalDependencies": [ - "@img/sharp-libvips-darwin-arm64" + "@img/sharp-libvips-darwin-arm64@1.2.4" ], "os": ["darwin"], "cpu": ["arm64"] }, + "@img/sharp-darwin-x64@0.33.5": { + "integrity": "sha512-fyHac4jIc1ANYGRDxtiqelIbdWkIuQaI84Mv45KvGRRxSAa7o7d1ZKAOBaYbnepLC1WqxfpimdeWfvqqSGwR2Q==", + "optionalDependencies": [ + "@img/sharp-libvips-darwin-x64@1.0.4" + ], + "os": ["darwin"], + "cpu": ["x64"] + }, "@img/sharp-darwin-x64@0.34.5": { "integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==", "optionalDependencies": [ - "@img/sharp-libvips-darwin-x64" + "@img/sharp-libvips-darwin-x64@1.2.4" ], "os": ["darwin"], "cpu": ["x64"] }, + "@img/sharp-libvips-darwin-arm64@1.0.4": { + "integrity": "sha512-XblONe153h0O2zuFfTAbQYAX2JhYmDHeWikp1LM9Hul9gVPjFY427k6dFEcOL72O01QxQsWi761svJ/ev9xEDg==", + "os": ["darwin"], + "cpu": ["arm64"] + }, "@img/sharp-libvips-darwin-arm64@1.2.4": { "integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==", "os": ["darwin"], "cpu": ["arm64"] }, + "@img/sharp-libvips-darwin-x64@1.0.4": { + "integrity": "sha512-xnGR8YuZYfJGmWPvmlunFaWJsb9T/AO2ykoP3Fz/0X5XV2aoYBPkX6xqCQvUTKKiLddarLaxpzNe+b1hjeWHAQ==", + "os": ["darwin"], + "cpu": ["x64"] + }, "@img/sharp-libvips-darwin-x64@1.2.4": { "integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==", "os": ["darwin"], "cpu": ["x64"] }, + "@img/sharp-libvips-linux-arm64@1.0.4": { + "integrity": "sha512-9B+taZ8DlyyqzZQnoeIvDVR/2F4EbMepXMc/NdVbkzsJbzkUjhXv/70GQJ7tdLA4YJgNP25zukcxpX2/SueNrA==", + "os": ["linux"], + "cpu": ["arm64"] + }, "@img/sharp-libvips-linux-arm64@1.2.4": { "integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==", "os": ["linux"], "cpu": ["arm64"] }, + "@img/sharp-libvips-linux-arm@1.0.5": { + "integrity": "sha512-gvcC4ACAOPRNATg/ov8/MnbxFDJqf/pDePbBnuBDcjsI8PssmjoKMAz4LtLaVi+OnSb5FK/yIOamqDwGmXW32g==", + "os": ["linux"], + "cpu": ["arm"] + }, "@img/sharp-libvips-linux-arm@1.2.4": { "integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==", "os": ["linux"], @@ -178,33 +232,64 @@ "os": ["linux"], "cpu": ["s390x"] }, + "@img/sharp-libvips-linux-x64@1.0.4": { + "integrity": "sha512-MmWmQ3iPFZr0Iev+BAgVMb3ZyC4KeFc3jFxnNbEPas60e1cIfevbtuyf9nDGIzOaW9PdnDciJm+wFFaTlj5xYw==", + "os": ["linux"], + "cpu": ["x64"] + }, "@img/sharp-libvips-linux-x64@1.2.4": { "integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==", "os": ["linux"], "cpu": ["x64"] }, + "@img/sharp-libvips-linuxmusl-arm64@1.0.4": { + "integrity": "sha512-9Ti+BbTYDcsbp4wfYib8Ctm1ilkugkA/uscUn6UXK1ldpC1JjiXbLfFZtRlBhjPZ5o1NCLiDbg8fhUPKStHoTA==", + "os": ["linux"], + "cpu": ["arm64"] + }, "@img/sharp-libvips-linuxmusl-arm64@1.2.4": { "integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==", "os": ["linux"], "cpu": ["arm64"] }, + "@img/sharp-libvips-linuxmusl-x64@1.0.4": { + "integrity": "sha512-viYN1KX9m+/hGkJtvYYp+CCLgnJXwiQB39damAO7WMdKWlIhmYTfHjwSbQeUK/20vY154mwezd9HflVFM1wVSw==", + "os": ["linux"], + "cpu": ["x64"] + }, "@img/sharp-libvips-linuxmusl-x64@1.2.4": { "integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==", "os": ["linux"], "cpu": ["x64"] }, + "@img/sharp-linux-arm64@0.33.5": { + "integrity": "sha512-JMVv+AMRyGOHtO1RFBiJy/MBsgz0x4AWrT6QoEVVTyh1E39TrCUpTRI7mx9VksGX4awWASxqCYLCV4wBZHAYxA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-arm64@1.0.4" + ], + "os": ["linux"], + "cpu": ["arm64"] + }, "@img/sharp-linux-arm64@0.34.5": { "integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==", "optionalDependencies": [ - "@img/sharp-libvips-linux-arm64" + "@img/sharp-libvips-linux-arm64@1.2.4" ], "os": ["linux"], "cpu": ["arm64"] }, + "@img/sharp-linux-arm@0.33.5": { + "integrity": "sha512-JTS1eldqZbJxjvKaAkxhZmBqPRGmxgu+qFKSInv8moZ2AmT5Yib3EQ1c6gp493HvrvV8QgdOXdyaIBrhvFhBMQ==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-arm@1.0.5" + ], + "os": ["linux"], + "cpu": ["arm"] + }, "@img/sharp-linux-arm@0.34.5": { "integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==", "optionalDependencies": [ - "@img/sharp-libvips-linux-arm" + "@img/sharp-libvips-linux-arm@1.2.4" ], "os": ["linux"], "cpu": ["arm"] @@ -233,26 +318,50 @@ "os": ["linux"], "cpu": ["s390x"] }, + "@img/sharp-linux-x64@0.33.5": { + "integrity": "sha512-opC+Ok5pRNAzuvq1AG0ar+1owsu842/Ab+4qvU879ippJBHvyY5n2mxF1izXqkPYlGuP/M556uh53jRLJmzTWA==", + "optionalDependencies": [ + "@img/sharp-libvips-linux-x64@1.0.4" + ], + "os": ["linux"], + "cpu": ["x64"] + }, "@img/sharp-linux-x64@0.34.5": { "integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==", "optionalDependencies": [ - "@img/sharp-libvips-linux-x64" + "@img/sharp-libvips-linux-x64@1.2.4" ], "os": ["linux"], "cpu": ["x64"] }, + "@img/sharp-linuxmusl-arm64@0.33.5": { + "integrity": "sha512-XrHMZwGQGvJg2V/oRSUfSAfjfPxO+4DkiRh6p2AFjLQztWUuY/o8Mq0eMQVIY7HJ1CDQUJlxGGZRw1a5bqmd1g==", + "optionalDependencies": [ + "@img/sharp-libvips-linuxmusl-arm64@1.0.4" + ], + "os": ["linux"], + "cpu": ["arm64"] + }, "@img/sharp-linuxmusl-arm64@0.34.5": { "integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==", "optionalDependencies": [ - "@img/sharp-libvips-linuxmusl-arm64" + "@img/sharp-libvips-linuxmusl-arm64@1.2.4" ], "os": ["linux"], "cpu": ["arm64"] }, + "@img/sharp-linuxmusl-x64@0.33.5": { + "integrity": "sha512-WT+d/cgqKkkKySYmqoZ8y3pxx7lx9vVejxW/W4DOFMYVSkErR+w7mf2u8m/y4+xHe7yY9DAXQMWQhpnMuFfScw==", + "optionalDependencies": [ + "@img/sharp-libvips-linuxmusl-x64@1.0.4" + ], + "os": ["linux"], + "cpu": ["x64"] + }, "@img/sharp-linuxmusl-x64@0.34.5": { "integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==", "optionalDependencies": [ - "@img/sharp-libvips-linuxmusl-x64" + "@img/sharp-libvips-linuxmusl-x64@1.2.4" ], "os": ["linux"], "cpu": ["x64"] @@ -274,6 +383,11 @@ "os": ["win32"], "cpu": ["ia32"] }, + "@img/sharp-win32-x64@0.33.5": { + "integrity": "sha512-MpY/o8/8kj+EcnxwvrP4aTJSWw/aZ7JIGR4aBeZkZw5B7/Jn+tY9/VNwtcoGmdT7GfggGIU4kygOMSbYnOrAbg==", + "os": ["win32"], + "cpu": ["x64"] + }, "@img/sharp-win32-x64@0.34.5": { "integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==", "os": ["win32"], @@ -306,7 +420,7 @@ "pkce-challenge", "raw-body@3.0.2", "zod", - "zod-to-json-schema" + "zod-to-json-schema@3.25.0_zod@3.25.76" ] }, "@protobufjs/aspromise@1.1.2": { @@ -1276,30 +1390,30 @@ "semver" ], "optionalDependencies": [ - "@img/sharp-darwin-arm64", - "@img/sharp-darwin-x64", - "@img/sharp-libvips-darwin-arm64", - "@img/sharp-libvips-darwin-x64", - "@img/sharp-libvips-linux-arm", - "@img/sharp-libvips-linux-arm64", + "@img/sharp-darwin-arm64@0.34.5", + "@img/sharp-darwin-x64@0.34.5", + "@img/sharp-libvips-darwin-arm64@1.2.4", + "@img/sharp-libvips-darwin-x64@1.2.4", + "@img/sharp-libvips-linux-arm@1.2.4", + "@img/sharp-libvips-linux-arm64@1.2.4", "@img/sharp-libvips-linux-ppc64", "@img/sharp-libvips-linux-riscv64", "@img/sharp-libvips-linux-s390x", - "@img/sharp-libvips-linux-x64", - "@img/sharp-libvips-linuxmusl-arm64", - "@img/sharp-libvips-linuxmusl-x64", - "@img/sharp-linux-arm", - "@img/sharp-linux-arm64", + "@img/sharp-libvips-linux-x64@1.2.4", + "@img/sharp-libvips-linuxmusl-arm64@1.2.4", + "@img/sharp-libvips-linuxmusl-x64@1.2.4", + "@img/sharp-linux-arm@0.34.5", + "@img/sharp-linux-arm64@0.34.5", "@img/sharp-linux-ppc64", "@img/sharp-linux-riscv64", "@img/sharp-linux-s390x", - "@img/sharp-linux-x64", - "@img/sharp-linuxmusl-arm64", - "@img/sharp-linuxmusl-x64", + "@img/sharp-linux-x64@0.34.5", + "@img/sharp-linuxmusl-arm64@0.34.5", + "@img/sharp-linuxmusl-x64@0.34.5", "@img/sharp-wasm32", "@img/sharp-win32-arm64", "@img/sharp-win32-ia32", - "@img/sharp-win32-x64" + "@img/sharp-win32-x64@0.34.5" ], "scripts": true }, @@ -1458,6 +1572,12 @@ "wrappy@1.0.2": { "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==" }, + "zod-to-json-schema@3.24.1_zod@3.25.76": { + "integrity": "sha512-3h08nf3Vw3Wl3PK+q3ow/lIil81IT2Oa7YpQyUUDsEWbXveMesdfK1xBd2RhCkynwZndAxixji/7SYJJowr62w==", + "dependencies": [ + "zod" + ] + }, "zod-to-json-schema@3.25.0_zod@3.25.76": { "integrity": "sha512-HvWtU2UG41LALjajJrML6uQejQhNJx+JBO9IflpSja4R03iNWfKXrj6W2h7ljuLyc1nKS+9yDyL/9tD1U/yBnQ==", "dependencies": [ @@ -1646,11 +1766,13 @@ "jsr:@std/fs@^1.0.20", "jsr:@std/path@^1.1.3", "jsr:@std/text@1", + "npm:@anthropic-ai/claude-agent-sdk@~0.1.55", "npm:@mkerix/transformers@3.8.0-patch.3", "npm:@modelcontextprotocol/sdk@1.23.0", "npm:express@5.1.0", "npm:fuse.js@7", "npm:json-schema-to-typescript@15.0.4", + "npm:zod-to-json-schema@3.24.1", "npm:zod@3.25.76" ] } diff --git a/openspec/changes/implement-intelligent-context-injection/tasks.md b/openspec/changes/implement-intelligent-context-injection/tasks.md index 5f7cabf..28d8827 100644 --- a/openspec/changes/implement-intelligent-context-injection/tasks.md +++ b/openspec/changes/implement-intelligent-context-injection/tasks.md @@ -6,49 +6,48 @@ This document outlines the implementation tasks for the intelligent context inje ## Phase 1: Foundation & Dependencies -### Task 1.1: Add Agent SDK Dependency +### Task 1.1: Add Agent SDK Dependency ✅ **Goal**: Install and configure Claude Agent SDK package -- Add `@anthropic-ai/claude-agent-sdk` to deno.json imports -- Run `deno cache` to download dependencies -- Verify types are available in IDE +- [x] Add `@anthropic-ai/claude-agent-sdk` to deno.json imports +- [x] Run `deno cache` to download dependencies +- [x] Verify types are available in IDE **Deliverable**: SDK available for import **Test**: Import SDK in test file, verify no errors --- -### Task 1.2: Create Skill Discovery Utility +### Task 1.2: Create Skill Discovery Utility ✅ **Goal**: Implement skill scanning from all sources (global, project, plugins) -- Create `src/utils/skill-discovery.ts` -- Implement `scanGlobalSkills()` to scan `~/.claude/skills/*/SKILL.md` -- Implement `scanProjectSkills()` to scan `.claude/skills/*/SKILL.md` -- Implement `loadInstalledPlugins()` to read `~/.claude/plugins/installed_plugins.json` -- Implement `scanPluginSkills(pluginPath)` to find plugin SKILL.md files -- Implement `parseSkillDescription(content)` to extract YAML frontmatter -- Implement `mergeSkills()` to combine all sources and deduplicate -- Handle missing directories and malformed YAML gracefully +- [x] Create `src/utils/skill-discovery.ts` +- [x] Implement `scanGlobalSkills()` to scan `~/.claude/skills/*/SKILL.md` +- [x] Implement `scanProjectSkills()` to scan `.claude/skills/*/SKILL.md` +- [x] Implement `loadInstalledPlugins()` to read `~/.claude/plugins/installed_plugins.json` +- [x] Implement `scanPluginSkills(pluginPath)` to find plugin SKILL.md files +- [x] Implement `parseSkillDescription(content)` to extract YAML frontmatter +- [x] Implement `mergeSkills()` to combine all sources and deduplicate +- [x] Handle missing directories and malformed YAML gracefully **Deliverable**: Utility that returns `Array<{name: string, description: string, source: string}>` **Test**: Unit tests with mock skill directories and SKILL.md files --- -### Task 1.3: Create Agent SDK Wrapper +### Task 1.3: Create Agent SDK Wrapper ✅ **Goal**: Build TypeScript wrapper for Claude Agent SDK with configuration for suggestion tasks -- Create `src/agent/suggestion.ts` -- Implement `suggestContext(userPrompt, skills)` function -- Configure Agent SDK query: +- [x] Create `src/agent/suggestion.ts` +- [x] Implement `suggestContext(userPrompt, skills)` function +- [x] Configure Agent SDK query: - Model: "haiku" (uses user's configured model) - - settingSources: ["user", "project", "local"] - allowedTools: [] - Custom systemPrompt for skill/tool classification -- Implement timeout protection (5s with AbortSignal) -- Implement response parsing and validation -- Handle all error cases (network errors, malformed response) -- Delegate authentication to Agent SDK +- [x] Implement timeout protection (5s with AbortSignal) +- [x] Implement response parsing and validation +- [x] Handle all error cases (network errors, malformed response) +- [x] Delegate authentication to Agent SDK **Deliverable**: Function that returns `Promise<{skills: string[], toolQueries: string[]}>` **Test**: Unit tests with mocked Agent SDK responses @@ -57,26 +56,26 @@ This document outlines the implementation tasks for the intelligent context inje ## Phase 2: CLI Command Implementation -### Task 2.1: Create CLI Command File +### Task 2.1: Create CLI Command File ✅ **Goal**: Scaffold the `context claude-usage-suggestion` command -- Create `src/cli/commands/context.ts` for the context command group -- Create `src/cli/commands/context/claude-usage-suggestion.ts` for the subcommand -- Define command using @cliffy/command -- Add `--prompt` and `--gateway-url` options -- Wire up to main CLI in `src/cli/main.ts` +- [x] Create `src/cli/commands/context.ts` for the context command group +- [x] Create `src/cli/commands/context/claude-usage-suggestion.ts` for the subcommand +- [x] Define command using @cliffy/command +- [x] Add `--prompt` and `--gateway-url` options +- [x] Wire up to main CLI in `src/cli/main.ts` **Deliverable**: Command appears in `toolscript context --help` **Test**: Run `toolscript context claude-usage-suggestion --help` --- -### Task 2.2: Implement Command Logic +### Task 2.2: Implement Command Logic ✅ **Goal**: Full end-to-end flow in CLI command -- Import skill discovery utility -- Import agent wrapper -- Implement command action: +- [x] Import skill discovery utility +- [x] Import agent wrapper +- [x] Implement command action: 1. Scan skills from all sources (global, project, plugins) 2. Call agent wrapper with user prompt and skills 3. Parse LLM response (skills and toolQueries) @@ -84,7 +83,7 @@ This document outlines the implementation tasks for the intelligent context inje 5. Format context text (skills + tools) 6. Wrap in hook JSON response 7. Output JSON to stdout -- Handle errors and return empty JSON `{}` on failures +- [x] Handle errors and return empty JSON `{}` on failures **Deliverable**: Working CLI command that outputs hook JSON response **Test**: Manual test with sample prompts, verify JSON output @@ -93,14 +92,14 @@ This document outlines the implementation tasks for the intelligent context inje --- -### Task 2.3: Add Gateway Search Integration +### Task 2.3: Add Gateway Search Integration ✅ **Goal**: Search gateway for MCP tools when URL provided -- Import gateway search functionality (or use existing toolscript search) -- Check if --gateway-url was provided -- For each toolQuery from LLM, search gateway -- Collect tool names and descriptions -- Handle search failures gracefully (skip failed queries) +- [x] Import gateway search functionality (or use existing toolscript search) +- [x] Check if --gateway-url was provided +- [x] For each toolQuery from LLM, search gateway +- [x] Collect tool names and descriptions +- [x] Handle search failures gracefully (skip failed queries) **Deliverable**: CLI finds relevant MCP tools when gateway URL provided **Test**: Test with and without --gateway-url, verify behavior @@ -109,15 +108,15 @@ This document outlines the implementation tasks for the intelligent context inje --- -### Task 2.4: Implement Context Formatting +### Task 2.4: Implement Context Formatting ✅ **Goal**: Format context text and wrap in hook JSON -- Format skill suggestions: Instruct to use Skill(name) -- Format tool suggestions: Instruct to use toolscript skill with tool names -- Combine skills + tools into unified message -- Use informative, helpful tone (not aggressive) -- Wrap in hook JSON: `{"hookSpecificOutput":{"additionalContext":"..."}}` -- Return empty JSON `{}` when nothing relevant +- [x] Format skill suggestions: Instruct to use Skill(name) +- [x] Format tool suggestions: Instruct to use toolscript skill with tool names +- [x] Combine skills + tools into unified message +- [x] Use informative, helpful tone (not aggressive) +- [x] Wrap in hook JSON: `{"hookSpecificOutput":{"additionalContext":"..."}}` +- [x] Return empty JSON `{}` when nothing relevant **Deliverable**: Well-formatted hook JSON responses **Test**: Verify JSON structure and context quality @@ -126,13 +125,13 @@ This document outlines the implementation tasks for the intelligent context inje --- -### Task 2.5: Add Fallback Behavior +### Task 2.5: Add Fallback Behavior ✅ **Goal**: Handle all error cases gracefully -- Return empty JSON `{}` on SDK auth errors -- Return empty JSON `{}` on timeout/network errors -- Return empty JSON `{}` on malformed LLM responses -- Return empty JSON `{}` on gateway search failures +- [x] Return empty JSON `{}` on SDK auth errors +- [x] Return empty JSON `{}` on timeout/network errors +- [x] Return empty JSON `{}` on malformed LLM responses +- [x] Return empty JSON `{}` on gateway search failures **Deliverable**: Command never crashes, always returns valid JSON **Test**: Test all error scenarios, verify valid JSON @@ -143,25 +142,25 @@ This document outlines the implementation tasks for the intelligent context inje ## Phase 3: Gateway URL Persistence -### Task 3.1: Modify SessionStart Hook Script +### Task 3.1: Modify SessionStart Hook Script ✅ **Goal**: Save gateway URL to temp file for hook access -- Edit `plugins/toolscript/scripts/session-start.sh` -- After starting gateway, write URL to `${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.url` -- Ensure file contains only URL (no extra text) -- Add logging of URL file path +- [x] Edit `plugins/toolscript/scripts/session-start.sh` +- [x] After starting gateway, write URL to `${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.url` +- [x] Ensure file contains only URL (no extra text) +- [x] Add logging of URL file path **Deliverable**: URL file created on session start **Test**: Start session, verify file exists and contains URL --- -### Task 3.2: Remove Static Context Injection +### Task 3.2: Remove Static Context Injection ✅ **Goal**: Remove hardcoded "MUST USE TOOLSCRIPT FIRST" message from SessionStart -- Edit `plugins/toolscript/scripts/session-start.sh` -- Remove the `additionalContext` field from JSON output -- Keep other functionality intact (gateway start, PID file, env var) +- [x] Edit `plugins/toolscript/scripts/session-start.sh` +- [x] Remove the `additionalContext` field from JSON output +- [x] Keep other functionality intact (gateway start, PID file, env var) **Deliverable**: SessionStart no longer injects aggressive context **Test**: Start session, verify no static context appears @@ -172,27 +171,26 @@ This document outlines the implementation tasks for the intelligent context inje ## Phase 4: UserPromptSubmit Hook Implementation -### Task 4.1: Create UserPromptSubmit Hook Script +### Task 4.1: Create UserPromptSubmit Hook Script ✅ **Goal**: Scaffold the hook script that will be called on each user prompt -- Create `plugins/toolscript/scripts/user-prompt-submit.sh` -- Make executable (`chmod +x`) -- Implement JSON input reading (user prompt, cwd, session_id) -- Add basic logging to temp file +- [x] Create `plugins/toolscript/scripts/user-prompt-submit.sh` +- [x] Make executable (`chmod +x`) +- [x] Implement JSON input reading (user prompt, cwd, session_id) **Deliverable**: Hook script that can be called manually **Test**: Echo JSON to script, verify logging works --- -### Task 4.2: Integrate CLI Command Call +### Task 4.2: Integrate CLI Command Call ✅ **Goal**: Hook reads gateway URL and calls CLI command -- In hook script, read gateway URL from ${TMPDIR}toolscript-gateway-${SESSION_ID}.url -- If URL missing/empty: Exit successfully without calling CLI -- If URL exists: call `toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL"` -- Output CLI response directly (no processing needed) -- Handle command failures gracefully (log and exit without output) +- [x] In hook script, read gateway URL from ${TMPDIR}toolscript-gateway-${SESSION_ID}.url +- [x] If URL missing/empty: Exit successfully without calling CLI +- [x] If URL exists: call `toolscript context claude-usage-suggestion --prompt "$PROMPT" --gateway-url "$GATEWAY_URL"` +- [x] Output CLI response directly (no processing needed) +- [x] Handle command failures gracefully (log and exit without output) **Deliverable**: Hook exits early without gateway, passes through CLI JSON when gateway exists **Test**: Test with and without gateway URL file @@ -201,13 +199,13 @@ This document outlines the implementation tasks for the intelligent context inje --- -### Task 4.3: Test Hook Behavior +### Task 4.3: Test Hook Behavior ✅ **Goal**: Verify hook correctly handles all cases -- Test with gateway URL present and relevant prompt (should output CLI JSON) -- Test with gateway URL missing (should exit without output) -- Test with CLI returning empty JSON (should pass through `{}`) -- Test with CLI errors (should exit without output) +- [x] Test with gateway URL present and relevant prompt (should output CLI JSON) +- [x] Test with gateway URL missing (should exit without output) +- [x] Test with CLI returning empty JSON (should pass through `{}`) +- [x] Test with CLI errors (should exit without output) **Deliverable**: Hook works correctly in all scenarios **Test**: Run hook with various inputs, verify correct behavior @@ -216,12 +214,12 @@ This document outlines the implementation tasks for the intelligent context inje --- -### Task 4.4: Register UserPromptSubmit Hook +### Task 4.4: Register UserPromptSubmit Hook ✅ **Goal**: Add hook to hooks.json so Claude Code calls it -- Edit `plugins/toolscript/hooks/hooks.json` -- Add UserPromptSubmit hook entry with matcher "prompt_input_submit" -- Point to `${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh` +- [x] Edit `plugins/toolscript/hooks/hooks.json` +- [x] Add UserPromptSubmit hook entry with matcher "prompt_input_submit" +- [x] Point to `${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh` **Deliverable**: Hook is registered and called by Claude Code **Test**: Submit prompt in Claude Code session, verify hook executes diff --git a/plugins/toolscript/hooks/hooks.json b/plugins/toolscript/hooks/hooks.json index c005294..84a0dac 100644 --- a/plugins/toolscript/hooks/hooks.json +++ b/plugins/toolscript/hooks/hooks.json @@ -10,6 +10,17 @@ ] } ], + "UserPromptSubmit": [ + { + "matcher": "prompt_input_submit", + "hooks": [ + { + "type": "command", + "command": "${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh" + } + ] + } + ], "SessionEnd": [ { "hooks": [ diff --git a/plugins/toolscript/scripts/session-end.sh b/plugins/toolscript/scripts/session-end.sh index 9de44ba..e8dcb25 100755 --- a/plugins/toolscript/scripts/session-end.sh +++ b/plugins/toolscript/scripts/session-end.sh @@ -10,8 +10,9 @@ INPUT_JSON=$(cat) # Extract session_id from JSON SESSION_ID=$(echo "$INPUT_JSON" | jq -r '.session_id') -# PID and log file locations based on session_id +# PID, URL, and log file locations based on session_id PID_FILE="${TMPDIR}toolscript-gateway-${SESSION_ID}.pid" +URL_FILE="${TMPDIR}toolscript-gateway-${SESSION_ID}.url" LOG_FILE="${TMPDIR}toolscript-gateway-${SESSION_ID}.log" # Read PID from file @@ -28,5 +29,6 @@ if [ -f "$PID_FILE" ]; then rm -f "$PID_FILE" fi -# Cleanup log file +# Cleanup URL and log files +rm -f "$URL_FILE" rm -f "$LOG_FILE" diff --git a/plugins/toolscript/scripts/session-start.sh b/plugins/toolscript/scripts/session-start.sh index 4d9a543..6724af9 100755 --- a/plugins/toolscript/scripts/session-start.sh +++ b/plugins/toolscript/scripts/session-start.sh @@ -10,8 +10,9 @@ INPUT_JSON=$(cat) # Extract session_id from JSON and export CLAUDE_SESSION_ID export CLAUDE_SESSION_ID=$(echo "$INPUT_JSON" | jq -r '.session_id') -# PID and log file locations based on session_id +# PID, URL, and log file locations based on session_id PID_FILE="${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.pid" +URL_FILE="${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.url" LOG_FILE="${TMPDIR}toolscript-gateway-${CLAUDE_SESSION_ID}.log" # Create CLAUDE_ENV_FILE if not set @@ -41,6 +42,9 @@ echo "$GATEWAY_PID" > "$PID_FILE" # Construct gateway URL GATEWAY_URL="http://localhost:$FREE_PORT" +# Save the gateway URL to file for UserPromptSubmit hook +echo "$GATEWAY_URL" > "$URL_FILE" + # Write environment variables to CLAUDE_ENV_FILE if [ -z "$CLAUDE_ENV_FILE" ]; then echo "ERROR: CLAUDE_ENV_FILE is not set, cannot persist environment variables" >> "$LOG_FILE" @@ -48,15 +52,3 @@ if [ -z "$CLAUDE_ENV_FILE" ]; then fi echo "export TOOLSCRIPT_GATEWAY_URL=$GATEWAY_URL" >> "$CLAUDE_ENV_FILE" - -# Inject context encouraging use of toolscript skill -cat <<'EOF' -{ - "hookSpecificOutput": { - "hookEventName": "SessionStart", - "additionalContext": "\n\n**🔧 TOOLSCRIPT MUST BE YOUR FIRST STEP**\n\n**MANDATORY:** Before EVERY task - searches, lookups, API calls, file operations, data queries, code analysis, web requests, documentation lookups - you MUST invoke the `toolscript:toolscript` skill FIRST.\n\n**Required workflow:**\n1. Invoke `toolscript:toolscript` skill immediately\n2. Execute `toolscript search \"\"` to discover MCP tools\n3. Execute the generated TypeScript code with `toolscript exec`\n4. Only fall back to built-in tools if NO MCP tools are found\n\n**This is NOT optional.** MCP tools via toolscript provide superior capabilities, better integrations, and more specialized functionality than built-in tools. You must actively search for them before defaulting to built-in capabilities.\n\n**BE PROACTIVE. CHECK FIRST. ALWAYS.**\n\n" - } -} -EOF - -exit 0 diff --git a/plugins/toolscript/scripts/user-prompt-submit.sh b/plugins/toolscript/scripts/user-prompt-submit.sh new file mode 100755 index 0000000..283daa2 --- /dev/null +++ b/plugins/toolscript/scripts/user-prompt-submit.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# UserPromptSubmit hook - Intelligent context injection based on user prompt + +set -e + +# Read and parse the input JSON +INPUT_JSON=$(cat) + +# Extract user prompt and session_id +USER_PROMPT=$(echo "$INPUT_JSON" | jq -r '.prompt') +SESSION_ID=$(echo "$INPUT_JSON" | jq -r '.session_id') + +# Gateway URL file location +URL_FILE="${TMPDIR}toolscript-gateway-${SESSION_ID}.url" + +# Exit early if no gateway URL file exists (no toolscript without gateway) +if [ ! -f "$URL_FILE" ]; then + exit 0 +fi + +# Read gateway URL from file +GATEWAY_URL=$(cat "$URL_FILE" 2>/dev/null || echo "") + +# Exit early if gateway URL is empty +if [ -z "$GATEWAY_URL" ]; then + exit 0 +fi + +# Call toolscript CLI command to generate intelligent context +# The command will: +# 1. Discover all skills +# 2. Use Claude Agent SDK to select relevant skills and tool queries +# 3. Search gateway for matching tools +# 4. Format context text +# 5. Output hook JSON response +toolscript context claude-usage-suggestion \ + --prompt "$USER_PROMPT" \ + --gateway-url "$GATEWAY_URL" 2>/dev/null || exit 0 diff --git a/plugins/toolscript/skills/toolscript/SKILL.md b/plugins/toolscript/skills/toolscript/SKILL.md index 4b47082..3de3ca3 100644 --- a/plugins/toolscript/skills/toolscript/SKILL.md +++ b/plugins/toolscript/skills/toolscript/SKILL.md @@ -1,39 +1,22 @@ --- name: toolscript -description: This skill should be used before performing any task to check for available MCP tools, including "search", "code lookup", "API calls", "file operations", "data queries". Also use when user asks to "call tool", "list tools", "discover tools". Discover and execute MCP tools via gateway. +description: Discover and execute MCP tools via gateway. Use when user asks to "call tool", "list tools", or before performing tasks that might have specialized MCP capabilities. --- # Toolscript Skill Discover and execute MCP tools through the toolscript gateway. -**Use proactively:** Before any operation, search for specialized MCP tools that might provide better capabilities. +**Use proactively:** Before operations, search for specialized MCP tools. -## Quick Start +## Workflow ```bash -# Search for tools and get TypeScript code -toolscript search "commit git changes" --output types +# 1. Search for tools and get TypeScript code +toolscript search "what you need" --output types -# Execute the generated code -toolscript exec '' -``` - -## Primary Workflow: Search-Based - -```bash -toolscript search "" --output types # Get code (use --threshold 0.1 for more) -toolscript exec '' # Execute inline -toolscript exec --file script.ts # Or from file -``` - -## Alternative: Browse-Based - -```bash -toolscript list-servers # List MCP servers -toolscript list-tools # List tools -toolscript get-types --filter # Get types -toolscript exec --file script.ts # Execute +# 2. Execute the generated code +toolscript exec '' ``` ## Toolscript Format @@ -46,9 +29,14 @@ const result = await tools.serverName.toolName({ }); ``` +## Alternative Workflows + +- **Direct access:** Use `toolscript get-types --filter ,<2nd-tool-name>` if you know the tools +- **Browse discovery:** Use `toolscript list-servers` and `toolscript list-tools ` + ## References -- **`references/commands.md`** - All commands and options -- **`references/configuration.md`** - Gateway, requirements, server setup -- **`references/troubleshooting.md`** - Diagnostics and fixes -- **`references/examples.md`** - Working examples +- `references/commands.md` - All commands and options +- `references/examples.md` - Working examples and workflows +- `references/configuration.md` - Gateway and server setup +- `references/troubleshooting.md` - Diagnostics and fixes diff --git a/plugins/toolscript/skills/toolscript/references/examples.md b/plugins/toolscript/skills/toolscript/references/examples.md index 6abedd2..900ea62 100644 --- a/plugins/toolscript/skills/toolscript/references/examples.md +++ b/plugins/toolscript/skills/toolscript/references/examples.md @@ -1,6 +1,43 @@ # Toolscript Examples -## Basic Example: Single Tool Call +## Discovery Workflows + +### Primary Workflow: Search-Based + +```bash +# Search for tools and get TypeScript code +toolscript search "commit git changes" --output types + +# Execute the generated code +toolscript exec '' +``` + +Use `--threshold 0.1` for more results: +```bash +toolscript search "database query" --output types --threshold 0.1 +``` + +### Alternative: Direct Tool Access + +If you already know which MCP tools to use: + +```bash +toolscript get-types --filter git_commit # Get specific tool's TypeScript types +toolscript exec '' # Execute inline +``` + +### Alternative: Browse-Based Discovery + +```bash +toolscript list-servers # List MCP servers +toolscript list-tools github # List tools from server +toolscript get-types --filter github # Get types for server +toolscript exec --file script.ts # Execute from file +``` + +## Code Examples + +### Basic Example: Single Tool Call ```typescript import { tools } from "toolscript"; diff --git a/src/agent/suggestion.ts b/src/agent/suggestion.ts new file mode 100644 index 0000000..c4d7f82 --- /dev/null +++ b/src/agent/suggestion.ts @@ -0,0 +1,183 @@ +/** + * Claude Agent SDK wrapper for intelligent context suggestion + */ + +import { query } from "@anthropic-ai/claude-agent-sdk"; +import { z } from "zod"; +import { zodToJsonSchema } from "zod-to-json-schema"; +import { join } from "@std/path"; +import { exists } from "@std/fs"; +import { getLogger } from "@logtape/logtape"; +import type { Skill } from "../utils/skill-discovery.ts"; + +const logger = getLogger(["toolscript", "agent", "suggestion"]); + +// Define schema with Zod +const SuggestionResultSchema = z.object({ + skills: z.array(z.string()).max(3).describe( + "Relevant skill names from the available skills list", + ), + toolQueries: z.array(z.string()).max(3).describe("Search queries for finding relevant MCP tools"), +}); + +export type SuggestionResult = z.infer; + +/** + * Load environment variables from ~/.claude/settings.json + */ +async function loadEnvFromSettings(): Promise> { + try { + const home = Deno.env.get("HOME") || Deno.env.get("USERPROFILE"); + if (!home) return {}; + + const settingsFile = join(home, ".claude", "settings.json"); + if (!await exists(settingsFile)) { + return {}; + } + + const content = await Deno.readTextFile(settingsFile); + const settings = JSON.parse(content); + + if (settings && typeof settings.env === "object" && settings.env !== null) { + // Return the env object as a record + return settings.env as Record; + } + + return {}; + } catch { + return {}; + } +} + +/** + * Create a prompt template for skill and tool selection + */ +function createPrompt(userPrompt: string, skills: Skill[]): string { + const skillList = skills + .map((s) => `- ${s.name}: ${s.description}`) + .join("\n"); + + return `You analyze user prompts and recommend relevant skills and tools. + +User's request: +--- +${userPrompt} +--- + +Available skills: +${skillList || "(No skills available)"} + +Based on the user's request, identify: +1. Which skills (if any) from the list would be useful for this task +2. What types of MCP tools to search for (if the task could benefit from MCP tools) + +Rules: +- Only include skills that are DIRECTLY relevant to the user's request +- For irrelevant prompts, return empty arrays +- Return ONE focused toolQuery per distinct action/intent (don't split unnecessarily) +- Each toolQuery should be specific and capture the complete intent (e.g., "kubernetes cluster management", "git operations") +- Maximum 3 skills and 3 tool queries, but prefer fewer focused queries over many generic ones +- Return empty arrays if nothing is relevant + +Examples: +- User: "hello" → no skills, no queries +- User: "search for React hooks examples" → skills: ["react-dev"], queries: ["code search"] +- User: "list k8s servers" → skills: ["toolscript"], queries: ["kubernetes cluster management"] +- User: "what's the weather?" → no skills, queries: ["weather API"] +- User: "commit and push changes" → skills: ["git-flow"], queries: ["git operations"]`; +} + +/** + * Use Claude Agent SDK to suggest relevant skills and tool queries + * + * @param userPrompt The user's input prompt + * @param skills Available skills from all sources + * @param timeoutMs Timeout in milliseconds (default: 5000) + * @returns Promise with suggested skills and tool queries + */ +export async function suggestContext( + userPrompt: string, + skills: Skill[], + timeoutMs: number = 8000, +): Promise { + try { + // Create abort controller for timeout + const abortController = new AbortController(); + const timeoutId = setTimeout(() => abortController.abort(), timeoutMs); + + try { + const prompt = createPrompt(userPrompt, skills); + logger.debug(`Calling Agent SDK with ${skills.length} skills`); + + // Load environment variables from settings.json + const settingsEnv = await loadEnvFromSettings(); + + // Merge with process env (process env overrides settings.json) + const mergedEnv = { + ...settingsEnv, + ...Deno.env.toObject(), + }; + + // Convert Zod schema to JSON Schema + const jsonSchema = zodToJsonSchema(SuggestionResultSchema, { $refStrategy: "none" }); + + // Query the Agent SDK with structured output + const response = query({ + prompt: prompt, + options: { + abortController, + model: "haiku", + allowedTools: [], + maxTurns: 5, + env: mergedEnv, + outputFormat: { + type: "json_schema", + schema: jsonSchema, + }, + }, + }); + + // Get the structured result + for await (const msg of response) { + if (msg.type === "result") { + clearTimeout(timeoutId); + + if (msg.subtype === "success" && msg.structured_output) { + // Validate and parse with Zod + const parsed = SuggestionResultSchema.safeParse(msg.structured_output); + if (parsed.success) { + logger.debug( + `Suggestion result: ${parsed.data.skills.length} skills, ${parsed.data.toolQueries.length} queries`, + ); + return parsed.data; + } else { + logger.error("Failed to validate structured output", { error: parsed.error }); + } + } + + // Handle errors + if (msg.subtype !== "success") { + logger.error(`Agent SDK returned error: ${msg.subtype}`); + } + + break; + } + } + + logger.debug("No assistant message received from Agent SDK"); + clearTimeout(timeoutId); + return { skills: [], toolQueries: [] }; + } finally { + clearTimeout(timeoutId); + } + } catch (error) { + // Handle all errors gracefully (auth errors, network errors, timeouts) + // Return empty suggestions - session should never crash + if (error instanceof Error && error.name === "AbortError") { + logger.debug("Agent SDK request timed out"); + } else { + logger.error(`Agent SDK error: ${error}`); + } + return { skills: [], toolQueries: [] }; + } +} diff --git a/src/cli/commands/context.ts b/src/cli/commands/context.ts new file mode 100644 index 0000000..75643fc --- /dev/null +++ b/src/cli/commands/context.ts @@ -0,0 +1,13 @@ +/** + * Context command group - Commands for generating context suggestions + */ + +import { Command } from "@cliffy/command"; +import { claudeUsageSuggestionCommand } from "./context/claude-usage-suggestion.ts"; + +/** + * Context command group + */ +export const contextCommand = new Command() + .description("Generate context suggestions for Claude Code") + .command("claude-usage-suggestion", claudeUsageSuggestionCommand); diff --git a/src/cli/commands/context/claude-usage-suggestion.ts b/src/cli/commands/context/claude-usage-suggestion.ts new file mode 100644 index 0000000..987a89e --- /dev/null +++ b/src/cli/commands/context/claude-usage-suggestion.ts @@ -0,0 +1,138 @@ +/** + * Claude usage suggestion command - Intelligent context injection for Claude Code hooks + */ + +import { Command } from "@cliffy/command"; +import { getLogger } from "@logtape/logtape"; +import type { HookJSONOutput } from "@anthropic-ai/claude-agent-sdk"; +import { discoverAllSkills } from "../../../utils/skill-discovery.ts"; +import { suggestContext } from "../../../agent/suggestion.ts"; +import type { SearchResult } from "../../../search/types.ts"; + +const logger = getLogger(["toolscript", "context", "suggestion"]); + +/** + * Search the gateway for tools matching a query + */ +async function searchGatewayForTools( + gatewayUrl: string, + query: string, + limit: number = 3, +): Promise { + try { + const searchParams = new URLSearchParams({ + q: query, + limit: String(limit), + }); + + const response = await fetch(`${gatewayUrl}/search?${searchParams}`); + + if (!response.ok) { + return []; + } + + const results: SearchResult[] = await response.json(); + return results; + } catch { + // Gateway not available or search failed + return []; + } +} + +/** + * Format context text from skills and tools + */ +function formatContextText(skills: string[], tools: SearchResult[]): string { + const parts: string[] = []; + + if (skills.length > 0) { + parts.push( + `The following skills may be helpful for this request: +${skills.map((s) => `- Skill("${s}")`).join("\n")}`, + ); + } + + if (tools.length > 0) { + const toolNames = tools.map((t) => t.tool.toolId).join(", "); + parts.push( + `Consider using Skill("toolscript") to access these MCP tools: ${toolNames}`, + ); + } + + return parts.join("\n\n"); +} + +/** + * Create hook JSON response + */ +function createHookResponse(contextText: string): HookJSONOutput { + if (contextText) { + return { + hookSpecificOutput: { + hookEventName: "UserPromptSubmit", + additionalContext: contextText, + }, + }; + } + // Return empty object for no context - hook system will treat this as success + return {}; +} + +/** + * Claude usage suggestion command + */ +export const claudeUsageSuggestionCommand = new Command() + .description("Generate intelligent context suggestions for Claude Code hooks") + .option("-p, --prompt ", "User's prompt", { required: true }) + .option("-g, --gateway-url ", "Gateway URL") + .env("TOOLSCRIPT_GATEWAY_URL=", "Gateway URL", { prefix: "TOOLSCRIPT_" }) + .action(async (options) => { + try { + // 1. Discover all skills + const skills = await discoverAllSkills(); + logger.debug(`Discovered ${skills.length} skills`); + + // 2. Use Agent SDK to suggest relevant skills and tool queries + const suggestion = await suggestContext(options.prompt, skills); + + // 3. Search gateway for tools (if gateway URL provided and tool queries exist) + let allTools: SearchResult[] = []; + + if (options.gatewayUrl && suggestion.toolQueries.length > 0) { + logger.debug(`Searching gateway for ${suggestion.toolQueries.length} tool queries`); + const toolSearches = suggestion.toolQueries.map((query) => + searchGatewayForTools(options.gatewayUrl!, query) + ); + + const toolResults = await Promise.all(toolSearches); + allTools = toolResults.flat(); + + // Deduplicate tools by toolId, keeping the highest score for each tool + const toolMap = new Map(); + for (const tool of allTools) { + const existing = toolMap.get(tool.tool.toolId); + // Keep this tool if it's new OR has a higher score than existing + if (!existing || tool.score > existing.score) { + toolMap.set(tool.tool.toolId, tool); + } + } + + // Sort by confidence score (highest first) and take top 5 + allTools = Array.from(toolMap.values()) + .sort((a, b) => b.score - a.score) + .slice(0, 5); + logger.debug(`Found ${allTools.length} unique tools after deduplication`); + } + + // 4. Format context text + const contextText = formatContextText(suggestion.skills, allTools); + + // 5. Create and output hook JSON response + const hookResponse = createHookResponse(contextText); + console.log(JSON.stringify(hookResponse, null, 2)); + } catch (error) { + // On any error, output empty JSON (graceful fallback) + logger.error("Error generating suggestion", { error }); + console.log(JSON.stringify({}, null, 2)); + } + }); diff --git a/src/cli/commands/search.ts b/src/cli/commands/search.ts index f8cfa63..cee96cb 100644 --- a/src/cli/commands/search.ts +++ b/src/cli/commands/search.ts @@ -6,25 +6,7 @@ import { Command, EnumType } from "@cliffy/command"; import { Table } from "@cliffy/table"; import { dedent } from "@std/text/unstable-dedent"; import { fetchTypes, formatTypesOutput } from "../utils/types-output.ts"; - -/** - * Search result from gateway - */ -interface SearchResultItem { - tool: { - serverName: string; - toolName: string; - toolId: string; - description: string; - }; - score: number; - scoreBreakdown: { - semantic: number; - fuzzy: number; - combined: number; - }; - reason?: string; -} +import type { SearchResult } from "../../search/types.ts"; /** * Format confidence score as percentage @@ -36,7 +18,7 @@ function formatScore(score: number): string { /** * Build confidence table in Markdown format */ -function buildConfidenceTable(results: SearchResultItem[]): string { +function buildConfidenceTable(results: SearchResult[]): string { const rows = results.map((r) => { return `| ${r.tool.toolId} | ${formatScore(r.score)} | ${r.reason || "match"} |`; }); @@ -103,7 +85,7 @@ export const searchCommand = new Command() Deno.exit(1); } - const results: SearchResultItem[] = await searchResponse.json(); + const results: SearchResult[] = await searchResponse.json(); if (results.length === 0) { console.log("No tools found matching your query."); @@ -132,7 +114,7 @@ export const searchCommand = new Command() /** * Output results in table format */ -function outputTableFormat(results: SearchResultItem[]): void { +function outputTableFormat(results: SearchResult[]): void { const table = new Table() .header(["Tool", "Confidence", "Description"]) .body( @@ -154,7 +136,7 @@ function outputTableFormat(results: SearchResultItem[]): void { */ async function outputTypesFormat( gatewayUrl: string, - results: SearchResultItem[], + results: SearchResult[], ): Promise { // Build tool filter from results const toolIds = results.map((r) => r.tool.toolId).join(","); diff --git a/src/cli/main.ts b/src/cli/main.ts index eeb6982..75a4545 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -11,6 +11,7 @@ import { listServersCommand, listToolsCommand } from "./commands/list.ts"; import { getTypesCommand } from "./commands/types.ts"; import { searchCommand } from "./commands/search.ts"; import { authCommand } from "./commands/auth.ts"; +import { contextCommand } from "./commands/context.ts"; import packageInfo from "../../deno.json" with { type: "json" }; /** @@ -26,7 +27,8 @@ const main = new Command() .command("list-servers", listServersCommand) .command("list-tools", listToolsCommand) .command("get-types", getTypesCommand) - .command("exec", execCommand); + .command("exec", execCommand) + .command("context", contextCommand); if (import.meta.main) { await main.parse(Deno.args); diff --git a/src/utils/skill-discovery.ts b/src/utils/skill-discovery.ts new file mode 100644 index 0000000..bbaa614 --- /dev/null +++ b/src/utils/skill-discovery.ts @@ -0,0 +1,296 @@ +/** + * Skill discovery utility for finding and parsing SKILL.md files + * from global, project, and plugin sources. + */ + +import { join } from "@std/path"; +import { exists } from "@std/fs"; +import { getLogger } from "@logtape/logtape"; + +const logger = getLogger(["toolscript", "utils", "skill-discovery"]); + +/** + * Get the global .claude directory path + */ +function getGlobalClaudeDirectory(): string | null { + const home = Deno.env.get("HOME") || Deno.env.get("USERPROFILE"); + if (!home) { + return null; + } + return join(home, ".claude"); +} + +export interface Skill { + name: string; + description: string; + source: "global" | "project" | "plugin"; + pluginName?: string; +} + +interface PluginMetadata { + installPath: string; + name: string; +} + +/** + * Parse YAML frontmatter from SKILL.md content + */ +export function parseSkillDescription( + content: string, +): { name: string; description: string } | null { + // Match YAML frontmatter between --- delimiters + const frontmatterRegex = /^---\n([\s\S]*?)\n---/; + const match = content.match(frontmatterRegex); + + if (!match) { + return null; + } + + const frontmatter = match[1]; + let name: string | null = null; + let description: string | null = null; + + // Parse name and description from YAML + for (const line of frontmatter.split("\n")) { + const nameMatch = line.match(/^name:\s*(.+)$/); + if (nameMatch) { + name = nameMatch[1].trim(); + continue; + } + + const descMatch = line.match(/^description:\s*(.+)$/); + if (descMatch) { + description = descMatch[1].trim(); + continue; + } + } + + if (name && description) { + return { name, description }; + } + + return null; +} + +/** + * Scan a directory for SKILL.md files + */ +async function scanSkillDirectory( + basePath: string, + source: "global" | "project" | "plugin", + pluginName?: string, +): Promise { + const skills: Skill[] = []; + + try { + const skillsPath = join(basePath, "skills"); + + if (!await exists(skillsPath)) { + return skills; + } + + // Iterate over subdirectories in skills/ + for await (const entry of Deno.readDir(skillsPath)) { + if (!entry.isDirectory) continue; + + const skillMdPath = join(skillsPath, entry.name, "SKILL.md"); + + if (await exists(skillMdPath)) { + try { + const content = await Deno.readTextFile(skillMdPath); + const parsed = parseSkillDescription(content); + + if (parsed) { + logger.debug( + `Found skill: ${parsed.name} (${source}${pluginName ? ` - ${pluginName}` : ""})`, + ); + skills.push({ + ...parsed, + source, + pluginName, + }); + } + } catch { + // Skip malformed files + logger.debug(`Failed to parse skill file: ${skillMdPath}`); + } + } + } + } catch { + // Directory doesn't exist or not accessible, return empty + } + + return skills; +} + +/** + * Scan global skills from ~/.claude/skills + */ +export async function scanGlobalSkills(): Promise { + const claudeDir = getGlobalClaudeDirectory(); + if (!claudeDir) { + return []; + } + + const result = await scanSkillDirectory(claudeDir, "global"); + logger.debug(`Found ${result.length} global skills`); + return result; +} + +/** + * Scan project skills from .claude/skills + */ +export async function scanProjectSkills(): Promise { + const cwd = Deno.cwd(); + const claudeDir = join(cwd, ".claude"); + return await scanSkillDirectory(claudeDir, "project"); +} + +/** + * Load enabled plugins from ~/.claude/settings.json + */ +async function loadEnabledPlugins(): Promise> { + try { + const claudeDir = getGlobalClaudeDirectory(); + if (!claudeDir) return new Set(); + + const settingsFile = join(claudeDir, "settings.json"); + + if (!await exists(settingsFile)) { + return new Set(); + } + + const content = await Deno.readTextFile(settingsFile); + const settings = JSON.parse(content); + + if ( + settings && typeof settings.enabledPlugins === "object" && settings.enabledPlugins !== null + ) { + // enabledPlugins is an object with plugin names as keys and boolean values + const enabledList = Object.entries(settings.enabledPlugins) + .filter(([_name, enabled]) => enabled === true) + .map(([name, _enabled]) => name); + + logger.debug(`Found ${enabledList.length} enabled plugins`); + return new Set(enabledList); + } + + return new Set(); + } catch (error) { + logger.error("Error loading plugin settings", { error }); + return new Set(); + } +} + +/** + * Load installed plugins metadata from ~/.claude/plugins/installed_plugins.json + */ +export async function loadInstalledPlugins(): Promise { + try { + const claudeDir = getGlobalClaudeDirectory(); + if (!claudeDir) { + return []; + } + + const pluginsFile = join(claudeDir, "plugins", "installed_plugins.json"); + + if (!await exists(pluginsFile)) { + return []; + } + + const content = await Deno.readTextFile(pluginsFile); + const data = JSON.parse(content); + + // Handle wrapper format with "plugins" key + let pluginsData = data; + if (typeof data === "object" && data !== null && "plugins" in data) { + pluginsData = data.plugins; + } + + // Load enabled plugins from settings + const enabledPlugins = await loadEnabledPlugins(); + + // pluginsData is an object with plugin names as keys + if (typeof pluginsData === "object" && pluginsData !== null) { + const plugins = Object.entries(pluginsData) + .filter(([name, _meta]: [string, unknown]) => { + // If enabledPlugins is empty, allow all (no settings file or no list) + if (enabledPlugins.size === 0) return true; + + // Otherwise, only include if it's in the enabled list + return enabledPlugins.has(name); + }) + .map(([name, meta]: [string, unknown]) => ({ + name, + installPath: typeof meta === "object" && meta !== null && "installPath" in meta + ? String(meta.installPath) + : typeof meta === "object" && meta !== null && "path" in meta + ? String(meta.path) + : "", + })); + logger.debug(`Loaded ${plugins.length} enabled plugins`); + return plugins; + } + + return []; + } catch (error) { + logger.error("Error loading plugins", { error }); + return []; + } +} + +/** + * Scan skills from a single plugin + */ +export async function scanPluginSkills(plugin: PluginMetadata): Promise { + return await scanSkillDirectory(plugin.installPath, "plugin", plugin.name); +} + +/** + * Merge skills from all sources and deduplicate by name + * Priority: project > global > plugin + */ +export function mergeSkills(skillLists: Skill[][]): Skill[] { + const skillMap = new Map(); + + // Process in reverse priority order (plugin, global, project) + // so later entries overwrite earlier ones + for (const skills of skillLists) { + for (const skill of skills) { + // Project skills have highest priority, then global, then plugin + const existing = skillMap.get(skill.name); + + if (!existing) { + skillMap.set(skill.name, skill); + } else { + const priorityOrder = { "project": 3, "global": 2, "plugin": 1 }; + if (priorityOrder[skill.source] > priorityOrder[existing.source]) { + skillMap.set(skill.name, skill); + } + } + } + } + + return Array.from(skillMap.values()); +} + +/** + * Discover all skills from global, project, and plugin sources + */ +export async function discoverAllSkills(): Promise { + // Scan global skills + const globalSkills = await scanGlobalSkills(); + + // Scan project skills + const projectSkills = await scanProjectSkills(); + + // Scan plugin skills + const plugins = await loadInstalledPlugins(); + const pluginSkillsLists = await Promise.all( + plugins.map((plugin) => scanPluginSkills(plugin)), + ); + const pluginSkills = pluginSkillsLists.flat(); + + // Merge all skills with priority: project > global > plugin + return mergeSkills([pluginSkills, globalSkills, projectSkills]); +} From 17be80d701b3103241ebc06f76df4bb6ad19a945 Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Tue, 2 Dec 2025 10:25:07 +0100 Subject: [PATCH 3/6] test: add tests for suggestion functionality --- src/agent/suggestion.test.ts | 189 +++++++++++++ .../context/claude-usage-suggestion.test.ts | 214 ++++++++++++++ .../context/claude-usage-suggestion.ts | 44 +-- src/utils/skill-discovery.test.ts | 260 ++++++++++++++++++ 4 files changed, 691 insertions(+), 16 deletions(-) create mode 100644 src/agent/suggestion.test.ts create mode 100644 src/cli/commands/context/claude-usage-suggestion.test.ts create mode 100644 src/utils/skill-discovery.test.ts diff --git a/src/agent/suggestion.test.ts b/src/agent/suggestion.test.ts new file mode 100644 index 0000000..7d8cb54 --- /dev/null +++ b/src/agent/suggestion.test.ts @@ -0,0 +1,189 @@ +/** + * Tests for Claude Agent SDK suggestion wrapper + */ + +import { assertEquals, assertExists } from "@std/assert"; +import { suggestContext } from "./suggestion.ts"; +import type { Skill } from "../utils/skill-discovery.ts"; + +// Mock skills for testing +const mockSkills: Skill[] = [ + { + name: "toolscript", + description: "Search and execute MCP tools via gateway", + source: "plugin", + pluginName: "toolscript", + }, + { + name: "react-dev", + description: "React development utilities and hooks", + source: "global", + }, + { + name: "git-flow", + description: "Git workflow automation", + source: "project", + }, +]; + +Deno.test({ + name: "suggestContext should return empty result on timeout", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const result = await suggestContext( + "hello world", + mockSkills, + 1, // Very short timeout to trigger abort + ); + + assertExists(result); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + }, +}); + +Deno.test({ + name: "suggestContext should handle empty skills array", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const result = await suggestContext( + "search for kubernetes tools", + [], + 100, // Short timeout for faster tests + ); + + assertExists(result); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + }, +}); + +Deno.test({ + name: "suggestContext should handle errors gracefully", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + // Test with invalid ANTHROPIC_API_KEY to trigger auth error + const originalKey = Deno.env.get("ANTHROPIC_API_KEY"); + + try { + Deno.env.set("ANTHROPIC_API_KEY", "invalid-key"); + + const result = await suggestContext( + "search for tools", + mockSkills, + 100, + ); + + assertExists(result); + // Should return empty arrays on error, not throw + assertEquals(result.skills, []); + assertEquals(result.toolQueries, []); + } finally { + // Restore original key + if (originalKey) { + Deno.env.set("ANTHROPIC_API_KEY", originalKey); + } else { + Deno.env.delete("ANTHROPIC_API_KEY"); + } + } + }, +}); + +Deno.test({ + name: "suggestContext should validate result structure", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const result = await suggestContext( + "test prompt", + mockSkills, + 100, + ); + + assertExists(result); + assertExists(result.skills); + assertExists(result.toolQueries); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + }, +}); + +Deno.test({ + name: "suggestContext should respect max array limits", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + // Even if the LLM returns more, Zod schema should enforce max 3 + const result = await suggestContext( + "complex query requiring many tools", + mockSkills, + 100, + ); + + assertExists(result); + assertEquals(result.skills.length <= 3, true); + assertEquals(result.toolQueries.length <= 3, true); + }, +}); + +Deno.test({ + name: "suggestContext should handle special characters in prompts", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const specialPrompts = [ + 'search for "react hooks" examples', + "find tools with @mentions", + "query with new\nlines", + "unicode: 🚀 emoji", + ]; + + for (const prompt of specialPrompts) { + const result = await suggestContext(prompt, mockSkills, 100); + assertExists(result); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + } + }, +}); + +Deno.test({ + name: "suggestContext should handle long prompts", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const longPrompt = "search for tools ".repeat(100); + + const result = await suggestContext(longPrompt, mockSkills, 100); + + assertExists(result); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + }, +}); + +Deno.test({ + name: "suggestContext should handle many skills", + sanitizeResources: false, + sanitizeOps: false, + async fn() { + const manySkills: Skill[] = Array.from({ length: 50 }, (_, i) => ({ + name: `skill-${i}`, + description: `Description for skill ${i}`, + source: "global" as const, + })); + + const result = await suggestContext( + "find relevant skills", + manySkills, + 100, + ); + + assertExists(result); + assertEquals(Array.isArray(result.skills), true); + assertEquals(Array.isArray(result.toolQueries), true); + }, +}); diff --git a/src/cli/commands/context/claude-usage-suggestion.test.ts b/src/cli/commands/context/claude-usage-suggestion.test.ts new file mode 100644 index 0000000..cc48613 --- /dev/null +++ b/src/cli/commands/context/claude-usage-suggestion.test.ts @@ -0,0 +1,214 @@ +/** + * Tests for Claude usage suggestion command + */ + +import { assertEquals, assertExists, assertStringIncludes } from "@std/assert"; +import type { SearchResult } from "../../../search/types.ts"; +import { + createHookResponse, + deduplicateAndSortTools, + formatContextText, +} from "./claude-usage-suggestion.ts"; + +Deno.test("formatContextText should format skills correctly", () => { + const result = formatContextText(["skill1", "skill2"], []); + + assertExists(result); + assertStringIncludes(result, 'Skill("skill1")'); + assertStringIncludes(result, 'Skill("skill2")'); + assertStringIncludes(result, "The following skills may be helpful"); +}); + +Deno.test("formatContextText should format tools correctly", () => { + const mockTools: SearchResult[] = [ + { + tool: { + toolId: "tool1", + serverName: "server1", + toolName: "tool1", + description: "First tool", + inputSchema: {}, + }, + score: 0.9, + scoreBreakdown: { + semantic: 0.85, + fuzzy: 0.95, + combined: 0.9, + }, + }, + { + tool: { + toolId: "tool2", + serverName: "server1", + toolName: "tool2", + description: "Second tool", + inputSchema: {}, + }, + score: 0.8, + scoreBreakdown: { + semantic: 0.75, + fuzzy: 0.85, + combined: 0.8, + }, + }, + ]; + + const result = formatContextText([], mockTools); + + assertExists(result); + assertStringIncludes(result, "tool1"); + assertStringIncludes(result, "tool2"); + assertStringIncludes(result, 'Skill("toolscript")'); + assertStringIncludes(result, "MCP tools"); +}); + +Deno.test("formatContextText should format both skills and tools", () => { + const mockTools: SearchResult[] = [ + { + tool: { + toolId: "tool1", + serverName: "server1", + toolName: "tool1", + description: "First tool", + inputSchema: {}, + }, + score: 0.9, + scoreBreakdown: { + semantic: 0.85, + fuzzy: 0.95, + combined: 0.9, + }, + }, + ]; + + const result = formatContextText(["skill1"], mockTools); + + assertExists(result); + assertStringIncludes(result, 'Skill("skill1")'); + assertStringIncludes(result, "tool1"); + assertEquals(result.includes("\n\n"), true); // Should have section separator +}); + +Deno.test("formatContextText should return empty string for no input", () => { + const result = formatContextText([], []); + assertEquals(result, ""); +}); + +Deno.test("createHookResponse should return hookSpecificOutput for non-empty context", () => { + const result = createHookResponse("test context"); + + assertExists(result); + assertEquals(typeof result, "object"); + + // Check that hookSpecificOutput exists and has the right properties + if ("hookSpecificOutput" in result && result.hookSpecificOutput) { + assertEquals(result.hookSpecificOutput.hookEventName, "UserPromptSubmit"); + // Type assertion needed because hookSpecificOutput is a union type + const output = result.hookSpecificOutput as { + hookEventName: string; + additionalContext?: string; + }; + assertEquals(output.additionalContext, "test context"); + } else { + throw new Error("hookSpecificOutput should exist for non-empty context"); + } +}); + +Deno.test("createHookResponse should return empty object for empty context", () => { + const result = createHookResponse(""); + + assertExists(result); + assertEquals(typeof result, "object"); + assertEquals(Object.keys(result).length, 0); +}); + +Deno.test("deduplicateAndSortTools should deduplicate by toolId and keep highest score", () => { + const allTools: SearchResult[] = [ + { + tool: { + toolId: "tool1", + serverName: "server1", + toolName: "tool1", + description: "", + inputSchema: {}, + }, + score: 0.8, + scoreBreakdown: { semantic: 0.7, fuzzy: 0.9, combined: 0.8 }, + }, + { + tool: { + toolId: "tool1", + serverName: "server1", + toolName: "tool1", + description: "", + inputSchema: {}, + }, + score: 0.9, // Higher score + scoreBreakdown: { semantic: 0.85, fuzzy: 0.95, combined: 0.9 }, + }, + { + tool: { + toolId: "tool2", + serverName: "server2", + toolName: "tool2", + description: "", + inputSchema: {}, + }, + score: 0.7, + scoreBreakdown: { semantic: 0.6, fuzzy: 0.8, combined: 0.7 }, + }, + ]; + + const result = deduplicateAndSortTools(allTools); + + assertEquals(result.length, 2); + + // tool1 should have the higher score (0.9) + const tool1 = result.find((t) => t.tool.toolId === "tool1"); + assertExists(tool1); + assertEquals(tool1.score, 0.9); +}); + +Deno.test("deduplicateAndSortTools should sort by score and limit results", () => { + const tools: SearchResult[] = [ + { + tool: { + toolId: "tool1", + serverName: "server1", + toolName: "tool1", + description: "", + inputSchema: {}, + }, + score: 0.5, + scoreBreakdown: { semantic: 0.4, fuzzy: 0.6, combined: 0.5 }, + }, + { + tool: { + toolId: "tool2", + serverName: "server2", + toolName: "tool2", + description: "", + inputSchema: {}, + }, + score: 0.9, + scoreBreakdown: { semantic: 0.85, fuzzy: 0.95, combined: 0.9 }, + }, + { + tool: { + toolId: "tool3", + serverName: "server3", + toolName: "tool3", + description: "", + inputSchema: {}, + }, + score: 0.7, + scoreBreakdown: { semantic: 0.65, fuzzy: 0.75, combined: 0.7 }, + }, + ]; + + const result = deduplicateAndSortTools(tools, 2); + + assertEquals(result.length, 2); + assertEquals(result[0].tool.toolId, "tool2"); // Highest score + assertEquals(result[1].tool.toolId, "tool3"); // Second highest +}); diff --git a/src/cli/commands/context/claude-usage-suggestion.ts b/src/cli/commands/context/claude-usage-suggestion.ts index 987a89e..e6f5762 100644 --- a/src/cli/commands/context/claude-usage-suggestion.ts +++ b/src/cli/commands/context/claude-usage-suggestion.ts @@ -39,10 +39,34 @@ async function searchGatewayForTools( } } +/** + * Deduplicate and sort tools by score + * Keeps the highest score for each unique toolId and returns top N results + */ +export function deduplicateAndSortTools( + tools: SearchResult[], + limit: number = 5, +): SearchResult[] { + // Deduplicate tools by toolId, keeping the highest score for each tool + const toolMap = new Map(); + for (const tool of tools) { + const existing = toolMap.get(tool.tool.toolId); + // Keep this tool if it's new OR has a higher score than existing + if (!existing || tool.score > existing.score) { + toolMap.set(tool.tool.toolId, tool); + } + } + + // Sort by confidence score (highest first) and take top N + return Array.from(toolMap.values()) + .sort((a, b) => b.score - a.score) + .slice(0, limit); +} + /** * Format context text from skills and tools */ -function formatContextText(skills: string[], tools: SearchResult[]): string { +export function formatContextText(skills: string[], tools: SearchResult[]): string { const parts: string[] = []; if (skills.length > 0) { @@ -65,7 +89,7 @@ ${skills.map((s) => `- Skill("${s}")`).join("\n")}`, /** * Create hook JSON response */ -function createHookResponse(contextText: string): HookJSONOutput { +export function createHookResponse(contextText: string): HookJSONOutput { if (contextText) { return { hookSpecificOutput: { @@ -107,20 +131,8 @@ export const claudeUsageSuggestionCommand = new Command() const toolResults = await Promise.all(toolSearches); allTools = toolResults.flat(); - // Deduplicate tools by toolId, keeping the highest score for each tool - const toolMap = new Map(); - for (const tool of allTools) { - const existing = toolMap.get(tool.tool.toolId); - // Keep this tool if it's new OR has a higher score than existing - if (!existing || tool.score > existing.score) { - toolMap.set(tool.tool.toolId, tool); - } - } - - // Sort by confidence score (highest first) and take top 5 - allTools = Array.from(toolMap.values()) - .sort((a, b) => b.score - a.score) - .slice(0, 5); + // Deduplicate and sort tools + allTools = deduplicateAndSortTools(allTools, 5); logger.debug(`Found ${allTools.length} unique tools after deduplication`); } diff --git a/src/utils/skill-discovery.test.ts b/src/utils/skill-discovery.test.ts new file mode 100644 index 0000000..6c1a7ce --- /dev/null +++ b/src/utils/skill-discovery.test.ts @@ -0,0 +1,260 @@ +/** + * Tests for skill discovery utilities + */ + +import { assertEquals, assertExists } from "@std/assert"; +import { + discoverAllSkills, + loadInstalledPlugins, + mergeSkills, + parseSkillDescription, + scanGlobalSkills, + scanPluginSkills, + scanProjectSkills, + type Skill, +} from "./skill-discovery.ts"; +import { join } from "@std/path"; + +Deno.test("parseSkillDescription should parse valid YAML frontmatter", () => { + const content = `--- +name: test-skill +description: A test skill for testing +--- + +# Test Skill + +This is the skill content.`; + + const result = parseSkillDescription(content); + assertExists(result); + assertEquals(result.name, "test-skill"); + assertEquals(result.description, "A test skill for testing"); +}); + +Deno.test("parseSkillDescription should return null for invalid frontmatter", () => { + const content = `# Test Skill + +No frontmatter here.`; + + const result = parseSkillDescription(content); + assertEquals(result, null); +}); + +Deno.test("parseSkillDescription should return null for incomplete frontmatter", () => { + const content = `--- +name: test-skill +--- + +Missing description.`; + + const result = parseSkillDescription(content); + assertEquals(result, null); +}); + +Deno.test("parseSkillDescription should handle multiline descriptions", () => { + const content = `--- +name: test-skill +description: A test skill with a long description +--- + +# Test Skill`; + + const result = parseSkillDescription(content); + assertExists(result); + assertEquals(result.name, "test-skill"); + assertEquals(result.description, "A test skill with a long description"); +}); + +Deno.test("mergeSkills should deduplicate by name", () => { + const skills: Skill[][] = [ + [ + { + name: "skill1", + description: "Plugin skill", + source: "plugin", + pluginName: "test-plugin", + }, + ], + [{ name: "skill1", description: "Global skill", source: "global" }], + ]; + + const result = mergeSkills(skills); + assertEquals(result.length, 1); + assertEquals(result[0].description, "Global skill"); + assertEquals(result[0].source, "global"); +}); + +Deno.test("mergeSkills should prioritize project > global > plugin", () => { + const skills: Skill[][] = [ + [ + { + name: "skill1", + description: "Plugin skill", + source: "plugin", + pluginName: "test-plugin", + }, + ], + [{ name: "skill1", description: "Global skill", source: "global" }], + [{ name: "skill1", description: "Project skill", source: "project" }], + ]; + + const result = mergeSkills(skills); + assertEquals(result.length, 1); + assertEquals(result[0].description, "Project skill"); + assertEquals(result[0].source, "project"); +}); + +Deno.test("mergeSkills should keep all unique skills", () => { + const skills: Skill[][] = [ + [ + { name: "skill1", description: "Plugin skill 1", source: "plugin" }, + { name: "skill2", description: "Plugin skill 2", source: "plugin" }, + ], + [{ name: "skill3", description: "Global skill", source: "global" }], + [{ name: "skill4", description: "Project skill", source: "project" }], + ]; + + const result = mergeSkills(skills); + assertEquals(result.length, 4); +}); + +Deno.test("mergeSkills should handle empty arrays", () => { + const skills: Skill[][] = [[], [], []]; + const result = mergeSkills(skills); + assertEquals(result.length, 0); +}); + +Deno.test("scanGlobalSkills should return empty array when no home directory", async () => { + // Save original env + const originalHome = Deno.env.get("HOME"); + const originalUserProfile = Deno.env.get("USERPROFILE"); + + try { + // Clear HOME and USERPROFILE + Deno.env.delete("HOME"); + Deno.env.delete("USERPROFILE"); + + const result = await scanGlobalSkills(); + assertEquals(result.length, 0); + } finally { + // Restore original env + if (originalHome) Deno.env.set("HOME", originalHome); + if (originalUserProfile) Deno.env.set("USERPROFILE", originalUserProfile); + } +}); + +Deno.test("scanProjectSkills should scan from current working directory", async () => { + // This test just verifies it doesn't throw and returns an array + const result = await scanProjectSkills(); + assertEquals(Array.isArray(result), true); +}); + +Deno.test("loadInstalledPlugins should return empty array when no plugins file", async () => { + // Save original env + const originalHome = Deno.env.get("HOME"); + + try { + // Set HOME to a non-existent directory + const tempDir = await Deno.makeTempDir(); + Deno.env.set("HOME", tempDir); + + const result = await loadInstalledPlugins(); + assertEquals(result.length, 0); + + // Cleanup + await Deno.remove(tempDir, { recursive: true }); + } finally { + // Restore original env + if (originalHome) Deno.env.set("HOME", originalHome); + } +}); + +Deno.test("scanPluginSkills should handle plugin with no skills directory", async () => { + const tempDir = await Deno.makeTempDir(); + + try { + const plugin = { + name: "test-plugin", + installPath: tempDir, + }; + + const result = await scanPluginSkills(plugin); + assertEquals(result.length, 0); + } finally { + await Deno.remove(tempDir, { recursive: true }); + } +}); + +Deno.test("scanPluginSkills should discover skills from plugin directory", async () => { + const tempDir = await Deno.makeTempDir(); + + try { + // Create a skill directory structure + const skillsDir = join(tempDir, "skills", "test-skill"); + await Deno.mkdir(skillsDir, { recursive: true }); + + // Create a SKILL.md file + const skillContent = `--- +name: test-plugin-skill +description: A skill from a test plugin +--- + +# Test Plugin Skill`; + + await Deno.writeTextFile(join(skillsDir, "SKILL.md"), skillContent); + + const plugin = { + name: "test-plugin", + installPath: tempDir, + }; + + const result = await scanPluginSkills(plugin); + assertEquals(result.length, 1); + assertEquals(result[0].name, "test-plugin-skill"); + assertEquals(result[0].description, "A skill from a test plugin"); + assertEquals(result[0].source, "plugin"); + assertEquals(result[0].pluginName, "test-plugin"); + } finally { + await Deno.remove(tempDir, { recursive: true }); + } +}); + +Deno.test("discoverAllSkills should merge all skill sources", async () => { + // This is an integration test that just verifies the function works + const result = await discoverAllSkills(); + assertEquals(Array.isArray(result), true); + // Verify all results have required fields + for (const skill of result) { + assertExists(skill.name); + assertExists(skill.description); + assertExists(skill.source); + } +}); + +Deno.test("parseSkillDescription should handle extra whitespace", () => { + const content = `--- +name: test-skill +description: A test skill with extra whitespace +--- + +# Test Skill`; + + const result = parseSkillDescription(content); + assertExists(result); + assertEquals(result.name, "test-skill"); + assertEquals(result.description, "A test skill with extra whitespace"); +}); + +Deno.test("parseSkillDescription should handle content with tabs", () => { + const content = `--- +name:\ttest-skill +description:\tA test skill with tabs +--- + +# Test Skill`; + + const result = parseSkillDescription(content); + assertExists(result); + assertEquals(result.name, "test-skill"); + assertEquals(result.description, "A test skill with tabs"); +}); From 9e004638d5b31c2bc5913a453067ffc1ee34cba7 Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Tue, 2 Dec 2025 10:49:49 +0100 Subject: [PATCH 4/6] docs: update spec with impl learnings --- .../specs/cli-interface/spec.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md b/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md index 6cf03e6..3fa6dfe 100644 --- a/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md +++ b/openspec/changes/implement-intelligent-context-injection/specs/cli-interface/spec.md @@ -31,11 +31,11 @@ The CLI SHALL provide a command under the context group for LLM-based usage sugg #### Scenario: Scan installed plugins - **WHEN** command executes -- **THEN** command reads `~/.claude/plugins/installed_plugins.json` to discover all installed plugins +- **THEN** command reads `~/.claude/settings.json` to discover enabled plugins only -#### Scenario: Discover skills from all plugins -- **WHEN** plugin paths are known -- **THEN** command scans `{plugin_path}/skills/*/SKILL.md` for each plugin to find available skills +#### Scenario: Discover skills from enabled plugins +- **WHEN** enabled plugin paths are known +- **THEN** command scans `{plugin_path}/skills/*/SKILL.md` for each enabled plugin to find available skills #### Scenario: Extract skill descriptions - **WHEN** SKILL.md file is read @@ -57,9 +57,9 @@ The CLI SHALL provide a command under the context group for LLM-based usage sugg - **WHEN** Agent SDK query is created - **THEN** model is set to "haiku" to use user's configured model for this category -#### Scenario: Load SDK config sources +#### Scenario: Load agent environment variables - **WHEN** Agent SDK query is created -- **THEN** `settingSources` includes ["user", "project", "local"] for full config support +- **THEN** only environment variables configured in the user's `~/.claude/settings.json` are passed to the agent #### Scenario: Disable tools for agent - **WHEN** Agent SDK query is created @@ -75,7 +75,7 @@ The CLI SHALL provide a command under the context group for LLM-based usage sugg #### Scenario: Timeout protection - **WHEN** LLM call takes too long -- **THEN** command aborts after 5 seconds and returns empty suggestions +- **THEN** command aborts after 8 seconds and returns empty suggestions #### Scenario: API error handling - **WHEN** Claude API returns error response From bac952158df4b08b112ed92a751492552a5430e6 Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Tue, 2 Dec 2025 12:09:13 +0100 Subject: [PATCH 5/6] chore: address review comments --- .../design.md | 12 ++++++------ plugins/toolscript/scripts/user-prompt-submit.sh | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/openspec/changes/implement-intelligent-context-injection/design.md b/openspec/changes/implement-intelligent-context-injection/design.md index 4e1ce55..eeec669 100644 --- a/openspec/changes/implement-intelligent-context-injection/design.md +++ b/openspec/changes/implement-intelligent-context-injection/design.md @@ -90,20 +90,20 @@ toolscript context claude-usage-suggestion: * systemPrompt: (custom, not Claude Code's) * allowedTools: [] * Single iteration (for await one response) -4. Parse LLM response (JSON with skills and toolQueries arrays) -5. If --gateway-url provided: +3. Parse LLM response (JSON with skills and toolQueries arrays) +4. If --gateway-url provided: - For each toolQuery: - Search gateway for matching tools - Collect tool names and descriptions -6. Format context text: +5. Format context text: - Skills: Instruct to use Skill(name) for each - Tools: Instruct to use toolscript skill with tool names - Combined: Unified message if both present -7. Wrap in hook JSON: +6. Wrap in hook JSON: - If context: `{"hookSpecificOutput":{"additionalContext":"formatted text"}}` - If no context: `{}` -8. Output JSON to stdout -9. Exit +7. Output JSON to stdout +8. Exit ``` ## Key Design Decisions diff --git a/plugins/toolscript/scripts/user-prompt-submit.sh b/plugins/toolscript/scripts/user-prompt-submit.sh index 283daa2..a6d7eb5 100755 --- a/plugins/toolscript/scripts/user-prompt-submit.sh +++ b/plugins/toolscript/scripts/user-prompt-submit.sh @@ -36,4 +36,4 @@ fi # 5. Output hook JSON response toolscript context claude-usage-suggestion \ --prompt "$USER_PROMPT" \ - --gateway-url "$GATEWAY_URL" 2>/dev/null || exit 0 + --gateway-url "$GATEWAY_URL" 2>> "${TMPDIR}toolscript-gateway-${SESSION_ID}.log" || exit 0 From e50e889cc95d639751c7e16dbbdfab25c003e595 Mon Sep 17 00:00:00 2001 From: Heiko Rothe Date: Tue, 2 Dec 2025 12:17:12 +0100 Subject: [PATCH 6/6] chore: refine skill instructions --- plugins/toolscript/skills/toolscript/SKILL.md | 6 +++++- .../skills/toolscript/references/commands.md | 9 +++++---- .../skills/toolscript/references/examples.md | 10 +++++++--- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/plugins/toolscript/skills/toolscript/SKILL.md b/plugins/toolscript/skills/toolscript/SKILL.md index 3de3ca3..dbb3df5 100644 --- a/plugins/toolscript/skills/toolscript/SKILL.md +++ b/plugins/toolscript/skills/toolscript/SKILL.md @@ -16,7 +16,11 @@ Discover and execute MCP tools through the toolscript gateway. toolscript search "what you need" --output types # 2. Execute the generated code -toolscript exec '' +# Single-line code: use exec directly +toolscript exec '' + +# Multi-line code: write to temp file and use -f flag +toolscript exec -f /tmp/script.ts ``` ## Toolscript Format diff --git a/plugins/toolscript/skills/toolscript/references/commands.md b/plugins/toolscript/skills/toolscript/references/commands.md index d1603f4..3981afe 100644 --- a/plugins/toolscript/skills/toolscript/references/commands.md +++ b/plugins/toolscript/skills/toolscript/references/commands.md @@ -92,14 +92,15 @@ This shows the exact TypeScript interface for the tool's parameters and return v After understanding the tool schema, execute it: -**Inline code:** +**Single-line code (use exec directly):** ```bash -toolscript exec '' +toolscript exec '' ``` -**From file:** +**Multi-line code (write to temp file and use -f flag):** ```bash -toolscript exec --file +# Write code to temp file first +toolscript exec -f /tmp/script.ts ``` ## 6. Gateway Status (Diagnostic) diff --git a/plugins/toolscript/skills/toolscript/references/examples.md b/plugins/toolscript/skills/toolscript/references/examples.md index 900ea62..2f2d44b 100644 --- a/plugins/toolscript/skills/toolscript/references/examples.md +++ b/plugins/toolscript/skills/toolscript/references/examples.md @@ -8,8 +8,11 @@ # Search for tools and get TypeScript code toolscript search "commit git changes" --output types -# Execute the generated code -toolscript exec '' +# Execute single-line code directly +toolscript exec '' + +# For multi-line code: write to temp file and use -f flag +toolscript exec -f /tmp/script.ts ``` Use `--threshold 0.1` for more results: @@ -23,7 +26,8 @@ If you already know which MCP tools to use: ```bash toolscript get-types --filter git_commit # Get specific tool's TypeScript types -toolscript exec '' # Execute inline +toolscript exec '' # Execute single-line code inline +toolscript exec -f /tmp/script.ts # Execute multi-line code from file ``` ### Alternative: Browse-Based Discovery