feat: Implement TOON (Token-Oriented Object Notation) for LLM data serialization #1480
Conversation
🦋 Changeset detectedLatest commit: 1d27201 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
WalkthroughThis PR introduces TOON (Token-Oriented Object Notation) serialization for LLM data to reduce token usage by 30-60%, along with cleanup of pre-release changeset files as part of transitioning to stable version 0.37.0. The implementation includes serialization utilities, LLM provider enhancement, AI services integration layer, CLI tooling, and comprehensive documentation. Changes
Sequence DiagramsequenceDiagram
participant Client as Client / CLI
participant AI as AI Services Unified
participant Factory as ToonProviderFactory
participant Adapter as Adapter Layer<br/>(LLM Provider)
participant Analyzer as TOON Analyzer
participant Serializer as TOON Serializer
participant LLM as LLM (Claude/etc)
Client->>AI: Call AI command<br/>(e.g., generateText)
AI->>Factory: getEnhancedProvider(name, provider)
alt Provider cached?
Factory-->>AI: Return cached enhanced provider
else New provider
Factory->>Adapter: enhanceProviderWithToon(provider)
Adapter->>Analyzer: analyzeToonSuitability(data)
Analyzer-->>Adapter: {suitable: true, savings: X%}
Adapter-->>Factory: Return wrapped provider
Factory->>Factory: Cache enhanced provider
Factory-->>AI: Return enhanced provider
end
AI->>Adapter: generateText(params)
Adapter->>Analyzer: analyzeToonSuitability(messages)
alt TOON beneficial & enabled?
Analyzer-->>Adapter: Yes, proceed
Adapter->>Serializer: jsonToToon(messageContent)
Serializer-->>Adapter: TOON-encoded data
Adapter->>Adapter: Inject TOON system prompt
else JSON fallback
Analyzer-->>Adapter: No/disabled
Adapter->>Adapter: Use original JSON
end
Adapter->>LLM: Call with data<br/>(JSON or TOON format)
LLM-->>Adapter: Response tokens
alt TOON was used
Adapter->>Serializer: toonToJson(response)
Serializer-->>Adapter: Parsed JSON result
else JSON response
Adapter->>Adapter: Use response as-is
end
Adapter-->>AI: Return result<br/>+ token savings metadata
AI-->>Client: Final result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Areas requiring extra attention:
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
…rialization Add comprehensive TOON format implementation that reduces LLM token usage by 30-60%: - Core TOON serializer with JSON ↔ TOON conversion and validation - LLM provider integration layer with automatic suitability analysis - Enhanced provider factory with caching and performance optimization - AI services integration hooks for seamless workflow compatibility - CLI utility for enable/disable, testing, and file conversion - Comprehensive documentation and usage guidelines - Full test suite for serialization functions Key features: - Intelligent data analysis (only uses TOON when beneficial) - Configurable thresholds for data size and savings percentage - Automatic fallback to JSON for unsuitable structures - Zero-config integration with existing Task Master workflows - 100% backward compatibility Resolves #1479 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Ralph Khreish <Crunchyman-ralph@users.noreply.github.com>
Crunchyman-ralph
force-pushed
the
claude/issue-1479-20251203-1231
branch
from
4601fe8 to
1d27201
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 18
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
apps/mcp/CHANGELOG.md (1)
3-43: Invalid changelog structure: "## null" is not a valid version identifier.All entries use "## null" as a version header, which violates changelog conventions and is not a valid semantic version. This appears to be an auto-generated placeholder from the changeset process that was never filled in with actual version data. Additionally, the dependency entry
@tm/core@nullis invalid.These entries must be removed or replaced with proper semantic versions (e.g.,
## 0.37.0,## 1.0.0) or an explicit## Unreleasedsection if appropriate. Verify the changeset configuration and version setup inpackage.jsonfor this package.packages/tm-bridge/CHANGELOG.md (1)
3-29: Invalid changelog structure: "## null" entries throughout file.Same issue as in apps/mcp/CHANGELOG.md—all version headers are "## null", which is not a valid semantic version. For an internal package (
@tm/bridge), verify whether this changelog should contain version entries at all, or if it should be cleaned to show only legitimate version milestones. Remove or replace all "## null" entries with proper versions or## Unreleased.apps/cli/CHANGELOG.md (1)
17-24: Multiple critical issues: invalid version headers and dependency versions.This section adds a new "## null" entry (lines 17–24) with an invalid dependency version
@tm/core@null. The file contains numerous "## null" entries throughout (lines 3–16 plus additional ones at lines 26, 33, 40, 49, 56, 63), all of which are invalid semantic versions. Additionally, the dependency entries reference@tm/core@null, which is not a valid version.For a public app changelog (apps/cli), these entries must be replaced with proper versions (e.g.,
## 0.37.0) and dependencies must reference actual released versions (e.g.,@tm/core@0.26.1). This pattern suggests the changeset automation failed to assign proper versions. Remove all "## null" entries and verify the changeset release configuration.
🧹 Nitpick comments (7)
.changeset/toon-serialization.md (2)
5-5: Use a single-line imperative changeset summary (drop “feat:” prefix).Keep the detailed body, but make the first line a concise, user-facing summary per repo rules.
Apply:
- feat: Add TOON (Token-Oriented Object Notation) for LLM data serialization + Add TOON (Token-Oriented Object Notation) serialization with provider integration and CLIBased on learnings, provide a concise single-line changeset summary.
21-21: Align CLI usage with existing command naming (task-master/tm) instead of node script.Project UX consistently surfaces CLI via
task-master/tm. Recommend exposing TOON through the main CLI and reflecting that here.Example:
- Usage: `node scripts/toon-cli.js enable --min-savings 10` + Usage: `tm toon enable --min-savings 10` + Alias: `task-master toon enable --min-savings 10`If the command is already wired, update docs; if not, add a thin CLI subcommand that forwards to the same implementation. Based on learnings, focus on functionality over cosmetic formatting.
src/serialization/toon-serializer.spec.js (1)
173-208: Relax hardcoded 20% savings threshold for complex dataThe
complex data structurestest assertssavings.savingsPercentageis> 20. That’s reasonable for the current serializer, but it’s brittle: small tweaks to the TOON encoding could drop savings slightly below 20% while still being correct and beneficial.Consider lowering this threshold or just asserting
> 0(or>= 10) and optionally snapshotting the detailedsavingsobject if you want to monitor changes without making the test flaky.src/serialization/llm-toon-adapter.js (2)
9-9: Unused import:validateToonRoundTrip.The
validateToonRoundTripfunction is imported but never used in this module. Remove it to avoid dead code.-import { jsonToToon, toonToJson, estimateTokenSavings, validateToonRoundTrip } from './toon-serializer.js'; +import { jsonToToon, estimateTokenSavings } from './toon-serializer.js';
15-38: Exported mutable config object may lead to inconsistent state.
TOON_CONFIGis exported directly and can be mutated from anywhere. WhileenableToon/disableToonprovide controlled mutation, external code could bypass these and mutate the object directly, leading to inconsistent state. Consider making the config private and exposing only getter/setter functions.src/ai-providers/toon-enhanced-provider.js (1)
37-80: Cache may serve stale enhanced providers after config changes.The factory caches enhanced providers by name but doesn't invalidate when TOON configuration changes. If
enableToon()is called with new config values, previously cached providers will still use the old enhancement. Consider either:
- Documenting that
clearCache()must be called after config changes, or- Storing a config hash with each cached provider to detect staleness.
src/serialization/toon-ai-services-integration.js (1)
199-203: Unnecessary dynamic import; these functions could be imported statically.
analyzeToonSuitabilityandestimateTokenSavingsare already available via./index.js. Using dynamic import here adds unnecessary overhead. Consider adding static imports at the top of the file.Add to the imports at the top:
import { ToonProviderFactory } from '../ai-providers/toon-enhanced-provider.js'; -import { enableToon, disableToon, getToonConfig } from './index.js'; +import { enableToon, disableToon, getToonConfig, analyzeToonSuitability, estimateTokenSavings } from './index.js'; import { log } from '../../scripts/modules/utils.js';Then remove the dynamic import:
- // Import suitability analysis - const { analyzeToonSuitability, estimateTokenSavings } = await import('./index.js'); - const suitability = analyzeToonSuitability(sampleTaskData);
| ## 0.37.0 | ||
|
|
||
| ### Minor Changes | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for TOON references and check changelog structure
echo "=== CHANGELOG.md structure (first 50 lines) ==="
head -50 CHANGELOG.md
echo -e "\n=== Searching for TOON references in docs ==="
fd -i 'toon' docs -type f 2>/dev/null | head -20 || echo "No TOON docs found"
echo -e "\n=== Searching for TOON in source code ==="
rg -l -i 'toon' src --type ts --type js 2>/dev/null | head -20 || echo "No TOON in source"
echo -e "\n=== Searching for TOON in CLI/scripts ==="
rg -l -i 'toon' scripts apps -i 2>/dev/null | head -20 || echo "No TOON in CLI"
echo -e "\n=== Checking for changeset.mdc ==="
find . -name 'changeset.mdc' -o -name '*changeset*' -type f | grep -E '\.(md|mdc)$' | head -10
echo -e "\n=== Checking package.json for TOON references ==="
rg -i 'toon' package.json -C 2 || echo "No TOON in package.json"
echo -e "\n=== Repository structure ==="
ls -la | head -20Repository: eyaltoledano/claude-task-master
Length of output: 4661
🏁 Script executed:
#!/bin/bash
# Check full CHANGELOG and changeset guidelines
echo "=== Full CHANGELOG.md (first 200 lines) ==="
head -200 CHANGELOG.md
echo -e "\n=== changeset.mdc content ==="
cat ./.cursor/rules/changeset.mdc
echo -e "\n=== Check for TOON in .changeset directory ==="
ls -la .changeset/
cat .changeset/*.md 2>/dev/null | head -100 || echo "No pending changesets"
echo -e "\n=== TOON file details ==="
wc -l src/serialization/toon-serializer.js src/ai-providers/toon-enhanced-provider.js scripts/toon-cli.js
echo -e "\n=== Check if TOON is exported from main index ==="
rg -n 'toon' src/serialization/index.js -C 3
rg -n 'toon' src/ai-providers/index.js -C 3 2>/dev/null || echo "No ai-providers index"
echo -e "\n=== Check for TOON in README or docs ==="
find . -name 'README*' -type f | xargs rg -l -i 'toon' 2>/dev/null || echo "TOON not in README"Repository: eyaltoledano/claude-task-master
Length of output: 24140
🏁 Script executed:
#!/bin/bash
# Check pending changeset and TOON documentation
echo "=== .changeset/toon-serialization.md content ==="
cat .changeset/toon-serialization.md
echo -e "\n=== Check for TOON docs directory ==="
find . -type d -name '*toon*' 2>/dev/null
echo -e "\n=== Search for TOON in docs folder ==="
find docs -type f \( -name '*.md' -o -name '*.mdx' \) 2>/dev/null | head -20
rg -l -i 'toon' docs/ 2>/dev/null || echo "TOON not in docs"
echo -e "\n=== Check TOON CLI for documentation ==="
head -50 scripts/toon-cli.js
echo -e "\n=== Search for GUIDE or tutorial mentioning TOON ==="
find . -name '*guide*' -o -name '*tutorial*' -type f | xargs rg -l -i 'toon' 2>/dev/null || echo "No guides found"Repository: eyaltoledano/claude-task-master
Length of output: 3265
🏁 Script executed:
#!/bin/bash
# Verify TOON guide and discoverability
echo "=== toon-integration-guide.md exists and content preview ==="
wc -l docs/toon-integration-guide.md
head -80 docs/toon-integration-guide.md
echo -e "\n=== Check if TOON is mentioned in docs/README.md ==="
cat docs/README.md | grep -i toon || echo "TOON not mentioned in docs README"
echo -e "\n=== Check if toon-integration-guide is linked from anywhere ==="
rg -n 'toon-integration-guide' --type md . || echo "Guide not linked from markdown files"
echo -e "\n=== Check main README.md for TOON reference ==="
head -200 README.md | grep -i toon || echo "TOON not in main README"
echo -e "\n=== Verify changeset summary matches guide scope ==="
echo "Changeset mentions:"
cat .changeset/toon-serialization.mdRepository: eyaltoledano/claude-task-master
Length of output: 3757
Link TOON integration guide from README.md and docs/README.md for discoverability.
The TOON integration guide (docs/toon-integration-guide.md) is comprehensive and the CLI commands are implemented, but the guide is not discoverable from primary documentation entry points. Per coding guidelines, update README.md and docs/README.md to reference the TOON integration guide so users can find it after the release. The changeset will generate the changelog entry automatically during the release process.
🤖 Prompt for AI Agents
In CHANGELOG.md around lines 3 to 6, the TOON integration guide is not
referenced from primary docs; update README.md and docs/README.md to add
discoverable links to docs/toon-integration-guide.md (use correct relative
paths), placing a short entry under the Guides/Integrations section in each file
(e.g., "TOON integration guide — docs/toon-integration-guide.md") so the guide
is visible from both the repo root README and docs index; ensure link text is
clear and the path works from each file's location.
| ### Minor Changes | ||
|
|
||
| - [#1452](https://github.com/eyaltoledano/claude-task-master/pull/1452) [`4046b3c`](https://github.com/eyaltoledano/claude-task-master/commit/4046b3ca4479adf0239679eb5ba18b7b4aec0749) Thanks [@eyaltoledano](https://github.com/eyaltoledano)! - Add Hamster integration for `parse-prd` command |
There was a problem hiding this comment.
Add a 0.37.0 “Minor Changes” entry for TOON (PR #1480) using project’s changelog format.
This release notes block is missing the TOON feature that this PR introduces. Add a bullet following the repo’s “[#PR] Thanks [author]! - Description” convention.
Proposed insertion right after the section header:
### Minor Changes
+- [#1480](https://github.com/eyaltoledano/claude-task-master/pull/1480) [`<commit>`](https://github.com/eyaltoledano/claude-task-master/commit/<commit>) Thanks [@Crunchyman-ralph](https://github.com/Crunchyman-ralph)! - Introduce TOON (Token‑Oriented Object Notation) serialization with JSON↔TOON helpers, provider suitability auto‑switching, and a CLI to enable/disable and convert data (30–60% token savings on suitable payloads).
+Replace with the merge commit SHA. As per changelog rules, keep it single-line at the bullet level and move detail to docs. Based on learnings, the project parses this format programmatically.
Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In CHANGELOG.md around lines 5 to 7, the "Minor Changes" section is missing the
0.37.0 TOON entry (PR #1480); add a single-line bullet immediately after the
"### Minor Changes" header following the repository's convention: -
[#1480](https://github.com/eyaltoledano/claude-task-master/pull/1480)
[`<merge-commit-sha>`](https://github.com/eyaltoledano/claude-task-master/commit/<merge-commit-sha>)
Thanks [@TOON](https://github.com/TOON)! - Add TOON integration for `parse-prd`
command, replacing <merge-commit-sha> with the actual merge commit SHA and
keeping the bullet to one line only.
| # TOON (Token-Oriented Object Notation) Integration Guide | ||
|
|
||
| ## Overview | ||
|
|
||
| TOON (Token-Oriented Object Notation) is a compact, schema-aware format that reduces LLM token usage by 30-60% versus standard JSON by eliminating syntactic overhead like braces, quotes, and repeated fields. | ||
|
|
||
| This implementation provides a serialization layer that converts JSON ↔ TOON at the LLM provider boundary, reducing token costs and latency while maintaining compatibility with existing Task Master workflows. | ||
|
|
||
| ## Benefits | ||
|
|
||
| - **30-60% token reduction** for structured data | ||
| - **Lower latency** due to smaller payload sizes | ||
| - **Cost savings** on LLM API calls | ||
| - **Seamless integration** with existing JSON workflows | ||
| - **Automatic fallback** to JSON for unsuitable data | ||
|
|
||
| ## Architecture | ||
|
|
||
| ### Core Components | ||
|
|
||
| 1. **TOON Serializer** (`src/serialization/toon-serializer.js`) | ||
| - Core conversion functions: `jsonToToon()`, `toonToJson()` | ||
| - Token savings estimation | ||
| - Round-trip validation | ||
|
|
||
| 2. **LLM Adapter** (`src/serialization/llm-toon-adapter.js`) | ||
| - Suitability analysis for data structures | ||
| - Provider wrapping for automatic TOON usage | ||
| - Configuration management | ||
|
|
||
| 3. **Provider Enhancement** (`src/ai-providers/toon-enhanced-provider.js`) | ||
| - Factory for creating TOON-enhanced providers | ||
| - Caching and performance optimization | ||
|
|
||
| 4. **AI Services Integration** (`src/serialization/toon-ai-services-integration.js`) | ||
| - Integration hooks for existing AI services | ||
| - Dynamic provider enhancement | ||
|
|
||
| ## TOON Format Specification | ||
|
|
||
| ### Basic Rules | ||
|
|
||
| - **Objects**: `{key:value key2:value2}` (no quotes around keys unless containing spaces) | ||
| - **Arrays**: `[item1 item2 item3]` (space-separated items) | ||
| - **Strings**: Only quoted if containing spaces or special characters | ||
| - **Numbers**: Raw numeric values | ||
| - **Booleans**: `true` / `false` | ||
| - **Null**: `null` | ||
|
|
||
| ### Examples | ||
|
|
||
| ```javascript | ||
| // JSON | ||
| { | ||
| "users": [ | ||
| {"id": 1, "name": "John", "active": true}, | ||
| {"id": 2, "name": "Jane", "active": false} | ||
| ], | ||
| "total": 2 | ||
| } | ||
|
|
||
| // TOON | ||
| {users:[{id:1 name:John active:true} {id:2 name:Jane active:false}] total:2} | ||
| ``` | ||
|
|
||
| ## Usage Guide | ||
|
|
||
| ### Command Line Interface | ||
|
|
||
| ```bash | ||
| # Enable TOON integration | ||
| node scripts/toon-cli.js enable --min-size 100 --min-savings 10 | ||
|
|
||
| # Check status | ||
| node scripts/toon-cli.js status | ||
|
|
||
| # Test with sample data | ||
| node scripts/toon-cli.js test --enable-first | ||
|
|
||
| # Convert JSON file to TOON | ||
| node scripts/toon-cli.js convert data.json -o data.toon | ||
|
|
||
| # Disable TOON integration | ||
| node scripts/toon-cli.js disable | ||
| ``` | ||
|
|
||
| ### Programmatic Usage | ||
|
|
||
| ```javascript | ||
| import { enableToonForAIServices, testToonWithTaskData } from './src/serialization/toon-ai-services-integration.js'; | ||
|
|
||
| // Enable TOON for all AI providers | ||
| await enableToonForAIServices({ | ||
| minDataSize: 100, // Only use TOON for data >= 100 chars | ||
| minSavingsThreshold: 10 // Only use TOON if >= 10% savings expected | ||
| }); | ||
|
|
||
| // Test with sample task data | ||
| const results = await testToonWithTaskData(); | ||
| console.log('Token savings:', results.savings.estimatedTokenSavingsPercentage + '%'); | ||
| ``` | ||
|
|
||
| ### Manual TOON Conversion | ||
|
|
||
| ```javascript | ||
| import { jsonToToon, toonToJson, estimateTokenSavings } from './src/serialization/index.js'; | ||
|
|
||
| const data = { tasks: [{ id: 1, title: 'Task 1', status: 'pending' }] }; | ||
|
|
||
| // Convert to TOON | ||
| const toonData = jsonToToon(data); | ||
| console.log('TOON:', toonData); | ||
| // Output: {tasks:[{id:1 title:"Task 1" status:pending}]} | ||
|
|
||
| // Convert back to JSON | ||
| const jsonData = toonToJson(toonData); | ||
| console.log('JSON:', jsonData); | ||
|
|
||
| // Estimate savings | ||
| const savings = estimateTokenSavings(data); | ||
| console.log(`Estimated token savings: ${savings.estimatedTokenSavingsPercentage}%`); | ||
| ``` | ||
|
|
||
| ## Configuration Options | ||
|
|
||
| ### Global TOON Configuration | ||
|
|
||
| ```javascript | ||
| const TOON_CONFIG = { | ||
| enabled: false, // Enable/disable globally | ||
| minDataSize: 100, // Minimum chars to consider TOON | ||
| minSavingsThreshold: 10, // Minimum % savings to use TOON | ||
| preferredStructures: [ // Data types that work well with TOON | ||
| 'arrays_of_objects', | ||
| 'flat_objects', | ||
| 'uniform_data' | ||
| ], | ||
| avoidStructures: [ // Data types to avoid with TOON | ||
| 'deeply_nested', | ||
| 'sparse_objects', | ||
| 'mixed_types' | ||
| ] | ||
| }; | ||
| ``` | ||
|
|
||
| ## Data Suitability Analysis | ||
|
|
||
| The system automatically analyzes data to determine TOON suitability: | ||
|
|
||
| ### Good Candidates for TOON | ||
|
|
||
| - **Arrays of uniform objects** (e.g., task lists, user records) | ||
| - **Flat object structures** with repeated keys | ||
| - **Large datasets** with consistent schema | ||
| - **API responses** with standard formats | ||
|
|
||
| ### Poor Candidates for TOON | ||
|
|
||
| - **Deeply nested objects** (>4 levels) | ||
| - **Sparse objects** with many null/undefined values | ||
| - **Mixed data types** within arrays | ||
| - **Small payloads** (<100 characters) | ||
|
|
||
| ## Performance Considerations | ||
|
|
||
| ### Token Savings Analysis | ||
|
|
||
| ```javascript | ||
| // Example: Task management data | ||
| const taskData = { | ||
| tasks: [ | ||
| { | ||
| id: 'task-1', | ||
| title: 'Implement authentication', | ||
| status: 'in-progress', | ||
| assignee: { id: 'user-123', name: 'John Doe' }, | ||
| tags: ['auth', 'security', 'backend'] | ||
| } | ||
| // ... more tasks | ||
| ] | ||
| }; | ||
|
|
||
| // Typical savings: 35-45% for uniform task data | ||
| // JSON: ~150 tokens → TOON: ~95 tokens (37% savings) | ||
| ``` | ||
|
|
||
| ### Runtime Overhead | ||
|
|
||
| - **Serialization**: ~1-2ms for typical payloads | ||
| - **Analysis**: ~0.5ms for suitability checking | ||
| - **Memory**: Minimal additional memory usage | ||
| - **Caching**: Enhanced providers are cached for reuse | ||
|
|
||
| ## Integration with Task Master Workflows | ||
|
|
||
| ### Existing Workflows That Benefit | ||
|
|
||
| 1. **Task List Operations** | ||
| ```javascript | ||
| // task-master list → Returns task arrays (excellent TOON candidate) | ||
| // 40-50% token savings typical | ||
| ``` | ||
|
|
||
| 2. **Task Generation from PRDs** | ||
| ```javascript | ||
| // task-master parse-prd → Large structured responses (good TOON candidate) | ||
| // 30-40% token savings typical | ||
| ``` | ||
|
|
||
| 3. **Complexity Analysis** | ||
| ```javascript | ||
| // task-master analyze-complexity → Structured analysis data (good TOON candidate) | ||
| // 25-35% token savings typical | ||
| ``` | ||
|
|
||
| ### Workflows That Don't Benefit | ||
|
|
||
| - **Simple text responses** (no structured data) | ||
| - **Error messages** (small, unstructured) | ||
| - **Single task queries** (small payloads) | ||
|
|
||
| ## Testing and Validation | ||
|
|
||
| ### Automated Testing | ||
|
|
||
| ```bash | ||
| # Run TOON serialization tests | ||
| npm test src/serialization/toon-serializer.spec.js | ||
|
|
||
| # Test full integration | ||
| node scripts/toon-cli.js test | ||
| ``` | ||
|
|
||
| ### Manual Testing | ||
|
|
||
| ```javascript | ||
| import { validateToonRoundTrip } from './src/serialization/index.js'; | ||
|
|
||
| const testData = { /* your data */ }; | ||
| const validation = validateToonRoundTrip(testData); | ||
|
|
||
| if (!validation.isValid) { | ||
| console.error('Round-trip validation failed:', validation.error); | ||
| } | ||
| ``` | ||
|
|
||
| ## Rollout Guidelines | ||
|
|
||
| ### Phase 1: Enable for Specific Data Types | ||
|
|
||
| 1. Start with **arrays of uniform objects** (task lists, user records) | ||
| 2. Monitor token savings and accuracy | ||
| 3. Gradually expand to more data types | ||
|
|
||
| ### Phase 2: Broaden Usage | ||
|
|
||
| 1. Enable for **flat object structures** | ||
| 2. Test with **complex task data** | ||
| 3. Monitor for any accuracy regressions | ||
|
|
||
| ### Phase 3: Full Deployment | ||
|
|
||
| 1. Enable for **all suitable data structures** | ||
| 2. Set production-ready thresholds | ||
| 3. Monitor cost savings and performance | ||
|
|
||
| ### Recommended Thresholds | ||
|
|
||
| - **Development**: `minDataSize: 50, minSavingsThreshold: 15` | ||
| - **Staging**: `minDataSize: 75, minSavingsThreshold: 12` | ||
| - **Production**: `minDataSize: 100, minSavingsThreshold: 10` | ||
|
|
||
| ## Monitoring and Metrics | ||
|
|
||
| ### Key Metrics to Track | ||
|
|
||
| - **Token savings percentage** per request type | ||
| - **Cost reduction** over time | ||
| - **Response accuracy** (no degradation) | ||
| - **Latency improvements** from smaller payloads | ||
| - **Error rates** (should remain unchanged) | ||
|
|
||
| ### Logging | ||
|
|
||
| ```javascript | ||
| // TOON usage is automatically logged | ||
| // Look for log entries like: | ||
| // "Using TOON serialization for generateText: 35% token savings expected" | ||
| // "TOON optimization saved approximately 45 tokens (32%)" | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Common Issues | ||
|
|
||
| 1. **Round-trip validation failures** | ||
| - Check for complex nested structures | ||
| - Verify special character handling | ||
|
|
||
| 2. **Poor savings performance** | ||
| - Adjust `minSavingsThreshold` | ||
| - Exclude unsuitable data types | ||
|
|
||
| 3. **Provider compatibility issues** | ||
| - Some providers may not work well with TOON instructions | ||
| - Use provider-specific configurations | ||
|
|
||
| ### Debugging | ||
|
|
||
| ```bash | ||
| # Enable debug logging | ||
| DEBUG=toon* node scripts/toon-cli.js test | ||
|
|
||
| # Check TOON configuration | ||
| node scripts/toon-cli.js status | ||
|
|
||
| # Validate specific data | ||
| node -e " | ||
| const { validateToonRoundTrip } = require('./src/serialization'); | ||
| console.log(validateToonRoundTrip({your: 'data'})); | ||
| " | ||
| ``` | ||
|
|
||
| ## Migration Path | ||
|
|
||
| ### From Standard JSON | ||
|
|
||
| 1. **No code changes required** - TOON works transparently | ||
| 2. **Enable gradually** using CLI or programmatic controls | ||
| 3. **Monitor performance** and adjust thresholds | ||
| 4. **Rollback easily** by disabling TOON integration | ||
|
|
||
| ### Compatibility | ||
|
|
||
| - **100% backward compatible** with existing JSON workflows | ||
| - **Automatic fallback** for unsuitable data | ||
| - **No changes required** to existing Task Master commands | ||
| - **Optional feature** that can be disabled anytime | ||
|
|
||
| ## Future Enhancements | ||
|
|
||
| ### Planned Improvements | ||
|
|
||
| - **Schema-aware TOON** using task/subtask schemas | ||
| - **Compression algorithms** for further token reduction | ||
| - **Provider-specific optimizations** based on model capabilities | ||
| - **Real-time savings metrics** in Task Master dashboard | ||
| - **A/B testing framework** for accuracy validation | ||
|
|
||
| ### Community Contributions | ||
|
|
||
| - Submit issues for data types that don't convert well | ||
| - Contribute provider-specific optimizations | ||
| - Share real-world usage statistics and savings data No newline at end of file |
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Align docs location with Mintlify setup and fix ESM example
The guide content itself is excellent and aligns well with the new TOON components, but there are two structural issues to address:
-
Docs location vs Mintlify
Project guidelines route documentation through the Mintlify site underapps/docs/, not the top‑leveldocs/directory. To keep the canonical docs in one place (and actually visible on docs.task-master.dev), please move this guide intoapps/docs/(e.g., as a dedicated TOON page) and, if desired, leave only a short pointer here. -
ESM vs CommonJS in the debugging snippet
The snippet near lines 318–321 uses:const { validateToonRoundTrip } = require('./src/serialization');
With
"type": "module"andsrc/serialization/index.jsexporting via ES modules, thisrequire()form will fail. Prefer an ESM-friendly example, e.g.:node -e "import('./src/serialization/index.js').then(m => console.log(m.validateToonRoundTrip({ your: 'data' })))"or show a small
.mjs/.jsscript usingimportinstead.
🤖 Prompt for AI Agents
In docs/toon-integration-guide.md lines 1–354: move this guide into the Mintlify
docs app (e.g., apps/docs/content/toon-integration-guide.md or the appropriate
docs page location) and leave a short pointer or redirect in the top-level docs
folder; additionally update the debugging snippet at ~lines 318–321 to use an
ESM-friendly example (show importing the ESM module via dynamic import with node
-e or an ES module script path) instead of require(), and ensure the import path
includes the explicit index.js where needed so it works with "type":"module".
|
|
||
| ## null |
There was a problem hiding this comment.
Invalid changelog entry: "## null" is not a valid version.
This file contains multiple "## null" entries (lines 3–13 plus the new entry at 14–15), which are invalid version headers. Remove all "## null" entries and replace with proper semantic versions or remove them entirely. Verify the changeset release configuration for this package.
🤖 Prompt for AI Agents
In packages/ai-sdk-provider-grok-cli/CHANGELOG.md around lines 14 to 15 (and
note similar entries at lines 3–13), there are invalid section headers "##
null"; remove every "## null" header from the file and either replace each with
a proper semantic version header (e.g., "## vX.Y.Z - YYYY-MM-DD") or delete the
empty entries entirely, and then check the package's changeset/release
configuration (changeset files and package.json release settings) to ensure
versions are generated correctly to prevent future "null" entries.
| ## null | ||
|
|
There was a problem hiding this comment.
Invalid changelog entry: "## null" is not a valid version.
Same issue as in other changelog files—multiple "## null" entries (lines 3–16 plus the new entry at 17–18) are invalid version headers. These appear to be leftover placeholders from the changeset process. Remove all "## null" entries and verify the changeset configuration for proper version assignment.
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)
17-17: Multiple headings with the same content
(MD024, no-duplicate-heading)
🤖 Prompt for AI Agents
In packages/build-config/CHANGELOG.md around lines 17 to 18, there is an invalid
version header "## null" (and others at lines 3–16) left from the changeset
process; remove all "## null" entries and replace them with the correct version
headers or remove the empty sections, then inspect and fix the changeset config
(changeset files in .changeset/) so that future changelogs receive proper
version values (ensure each changeset has a valid release type and package
field, run the changeset versioning command to regenerate the changelog and
confirm the null headers are gone).
| // Dynamically modify ai-services-unified.js to use TOON-enhanced providers | ||
| const aiServices = await import('../../scripts/modules/ai-services-unified.js'); | ||
|
|
||
| // Store reference to original _getProvider if it exists and we haven't patched it | ||
| if (aiServices._getProvider && !originalGetProvider) { | ||
| originalGetProvider = aiServices._getProvider; | ||
|
|
||
| // Patch the _getProvider function | ||
| aiServices._getProvider = function(providerName) { | ||
| const provider = originalGetProvider(providerName); | ||
| if (provider) { | ||
| return ToonProviderFactory.getEnhancedProvider(providerName, provider); | ||
| } | ||
| return provider; | ||
| }; | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check if _getProvider is exported from ai-services-unified.js and how it's defined
rg -n '_getProvider' scripts/modules/ai-services-unified.js -A 5Repository: eyaltoledano/claude-task-master
Length of output: 640
🏁 Script executed:
# Check the overall structure of ai-services-unified.js to understand exports
head -100 scripts/modules/ai-services-unified.jsRepository: eyaltoledano/claude-task-master
Length of output: 2645
🏁 Script executed:
# Check if there are any exports at the end of the file
tail -50 scripts/modules/ai-services-unified.jsRepository: eyaltoledano/claude-task-master
Length of output: 1258
🏁 Script executed:
# Verify the complete module namespace by checking what's actually available on the imported module
grep -n "export" scripts/modules/ai-services-unified.js | tail -20Repository: eyaltoledano/claude-task-master
Length of output: 87
🏁 Script executed:
# Double-check: search for any other export statements or export default
rg -n "export\s+(default|{|\*)" scripts/modules/ai-services-unified.jsRepository: eyaltoledano/claude-task-master
Length of output: 191
Attempted patch targets a non-exported function and will never execute.
_getProvider is a private internal function in ai-services-unified.js and is not included in the module's exports. The condition if (aiServices._getProvider && !originalGetProvider) will always be falsy, making this monkey-patching logic dead code.
To integrate TOON enhancements, consider:
- Exporting
_getProvideror a wrapper function fromai-services-unified.jsif patching is necessary. - Using a different integration pattern, such as a provider wrapper or middleware layer, that doesn't require modifying the module namespace.
🧰 Tools
🪛 GitHub Actions: CI
[error] 34-34: await is only allowed within async functions and at the top levels of modules.
🤖 Prompt for AI Agents
In src/serialization/toon-ai-services-integration.js around lines 33 to 48, the
current monkey-patch checks aiServices._getProvider which is not exported and
thus the condition is always false; to fix either (A) add an explicit
export/wrapper in ai-services-unified.js (e.g., export _getProvider or a new
getProviderProxy) and then import that exported function here and wrap it via
ToonProviderFactory.getEnhancedProvider, or (B) avoid patching internals: import
the public provider-creation API from ai-services-unified.js and wrap its
returned provider instances (or register a middleware/provider-factory wrapper)
so that every provider returned is passed through
ToonProviderFactory.getEnhancedProvider before use; implement one of these
approaches and remove the dead conditional monkey-patch.
| export function disableToonForAIServices() { | ||
| if (!toonIntegrationEnabled) { | ||
| log('warn', 'TOON integration for AI services is not currently enabled'); | ||
| return; | ||
| } |
There was a problem hiding this comment.
Critical: Function uses await but is not declared async.
The pipeline correctly flags that await at line 78 is used in a non-async function. This is a syntax error.
-export function disableToonForAIServices() {
+export async function disableToonForAIServices() {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| export function disableToonForAIServices() { | |
| if (!toonIntegrationEnabled) { | |
| log('warn', 'TOON integration for AI services is not currently enabled'); | |
| return; | |
| } | |
| export async function disableToonForAIServices() { | |
| if (!toonIntegrationEnabled) { | |
| log('warn', 'TOON integration for AI services is not currently enabled'); | |
| return; | |
| } |
🤖 Prompt for AI Agents
In src/serialization/toon-ai-services-integration.js around lines 66–70 (and
note the await used at line 78), the function disableToonForAIServices uses
await but is not declared async; update the function signature to be async
(export async function disableToonForAIServices()) so the await is valid, and
ensure callers handle the returned Promise (or alternatively remove/replace the
await with Promise handling if you don't want an async function).
| /** | ||
| * TOON (Token-Oriented Object Notation) Serialization Utilities | ||
| * | ||
| * TOON is a compact, schema-aware format that reduces token usage by 30-60% versus JSON | ||
| * by eliminating syntactic overhead (braces, quotes, repeated fields). | ||
| * | ||
| * This implementation provides conversion between JSON and TOON format | ||
| * for LLM data serialization at the provider boundary layer. | ||
| */ | ||
|
|
||
| import { log } from '../../scripts/modules/utils.js'; | ||
|
|
||
| /** | ||
| * Convert JSON data to TOON format | ||
| * @param {any} data - The data to convert to TOON | ||
| * @param {object} options - Conversion options | ||
| * @param {boolean} options.preserveTypes - Whether to preserve type information | ||
| * @returns {string} TOON formatted string | ||
| */ | ||
| export function jsonToToon(data, options = { preserveTypes: true }) { | ||
| try { | ||
| if (data === null) return 'null'; | ||
| if (data === undefined) return 'undefined'; | ||
|
|
||
| // Handle primitive types | ||
| if (typeof data === 'string') { | ||
| // Escape special characters and wrap in quotes only if necessary | ||
| return data.includes(' ') || data.includes('\n') ? `"${data.replace(/"/g, '\\"')}"` : data; | ||
| } | ||
| if (typeof data === 'number') return data.toString(); | ||
| if (typeof data === 'boolean') return data.toString(); | ||
|
|
||
| // Handle arrays - use space-separated values instead of brackets | ||
| if (Array.isArray(data)) { | ||
| if (data.length === 0) return '[]'; | ||
| const items = data.map(item => jsonToToon(item, options)); | ||
| return `[${items.join(' ')}]`; | ||
| } | ||
|
|
||
| // Handle objects - use key:value pairs separated by spaces | ||
| if (typeof data === 'object') { | ||
| const entries = Object.entries(data); | ||
| if (entries.length === 0) return '{}'; | ||
|
|
||
| const pairs = entries.map(([key, value]) => { | ||
| const toonValue = jsonToToon(value, options); | ||
| // Use colon separator without spaces for compactness | ||
| return `${key}:${toonValue}`; | ||
| }); | ||
|
|
||
| return `{${pairs.join(' ')}}`; | ||
| } | ||
|
|
||
| return String(data); | ||
| } catch (error) { | ||
| log('error', `Failed to convert JSON to TOON: ${error.message}`); | ||
| throw new Error(`TOON serialization error: ${error.message}`); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Convert TOON format back to JSON | ||
| * @param {string} toonData - The TOON formatted string | ||
| * @returns {any} Parsed JSON data | ||
| */ | ||
| export function toonToJson(toonData) { | ||
| try { | ||
| if (!toonData || typeof toonData !== 'string') { | ||
| throw new Error('Invalid TOON data: must be a non-empty string'); | ||
| } | ||
|
|
||
| const trimmed = toonData.trim(); | ||
| if (!trimmed) return null; | ||
|
|
||
| // Handle primitive values | ||
| if (trimmed === 'null') return null; | ||
| if (trimmed === 'undefined') return undefined; | ||
| if (trimmed === 'true') return true; | ||
| if (trimmed === 'false') return false; | ||
|
|
||
| // Handle numbers | ||
| if (/^-?\d+(\.\d+)?$/.test(trimmed)) { | ||
| return Number(trimmed); | ||
| } | ||
|
|
||
| // Handle quoted strings | ||
| if (trimmed.startsWith('"') && trimmed.endsWith('"')) { | ||
| return trimmed.slice(1, -1).replace(/\\"/g, '"'); | ||
| } | ||
|
|
||
| // Handle arrays | ||
| if (trimmed.startsWith('[') && trimmed.endsWith(']')) { | ||
| const content = trimmed.slice(1, -1).trim(); | ||
| if (!content) return []; | ||
|
|
||
| const items = parseTooonItems(content); | ||
| return items.map(item => toonToJson(item)); | ||
| } | ||
|
|
||
| // Handle objects | ||
| if (trimmed.startsWith('{') && trimmed.endsWith('}')) { | ||
| const content = trimmed.slice(1, -1).trim(); | ||
| if (!content) return {}; | ||
|
|
||
| const result = {}; | ||
| const pairs = parseTooonItems(content); | ||
|
|
||
| for (const pair of pairs) { | ||
| const colonIndex = pair.indexOf(':'); | ||
| if (colonIndex === -1) { | ||
| throw new Error(`Invalid TOON object pair: ${pair}`); | ||
| } | ||
|
|
||
| const key = pair.substring(0, colonIndex).trim(); | ||
| const value = pair.substring(colonIndex + 1).trim(); | ||
| result[key] = toonToJson(value); | ||
| } | ||
|
|
||
| return result; | ||
| } | ||
|
|
||
| // Handle unquoted strings | ||
| return trimmed; | ||
|
|
||
| } catch (error) { | ||
| log('error', `Failed to convert TOON to JSON: ${error.message}`); | ||
| throw new Error(`TOON deserialization error: ${error.message}`); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Parse TOON items while respecting nested structures | ||
| * @param {string} content - Content to parse | ||
| * @returns {string[]} Array of parsed items | ||
| */ | ||
| function parseTooonItems(content) { | ||
| const items = []; | ||
| let current = ''; | ||
| let depth = 0; | ||
| let inQuotes = false; | ||
| let escaped = false; | ||
|
|
||
| for (let i = 0; i < content.length; i++) { | ||
| const char = content[i]; | ||
|
|
||
| if (escaped) { | ||
| current += char; | ||
| escaped = false; | ||
| continue; | ||
| } | ||
|
|
||
| if (char === '\\') { | ||
| escaped = true; | ||
| current += char; | ||
| continue; | ||
| } | ||
|
|
||
| if (char === '"' && !escaped) { | ||
| inQuotes = !inQuotes; | ||
| current += char; | ||
| continue; | ||
| } | ||
|
|
||
| if (inQuotes) { | ||
| current += char; | ||
| continue; | ||
| } | ||
|
|
||
| if (char === '{' || char === '[') { | ||
| depth++; | ||
| current += char; | ||
| } else if (char === '}' || char === ']') { | ||
| depth--; | ||
| current += char; | ||
| } else if (char === ' ' && depth === 0) { | ||
| if (current.trim()) { | ||
| items.push(current.trim()); | ||
| current = ''; | ||
| } | ||
| } else { | ||
| current += char; | ||
| } | ||
| } | ||
|
|
||
| if (current.trim()) { | ||
| items.push(current.trim()); | ||
| } | ||
|
|
||
| return items; | ||
| } | ||
|
|
||
| /** | ||
| * Estimate token savings from using TOON vs JSON | ||
| * @param {any} data - The data to analyze | ||
| * @returns {object} Analysis of token savings | ||
| */ | ||
| export function estimateTokenSavings(data) { | ||
| try { | ||
| const jsonString = JSON.stringify(data); | ||
| const toonString = jsonToToon(data); | ||
|
|
||
| const jsonLength = jsonString.length; | ||
| const toonLength = toonString.length; | ||
| const savings = jsonLength - toonLength; | ||
| const savingsPercentage = jsonLength > 0 ? (savings / jsonLength) * 100 : 0; | ||
|
|
||
| // Rough token estimation (1 token ≈ 4 characters for English text) | ||
| const estimatedJsonTokens = Math.ceil(jsonLength / 4); | ||
| const estimatedToonTokens = Math.ceil(toonLength / 4); | ||
| const tokenSavings = estimatedJsonTokens - estimatedToonTokens; | ||
|
|
||
| return { | ||
| jsonLength, | ||
| toonLength, | ||
| characterSavings: savings, | ||
| savingsPercentage: Math.round(savingsPercentage * 100) / 100, | ||
| estimatedJsonTokens, | ||
| estimatedToonTokens, | ||
| estimatedTokenSavings: tokenSavings, | ||
| estimatedTokenSavingsPercentage: estimatedJsonTokens > 0 ? Math.round((tokenSavings / estimatedJsonTokens) * 10000) / 100 : 0 | ||
| }; | ||
| } catch (error) { | ||
| log('error', `Failed to estimate token savings: ${error.message}`); | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Validate that TOON data can be round-tripped (JSON -> TOON -> JSON) | ||
| * @param {any} data - The data to validate | ||
| * @returns {object} Validation results | ||
| */ | ||
| export function validateToonRoundTrip(data) { | ||
| try { | ||
| const toonData = jsonToToon(data); | ||
| const reconstructedData = toonToJson(toonData); | ||
|
|
||
| // Deep comparison for validation | ||
| const isValid = JSON.stringify(data) === JSON.stringify(reconstructedData); | ||
|
|
||
| return { | ||
| isValid, | ||
| original: data, | ||
| toon: toonData, | ||
| reconstructed: reconstructedData, | ||
| ...(isValid ? {} : { | ||
| error: 'Data mismatch after round-trip conversion' | ||
| }) | ||
| }; | ||
| } catch (error) { | ||
| return { | ||
| isValid: false, | ||
| error: error.message, | ||
| original: data | ||
| }; | ||
| } | ||
| } No newline at end of file |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check package.json for @toon-format/toon and current dependencies
cat package.json | head -50Repository: eyaltoledano/claude-task-master
Length of output: 2260
🏁 Script executed:
# Search for references to Issue #1479 in the codebase
rg "#1479|Issue.*1479|issue.*1479" -iRepository: eyaltoledano/claude-task-master
Length of output: 57
🏁 Script executed:
# Search for any references to @toon-format/toon in the codebase
rg "@toon-format/toon|toon-format"Repository: eyaltoledano/claude-task-master
Length of output: 57
🏁 Script executed:
# Search for TODO or FIXME comments related to TOON implementation
rg "TODO|FIXME" src/serialization/toon-serializer.js -A 2 -B 2Repository: eyaltoledano/claude-task-master
Length of output: 57
🏁 Script executed:
# Check for any documentation about TOON format decisions
fd -e md -e txt | xargs grep -l "TOON\|toon" 2>/dev/null | head -20Repository: eyaltoledano/claude-task-master
Length of output: 105
🏁 Script executed:
# Read the TOON integration guide
cat docs/toon-integration-guide.mdRepository: eyaltoledano/claude-task-master
Length of output: 9766
🏁 Script executed:
# Get full package.json dependencies
cat package.json | grep -A 100 '"dependencies"'Repository: eyaltoledano/claude-task-master
Length of output: 2832
🏁 Script executed:
# Check if there are any other references to official TOON spec
rg "toon.*spec|official.*toon|toon.*library" -iRepository: eyaltoledano/claude-task-master
Length of output: 308
🌐 Web query:
@toon-format/toon npm package
💡 Result:
@toon-format/toon is the official TypeScript/JavaScript reference implementation of TOON (Token-Oriented Object Notation) — a compact, LLM-friendly, lossless encoding of JSON. It provides encode/decode functions, a CLI (npx @toon-format/cli), streaming helpers, and API docs on the project site. Install with npm: npm install @toon-format/toon. [1][2][3]
Sources:
- GitHub repository / reference implementation and README. [1]
- Official API/docs (toonformat.dev) with installation and examples. [2]
- npm/CDN package listing (jsDelivr/npm) showing published package. [3]
🏁 Script executed:
# Search for any GitHub issue numbers or references in the codebase
rg "issue|Issue|#[0-9]{3,}" -i | grep -i "toon\|1479" | head -20Repository: eyaltoledano/claude-task-master
Length of output: 279
🏁 Script executed:
# Check git history or comments for issue references
rg "1479" -r .Repository: eyaltoledano/claude-task-master
Length of output: 57
🏁 Script executed:
# Look for any comments or documentation explaining the design decision
rg "custom|implementation|official|spec|toon-format" src/serialization/toon-serializer.js -B 2 -A 2Repository: eyaltoledano/claude-task-master
Length of output: 675
🏁 Script executed:
# Check if there are architecture decision records or design docs
fd -e md | xargs grep -l "toon.*design\|toon.*decision\|why.*toon\|custom.*toon" 2>/dev/nullRepository: eyaltoledano/claude-task-master
Length of output: 59
🏁 Script executed:
# Search for any TODO or FIXME about using official package
rg "TODO|FIXME|XXX" src/serialization/ -A 3Repository: eyaltoledano/claude-task-master
Length of output: 57
🌐 Web query:
TOON format specification official standard compatibility
💡 Result:
Summary — official status & compatibility
-
Official specification and reference implementation: the TOON spec and reference code are hosted in the toon-format organization on GitHub (spec repository is the authoritative spec). The spec is published there (current spec version noted in repo). [1]
-
JSON compatibility: TOON is a lossless serialization of the JSON data model (objects, arrays, primitives) and guarantees deterministic round-trip JSON ↔ TOON conversion per the spec and reference implementation. [1][2]
-
Media type & extension: the spec defines a provisional media type text/toon and the .toon file extension (UTF‑8). IANA registration is listed as pending in the spec. [1]
-
Conformance & ecosystem: the spec includes ABNF grammar, conformance tests/fixtures and a versioning policy; multiple language implementations and tooling (TypeScript, Python, Rust, .NET, etc.) and a test-suite are maintained to ensure compatibility. [1][2][3]
-
Status: the spec repository lists the specification as a working draft and follows semantic versioning; implementers should follow the spec’s VERSIONING and CHANGELOG for compatibility guarantees. [1]
Sources
[1] TOON format specification (official repo/spec).
[2] TOON reference implementation / project repo.
[3] TOON documentation / toolkit (implementation and docs).
🏁 Script executed:
# Check if there are any tests that validate against official TOON spec
find . -name "*toon*.test.*" -o -name "*toon*.spec.*" | head -10Repository: eyaltoledano/claude-task-master
Length of output: 118
🏁 Script executed:
# Check the test file for the serializer
find . -name "toon-serializer*" -type fRepository: eyaltoledano/claude-task-master
Length of output: 157
🏁 Script executed:
# Look for any edge case handling or spec compliance tests
rg "spec|compliance|edge.*case" src/serialization/ -iRepository: eyaltoledano/claude-task-master
Length of output: 461
🏁 Script executed:
# Check package.json for any TOON-related dev dependencies or scripts
cat package.json | grep -A 50 '"devDependencies"' | grep -i toonRepository: eyaltoledano/claude-task-master
Length of output: 57
🏁 Script executed:
# Read the test file to see what edge cases are covered and understand the testing approach
cat src/serialization/toon-serializer.spec.jsRepository: eyaltoledano/claude-task-master
Length of output: 6122
Use the official @toon-format/toon package or document the rationale for this custom implementation
The official TOON format reference implementation (@toon-format/toon) exists and provides a maintained, spec-compliant encoder/decoder with ABNF grammar and conformance test fixtures. This module implements a custom TOON-like parser instead without documented justification.
If delegating to @toon-format/toon is feasible, prefer it to ensure spec compliance and reduce maintenance burden. If a custom simplified variant is intentional, explicitly document why (e.g., performance, licensing, feature subset) and add test coverage for edge cases that may diverge from the official spec—such as keys with spaces, special character handling, and JSON methods like toJSON. The current tests validate only basic round-trip conversion and do not verify spec conformance.
🧰 Tools
🪛 GitHub Actions: CI
[error] 1-1: Formatter would print different content. Code formatting issues detected in this file.
| export function jsonToToon(data, options = { preserveTypes: true }) { | ||
| try { | ||
| if (data === null) return 'null'; | ||
| if (data === undefined) return 'undefined'; | ||
|
|
||
| // Handle primitive types | ||
| if (typeof data === 'string') { | ||
| // Escape special characters and wrap in quotes only if necessary | ||
| return data.includes(' ') || data.includes('\n') ? `"${data.replace(/"/g, '\\"')}"` : data; | ||
| } | ||
| if (typeof data === 'number') return data.toString(); | ||
| if (typeof data === 'boolean') return data.toString(); | ||
|
|
||
| // Handle arrays - use space-separated values instead of brackets | ||
| if (Array.isArray(data)) { | ||
| if (data.length === 0) return '[]'; | ||
| const items = data.map(item => jsonToToon(item, options)); | ||
| return `[${items.join(' ')}]`; | ||
| } | ||
|
|
||
| // Handle objects - use key:value pairs separated by spaces | ||
| if (typeof data === 'object') { | ||
| const entries = Object.entries(data); | ||
| if (entries.length === 0) return '{}'; | ||
|
|
||
| const pairs = entries.map(([key, value]) => { | ||
| const toonValue = jsonToToon(value, options); | ||
| // Use colon separator without spaces for compactness | ||
| return `${key}:${toonValue}`; | ||
| }); | ||
|
|
||
| return `{${pairs.join(' ')}}`; | ||
| } | ||
|
|
||
| return String(data); | ||
| } catch (error) { | ||
| log('error', `Failed to convert JSON to TOON: ${error.message}`); | ||
| throw new Error(`TOON serialization error: ${error.message}`); | ||
| } | ||
| } |
There was a problem hiding this comment.
Fix empty-string round‑trip bug in jsonToToon/toonToJson
There’s a real correctness issue around empty strings:
jsonToToon('')currently returns''(no quotes).- For
{ key: '' }you produce{key:}. - In
toonToJson,trimmed === ''hitsif (!trimmed) return null;, so:toonToJson('')returnsnull.toonToJson('{key:}')returns{ key: null }.
This means validateToonRoundTrip will report isValid: false for any data containing empty strings, and more importantly your serializer mutates '' → null for real payloads, which conflicts with the stated “100% backward compatibility” goal.
A minimal fix would be:
- Always quote empty strings when serializing:
- if (typeof data === 'string') {
- // Escape special characters and wrap in quotes only if necessary
- return data.includes(' ') || data.includes('\n') ? `"${data.replace(/"/g, '\\"')}"` : data;
- }
+ if (typeof data === 'string') {
+ const escaped = data.replace(/"/g, '\\"');
+ // Preserve empty strings explicitly
+ if (data === '') {
+ return '""';
+ }
+ // Escape special characters and wrap in quotes only if necessary
+ return data.includes(' ') || data.includes('\n') ? `"${escaped}"` : escaped;
+ }- Treat empty TOON tokens as empty strings, not null:
- const trimmed = toonData.trim();
- if (!trimmed) return null;
+ const trimmed = toonData.trim();
+ // Preserve empty strings for round-trip correctness
+ if (trimmed === '') return '';With these changes, both '' and object fields containing '' will round-trip correctly. I’d also add focused tests in toon-serializer.spec.js for:
jsonToToon('')/toonToJson('""')validateToonRoundTrip('')validateToonRoundTrip({ key: '' })
Finally, CI is also failing with a Biome formatter mismatch on this file; after code changes, please re-run the formatter so it passes.
Also applies to: 66-129, 193-257
🤖 Prompt for AI Agents
src/serialization/toon-serializer.js around lines 20-59 (also apply same logic
to blocks at 66-129 and 193-257): the serializer currently emits unquoted empty
strings and the parser treats empty tokens as null, breaking round-trip for ''.
Change jsonToToon so that when typeof data === 'string' and data === '' it
returns '""' (and when serializing objects/keys ensure empty string values
become key:""); change toonToJson so that an empty TOON token is parsed as an
empty string rather than null (replace the generic if (!trimmed) return null
check with an explicit check that distinguishes missing/absent tokens from an
empty token and return '' for an empty token); add targeted tests for
jsonToToon(''), toonToJson('""'), validateToonRoundTrip(''),
validateToonRoundTrip({ key: '' }); and re-run the Biome formatter before
pushing.
| import { describe, it, expect } from '@jest/globals'; | ||
| import { jsonToToon, toonToJson, estimateTokenSavings, validateToonRoundTrip } from './toon-serializer.js'; | ||
|
|
||
| describe('TOON Serializer', () => { | ||
| describe('jsonToToon', () => { | ||
| it('should convert primitive values correctly', () => { | ||
| expect(jsonToToon(42)).toBe('42'); | ||
| expect(jsonToToon(true)).toBe('true'); | ||
| expect(jsonToToon(false)).toBe('false'); | ||
| expect(jsonToToon(null)).toBe('null'); | ||
| expect(jsonToToon('hello')).toBe('hello'); | ||
| expect(jsonToToon('hello world')).toBe('"hello world"'); | ||
| }); | ||
|
|
||
| it('should convert simple objects to TOON format', () => { | ||
| const input = { name: 'John', age: 30 }; | ||
| const result = jsonToToon(input); | ||
| expect(result).toBe('{name:John age:30}'); | ||
| }); | ||
|
|
||
| it('should convert arrays to TOON format', () => { | ||
| const input = ['apple', 'banana', 'cherry']; | ||
| const result = jsonToToon(input); | ||
| expect(result).toBe('[apple banana cherry]'); | ||
| }); | ||
|
|
||
| it('should handle nested objects', () => { | ||
| const input = { | ||
| user: { | ||
| name: 'John', | ||
| age: 30 | ||
| }, | ||
| active: true | ||
| }; | ||
| const result = jsonToToon(input); | ||
| expect(result).toBe('{user:{name:John age:30} active:true}'); | ||
| }); | ||
|
|
||
| it('should handle arrays of objects', () => { | ||
| const input = [ | ||
| { name: 'John', age: 30 }, | ||
| { name: 'Jane', age: 25 } | ||
| ]; | ||
| const result = jsonToToon(input); | ||
| expect(result).toBe('[{name:John age:30} {name:Jane age:25}]'); | ||
| }); | ||
|
|
||
| it('should handle empty containers', () => { | ||
| expect(jsonToToon([])).toBe('[]'); | ||
| expect(jsonToToon({})).toBe('{}'); | ||
| }); | ||
|
|
||
| it('should escape quotes in strings', () => { | ||
| const input = { message: 'She said "hello"' }; | ||
| const result = jsonToToon(input); | ||
| expect(result).toBe('{message:"She said \\"hello\\""}'); | ||
| }); | ||
| }); | ||
|
|
||
| describe('toonToJson', () => { | ||
| it('should convert primitive values correctly', () => { | ||
| expect(toonToJson('42')).toBe(42); | ||
| expect(toonToJson('true')).toBe(true); | ||
| expect(toonToJson('false')).toBe(false); | ||
| expect(toonToJson('null')).toBe(null); | ||
| expect(toonToJson('hello')).toBe('hello'); | ||
| expect(toonToJson('"hello world"')).toBe('hello world'); | ||
| }); | ||
|
|
||
| it('should convert TOON objects back to JSON', () => { | ||
| const input = '{name:John age:30}'; | ||
| const result = toonToJson(input); | ||
| expect(result).toEqual({ name: 'John', age: 30 }); | ||
| }); | ||
|
|
||
| it('should convert TOON arrays back to JSON', () => { | ||
| const input = '[apple banana cherry]'; | ||
| const result = toonToJson(input); | ||
| expect(result).toEqual(['apple', 'banana', 'cherry']); | ||
| }); | ||
|
|
||
| it('should handle nested structures', () => { | ||
| const input = '{user:{name:John age:30} active:true}'; | ||
| const result = toonToJson(input); | ||
| expect(result).toEqual({ | ||
| user: { | ||
| name: 'John', | ||
| age: 30 | ||
| }, | ||
| active: true | ||
| }); | ||
| }); | ||
|
|
||
| it('should handle arrays of objects', () => { | ||
| const input = '[{name:John age:30} {name:Jane age:25}]'; | ||
| const result = toonToJson(input); | ||
| expect(result).toEqual([ | ||
| { name: 'John', age: 30 }, | ||
| { name: 'Jane', age: 25 } | ||
| ]); | ||
| }); | ||
|
|
||
| it('should handle empty containers', () => { | ||
| expect(toonToJson('[]')).toEqual([]); | ||
| expect(toonToJson('{}')).toEqual({}); | ||
| }); | ||
|
|
||
| it('should handle quoted strings with escaped quotes', () => { | ||
| const input = '{message:"She said \\"hello\\""}'; | ||
| const result = toonToJson(input); | ||
| expect(result).toEqual({ message: 'She said "hello"' }); | ||
| }); | ||
| }); | ||
|
|
||
| describe('estimateTokenSavings', () => { | ||
| it('should calculate token savings for typical data', () => { | ||
| const data = { | ||
| users: [ | ||
| { id: 1, name: 'John', email: 'john@example.com', active: true }, | ||
| { id: 2, name: 'Jane', email: 'jane@example.com', active: false } | ||
| ] | ||
| }; | ||
|
|
||
| const savings = estimateTokenSavings(data); | ||
| expect(savings).toBeDefined(); | ||
| expect(savings.characterSavings).toBeGreaterThan(0); | ||
| expect(savings.savingsPercentage).toBeGreaterThan(0); | ||
| expect(savings.estimatedTokenSavings).toBeGreaterThan(0); | ||
| }); | ||
|
|
||
| it('should handle edge cases', () => { | ||
| expect(estimateTokenSavings(null)).toBeDefined(); | ||
| expect(estimateTokenSavings({})).toBeDefined(); | ||
| expect(estimateTokenSavings([])).toBeDefined(); | ||
| }); | ||
| }); | ||
|
|
||
| describe('validateToonRoundTrip', () => { | ||
| it('should validate successful round-trips', () => { | ||
| const testCases = [ | ||
| { name: 'John', age: 30 }, | ||
| [1, 2, 3, 'hello'], | ||
| { users: [{ id: 1, active: true }] }, | ||
| 42, | ||
| 'hello world', | ||
| true, | ||
| null | ||
| ]; | ||
|
|
||
| for (const testCase of testCases) { | ||
| const validation = validateToonRoundTrip(testCase); | ||
| expect(validation.isValid).toBe(true); | ||
| expect(validation.error).toBeUndefined(); | ||
| } | ||
| }); | ||
|
|
||
| it('should provide detailed validation results', () => { | ||
| const data = { test: 'data' }; | ||
| const validation = validateToonRoundTrip(data); | ||
|
|
||
| expect(validation).toHaveProperty('isValid'); | ||
| expect(validation).toHaveProperty('original'); | ||
| expect(validation).toHaveProperty('toon'); | ||
| expect(validation).toHaveProperty('reconstructed'); | ||
| expect(validation.original).toEqual(data); | ||
| }); | ||
| }); | ||
|
|
||
| describe('complex data structures', () => { | ||
| it('should handle task management data', () => { | ||
| const taskData = { | ||
| id: 'task-123', | ||
| title: 'Implement TOON serialization', | ||
| status: 'in-progress', | ||
| priority: 'high', | ||
| assignee: { | ||
| name: 'John Doe', | ||
| email: 'john@taskmaster.dev' | ||
| }, | ||
| subtasks: [ | ||
| { | ||
| id: 'subtask-1', | ||
| title: 'Create TOON serializer', | ||
| status: 'done' | ||
| }, | ||
| { | ||
| id: 'subtask-2', | ||
| title: 'Add LLM integration', | ||
| status: 'in-progress' | ||
| } | ||
| ], | ||
| tags: ['feature', 'optimization', 'llm'], | ||
| metadata: { | ||
| created: '2024-12-03T12:30:00Z', | ||
| updated: '2024-12-03T13:45:00Z' | ||
| } | ||
| }; | ||
|
|
||
| const validation = validateToonRoundTrip(taskData); | ||
| expect(validation.isValid).toBe(true); | ||
|
|
||
| const savings = estimateTokenSavings(taskData); | ||
| expect(savings.savingsPercentage).toBeGreaterThan(20); // Should provide significant savings | ||
| }); | ||
| }); | ||
| }); No newline at end of file |
There was a problem hiding this comment.
Good coverage, but fix formatting to satisfy CI
The spec file nicely exercises all four public APIs (including nested/complex task data), but CI is currently failing with “formatter would print different content” for this file. Please run the configured formatter (e.g. npm run format / npm run format-check) so Biome agrees with the committed layout before merging.
🤖 Prompt for AI Agents
In src/serialization/toon-serializer.spec.js around lines 5 to 210, the test
file formatting does not match the project's formatter (Biome), causing CI to
fail; run the configured formatter (e.g. npm run format) or the format-check
(npm run format-check) locally, apply the resulting changes
(whitespace/quotes/linebreaks/indentation), stage and commit the formatted file,
and re-run the format-check to ensure CI will pass.
|
Hi @Crunchyman-ralph |
|
@Ivlad003 hell yeah dude, fork it and make it your own ! |
What type of PR is this?
Description
Add comprehensive TOON format implementation that reduces LLM token usage by 30-60%:
Key features:
Related Issues
How to Test This
# Example commands or stepsExpected result:
Contributor Checklist
npm run changesetnpm testnpm run format-check(ornpm run formatto fix)Changelog Entry
For Maintainers
Summary by CodeRabbit
New Features
Documentation
Tests
✏️ Tip: You can customize this high-level summary in your review settings.