Multi-line strings with special characters cause shell escaping problems

[konard]

🐛 Bug Description

Multi-line strings containing special shell characters get corrupted when passed through command-stream's $ template literal with the echo command. This is a critical issue for scripts that need to handle complex text content like README files, configuration files, or any multi-line text with special characters.

🔴 Impact

This bug makes it impossible to reliably write multi-line content with special characters using shell commands through command-stream. It affects any use case involving:

• README files with code examples
• Configuration files with paths and variables
• Documentation with markdown formatting
• Any text containing backticks, quotes, dollar signs, or backslashes

📝 Detailed Problem Analysis

When using echo with multi-line strings containing special characters, the shell interprets these characters instead of treating them as literal text:

1. Backticks (`) get interpreted as command substitution
2. Dollar signs ($) trigger variable expansion
3. Quotes (single and double) get mangled during escaping
4. Backslashes (\) are processed as escape characters
5. Newlines may be lost or corrupted

🔄 Reproduction Steps

const complexContent = \`# Test Repository

This is a test repository with \\\`backticks\\\` and "quotes".

## Code Example
\\\`\\\`\\\`javascript
const message = "Hello, World!";
console.log(\\\`Message: \\\${message}\\\`);
\\\`\\\`\\\`

## Special Characters
- Single quotes: 'test'
- Double quotes: "test"
- Backticks: \\\`test\\\`
- Dollar signs: $100
- Backslashes: C:\\\\Windows\\\\System32\`;

// ❌ This fails or corrupts the content
await $\`echo "\${complexContent}" > \${testFile}\`;

// The content gets mangled due to shell interpretation

✅ Expected Behavior

The content should be written to the file exactly as specified, preserving all special characters without any shell interpretation.

❌ Actual Behavior

• Content gets corrupted due to shell escaping issues
• Special characters are interpreted by the shell
• The resulting file contains mangled or incomplete text
• May cause syntax errors if the content includes code

🔧 Workarounds

Workaround 1: Use fs.writeFile (Recommended)

// ✅ Most reliable for any content
await fs.writeFile(testFile, complexContent);

Workaround 2: Use heredoc with quoted delimiter

// ✅ Prevents shell interpretation
await $\`cat << 'EOF' > \${testFile}
\${complexContent}
EOF\`;

Workaround 3: Avoid echo with complex strings

Simply don't use echo for multi-line or complex content - use Node.js fs methods instead.

🎯 Suggested Fix

1. Implement proper escaping for multi-line strings in command-stream
2. Consider detecting multi-line content and automatically using a safer method
3. Add warnings in documentation about this limitation
4. Provide a built-in safe write method that handles all content types

📊 Test Coverage

The issue includes a comprehensive test script that:

• Sets up complex multi-line content with various special characters
• Attempts to write it using echo (demonstrates the bug)
• Verifies content corruption
• Shows working workarounds

🔗 References

• Full test case: https://github.com/deep-assistant/hive-mind/blob/main/command-stream-issues/issue-01-multiline-strings.mjs
• Related shell escaping documentation: https://www.gnu.org/software/bash/manual/html_node/Quoting.html

📋 Checklist for Fix

• Implement proper escaping for multi-line strings
• Add tests for various special character combinations
• Update documentation with limitations and best practices
• Consider adding a safe write utility method
• Ensure backward compatibility