A Python implementation of a VCDIFF (RFC 3284) decoder library and command-line tool for efficient binary differencing and compression.
The library provides a VCDIFF decoder that can decode delta files created according to RFC 3284 - The VCDIFF Generic Differencing and Compression Data Format. VCDIFF is a format for expressing one data stream as a variant of another data stream, commonly used for binary differencing, compression, and patch applications.
- Python Library: RFC 3284 compliant VCDIFF decoding with clean, Pythonic API
- Command-Line Tool: Apply deltas and inspect VCDIFF file structure
- Comprehensive Validation: Support for all VCDIFF instruction types (ADD, COPY, RUN)
- Address Caching: Efficient decoding with proper address cache implementation
- Checksum Validation: Full Adler-32 checksum validation support
- Robust Error Handling: Detailed error messages for debugging malformed files
- Extensive Testing: 85 test cases with reference implementation validation
- Type Safety: Full type hints for better development experience
- Application Headers: This implementation does not handle application header information
- Secondary Compression: This decoder does not support secondary compression (e.g., gzip, bzip2)
- Compatibility: Works with VCDIFF deltas created using
xdelta3 -e -S -A(no secondary compression, no application header)
- VCD_ADLER32: This implementation detects and parses the VCD_ADLER32 extension (bit 0x04 in window indicator)
- Non-standard Extension: The Adler-32 checksum is not part of RFC 3284 but is supported by some implementations
- Validation: Full Adler-32 checksum validation is implemented and performed during decoding
- Display: Checksums are displayed in the CLI output as
Adler32: 0x########
pip install vcdiff-decoderpoetry installThis repository includes the VCDIFF test suite as a git submodule. To clone the repository with all test cases:
git clone --recursive https://github.com/ably/vcdiff-python.gitIf you've already cloned the repository without the submodule, initialize it:
git submodule update --init --recursiveTo update the test suite submodule to the latest version:
git submodule update --remoteimport vcdiff_decoder
# Read the source file
with open("original.txt", "rb") as f:
source = f.read()
# Read the VCDIFF delta file
with open("changes.vcdiff", "rb") as f:
delta_data = f.read()
# Apply the delta to reconstruct the target
try:
result = vcdiff_decoder.decode(source, delta_data)
print(f"Decoded result: {result}")
except vcdiff_decoder.VCDIFFError as e:
print(f"Decoding failed: {e}")Decodes a VCDIFF delta file using the provided source data and returns the reconstructed target data.
Parameters:
source: The original source data (may be empty for deltas that don't reference source)delta: The VCDIFF delta file data
Returns:
- Decoded target data as bytes
Raises:
VCDIFFError: If decoding fails (malformed delta, checksum validation failure, etc.)
Creates a new decoder instance with the specified source data. Useful for decoding multiple deltas against the same source.
Parameters:
source: The source data for decoding operations
Returns:
- A
Decoderinstance that can be used to decode multiple deltas
Decodes a single VCDIFF delta using the decoder's source data.
The decoder provides detailed error messages for various failure conditions:
- Invalid VCDIFF format or magic bytes
- Malformed varint encoding
- Out-of-bounds memory access attempts
- Checksum validation failures
- Truncated or corrupted delta files
Exception Hierarchy:
VCDIFFError: Base exception for all VCDIFF-related errorsInvalidMagicError: Invalid VCDIFF magic bytesInvalidVersionError: Unsupported VCDIFF versionInvalidFormatError: Malformed delta structureCorruptedDataError: Data corruption detectedInvalidChecksumError: Checksum validation failure
For comprehensive testing, this project uses xdelta3 as a reference implementation to verify the correctness of the decoder.
brew install xdeltasudo apt-get install xdelta3To run the Python unit tests:
poetry run pytestTo run tests with coverage:
poetry run pytest --cov=vcdiff tests/To run the comprehensive test suite against the VCDIFF test cases (requires submodule):
poetry run pytest tests/test_vcdiff.py -vThe test suite includes:
- 20 general positive tests: Valid VCDIFF files that should decode successfully
- 33 targeted negative tests: Invalid VCDIFF files that should be rejected with appropriate errors
- 32 targeted positive tests: Specific feature validation tests
- Total: 85 test cases with 100% pass rate
- General Positive Tests: 20/20 passed ✅
- Targeted Negative Tests: 33/33 passed ✅
- Targeted Positive Tests: 32/32 passed ✅
Contributions are welcomed. Please follow these guidelines:
- Fork the repository
- Clone your fork with submodules:
git clone --recursive <your-fork-url> - Create a feature branch:
git checkout -b feature/your-feature-name - Make your changes
- Test your changes thoroughly
- Submit a pull request
- Code Style: Follow PEP 8 formatting (
black .for automatic formatting) - Type Hints: All new code should include proper type annotations
- Testing: All new features must include tests
- Documentation: Update documentation for any API changes
- Commits: Use clear, descriptive commit messages
Ensure your contribution passes all checks:
# Run all tests
pytest
# Format code
black .
# Lint code
flake8 vcdiff/
# Type checking
mypy vcdiff/When reporting bugs, please include:
- Python version
- Operating system
- Minimal reproduction case
- Expected vs actual behavior
- Sample VCDIFF files (if applicable)
For new features, please:
- Check existing issues first
- Describe the use case
- Provide RFC 3284 references if applicable
- Consider backwards compatibility
- Python: 3.8 or higher
- Development Dependencies:
pytest>=7.0.0pytest-cov>=4.0.0black>=22.0.0flake8>=5.0.0mypy>=1.0.0
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
- RFC 3284: The VCDIFF Generic Differencing and Compression Data Format
- xdelta3: VCDIFF binary diff tool
- Go VCDIFF Implementation - Algorithmically compatible sibling project