feat: add optional drift-core rust acceleration path

.dockerignore

@@ -20,6 +20,8 @@ __pycache__/
 # Build artifacts
 build/
 dist/
+target/
+**/target/
 *.egg-info/
 .eggs/
 eggs/

.github/workflows/e2e.yml

@@ -103,6 +103,7 @@ jobs:
         env:
           DOCKER_DEFAULT_PLATFORM: linux/amd64
           TUSK_CLI_VERSION: ${{ steps.tusk-version.outputs.version }}
+          TUSK_USE_RUST_CORE: "1"
         run: |
           chmod +x ./drift/instrumentation/${{ matrix.library }}/e2e-tests/run.sh
           cd ./drift/instrumentation/${{ matrix.library }}/e2e-tests && ./run.sh 8000
@@ -180,6 +181,7 @@ jobs:
         env:
           DOCKER_DEFAULT_PLATFORM: linux/amd64
           TUSK_CLI_VERSION: ${{ steps.tusk-version.outputs.version }}
+          TUSK_USE_RUST_CORE: "1"
         run: |
           chmod +x ./drift/stack-tests/${{ matrix.test }}/run.sh
           cd ./drift/stack-tests/${{ matrix.test }} && ./run.sh 8000

.gitignore

@@ -229,4 +229,6 @@ __marimo__/
 # Coverage
 coverage.lcov
 coverage.xml
-.coverage
+.coverage
+
+experimental/

docs/environment-variables.md

@@ -130,6 +130,33 @@ TUSK_SAMPLING_RATE=0.1 python app.py
 
 For more details on sampling rate configuration methods and precedence, see the [Initialization Guide](./initialization.md#configure-sampling-rate).
 
+## Rust Core Flags
+
+These variables control optional Rust-accelerated paths in the SDK.
+
+| Variable | Description | Default |
+| --- | --- | --- |
+| `TUSK_USE_RUST_CORE` | Enables Rust binding usage when available (`1`, `true`, `yes`) | `0` (disabled) |
+| `TUSK_SKIP_PROTO_VALIDATION` | Skips expensive protobuf validation in hot path (`1`, `true`, `yes`) | `0` (disabled) |
+
+**Notes:**
+
+- The SDK is fail-open: if Rust bindings are unavailable or a Rust call fails, it falls back to Python implementation.
+- `TUSK_USE_RUST_CORE` does not install Rust bindings automatically. The `drift-core-python` package still must be installed in your environment.
+- `TUSK_SKIP_PROTO_VALIDATION` is performance-focused and should be used with confidence in parity tests and serialization correctness.
+
+See [`rust-core-bindings.md`](./rust-core-bindings.md) for more details.
+
+**Example usage:**
+
+```bash
+# Enable Rust path (if drift-core-python is installed)
+TUSK_USE_RUST_CORE=1 python app.py
+
+# Enable Rust path and skip proto validation
+TUSK_USE_RUST_CORE=1 TUSK_SKIP_PROTO_VALIDATION=1 python app.py
+```
+
 ## Connection Variables
 
 These variables configure how the SDK connects to the Tusk CLI during replay:
@@ -142,9 +169,7 @@ These variables configure how the SDK connects to the Tusk CLI during replay:
 
 These are typically set automatically by the Tusk CLI and do not need to be configured manually.
 
----
-
-## Related Documentation
+## Related Docs
 
 - [Initialization Guide](./initialization.md) - SDK initialization parameters and config file settings
 - [Quick Start Guide](./quickstart.md) - Record and replay your first trace

docs/rust-core-bindings.md

@@ -0,0 +1,91 @@
+# Rust Core Bindings
+
+This document explains how Rust acceleration works in the Python SDK, how to enable it, and what fallback behavior to expect.
+
+## Overview
+
+The SDK can offload selected hot-path logic to Rust through optional Python bindings (`drift-core-python`), defined in the [`drift-core`](https://github.com/Use-Tusk/drift-core) repository. This is controlled by environment flags and is designed to fail open.
+
+At a high level:
+
+- Python SDK logic remains the source of truth.
+- Rust paths are opportunistic optimizations.
+- If Rust is unavailable or fails at runtime, SDK behavior falls back to Python equivalents.
+
+## Enablement
+
+Set:
+
+```bash
+TUSK_USE_RUST_CORE=1
+```
+
+Truthy values are `1`, `true`, and `yes` (case-insensitive). Any other value is treated as disabled.
+
+## Installation Requirements
+
+Rust acceleration requires the `drift-core-python` package to be installed in the runtime environment.
+
+Notes:
+
+- The SDK does not auto-install this package at runtime.
+- If the package is missing or cannot be imported on a machine, the SDK continues on Python code paths.
+
+You can install the SDK with Rust bindings via extras:
+
+```bash
+pip install "tusk-drift-python-sdk[rust]"
+```
+
+## Wheel Platform Coverage
+
+Based on the current `drift-core` publish workflow, prebuilt wheels are built for:
+
+- Linux `x86_64-unknown-linux-gnu`
+- Linux `aarch64-unknown-linux-gnu`
+- macOS Apple Silicon `aarch64-apple-darwin`
+- Windows `x86_64-pc-windows-msvc`
+
+Likely missing prebuilt wheels (source build fallback required) include:
+
+- macOS Intel (`x86_64-apple-darwin`)
+- Linux musl targets (e.g. Alpine)
+- Windows ARM64
+- Other uncommon Python/platform combinations not covered by release artifacts
+
+If no wheel matches the environment, `pip` may attempt a source build of `drift-core-python`, which typically requires a Rust toolchain and native build prerequisites.
+
+## Fallback Behavior
+
+The bridge module is fail-open:
+
+- Rust calls are guarded.
+- On import failures or call exceptions, the corresponding helper returns `None`.
+- Calling code then uses the existing Python implementation.
+
+This means users do not need Rust installed to run the SDK when Rust acceleration is disabled or unavailable.
+
+## Optional Performance Flag
+
+```bash
+TUSK_SKIP_PROTO_VALIDATION=1
+```
+
+This skips expensive protobuf validation checks in hot paths.
+
+Use with care:
+
+- Recommended only when parity/smoke tests are healthy.
+- Keep it off in environments where strict serialization verification is preferred.
+
+## Practical Guidance
+
+- Default production-safe posture: leave Rust disabled unless you have tested your deployment matrix.
+- Performance posture: enable Rust + benchmark on your workloads before broad rollout.
+- Reliability posture: keep parity tests and smoke tests in CI to detect drift between Python and Rust paths.
+
+## Related Docs
+
+- [Environment Variables](./environment-variables.md)
+- [Initialization Guide](./initialization.md)
+- [Context Propagation](./context-propagation.md)

drift/core/json_schema_helper.py

@@ -10,6 +10,8 @@
 from enum import Enum
 from typing import Any
 
+from .rust_core_binding import deterministic_hash_jsonable, normalize_and_hash_jsonable, normalize_json_jsonable
+
 
 class JsonSchemaType(Enum):
     UNSPECIFIED = 0
@@ -132,11 +134,49 @@ def generate_schema(data: Any, schema_merges: SchemaMerges | None = None) -> Jso
 
     @staticmethod
     def generate_schema_and_hash(data: Any, schema_merges: SchemaMerges | None = None) -> SchemaComputationResult:
-        normalized = JsonSchemaHelper._normalize_data(data)
-        decoded = JsonSchemaHelper._decode_with_merges(normalized, schema_merges)
+        # Convert non-JSON primitives once. Rust/Python paths consume this shared shape.
+        sanitized = JsonSchemaHelper._to_jsonable(data)
+
+        normalized: Any
+        decoded: Any
+        decoded_value_hash: str
+
+        # If there are no merges, use one coarse Rust call for normalize+hash.
+        if not schema_merges:
+            rust_normalized = normalize_and_hash_jsonable(sanitized)
+            if rust_normalized is not None:
+                normalized, decoded_value_hash = rust_normalized
+            else:
+                normalized = json.loads(json.dumps(sanitized))
+                decoded_value_hash = JsonSchemaHelper.generate_deterministic_hash(normalized)
+            decoded = normalized
+        else:
+            # Merges require decode before value-hash. Let Rust handle normalize only.
+            rust_normalized_only = normalize_json_jsonable(sanitized)
+            if rust_normalized_only is not None:
+                normalized = rust_normalized_only
+            else:
+                normalized = json.loads(json.dumps(sanitized))
+
+            decoded = JsonSchemaHelper._decode_with_merges(normalized, schema_merges)
+            rust_decoded_hash = deterministic_hash_jsonable(decoded)
+            if rust_decoded_hash is not None:
+                decoded_value_hash = rust_decoded_hash
+            else:
+                sorted_decoded = JsonSchemaHelper._sort_object_keys(decoded)
+                payload = json.dumps(sorted_decoded, ensure_ascii=False, separators=(",", ":"))
+                decoded_value_hash = hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
         schema = JsonSchemaHelper.generate_schema(decoded, schema_merges)
-        decoded_value_hash = JsonSchemaHelper.generate_deterministic_hash(decoded)
-        decoded_schema_hash = JsonSchemaHelper.generate_deterministic_hash(schema.to_primitive())
+
+        schema_primitive = schema.to_primitive()
+        rust_schema_hash = deterministic_hash_jsonable(schema_primitive)
+        if rust_schema_hash is not None:
+            decoded_schema_hash = rust_schema_hash
+        else:
+            sorted_schema = JsonSchemaHelper._sort_object_keys(schema_primitive)
+            payload = json.dumps(sorted_schema, ensure_ascii=False, separators=(",", ":"))
+            decoded_schema_hash = hashlib.sha256(payload.encode("utf-8")).hexdigest()
         return SchemaComputationResult(
             schema=schema,
             decoded_value_hash=decoded_value_hash,
@@ -145,6 +185,9 @@ def generate_schema_and_hash(data: Any, schema_merges: SchemaMerges | None = Non
 
     @staticmethod
     def generate_deterministic_hash(data: Any) -> str:
+        rust_hash = deterministic_hash_jsonable(data)
+        if rust_hash is not None:
+            return rust_hash
         sorted_data = JsonSchemaHelper._sort_object_keys(data)
         payload = json.dumps(sorted_data, ensure_ascii=False, separators=(",", ":"))
         return hashlib.sha256(payload.encode("utf-8")).hexdigest()