feat: add optional drift-core rust acceleration path

[jy-tan]

Summary

Integrate optional drift-core-python acceleration into the Python SDK behind fail-open feature flags, and wire e2e/benchmark flows to use SDK extras instead of ad-hoc version-pinned installs. See drift-core.

Changes

• Add optional Rust binding dependency via pyproject.toml extra:
  • rust = ["drift-core-python>=0.1.6"]
• Introduce Rust bridge module:
  • add drift/core/rust_core_binding.py
  • load drift_core only when TUSK_USE_RUST_CORE is enabled
  • keep fail-open behavior (fallback to Python logic on import/call failure)
• Hook Rust-backed paths into tracing/export hot paths:
  • payload/schema/hash conversion flow updates in core tracing helpers
  • span/export protobuf-byte construction path updates
  • mixed-path fallback behavior improvements
• Update e2e benchmark setup to consume SDK [rust] extra instead of direct version-plumbed binding installs:
  • changes in drift/instrumentation/e2e_common/base_runner.py
  • docker compose updates for Django/FastAPI/Flask e2e suites
• Document new runtime controls and setup:
  • expand docs/environment-variables.md with Rust flags
  • add docs/rust-core-bindings.md
• Refresh ignore/config support files (.gitignore, .dockerignore) and dependency lock updates (uv.lock)

Notes

• Rust path remains opt-in via TUSK_USE_RUST_CORE.
• SDK remains fail-open by design: missing/failed Rust binding calls fall back to Python behavior.

[cubic-dev-ai]

4 issues found across 17 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="drift/core/json_schema_helper.py">

<violation number="1" location="drift/core/json_schema_helper.py:163">
P2: Inconsistent `None` check: this line uses `or` (falsy check), but `generate_deterministic_hash` uses `is not None`. An empty-string return would behave differently. Additionally, when `rust_decoded_hash` is `None`, the fallback `generate_deterministic_hash` will redundantly call `deterministic_hash_jsonable(decoded)` a second time. Use an explicit `if ... is not None` check and avoid the double call.</violation>

<violation number="2" location="drift/core/json_schema_helper.py:169">
P2: Same inconsistent `or` vs `is not None` pattern and redundant Rust call as the `decoded_value_hash` line. Use an explicit `is not None` check for consistency with `generate_deterministic_hash`.</violation>
</file>

<file name="drift/core/tracing/otel_converter.py">

<violation number="1" location="drift/core/tracing/otel_converter.py:462">
P2: The Rust fast-path now replaces input/output schemas with empty JsonSchema objects when `span_proto_bytes` is present, but filesystem export still serializes `span.input_schema`/`span.output_schema`. This drops schema data whenever Rust acceleration is enabled. Keep the schema conversion so downstream exports retain schema content.</violation>
</file>

<file name="drift/core/tracing/adapters/api.py">

<violation number="1" location="drift/core/tracing/adapters/api.py:209">
P2: `span.to_proto()` drops dict metadata because `clean_span_to_proto` only serializes `metadata.__dict__`. This regresses span exports (metadata omitted) when `proto_span_bytes` is missing or Rust serialization falls back. Consider using `_transform_span_to_protobuf` for Python serialization or updating `clean_span_to_proto` to handle dict metadata before switching to `to_proto`.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}