`SensitiveString::serialize` destroys values during figment/serde round-trips

[kazmosahebi]

Problem

SensitiveString::serialize (src/sensitive.rs:84-88) always returns the literal constant ***REDACTED***:

impl serde::Serialize for SensitiveString {
    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
        serializer.serialize_str(REDACTED)
    }
}

This is correct for log dumps / /config endpoint output. But any DFE service that does a serde round-trip on its Config — serialize to a Value/dict, merge env overrides, deserialize back — irretrievably destroys every SensitiveString field. The deserialized struct contains SensitiveString::from("***REDACTED***") instead of the original value.

Reproducer

dfe-loader's Config::load calls apply_figment_env:

let figment = Figment::from(Serialized::defaults(&*config))   // ← serializes config
    .merge(Env::prefixed(&format!("{ENV_PREFIX}_")).split("__"));
*config = figment.extract().map_err(...)?;                    // ← deserializes back

After this call, config.clickhouse.password is SensitiveString("***REDACTED***") regardless of what the YAML or env vars provided. Confirmed end-to-end:

1. YAML: clickhouse.password: env:CLICKHOUSE_PASSWORD
2. After serde_yaml_ng::from_str: password = SensitiveString("env:CLICKHOUSE_PASSWORD") ✓
3. After apply_figment_env: password = SensitiveString("***REDACTED***") ✗
4. Downstream consumer (ClickHouse HTTP client) authenticates with the literal "***REDACTED***" string → auth fails.

Confirmed prior-art workaround

dfe-transform-vector has explicit documentation acknowledging this bug:

NOTE: password is String, not SensitiveString, because the config goes through a figment serialize→merge→deserialize round-trip in apply_figment_env(). SensitiveString serialises as ***REDACTED*** which destroys the value during the round-trip.

…and uses plain String for its SASL password to work around it. That's safe for vector because the value is interpolated by Vector at runtime, but it forfeits all the masking guarantees SensitiveString was supposed to provide.

Why this matters now

The dfe-loader env:VAR credential feature (hyperi-io/dfe-loader#56, depends on #40) is fundamentally blocked by this bug: by the time the credential resolver runs at startup, the spec string in config.clickhouse.password has already been replaced with "***REDACTED***" by the figment round-trip, so the resolver has nothing to resolve.

Local diagnostic (revert before committing): temporarily changing password: SensitiveString → password: String in dfe-loader confirms make dev starts cleanly and clickhouse auth succeeds.

Proposed fixes (pick one)

Option A — Thread-local exposure flag (recommended)

Add a process-local toggle to SensitiveString::serialize:

thread_local! {
    static EXPOSE: Cell<bool> = Cell::new(false);
}

impl Serialize for SensitiveString {
    fn serialize<S: Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
        if EXPOSE.with(|e| e.get()) {
            s.serialize_str(&self.0)   // expose
        } else {
            s.serialize_str(REDACTED)  // redact (default)
        }
    }
}

pub fn expose_during<F, R>(f: F) -> R
where F: FnOnce() -> R {
    EXPOSE.with(|e| e.set(true));
    let r = f();  // ideally drop-guard if f can panic
    EXPOSE.with(|e| e.set(false));
    r
}

Consumers wrap their figment round-trip in expose_during(|| { ... }). Defaults stay redacted, no per-field enumeration, single point of change. Bonus: dfe-transform-vector can drop its String workaround and reinstate SensitiveString.

Option B — SensitiveStringExposed wrapper

Provide a separate type that serializes its inner value verbatim. Consumers convert temporarily before the round-trip. Verbose at the call site, no global state.

Option C — Field-level #[serde(serialize_with = ...)] opt-in

Provide a serialize_exposed free function consumers can wire onto specific fields. Doesn't fix the underlying type but works as an escape hatch.

Acceptance

• A SensitiveString round-tripped through serde_json::to_value → serde_json::from_value preserves its value (when exposure is enabled).
• Default behaviour (no exposure enabled) continues to redact — every existing test must still pass.
• Documentation explains when to use the exposure helper (config processing) vs. when not to (logging, dumps).
• dfe-loader's apply_figment_env adopts the new helper; dfe-loader integration test verifies clickhouse password survives the cascade.

Related

• hyperi-io/dfe-loader#56 — env:VAR credentials feature (blocked by this)
• Extract credential::resolve from dfe-fetcher into shared module #40 — credential resolver extraction (Phase 1 of the env:VAR work)
• dfe-transform-vector's String-instead-of-SensitiveString workaround