Skip to content

ParquetWriterExec's derived Debug prints object store options, which can hold secret keys #7558

Open

Description

@dwsmith1983

Describe the bug

Found while working on #7128. ParquetWriterExec in native/core/src/execution/operators/parquet_writer.rs derives Debug, so {:?} of the operator prints output_path and object_store_options as they are. The options come from NativeConfig.extractObjectStoreOptions and can hold credentials such as fs.s3a.secret.key or an Azure SAS token, and the path can carry a password or a SAS query.

Comet does not format the plan with {:?} today, so nothing leaks yet. A debug log, a dbg!, a test assertion or a DataFusion error that formats the plan with Debug would print the secrets.

Steps to reproduce

Format a ParquetWriterExec built with fs.s3a.secret.key in its options with {:?}. The key's value appears in the output.

Expected behavior

Debug for ParquetWriterExec prints the path without credentials (the helpers #7127 adds) and only the keys of object_store_options, not their values.

Additional context

The plan display (DisplayAs) and the writer's messages are handled in the #7128 pull requests; this is the remaining way the operator can print its options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions