Skip to content

Native object-store errors and debug logs print whole URLs outside the Azure store #7128

Open

Description

@dwsmith1983

Describe the bug

Found in #7127, which keeps URL passwords and SAS queries out of the Azure store, the object store resolution in parquet_support.rs, and the driver's ABFS messages. Several other native messages still print a whole URL, so a password in the userinfo or a token in the query would reach task errors and logs. Line numbers are as of 39380cc.

  • native/core/src/parquet/objectstore/s3.rs:72 and :76: ObjectStoreScheme::parse(url) quotes the whole URL in Unable to recognise URL "{url}", and Scheme of URL is not S3: {url}.
  • native/core/src/parquet/objectstore/s3_blob_fs_support.rs:138: the whole URL when the bucket name is missing. :171: Error parsing URL {path} prints the raw string.
  • native/core/src/parquet/parquet_support.rs:794-796 (feature hdfs-opendal): get_name_node_uri returns InvalidPath with the whole URL for a hostless URL, so a query would be echoed.
  • native/core/src/parquet/parquet_support.rs:1016 (parse_url for schemes other than S3, Azure and libhdfs): the full URL goes to object_store's builders, whose errors can quote it.
  • native/core/src/execution/operators/parquet_writer.rs:312, :326, :342, :407 and the HDFS writer messages at :131-219: errors format the raw output path. The JVM only sends file: and hdfs: paths here, so only an hdfs://user:pw@... path would carry credentials.

None of these is easy to reach from a normal Spark file listing, which doesn't put credentials or a query in the paths it hands the native scan. So this is defense in depth, like #7127.

Steps to reproduce

Unit-level: call the functions above with a URL like s3://KEY:SECRET@bucket/path?sig=TOKEN, or with an unrecognised scheme carrying the same userinfo and query, and look at the error text.

Expected behavior

Messages name a URL by its scheme and host. They never include the password or the query. The helpers in native/core/src/parquet/objectstore/url_redaction.rs from #7127 show one way to do it.

Activity

  1. dwsmith1983 commented on Oct 21, 2026

    @dwsmith1983
    ContributorAuthor

    Two more sites, found in #7127 (review), both in the Delta contrib:

    • native/core/src/execution/planner/delta_spark_scan.rs:336: redacted_url_display returns a URL without userinfo unchanged, and after masking userinfo it appends the query again, so a SAS token or signature survives either way. It could move next to the helpers in url_redaction.rs and drop the query.
    • contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:574: objectStoreRejectedPathReason prints uri.toString when there is no userinfo, which includes the query.
  2. added
    bugSomething isn't working
    priority:criticalData corruption, silent wrong results, security issues
    area:scanParquet scan / data reading
    and removed on Oct 26, 2026
  3. added this to the 1.3.0 milestone on Oct 26, 2026
  4. dwsmith1983 commented on Nov 7, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingbugSomething isn't workingpriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions