fsspec: Support token in ADLS

[kevinjqliu]

Rationale for this change

Closes #2328

Pass token string as credential in AzureBlobFileSystem, https://fsspec.github.io/adlfs/api/
This is a known workaround as described by Azure/azure-sdk-for-python#9075 (comment)

I've also made a feature request for java implementation apache/iceberg#13818

Note, pyarrow.fs.AzureFileSystem does not currently expose credential as a parameter

Are these changes tested?

Yes, manually since azurite does not integrate with entra.

heres a repro script

from azure.identity import DefaultAzureCredential
import pyarrow as pa

credential = DefaultAzureCredential()
token = credential.get_token("https://storage.azure.com/.default").token

warehouse = "abfss://kevinliu_demo_ws@daily-onelake.blob.fabric.microsoft.com/kevinliu_demo_lh.Lakehouse/Files"
account_name = "daily-onelake"
account_host = f"{account_name}.blob.fabric.microsoft.com"

catalog = load_catalog("default", **{
    "type": "in-memory", 
    "warehouse": warehouse,
    "adls.account-name": account_name,
    "adls.account-host": account_host,
    "adls.token": token,
})

catalog.create_namespace_if_not_exists("default")
try:
    catalog.drop_table("default.test")
except:
    ...


TEST_DATA = {
    "id": [1, 2, 3, 1, 1],
    "name": ["AB", "CD", "EF", "CD", "EF"],
}
arrow_table = pa.Table.from_pydict(TEST_DATA)
tbl = catalog.create_table_if_not_exists("default.test", schema=arrow_table.schema)
tbl.append(arrow_table)
tbl.scan().to_arrow()

Are there any user-facing changes?

[Fokko]

Makes sense, thanks for fixing this @kevinjqliu

@djouallah looking forward seeing your findings 👍

[djouallah]

@Fokko yes it works :)

[Fokko]

@djouallah Awesome, thanks for letting us know!