Skip to content

Commit ceca117

Browse files
committed
docs: explain S3 object-store configuration for SQL
Signed-off-by: Yifan Chen <emecii23@gmail.com>
1 parent b6c6f5b commit ceca117

1 file changed

Lines changed: 27 additions & 0 deletions

File tree

docs/source/user-guide/data-sources.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,11 +100,16 @@ Supported Object Stores are
100100
- {py:class}`~datafusion.object_store.MicrosoftAzure`
101101

102102
```python
103+
import os
104+
105+
from datafusion import SessionContext
103106
from datafusion.object_store import AmazonS3
104107

105108
region = "us-east-1"
106109
bucket_name = "yellow-trips"
107110

111+
ctx = SessionContext()
112+
108113
s3 = AmazonS3(
109114
bucket_name=bucket_name,
110115
region=region,
@@ -120,6 +125,28 @@ ctx.register_parquet("trips", path)
120125
ctx.table("trips").show()
121126
```
122127

128+
### Use S3 in SQL
129+
130+
Configure S3 access on an {py:class}`~datafusion.object_store.AmazonS3` object and
131+
register it on the context before issuing SQL that uses an `s3://` location. AWS
132+
credentials are not SQL `OPTIONS`: `aws.*` is not a recognized SQL configuration
133+
namespace.
134+
135+
After registering the object store above, a SQL external table can use the same
136+
S3 path:
137+
138+
```python
139+
ctx.sql(
140+
f"""
141+
CREATE EXTERNAL TABLE trips_sql
142+
STORED AS PARQUET
143+
LOCATION '{path}'
144+
"""
145+
).collect()
146+
147+
ctx.sql("SELECT count(passenger_count) FROM trips_sql").show()
148+
```
149+
123150
## Other DataFrame Libraries
124151

125152
DataFusion can import DataFrames directly from other libraries, such as

0 commit comments

Comments
 (0)