IO Error: Cannot open file, while connecting to S3

[klonuo]

What happens?

When I try to connect, it prepends my working folder in provided db path

To Reproduce

In [6]: config.keys()
Out[6]: dict_keys(['s3_region', 's3_endpoint', 's3_access_key_id', 's3_secret_access_key'])

In [7]: c = duckdb.connect('s3://air_pollution/duckdb/mk.duckdb', config=config)
---------------------------------------------------------------------------
IOException                               Traceback (most recent call last)
Cell In[7], line 1
----> 1 c = duckdb.connect('s3://air_pollution/duckdb/mk.duckdb', config=config)

IOException: IO Error: Cannot open file "c:\temp\s3:\\air_pollution\duckdb\mk.duckdb": The filename, directory name, or volume label syntax is incorrect.


In [8]: %pwd
Out[8]: 'c:\\temp'

In [9]: duckdb.__version__
Out[9]: '1.4.2'

OS:

Windows

DuckDB Package Version:

1.4.2

Python Version:

3.12

Full Name:

klo

Affiliation:

acme

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[evertlammerts]

connect(path) supports the local filesystem only, unless you use fsspec.

Try instead something like:

con = duckdb.connect(config=config)
con.sql("LOAD HTTPFS")
con.sql("SELECT * FROM 's3://air_pollution/duckdb/mk.duckdb'")

See more information here.

[klonuo]

Thanks, I managed to connect with httpfs, but cannot attach in R/W mode:

n [48]: conn.execute("ATTACH 's3://air_pollution/duckdb/mk.duckdb' AS db (READ_WRITE)")
---------------------------------------------------------------------------
NotImplementedException                   Traceback (most recent call last)
Cell In[48], line 1
----> 1 conn.execute("ATTACH 's3://air_pollution/duckdb/mk.duckdb' AS db (READ_WRITE)")

NotImplementedException: Not implemented Error: Cannot open an HTTP file for both reading and writing

I tried also fsspec, but as I see I can only download (fs.get) the database locally work on it, then upload (fs.put), which is overkill as I wanted to update the database on a daily bases, and the database is relatively large

I guess my only way is to go with parquet files...

[paultiq]

FYI: https://duckdb.org/docs/stable/guides/network_cloud_storage/duckdb_over_https_or_s3 documents the limitation that "Only read-only connections are allowed"

[klonuo]

Ok, thanks, if anyone reads this I ended up with Iceberg catalog for R/W access as the simplest way for my use case