Skip to content

aws.XXX options are not recognized in SQL API for CREATE EXTERNAL TABLE with AWS S3 location #970

Open
@Nick-Nal

Description

@Nick-Nal

Describe the bug
Hi

I was trying to create external table from AWS S3 using examples from Datafusion CLI docs and SQL DDL docs, see To Reproduce section below. But looks like AWS-specific options for external tables doesn't work with SQL API in the same way as it described in CLI docs

Exception: DataFusion error: Configuration("Could not find config namespace \"aws\"")

I can successfully register and read the same parquet with the same credentials via register_object_store + register_parquet functions, so there's no problem with the data or S3 connection

To Reproduce

import datafusion

ctx = datafusion.SessionContext()

raw_data_df = ctx.sql(f"""
    create external table raw_data
    stored as parquet
    options
    (
        'aws.access_key_id' '***',
        'aws.secret_access_key' '***',
        'aws.region' 'eu-central-1'
    ) 
    location 's3://some-bucket/some-folder/'
""")

raw_data_df.show()

Expected behavior
create external table command works with aws.xxx options

Additional context
v 42.0.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions