is_s3_url will not work

[cpcloud]

as per the amazon docs

"Any bucket with a DNS compatible name may be referenced as follows: http://[BucketName].s3.amazonaws.com/[Filename], for example, http://images.johnsmith.net.s3.amazonaws.com/mydog.jpg. Using CNAME you can map images.johnsmith.net to an Amazon S3 host name so the previous URL could become: http://images.johnsmith.net/mydog.jpg."

this function is looking at the scheme which will never test as 's3' since that is not (to my knowledge) a valid URL protocol

[cpcloud]

there are also no tests for this not sure why it was merged...cc #3504 .

[jreback]

as a side issue I think the authentication parms for this should be passable as parameters (and only fall back to env variables); maybe a little tricky to allow this though

[cpcloud]

this:

# Assuming AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
# are environment variables

is kind of scary since u can just check the key membership in os.environ

[cpcloud]

should there be a read_s3 function associated with this? otherwise it just reads into a string. as it stands it won't ever read an s3 bucket.

[cpcloud]

should just try to import if that passes try to read if that fails then continue and try from other sources

[cpcloud]

@pjob can u write some tests for this and fix the behavior accordingly?

[cpcloud]

i think this should be removed until the original author submits a working patch with tests...this is currently dead code

Some notes:

• yes, the s3:// scheme is not an IANA registered scheme. This is a common form of abuse (redis, magnet links, db DSN's also borrow this syntax, e.g. postgres).

In [3]: from urlparse import urlparse
In [4]: urlparse("s3://bucketname/my/path/to/data.csv").scheme
Out[4]: 's3'

The python standard library considers this reasonable.

• The AWS docs you quoted cannot differentiate between a user trying to load a public HTTP(s) asset
  which just happens to be served from s3 vs loading a private asset which uses HTTP(s) but requires authorization. That's the reason for using the s3:// scheme.
• Adding a test would require putting a private bucket up and sharing (IAM) keys to it. The code merged was minimal and I was satisfied with a smoke test.
• The reason for merging was being useful and small at the same time.
• Practically everyone who develops in AWS has those envars defined in their .profile as they are
  the same ones used by the official CLI tools and their clones and client libraries which access
  AWS services, boto is one case.
• I'm not sure what's "scary" about that comment. If the envars are undefined you should get a familiar authentication exception from boto.
• S3 buckets are containers of keys not containers of data. You wouldn't "read" a bucket in the usual sense.
  not sure what you meant.
• This works just fine and has since merging. Perhaps the point you're making is that a usage
  example in the docs would have been in order. I agree.

df=pd.read_csv('s3://bucketname/my/faux/path/to/data.csv')

[cpcloud]

@y-p thanks for the notes. sounds good. didn't know that about those other system. sorry for the noise. will open a doc issue