still doesn't work with boto3 StreamingBody

[uiur]

ref: #16135 #16150

Example

s3_object = client.get_object(Bucket=bucket, Key=key)
result = read_csv(s3_object["Body"])
# ValueError: Invalid file path or buffer object type: <class 'botocore.response.StreamingBody'>

Problem description

issue: is_file_like requirements are too strict for boto3 S3 objects #16135
pull request: #16150

This issue was closed but it's not working. I've found that the test is skipped and has a bug.
I'll send a pull request to reproduce the behavior.

I think it's because botocore.response.StreamingBody doesn't have __iter__ so is_file_like returns False.

INSTALLED VERSIONS

commit: c8dcf19
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL:
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.21.0.dev+317.gc8dcf19
pytest: 3.1.3
pip: 7.1.2
setuptools: 36.2.6
Cython: 0.26
numpy: 1.13.1
scipy: None
pyarrow: None
xarray: None
IPython: 5.0.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: 0.1.2
pandas_gbq: None
pandas_datareader: None

[uiur]

cc @gfyoung

[gfyoung]

@uiureo : Thanks for the report! From our conversation back in that issue, we got the impression that the PR would address this issue. Network tests are more difficult to reproduce because they are out of our control, but that's a little unfortunate that the purported patch failed to address this.

[gfyoung]

One issue currently is that we are facing this delicate balancing act between accepting as many "valid" IO objects as possible without allowing for invalid ones. We check for __iter__ because of #15337, as that is a robust method of rejecting such objects.

For your patch to be robust, you will need to find a way that rejects mock.Mock() while allowing for StreamingBody. That would be fantastic if you could!

[uiur]

This issue would be resolved if this PR boto/botocore#1195 is merged at botocore.

[gfyoung]

@uiureo : Thanks for letting us know! That would definitely be one way to resolve this issue.

[stonefly]

It's been widely accepted that as long as an object has read or write function it can be seen as a file-like object.

Much more importantly, this is the behavior before pandas 0.20.*.

A lot of codes relying on this assumption now stop working.

I'm wondering what is the point to re-define what is file-like here? Is the iter actually used in the implementation of read_csv for example?