Feature to read csv from hdfs:// URL

[AbdealiLoKo]

When running pandas in AWS, The following works perfectly fine:

pd.read_csv("s3://mybucket/data.csv")

But running the following, does not:

pd.read_csv("hdfs:///tmp/data.csv")

It would be a good user experience to allow for the hdfs:// schema too similar to how http, ftp, s3, and file are valid schemas right now.

[TomAugspurger]

Based on the limited bit I know, dealing with authentication and all that can be a rabbit hole.

If you want to put together a prototype based around http://hdfs3.readthedocs.io/en/latest/, I think we'd add it.

[AbdealiLoKo]

How should this be implemented ? Should there also be a read_hdfs like the read_s3 ?

[TomAugspurger]

See https://github.com/pandas-dev/pandas/blob/dc4b0708f36b971f71890bfdf830d9a5dc019c7b/pandas/io/s3.py and

pandas/pandas/io/common.py

Line 94 in 2f9d4fb

def _is_s3_url(url):

and search for uses of _is_s3_url.

[jreback]

i believe we can either use hdfs (similar to s3fs and/or pyarrow for this); would be similar to the way we do s3 atm

[jreback]

https://hdfs3.readthedocs.io/en/latest/

[jreback]

https://arrow.apache.org/docs/python/filesystems.html

[AbdealiLoKo]

So, here is a quick comparison:

• hdfs: Package for connecting to WebFS and HttpFS which are REST protocols to access HDFS data
• hdfs3: Wrapper on the library libhdfs3 which needs to be installed independently
• pyarrow: Supports both engines the native libhdfs and separately installed libhdfs3
• cyhdfs: Cython wrapper for native libhdfs

These seem to be the active (have their latest release in 2017) options.

As pandas already has a pyarrow engine for parquet, it looks like having pyarrow with the native libhdfs would be the most universal option.