to_csv to Google Cloud Storage ignores encoding

[vackosar]

Code Sample, a copy-pastable example if possible

        df.to_csv(path_or_buf='gs://mybucket/my_file', encoding='utf8')

Problem description

Instead of expected UTF-8 file is stored in latin-1 on Windows. I can also see that current implementation of method pandas.io.gcs.get_filepath_or_buffer ignores encoding argument.

I can submit PR if are open to it.

Expected Output

Uploaded file encoded in UTF8 even on Windows.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-17-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: None
pip: 10.0.1
setuptools: 39.1.0
Cython: None
numpy: 1.16.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: 0.9.0
pandas_datareader: None
gcsfs: 0.2.1

[WillAyd]

Not sure this is even supported by gcsfs - I don't see encoding as an available option there:

https://github.com/dask/gcsfs/blob/523eb65b3e7feb05f9c10ce84523d1058716fecf/gcsfs/core.py#L1133

Might need to start upstream if wanted to make this possible

[mstauber1]

I've encountered this issue with AWS s3 as well. Encoding is ignored.

[EgorBEremeev]

Not sure this is even supported by gcsfs - I don't see encoding as an available option there:

https://github.com/dask/gcsfs/blob/523eb65b3e7feb05f9c10ce84523d1058716fecf/gcsfs/core.py#L1133

Might need to start upstream if wanted to make this possible

Hi, @WillAyd
In the current gcsf master GCSFileSystem.open() has been removed and fsspec.AbstractFileSystem.open() has works instead:

https://github.com/intake/filesystem_spec/blob/4c66e096d32dafe264e2d6707992ee6935685944/fsspec/spec.py#L717

where applying of passed encoding for the text reading\writing is now implemented:

        if "b" not in mode:
            mode = mode.replace("t", "") + "b"

            text_kwargs = {
                k: kwargs.pop(k)
                for k in ["encoding", "errors", "newline"]
                if k in kwargs
            }
            return io.TextIOWrapper(
                self.open(path, mode, block_size, **kwargs), **text_kwargs
            )

Note also this issue from gcsfs

So, it looks that in pandas ignoring of encoding parameter happens because in the pandas.io.gcs.get_filepath_or_buffer the mode = 'rb' is passed to call of GCSFileSystem.open(filepath_or_buffer, mode)

Tracing back to the moment of the first actual setting the mode parameter we have stop on this line:

pandas.io.common.py

def get_filepath_or_buffer(
    filepath_or_buffer, encoding=None, compression=None, mode=None
)

It is so, because in the call of get_filepath_or_buffer() performed from here

pandas/pandas/io/parsers.py

Line 430 in 29d6b02

fp_or_buf, _, compression, should_close = get_filepath_or_buffer(

we do not pass value of mode and default mode=None works.

As I could suggest for read_csv() we need pass mode=r and for to_csv() we need pass mode=w in the call of get_filepath_or_buffer(). But I'm not sure where it's better to implement this change.