Description
Code Sample, a copy-pastable example if possible
import pandas as pd
>>> df = pd.read_json("file://localhost/home/ubuntu/test.json", lines=True)
>>> df.to_json("test-out.json", orient="records", lines=True)
>>> df = pd.read_json("file://localhost/home/ubuntu/test2.json", lines=True)
>>> df.to_json("test2-out.json", orient="records", lines=True)Problem description
I noticed this issue when I saw that a ~8500 row dframe became a ~3400 line file, even though in theory "lines=True" should mean that the number of rows and number of lines are equal. I dug into the code a bit, and it turns out that convert_json_to_lines() does not correctly insert newlines if the json contains a backslash before a double quote, even if the backslash itself is escaped.
(https://github.com/pandas-dev/pandas/blob/v0.19.2/pandas/lib.pyx#L1114)
Here are the test files I used with the code sample above, and the outputs I got:
ubuntu@ip:~$ cat test.json
{"test": "qwer"}
{"test": "asdf \\"}
{"test": "zxcv"}
ubuntu@ip:~$ cat test-out.json
{"test":"qwer"}
{"test":"asdf \\"},{"test":"zxcv"}ubuntu@ip:~$
ubuntu@ip:~$ cat test2.json
{"test": "123"}
{"test": "234 \\"}
{"test": "345"}
{"test": "456 \\"}
{"test": "567"}
ubuntu@ip:~$ cat test2-out.json
{"test":"123"}
{"test":"234 \\"},{"test":"345"},{"test":"456 \\"}
{"test":"567"}ubuntu@ip:~$
Expected Output
"-out" files should be identical to originals.
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-68-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.19.1
nose: None
pip: 9.0.1
setuptools: 27.2.0
Cython: None
numpy: 1.11.2
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: 2.43.0
pandas_datareader: None