BUG: Fix #15344 by backporting ujson usage of PEP 393 API

[tobgu]

Make use of the PEP 393 API to avoid expanding single byte ascii characters into four byte unicode characters when encoding objects to json.

• closes Calling DataFrame.to_json() increases data frame memory usage in Python 3.6 #15344
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

[jreback]

lgtm. can you add a whatsnew note? (Bug fix is fine)

[jreback]

https://travis-ci.org/pandas-dev/pandas/jobs/200143310

we have a c-linter :<

[tobgu]

Ouch :-). Hopefully fixed now.

[jreback]

thanks @tobgu

keep em coming!

I am still puzzled why the encoding of a new UTF8 string, that returns a new reference is affecting existing references. Though this maybe some deep things going on that I don't understand. If you have time I would search / post an issue on CPython to understand the semantics here. Its also possible that this is the reason this function is being deprecated (because of these side effects).

PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
Return value: New reference.
Encode the Py_UNICODE buffer s of the given size using UTF-8 and return a Python bytes object. Return NULL if an exception was raised by the codec.

Part of the old-style Py_UNICODE API; please migrate to using PyUnicode_AsUTF8String() or PyUnicode_AsUTF8AndSize().
Deprecated since version 3.3, will be removed in version 4.0.

[tobgu]

I think the culprit in this case is the PyUnicode_AS_UNICODE macro that's described as inefficient and something you should migrate away from:
https://www.python.org/dev/peps/pep-0393/#porting-guidelines
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AS_UNICODE

It seems like there's actually an allocation for a full strlen*4 byte unicode string here that would be the root cause of the observed behaviour:
https://github.com/python/cpython-mirror/blob/9aa885c9df5b6895f70272b6abc24ef0b1fb8b46/Objects/unicodeobject.c#L4089

Haven't dug into it further than that...

Thanks @jreback and @chris-b1 for helping out with this! It would have been a show stopper for the application that I'm currently working on. Hope to see it released soon! :-)

[jreback]

thanks for looking @tobgu, that is some nasty code.