Plotting performance deterioration on DataFrames with date-index

[davaco]

The below code-sample gives a big performance drop on plotting in pandas 0.12, compared to pandas 0.11:

In Pandas 0.12 : "Ran in 0:00:29.542475 secs"
In Pandas 0.11 : "Ran in 0:00:06.653506 secs"

It only happens on a date-indexed DataFrame:

from pandas import *
from numpy.random import randn

N = 10000
M = 25

df = DataFrame(randn(N,M), index=date_range('1/1/1975', periods=N))

t0 = datetime.now()
df.plot()
print("Ran in %s secs" % (datetime.now() - t0))

[jorisvandenbossche]

As I also wrote on the mailing list, I can confirm this. For me it gives 35 s (0.12) vs 9 s (0.11) on Windows 7 (also both Matplotlib 1.2.1).

You can see it here (together with the result of %prun):
http://nbviewer.ipython.org/5868420/pandas-slow-plotting-011.ipynb
http://nbviewer.ipython.org/5868420/pandas-slow-plotting-012.ipynb

I looked a little bit into it, and I am not an expert at all but I thought to share some insights (for the case it would be useful):

• The factual reason it is so slow in 0.12 is due to the fact that it searches the config options 250025 times (for each point), because it searches for 'display.encoding' (this is the reason you see a lot of re in the %prun output). This is introduced by this commit: ae50103, changing the unicode repr of PeriodIndex. Just removing this commit does reduce it from ca 35 to ca 15 s.
• This is off course not the real reason, because the index of all points should not converted to a string at all! Why it is converted to a string I do not know. I tried to debug it, and I end up on line https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L131 with date being a PeriodIndex with 10000 elements. Why I end up there I don't get, because the mapping of the PeriodIndex (https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L115) should feed individual elements inside the get_datevalue instead of the whole Index.
• Plotting these timeserieses is very slow, also in the less slow 0.11. Almost all time is taken inside converter.py, and more specific in the mapping of all points in the index to get_datevalue (https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L121). Some observations:
  1. The function get_datevalue is called for every point in the dataframe, so 10000x25 in the example. So it is called for every column in the dataframe, while the index is the same for all columns. This seems redundant, and should ideally only be done once?.
  2. The function get_datevalue is performed on the individual elements of the PeriodIndex to get the ordinal value. This seems not really efficient to me, and I would think this can be vectorised. With values being a PeriodIndex I think values.asfreq(axis.freq).values is equivalent to (but much faster than) values.map(lambda x: get_datevalue(x, axis.freq))?

[jorisvandenbossche]

After another look, I might have figured it out. Short version: with this commit (jorisvandenbossche@d63e77c) the time goes down from 30s to only 210ms!

---

Long version:

• My second bullet of previous comment said the reason of the bad performance is that all index points are converted to a string. The reason is as follows:
  • At line https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L115, the PeriodIndex is mapped on the function get_datevalue. This function only handles individual elements and otherwise will raise an error (https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L131). map will first try to apply the function to the whole PeriodIndex, the error is captured, and then map will apply the function on the individual elements of the PeriodIndex. But because of the error is raised, the PeriodIndex is converted to a string in the error message (ValueError("Unrecognizable date '%s'" % date))
• I gave it a try to improve the situation, and with this commit (jorisvandenbossche@d63e77c) the conversion to a string is avoided, as well as just the .values is faster than .map() (see point 2 in previous comment):
  • And it seems a lot faster! 210 ms instead of 30s! (or ca 7-10s in 0.11)
    See http://nbviewer.ipython.org/5868420/pandas-slow-plotting-013dev-patched.ipynb (version with commit)
    http://nbviewer.ipython.org/5868420/pandas-slow-plotting-013dev.ipynb (version without commit)

If you think this is a sensible change, I put it in a PR (in every case, travis passes, but I don't know if this is much tested).

[jreback]

@jorisvandenbossche seems reasonable
pls put in a PR - u can do some tests (to at least ensure that their is a valid plot)

[jreback]

closed by #4722