API: Implement pipe protocol, a method for extensible method chaining

[TomAugspurger]

Today @shoyer and I were talking about a new "protocol" that will let us sidestep the whole macro / method chaining issue. The basic idea is that pandas objects define a pipe method (ideally other libraries will implement this to, assuming this is useful).

Based on the discussions below, we're leaning towards a method like

def pipe(self, func, *args, **kwargs):
    pipe_func = getattr(func, '__pipe_func__', func)
    return pipe_func(self, *args, **kwargs)

That's it. This lets you write code like:

import seaborn as sns

(sns.load_dataset('iris')
    .query("sepal_length > 5")
    .assign(sepal_ratio = lambda df: df.sepal_length / df.sepal_width)
    .pipe(lambda df: sns.violinplot(x='species', y='sepal_ratio', data=df))
)

seaborn didn't have to do anything! If the DataFrame is the first argument to the function, things are even simpler:

def f(x):
    return x / 2

df['sepal_length'].pipe(f)

Users or libraries can work around the need for the (somewhat ugly) lambda _:, by using the __pipe_func__ attribute of the function being piped in. This is where a protocol (de facto or official) would be useful, since libraries that know nothing else about each other can rely on it. As an example, consider seaborn's violin plot, which expects a DataFrame as its fourth argument, data. Seaborn can define a simple decorator to attach a __pipe_func__ attribute, allowing it to define how it expects to be piped to.

def pipeable(func):
    def pipe_func(data, *args, **kwargs):
        return func(*args, data=data, **kwargs)
    func.__pipe_func__ = pipe_func
    return func

# now we decorate all the Seaborn methods as pipeable

@pipeable
def violinplot(x=None, y=None, hue=None, data=None, ...):
    # ...   

And users write

(sns.load_dataset('iris')
    .query("sepal_length > 5")
    .assign(sepal_ratio = lambda x: x.sepal_length / x.sepal_width)
    .pipe(sns.violinplot, x='species', y='sepal_ratio')
)

Why?

Heavily nested function calls are bad. They're hard to read, and can easily introduce bugs. Consider:

# f, g, and h are functions that take and receive a DataFrame
result = f(g(h(df), arg1=1), arg2=2, arg3=3)

For pandas, the approach has been to add f, g, and h as methods to (say) DataFrame

(df.h()
   .g(arg1=1)
   .f(arg2=2, arg3=3)
)

The code is certainly cleaner. It reads and flows top to bottom instead of inside-out. The function arguments are next to the function calls. But there's a hidden cost. DataFrame has something like 200+ methods, which is crazy. It's less flexible for users since it's hard to get their own functions into pipelines (short of monkey-patching). With .pipe, we can

(df.pipe(h)
   .pipe(g, arg1=1)
   .pipe(f, arg2=2, arg3=3)
)

The other way around the nested calls is using temporary variables:

r1 = h(df)
r2 = g(r1, arg1=1)
r3 = f(r2, arg2=2, arg3=3)

Which is better, but not as good as the .pipe solution.

---

A relevant thread on python-ideas, started by @mrocklin: https://mail.python.org/pipermail/python-ideas/2015-March/032745.html

This doesn't achieve everything macros could. We still can't do things like df.plot(x=x_col, y=y_col) where x_col and y_col are captured by df's namespace. But it may be good enough.

Going to cc a bunch of people here, who've had interest in the past.

@shoyer
@mrocklin
@datnamer
@dalejung

[TomAugspurger]

I really like how simple the implementation is. No PEPs, no hacking on CPython, others should be able to implement the same protocol.

One (maybe) drawback is that we must rely on people to be responsible and not modify the self being passed in. But that's a very pythonic sort of "problem". Anyone, I look forward to the discussion :)

[jreback]

http://toolz.readthedocs.org/en/latest/api.html#toolz.functoolz.pipe

[mrocklin]

I like the idea of our community establishing and using protocols more heavily. Will ponder this one for a while.

[TomAugspurger]

@jreback I've used that in the past, and this would be very similar. toolz.pipe is less convenient for interactive use though, I think. You need to define each function ahead of time, including simple functions to get the data to pipe into the correct position for functions that don't expect data as the first argument. We'll still have to do that with df.pipe, but I think it's easier to do as you work through the chain.

[shoyer]

We might also extend pipe to allow functions to customize which argument is used for chaining. This would be useful for seaborn, which currently takes dataframes in the data keyword argument:

# in pandas
def pipe(self, func, *args, **kwargs):
    if hasattr(func, 'pipe_arg'):
        kwargs[func.pipe_arg] = self
        return func(*args, **kwargs)
    return func(self, *args, **kwargs)

# in seaborn (personally, I would probably implement this with a decorator)
sns.violinplot.pipe_arg = 'data'

# in user code
df.pipe(sns.violinplot, x='species', y='sepal_ratio')

Alternatively, we could just ask @mwaskom to break Seaborn's API again ;).

[mrocklin]

I agree that we can probably make something more pandas-y than toolz.pipe which, while convenient in its own way, doesn't jive particularly well with the pandas flow.

[datnamer]

Is it possible to get infix syntax using a decorator? https://github.com/JulienPalard/Pipe

[TomAugspurger]

That's done by overriding __or__ though, right? I wouldn't feel comfortable doing that in library code.

[shoyer]

@datnamer Yes, but not satisfactorily without macros. The problem is that to get df >> sns.violinplot('x', 'y') to work properly, you need to allow sns.violinplot('x', 'y'), even though that would usually raise an error because the required data argument is not provided.

So turning functions into pipe objects means that their non-piped usability is compromised. There are ways around this, e.g., using tuples like dask: df >> (sns.violinplot, 'x', 'y'), but >> (with the spaces) is only one less character than .pipe (which is much more explicit).

[mrocklin]

This isn't a viable candidate for a Pandas solution, but I thought I'd point out toolz.curry which lives in the same space.

[sinhrks]

Following is a list of pipes which R's magrittr package provides (the package is used by dplyr).

http://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html

• pipe: current example (and main usecase). %>% in magrittr.
• tee: apply the function to left value, and return the left value as it is. %T>% in magrittr.
• pipe and assignment: corresponding to panda's inplace option, %<>% in magrittr.

If any of them is useful, preparing an separate kw (or method) might be an option.

[TomAugspurger]

tee should be quite easy as a separate method. Just apply func and return self.

df.tee(sns.violinplot, x='x', y='y')
    .pipe(sm.OLS.from_formula('y ~ x'))

I do consider this experimental though. My initial thought was to start slow and see how things go so I didn't include that in the initial proposal.

[tacaswell]

👍 I like this in general as an alternative to the current df.plot. It also fits pretty well with where I (speaking only for me) want to push the mpl API.

tee worries me a bit as you lose the ability to get the Artists back which is important if you want to do anything interactive.

Instead of pushing the decorator out to other libraries, I think it would be better to encourage end user to decorate on the fly, ex

pd_violin = pd.pipeify(sns.violin)
artists = df.pipe(pd_violin, x='foo', y='bar')

or to give pipe a reserved kwarg like _pipe_target so

def pipe(self, func, _pipe_target=None, *args, **kwargs):
    if pipe_target is not None:
        kwargs[_pipe_target] = self
        return func(*args, **kwargs)
    else:
        return func(self, *args, **kwargs)

[TomAugspurger]

Another drawback is that libraries wishing to support pandas <0.17 will need to implement there own pipeable decorator. I've got one in pandas/util/decorators.py

def pipeable(pipe_arg):
    def decorate(func):
        setattr(func, 'pipe_arg', pipe_arg)

        @wraps(func)
        def wrapper(*args, **kwargs):
            return func(*args, **kwargs)
        return wrapper
    return decorate

It's not difficult, but it's essential that everyone agrees on the name for pipe_arg. This is a point in favor of @tacaswell's suggestion, but I'm pretty attached to simply being able to write df.pipe(sns.violinplot, 'x', 'y') :)

[TomAugspurger]

@josef-pkt would statsmodels be interested in adding the decorators to support this? I see it working quite nicely with the formula API. I'll send over a PR if you want. I believe you're hoping to have a release in the not too distant future? It'd be nice to lay the groundwork in statsmodels 0.7. Even if that comes out before the next version of pandas, adding the decorators should entirely transparent to pre 0.17 pandas and just work for pandas >= 0.17.