Protocols consolidation for releases

[josevalim]

Protocols main goal is to allow code extensibility. Athough we compile Elixir's built-in clauses to fast protocol dispatches, dispatching a protocol for a record goes through a slower process:

1. The record name is converted to a list and it is verified it starts with Elixir-
2. The generate the target implementation module by concatenating the Protocol name with the Record name
3. We attempt to dispatch to the implementation module
4. If the dispatch fails, we go to the fallback branch, which may be to invoke the Tuple implementation (after all, all records are Tuple), the Any implementation or raise an exception

This lookup is essential for the extension mechanism. However, in some cases, it may inflict a too big performance overhead. Based on this concern, we have received a couple proposals to improve this situation. One of those are outlined in #802, which allows us to inline implementations into the protocol. The problem with the proposal above is that it gives preference to the types known upfront, sacrificying the extensibility mechanism protocols are meant to bring in the first place!

This proposal intents to solve this problem without introducing side effects.

The release process

Erlang applications and projects are familar with the term "releases". When you bundle a release, your project is compiled and ready to be used in production. You can read more about releases on OTP's website.

In other words, when building a release, the software knows all protocols available and all of their implementations, meaning we can generate a protocol with a fast dispatch logic to all available implementations. This ensures protocol will be fast for both positives (a target exists) and negative cases (the target does not exist).

In terms of a code, the Protocol API would look like:

Protocol.consolidate(MyProtocol)

This code will look at all load paths, gather all implementations available and recompile the protocol using the fast dispatch clauses. If known up-front, the list of implementations could be given as argument:

Protocol.consolidate(MyProtocol, [Binary, List, SomeRecord])

This approach has two caveats:

1. If a protocol is consolidated, code loading a new implementation for a new record to production servers won't work if a new protocol consolidation is not available
2. Some people may want to define records and implementations at runtime, although very unlikely, they will probably want a way to disable this feature

Although Elixir doesn't ship with its own release tool yet, the available solutions would be rewriten to rely use this mechanism. By the time Elixir has its own release tool (planned for 0.9), we can integrate this behaviour as part of the release process.

[ericmj]

I really like this idea. I'm not really sure how this would work, but could caveat 1 be solved by an appup tool?

[josevalim]

@ericmj Both caveats could be solved somehow, that's why I called them caveats and not something like drawback. :) And for the first case, you are right, appups would catch it. :)

[meh]

Can we have a helper to check if a record implements a protocol?

[meh]

Can we have additional non-protocol definitions in the defprotocol?

It could be helpful to have helpers or other things which are supposed to work with protocol implementing records but aren't supposed to end up in protocol implementations.

[josevalim]

We can't because that would make consolidation impossible. We can only consolidate because Elixir has control of all the code inside the protocol, so we can recompile it as many times we want.

[meh]

Oh 🐼

Oh well.

[josevalim]

Note: this is also an opportunity to clean up @only and @except attributes.