Add blog article about the new collections performance #832
Conversation
| based operation implementations are about 25% slower than builder based implementations | ||
| and I’ve explained how we restored builder based implementations on strict collections. | ||
|
|
||
| I expect the new collections to be as fast or slightly faster than the previous collections. |
There was a problem hiding this comment.
s/fast or slightly faster/equally fast or slightly faster/s
?
| [merged](https://github.com/scala/collection-strawman/pull/342) | ||
| a completely new implementation of immutable `Set` and `Map` based on [compressed | ||
| hash-array mapped prefix-trees](https://michael.steindorfer.name/publications/oopsla15.pdf). | ||
| This data structure has a smaller memory footprint than the old `HashSet` and `HashMap`, |
There was a problem hiding this comment.
Can you include concrete numbers here? Sounds amazing!
| them to be slower since the execution path, when calling an operation, can be made | ||
| exactly the same as in the old collections. | ||
|
|
||
| ## Conclusion |
There was a problem hiding this comment.
I would include "correct first" somewhere in the conclusion, it's an interesting point worth repeating.
|
|
||
| In a [previous blog post](/blog/2017/11/28/view-based-collections.html), I explained | ||
| how [Scala 2.13’s new collections](http://www.scala-lang.org/blog/2017/02/28/collections-rework.html) | ||
| have been designed so that the default implementations of transformation operations work |
| how [Scala 2.13’s new collections](http://www.scala-lang.org/blog/2017/02/28/collections-rework.html) | ||
| have been designed so that the default implementations of transformation operations work | ||
| with both strict and non-strict types of collections. In essence, we abstract over | ||
| the evaluation mode (strict or non strict) of concrete collection types. |
| builder based versions. How much slower exactly varies with the type of collection | ||
| (e.g. `List`, `Vector`, `Set`), the operation (e.g. `map`, `flatMap`, `filter`) | ||
| and the number of elements in the collection. In my benchmark on `Vector`, on | ||
| the `map`, `filter` and `flatMap` operations, with 1 element to 7 million of |
There was a problem hiding this comment.
with 1 to 7 million of elements
| For reference, the source code of the new collections is available in | ||
| [this GitHub repository](https://github.com/scala/collection-strawman). | ||
|
|
||
| ## Overhead Of View Based Implementations |
| Here we use `StrictOptimizedSeqOps`, which is a specialization of `StrictOptimizedIterableOps` | ||
| for `Seq` collections. | ||
|
|
||
| ## Is The View Based Design Worth It? |
|
|
||
| ## Overhead Of View Based Implementations | ||
|
|
||
| Let’s be clear: the view based implementations are in general slower than their |
| ## How To Fix That Performance Regression? | ||
|
|
||
| Our solution is simply to go back to builder based implementations for strict collections: we | ||
| override the default view based implementations with more efficient builder based |
| These charts show the execution time (vertically) of the `filter`, `map` and `flatMap` | ||
| operations, according to the number of elements (horizontally). Note that scales are | ||
| logarithmic in both axis. The blue line shows the performance of the old `Vector`, | ||
| the green line shows the performance of the new `Vector` if it used only view based |
| Again, the answer depends on the type of collection, the operations and the number of elements. | ||
| My `Vector` benchmarks show a 20% speedup on average: | ||
|
|
||
|  |
There was a problem hiding this comment.
unit for the y-axis? s or ms?
the title should be: Vector.filter (log-scaled)
| ## Is The View Based Design Worth It? | ||
|
|
||
| In my previous article I explained that a drawback of the old builder based design was that, | ||
| on non strict collections (e.g. `Stream` or `View`), we had to carefully override all the |
|
|
||
| In my previous article I explained that a drawback of the old builder based design was that, | ||
| on non strict collections (e.g. `Stream` or `View`), we had to carefully override all the | ||
| default implementations of transformation operations to make them non strict. |
| default implementations of transformation operations to make them non strict. | ||
|
|
||
| Now it seems that the situation is just reversed: the default implementations work well | ||
| with non strict collections, but we have to override them in strict collections. |
| So, is the new design worth it? To answer this question I will quote a comment posted | ||
| by Stefan Zeiger [here](https://www.reddit.com/r/scala/comments/7g52cy/let_them_be_lazy/dqixt8d/): | ||
|
|
||
| > The lazy-by-default approach is mostly beneficial when you're implementing lazy |
|
|
||
| These charts show the execution time (vertically) of the `filter`, `map` and `flatMap` | ||
| operations, according to the number of elements (horizontally). Note that scales are | ||
| logarithmic in both axis. The blue line shows the performance of the old `Vector`, |
|
|
||
| Since operation implementations end up being the same, why do we get better performance | ||
| at all? Well, these numbers are specific to `Vector`, and are due to the fact that | ||
| we more agressively inlined a few critical methods. I don’t expect the new collections |
| we more agressively inlined a few critical methods. I don’t expect the new collections | ||
| to be *always* 20% faster than the old collections. However, there is no reason for | ||
| them to be slower since the execution path, when calling an operation, can be made | ||
| exactly the same as in the old collections. |
There was a problem hiding this comment.
exactly the same => the same
| title: On Performance of the New Collections | ||
| --- | ||
|
|
||
| In a [previous blog post](/blog/2017/11/28/view-based-collections.html), I explained |
There was a problem hiding this comment.
I would group all those paragraphs. until Overhead Of View Based Implementations
|
|
||
| ## Overhead Of View Based Implementations | ||
|
|
||
| Let’s be clear: the view based implementations are in general slower than their |
| Let’s be clear: the view based implementations are in general slower than their | ||
| builder based versions. How much slower exactly varies with the type of collection | ||
| (e.g. `List`, `Vector`, `Set`), the operation (e.g. `map`, `flatMap`, `filter`) | ||
| and the number of elements in the collection. In my benchmark on `Vector`, on |
|
|
||
| ## How To Fix That Performance Regression? | ||
|
|
||
| Our solution is simply to go back to builder based implementations for strict collections: we |
| ## How To Fix That Performance Regression? | ||
|
|
||
| Our solution is simply to go back to builder based implementations for strict collections: we | ||
| override the default view based implementations with more efficient builder based |
|
|
||
| Our solution is simply to go back to builder based implementations for strict collections: we | ||
| override the default view based implementations with more efficient builder based | ||
| ones. We actually and up with the same implementations as in the old collections. |
There was a problem hiding this comment.
We actually and up with ?
|
|
||
| ## Is The View Based Design Worth It? | ||
|
|
||
| In my previous article I explained that a drawback of the old builder based design was that, |
There was a problem hiding this comment.
In my previous article,
|
|
||
| ## Is The View Based Design Worth It? | ||
|
|
||
| In my previous article I explained that a drawback of the old builder based design was that, |
There was a problem hiding this comment.
This sentence could be split into two.
In my previous article, I explained that a drawback of the old builder based design. On non-strict collections (e.g. Stream or View), we had to carefully override all the default implementations of transformation operations to make them non-strict.
| > implementation for a strict collection type you only suffer a small performance | ||
| > impact but it's still correct. | ||
|
|
||
| In short: implementations are **correct first** in the new design but you might want to |
|
Just a heads up that I defined the release date for this article to today. Let me know if you need more time to review or if I should merge it. |
| the green line shows the performance of the new `Vector` if it used only view based | ||
| implementations, and the red line shows the actual performance of the new `Vector` | ||
| (with strict optimized implementations). Benchmark source code and numbers can be found | ||
| [here](https://gist.github.com/julienrf/f1cb2b062cd9783a35e2f35778959c76). |
There was a problem hiding this comment.
In the logarithmic scale it's not obvious to see the speedup factor. could you say something about it?
There was a problem hiding this comment.
I mention that the scale is logarithmic and I’ve given the speedup factors in the text (view-based implementations are 25% slower than builder-based ones, and the new collections are 35% faster than the old collections).
What should I add?
There was a problem hiding this comment.
Ah I missed the mention of 35 above the charts. I guess my concern is that the graphs the thing that really jumps at people's eyes. The current for with the logarithmic scale doesn't really visualize the nice improvements. Maybe you could do some bars on a linear scale, normalize the old collection to 1, and add time values to the bars? Something like https://docs.google.com/spreadsheets/d/1Jcw3iG5sx_Xo-3svjb_qmB4Fprz5sEX7nCFpCd4EcWQ/edit?usp=sharing
There was a problem hiding this comment.
Yeah I was considering of doing something exactly like that! OK, let’s do it then.
| > impact but it's still correct. | ||
|
|
||
| In short, implementations are **correct first** in the new design but you might want to | ||
| override them for performance reasons on strict collections. |
There was a problem hiding this comment.
This sentence seems like the crux of the whole post. I suggest saying something this right at the top. Many people will read partway in and then bail, so you should hit the takeaways early, then proceed with more detailed explanations.
|
|
||
| ## Performance Comparison With 2.12’s Collections | ||
|
|
||
| Talking about performance, how performant are the new collections compared to the old ones? |
There was a problem hiding this comment.
Again, I suggest including some brief answer to this very near the top of the whole blog post.
|
Alright, thanks all for your reviews, I’ve improved the charts and mentioned the takeaways in the first sections. I can squash the commits but I think the “squash and merge” button should just work. |
julienrf
force-pushed
the
new-collections-performance
branch
from
964faed to
6036b26
Compare
|
I updated the publication date to today. |
|
Looks good! |
No description provided.