`scala.testing.Benchmark` is deprecated and should no longer be mentioned

[SethTisue]

it appears in

es/overviews/parallel-collections/performance.md
ja/overviews/parallel-collections/performance.md
overviews/parallel-collections/performance.md
zh-cn/overviews/parallel-collections/performance.md

[DarkDimius]

+ ru/overviews/parallel-collections/performance.md

[jiminhsieh]

@SethTisue Is it just simple remove anything related to scala.testing.Benchmark? It would remove lots of document. Or rewrite that part of document with other way of benchmark. Thanks!

[SethTisue]

I think that whole "Microbenchmarking example" section could be cut way back, with the specifics in the second half simply removed. In its place, you could link to https://github.com/scala/scala/blob/2.12.x/test/benchmarks/README.md as an example of proper benchmarking code.

[jiminhsieh]

I would try to send the PR, let you see does it works as you expect. By the way, in my opinion, the second half of How big should a collection be to parallel? should be remove or rewrite too.

[SethTisue]

yeah, I agree that could be condensed substantially.

[jiminhsieh]

Hi @SethTisue I have some ideas about How big should a collection be to go parallel?. I change the code snippet with JMH instead of scala.testing.Benchmark and JDK7 to JDK8. The results of the code snippet would be the results of JMH. Like this:

[info] Benchmark                    Mode  Cnt        Score       Error  Units
[info] TestBenchmark.iterator      thrpt   30      418.637 ±     7.991  ops/s
[info] TestBenchmark.list          thrpt   30      348.523 ±     9.162  ops/s
[info] TestBenchmark.range         thrpt   30      367.331 ±     8.731  ops/s
[info] TestBenchmark.readFromFile  thrpt   30  3466096.653 ± 40711.291  ops/s
[info] TestBenchmark.stream        thrpt   30      112.702 ±     1.214  ops/s

BTW, I have similar CPU which is E3 quad-core processor.

[SethTisue]

yes, that sounds like a good change!

[jiminhsieh]

@SethTisue I did some benchmarks locally. There is the source code that I did the benchmark.

Environment:

• CPU: E3-1231 v3 without hyper-threading and turbo boost
• RAM: 16G, I allocated 8G for each benchmark.
• OS: Debian 8 with limitation(2.5GHz) of CPU frequency.
• JVM: Oracle 1.8.0_144-b01
• Scala: 2.12.3

Result:

[info] MapBench.map           120000       N/A  thrpt   45    91.299 ±   1.214  ops/s
[info] MapBench.map            15000       N/A  thrpt   45   702.102 ±   1.874  ops/s
[info] ParMapBench.map        120000         1  thrpt   45    98.208 ±   0.464  ops/s
[info] ParMapBench.map        120000         2  thrpt   45   188.050 ±   1.217  ops/s
[info] ParMapBench.map        120000         4  thrpt   45   287.804 ±   2.439  ops/s
[info] ParMapBench.map         15000         1  thrpt   45   872.125 ±   2.228  ops/s
[info] ParMapBench.map         15000         2  thrpt   45  1578.931 ±   8.447  ops/s
[info] ParMapBench.map         15000         4  thrpt   45  2556.409 ±  14.718  ops/s
[info] ParVectorBench.reduce  250000         1  thrpt   45   582.542 ±  12.397  ops/s
[info] ParVectorBench.reduce  250000         2  thrpt   45   655.938 ±  12.524  ops/s
[info] ParVectorBench.reduce  250000         4  thrpt   45   927.864 ±   7.750  ops/s
[info] ParVectorBench.reduce  120000         1  thrpt   45   944.875 ± 137.443  ops/s
[info] ParVectorBench.reduce  120000         2  thrpt   45  1481.952 ±  34.182  ops/s
[info] ParVectorBench.reduce  120000         4  thrpt   45  2468.674 ±  42.307  ops/s
[info] VectorBench.reduce     250000       N/A  thrpt   45   495.297 ±   1.650  ops/s
[info] VectorBench.reduce     120000       N/A  thrpt   45  1116.625 ±   2.346  ops/s

Do you have any suggestion with this benchmark? Mybe I could change to different size or add bigger size of collection for the benchamrk? Thanks! :)

[SethTisue]

@jiminhsieh this isn't an area I've worked in myself, so I can't give good detailed feedback. perhaps @Ichoran or @axel22 have thoughts on it. doing a deep dive on parallel collections performance is outside the scope of this ticket, anyway — I don't mean to discourage you from going deeper on it if it interests you, but in the meantime it would be nice to just get a basic "get rid of scala.testing.Benchmark" fix in

[Ichoran]

In most cases, parallel collections performance is substantially behind what you can get with Java 8 Streams, or even with manual approaches with Future, so I am not sure it's worth spending much time on it.

…

On Tue, Sep 12, 2017 at 1:43 PM, Seth Tisue ***@***.***> wrote: @jiminhsieh <https://github.com/jiminhsieh> this isn't an area I've worked in myself, so I can't give good detailed feedback. perhaps @Ichoran <https://github.com/ichoran> or @axel22 <https://github.com/axel22> have thoughts on it. doing a deep dive on parallel collections performance is outside the scope of this ticket, anyway — I don't mean to discourage you from going deeper on it if it interests you, but in the meantime it would be nice to just get a basic "get rid of scala.testing.Benchmark" fix in — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#752 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACMTFHHzo3ZKRsrbV4wTTdUs07dkUth6ks5shuz2gaJpZM4MtrJv> .

[axel22]

@jiminhsieh Parallel collections are not specialized for primitive types, so you should not expect to get too much performance from Vector[Int] or ParVector[Int] - most of what you're measuring in map operations is boxing. To compare, I suggest testing collections that contain objects (e.g. Vector[String]) and do some operation on those objects.

[jiminhsieh]

@Ichoran @axel22 Thanks for all of your indications! I would try to work on what you mentioned in my own experiments.
@SethTisue Since it will need deeper knowledge to get informative results of benchmark, I would just remove doc related to scala.testing.Benchmark.