fix benchmark infrastructure

[liufengyun]

@DarkDimius when you have time, could you please give more detailed description of this task? Thanks a lot.

Todos:

• fix resultDir error (workaround in Workaround for ScalaMeter incorrect report path #1218 and ScalaMeter PR )
• ? simplify tests to just 3 groups : Scala collections, dotty and test set -- problem of data migration
• don't use ExponentialBackoff, we can afford to keep result data for each day
• daily backup regression data
  • added as post step: tar -zcvf "backup/report-$(date '+%F').tar.gz" benchtmp/
• implement new UI for benchmarks
• reconstruct compileStdLib.dat from compileStdLib.dsv
• disable Ycheck:all

Two issues above involves migrate old regression data to new format, which may be quite involved.

[DarkDimius]

The main command that is used to run benchmarks:

"dotty-bench/test:run -CscopeFilter '(pos|dotc|run|java|compileStdLib).*' -CresultDir benchtmp"

[DarkDimius]

#156 PR has link to the website generated by scalameter with published benchmarks. In order to show graphs(like here: https://goo.gl/HNJ0dt) in pretty way, you would need to drag the "date" axis to become "x-axis".

http://ci.d-d.me has jenkins web-interface. It runs benchmarks every night, but only if master branch has changed.

[liufengyun]

As an update, the benchmark infrastructure breaks because we have one corrupted serialised file in the report folder(tests.compileStdLib.compileStdLib.dat is 0 byte).

The simple solution is to remove the file. As we've no previous backup of the data, it means we lost the regression data for scala collection compilation. Now the benchmark is back to normal.

However, I do notice another problem:

• A bug in ScalaMeter caused command line option -CresultDir is ignored (workaround in Workaround for ScalaMeter incorrect report path #1218 ).

[DarkDimius]

• simplify tests to just 3 groups : Scala collections, dotty and test set -- problem of data migration
• use a fixed version of dotty and test set for benchmarks -- the objects should not change.

I'm not sure I understand what you mean. Could you please elaborate?

[liufengyun]

simplify tests to just 3 groups

Currently we have more than 20 groups, it's a little difficult to get insight with so many groups. I mean we can merge related groups into just 3 groups: scala collection, dotty and test set.

use a fixed version of dotty and test set for benchmarks

Current benchmark measures compilation time for dotty itself and also the test set. However, dotty as well as the test set are changing all the time in development, an increase in compilation time doesn't necessarily imply degradation in performance. To make benchmarks comparable and more meaningful, I think it's better to compile a fixed version of dotty.

[DarkDimius]

As we've no previous backup of the data, it means we lost the regression data for scala collection compilation. Now the benchmark is back to normal.

https://d-d.me/tnc/dotty/tests.compileStdLib.compileStdLib.dsv - this is the backup in a different format.

[liufengyun]

Thanks @DarkDimius , the dsv has part of the information of dat, but it's not a serialisation(it's generated by a reporter, instead of a persistor), thus can't be used to restore the dat file.

[DarkDimius]

Well, for graphs we actually don't need the dat file, as only dsvs are used.
I'm not aware if it would be easy to to make ScalaMeter append to existing dsv without a matching dat file.

[liufengyun]

That would require deep hacking of ScalaMeter, make the infrastructure difficult to maintain.

However, the dsv contains the most essential data of dat. Maybe I can try to use that information to reconstruct dat.

[liufengyun]

I've just reconstructed the dat file from dsv file using a short program.

[liufengyun]

It seems the ScalaMeter web app hasn't been maintained for 3 years. It's difficult to use or customize. I kind of want to develop our own HTMLReporter, and possibly contribute back to ScalaMeter.

I estimate it would take about 30 to 40 hours to get it working. I'm not sure if it's worth the effort.

[liufengyun]

I've spent one day rewriting the web app for benchmarks:

https://d-d.me/tnc/dotty/web/

It's a pure static web app, without any change in ScalaMeter or infrastructure, except a symbolic link to ScalaMeter HTML report files.

One small but useful feature is that you can select the charts you are interested and they are remembered in your browser.

BTW, the reconstruction of dat from dsv caused runtime type casting error between long and int. I'll investigate and propose a robust reconstruction.

[odersky]

This looks incredibly helpful! Now, do I read that right that between
december and march compile-time performance halved for most benchmarks? If
yes, is there a way to get more datapoints to identify where this comes
from?

On Sun, Apr 17, 2016 at 2:28 AM, liu fengyun notifications@github.com
wrote:

I've spent one day rewriting the web app for benchmarks:

https://d-d.me/tnc/dotty/web/

It's a pure static web app, without any change in ScalaMeter or
infrastructure, except a symbolic link to ScalaMeter HTML report files.

One small but useful feature is that you can select the charts you are
interested and they are remembered in your browser.

BTW, the reconstruction of dat from dsv caused runtime type casting error
between long and int. I'll investigate and propose a robust
reconstruction.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1210 (comment)

Martin Odersky
EPFL and Lightbend