Run benchmarks directly from sbt #2913
Conversation
| @@ -828,7 +828,8 @@ object Build { | |||
| dependsOn(`dotty-compiler`). | |||
| settings(commonNonBootstrappedSettings). | |||
| settings( | |||
| mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench") // custom main for jmh:run | |||
| mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench"), // custom main for jmh:run | |||
There was a problem hiding this comment.
Could you please print the entire command line that was used to launch the behcnmark vm?
There was a problem hiding this comment.
@liufengyun do you know if that is possible without the jmh plugin?
There was a problem hiding this comment.
When the command runs, we'll have the output in the console, which shows the VM options:
# JMH 1.18 (released 134 days ago)
# VM version: JDK 1.8.0_131, VM 25.131-b11
# VM invoker: /usr/lib/jvm/java-8-oracle/jre/bin/java
# VM options: -Xbootclasspath/a:some/jars
# Warmup: 10 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: dotty.tools.benchmarks.Worker.compile
| .timeUnit(TimeUnit.MILLISECONDS) | ||
| .forks(1) | ||
| .warmupIterations(12) | ||
| .measurementIterations(20) | ||
| .resultFormat(ResultFormatType.CSV) | ||
| .result("result.csv") |
There was a problem hiding this comment.
We can remove this file, and there'll be no result.csv, which is useless.
project/Build.scala
Outdated
| @@ -828,7 +828,8 @@ object Build { | |||
| dependsOn(`dotty-compiler`). | |||
| settings(commonNonBootstrappedSettings). | |||
| settings( | |||
| mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench") // custom main for jmh:run | |||
| mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench"), // custom main for jmh:run | |||
| javaOptions += "-DBENCH_CLASS_PATH=" + Attributed.data((fullClasspath in Compile).value).mkString("", ":", "") | |||
| ). | |||
| enablePlugins(JmhPlugin). | |||
| settings(packSettings). | |||
There was a problem hiding this comment.
As we don't use sbt-pack, it's fine to just remove the settings.
|
If it is not, this should be a strong reason to prefer a bash script.
…On 25 July 2017 4:43:58 pm Nicolas Stucki ***@***.***> wrote:
nicolasstucki commented on this pull request.
> @@ -828,7 +828,8 @@ object Build {
dependsOn(`dotty-compiler`).
settings(commonNonBootstrappedSettings).
settings(
- mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench") //
custom main for jmh:run
+ mainClass in (Jmh, run) := Some("dotty.tools.benchmarks.Bench"), //
custom main for jmh:run
@liufengyun do you know if that is possible with the jmh plugin?
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#2913 (comment)
|
|
We can add a script sbt "dotty-bench/jmh:run $@" |
|
@DarkDimius this is only intended as way quick to benchmark changes in the compiler itself. Hence sbt is the simplest option to make sure that everything is up to date with the latest changes in the compiler code. If we want reproducible benchmarks from command line we should use the benchmarks from https://github.com/liufengyun/bench instead. |
|
@liufengyun I added a couple of new commits. |
|
@nicolasstucki I thought I already merged this PR, well done :) |
To benchmark use
Setting up forks, warmups, ireations can be done in
bench/src/main/scala/Benchmarks.scala