code coverage support

[johnynek]

This would look like using:

https://github.com/scoverage/scalac-scoverage-plugin

to get the scala test rules to emit code coverage reports, then aggregating them across the repo.

It is related to bazelbuild/bazel#1118 but I don't think it is actually a blocker. We may just be able to produce coverage output of each test rule, then an aspect to walk all the rules and aggregate a total.

[softprops]

Has anyone started working on this? This is going to become a priority from my team soon. We're working on replacing our reliance on sbt in our jenkins pipeline with bazel. One part of that is figuring out the test report and codecov ( scoverage ) stories.

[ittaiz]

I don't know (and think) anyone started working on it but I'll add that there's been a lot of discussion on various coverage issues on bazel itself so I'd recommend anyone tackling this to read a bit beforehand

…

On Fri, 5 May 2017 at 20:40 doug tangren ***@***.***> wrote: Has anyone started working on this? This is going to become a priority from my team soon. We're working on replacing our reliance on sbt in our jenkins pipeline with bazel. One part of that is figuring out the test report and codecov ( scoverage ) stories. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF4kLM62g_5DrI10vlWYAwmLVb7Dfks5r217wgaJpZM4NG3kM> .

[softprops]

👍

I was looking through the machinery that ties scoverage into sbt, and it looks like it may be possible to to just opt into code coverage for a regular scala_test by threading through a few additional arguments flags: namely: -Xplugin:path/to/plugin and -P:scoverage:dataDir:path/to/scoverage-data

[softprops]

opt in meaning maybe this is possible today without any additional changes and when the coverage story in bazel settles something like this could then just become a conveniences of being instrumented by default with running bazel cover

[johnynek]

The scala side should be easy. What I don't know is how to plug in to bazel's expectations (if they are even standard). Like where should we write the coverage files to? What should the format be?

…

On Fri, May 5, 2017 at 08:41 doug tangren ***@***.***> wrote: opt in meaning maybe this is possible today without any additional changes and when the coverage story in bazel settles something like this could then just become a conveniences of being instrumented by default with running bazel cover — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdgmLCVqU5WVE95c7FE6dhkMXP38Tks5r221OgaJpZM4NG3kM> .

[softprops]

My guess is it'll be the same as the test_filter flag story. When x is present in the env ( or args ) do y. My guess about where the conventional place to write reports would be customizable with rule dependent defaults, like junit test reports

[ittaiz]

I really suggest searching the Bazel github issues with the term coverage. There have been several discussions about it in the issues.

…

On Fri, 5 May 2017 at 21:56 doug tangren ***@***.***> wrote: My guess is it'll be the same as the test_filter flag story. When x is present in the env ( or args ) do y. My guess about where the conventional place to write reports would be customizable with rule dependent defaults, like junit test reports — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFzVFzcvA0v4oPL3Z6UF9Srf6YWNZks5r23DSgaJpZM4NG3kM> .

[softprops]

@johnynek do you know ( specifically for scalac ) if there needs to be anything special done for scalac plugins to set up a special kind of classpath separate form the classpath of the files being compiled?

I've now optimistically convinced myself this should be more straightforward that I originally thought so I couldn't help trying out a POC to close out my friday on a happy note.

my WORKSPACE

# other stuff ...

filegroup(
    name = "scoverage_plugin",
    srcs = ["sbt-plugins/scalac-scoverage-plugin_2.11-1.2.0.jar"],
    visibility = ["//visibility:public"],
)

java_import(
    name = "sbt-plugins",
    jars=glob([
    "sbt-plugins/scalac-scoverage*.jar"],
    exclude = [
            "sbt-plugins/scala-library*jar",
            "sbt-plugins/scala-xml*jar"
    ]),
    visibility = ["//visibility:public"],
)

my BUILD

scala_test(
    name = "foo",
    size = "small", 
    srcs = glob(
        [
            "src/test/scala/**/*.scala",
            "src/test/java/**/*.java",
        ],
    ),
    jvm_flags = [
        "-Dfile.encoding=UTF-8",
    ],
    plugins = ["@meetuplib//:scoverage_plugin"], # the scoverage scalac plugin jar
    scalacopts = [
      "-P:scoverage:dataDir:/path/to/target/scovtest/scoverage-data",
      "-Yrangepos"
    ],
   deps = [
      # other stuff....
        "@meetuplib//:sbt-plugins" # puts scoverage deps on my classpath
    ],
)

when I run bazel test -s --progress_report_interval 1 --test_output streamed --strategy=Scalac=standalone //foo:* I get a class not found error on scala.xml.NamespaceBinding. Scala xml is a dependency of scoverage but I've already confirmed it's on my classpath. I checked my foo-test_worker_input file to be sure.

[info] Instrumentation completed [50790 statements]
error: java.lang.NoClassDefFoundError: scala/xml/NamespaceBinding
	at scoverage.ScoverageInstrumentationComponent$$anon$1.run(plugin.scala:121)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1501)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1486)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1481)
	at scala.tools.nsc.Global$Run.compile(Global.scala:1582)
	at scala.tools.nsc.Driver.doCompile(Driver.scala:32)
	at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
	at scala.tools.nsc.Driver.process(Driver.scala:51)
	at io.bazel.rulesscala.scalac.ScalacProcessor.compileScalaSources(ScalacProcessor.java:207)
	at io.bazel.rulesscala.scalac.ScalacProcessor.processRequest(ScalacProcessor.java:77)
	at io.bazel.rulesscala.worker.GenericWorker.run(GenericWorker.java:125)
	at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:42)
Caused by: java.lang.ClassNotFoundException: scala.xml.NamespaceBinding
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 12 more

cat bazel-out/local-fastbuild/bin/foo/foo-test_worker_input
Classpath: # includes external/scala/lib/scala-xml_2.11-1.0.4.jar among otherthings
EnableIjar: False
Files: # bunch of sources
IjarCmdPath:
IjarOutput:
JarOutput: bazel-out/local-fastbuild/bin/foo/foo-test.jar
JavacOpts:
JavacPath: external/local_jdk/bin/javac
JavaFiles:  # bunch of sources
JvmFlags: -J-Dfile.encoding=UTF-8
Manifest: bazel-out/local-fastbuild/bin/foo/foo-test_MANIFEST.MF
Plugins: external/meetuplib/sbt-plugins/scalac-scoverage-plugin_2.11-1.2.0.jar
PrintCompileTime: False
ResourceDests: /log4j.properties,/malformed-manifest.json,/test-velocity.properties,/valid-manifest.json
ResourceJars:
ResourceSrcs: # bunch of sources
ResourceStripPrefix:
ScalacOpts: -P:scoverage:dataDir:/path/to/target/scovtest/scoverage-data,-Yrangepos
SourceJars:

The only thing that turned up in a lazy google search was this issue that popped on up the intellij mailing list. I didn't quite get the solution in that case or how it may be part of the problem in mine.

[sdtwigg]

I was actually speaking with ahirreddy@ about this the other day. I think the plugin pulls classes from the bootclasspath. Was eventually planning to verify this and fixup the rules to also setup the bootclasspath properly. Edit: You just need to add the dependencies as other (fake) -Xplugin. This makes a more structured deploy_jar construction tempting.

It might be easier just to build the plugin as a deploy_jar for now though.

[softprops]

@sdtwigg I'm more of a consumer than a producer in this case. I'm using the scoverage plugin and runtime jars from maven central and trying to provide them as a plugin value to a scala_test rule. Can you draw me a rough sketch of what you mean?

[softprops]

Now I'm thinking there may be something to do with the bootclasspath which may actually be related to an open bug with scala.

One thing I failed to mention above when setting up my test was that initially I didn't have the scalac-scoverage-runtime jar on my classpath which caused a similar issue. After adding that to my scala_tests' deps, that problem was resolved. The next issue I ran into was the xml.NameSpace error.

I tried the change mentioned in the issue by adding the -nobootcp to my scalacopts. Same issue.

I'm a little confused because the identity of this class technically didn't change between scala 2.10 and 2.11 so either way I expected it to be on the classpath one way or another.

[softprops]

I have a little bit more mental energy to this but I'm still stumped on the scala.xml.NameSpace issue. Though I was reminded that in order for scoverage to instrument code it needs to be applied to my scala_library targets ( not my scala_test ) targets which makes things a bit more awkward

[johnynek]

we should be able to use a target in the current repo as a plugin. If we can't now, that shouldn't be too much work. Then, we could just make a deploy jar ourselves by linking in the full path.

[sdtwigg]

Sorry, I never really described what deploy_jar is although seems like you found out but just in case: The deploy_jar is a fat-jar that contains the current compiled target and all its transitive dependencies (both runtime and compile time). It is analogous to the deploy_jar you can get from a java_binary (although not executable if from scala_library).

I realized over the weekend that just directly using a deploy_jar from one of the rules_scala rules probably won't work because I don't think it will put the plugin.xml in the right place. So, would need to take the deploy_jar from the rules and have another action/genrule put the plugin.xml inside that fat jar.

I evntually wanted to demonstrate this by replacing the linter plugin in the tests with a custom rolled one (both to demonstrate how to make plugins and drop a rules_scala dependency). Unfortunately, I am really reluctant to say I have the time to do this :( . Had spent my 'free-work' time over weekend pushing through the bazel changes outlined in: #57 (comment)

PS: Do you think it is a problem that using two plugins might have redundant classes in their fat-jars? (Assuming that the classes are identical duplicates then it seems like it would just be like a 'stylistic' issue versus a functional one.)