Add 'Sciss/Lucre' to Community Build

[Sciss]

No description provided.

[Sciss]

Hmmm. I could run testOnly for scissLucre, but apparently missed scalatest for some tests of dependencies. When I add dependencies = List(scalatest), I get build error from scalactic:

Generated /home/hhrutz/Documents/devel/dotty/community-build/community-projects/scalatest/dotty/scalactic/target/scala-3.0.0-M3/src_managed/main/org/scalactic/ColCompatHelper.scala
Generated /home/hhrutz/Documents/devel/dotty/community-build/community-projects/scalatest/dotty/scalactic/target/scala-3.0.0-M3/src_managed/main/org/scalactic/ArrayHelper.scala
[error] -- [E007] Type Mismatch Error: /home/hhrutz/Documents/devel/dotty/community-build/community-projects/scalatest/dotty/scalactic/src/main/scala/org/scalactic/anyvals/NumericString.scala:1541:4 
[error] 1541 |    value.lines
[error]      |    ^^^^^^^^^^^
[error]      |    Found:    java.util.stream.Stream[String]
[error]      |    Required: Iterator[String]
[error] one error found
[error] (scalacticDotty / Compile / compileIncremental) Compilation failed

Don't know what to do.

[smarter]

[error] 1541 |    value.lines
[error]      |    ^^^^^^^^^^^
[error]      |    Found:    java.util.stream.Stream[String]
[error]      |    Required: Iterator[String]

This happens when running on a recent JDK which added String#lines that returns a Stream, it needs to be fixed upstream in scalatest but our community build runs on Java 8 and if you run it locally on Java 8 it should work.

[Sciss]

[error] 1541 |    value.lines
[error]      |    ^^^^^^^^^^^
[error]      |    Found:    java.util.stream.Stream[String]
[error]      |    Required: Iterator[String]

This happens when running on a recent JDK which added String#lines that returns a Stream, it needs to be fixed upstream in scalatest but our community build runs on Java 8 and if you run it locally on Java 8 it should work.

Ah! Yes, I had switched back to JDK 11 for other project, whereas before I ran successfully on JDK 8. So hopefully this test run succeeds now.

[smarter]

Ah, it's fixed upstream already: scalatest/scalatest@543f9ac

[smarter]

Dotty CI / community_build_b (pull_request) Successful in 87m

This is a > 20 minutes increased runtime for this job, I assume most of that comes from tests, could you try running less tests (for example, no tests for the intermediate projects and/or less tests for lucre itself) to reduce the running time?

[Sciss]

I don't know if this shaves off much, the dependencies are very small, the only one that runs substantial tests is Scala STM, which I think would anyway be a project that should be tested in the CB.

When you look at the raw log, beginning at 2020-12-08T01:43:47.6464622Z

• preparing the build of Lucre (dependencies): ca. 4 minutes
• compilation: 1.5 minutes
• test run: 6 minutes (it does a lot of data structure and database-baked testing that uses disk I/O, so it's inherently taking that time)

That's around 11.5 minutes. In comparison, Scala STM, beginning at 2020-12-08T00:54:20.8986504Z

• preparing the build of Scala STM (dependencies): ca. 0.5 minutes
• compilation: 1.5 minutes
• test run: <1 minute

Now we're at 14.5 minutes. Add another 30 seconds for the other eight dependencies (each publish takes around 30 seconds), then we sum up to around 20 minutes.

[Sciss]

So to summarise, I don't think removing intermediate tests will bring down the time much; maybe two minutes or so. Running less (selected) tests for Lucre is not a good idea; here it would require that the tests be parametrised, for example to build smaller data structures; this could shave off max. three minutes I think.

[griggt]

Moving the tests for Lucre and its dependencies from CommunityBuildTestB to CommunityBuildTestA in CommunityBuildTest.scala may shave off another ~3.5 minutes (the build/publish time for scalatest), since scalatest is already built and published in CommunityBuildA, but not CommunityBuildB.

[Sciss]

Ok, should I try that - move to A, disable all tests but stm/lucre?

[smarter]

That sounds good to me, I also think we could try splititng the community build into more concurrent jobs now that we can run more jobs at once (#10703) but that can be explored separately.

[griggt]

we could try splititng the community build into more concurrent jobs

I had the same thought... how many runners do we have now?

[smarter]

I had the same thought... how many runners do we have now?

14 right now, but I'm monitoring how things go and will increase it gradually until the machines are fully used.

[smarter]

LGTM, thank you!