Add sbt incremental compilation support #1244

smarter · 2016-05-02T23:07:06Z

To test this with sbt, see
https://github.com/lampepfl/dotty/wiki/Using-Dotty-with-sbt

The following flags are added:

-Yforce-sbt-phases: Run the phases used by sbt for incremental compilation
(ExtractDependencies and ExtractAPI) even if the compiler is ran outside of
sbt, for debugging.
-Ydump-sbt-inc: For every compiled foo.scala, output the sbt API
representation and dependencies in foo.inc, implies -Yforce-sbt-phases.

This commit introduces two new phases which do not transform trees:

ExtractDependencies which extracts the dependency information of the current
compilation unit and sends it to sbt via callbacks
ExtractAPI which creates a representation of the API of the current compilation
unit and sends it to sbt via callbacks

Briefly, when a file changes sbt will recompile it, if its API has
changed (determined by what ExtractAPI sent) then sbt will determine
which reverse-dependencies (determined by what ExtractDependencies
sent) of the API have to be recompiled depending on what changed.
See http://www.scala-sbt.org/0.13/docs/Understanding-Recompilation.html for
more information on how sbt incremental compilation works.

This phase was originally based on
https://github.com/adriaanm/scala/tree/sbt-api-consolidate/src/compiler/scala/tools/sbt
which attempts to integrate the sbt phases into scalac (and is itself based
on https://github.com/sbt/sbt/tree/0.13/compile/interface/src/main/scala/xsbt),
but it has been heavily refactored and adapted to Dotty. The main
functional differences are:

ExtractDependencies runs right after Frontend (so that we don't lose
dependency informations because of the simplifications done by PostTyper),
but ExtractAPI runs right after PostTyper (so that SuperAccessors are
part of the API).
ExtractAPI only extract types as they are defined and never "as seen
from" some some specific prefix, see its documentation for more details.
ExtractDependenciesTraverser and ExtractUsedNames have been fused into
one tree traversal in ExtractDependenciesCollector.

TODO: Try to run these phases in parallel with the rest of the compiler
pipeline since they're independent (except for the sbt callbacks in GenBCode) ?

smarter · 2016-05-02T23:17:00Z

To run the scripted tests in dotty-bridge:

$ sbt
> scripted source-dependencies/*

DarkDimius · 2016-05-04T10:38:49Z

interfaces/src/main/java/dotty/tools/dotc/interfaces/CompilerCallback.java

@@ -25,4 +27,12 @@
   *                        Example: ./src/library/scala/collection/Seq.scala
   */
  default void onSourceCompiled(SourceFile source) {};
+
+  default void sourceDependency(File dependsOn, File source, xsbti.DependencyContext context) {};


I dont see what a method does given its name. Should it be addSourceDependency?

I'll remove these extra callbacks, they're not needed now that I added Context#sbtCallback instead.

DarkDimius · 2016-05-08T19:21:09Z

src/dotty/tools/dotc/config/ScalaSettings.scala

@@ -160,6 +160,8 @@ class ScalaSettings extends Settings.SettingGroup {
  val YtestPickler = BooleanSetting("-Ytest-pickler", "self-test for pickling functionality; should be used with -Ystop-after:pickler")
  val YcheckReentrant = BooleanSetting("-Ycheck-reentrant", "check that compiled program does not contain vars that can be accessed from a global root.")
  val YkeepComments = BooleanSetting("-Ykeep-comments", "Keep comments when scanning source files.")
+  val YforceSbtPhase = BooleanSetting("-Yforce-sbt-phase", "Run the sbt incremental compiler phase even if the compiler is ran outside of sbt, for debugging.")
+  val YdumpSbtApi = BooleanSetting("-Ydump-sbt-api", "For every compiled foo.scala, output the sbt API representation in foo.api, implies -Yforce-sbt-phase.")


dependsOn(YforceSbtPhase)?

Looks like this would force the user to write -Yforce-sbt-phase -Ydump-sbt-api, I can do the change if you want but I don't think it's very useful

smarter · 2016-05-10T00:26:20Z

I didn't realize this before but there's a single test that does not pass now: trait-super because it relies on the presence or absence of superaccesors in the API to know what should be recompiled, but in dotty superaccessors are created by PostTyper but the sbt phase is run before PostTyper so that it can see constant types and trees of types (and so that the constants test pass), I don't know if there's a way to fix this without doing some phase-travelling. Anyway, this is something that can be fixed later.

This involved some mad ClassLoader hacks, see `CompilerClassLoader`. Starting from this commit, dotty-bridge will only work with a dotty where scala/scala3#1244 has been merged.

smarter · 2016-05-16T21:52:36Z

OK I just fixed trait-super by splitting the sbt phase into two phases, ExtractDependencies before PostTyper and ExtractAPI after PostTyper (they were already completely independent before, but ran in the same phase one after the other), this works out because:

ExtractDependencies needs to be before PostTyper to avoid loosing dependency informations (see constants for an example where this matters).
ExtractAPI needs to be after PostTyper because SuperAccessors need to be part of the API (see again trait-super for an example where this matter). Unlike ExtractDependencies, the simplication to trees done by PostTyper do not affect this phase because it only cares about definitions, and PostTyper does not change definitions.

I've also cleaned up the changes needed to dotty-bridge, they now sit in smarter/dotty-bridge#1 and will be merged once this PR has been merged.

smarter · 2016-05-16T21:53:59Z

/cc @gkossakowski and @adriaanm who might be interested to know that this exist (also thanks to both of you for the discussions we had on incremental compilation!)

quick hack to experiment with stability/perf of api extraction I tried to stay as close to the way it's done in sbt, EXCEPT for fusing all traversals into one tree walk (there's still some local redundant tree.foreach'ing, but should be ok) i do believe integrating sbt's compiler interface into the compiler is the way forward

This involved some mad ClassLoader hacks, see `CompilerClassLoader`. Starting from this commit, dotty-bridge will only work with a dotty where scala/scala3#1244 has been merged.

DarkDimius · 2016-05-27T09:01:58Z

Needs rebasing.

DarkDimius · 2016-05-27T09:03:51Z

src/dotty/tools/dotc/Compiler.scala

@@ -41,56 +41,58 @@ class Compiler {
   */
  def phases: List[List[Phase]] =
    List(
-      List(new FrontEnd),           // Compiler frontend: scanner, parser, namer, typer


I'd prefer not to loose git history here.
Indentation is not that important.

Additionally, it will introduce a lot of merge conflicts with linker.

OK, will revert formatting change (though note that git blame -w and git diff -w make this less important)

DarkDimius · 2016-05-27T09:33:07Z

I've done a quick review, as I'm not into details of xsbti.api.

I'd be curious to see how much does introduction of this phase affect compilation time of dotty&stdlib.

Otherwise LGTM

smarter · 2016-05-28T02:12:20Z

I'd be curious to see how much does introduction of this phase affect compilation time of dotty&stdlib.

I spent a bunch of time optimizing it and currently it adds a few seconds to compilation, most of that time is just because of the full AST traversal done by ExtractDependenciesCollector which I don't think we can avoid (I don't think ExtractDependencies can be fused with any other phase). It would be interesting to see if we could just run it in a separate thread.

SymDenotations#topLevelClass: don't throw an exception Symbols#associateFile: avoid infinite loop

This works similarly to `Trees#TreeTraverser`.

To test this with sbt, see https://github.com/lampepfl/dotty/wiki/Using-Dotty-with-sbt The following flags are added: - -Yforce-sbt-phases: Run the phases used by sbt for incremental compilation (ExtractDependencies and ExtractAPI) even if the compiler is ran outside of sbt, for debugging. - -Ydump-sbt-inc: For every compiled foo.scala, output the API representation and dependencies used for sbt incremental compilation in foo.inc, implies -Yforce-sbt-phases. This commit introduces two new phases which do not transform trees: - `ExtractDependencies` which extracts the dependency information of the current compilation unit and sends it to sbt via callbacks - `ExtractAPI` which creates a representation of the API of the current compilation unit and sends it to sbt via callbacks Briefly, when a file changes sbt will recompile it, if its API has changed (determined by what `ExtractAPI` sent) then sbt will determine which reverse-dependencies (determined by what `ExtractDependencies` sent) of the API have to be recompiled depending on what changed. See http://www.scala-sbt.org/0.13/docs/Understanding-Recompilation.html for more information on how sbt incremental compilation works. This phase was originally based on https://github.com/adriaanm/scala/tree/sbt-api-consolidate/src/compiler/scala/tools/sbt which attempts to integrate the sbt phases into scalac (and is itself based on https://github.com/sbt/sbt/tree/0.13/compile/interface/src/main/scala/xsbt), but it has been heavily refactored and adapted to Dotty. The main functional differences are: - ExtractDependencies runs right after Frontend (so that we don't lose dependency informations because of the simplifications done by PostTyper), but ExtractAPI runs right after PostTyper (so that SuperAccessors are part of the API). - `ExtractAPI` only extract types as they are defined and never "as seen from" some some specific prefix, see its documentation for more details. - `ExtractDependenciesTraverser` and `ExtractUsedNames` have been fused into one tree traversal in `ExtractDependenciesCollector`. TODO: Try to run these phases in parallel with the rest of the compiler pipeline since they're independent (except for the sbt callbacks in `GenBCode`) ?

liufengyun · 2016-05-30T08:10:08Z

We have a big improvement in performance, I guess it's due to incremental compilation:

https://d-d.me/tnc/dotty/web/

smarter · 2016-05-30T13:03:31Z

That's very weird, the incremental compilation cannot possibily be used by these tests, they're not set up to be run through sbt and the dotty-bridge

This involved some mad ClassLoader hacks, see `CompilerClassLoader`. Starting from this commit, dotty-bridge will only work with a dotty where scala/scala3#1244 has been merged.

Blaisorblade · 2016-06-22T12:59:20Z

ExtractAPI runs right after PostTyper (so that SuperAccessors are part of the API).

@smarter Can you confirm this prevents issues like sbt/sbt#466, as I'm thinking?

To summarize: nowadays adding super calls to a trait requires recompiling implementing classes/objects, but SBT does not manage, because superaccessors is ignored by SBT (in sbt/sbt#466 (comment) I wondered whether that could be fixed). It seems your design prevents such problems?

smarter force-pushed the add/sbt-phase branch from 717f695 to 36f22d6 Compare May 2, 2016 23:15

DarkDimius reviewed May 4, 2016
View reviewed changes

smarter force-pushed the add/sbt-phase branch 2 times, most recently from 94c6809 to 5c5bdec Compare May 7, 2016 20:38

smarter changed the title ~~[DO NOT MERGE] Add sbt incremental compilation support via a special phase~~ Add sbt incremental compilation support via a special phase May 7, 2016

DarkDimius reviewed May 8, 2016
View reviewed changes

smarter force-pushed the add/sbt-phase branch 4 times, most recently from fd79ba0 to 341e76c Compare May 9, 2016 23:10

smarter force-pushed the add/sbt-phase branch 2 times, most recently from 401ca9e to fffa04e Compare May 16, 2016 21:38

smarter changed the title ~~Add sbt incremental compilation support via a special phase~~ Add sbt incremental compilation support May 16, 2016

smarter force-pushed the add/sbt-phase branch 2 times, most recently from 55340fe to 5dae0a7 Compare May 16, 2016 21:41

smarter mentioned this pull request May 16, 2016

Add support for incremental compilation smarter/dotty-bridge#1

Merged

smarter force-pushed the add/sbt-phase branch 2 times, most recently from d318f9f to 64d2b47 Compare May 16, 2016 22:32

DarkDimius reviewed May 27, 2016
View reviewed changes

smarter added 5 commits May 28, 2016 21:47

Avoid issues when calling methods on root package symbol

ca2fb9d

SymDenotations#topLevelClass: don't throw an exception Symbols#associateFile: avoid infinite loop

Fix memberExcluding for ClassInfo

e9adc8a

Fix typo in Desugar#refinedTypeToClass documentation

5cf997c

Add Types#TypeTraverser for convenience

e5ce083

This works similarly to `Trees#TreeTraverser`.

Document why Trees#SelectFromTypeTree cannot be a TypTree

bcdddd9

smarter force-pushed the add/sbt-phase branch from 64d2b47 to 65d1489 Compare May 28, 2016 19:49

smarter force-pushed the add/sbt-phase branch from 65d1489 to 4c865c5 Compare May 28, 2016 20:23

smarter merged commit fabe697 into scala:master May 29, 2016

smarter mentioned this pull request Sep 14, 2016

zinc's handling of As Seen From sbt/zinc#174

Closed

felixmulder mentioned this pull request Sep 26, 2016

compiling a project's code with dotty cvogt/cbt#166

Closed

allanrenucci deleted the add/sbt-phase branch December 14, 2017 16:58

Add sbt incremental compilation support #1244

Add sbt incremental compilation support #1244

Uh oh!

Conversation

smarter commented May 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkDimius May 4, 2016

Choose a reason for hiding this comment

Uh oh!

smarter May 7, 2016

Choose a reason for hiding this comment

Uh oh!

DarkDimius May 8, 2016

Choose a reason for hiding this comment

Uh oh!

smarter May 8, 2016

Choose a reason for hiding this comment

Uh oh!

smarter commented May 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented May 16, 2016

Uh oh!

smarter commented May 16, 2016

Uh oh!

DarkDimius commented May 27, 2016

Uh oh!

DarkDimius May 27, 2016

Choose a reason for hiding this comment

Uh oh!

smarter May 28, 2016

Choose a reason for hiding this comment

Uh oh!

DarkDimius commented May 27, 2016

Uh oh!

smarter commented May 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liufengyun commented May 30, 2016

Uh oh!

smarter commented May 30, 2016

Uh oh!

Blaisorblade commented Jun 22, 2016

Uh oh!

Uh oh!

smarter commented May 2, 2016 •

edited

Loading

smarter commented May 2, 2016 •

edited

Loading

smarter commented May 10, 2016 •

edited

Loading

smarter commented May 28, 2016 •

edited

Loading