|
| 1 | +--- |
| 2 | +layout: blog-detail |
| 3 | +post-type: blog |
| 4 | +by: Alexandre Archambault, Sébastien Doeraene |
| 5 | +title: "Better management of transitive dependencies and conflicts" |
| 6 | +--- |
| 7 | + |
| 8 | +Early in 2019, the Scala Center advisory board accepted [a proposal from Spotify](https://github.com/scalacenter/advisoryboard/blob/master/proposals/020-sbt-transitive-dependencies-conflicts.md), whose goal is to improve the handling of conflicts in dependency management in sbt. |
| 9 | + |
| 10 | +The work we've been doing may interest you if: |
| 11 | + |
| 12 | +* You've run into trouble with version conflicts between dependencies and didn't know what to do or where to look for advice. |
| 13 | +* You want more control on how version conflicts are resolved in your build. |
| 14 | +* You want automated assistance with deciding whether a version upgrade might break something in your project. |
| 15 | + |
| 16 | +At the time we started this project, Ivy was still used by default to resolve libraries in sbt, but efforts were in progress to use coursier instead, so we focused on the latter as a vehicle for improvements. |
| 17 | +Since then, coursier support in sbt [became official](https://github.com/sbt/sbt/pull/4614), and ships by default in sbt 1.3.0. |
| 18 | +Besides having a simpler model than the one used before in sbt or in Ivy, coursier has the advantage of being usable from the command line, and can benefit other users of the coursier API. |
| 19 | + |
| 20 | +For the third point about static analysis, we found [the missinglink tool](https://github.com/spotify/missinglink) for Maven, originating from Spotify themselves. |
| 21 | +This tool can analyze the classpath of a Maven project, and detect ahead of time (at "link" time) whether binary incompatibilities can be encountered at run-time. |
| 22 | +The only issue was that it was not usable from sbt, rendering it useless for a large subset of Scala users. |
| 23 | +We built and released [an sbt plugin for missinglink](https://github.com/scalacenter/sbt-missinglink) to address this hole. |
| 24 | + |
| 25 | +## Version ordering and reconciliation |
| 26 | + |
| 27 | +While most people have a simple intuition about library resolution (a newer version of a library gets picked over an older version), the details of how versions are compared and reconciled are quite complicated. |
| 28 | +We contributed [a dedicated documentation page](https://get-coursier.io/docs/other-version-handling) on the coursier website to help users understand how resolution works in the not-so-trivial cases. |
| 29 | +In order to improve the experience of upgrading from sbt 1.2.x and Ivy-based resolution, we [tweaked resolution in coursier 2.0.0-RC3-4](https://github.com/coursier/coursier/pull/1348) with the help of Eugene Yokota to be more in line with Ivy, the former sbt behavior, and the [semantic versioning specification](https://semver.org). |
| 30 | +The documentation page should be updated accordingly soon. |
| 31 | + |
| 32 | +On top of that, Eugene Yokota recently wrote [a blog post](http://eed3si9n.com/dependency-resolver-semantics) comparing how versions are compared and reconciled in both Ivy and coursier. |
| 33 | + |
| 34 | +## Strict conflict manager |
| 35 | + |
| 36 | +By default, both Ivy and coursier tolerate that the dependency graph contains two different versions of a library, in which case they pick a winner. |
| 37 | +Sometimes, we want more control, and do not want any conflict resolution to happen. |
| 38 | +Instead, we would like an error to be reported if there are two different versions of a library. |
| 39 | +A *strict* conflict manager does precisely that, and we added one to coursier. |
| 40 | +Instructions follow for enabling the strict conflict manager 1) in an sbt build, and 2) from the coursier command line. |
| 41 | + |
| 42 | +### sbt |
| 43 | + |
| 44 | +From sbt, enable the strict conflict manager either through the original `conflictManager` key, like |
| 45 | + |
| 46 | +```scala |
| 47 | +conflictManager := ConflictManager.strict |
| 48 | +``` |
| 49 | + |
| 50 | +or enable it in a possibly more fine-grained way if [sbt-coursier](https://get-coursier.io/docs/sbt-coursier) is around with |
| 51 | + |
| 52 | +```scala |
| 53 | +versionReconciliation += "*" % "*" % "strict" |
| 54 | +``` |
| 55 | + |
| 56 | +(Note that to use sbt-coursier from an sbt 1.3.x project, the [coursier-based sbt launcher](https://github.com/coursier/sbt-launcher) is required. |
| 57 | +Get it via its [custom sbt-extras runner](https://github.com/coursier/sbt-extras/blob/master/sbt) or generate one with `coursier bootstrap sbt-launcher && ./sbt`.) |
| 58 | + |
| 59 | +The `update` task will then succeed only if the strict checks pass. |
| 60 | +If they don't, either force the versions of faulty dependencies with `dependencyOverrides`, like |
| 61 | + |
| 62 | +```scala |
| 63 | +dependencyOverrides += "org.typelevel" % "cats-core_2.12" % "1.5.0" |
| 64 | +``` |
| 65 | + |
| 66 | +or adjust their version reconciliation, like |
| 67 | + |
| 68 | +```scala |
| 69 | +versionReconciliation ++= Seq( |
| 70 | + "org.typelevel" %% "cats-core" % "relaxed", // "semver" reconciliation is also available |
| 71 | + "*" % "*" % "strict" |
| 72 | +) |
| 73 | +``` |
| 74 | + |
| 75 | +A number of version reconciliation types are available: |
| 76 | + |
| 77 | +* `"strict"` requires all dependees to depend on the exact selected version (if they depend on an interval, the selected version only needs to be contained in it) |
| 78 | +* `"semver"` requires all dependees to depend on the same major version as the selected version (if they depend on an interval, the selected version only needs to be contained in it too) |
| 79 | +* `"default"` is the default version reconciliation in coursier, enabled in the coursier CLI or with the sbt-coursier plugin. |
| 80 | + It is described in more detail [in this page](https://get-coursier.io/docs/other-version-handling.html#reconciliation). |
| 81 | +* `"relaxed"` does not trigger any conflict. |
| 82 | + It is the default in the coursier support of sbt 1.3.x, which allows for better compatibility with former sbt versions. |
| 83 | + It ignores the lowest versions and version intervals, until they can be reconciled with the same algorithm as `"default"`. |
| 84 | + |
| 85 | +Since reconciliation strategies can be set per artifact, or per organization, it becomes much easier to declaratively tell sbt, for example, that some project adheres to SemVer, and that it is fine to resolve dependencies on that specific project according to SemVer, but not others. |
| 86 | + |
| 87 | +### CLI |
| 88 | + |
| 89 | +From the command line, we can use the new strict conflict manager of coursier with: |
| 90 | + |
| 91 | +```bash |
| 92 | +$ coursier resolve --strict \ |
| 93 | + org.typelevel:cats-effect_2.12:2.0.0 \ |
| 94 | + org.typelevel:cats-core_2.12:1.5.0 |
| 95 | +Resolution error: Unsatisfied rule Strict(*:*): Found evicted dependencies: |
| 96 | + |
| 97 | +org.typelevel:cats-core_2.12:2.0.0 (1.5.0 wanted) |
| 98 | +└─ org.typelevel:cats-core_2.12:1.5.0 |
| 99 | +``` |
| 100 | + |
| 101 | +To ignore some strict checks, one can exclude the faulty dependency with |
| 102 | + |
| 103 | +```bash |
| 104 | +$ coursier resolve \ |
| 105 | + --strict-exclude 'org.typelevel:cats-core*' \ |
| 106 | + org.typelevel:cats-effect_2.12:2.0.0 \ |
| 107 | + org.typelevel:cats-core_2.12:1.5.0 |
| 108 | +org.scala-lang:scala-library:2.12.9:default |
| 109 | +org.typelevel:cats-core_2.12:2.0.0:default |
| 110 | +org.typelevel:cats-effect_2.12:2.0.0:default |
| 111 | +org.typelevel:cats-kernel_2.12:2.0.0:default |
| 112 | +org.typelevel:cats-macros_2.12:2.0.0:default |
| 113 | +``` |
| 114 | + |
| 115 | +(passing `--strict-exclude` automatically enables `--strict`), or only include a subset of the dependencies in the strict checks, like |
| 116 | + |
| 117 | +```bash |
| 118 | +$ coursier resolve \ |
| 119 | + --strict-include 'org.scala-lang:*' \ |
| 120 | + org.typelevel:cats-effect_2.12:2.0.0 \ |
| 121 | + org.typelevel:cats-core_2.12:1.5.0 |
| 122 | +org.scala-lang:scala-library:2.12.9:default |
| 123 | +org.typelevel:cats-core_2.12:2.0.0:default |
| 124 | +org.typelevel:cats-effect_2.12:2.0.0:default |
| 125 | +org.typelevel:cats-kernel_2.12:2.0.0:default |
| 126 | +org.typelevel:cats-macros_2.12:2.0.0:default |
| 127 | +``` |
| 128 | + |
| 129 | +Alternatively, forcing the version of the faulty dependency makes the strict checks ignore it: |
| 130 | + |
| 131 | +```bash |
| 132 | +$ coursier resolve \ |
| 133 | + --strict \ |
| 134 | + --strict-exclude 'org.scala-lang:*' \ |
| 135 | + --force-version org.typelevel:cats-core_2.12:1.5.0 \ |
| 136 | + org.typelevel:cats-effect_2.12:2.0.0 \ |
| 137 | + org.typelevel:cats-core_2.12:1.5.0 |
| 138 | +org.scala-lang:scala-library:2.12.9:default |
| 139 | +org.scala-lang:scala-reflect:2.12.6:default |
| 140 | +org.typelevel:cats-core_2.12:1.5.0:default |
| 141 | +org.typelevel:cats-effect_2.12:2.0.0:default |
| 142 | +org.typelevel:cats-kernel_2.12:1.5.0:default |
| 143 | +org.typelevel:cats-macros_2.12:1.5.0:default |
| 144 | +org.typelevel:machinist_2.12:0.6.6:default |
| 145 | +``` |
| 146 | + |
| 147 | +## Static analysis checks |
| 148 | + |
| 149 | +To leverage the power of the static analysis offered by [missinglink library](https://github.com/spotify/missinglink) in sbt projects, we have developed [sbt-missinglink](https://github.com/scalacenter/sbt-missinglink). |
| 150 | +The readme explains how to use it, but the basics are simple. |
| 151 | + |
| 152 | +First, add the following line in `project/plugins.sbt`: |
| 153 | + |
| 154 | +```scala |
| 155 | +addSbtPlugin("ch.epfl.scala" % "sbt-missinglink" % "0.1.0") |
| 156 | +``` |
| 157 | + |
| 158 | +then simply run the following task for the project you want to test: |
| 159 | + |
| 160 | +``` |
| 161 | +> theProject/missinglinkCheck |
| 162 | +``` |
| 163 | + |
| 164 | +This will check that the transitive dependencies of your project do not exhibit any binary compatibility conflict, assuming that the methods of your `Compile` configuration (in `src/main/`) are all called. |
| 165 | +You can add that task to your CI script, for example. |
| 166 | + |
| 167 | +If the checks succeed, you have a guarantee that no `LinkageError` can happen at run-time, unless run-time reflection is involved, making static analysis impossible. |
| 168 | + |
| 169 | +The plugin is very simple at the moment, and will be extended to support more features of `missinglink`, in particular the ability to add exclusions to the checks performed. |
| 170 | +If `missinglink` itself proves insufficient for the needs of Scala developers, we may also contribute improvements to it in the future. |
| 171 | + |
| 172 | +## Conclusion |
| 173 | + |
| 174 | +We have seen a few new ways you can improve the management of your transitive dependencies. |
| 175 | +In particular, we have introduced: |
| 176 | + |
| 177 | +* *reconciliation* strategies, which let you declaratively specify the compatibility guarantees of individual libraries, allowing coursier to do a better, safer job at conflict resolution, and |
| 178 | +* `sbt-missinglink`, an sbt plugin to easily check *ahead of time* that your resolved transitive dependencies are actually binary compatible. |
0 commit comments