Restructure Spark Project Cross Compilation

[jbaiera]

The way that ES-Hadoop currently handles cross compiling the Spark integration for various versions of Scala has reached a point where it needs to be reconsidered. Since code compiled with one major version of Scala is not binary compatible with a Scala runtime on a different major version, most Scala based projects must re-compile and release separate artifacts for each supported version of Scala.

This process of cross compilation is supported natively on SBT, but since this project is based first and foremost in Java and makes extensive use of the Elasticsearch project's testing facilities by ways of Gradle plugins, converting to SBT in order to fix the problems we are seeing is not an option.

The current process for cross compiling our Scala libraries in the project is to make use of an in-house gradle plugin that recursively launches the gradle build with a different version of Scala specified. The child build process performs the variant assembly, taking care to rename artifacts and the like as needed.

We've run into a number of problems with this process though:

1. With a recursive build Gradle cannot apply build optimizations in a normal fashion.
2. As core Gradle logic around project configuration changes, our cross compile plugin logic has broken, caused maintenance issues, and delayed upgrades.
3. As Spark and Scala release new supported variants on very different release schedules, the versions that ES-Hadoop support begin to diverge. For example, Spark 2.x does not support Scala 2.10 any more, and Spark 1.6 does not support Scala 2.12. The direction we want to take the project's structure would make our cross compile process incompatible with supporting new versions of Spark that diverge in their Scala support outside of major releases.
4. Testing the different variants requires a complicated array of CI configurations.

A potential solution to this issue that we are actively investigating is the usage of officially supported Gradle variant artifacts to organize the build logic. By using Gradle's variant system we solve the following problems:

1. All variants are built with one execution of the Gradle command, which allows more build optimizations to be applied.
2. Since the variant configuration is officially supported (unlike nested and recursive builds) we are better insulated to breakages when trying to upgrade.
3. We can correctly model the project as it always should have been while still supporting new Spark and Scala versions that diverge from the earlier supported versions of both.
4. We can test all the variants in one build.

[nfx]

I just wonder - would it make sense to move spark integration out to separate project?

[jbaiera]

I'm assuming you are suggesting that we use SBT for the Spark integration portion of the project. I'm -1 on this idea for the time being. Moving the integration to another project/build tool invites a different set of build challenges, mostly around the process of testing, license checking, and other functionalities we've come to depend on with Gradle and Elasticsearch Build Tools. I think it unwise to increase the build tool footprint of the project. Getting the benefit of using SBT for native cross compilation would just not offset the costs of adopting the tool enough (at this time). I'd be more amenable to the idea if the Gradle variant approach does not end up working correctly.

[mmigdiso]

hey @jbaiera , I see your point and as a newbie in es-hadoop code, I feel your pain. But on the other hand, don't you think that es-hadoop should somehow support the recent releases of spark? spark 3.0 is now out and many companies are already evaluating the migration plans for it. The ES-hadoop package is pretty critical for the spark users and will be a blocker for the spark3.0 migration for many.

[jbaiera]

@mmigdiso I completely understand the push for supporting the newer versions of Spark, but this work is very much needed in order to get there while still satisfying our compatibility requirements. Gradle's support for Scala is lacking in the cross compilation area, and in order to get where we need to be there are some complicated changes that need to be made. A decent number of PR's have already been merged that deal with this issue. I'll go through and link them up to it for the sake of visibility. The goal here is to have a build process that can handle the changes made to the Spark and Scala ecosystem going forward with minimal turn around time, starting with Spark 2.4.3 and 3.0 as well as Spark 2.12 and above.