[JAVA] [GRADLE] OOM Issue with GitHub Autobuilder for Kotlin

[JLLeitschuh]

Description of the issue

Currently getting an OOM error with the CodeQL Autobuilder when attempting to compile Kotlin.

https://github.com/JLLeitschuh/ktlint-gradle/actions/runs/14649396135/job/41111305233?pr=861#step:5:121

JLLeitschuh/ktlint-gradle#861

Build Scan: https://scans.gradle.com/s/mfzgqpzmjgq6q

The stack trace error is too long for a GitHub issue, so I've attached it here. It looks like the code injected inside the compiler is leading to an OOM in the JetBrains Kotlin compiler

logs_37644012249.zip

[redsun82]

👋 @JLLeitschuh I'm sorry to hear you had problems with setting up CodeQL for your kotlin project.

I don't think this should happen with autobuild in your case. As far as I could test moving to a manual build was successful, so I opened JLLeitschuh/ktlint-gradle#865 with the proposed changes:

• move to build-mode manual from autobuild
• move the gradle setup before codeql init: this is probably not required for it to work, but it's good hygiene, as otherwise setup steps get analyzed by CodeQL as build steps, which adds overhead and may give unwanted data in the CodeQL database
• provide an explicit command for the build to be analyzed instead of relying on autobuild: you may want to tweak this if it does not cover what you want being analyzed in your project.

In any case I will reach out with this to our internal team, as this might be an autobuild bug.

[JLLeitschuh]

The concern I have with this solution is that the init script that the autobuilder uses won't be injected into the Gradle build with this solution. I don't know how essential the init script is or not. There doesn't seem to be a way to get that init script out of the environment as far as I can tell.

This was the contents of that init script as of back in May of 2024:

allprojects { 
  try {
    tasks.withType(JavaCompile).configureEach {
      outputs.doNotCacheIf("CodeQL analysis", {true})
    }

    tasks.withType(ScalaCompile).configureEach {
      outputs.doNotCacheIf("CodeQL analysis", {true})
    }

    tasks.withType(GroovyCompile).configureEach {
      outputs.doNotCacheIf("CodeQL analysis", {true})
    }
  }
  catch(e) {
    println e
    println "Warning: CodeQL: configureEach not supported on this Gradle version. Using fallback instead."
    tasks.withType(JavaCompile) {
      try {
        outputs.cacheIf({false})
      }
      catch(e2) {
        println e2
        println "Warning: CodeQL: cacheIf not supported on this Gradle version."
      }
    }
    tasks.withType(ScalaCompile) {
      try {
        outputs.cacheIf({false})
      }
      catch(e2) {
        println e2
        println "Warning: CodeQL: cacheIf not supported on this Gradle version."
      }
    }
    tasks.withType(GroovyCompile) {
      try {
        outputs.cacheIf({false})
      }
      catch(e2) {
        println e2
        println "Warning: CodeQL: cacheIf not supported on this Gradle version."
      }
    }
  }

  // KotlinCompile comes from a plugin which the build may or may not
  // use, so we can't use it here. We therefore check task names
  // instead.
  try {
    tasks.matching { task -> task.name.equals( 'compileKotlin' ) }.configureEach {
      try {
        outputs.doNotCacheIf("CodeQL analysis", {true})
      }
      catch(e) {
        println e
        println "Warning: CodeQL: Failed to disable cache for Kotlin."
      }
    }
  }
  catch(e) {
    try {
      tasks.matching { task -> task.name.equals( 'compileKotlin' ) }.all {
        try {
          outputs.cacheIf({false})
        }
        catch(e2) {
          println e2
          println "Warning: CodeQL: Fallback failed to disable cache for Kotlin."
        }
      }
    }
    catch(e2) {
      println e
      println e2
      println "Warning: CodeQL: Failed to configure cache disabled for Kotlin."
    }
  }
}

[JLLeitschuh]

The downside of using the --no-build-cache solution as you've done in this pull request is that it disables the build cache on all tasks, not just the compiler tasks. The init script provided by the autobuilder seems to explicitly only target the compilation tasks, which is optimal. Especially on massive builds like the build of gradle/gradle which I worked to get CodeQL working on a few years ago.

[redsun82]

I see, let me pull in someone who can help out here.

[smowton]

I took a look at this, and it seems that simply tweaking the default limit for the Kotlin daemon did the trick.

At least on my machine the default memory limit is pretty severe -- it runs with -Xmx512m. Because the Kotlin extractor currently operates as a compiler plugin, we don't get the chance to influence that; the limit is set by the time the daemon spawns a Kotlin compilation and so we notice we need to inject our extraction plugin. This also means if the existing build is pretty memory-hungry then our extractor can push things over the edge.

The simplest solution is to write org.gradle.jvmargs=-Xmx4g (or quite possibly some lower value) to plugin/gradle.properties. I note by the way the root directory already has org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m but this isn't applied when running from the plugin subdirectory; this doesn't affect the overall limit though AFAIK.