Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As reported by @klensy on Zulip, Github Actions have recently updated their Docker version from 20.x to 23.x, which enabled the BuildKit build backend by default.
This broke our way of performing Docker layer caching on CI, which immediately made all non-PR CI builds (including try builds) ~1 hour longer (Docker caching didn't work on PR builds before, so it wasn't affected). The moment this started happening can be seen here.
The problem is with the following command:
which returns the intermediate layers as
<missing>
, if BuildKit is enabled. This was investigated by @klensy in #114621. Thanks for that!I will continue experimenting with how we can enable the cache with BuildKit in #114762, but for the time being, I think that we should just hotfix this.
This PR reverts the build backend back to the old one, which fixes the caching. However, we also have to bust the cache of all Dockerfiles, otherwise caching would only start kicking in for them the next time they are updated (or the next time GH updates their docker version). Because when the Docker version was updated the last time, the Dockerfiles were cached on S3 with basically an empty cache, and unless we bust it, even after reverting to the old build engine, the CI script would just download the empty cache and rebuild the Dockerfile from scratch, thus nullifying our fix.
r? @Mark-Simulacrum