Description
Currently the ggml, llama.cpp and whisper.cpp projects share the same source of the ggml library, but have different CMake scripts. The scripts are adapted to the specifics of the projects and are quite similar with each other - all of them build ggml. Still, there are differences due to manually rewriting them and applying changes from one repo to another
The goal in this task is to unify, deduplicate and streamline the build process of ggml with proper CMake scripts that are shared across the projects. This will simplify changes in the future and will also help other 3rd party projects that depend on ggml
More on this topic has been discussed in:
- Feature: A unified configure (and build) process for llama.cpp #5890
- add way to compile ggml with intel mkl ggml#804
To achieve that, the ggml-related sources in llama.cpp and whisper.cpp would likely have to be reorganized in a subfolder to emulate a submodule. We avoid usage of actual git submodules since I consider that it has some disadvantages. Instead, we do manual synchronization with sync scripts across the 3 repos. In any case, after we complete this task it would be much simpler to switch to a ggml submodule if we decide to do so in the future
Regarding the existing Makefiles in llama.cpp and whisper.cpp - we should keep those as an alternative build system. It does not have to support all possible backends and configurations as the primary CMake build would do so. Makefile maintenance will be low priority
After the build system is improved, we can consider extending it with build-time generated configuration (e.g. config.h) for increased compatibility as suggested in #5890. But for now this remains low priority