Allow libraries to reuse binaries compiled with older NIF versions

[cocoa-xu]

Hi this PR added a feature that allows libraries to reuse binaries compiled with older NIF versions instead of compiling the same code over and over again for all NIF versions they can support.

The main motivation is to reduce CI build time and release size as older NIF libraries can be used with newer Erlang/OTP versions.

This feature is done by changing how users can config the :make_precompiler_nif_versions key. And now there are two ways configure elixir_make on how to compile and reuse precompiled binaries.

Use minimum NIF version (recommended)

The first way is to use the minimum NIF version that is available to the current target. For example, if the NIF library
only uses functions and feature available in NIF version 2.14, the following configuration should be set:

make_precompiler_nif_versions: [
  use_minimum_version: true,
  minimum_version: "2.14"
]

In this way, we can reuse the same precompiled binary for newer Erlang/OTP versions instead of compiling the same code over and over again. And if the user is using the NIF library on an older Erlang/OTP version, elixir_make will try to compile the NIF library for the target.

Additionally, if the NIF library can optionally use functions and/or features that are only available in NIF version 2.17, then the following configuration can be set to provide better support for newer Erlang/OTP versions while still providing support for older Erlang/OTP versions:

make_precompiler_nif_versions: [
  use_minimum_version: true,
  minimum_version: fn _target, current_nif_version ->
    if current_nif_version >= "2.17" do
        "2.17"
    else
        "2.14"
    end
  end
]

The above example tells elixir_make that it can reuse binaries compiled with NIF version 2.17 if the current host supports it; otherwise, it will use binaries compiled with NIF version 2.14.

Users can further change the above example to use different target name to control which feature set is available. For example, to use NIF version 2.17 for any aarch64 targets and 2.14 for other targets:

make_precompiler_nif_versions: [
  use_minimum_version: true,
  minimum_version: fn target, current_nif_version ->
    if current_nif_version >= "2.17" and String.contains?(target, "aarch64") do
        "2.17"
    else
        "2.14"
    end
  end
]

However, the versions_for_target sub-key must be set to inform :elixir_make that the precompiled artefacts may have different NIF versions for different targets. For example,

make_precompiler_nif_versions: [
  use_minimum_version: true,
  minimum_version: fn target, current_nif_version ->
    if current_nif_version >= "2.17" and String.contains?(target, "aarch64") do
        "2.17"
    else
        "2.14"
    end
  end,
  versions_for_target: fn target ->
    if String.contains?(target, "aarch64") do
      ["2.17", "2.14"]
    else
      ["2.14"]
    end
  end
]

This information is used to determine all available precompiled artefacts and is used to fetch all of them and generate the checksum file.

make_precompiler_nif_versions: [
  use_minimum_version: true,
  minimum_version: fn target, current_nif_version ->
    if current_nif_version >= "2.17" do
        "2.17"
    else
        "2.14"
    end
  end
]

Use exact NIF version

The second (and the old) way is to use the exact NIF version that is available to the current target. In this case, a list of supported NIF versions should be set in key versions. For example,

make_precompiler_nif_versions: [versions: ["2.14", "2.15", "2.16"]]

The above example tells elixir_make that precompiled artefacts are available for these NIF versions. If there's no precompiled artefacts available on the current host, elixir_make will try to compile the NIF library.

If you'd like to aim for an older NIF version, say 2.15 for Erlang/OTP 23 and 24, then you need to setup CI correspondingly and set the value of this key to [versions: ["2.15", "2.16"]]. This optional key will only be checked when downloading precompiled artefacts.

For some platforms maybe we only have precompiled artefacts after a certain NIF version, say for x86_64 Windows we have precompiled artefacts available when NIF version >= 2.16 while other platforms have precompiled artefacts available from NIF version >= 2.15.

In such case we can inform :elixir_make that Windows targets don't have precompiled artefacts available except for NIF version 2.16 by passing a function to the availability sub-key.

defp target_available_for_nif_version?(target, nif_version) do
  if String.contains?(target, "windows") do
    nif_version == "2.16"
  else
    true
  end
end

The default value for make_precompiler_nif_versions is

[versions: ["#{:erlang.system_info(:nif_version)}"]]

[josevalim]

Thank you @cocoa-xu! I think this is an important feature to have indeed.

I was thinking though, perhaps we should have it enabled by default. This means that, as soon as we release a new version of elixir_make, everyone can benefit from it without publishing new assets. The default behaviour would be to find biggest major version still supported. For example, this:

make_precompiler_nif_versions: [
  versions: ["2.14", "2.15"],
]

Now imagine that "2.16" is released, we can do this:

defp nif_version_to_tuple(nif_version) do
  [major, minor | _] = String.split(nif_version, ".")
  {String.to_integer(major), String.to_integer(minor)}
end

defp find_nif_version(nif_version, versions) do
  if nif_version in versions do
    # use it
  else
    # default fallback logic
    {major, minor} = nif_version_to_tuple(nif_version)
    versions = Enum.map(versions, &nif_version_to_tuple/1)

    # Get all matching major versions, earlier than the current version
    # and their distance. We want the closest (smallest distance).
    candidates =
      for version <- versions,
          {^major, candidate_minor} <- [nif_version_to_tuple(version)],
          candidate_minor <= minor,
          do: {minor - candidate_minor, version}

    case Enum.sort(candidates) do
      [{_, version} | _] -> version
      _ -> # no candidates
    end
  end
end

The second part of the patch is to make it customizable. I think we can provide a :fallback_version option, which is pretty much the same option as you defined for minimum_version, except with a different name and it is only invoked in case we can't find a matching name:

make_precompiler_nif_versions: [
  fallback_version: fn target, current_nif_version ->
    if current_nif_version >= "2.17" and String.contains?(target, "aarch64") do
        "2.17"
    else
        "2.14"
    end
  end
]

WDYT? Would this work?

[josevalim]

Well, my snippets above need to be slightly modified to only pass versions that are available. Perhaps, the fallback_version function should receive three arguments target, current_nif_version, available_nif_versions (which are all versions after the availability check).

[cocoa-xu]

Hi @josevalim, I think the suggested changes should work. I've pushed some new commits to do it. Probably we can deprecate the availability key in this PR, and change how we use the versions key in #80?

[josevalim]

Thank you @cocoa-xu!

Probably we can deprecate the availability key in this PR, and change how we use the versions key in #80?

I would prefer to keep the pull request smaller. So I would add fallback version to this PR and we deprecate availability and add further features later. This way we can discuss each change individually, instead of juggling several at once. :)

[cocoa-xu]

Thank you @cocoa-xu!

Probably we can deprecate the availability key in this PR, and change how we use the versions key in #80?

I would prefer to keep the pull request smaller. So I would add fallback version to this PR and we deprecate availability and add further features later. This way we can discuss each change individually, instead of juggling several at once. :)

No problem, I've reverted related changes and only added fallback_version for this PR/

[josevalim]

💚 💙 💜 💛 ❤️