Consider clearly separating eval-time vs build-time expressions

[michaelpj]

Often we have builds that do some work at eval-time, and some at build-time. This requires some careful managing of evalPackages etc., and getting it wrong leads to issues like that fixed in #810.

Here's a hypothesis: any given derivation will be almost entirely build-time or eval-time, and any eval-time deps will be few enough that they could be passed in specially.

This suggests a way of structuring our expressions: everything uses only an unqualified pkgs, and we select as-late-as-possible whether we instantiate it with buildPackages/evalPackages/whatever.

For example, I think lib/call-cabal-project-to-nix.nix should be run entirely at eval time, and so should just have a single pkgs: the eval packages. However, it in fact gets passed a variety of things from both evalPackages and buildPackages, which frankly seems suspicious.

[angerman]

@michaelpj I wholeheartedly agree with this in spirit! We do need to take care of IFD issues wrt to platform dependencies. The use of IFDs is primarily to bridge the gap between the nix world, and the real world where we don't want to reimplement logic in nix. However the real world often has implicit platform dependencies (configure script discoveries, cabal configure being platofrm dependent, ...). This is also one of the reasons Stackage LTSs are essentially linux only. They happen to work on macOS mostly due to macOS ~= linux, and on windows as long as you don't do fancy things (the LTSs have occasionally not even mentioned Win32 as an available package).

@hamishmack did a great job with the dummy-ghc's so that we can effectively pretend to have a target ghc and trick cabal, but this is all so hacky :-(

@phadej has been working on a cabal-solver tool, which I hope will be able to eliminate most of these issues, by not using the ghc binary, but being explicit about the inputs (arch, os, ...)

[hamishmack]

For example, I think lib/call-cabal-project-to-nix.nix should be run entirely at eval time, and so should just have a single pkgs: the eval packages. However, it in fact gets passed a variety of things from both evalPackages and buildPackages, which frankly seems suspicious.

The derivation that captures the output of ghc and ghc-pkg (dummy-ghc-data) has to run on the build platform.

[michaelpj]

The derivation that captures the output of ghc and ghc-pkg (dummy-ghc-data) has to run on the build platform.

So how does this work if you eval on a machine with a platform different to the build platform? You either have to have a remote builder of the right platform or hope it's cached? I thought part of the point of the evalPackages thing was to avoid the situation where you end up unable to eval for another platform because it requires you to build things on that platform?

I still think it would be good if this was at least very visible at the top level, e.g.:

callCabalProjectToNix = import ./call-cabal-project-to-nix {
   # Runs at eval time
   pkgs = evalPackages;
   # IMPORTANT: this still depends on dummy ghcs from the build platform, important that these are cached
   dummy-ghc = buildPackages.dummyGhc 
}

[hamishmack]

So how does this work if you eval on a machine with a platform different to the build platform?

The output of ghc is captured on the build platform by dummy-ghc-data and that data is used by the dummy-ghc scripts that run on the eval platform.

You either have to have a remote builder of the right platform or hope it's cached?

The captured output (dummy-ghc-data) is materialized in haskell.nix.

I think we could refactor it so that the dummy-ghc-data is derivations are attributes of the ghc derivations. Then I think we could pass in compilers instead dummy-ghc.

callCabalProjectToNix = import ./call-cabal-project-to-nix {
   pkgs = evalPackages;
   compilers = buildPackages.haskell-nix.compilers;
}

[michaelpj]

The captured output (dummy-ghc-data) is materialized in haskell.nix.

Ah I see, so materialized things are a bit odd, in that they nominally have a build platform, but you only need that if you're doing checkMaterialization or similar. Tricky.

Again, I guess what I want is mostly just to have the tricky platform stuff very obvious.

I think we could refactor it so that the dummy-ghc-data is derivations are attributes of the ghc derivations. Then I think we could pass in compilers instead dummy-ghc.

I actually prefer it the way it is, since at least if we just pass in the dummy bits we can assert "and these are always materialized", whereas if we passed in the whole compiler you could conceivably be using it, which really would require the build platform.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michaelpj]

I still think this is a good idea.

[hamishmack]

Everywhere evalPackages is used it is now passed in explicitly (there is no pkgs.evalPackages that might magically start running stuff on the eval system). I think that probably helps.

What file would be a good place to start? Is lib/cabal-projecct-parser.nix still of concern?

[michaelpj]

I agree that helps a lot. I don't think this is a high priority though, and maybe it would just end up being more trouble than it's worth 🤷

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.