new-build --enable-benchmarks does not cause benchmarks to be built

[mrkkrp]

Describe the bug

Passing --enable-benchmarks to new-build does not cause benchmarks to be built.

To Reproduce

Here is Travis CI log for megaparsec which has broken benchmarks but still it's all green:

https://travis-ci.org/mrkkrp/megaparsec/jobs/585309471#L296

The exact command in this case is:

cabal new-build megaparsec megaparsec-tests --enable-tests --enable-benchmarks --flags=dev

yet even simpler command also doesn't build benchmarks:

cabal new-build --enable-benchmarks

It returns with exit code 0 even though the benchmarks are broken and do not compile.

Please use version-prefixed commands (e.g. v2-build or v1-build) to avoid ambiguity.

I got at first a different result with v2-build: it failed as it should. Then after that new-build also started to fail. Then later it started to report that the build is "up to date" returning 0 (the code is still broken).

Expected behavior

When --enable-benchmarks is passed, all benchmarks should be built and if they do not compile we should reliably get non-zero exit status no matter what.

System informataion

• Operating system: NixOS
• cabal version 2.4.1.0
• ghc version 8.6.5

[mrkkrp]

Here we can see clearly that despite --enable-benchmarks no benchmarks are considered for building even though the megaparsec package has two benchmarks: bench-speed and bench-memory.

[hvr]

nevermind; disregard the comment below -- I misunderstood your report at first

---

@mrkkrp Btw, cabal new-build w/o any target is context-specific (i.e. CWD specific;similar to how make works)

iow, does the all target (i.e. "all enabled components") build them

cabal v2-build --enable-benchmark all

or specifically only the benchmarks via the special target

cabal v2-build --enable-benchmark all:benches

which denotes the enabled benchmarks of all components do what you expect?

The target syntax is documented also at https://cabal.readthedocs.io/en/latest/nix-local-build.html#cabal-v2-build

[hvr]

@mrkkrp ok, here's what's going on:

the target megaparsec refers to the library component of megaparsec, not to the package megaparsec

in other words, the statement

cabal new-build megaparsec megaparsec-tests --enable-tests --enable-benchmarks --flags=dev

gets resolved into

cabal new-build lib:megaparsec lib:megaparsec-tests --enable-tests --enable-benchmarks --flags=dev

due to the target-resolution rules in the light of ambiguities; whereas you intended it to denote

cabal new-build :pkg:megaparsec :pkg:megaparsec-tests --enable-tests --enable-benchmarks --flags=dev

does this make any sense?

Fwiw, is there any reason you didn't simply use

cabal new-build all --enable-tests --enable-benchmarks --flags=dev

which doesn't require you to enumerate the local packages explicitly?

[mrkkrp]

I don't think I knew about the all magic.

Why do test suites get built though? They are other components, different from libraries, just like benchmarks. What cofuses me now is that test suites do get built, but not benchmarks.

[hvr]

@mrkkrp you're right; something's inconsistent here; the target doesn't seem to be resolved the way I described it... needs more investigation

[jneira]

Just hit this, i think for the third time
And nowadays things are worse cause you can skip the all target, cabal build --enable-benchmarks does not build local packages benchmarks but cabal build all --enable-benchmarks continue doing it as described above

[jneira]

I just double checked in the hls repo:

PS D:\hls> cabal build  --enable-benchmarks
Warning: Requested index-state 2021-12-29T12:30:08Z is newer than
'hackage.haskell.org'! Falling back to older state (2021-12-29T11:45:44Z).
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - ghcide-1.5.0.1 (exe:ghcide-test-preprocessor) (first run)
 - hie-compat-0.2.1.0 (lib) (first run)
 - hls-graph-1.5.1.1 (lib) (first run)
 - hiedb-0.4.1.0 (lib) (requires build)
 - hls-plugin-api-1.2.0.2 (lib) (first run)
 - ghcide-1.5.0.1 (lib) (first run)
.... hls plugins
 - haskell-language-server-1.5.1.0 (lib) (first run)
 - hls-refine-imports-plugin-1.0.0.2 (lib) (first run)
 - haskell-language-server-1.5.1.0 (exe:haskell-language-server-wrapper) (first run)
 - haskell-language-server-1.5.1.0 (exe:haskell-language-server) (first run)
 - haskell-language-server-1.5.1.0 (test:wrapper-test) (first run)
 - haskell-language-server-1.5.1.0 (test:func-test) (first run)
......

PS D:\hls> cabal build  all --enable-benchmarks
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - ghcide-1.5.0.1 (exe:ghcide-test-preprocessor) (first run)
 - hie-compat-0.2.1.0 (lib) (first run)
 - hls-graph-1.5.1.1 (lib) (first run)
 - hiedb-0.4.1.0 (lib) (requires build)
 - hls-plugin-api-1.2.0.2 (lib) (first run)
 - ghcide-1.5.0.1 (lib) (first run)
 - hls-test-utils-1.1.0.2 (lib) (first run)
... hls plugins
 - haskell-language-server-1.5.1.0 (lib) (first run)
 - ghcide-1.5.0.1 (exe:ghcide) (first run)
 - cubicbezier-0.6.0.6 (lib) (requires build)
 - diagrams-lib-1.4.5.1 (lib) (requires build)
 ... hls plugin tests
 - haskell-language-server-1.5.1.0 (exe:haskell-language-server-wrapper) (first run)
 - ghcide-1.5.0.1 (test:ghcide-tests) (first run)
 - ghcide-1.5.0.1 (exe:ghcide-bench) (first run)
 - hls-refine-imports-plugin-1.0.0.2 (test:tests) (first run)
 - haskell-language-server-1.5.1.0 (exe:haskell-language-server) (first run)
 - haskell-language-server-1.5.1.0 (test:wrapper-test) (first run)
 - haskell-language-server-1.5.1.0 (test:func-test) (first run)
 - shake-bench-0.1.0.2 (lib) (first run)
 - ghcide-1.5.0.1 (bench:benchHist) (first run)
.....

You can see in the second build and not in the first one shake-bench a local package used by ghcide-1.5.0.1 (bench:benchHist)

[Anton-Latukha]

In continuation of #6259 (comment).

In HLS we observed:

cabal build all --project-file=cabal-ghc901.project

Fails (at the state of haskell/haskell-language-server@4ffdf45).

The .project uses -stylishhaskell cabal flag:

flag stylishHaskell
  description: Enable stylishHaskell plugin
  default:     True
  manual:      True

to disable the hls-stylish-haskell-plugin, because stylish-haskell 0.13 (while has base <5), in reality does not support 9.0.1 & 9.2.1.

In CI dev workflows we running cabal build --project-file=cabal-ghc901.project as a base, so the builds & tests & benchmarks for all enabled targets to succeed. While the cabal all overrules all flag & .project configuration & starts to target & build the hls-stylish-haskell-plugin & so starts to build stylish-haskell 0.13 on unsupported GHC versions, which of course results in the error return code for the build. This overruling of .project settings was weird to observe. It is probably just a note may belong into a separate report or may be "works as intended".

The raw log of discussion in HLS: https://matrix.to/#/!oOjZFsoNYPAbTEgSOA:libera.chat/$0oqtvFoHlPe1At_LwJSQKrfboCKDcMvimyZUt8DS_jE?via=libera.chat&via=matrix.org&via=monoid.al

[jneira]

Correction about #6259 (comment)

• cabal build is not equivalent to cabal build all (yet). I think we talked about make them equivalent at some point but it is not implemented
• It seems in hls we have a top level package (hls itself) and cabal build is equivalent to cabal build pkg:haskell-language-server
  • it turns out that hat top level package has the rest of local packages as dependencies so build hls make build all subpackages libs
  • the layout is somewhat similar to megaparsec, which has a top level package megaparsec and a subpackage megaparsec-tests (but our top level package has no benchmarks)

So sorry for the confusion, as penitence will try to reproduce the original issue in megaparsec with last cabal

[jneira]

I can reproduce the first case with a recent cabal built from master:

D:\ws\haskell\issues\megaparsec>cabal v2-build megaparsec megaparsec-tests --enable-tests --enable-benchmarks --flags=dev -w ghc-8.10.7 --dry-run
....
 - megaparsec-tests-9.2.0 (lib) (first run)
 - megaparsec-tests-9.2.0 (test:tests) (first run)

• also reproduced for cabal v2-build megaparsec(without megaparsec-tests)

D:\ws\haskell\issues\megaparsec>cabal v2-build megaparsec --enable-tests --enable-benchmarks --flags=dev -w ghc-8.10.7 --dry-run
....
 - megaparsec-tests-9.2.0 (lib) (first run)

• but no for cabal build --enable-benchmarks

D:\ws\haskell\issues\megaparsec>cabal v2-build --enable-benchmarks -w ghc-8.10.7
 --dry-run
Build profile: -w ghc-8.10.7 -O1
In order, the following would be built (use -v for more details):
 - megaparsec-9.2.0 (lib) (first run)
 - megaparsec-9.2.0 (bench:bench-memory) (first run)
 - megaparsec-9.2.0 (bench:bench-speed) (first run)

So the bug is still present and it seems nowadays cabal build without targets is not exactly equivalent to use explicit targets