Skip to content

Commit 48b8674

Browse files
sevanspowellsevanspowell
authored
Add support for coverage (#762)
- Added the ability to generate coverage reports for packages and projects. - Outputs mix and tix information, as well as a HTML report. - Added the "doCoverage" module option that allows users to choose packages to enable coverage for. - Added a "doCoverage" flag to the component builder that outputs HPC information when coverage is enabled. - Added the "overrideModules" library function to make it more ergonomic fo users to enable coverage on existing projects. - Modified the "check" builder to also output ".tix" files (if they exist). This information is required to generate the coverage report. - Added a test for coverage.
1 parent 742932a commit 48b8674

File tree

43 files changed

+1009
-42
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+1009
-42
lines changed

‎builder/comp-builder.nix

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,9 @@ let self =
5353
, enableExecutableProfiling ? component.enableExecutableProfiling
5454
, profilingDetail ? component.profilingDetail
5555

56+
# Coverage
57+
, doCoverage ? component.doCoverage
58+
5659
# Data
5760
, enableSeparateDataOutput ? component.enableSeparateDataOutput
5861

@@ -118,6 +121,7 @@ let
118121
(enableFeature enableExecutableProfiling "executable-profiling")
119122
(enableFeature enableStatic "static")
120123
(enableFeature enableShared "shared")
124+
(enableFeature doCoverage "coverage")
121125
] ++ lib.optionals (stdenv.hostPlatform.isMusl && (haskellLib.isExecutableType componentId)) [
122126
# These flags will make sure the resulting executable is statically linked.
123127
# If it uses other libraries it may be necessary for to add more
@@ -358,6 +362,11 @@ let
358362
fi
359363
done
360364
'')
365+
+ (lib.optionalString doCoverage ''
366+
mkdir -p $out/share
367+
cp -r dist/hpc $out/share
368+
cp dist/setup-config $out/
369+
'')
361370
}
362371
runHook postInstall
363372
'' + (lib.optionalString keepSource ''

‎changelog.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,15 @@
11
This file contains a summary of changes to Haskell.nix and `nix-tools`
22
that will impact users.
33

4+
## Sep 8, 2020
5+
* Added the ability to generate coverage reports for packages and
6+
projects.
7+
* Added the `doCoverage` module option that allows users to choose
8+
packages to enable coverage for.
9+
* Added a `doCoverage` flag to the component builder that outputs HPC
10+
information when coverage is enabled.
11+
* Added test for coverage.
12+
413
## July 21, 2020
514
* Removed `components.all`, use `symlinkJoin` on components.exes or
615
`shellFor` if you need a shell.

‎docs/dev/coverage.md

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Developer Coverage Overview
2+
3+
## Building
4+
5+
The implementation of coverage starts with the "doCoverage" flag on
6+
the builder in `comp-builder.nix`. The doCoverage flag enables and
7+
disables the Cabal coverage flag and copies any generated coverage
8+
data to "$out/share/hpc".
9+
10+
### Mix and tix files
11+
12+
The coverage information for any derivation consists of "mix" and
13+
"tix" files.
14+
15+
Mix files record static information about a source file and are
16+
generated at build time. They primarily contain a path to the source
17+
file and information about expressions and regions of the source file,
18+
which are later referenced by tix files.
19+
20+
Tix files contain dynamic information about a test run, recording when
21+
a portion of a source file is touched by a test. These are generated
22+
when the test is run.
23+
24+
### Multiple local packages
25+
26+
In the context of multiple local packages, there are a few types of
27+
coverage we might be interested in:
28+
- How well does the tests for this package cover the package library?
29+
- How well does the tests for this package cover the libraries of
30+
other packages in this project?
31+
- Both of the above.
32+
33+
To facilitate expressing any of these classifications of coverage, the
34+
`lib/cover.nix` function provides the `mixLibraries` argument. If
35+
you're just interested in how the tests cover the package library, you
36+
provide that library as an argument to `mixLibraries`. If you're
37+
interested in how the tests also cover other local packages in the
38+
project, you can also provide those libraries as arguments to
39+
mixLibraries.
40+
41+
The `projectCoverageReport` and `coverageReport` attributes that are
42+
provided by default on projects and packages respectively provide
43+
coverage information for *all* local packages in the project. This is
44+
to mimic the behaviour of Stack, which seems to be the expectation of
45+
most people. Of course, you can use the `projectCoverageReport` and
46+
`coverageReport` functions to construct your own custom coverage
47+
reports (as detailed in the [coverage tutorial](../tutorials/coverage.md#custom)).
48+
49+
## Coverage reports
50+
51+
### Package reports
52+
53+
The coverage information generated will look something like this:
54+
55+
```bash
56+
/nix/store/...-my-project-0.1.0.0-coverage-report/
57+
└── share
58+
└── hpc
59+
└── vanilla
60+
├── html
61+
│   └── my-library-0.1.0.0
62+
│   ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
63+
│ │ ├── My.Lib.Config.hs.html
64+
│ │ ├── My.Lib.Types.hs.html
65+
│ │ └── My.Lib.Util.hs.html
66+
│   ├── hpc_index_alt.html
67+
│   ├── hpc_index_exp.html
68+
│   ├── hpc_index_fun.html
69+
│   └── hpc_index.html
70+
├── mix
71+
│   └── my-library-0.1.0.0
72+
│   └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
73+
│      ├── My.Lib.Config.mix
74+
│      ├── My.Lib.Types.mix
75+
│      └── My.Lib.Util.mix
76+
└── tix
77+
└── my-library-0.1.0.0
78+
├── my-library-0.1.0.0.tix
79+
├── my-test-1
80+
│   └── my-test-1.tix
81+
└── unit-test
82+
└── unit-test.tix
83+
```
84+
85+
- The mix files are copied verbatim from the library built with
86+
coverage.
87+
- The tix files for each test are copied from the check run verbatim
88+
and are output to ".../tix/<libraryname>/<testname>/<testname>.tix".
89+
- The tix files for each library are generated by summing the tix
90+
files for each test, but excluding any test modules. This tix file
91+
is output to ".../tix/<libraryname>/<libraryname>.tix".
92+
- Test modules are determined by inspecting the plan for the project
93+
(i.e. for the project "my-project" and test-suite "my-test-1", the
94+
test modules are read from:
95+
`my-project.checks.my-test-1.config.modules`)
96+
- The hpc HTML reports for each library are generated from their
97+
respective tix files (i.e. the
98+
`share/hpc/vanilla/html/my-library-0.1.0.0` report is generated from
99+
the
100+
`share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix`
101+
file)
102+
103+
### Project-wide reports
104+
105+
The coverage information for an entire project will look something
106+
like this:
107+
108+
```bash
109+
/nix/store/...-coverage-report
110+
└── share
111+
└── hpc
112+
└── vanilla
113+
├── html
114+
│   ├── index.html
115+
│   ├── all
116+
│   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
117+
│ │   │ ├── My.Lib.Config.hs.html
118+
│ │   │ ├── My.Lib.Types.hs.html
119+
│ │   │ └── My.Lib.Util.hs.html
120+
│ │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
121+
│ │ │   ├── Other.Lib.A.hs.html
122+
│ │ │   └── Other.Lib.B.hs.html
123+
│   │   ├── hpc_index_alt.html
124+
│   │   ├── hpc_index_exp.html
125+
│   │   ├── hpc_index_fun.html
126+
│   │   └── hpc_index.html
127+
│   ├── my-library-0.1.0.0
128+
│   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
129+
│ │   │ ├── My.Lib.Config.hs.html
130+
│ │   │ ├── My.Lib.Types.hs.html
131+
│ │   │ └── My.Lib.Util.hs.html
132+
│   │   ├── hpc_index_alt.html
133+
│   │   ├── hpc_index_exp.html
134+
│   │   ├── hpc_index_fun.html
135+
│   │   └── hpc_index.html
136+
│ └── other-libray-0.1.0.0
137+
│ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
138+
│ │   ├── Other.Lib.A.hs.html
139+
│ │   └── Other.Lib.B.hs.html
140+
│ ├── hpc_index_alt.html
141+
│ ├── hpc_index_exp.html
142+
│ ├── hpc_index_fun.html
143+
│ └── hpc_index.html
144+
├── mix
145+
│   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
146+
│   │   ├── My.Lib.Config.mix
147+
│   │   ├── My.Lib.Types.mix
148+
│   │   └── My.Lib.Util.mix
149+
│   └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
150+
│   ├── Other.Lib.A.mix
151+
│   └── Other.Lib.B.mix
152+
└── tix
153+
├── all
154+
│   └── all.tix
155+
├── my-library-0.1.0.0
156+
│ ├── my-library-0.1.0.0.tix
157+
│ ├── my-test-1
158+
│ │   └── my-test-1.tix
159+
│ └── unit-test
160+
│ └── unit-test.tix
161+
└── another-library-0.1.0.0
162+
├── another-library-0.1.0.0.tix
163+
├── my-test-2
164+
│   └── my-test-2.tix
165+
└── unit-test
166+
└── unit-test.tix
167+
```
168+
169+
All of the coverage information is copied verbatim from the coverage
170+
reports for each of the constituent packages. A few additions are
171+
made:
172+
- `tix/all/all.tix` is generated from the union of all the library
173+
tix files.
174+
- We use this file when generating coverage reports for
175+
"coveralls.io".
176+
- An index page (`html/index.html`) is generated which links to the
177+
HTML coverage reports of the constituent packages.
178+
- A synthetic HTML report is generated from the `tix/all/all.tix`
179+
file. This shows the union of all the coverage information
180+
generated by each constituent coverage report.

‎docs/tutorials/coverage.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Coverage
2+
3+
haskell.nix can generate coverage information for your package or
4+
project using Cabal's inbuilt hpc support.
5+
6+
## Prerequisites
7+
8+
To get a sensible coverage report, you need to enable coverage on each
9+
of the packages of your project:
10+
11+
```nix
12+
pkgs.haskell-nix.project {
13+
src = pkgs.haskell-nix.haskellLib.cleanGit {
14+
name = "haskell-nix-project";
15+
src = ./.;
16+
};
17+
compiler-nix-name = "ghc884";
18+
19+
modules = [{
20+
packages.$pkg.components.library.doCoverage = true;
21+
}];
22+
}
23+
```
24+
25+
If you would like to make coverage optional, add an argument to your nix expression:
26+
27+
```nix
28+
{ withCoverage ? false }:
29+
30+
pkgs.haskell-nix.project {
31+
src = pkgs.haskell-nix.haskellLib.cleanGit {
32+
name = "haskell-nix-project";
33+
src = ./.;
34+
};
35+
compiler-nix-name = "ghc884";
36+
37+
modules = pkgs.lib.optional withCoverage [{
38+
packages.$pkg.components.library.doCoverage = true;
39+
}];
40+
}
41+
```
42+
43+
## Per-package
44+
45+
```bash
46+
nix-build default.nix -A "projectWithCoverage.$pkg.coverageReport"
47+
```
48+
49+
This will generate a coverage report for the package you requested.
50+
All tests that are enabled (configured with `doCheck == true`) are
51+
included in the coverage report.
52+
53+
See the [developer coverage docs](../dev/coverage.md#package-reports) for more information.
54+
55+
## Project-wide
56+
57+
```bash
58+
nix-build default.nix -A "projectWithCoverage.projectCoverageReport"
59+
```
60+
61+
This will generate a coverage report for all the local packages in
62+
your project.
63+
64+
See the [developer coverage docs](../dev/coverage.md#project-wide-reports) for more information.
65+
66+
## Custom
67+
68+
By default, the behaviour of the `coverageReport` attribute is to
69+
generate a coverage report that describes how that package affects the
70+
coverage of all local packages (including itself) in the project.
71+
72+
The default behaviour of `projectCoverageReport` is to sum the
73+
default coverage reports (produced by the above process) of all local
74+
packages in the project.
75+
76+
You can modify this behaviour by using the `coverageReport` and
77+
`projectCoverageReport` functions found in the haskell.nix library:
78+
79+
```nix
80+
let
81+
inherit (pkgs.haskell-nix) haskellLib;
82+
83+
project = haskellLib.project {
84+
src = pkgs.haskell-nix.haskellLib.cleanGit {
85+
name = "haskell-nix-project";
86+
src = ./.;
87+
};
88+
compiler-nix-name = "ghc884";
89+
90+
modules = [{
91+
packages.$pkgA.components.library.doCoverage = true;
92+
packages.$pkgB.components.library.doCoverage = true;
93+
}];
94+
};
95+
96+
# Generate a coverage report for $pkgA that only includes the
97+
# unit-test check and only shows coverage information for $pkgA, not
98+
# $pkgB.
99+
custom$pkgACoverageReport = haskellLib.coverageReport rec {
100+
name = "$pkgA-unit-tests-only"
101+
inherit (project.$pkgA.components) library;
102+
checks = [project.$pkgA.components.checks.unit-test];
103+
# Note that this is the default value of the "mixLibraries"
104+
# argument and so this line isn't really necessary.
105+
mixLibraries = [project.$pkgA.components.library];
106+
};
107+
108+
custom$pkgBCoverageReport = haskellLib.coverageReport rec {
109+
name = "$pkgB-unit-tests-only"
110+
inherit (project.$pkgB.components) library;
111+
checks = [project.$pkgB.components.checks.unit-test];
112+
mixLibraries = [project.$pkgB.components.library];
113+
};
114+
115+
# Generate a project coverage report that only includes the unit
116+
# tests of the project, and only shows how each unit test effects
117+
# the coverage of it's package, and not other packages in the
118+
# project.
119+
allUnitTestsProjectReport = haskellLib.projectCoverageReport [custom$pkgACoverageReport custom$pkgBCoverageReport];
120+
in {
121+
inherit project custom$pkgACoverageReport custom$pkgBCoverageReport allUnitTestsProjectCoverageReport;
122+
}
123+
```

‎lib/check.nix

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ in stdenv.mkDerivation ({
1515
src = drv.source or (srcOnly drv);
1616

1717
passthru = {
18-
inherit (drv) identifier config configFiles executableToolDepends cleanSrc env;
18+
inherit (drv) identifier config configFiles executableToolDepends cleanSrc env exeName;
1919
};
2020

2121
inherit (drv) meta LANG LC_ALL buildInputs nativeBuildInputs;
@@ -27,11 +27,14 @@ in stdenv.mkDerivation ({
2727
# If doCheck or doCrossCheck are false we may still build this
2828
# component and we want it to quietly succeed.
2929
buildPhase = ''
30-
touch $out
30+
mkdir $out
3131
3232
runHook preCheck
3333
34-
${toString component.testWrapper} ${drv}/bin/${drv.exeName} ${lib.concatStringsSep " " component.testFlags} | tee $out
34+
${toString component.testWrapper} ${drv}/bin/${drv.exeName} ${lib.concatStringsSep " " component.testFlags} | tee $out/test-stdout
35+
36+
# Copy over tix files, if they exist
37+
find . -iname '${drv.exeName}.tix' -exec mkdir -p $out/share/hpc/vanilla/tix/${drv.exeName} \; -exec cp {} $out/share/hpc/vanilla/tix/${drv.exeName}/ \;
3538
3639
runHook postCheck
3740
'';

0 commit comments

Comments
 (0)