First iteration of coverage reports #762
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| # Developer Coverage Overview | ||
|
|
||
| ## Building | ||
|
|
||
| The implementation of coverage starts with the "doCoverage" flag on | ||
| the builder in `comp-builder.nix`. The doCoverage flag enables and | ||
| disables the Cabal coverage flag and copies any generated coverage | ||
| data to "$out/share/hpc". | ||
|
|
||
| ### Mix and tix files | ||
|
|
||
| The coverage information for any derivation consists of "mix" and | ||
| "tix" files. | ||
|
|
||
| Mix files record static information about a source file and are | ||
| generated at build time. They primarily contain a path to the source | ||
| file and information about expressions and regions of the source file, | ||
| which are later referenced by tix files. | ||
|
|
||
| Tix files contain dynamic information about a test run, recording when | ||
| a portion of a source file is touched by a test. These are generated | ||
| when the test is run. | ||
|
|
||
| ### Multiple local packages | ||
|
|
||
| In the context of multiple local packages, there are a few types of | ||
| coverage we might be interested in: | ||
| - How well does the tests for this package cover the package library? | ||
| - How well does the tests for this package cover the libraries of | ||
| other packages in this project? | ||
| - Both of the above. | ||
|
|
||
| To facilitate expressing any of these classifications of coverage, the | ||
| `lib/cover.nix` function provides the `mixLibraries` argument. If | ||
| you're just interested in how the tests cover the package library, you | ||
| provide that library as an argument to `mixLibraries`. If you're | ||
| interested in how the tests also cover other local packages in the | ||
| project, you can also provide those libraries as arguments to | ||
| mixLibraries. | ||
|
|
||
| The `projectCoverageReport` and `coverageReport` attributes that are | ||
| provided by default on projects and packages respectively provide | ||
| coverage information for *all* local packages in the project. This is | ||
| to mimic the behaviour of Stack, which seems to be the expectation of | ||
| most people. Of course, you can use the `projectCoverageReport` and | ||
| `coverageReport` functions to construct your own custom coverage | ||
| reports (as detailed in the [coverage tutorial](../tutorials/coverage.md#custom)). | ||
|
|
||
| ## Coverage reports | ||
|
|
||
| ### Package reports | ||
|
|
||
| The coverage information generated will look something like this: | ||
|
|
||
| ```bash | ||
| /nix/store/...-my-project-0.1.0.0-coverage-report/ | ||
| └── share | ||
| └── hpc | ||
| └── vanilla | ||
| ├── html | ||
| │ └── my-library-0.1.0.0 | ||
| │ ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf | ||
| │ │ ├── My.Lib.Config.hs.html | ||
| │ │ ├── My.Lib.Types.hs.html | ||
| │ │ └── My.Lib.Util.hs.html | ||
| │ ├── hpc_index_alt.html | ||
| │ ├── hpc_index_exp.html | ||
| │ ├── hpc_index_fun.html | ||
| │ └── hpc_index.html | ||
| ├── mix | ||
| │ └── my-library-0.1.0.0 | ||
| │ └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf | ||
| │ ├── My.Lib.Config.mix | ||
| │ ├── My.Lib.Types.mix | ||
| │ └── My.Lib.Util.mix | ||
| └── tix | ||
| └── my-library-0.1.0.0 | ||
| ├── my-library-0.1.0.0.tix | ||
| ├── my-test-1 | ||
| │ └── my-test-1.tix | ||
| └── unit-test | ||
| └── unit-test.tix | ||
| ``` | ||
|
|
||
| - The mix files are copied verbatim from the library built with | ||
| coverage. | ||
| - The tix files for each test are copied from the check run verbatim | ||
| and are output to ".../tix/<libraryname>/<testname>/<testname>.tix". | ||
| - The tix files for each library are generated by summing the tix | ||
|
There was a problem hiding this comment. Is this a standard thing to do? I would have hoped that maybe there were existing tools for working with tix files out there? Maybe we should consider writing one, and then just calling it? I'm a little uncomfortable having sophisticated logic like this just written in bash in a random Nix builder. There was a problem hiding this comment. This applies to the other clever coverage file handling you mention elsewhere. |
||
| files for each test, but excluding any test modules. This tix file | ||
| is output to ".../tix/<libraryname>/<libraryname>.tix". | ||
| - Test modules are determined by inspecting the plan for the project | ||
| (i.e. for the project "my-project" and test-suite "my-test-1", the | ||
| test modules are read from: | ||
| `my-project.checks.my-test-1.config.modules`) | ||
| - The hpc HTML reports for each library are generated from their | ||
| respective tix files (i.e. the | ||
| `share/hpc/vanilla/html/my-library-0.1.0.0` report is generated from | ||
| the | ||
| `share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix` | ||
| file) | ||
|
|
||
| ### Project-wide reports | ||
|
|
||
| The coverage information for an entire project will look something | ||
| like this: | ||
|
|
||
| ```bash | ||
| /nix/store/...-coverage-report | ||
| └── share | ||
| └── hpc | ||
| └── vanilla | ||
| ├── html | ||
| │ ├── index.html | ||
| │ ├── all | ||
| │ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV | ||
| │ │ │ ├── My.Lib.Config.hs.html | ||
| │ │ │ ├── My.Lib.Types.hs.html | ||
| │ │ │ └── My.Lib.Util.hs.html | ||
| │ │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf | ||
| │ │ │ ├── Other.Lib.A.hs.html | ||
| │ │ │ └── Other.Lib.B.hs.html | ||
| │ │ ├── hpc_index_alt.html | ||
| │ │ ├── hpc_index_exp.html | ||
| │ │ ├── hpc_index_fun.html | ||
| │ │ └── hpc_index.html | ||
| │ ├── my-library-0.1.0.0 | ||
| │ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV | ||
| │ │ │ ├── My.Lib.Config.hs.html | ||
| │ │ │ ├── My.Lib.Types.hs.html | ||
| │ │ │ └── My.Lib.Util.hs.html | ||
| │ │ ├── hpc_index_alt.html | ||
| │ │ ├── hpc_index_exp.html | ||
| │ │ ├── hpc_index_fun.html | ||
| │ │ └── hpc_index.html | ||
| │ └── other-libray-0.1.0.0 | ||
| │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf | ||
| │ │ ├── Other.Lib.A.hs.html | ||
| │ │ └── Other.Lib.B.hs.html | ||
| │ ├── hpc_index_alt.html | ||
| │ ├── hpc_index_exp.html | ||
| │ ├── hpc_index_fun.html | ||
| │ └── hpc_index.html | ||
| ├── mix | ||
| │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV | ||
| │ │ ├── My.Lib.Config.mix | ||
| │ │ ├── My.Lib.Types.mix | ||
| │ │ └── My.Lib.Util.mix | ||
| │ └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf | ||
| │ ├── Other.Lib.A.mix | ||
| │ └── Other.Lib.B.mix | ||
| └── tix | ||
| ├── all | ||
| │ └── all.tix | ||
| ├── my-library-0.1.0.0 | ||
| │ ├── my-library-0.1.0.0.tix | ||
| │ ├── my-test-1 | ||
| │ │ └── my-test-1.tix | ||
| │ └── unit-test | ||
| │ └── unit-test.tix | ||
| └── another-library-0.1.0.0 | ||
| ├── another-library-0.1.0.0.tix | ||
| ├── my-test-2 | ||
| │ └── my-test-2.tix | ||
| └── unit-test | ||
| └── unit-test.tix | ||
| ``` | ||
|
|
||
| All of the coverage information is copied verbatim from the coverage | ||
| reports for each of the constituent packages. A few additions are | ||
| made: | ||
| - `tix/all/all.tix` is generated from the union of all the library | ||
| tix files. | ||
| - We use this file when generating coverage reports for | ||
| "coveralls.io". | ||
| - An index page (`html/index.html`) is generated which links to the | ||
| HTML coverage reports of the constituent packages. | ||
| - A synthetic HTML report is generated from the `tix/all/all.tix` | ||
| file. This shows the union of all the coverage information | ||
| generated by each constituent coverage report. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Coverage | ||
|
|
||
| haskell.nix can generate coverage information for your package or | ||
| project using Cabal's inbuilt hpc support. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| To get a sensible coverage report, you need to enable coverage on each | ||
| of the packages of your project: | ||
|
|
||
| ```nix | ||
| pkgs.haskell-nix.project { | ||
| src = pkgs.haskell-nix.haskellLib.cleanGit { | ||
| name = "haskell-nix-project"; | ||
| src = ./.; | ||
| }; | ||
| compiler-nix-name = "ghc884"; | ||
|
|
||
| modules = [{ | ||
| packages.$pkg.components.library.doCoverage = true; | ||
| }]; | ||
| } | ||
| ``` | ||
|
|
||
| If you would like to make coverage optional, add an argument to your nix expression: | ||
|
|
||
| ```nix | ||
| { withCoverage ? false }: | ||
|
|
||
| pkgs.haskell-nix.project { | ||
| src = pkgs.haskell-nix.haskellLib.cleanGit { | ||
| name = "haskell-nix-project"; | ||
| src = ./.; | ||
| }; | ||
| compiler-nix-name = "ghc884"; | ||
|
|
||
| modules = pkgs.lib.optional withCoverage [{ | ||
| packages.$pkg.components.library.doCoverage = true; | ||
| }]; | ||
| } | ||
| ``` | ||
|
|
||
| ## Per-package | ||
|
|
||
| ```bash | ||
| nix-build default.nix -A "projectWithCoverage.$pkg.coverageReport" | ||
| ``` | ||
|
|
||
| This will generate a coverage report for the package you requested. | ||
| All tests that are enabled (configured with `doCheck == true`) are | ||
| included in the coverage report. | ||
|
|
||
| See the [developer coverage docs](../dev/coverage.md#package-reports) for more information. | ||
|
|
||
| ## Project-wide | ||
|
|
||
| ```bash | ||
| nix-build default.nix -A "projectWithCoverage.projectCoverageReport" | ||
| ``` | ||
|
|
||
| This will generate a coverage report for all the local packages in | ||
| your project. | ||
|
|
||
| See the [developer coverage docs](../dev/coverage.md#project-wide-reports) for more information. | ||
|
|
||
| ## Custom | ||
|
|
||
| By default, the behaviour of the `coverageReport` attribute is to | ||
| generate a coverage report that describes how that package affects the | ||
| coverage of all local packages (including itself) in the project. | ||
|
|
||
| The default behaviour of `projectCoverageReport` is to sum the | ||
| default coverage reports (produced by the above process) of all local | ||
| packages in the project. | ||
|
|
||
| You can modify this behaviour by using the `coverageReport` and | ||
| `projectCoverageReport` functions found in the haskell.nix library: | ||
|
|
||
| ```nix | ||
| let | ||
| inherit (pkgs.haskell-nix) haskellLib; | ||
|
|
||
| project = haskellLib.project { | ||
| src = pkgs.haskell-nix.haskellLib.cleanGit { | ||
| name = "haskell-nix-project"; | ||
| src = ./.; | ||
| }; | ||
| compiler-nix-name = "ghc884"; | ||
|
|
||
| modules = [{ | ||
| packages.$pkgA.components.library.doCoverage = true; | ||
| packages.$pkgB.components.library.doCoverage = true; | ||
| }]; | ||
| }; | ||
|
|
||
| # Generate a coverage report for $pkgA that only includes the | ||
| # unit-test check and only shows coverage information for $pkgA, not | ||
| # $pkgB. | ||
| custom$pkgACoverageReport = haskellLib.coverageReport rec { | ||
| name = "$pkgA-unit-tests-only" | ||
| inherit (project.$pkgA.components) library; | ||
| checks = [project.$pkgA.components.checks.unit-test]; | ||
| # Note that this is the default value of the "mixLibraries" | ||
| # argument and so this line isn't really necessary. | ||
| mixLibraries = [project.$pkgA.components.library]; | ||
| }; | ||
|
|
||
| custom$pkgBCoverageReport = haskellLib.coverageReport rec { | ||
| name = "$pkgB-unit-tests-only" | ||
| inherit (project.$pkgB.components) library; | ||
| checks = [project.$pkgB.components.checks.unit-test]; | ||
| mixLibraries = [project.$pkgB.components.library]; | ||
| }; | ||
|
|
||
| # Generate a project coverage report that only includes the unit | ||
| # tests of the project, and only shows how each unit test effects | ||
| # the coverage of it's package, and not other packages in the | ||
| # project. | ||
| allUnitTestsProjectReport = haskellLib.projectCoverageReport [custom$pkgACoverageReport custom$pkgBCoverageReport]; | ||
| in { | ||
| inherit project custom$pkgACoverageReport custom$pkgBCoverageReport allUnitTestsProjectCoverageReport; | ||
| } | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great doc