First iteration of coverage reports #762

sevanspowell · 2020-07-13T08:52:43Z

First attempt at adding support for coverage reports to haskell.nix.
doCoverage needs to be enabled on each component you want coverage for.
The exclusion of test modules is missing, I don't know yet how I'm
going to do that. Any advice would be appreciated. I know the plan
contains the test modules, but I'm not sure how to get access to
the plan.
You can then build with nix-build <component>.coverageReport.
We also need to add support for a project-wide coverage report, not
just per-component.
This does not upload to coveralls yet.

michaelpj · 2020-07-13T09:17:16Z

For configuring this: cabal.project has a coverage flag. Normally we try and get this sort of thing from cabal if we can - could we do that here?

rvl

Looking good so far...

lib/check.nix

lib/cover.nix

modules/plan.nix

angerman · 2020-08-03T01:42:56Z

I'm not sure what the purpose of the cq package is? If you just want to get modules for a cabal component, you could reuse nix-tools:cabal-to-nix which can include all the cabal file information and turn them into a nix expression (https://github.com/input-output-hk/nix-tools/blob/2d817aa6fc5ac3917400e88f205c934107ad3fe9/lib/Cabal2Nix.hs#L269-L278). We don't usually do this, as it increases the hackage.nix closure, but it can provide the information.

sevanspowell · 2020-08-06T01:50:55Z

I'm not sure what the purpose of the cq package is? If you just want to get modules for a cabal component, you could reuse nix-tools:cabal-to-nix which can include all the cabal file information and turn them into a nix expression (https://github.com/input-output-hk/nix-tools/blob/2d817aa6fc5ac3917400e88f205c934107ad3fe9/lib/Cabal2Nix.hs#L269-L278). We don't usually do this, as it increases the hackage.nix closure, but it can provide the information.

Thanks @angerman. This is exactly the kind of thing I was looking for. I whipped up the cq tool in half an hour to do what I needed, but I was waiting for someone to tell me I don't need it.

rvl · 2020-08-13T06:04:10Z

This looks pretty awesome - when can we use it?

Could there be a Haskell.nix test case for the coverage report of a single component?
I would remove the project-wide coverage stuff and do that in the next PR.
Likewise excluding tests from coverage.
Uploading to coveralls.io should not be done from nix. We may be able to make an external script to do it.

sevanspowell · 2020-08-14T02:46:51Z

This looks pretty awesome - when can we use it?

* Could there be a Haskell.nix test case for the coverage report of a single component?

* I would remove the project-wide coverage stuff and do that in the next PR.

* Likewise excluding tests from coverage.

* Uploading to coveralls.io should not be done from nix. We may be able to make an external script to do it.

👍 We definitely need a test case
I have done the project-wide coverage stuff already. Was there a reason to remove it other than time constraints?
I have also done the excluding tests from coverage already.
Totally agree. That was part of what led me down this path in the
first place. We were running stack-hpc-coveralls inside a nix-shell. Here's how I see it being used now:
https://github.com/input-output-hk/cardano-shell/compare/feature/SRE-90-use-cabal-for-coverage?expand=1
Nix is used to generate the script which invokes hpc-coveralls (mostly for
convenience/usability), then that script is executed by the CI machine outside of a nix sandbox. The idea being that the generation of the script is actually reproducible, but executing the script is not.

sevanspowell · 2020-08-14T02:47:03Z

Just working on doco, I'll have this out soon.

sevanspowell · 2020-08-14T03:00:47Z

Hey all :)

I'm re-requesting review because the core parts have stabilized and I'd really like your input on how it looks.

The documentation I've written might be helpful for understanding how the pieces fit together.

I've had separate discussions with a few of you out-of-band about what policy we should adopt towards enabling coverage. Ideally we'd like to get the "do coverage or not" option from the cabal.project/cabal side. Unfortunately, that attribute isn't available to us in the plan.json yet, see haskell/cabal#6343 (comment).

So until that is addressed, I'm going for: "if the user asks for a coverage report, force it on". Unfortunately, I'm not quite there yet :( I've outlined the key deficiency in the doco:

It is currently required that you enable coverage for each library you
want coverage for prior to attempting to generate a coverage report. I
hope to fix this before merging this PR:

  haskell-nix.cabalProject ({
    src = pkgs.haskell-nix.haskellLib.cleanGit {
      name = "haskell-nix-project";
      src = ./.;
    };
    modules = [
      {
        packages.package-1.components.library.doCoverage = true;
        packages.package-2.components.library.doCoverage = true;
      }
    ];
  });

I believe this deficiency exists because even though I do this:

  buildWithCoverage = builtins.map (d: d.covered);

  libraryCovered    = library.covered;
  testsWithCoverage = buildWithCoverage tests;

That does not enable coverage on the library dependency that already exists within each test. So that libraryCovered attribute might get me some .mix files, but it does not include the library in the coverage run the tests do, because as far as the tests are concerned, none of their dependencies have coverage enabled.

I'm not sure how to fix this, any insight would be amazing!

Until then I'll work on doco and a test as much as I can between other things.

michaelpj · 2020-08-14T11:07:13Z

It is currently required that you enable coverage for each library you
want coverage for prior to attempting to generate a coverage report. I
hope to fix this before merging this PR:

Does it matter if we have coverage enabled permanently? i.e. is it bad if our production builds of things use a library which was built with coverage available?

If not, then here's an alternative suggestion: require users to enable coverage explicitly, and assert in the coverage report generation that it's on. Then, yes, you have to set it manually, but at least it tells you what to do.

I'm a bit concerned about automatically turning it on in that it may require us to rebuild things unnecessarily (but I guess at least this will only be the libraries in the current project, not dependencies, right?).

michaelpj

Generally looks good, great doc!

michaelpj · 2020-08-14T11:09:44Z

docs/dev/coverage.md

+
+- The mix files are copied verbatim from the builds with coverage.
+- The tix files for each test are copied from the check run verbatim.
+- The tix files for each library are generated by summing the tix


Is this a standard thing to do? I would have hoped that maybe there were existing tools for working with tix files out there? Maybe we should consider writing one, and then just calling it? I'm a little uncomfortable having sophisticated logic like this just written in bash in a random Nix builder.

This applies to the other clever coverage file handling you mention elsewhere.

docs/dev/coverage.md

michaelpj · 2020-08-14T11:11:39Z

docs/dev/coverage.md

@@ -0,0 +1,173 @@
+# Developer Coverage Overview


docs/tutorials/coverage.md

lib/cover.nix

lib/default.nix

michaelpj · 2020-08-14T11:25:01Z

overlays/haskell.nix

+                  inherit (callProjectResults) index-state;
+                };
+            in project // {
+                projectCoverageReport = haskellLib.projectCoverageReport (haskellLib.selectProjectPackages project.hsPkgs);


I'd be tempted to make people construct this call themselves, i.e. document projectCoverageReport and tell people to just pass in the packages they want coverage for. Maybe that's my inner Haskeller preferring functions over magic attributes, but it's harder to work out what to do in Nix land.

Addressed these here: 29ac893

michaelpj · 2020-08-14T11:29:02Z

overlays/haskell.nix

                      components = final.lib.mapAttrs (n: v:
                        if n == "library" || n == "all"
                          then v // { inherit project package; }
                          else final.lib.mapAttrs (_: c: c // { inherit project package; }) v
                      ) package'.components;
                      inherit project;
+
+                      coverageReport = haskellLib.coverageReport {


Ditto? although this one is more complex...

Addressed these here: 29ac893

rvl

It looks very good, and I generally second Michael's review comments.

I'm happy to see a test case as well.

I did suggest removing any partially complete features, and doing them in another PR. But if it's all done then great.

sevanspowell · 2020-08-18T02:32:35Z

It is currently required that you enable coverage for each library you
want coverage for prior to attempting to generate a coverage report. I
hope to fix this before merging this PR:

Does it matter if we have coverage enabled permanently? i.e. is it bad if our production builds of things use a library which was built with coverage available?

If not, then here's an alternative suggestion: require users to enable coverage explicitly, and assert in the coverage report generation that it's on. Then, yes, you have to set it manually, but at least it tells you what to do.

I'm a bit concerned about automatically turning it on in that it may require us to rebuild things unnecessarily (but I guess at least this will only be the libraries in the current project, not dependencies, right?).

👍 It seems symptomatic of a design issue that I have to turn on coverage for the entire project permanently to generate the coverage report. I actually want the project to stay the same, but pass some of the packages to a coverage function and have it handle making a copy of those packages with coverage enabled. You will end up compiling the package twice, once with coverage, and once without, but that's only when you exercise a code path involving the coverage report. I doubt we'll make it a required build. At this stage I only foresee it being used by the CI machines in a build step to generate a report which it then sends to coveralls.io, or by a user locally, to ensure their changes maintain coverage.

I'll pursue turning the projectCoverage and coverageReport attributes into functions.

sevanspowell · 2020-08-18T02:39:09Z

It looks very good, and I generally second Michael's review comments.

I'm happy to see a test case as well.

I did suggest removing any partially complete features, and doing them in another PR. But if it's all done then great.

Thanks :) Here's my first attempt at a test: ee6f069.

sevanspowell · 2020-08-19T04:30:37Z

Just re-surfacing this comment as it doesn't appear here:

Your observation about the problem with covered makes me wonder if this works. What happens if I do someTestComponent.profiled? Do I get the test built with profiling, but not the library? thinking

Yeah, so that's a really good question.

I've figured out the issue I was encountering with covered so I understand this a bit more. The key step I was missing was this line in the comp-builder.nix file:

haskell.nix/builder/comp-builder.nix

Line 76 in e340658

inherit component fullName flags needsProfiling;

So when you build someTestComponent.profiled, the component builder enables library profiling on that derivation. That's enough to profile the test (and maybe it's immediate dependencies? I don't know how "--enable-library-profiling" works). But it's important that the request for profiling is then fed to make-config-files.nix, which will then call the profiled attribute on the dependencies of the test component, and this will recurse down the entire dependency tree:

haskell.nix/builder/make-config-files.nix

Line 50 in e340658

(if needsProfiling then (x: map (p: p.profiled or p) x) else x: x)

I needed to do something very similar with the covered attribute, but I only wanted it to apply to the immediate library dependency of the test component. The user doesn't care if their test fully covers the bytestring or transformers library. I've got this working here: f3a6bdc, but it's quite inelegant. Nevertheless, it will enable me to improve the UX of this PR a bit.

cc @michaelpj

sevanspowell · 2020-08-19T04:43:15Z

Ditto:

This isn't quite strong enough: you might also need to build any internal libraries with coverage. Generally, I think this an argument for requiring the user to specify the coverage options, then it's on them to get this right!

I've fixed this in f3a6bdc, but it's all "convenience" and doesn't ask the user to do much. I'm not sure how to approach asking them to do more, although I agree it'd be more elegant:

If I ask the user to enable coverage in their project like this:

  project = pkgs.haskell-nix.project {
    # 'cleanGit' cleans a source directory based on the files known by git
    src = pkgs.haskell-nix.haskellLib.cleanGit {
      name = "cardano-shell"
      src = ./.;
    };
    # For `cabal.project` based projects specify the GHC version to use.
    compiler-nix-name = "ghc864"; # Not used for `stack.yaml` based projects.

    modules = [{
      packages.cardano-launcher.components.library.doCoverage = true;
      packages.cardano-launcher.components.tests.cardano-launcher-test.doCoverage = true;
      
      packages.cardano-shell.components.library.doCoverage = true;
    }];
  };

I'm dooming that project to have coverage enabled for ALL builds from this point on.

I suppose I could ask the user to create a duplicate project for coverage builds, but I'm not sure about the elegance of that.

Maybe I want something like:

  project.withExtraModules({
      packages.cardano-launcher.components.library.doCoverage = true;
      packages.cardano-launcher.components.tests.cardano-launcher-test.doCoverage = true;
      
      packages.cardano-shell.components.library.doCoverage = true;
  }).projectCoverageReport'

At the moment, the magic attributes nicely sidestep the above problem (albeit the magic is fairly brittle):

  project.projectCoverageReport'

sevanspowell · 2020-08-21T06:02:15Z

I think I've addressed every comment bar creating an external tool.

The coverage implementation is now significantly simpler, and I've documented the new usage here.

I will work on the external tool to simplify the Nix logic in each builder. I agree that it's a lot of logic in Nix. I'm not sure if it will buy us a lot, but we'll see.

rvl · 2020-09-07T05:15:19Z

Feel free to rebase on latest master, fix merge conflicts, and squash @sevanspowell.

sevanspowell · 2020-09-07T08:12:04Z

Thanks @rvl

I didn't end up creating an external tool to munge the coverage information. I tried to approach it a couple of ways, but ultimately it didn't buy us enough to justify maintaining a separate tool. So I've simplified the logic in each of the core coverage bash scripts to make things easier to understand and maintain.

michaelpj

LGTM! have not tested it, though

michaelpj · 2020-09-07T09:50:39Z

lib/check.nix


    runHook preCheck

-    ${toString component.testWrapper} ${drv}/bin/${drv.exeName} ${lib.concatStringsSep " " component.testFlags} | tee $out
+    ${toString component.testWrapper} ${drv}/bin/${drv.exeName} ${lib.concatStringsSep " " component.testFlags} | tee $out/test-stdout


👍 for test-stdout

michaelpj · 2020-09-07T09:51:07Z

lib/cover.nix

+  #   main-is: Spec.hs
+  # still generates a "Main.mix" file with the contents:
+  #   Mix "Spec.hs" ...
+  # Hence we can hardcode the name "Main" here.


michaelpj · 2020-09-07T09:51:32Z

lib/cover.nix

+  testsAsList = lib.attrValues tests;
+  checks      = builtins.map (d: haskellLib.check d) testsAsList;
+
+  # Exclude test modules from tix file. The Main module is


Why do we exclude test modules?

Added a comment: https://github.com/input-output-hk/haskell.nix/pull/762/files#diff-4bba04d96602cc233c6b24d553f66222R16

michaelpj · 2020-09-07T09:52:57Z

lib/cover.nix

+  mixSearchDirs = map (info: if info.searchSubDir == null then info.rootDir else info.rootDir + "/" + info.searchSubDir) mixInfo;
+
+in pkgs.runCommand (identifier + "-coverage-report")
+  { buildInputs = (with pkgs; [ ghc ]); }


Should we use the same GHC as the project uses here? Otherwise we might a) incur a dependency on another GHC, which can be expensive, and b) the behaviour of hpc might be different.

Sounds good, fixed: https://github.com/input-output-hk/haskell.nix/pull/762/files#diff-4bba04d96602cc233c6b24d553f66222R41

michaelpj · 2020-09-07T09:53:10Z

lib/cover-project.nix

+  </html>
+  '';
+in pkgs.runCommand "project-coverage-report"
+  { buildInputs = (with pkgs; [ghc]); }


Same comment about choice of GHC here.

Fixed: https://github.com/input-output-hk/haskell.nix/pull/762/files#diff-8ee3808821db3b9f819360e568c5004aR51

michaelpj · 2020-09-07T09:53:51Z

lib/default.nix

+  };
+
+  # Do coverage of a project
+  projectCoverageReport' = import ./cover-project.nix {


Why the ticks? there aren't unticked versions here that I can see?

There are the ones which are attributes on projects, but I think that's clear enough. We use the ticks when they're actually attributes of the same attribute set.

Fixed: https://github.com/input-output-hk/haskell.nix/pull/762/files#diff-a013e20924130857c649dd17226282ffR235

sevanspowell · 2020-09-08T03:54:33Z

Just debugging the tests, I can't get them to build with the new "choose a ghc" test infra yet.

sevanspowell · 2020-09-08T04:03:30Z

Fixed I think.

rvl · 2020-09-08T13:51:17Z

@sevanspowell I am trying this branch and the coverage generation is working pretty well 👌.

But projectCoverageReport does not work with how we run tests in cardano-wallet. We have a recursive set of checks derivations. Ideally these could be passed directly as an argument of projectCoverageReport.

Edit: also I have found that the coverageReport attribute is missing from packages.

rvl

Looks awesome, thanks @sevanspowell.

As it is, the projectCoverageReport function is not quite suitable for cardano-wallet, but it will work for other projects, so let's merge.

You will need to squash then rebase again of course. What is going on with the CI fails - are they also happening on master?

Long term, I think we should either modify hpc markup so that it can produce multi-package HTML reports (like stack hpc does), or make a standalone tool.

- Added the ability to generate coverage reports for packages and projects. - Outputs mix and tix information, as well as a HTML report. - Added the "doCoverage" module option that allows users to choose packages to enable coverage for. - Added a "doCoverage" flag to the component builder that outputs HPC information when coverage is enabled. - Added the "overrideModules" library function to make it more ergonomic fo users to enable coverage on existing projects. - Modified the "check" builder to also output ".tix" files (if they exist). This information is required to generate the coverage report. - Added a test for coverage.

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io> Co-authored-by: Samuel Evans-Powell <mail@sevanspowell.net>

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> 2223: Enable 100 wallet scenario in latency benchmark r=Anviking a=Anviking # Issue Number ADP-469, ADP-473 # Overview - [x] Simply enable `Latencies for 100 fixture wallets` - [x] Delete all wallets in between all latency scenarios - [x] Rename DSL helper `tearDown` to `deleteAllWallets`, and re-use it more # Comments ``` $ stack bench cardano-wallet Benchmark latency: RUNNING... Non-cached run getNetworkInfo - 20.7 ms Latencies for 2 fixture wallets scenario listWallets - 1.4 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 2.2 ms postTransactionFee - 86.4 ms listStakePools - 1.2 ms getNetworkInfo - 0.0 ms Latencies for 10 fixture wallets scenario listWallets - 4.8 ms getWallet - 0.4 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 2.0 ms postTransactionFee - 93.2 ms listStakePools - 1.3 ms getNetworkInfo - 0.0 ms Latencies for 100 fixture wallets listWallets - 59.9 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.2 ms listTransactions - 2.6 ms postTransactionFee - 139.2 ms listStakePools - 1.1 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 10 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 0.9 ms listTransactions - 3.8 ms postTransactionFee - 104.2 ms listStakePools - 1.3 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 20 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 5.5 ms postTransactionFee - 97.0 ms listStakePools - 1.2 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 100 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 14.5 ms postTransactionFee - 95.1 ms listStakePools - 1.7 ms getNetworkInfo - 0.1 ms Latencies for 10 fixture wallets with 10 txs scenario listWallets - 4.4 ms getWallet - 0.5 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 4.3 ms postTransactionFee - 106.6 ms listStakePools - 1.3 ms getNetworkInfo - 0.0 ms Latencies for 10 fixture wallets with 20 txs scenario listWallets - 4.2 ms getWallet - 0.4 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 4.5 ms postTransactionFee - 106.6 ms listStakePools - 1.3 ms getNetworkInfo - 0.1 ms Latencies for 10 fixture wallets with 100 txs scenario listWallets - 4.5 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 0.9 ms listTransactions - 15.8 ms postTransactionFee - 96.8 ms listStakePools - 1.6 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 100 utxos scenario listWallets - 1.7 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 12.2 ms postTransactionFee - 95.9 ms listStakePools - 1.4 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 200 utxos scenario listWallets - 1.7 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 25.1 ms postTransactionFee - 96.6 ms listStakePools - 1.4 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 500 utxos scenario listWallets - 1.6 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 69.7 ms postTransactionFee - 87.9 ms listStakePools - 1.5 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 1000 utxos scenario ```   Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io> Co-authored-by: Samuel Evans-Powell <mail@sevanspowell.net> Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io> Co-authored-by: Samuel Evans-Powell <mail@sevanspowell.net>

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

2124: Add haskell program coverage reports to Hydra tests r=KtorZ a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> 2213: ADP-455: return unsigned delegation certificate r=KtorZ a=hasufell # Issue Number #2200 # Overview - [x] Added Api types - [x] Refactored some of the Transaction layer (still WIP) - [x] Factored out certificate creation from `joinStakePool` - [x] Added handler for joining stake pool - [x] Add handler for quitting stake pool - [x] serialization of certificates - [x] lots of cleanup - [x] fix jormungandr - [x] fix tests - [ ] add tests # Comments   Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io> Co-authored-by: Samuel Evans-Powell <mail@sevanspowell.net> Co-authored-by: Julian Ospald <julian.ospald@iohk.io> Co-authored-by: KtorZ <matthias.benkort@gmail.com> Co-authored-by: Matthias Benkort <5680256+KtorZ@users.noreply.github.com>

- Added the ability to generate coverage reports for packages and projects. - Outputs mix and tix information, as well as a HTML report. - Added the "doCoverage" module option that allows users to choose packages to enable coverage for. - Added a "doCoverage" flag to the component builder that outputs HPC information when coverage is enabled. - Added the "overrideModules" library function to make it more ergonomic fo users to enable coverage on existing projects. - Modified the "check" builder to also output ".tix" files (if they exist). This information is required to generate the coverage report. - Added a test for coverage.

sevanspowell requested review from angerman, hamishmack and rvl July 13, 2020 08:52

sevanspowell self-assigned this Jul 13, 2020

rvl reviewed Jul 14, 2020

View reviewed changes

lib/check.nix Outdated Show resolved Hide resolved

michaelpj reviewed Jul 14, 2020

View reviewed changes

sevanspowell requested review from rvl and michaelpj August 14, 2020 03:00

sevanspowell marked this pull request as ready for review August 14, 2020 03:07

michaelpj reviewed Aug 14, 2020

View reviewed changes

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from 01d061c to ee6f069 Compare August 17, 2020 03:52

rvl reviewed Aug 18, 2020

View reviewed changes

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch 4 times, most recently from debc841 to f3a6bdc Compare August 19, 2020 02:34

sevanspowell requested review from michaelpj and rvl August 19, 2020 04:45

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from 8312f62 to b1ba5a6 Compare September 7, 2020 07:42

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from 4e07cb0 to 73a1ddc Compare September 7, 2020 08:23

michaelpj approved these changes Sep 7, 2020

View reviewed changes

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from 2ac9f01 to b5cbbf9 Compare September 8, 2020 03:52

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from b5cbbf9 to ce74990 Compare September 8, 2020 04:03

rvl mentioned this pull request Sep 9, 2020

Add haskell program coverage reports to Hydra tests cardano-foundation/cardano-wallet#2124

Merged

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from bbdac76 to 69902c7 Compare September 14, 2020 04:21

sevanspowell mentioned this pull request Sep 14, 2020

Use haskell.nix for coverage input-output-hk/cardano-shell#379

Merged

rvl approved these changes Sep 25, 2020

View reviewed changes

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch 2 times, most recently from 7a4e069 to 4807434 Compare September 28, 2020 08:19

sevanspowell force-pushed the feature/SRE-125-add-coverage-support branch from 54d6da6 to 0e3727e Compare September 29, 2020 01:27

sevanspowell merged commit 48b8674 into master Sep 29, 2020

iohk-bors bot deleted the feature/SRE-125-add-coverage-support branch September 29, 2020 03:56

rvl added a commit to cardano-foundation/cardano-wallet that referenced this pull request Oct 1, 2020

Bump haskell.nix to latest master

d81ff3e

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

rvl added a commit to cardano-foundation/cardano-wallet that referenced this pull request Oct 6, 2020

Bump haskell.nix to latest master

81fbeef

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

rvl added a commit to cardano-foundation/cardano-wallet that referenced this pull request Oct 9, 2020

Bump haskell.nix to latest master

1ee6714

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

KtorZ pushed a commit to cardano-foundation/cardano-wallet that referenced this pull request Oct 16, 2020

Bump haskell.nix to latest master

87666d5

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

First iteration of coverage reports #762

First iteration of coverage reports #762

Uh oh!

Conversation

sevanspowell commented Jul 13, 2020

Uh oh!

michaelpj commented Jul 13, 2020

Uh oh!

rvl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

angerman commented Aug 3, 2020

Uh oh!

sevanspowell commented Aug 6, 2020

Uh oh!

rvl commented Aug 13, 2020

Uh oh!

sevanspowell commented Aug 14, 2020

Uh oh!

sevanspowell commented Aug 14, 2020

Uh oh!

sevanspowell commented Aug 14, 2020

Uh oh!

michaelpj commented Aug 14, 2020

Uh oh!

michaelpj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvl left a comment

Choose a reason for hiding this comment

Uh oh!

sevanspowell commented Aug 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sevanspowell commented Aug 18, 2020

Uh oh!

sevanspowell commented Aug 19, 2020

Uh oh!

sevanspowell commented Aug 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sevanspowell commented Aug 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rvl commented Sep 7, 2020

Uh oh!

sevanspowell commented Sep 7, 2020

Uh oh!

michaelpj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sevanspowell commented Aug 18, 2020 •

edited

Loading

sevanspowell commented Aug 19, 2020 •

edited

Loading

sevanspowell commented Aug 21, 2020 •

edited

Loading

rvl commented Sep 8, 2020 •

edited

Loading