[WIP] Azure: Publish code coverage results #4746

gerardroche · 2019-09-26T15:48:09Z

Add an Azure Publish Code Coverage Results task
Add make gcovr-html to generate a gcovr test coverage report in HTML
Add make gcovr-xml to generate a gcovr test coverage report in XML
Remove test target dependency from make lcov-html; run the two targets together instead: make test lcov-html.

See: https://externals.io/message/107113,
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops,
and #4759.

azure/configure.yml

nikic · 2019-09-30T10:24:49Z

It looks like the code coverage artefact is successfully published -- I can download it and browse it locally. However, during the coverage publish task we have:

##[warning]No coverage data found. Check the build errors/warnings for more details.

and we also don't see code coverage in the Azure Pipelines UI, something isn't working quite right yet. Not really obvious to me what that is though.

nikic · 2019-09-30T10:37:06Z

Azure Pipelines requires the Cobertura XML to be version 4. gcovr support for that was added in gcovr/gcovr@e8fdcee in version 3.4. Ubuntu 16.04 comes with gcovr 3.2 :(

It should be possible to fix this by installing gcovr from pip instead of apt-get.

gerardroche · 2019-09-30T11:44:11Z

It looks like the code coverage artefact is successfully published -- I can download it and browse it locally. However, during the coverage publish task we have:

##[warning]No coverage data found. Check the build errors/warnings for more details.

and we also don't see code coverage in the Azure Pipelines UI, something isn't working quite right yet. Not really obvious to me what that is though.

Yes. I'm having difficulties to get it working. It's being published as a build artefact but not appearing as a tab in the interface. There might be an option in azure to enable/disable it, or xml generated by gcovr is not compatible.

Azure Pipelines requires the Cobertura XML to be version 4. gcovr support for that was added in gcovr/gcovr@e8fdcee in version 3.4. Ubuntu 16.04 comes with gcovr 3.2 :(

It should be possible to fix this by installing gcovr from pip instead of apt-get.

Yes, I think this is the main problem. I'm using gcovr v3.4 locally which is obviously why I'm having no issues locally.

I was hoping to avoid using pip. Another option to gcovr is to use an lcov info file to cobertura convertor (https://github.com/eriwen/lcov-to-cobertura-xml), but it requires pip too.

nikic · 2019-09-30T13:21:39Z

azure/coverage_job.yml

+    steps:
+    - template: apt.yml
+      parameters:
+        packages: lcov


The only reason lcov is needed are the configure checks, right? We aren't actually using it anymore? If that's the case, I think we should adjust configure.ac to also check for gcovr and allow the build if one of them exists.

Yes, lcov can be removed.

I'm not familiar with configure.ac, was looking at it last night. I guess you should be able to build, but only generate a coverage report if one exists.

Currently there two ways to generate coverage: make lcov-html or gcovr-html depending on your preference. We could add make coverage or make code-coverage and default it to lcov or gcovr whichever one exists.

Yeah, I think it might make sense to just remove all those checks from configure.ac completely and make --enable-gcov only about building with the necessary instrumentation. We'd have to adjust the $(LTP) subst in the Makefile.gcov file as well though. I guess we would just add something like LTP ?= lcov to the top, but I have no idea if that's a GNU make-ism or not.

azure/coverage_job.yml

build/Makefile.gcov

nikic · 2019-09-30T15:22:34Z

azure/coverage_job.yml

+          -e '.*/ext/mbstring/libmbfl/.*' \
+          -e '.*/ext/opcache/jit/libudis86/.*' \
+          -e '.*/ext/pcre/pcre2lib/.*' \
+          -e '.*/ext/xmlrpc/libxmlrpc/.*'


Based on https://dev.azure.com/phpazuredevops/PHP/_build/results?buildId=2971&view=codecoverage-tab, it doesn't look like these excludes are working. Docs are not super clear, but possibly the leading .*/ is too much.

Yes, gcovr is emitting warnings (see log at below).

The date lib (ext/date/lib) causes warnings, in lcov and gcovr, related not being able to open file on <stdout>. I don't know reason for these for these warnings. In lcov these warnings can be prevented by excluding */<stdout>:

$(LTP) --output-file $@ --remove $@ '*/<stdout>' $(LCOV_EXCLUDES)

In gcovr we need to use the --exclude-directories:

--exclude-directories 'ext/date/lib$$' \ $(foreach lib, $(GCOVR_EXCLUDES), -e $(lib))

Log

========================== Starting Command Output =========================== [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/2d277420-04f0-44e9-95fb-913a6b2e7ce5.sh (WARNING) GCOV produced the following errors processing /home/vsts/work/1/s/ext/date/lib/parse_iso_intervals.gcda: Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> Could not open output file 'parse_iso_intervals.re.gcov' Could not open output file '<stdout>.gcov' Cannot open source file parse_iso_intervals.re Cannot open source file <stdout> (gcovr could not infer a working directory that resolved it.) (WARNING) GCOV produced the following errors processing /home/vsts/work/1/s/ext/date/lib/parse_date.gcda: Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Cannot open source file parse_date.re Cannot open source file <stdout> Could not open output file 'parse_date.re.gcov' Could not open output file '<stdout>.gcov' Cannot open source file parse_date.re Cannot open source file <stdout> (gcovr could not infer a working directory that resolved it.) lines: 72.1% (189432 out of 262876) branches: 51.6% (123168 out of 238880)

azure/coverage_job.yml

* Add an Azure Publish Code Coverage Results task * Add `make gcovr-html` to generate a gcovr test coverage report in HTML * Add `make gcovr-xml` to generate a gcovr test coverage report in XML * Remove `test` target dependency from `make lcov-html`; Run the two targets together instead: `make test lcov-html`. Re: php#4739 (comment) See: https://externals.io/message/107113, https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops, and php#4759.

nikic · 2019-10-11T15:47:16Z

Merged as db54b0f with a couple more tweaks. Coverage should show up on the next scheduled builds.

Thanks a lot for working on this!

gerardroche force-pushed the azure-code-coverage branch 8 times, most recently from ddde34e to 2923924 Compare September 29, 2019 20:31

krakjoe added Category: Tests Waiting on Author labels Sep 30, 2019

nikic reviewed Sep 30, 2019

View reviewed changes

azure/configure.yml Outdated Show resolved Hide resolved

gerardroche force-pushed the azure-code-coverage branch from aa3a451 to 92b0bc2 Compare September 30, 2019 11:02

nikic mentioned this pull request Sep 30, 2019

Test #4759

Closed

gerardroche force-pushed the azure-code-coverage branch from c83e349 to 385d36c Compare September 30, 2019 12:50

nikic reviewed Sep 30, 2019

View reviewed changes

azure/coverage_job.yml Show resolved Hide resolved

nikic reviewed Sep 30, 2019

View reviewed changes

build/Makefile.gcov Outdated Show resolved Hide resolved

nikic reviewed Sep 30, 2019

View reviewed changes

azure/coverage_job.yml Outdated Show resolved Hide resolved

gerardroche force-pushed the azure-code-coverage branch 3 times, most recently from 5a8e917 to 46143ec Compare October 1, 2019 14:15

gerardroche and others added 5 commits October 11, 2019 16:55

Enable only coverage job

3e5e98d

Don't install lcov (no longer needed)

bd27cf9

Share install step across jobs

310bd51

Reuse "make gcovr-xml" in azure job

d1f0eb8

nikic force-pushed the azure-code-coverage branch from 46143ec to d1f0eb8 Compare October 11, 2019 15:04

nikic closed this Oct 11, 2019

gerardroche deleted the azure-code-coverage branch October 15, 2019 13:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Azure: Publish code coverage results #4746

[WIP] Azure: Publish code coverage results #4746

Uh oh!

gerardroche commented Sep 26, 2019 •

edited

Loading

Uh oh!

Uh oh!

nikic commented Sep 30, 2019

Uh oh!

nikic commented Sep 30, 2019

Uh oh!

gerardroche commented Sep 30, 2019

Uh oh!

nikic Sep 30, 2019

Uh oh!

gerardroche Sep 30, 2019

Uh oh!

nikic Sep 30, 2019

Uh oh!

Uh oh!

Uh oh!

nikic Sep 30, 2019

Uh oh!

gerardroche Oct 1, 2019

Uh oh!

Uh oh!

nikic commented Oct 11, 2019

Uh oh!

Uh oh!

[WIP] Azure: Publish code coverage results #4746

[WIP] Azure: Publish code coverage results #4746

Uh oh!

Conversation

gerardroche commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nikic commented Sep 30, 2019

Uh oh!

nikic commented Sep 30, 2019

Uh oh!

gerardroche commented Sep 30, 2019

Uh oh!

nikic Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

gerardroche Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

nikic Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nikic Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

gerardroche Oct 1, 2019

Choose a reason for hiding this comment

Log

Uh oh!

Uh oh!

nikic commented Oct 11, 2019

Uh oh!

Uh oh!

gerardroche commented Sep 26, 2019 •

edited

Loading