XML output from xmlrunner is not valid to the xUnit Plugin versions xunit-1.104 and above

[bewczardski]

The junit-10.xsd used for testing does not match the junit-10.xsd file the xUnit plugin uses. https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

Updating the file in this project and running unit tests results in the following errors:

======================================================================
ERROR: test_junitxml_xsd_validation_empty_properties (tests.testsuite.XMLTestRunnerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub Repos\unittest-xml-reporting\tests\testsuite.py", line 658, in test_junitxml_xsd_validation_empty_properties
    validate_junit_report(output)
  File "C:\GitHub Repos\unittest-xml-reporting\tests\testsuite.py", line 50, in validate_junit_report
    JUnitSchema.assertValid(document)
  File "src\lxml\etree.pyx", line 3537, in lxml.etree._Validator.assertValid
DocumentInvalid: Element 'testcase', attribute 'file': The attribute 'file' is not allowed., line 4

======================================================================
ERROR: test_junitxml_xsd_validation_order (tests.testsuite.XMLTestRunnerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\GitHub Repos\unittest-xml-reporting\tests\testsuite.py", line 643, in test_junitxml_xsd_validation_order
    validate_junit_report(output)
  File "C:\GitHub Repos\unittest-xml-reporting\tests\testsuite.py", line 50, in validate_junit_report
    JUnitSchema.assertValid(document)
  File "src\lxml\etree.pyx", line 3537, in lxml.etree._Validator.assertValid
DocumentInvalid: Element 'testcase', attribute 'file': The attribute 'file' is not allowed., line 7

----------------------------------------------------------------------

These are not all the errors however, when using xmlrunner with the xUnit plugin for one of my build jobs, the following XML warnings are reported:

WARNING: At line 3 of file:/E:/Test_Jenkins/workspace/gin_Testing_junit-plugin-testing/unittest-reports/TEST-testValidationEntryPoint.TestEntryPoint-20190618152127.xml:cvc-complex-type.3.2.2: Attribute 'file' is not allowed to appear in element 'testcase'.
WARNING: At line 3 of file:/E:/Test_Jenkins/workspace/gin_Testing_junit-plugin-testing/unittest-reports/TEST-testValidationEntryPoint.TestEntryPoint-20190618152127.xml:cvc-complex-type.3.2.2: Attribute 'line' is not allowed to appear in element 'testcase'.
WARNING: At line 3 of file:/E:/Test_Jenkins/workspace/gin_Testing_junit-plugin-testing/unittest-reports/TEST-testValidationEntryPoint.TestEntryPoint-20190618152127.xml:cvc-complex-type.3.2.2: Attribute 'timestamp' is not allowed to appear in element 'testcase'.

[Pietro-G]

Experiencing precisely the same issue +1

[AmauryOrtega]

Exactly the same behaviour. Tried using the XSD provided in the readme using this tool but no luck.

[dnozay]

The version used was added in 2014 to xunit plugin by @gboissinot:

https://github.com/jenkinsci/xunit-plugin/blob/14c6e39c38408b9ed6280361484a13c6f5becca7/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

@nfalco79 made several backwards incompatible changes to the schema while keeping the same version in 2018.

https://github.com/jenkinsci/xunit-plugin/commits/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

A lot of tools came around between 2014 and 2018; if the claim was if you use unittest-xml-reporting and create an xml report, jenkins xunit plugin would work out of the box; that was true until sometime in 2018.

[dnozay]

if you care to try with a version earlier than xunit-2.2.4, please let me know.

[ncarrier]

Hi, for anyone interested in a workaround, I've posted one there: #209 (comment), which works for me and with some adjustments, may work for others...

[nfalco79]

In the README description you report:

A unittest test runner that can save test results to XML files in xUnit format.

xUnit does not define an own XML format, it translates external report format into an internal format that use to build a result graph using threasholds.

JUnit does not define a standard, it is a library. There is an attempt by the JUnit team to create a sort of ISO format for the future (but it is firm).

Since xUnit 2.x supports only a defined pair of defined JUnit format (Ant and Maven, the latest has been evolved recently). This breaking changes has been documented here: https://wiki.jenkins.io/display/JENKINS/xUnit+Plugin

[pbudzon]

if you care to try with a version earlier than xunit-2.2.4, please let me know.

@dnozay I tried with previous versions of xunit (from here: https://updates.jenkins.io/download/plugins/xunit/). Went as far back as version 2.0.4, results in each:

WARNING: The result file (...) for the metric 'JUnit' is not valid. The result file has been skipped.
(...) Attribute 'timestamp' is not allowed to appear in element 'testcase'.
(...) Attribute 'line' is not allowed to appear in element 'testcase'.

So basically the same as reported.

[dnozay]

@dnozay I tried with previous versions of xunit (from here: https://updates.jenkins.io/download/plugins/xunit/). Went as far back as version 2.0.4, results in each:

yes, put the wrong version, xunit 1.100 is the version that had been around from 2014-2018; mid-2018 is when the bug reports started coming in and that lines up with more strict XSD validation. it looks like xunit 1.104 is the first version published in 2018 which may present the issue.

I made additional changes based on #209; but the gist is that you need to perform XSL transformation (now described in readme) or chose Custom tool report type in xunit plugin with your own XSL file

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:output method="xml" indent="yes" />
    <!-- /dev/null for these attributes -->
    <xsl:template match="//testcase/@file" />
    <xsl:template match="//testcase/@line" />
    <xsl:template match="//testcase/@timestamp" />
    <!-- copy the rest -->
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

In this PR there are 2 ideas as well: relax the XSD validation, or provide another report type.
jenkinsci/xunit-plugin#81

I am also going to repeat my comment from
#209

To go back to this earlier comment; I don't plan on dropping the attributes because they provide good debugging information; testsuites are collection of testcases, and testcases can be defined in different file at a certain line number and based on how tests are run, the start time held in timestamp could be useful to debug timing issues.

For those using xunit plugin in Jenkins, the junit plugin seems to work out of the box without issues (it does not perform XSD validation AFAIK), so if you don't have specific requirements, it's worth trying out.

I'm going to keep this issue open until jenkinsci/xunit-plugin#81 is closed. If someone has the time to work on it, I'd advise adding the other report type directly to the plugin, but I couldn't find enough time / info in the wiki.