Skip to content

Update reporting.md #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 3, 2019
Merged

Update reporting.md #442

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 15 additions & 22 deletions docs/reporting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The `mftf` tool logs output continuously to the `dev/tests/acceptance/mftf.log`

## Command line

The MFTF reports about its progress during test run when you run the `mftf` CLI tool with [`run:test`][] or [`run:group`][] commands.
MFTF reports about its progress during test run when you run the `mftf` CLI tool with [`run:test`][] or [`run:group`][] commands.

The report can contain three main parts:

Expand All @@ -35,14 +35,7 @@ The following sections demonstrate an example interpretation of a complete log s

### Pre-run check report

First, the MFTF reports about issues with environment.
In our case, there is an issue with PHP library loading.

```terminal
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/pecl/20160303/php_intl.dll' - dlopen(/usr/local/lib/php/pecl/20160303/php_intl.dll, 9): image not found in Unknown on line 0
```

Next, the MFTF returns `DEPRECATION` reports alerting you that required test components are missing in XML test definitions.
First, MFTF returns `DEPRECATION` warnings alerting you that required test components are missing in XML test definitions.

```terminal
DEPRECATION: Test AdminFilteringCategoryProductsUsingScopeSelectorTest is missing required annotations.{"testName":"AdminFilteringCategoryProductsUsingScopeSelectorTest","missingAnnotations":"stories"}
Expand All @@ -51,7 +44,7 @@ DEPRECATION: Test AdminRemoveProductWeeeAttributeOptionTest is missing required
Generate Tests Command Run
```

`Generate Tests Command Run` indicates the moment when the MFTF has run the tests generation command actually.
`Generate Tests Command Run` indicates that test generation is finished and tests are able to be executed.

### Test execution report

Expand All @@ -72,7 +65,7 @@ Magento\FunctionalTestingFramework.functional Tests (2) ------------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTes
```

After the test generation command (mentioned in the previous section), the MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `2.3.9` that uses `PHPUnit` of version `6.5.13`.
After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `2.3.9` that uses `PHPUnit` of version `6.5.13`.

The tool runs `2 Tests` using the configuration defined in the `functional` suite under the `Magento\FunctionalTestingFramework` namespace.
The corresponding configuration file is `acceptance/tests/functional.suite.yml`.
Expand Down Expand Up @@ -154,7 +147,7 @@ I save screenshot
FAIL
```

When a test step fails, the MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs.
When a test step fails, MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs.
`I save screenshot` follows the failing test step `I see "#something"` in our case.

A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats:
Expand All @@ -176,7 +169,7 @@ Actions after `FAIL` are run as a part of the [`after`][] hook of the test.

### Test result report

After the MFTF completed test execution, it generates a general report about test results along with detailed information about each fail.
After MFTF completed test execution, it generates a general report about test results along with detailed information about each fail.

```terminal
--------------------------------------------------------------------------------
Expand All @@ -192,7 +185,7 @@ First you see warnings and deprecations.
The `DEPRECATION` here is thrown by an MFTF dependency (Symfony) that is out of the scope for test writers and should be considered by MFTF contributors.
If you encounter this type of reporting, [report an issue][].

Then, the MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM.
Then, MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM.
And, finally, that there was `1 failure`.

Next, the report provides details about the test failure.
Expand Down Expand Up @@ -268,19 +261,25 @@ FAILURES!
Tests: 2, Assertions: 3, Failures: 1.
```

The MFTF encountered failures due to the last test run, that included *2* tests with *3* assertions.
MFTF encountered failures due to the last test run, that included *2* tests with *3* assertions.
*1* assertion fails.

## Allure

Each time you run tests, the MFTF appends an XML file with results at the `tests/_output/allure-results/` directory.
Each time you run tests, MFTF appends an XML file with results at the `tests/_output/allure-results/` directory.

The official [Allure Test Report][] documentation is well-covered, so we'll list only the CLI commands that you would need for your day-to-day work.

<div class="bs-callout bs-callout-info">
The following commands are relative to the Magento installation directory.
</div>

To generate the HTML Allure report in a temporary folder and open the report in your default web browser:

```bash
allure serve dev/tests/acceptance/tests/_output/allure-results/
```

To generate a report to the `allure-report/` at the current directory:

```bash
Expand Down Expand Up @@ -334,12 +333,6 @@ To clean up existing reports before generation (for example after getting new re
allure generate dev/tests/acceptance/tests/_output/allure-result --clean
```

To generate the HTML Allure report in a temporary folder and open the report in your default web browser:

```bash
allure serve dev/tests/acceptance/tests/_output/allure-results/
```

Refer to the [Reporting section][] for more Allure CLI details.

<!-- Link definitions -->
Expand Down