Skip to content

MQE-2150: group value "skip" no longer causes test to be skipped #726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 4, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,22 @@ use Yandex\Allure\Adapter\Model\SeverityLevel;
use Yandex\Allure\Adapter\Annotation\TestCaseId;

/**
* @Title("[NO TESTCASEID]: skippedNoIssuesTest")
* @Description("<h3>Test files</h3>verification/TestModule/Test/SkippedTest/SkippedTestNoIssues.xml<br>")
* @Title("[NO TESTCASEID]: GroupSkipGenerationTestTitle")
* @Description("GroupSkipGenerationTestDescription<h3>Test files</h3>verification/TestModule/Test/GroupSkipGenerationTest.xml<br>")
* @group skip
*/
class SkippedTestNoIssuesCest
class GroupSkipGenerationTestCest
{
/**
* @Stories({"skippedNo"})
* @Stories({"GroupSkipGenerationTestStory"})
* @Severity(level = SeverityLevel::MINOR)
* @Features({"TestModule"})
* @Parameter(name = "AcceptanceTester", value="$I")
* @param AcceptanceTester $I
* @return void
* @throws \Exception
*/
public function SkippedTestNoIssues(AcceptanceTester $I, \Codeception\Scenario $scenario)
public function GroupSkipGenerationTest(AcceptanceTester $I)
{
$scenario->skip("This test is skipped due to the following issues:\nNo issues have been specified.");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
-->
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
<test name="SkippedTestNoIssues">
<test name="GroupSkipGenerationTest">
<annotations>
<stories value="skippedNo"/>
<title value="skippedNoIssuesTest"/>
<description value=""/>
<stories value="GroupSkipGenerationTestStory"/>
<title value="GroupSkipGenerationTestTitle"/>
<description value="GroupSkipGenerationTestDescription"/>
<severity value="AVERAGE"/>
<group value="skip"/>
</annotations>
Expand Down
22 changes: 22 additions & 0 deletions dev/tests/verification/Tests/GroupSkipGenerationTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<?php
/**
* Copyright © Magento, Inc. All rights reserved.
* See COPYING.txt for license details.
*/
namespace tests\verification\Tests;

use tests\util\MftfTestCase;

class GroupSkipGenerationTest extends MftfTestCase
{
/**
* Tests group skip test generation
*
* @throws \Exception
* @throws \Magento\FunctionalTestingFramework\Exceptions\TestReferenceException
*/
public function testGroupSkipGenerationTest()
{
$this->generateAndCompareTest('GroupSkipGenerationTest');
}
}
11 changes: 0 additions & 11 deletions dev/tests/verification/Tests/SkippedGenerationTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,4 @@ public function testMultipleSkippedIssuesGeneration()
{
$this->generateAndCompareTest('SkippedTestTwoIssues');
}

/**
* Tests skipped test generation with no specified issues. Will be deprecated after MFTF 3.0.0
*
* @throws \Exception
* @throws \Magento\FunctionalTestingFramework\Exceptions\TestReferenceException
*/
public function testSkippedNoIssueGeneration()
{
$this->generateAndCompareTest('SkippedTestNoIssues');
}
}
2 changes: 1 addition & 1 deletion docs/merging.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ To add a `<test>`, create a new `*Test.xml` file or add a `<test>` node to an ex
## Remove a test

Tests cannot be removed while merging.
If a [`<test>`][tests] must be skipped due to a module completely invalidating a function, you can add the test to the `skip` group.
If a [`<test>`][tests] must be skipped due to a module completely invalidating a function, you can use `skip` annotation to skip the test.

Learn more about running tests with different options using [`mftf`] or [`codecept`] commands.

Expand Down
2 changes: 1 addition & 1 deletion docs/suite.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The format of a suite:
- must not match any existing group value.
For example, the suite `<suite name="ExampleTest">` will fail during test run if any test contains in annotations `<group value="ExampleTest">`.
- must not be `default` or `skip`. Tests that are not in any suite are generated under the `default` suite.
The suite name `skip` is synonymous to including a test in the `<group value="skip"/>`, which will be deprecated in MFTF 3.0.0.
The suite name `skip` is synonymous to including a test in the `<group value="skip"/>`.
- can contain letters, numbers, and underscores.
- should be upper camel case.

Expand Down
2 changes: 1 addition & 1 deletion docs/test/annotations.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Add `<skip>` to the test to skip it during test run.

Attribute|Type|Use|Definition
---|---|---|---
`value`|string|required|A value that is used to group tests. It should be lower case. `skip` is reserved to ignore content of the test and generate an empty test.
`value`|string|required|A value that is used to group tests. It should be lower case.

#### Example

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,11 +161,8 @@ public function getDeprecated()
*/
public function isSkipped()
{
// TODO deprecation|deprecate MFTF 3.0.0 remove elseif when group skip is no longer allowed
if (array_key_exists('skip', $this->annotations)) {
return true;
} elseif (array_key_exists('group', $this->annotations) && (in_array("skip", $this->annotations['group']))) {
return true;
}
return false;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,6 @@ public function extractAnnotations($testAnnotations, $filename, $validateAnnotat
foreach ($annotationData as $annotationValue) {
$annotationValues[] = $annotationValue[self::ANNOTATION_VALUE];
}
// TODO deprecation|deprecate MFTF 3.0.0
if ($annotationKey == "group" && in_array("skip", $annotationValues)) {
LoggingUtil::getInstance()->getLogger(AnnotationExtractor::class)->warning(
"Use of group skip will be deprecated in MFTF 3.0.0. Please update tests to use skip tags.",
["test" => $filename]
);
}

$annotationObjects[$annotationKey] = $annotationValues;
}
Expand Down