Merge branch 'develop' into ReleaseCandidateTest

Author: okolesnyk

.gitignore

@@ -19,4 +19,4 @@ dev/tests/mftf.log
 dev/tests/docs/*
 dev/tests/_output
 dev/tests/functional.suite.yml
-mftf-annotations-static-check.txt
+

docs/backward-incompatible-changes.md

@@ -0,0 +1,163 @@
+# MFTF 3.0.0 backward incompatible changes
+
+This page highlights backward incompatible changes between releases that have a major impact and require detailed explanation and special instructions to ensure third-party tests continue working with Magento core tests.
+
+## Minimum supported PHP version changes
+
+We changed the minimum PHP version requirement from 7.0 to 7.3. Because of the PHP version requirement change, this MFTF version only supports Magento 2.4 or later.
+
+## Folder structure changes
+
+We removed support to read test modules from the deprecated path `dev/tests/acceptance/tests/functional/Magento/FunctionalTest`. If there are test modules in this path, they should be moved to `dev/tests/acceptance/tests/functional/Magento`. 
+
+## XSD schema changes
+
+-  Files under test modules `ActionGroup`, `Page`, `Section`, `Test` and `Suite` only support a single entity per file. 
+-  The `file` attribute from `<module>` has been removed from the suite schema. `<module file=""/>` is no longer supported in suites.
+-  Metadata filename format changed to ***`*Meta.xml`***.
+-  Only nested assertion syntax will be supported. See the [assertions page](./docs/test/assertions.md) for details. Here is an example of the nested assertion syntax:
+   ```xml
+   <assertEquals stepKey="assertAddressOrderPage">
+      <actualResult type="const">$billingAddressOrderPage</actualResult>
+      <expectedResult type="const">$shippingAddressOrderPage</expectedResult>
+   </assertEquals>
+   ```
+
+### Upgrading tests to the new schema
+
+The following table lists the upgrade scripts that are available to upgrade tests to the new schema.
+
+| Script name           | Description                                                                                               |
+|-----------------------|-----------------------------------------------------------------------------------------------------------|
+|`splitMultipleEntitiesFiles`| Splits files that have multiple entities into multiple files with one entity per file. |
+|`upgradeAssertionSchema`| Updates assert actions that uses the old assertion syntax into the new nested syntax.|
+|`renameMetadataFiles`| Renames Metadata filenames to `*Meta.xml`.|
+|`removeModuleFileInSuiteFiles`| Removes occurrences of `<module file=""/>` from all `<suite>`s.|
+|`removeUnusedArguments`| Removes unused arguments from action groups.|
+|`upgradeTestSchema`| Replaces relative schema paths to URN in test files.| 
+
+To run the upgrade tests:
+
+1. Run `bin/mftf reset --hard` to remove old generated configurations.
+1. Run `bin/mftf build:project` to generate new configurations.
+1. Run `bin/mftf upgrade:tests`. [See command page for details](./docs/commands/mftf.md#upgradetests).
+1. Lastly, try to generate all tests. Tests should all be generated as a result of the upgrades. If not, the most likely issue will be a changed XML schema. Check error messaging and search your codebase for the attributes listed.
+
+## MFTF commands
+
+`--debug` option `NONE` removed for strict schema validation. Ensure there are no schema validation errors in test modules before running MFTF commands.
+
+## MFTF actions
+
+### `executeInSelenium` and `performOn` removed
+
+**Action**: Deprecated actions `executeInSelenium` and `performOn` are removed in favor of new action `helper`.
+
+**Reason**: `executeInSelenium` and `performOn` allowed custom PHP code to be written inline inside of XML files which was difficult to maintain, troubleshoot, and modify.
+
+**Details**: 
+
+The `helper` allows test writers to solve advanced requirements beyond what MFTF offers out of the box. See [custom-helpers](./docs/custom-helpers.md) for more information on usage. 
+
+Here is an example of using `helper` in place of `executeSelenium` to achieve same workflow.
+
+Old usage:
+
+```xml
+<executeInSelenium function="function ($webdriver) use ($I) {
+        $heading = $webdriver->findElement(\Facebook\WebDriver\WebDriverBy::xpath('//div[contains(@class, \'inline-wysiwyg\')]//h2'));
+        $actions = new \Facebook\WebDriver\Interactions\WebDriverActions($webdriver);
+        $actions->moveToElement($heading, {{TinyMCEPartialHeadingSelection.startX}}, {{TinyMCEPartialHeadingSelection.startY}})
+         ->clickAndHold()
+         ->moveToElement($heading, {{TinyMCEPartialHeadingSelection.endX}}, {{TinyMCEPartialHeadingSelection.endY}})
+         ->release()
+         ->perform();
+        }" stepKey="selectHeadingTextInTinyMCE"/>
+```    
+
+New usage:
+
+```xml
+<helper class="\Magento\PageBuilder\Test\Mftf\Helper\SelectText" method="selectText" stepKey="selectHeadingTextInTinyMCE">
+    <argument name="context">//div[contains(@class, 'inline-wysiwyg')]//h2</argument>
+    <argument name="startX">{{TinyMCEPartialHeadingSelection.startX}}</argument>
+    <argument name="startY">{{TinyMCEPartialHeadingSelection.startY}}</argument>
+    <argument name="endX">{{TinyMCEPartialHeadingSelection.endX}}</argument>
+    <argument name="endY">{{TinyMCEPartialHeadingSelection.endY}}</argument>
+</helper>
+```
+
+### `pauseExecution` removed
+
+**Action**: `pauseExecution` is removed in favor of `pause`.
+
+**Reason**: `[WebDriver]pauseExecution` is removed in Codeception 3 in favor of `I->pause()`.
+
+**Details**: 
+
+See the [actions page for details](./docs/test/actions.md#pause). Here is a usage example:
+
+```xml
+<pause stepKey="pauseExecutionKey"/>
+```
+
+### Removed assert actions
+
+**Action**: Assert actions `assertInternalType`, `assertNotInternalType` and `assertArraySubset` are removed.
+
+**Reason**: PHPUnit 9 has dropped support for these assertions.
+
+### Updated assert actions
+
+**Action**: The `delta` attribute has been removed from `assertEquals` and `assertNotEquals`. Instead, new assert actions have been introduced:
+
+ -  `assertEqualsWithDelta`
+ -  `assertNotEqualsWithDelta` 
+ -  `assertEqualsCanonicalizing`
+ -  `assertNotEqualsCanonicalizing`
+ -  `assertEqualsIgnoringCase`
+ -  `assertNotEqualsIgnoringCase`
+
+**Reason**: PHPUnit 9 has dropped support for optional parameters for `assertEquals` and `assertNotEquals` and has introduced these new assertions.
+
+**Details**: 
+
+Usage of `assertEquals` or `assertNotEquals` with a specified `delta`, should be replaced with appropriate assertion from the above list.
+
+### `assertContains` supports only iterable haystacks
+
+**Action**: `assertContains` and `assertNotContains` now only supports iterable haystacks. These assert actions have been added to work with string haystacks:
+
+-  `assertStringContainsString`
+-  `assertStringNotContainsString`
+-  `assertStringContainsStringIgnoringCase`
+-  `assertStringNotContainsStringIgnoringCase`
+
+**Reason**: With PHPUnit 9, `assertContains` and `assertNotContains` only allows iterable haystacks. New assertions have been introduced to support string haystacks.
+
+**Details**: 
+
+Usages of `assertContains` and `assertNotContains` with string haystacks should be replaced with appropriate assertion from the above list.
+
+Usage example for string haystacks:
+
+```xml
+<assertStringContainsString stepKey="assertDiscountOnPrice2">
+  <actualResult type="const">$grabSimpleProdPrice2</actualResult>
+  <expectedResult type="string">$110.70</expectedResult>
+</assertStringContainsString>
+```
+
+### `formatMoney` removed
+
+**Action**: `formatMoney` has been removed in favor of `formatCurrency`.
+
+**Reason**: PHP 7.4 has deprecated use of `formatMoney`. 
+
+**Details**: Format input to specified currency according to the locale specified. 
+
+Usage example:
+
+```xml
+<formatCurrency userInput="1234.56789000" locale="de_DE" currency="USD" stepKey="usdInDE"/>
+```

docs/commands/mftf.md

@@ -430,13 +430,15 @@ The example parameters are taken from the `etc/config/.env.example` file.
 
 ### `static-checks`
 
-Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to.
+Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to. 
 Behavior for determining what tests to run is as follows:
 
 *  If test names are specified, only those tests are run.
 *  If no test names are specified, tests are run according to `staticRuleset.json`.
 *  If no `staticRuleset.json` is found, all tests are run.
 
+Static checks errors are written to *.txt files under TEST_BP/tests/_output/static-results/
+
 #### Usage
 
 ```bash

src/Magento/FunctionalTestingFramework/StaticCheck/ActionGroupArgumentsCheck.php

@@ -65,7 +65,7 @@ public function execute(InputInterface $input)
 
         $this->output = $this->scriptUtil->printErrorsToFile(
             $this->errors,
-            self::ERROR_LOG_FILENAME,
+            StaticChecksList::getErrorFilesPath() . DIRECTORY_SEPARATOR . self::ERROR_LOG_FILENAME . '.txt',
             self::ERROR_LOG_MESSAGE
         );
     }

src/Magento/FunctionalTestingFramework/StaticCheck/AnnotationsCheck.php

@@ -85,7 +85,7 @@ public function execute(InputInterface $input)
         $scriptUtil = new ScriptUtil();
         $this->output = $scriptUtil->printErrorsToFile(
             $this->errors,
-            self::ERROR_LOG_FILENAME,
+            StaticChecksList::getErrorFilesPath() . DIRECTORY_SEPARATOR . self::ERROR_LOG_FILENAME . '.txt',
             self::ERROR_LOG_MESSAGE
         );
     }

src/Magento/FunctionalTestingFramework/StaticCheck/DeprecatedEntityUsageCheck.php

@@ -128,7 +128,7 @@ public function execute(InputInterface $input)
         // Hold on to the output and print any errors to a file
         $this->output = $this->scriptUtil->printErrorsToFile(
             $this->errors,
-            self::ERROR_LOG_FILENAME,
+            StaticChecksList::getErrorFilesPath() . DIRECTORY_SEPARATOR . self::ERROR_LOG_FILENAME . '.txt',
             self::ERROR_LOG_MESSAGE
         );
     }

src/Magento/FunctionalTestingFramework/StaticCheck/StaticChecksList.php

@@ -7,13 +7,17 @@
 
 namespace Magento\FunctionalTestingFramework\StaticCheck;
 
+use Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException;
+use Magento\FunctionalTestingFramework\Util\Path\FilePathFormatter;
+
 /**
  * Class StaticChecksList has a list of static checks to run on test xml
  * @codingStandardsIgnoreFile
  */
 class StaticChecksList implements StaticCheckListInterface
 {
     const DEPRECATED_ENTITY_USAGE_CHECK_NAME = 'deprecatedEntityUsage';
+    const STATIC_RESULTS = 'tests' . DIRECTORY_SEPARATOR .'_output' . DIRECTORY_SEPARATOR . 'static-results';
 
     /**
      * Property contains all static check scripts.
@@ -22,10 +26,18 @@ class StaticChecksList implements StaticCheckListInterface
      */
     private $checks;
 
+    /**
+     * Directory path for static checks error files
+     *
+     * @var string
+     */
+    private static $errorFilesPath = null;
+
     /**
      * Constructor
      *
      * @param array $checks
+     * @throws TestFrameworkException
      */
     public function __construct(array $checks = [])
     {
@@ -35,6 +47,11 @@ public function __construct(array $checks = [])
             self::DEPRECATED_ENTITY_USAGE_CHECK_NAME => new DeprecatedEntityUsageCheck(),
             'annotations' => new AnnotationsCheck()
         ] + $checks;
+
+        // Static checks error files directory
+        if (null === self::$errorFilesPath) {
+            self::$errorFilesPath = FilePathFormatter::format(TESTS_BP) . self::STATIC_RESULTS;
+        }
     }
 
     /**
@@ -44,4 +61,12 @@ public function getStaticChecks()
     {
         return $this->checks;
     }
+
+    /**
+     * Return the directory path for the static check error files
+     */
+    public static function getErrorFilesPath()
+    {
+        return self::$errorFilesPath;
+    }
 }

src/Magento/FunctionalTestingFramework/StaticCheck/TestDependencyCheck.php

@@ -121,7 +121,7 @@ public function execute(InputInterface $input)
         // hold on to the output and print any errors to a file
         $this->output = $this->scriptUtil->printErrorsToFile(
             $this->errors,
-            self::ERROR_LOG_FILENAME,
+            StaticChecksList::getErrorFilesPath() . DIRECTORY_SEPARATOR . self::ERROR_LOG_FILENAME . '.txt',
             self::ERROR_LOG_MESSAGE
         );
     }

src/Magento/FunctionalTestingFramework/Util/Logger/MftfLogger.php

@@ -7,28 +7,53 @@
 namespace Magento\FunctionalTestingFramework\Util\Logger;
 
 use Magento\FunctionalTestingFramework\Config\MftfApplicationConfig;
-use Monolog\Handler\StreamHandler;
+use Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException;
+use Monolog\Handler\HandlerInterface;
 use Monolog\Logger;
 
 class MftfLogger extends Logger
 {
+    /**
+     * MFTF execution phase
+     *
+     * @var string
+     */
+    private $phase;
+
+    /**
+     * MftfLogger constructor.
+     *
+     * @param string             $name
+     * @param HandlerInterface[] $handlers
+     * @param callable[]         $processors
+     * @throws TestFrameworkException
+     */
+    public function __construct($name, array $handlers = [], array $processors = [])
+    {
+        parent::__construct($name, $handlers, $processors);
+        $this->phase = MftfApplicationConfig::getConfig()->getPhase();
+    }
+
     /**
      * Prints a deprecation warning, as well as adds a log at the WARNING level.
+     * Suppresses logging during execution phase.
      *
      * @param string  $message The log message.
      * @param array   $context The log context.
      * @param boolean $verbose
      * @return void
-     * @throws \Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException
      */
     public function deprecation($message, array $context = [], $verbose = false)
     {
         $message = "DEPRECATION: " . $message;
-        // Suppress print during unit testing
-        if (MftfApplicationConfig::getConfig()->getPhase() !== MftfApplicationConfig::UNIT_TEST_PHASE && $verbose) {
+        // print during test generation
+        if ($this->phase === MftfApplicationConfig::GENERATION_PHASE && $verbose) {
             print ($message . json_encode($context) . "\n");
         }
-        parent::warning($message, $context);
+        // suppress logging during test execution
+        if ($this->phase !== MftfApplicationConfig::EXECUTION_PHASE) {
+            parent::warning($message, $context);
+        }
     }
 
     /**
@@ -38,34 +63,36 @@ public function deprecation($message, array $context = [], $verbose = false)
      * @param array   $context The log context.
      * @param boolean $verbose
      * @return void
-     * @throws \Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException
      */
     public function criticalFailure($message, array $context = [], $verbose = false)
     {
         $message = "FAILURE: " . $message;
         // Suppress print during unit testing
-        if (MftfApplicationConfig::getConfig()->getPhase() !== MftfApplicationConfig::UNIT_TEST_PHASE && $verbose) {
+        if ($this->phase !== MftfApplicationConfig::UNIT_TEST_PHASE && $verbose) {
             print ($message . implode("\n", $context) . "\n");
         }
         parent::critical($message, $context);
     }
 
     /**
      * Adds a log record at the NOTICE level.
+     * Suppresses logging during execution phase.
      *
      * @param string  $message
      * @param array   $context
      * @param boolean $verbose
      * @return void
-     * @throws \Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException
      */
     public function notification($message, array $context = [], $verbose = false)
     {
         $message = "NOTICE: " . $message;
-        // Suppress print during unit testing
-        if (MftfApplicationConfig::getConfig()->getPhase() !== MftfApplicationConfig::UNIT_TEST_PHASE && $verbose) {
-            print ($message . implode("\n", $context) . "\n");
+        // print during test generation
+        if ($this->phase === MftfApplicationConfig::GENERATION_PHASE && $verbose) {
+            print ($message . json_encode($context) . "\n");
+        }
+        // suppress logging during test execution
+        if ($this->phase !== MftfApplicationConfig::EXECUTION_PHASE) {
+            parent::notice($message, $context);
         }
-        parent::notice($message, $context);
     }
 }