Merge pull request #116 from magento/develop

MQE-953: Merge the release branch to master

Author: imeron2433

.env.example

@@ -0,0 +1,40 @@
+#Copyright © Magento, Inc. All rights reserved.
+#See COPYING.txt for license details.
+
+#*** Set the base URL for your Magento instance ***#
+MAGENTO_BASE_URL=http://devdocs.magento.com/
+
+#*** Set the Admin Username and Password for your Magento instance ***#
+MAGENTO_BACKEND_NAME=admin
+MAGENTO_ADMIN_USERNAME=admin
+MAGENTO_ADMIN_PASSWORD=123123q
+
+#*** Path to CLI entry point and command parameter name. Uncomment and change if folder structure differs from standard Magento installation
+#MAGENTO_CLI_COMMAND_PATH=dev/tests/acceptance/utils/command.php
+#MAGENTO_CLI_COMMAND_PARAMETER=command
+
+#*** Selenium Server Protocol, Host, Port, and Path, with local defaults. Uncomment and change if not running Selenium locally.
+#SELENIUM_HOST=127.0.0.1
+#SELENIUM_PORT=4444
+#SELENIUM_PROTOCOL=http
+#SELENIUM_PATH=/wd/hub
+
+#*** Browser for running tests, default chrome. Uncomment and change if you want to run tests on another browser (ex. firefox).
+BROWSER=chrome
+
+#*** Uncomment and set host & port if your dev environment needs different value other than MAGENTO_BASE_URL for Rest API Requests ***#
+#MAGENTO_RESTAPI_SERVER_HOST=
+#MAGENTO_RESTAPI_SERVER_PORT=
+
+#*** Uncomment these properties to set up a dev environment with symlinked projects ***#
+#TESTS_BP=
+#FW_BP=
+#TESTS_MODULE_PATH=
+
+#*** These properties impact the modules loaded into MFTF, you can point to your own full path, or a custom set of modules located with the core set
+MODULE_WHITELIST=Magento_Framework,Magento_ConfigurableProductWishlist,Magento_ConfigurableProductCatalogSearch
+#CUSTOM_MODULE_PATHS=
+
+#*** Bool property which allows the user to toggle debug output during test execution
+#MFTF_DEBUG=
+#*** End of .env ***#

.gitignore

@@ -11,3 +11,6 @@ build/*
 clover.xml
 coverage/
 .vscode
+codeception.yml
+dev/tests/functional/MFTF.suite.yml
+dev/tests/functional/_output

CHANGELOG.md

@@ -1,6 +1,56 @@
 Magento Functional Testing Framework Changelog
 ================================================
 
+2.2.0
+-----
+### Enhancements  
+* Traceability
+    * Javascript errors are now logged and reported in test output.
+    * Test failures are no longer overwritten by failures in an `<after>` hook.
+    * Tests will no longer execute an `<after>` hook twice if a failure triggered in the `<after>` hook.
+    * Tests marked with `<group value="skip">` will now appear in generated Allure reports.
+        * Along with the above, the `robo group` command no longer omits the `skip` group (skipped tests are picked up but not fully executed).
+* Modularity
+    * MFTF no longer relies on relative pathing to determine its path to tests or Magento (favoring composer information if available).
+    * Tests and test materials are now read in from Magento modules as well as extensions in addition to `dev/tests/acceptance`.
+        * See DevDocs `Getting Started` for details on expected paths and merge order.
+* Customizability
+    * Creation of Suites is now supported
+        * `<suite>` can include tests via `name`, module, or `<group>` tags.
+        * Consolidation of preconditions can be achieved via use of `<before/after>` tags in a `<suite>`
+            * All normal test actions are supported
+            * Data returned from actions is not available for reference in subsequent tests (`createData` or `grab` actions).
+        * `robo generate:tests` generates all suites and tests, and can be given a JSON configuration to generate specific test/suites.
+        * See MFTF Devdocs "Suite" page for more details.
+    * `<deleteData>` may now be called against a `url` instead of a stepKey reference.
+    * `<dragAndDrop>` may now be given an additional `x/y` offset.
+    * `<executeJS>` now returns a variable based on what the executed script returns.
+    * Added `<element>` `type="block"`.
+    * `<page>` elements may now be blank (contain no child sections).
+* Maintainability
+    * `robo generate:tests --config parallel` now accepts a `--lines` argument, for grouping and sorting based on test length.
+    * `robo generate:tests` now checks for:
+        * Duplicate step keys within an `actionGroup`.
+        * Ambiguous or invalid `stepKey` references (in merge files).
+    * `robo generate:tests` now suppresses warnings by default. The command now accepts a `--verbose` flag to show full output including warnings.
+
+### Fixes
+* Exception message for the `<conditionalClick>` action now correctly references the `selector` given.
+* Usage of multiple parameterized elements in a `selector` now correctly resolves all element references.
+* Usage of multiple uniqueness references on the same entity now generate correctly.
+* Persisted entity references are correctly interpolated with `<page>` url of `type="admin"`.
+* Metadata that contains 2 or more params in its `url` now correctly resolve parameters.
+* Arguments can now be passed to `x` and `y` attributes in `actionGroup`.
+* Arguments can now be passed to nested `<assert*>` action elements.
+* The `<seeInField>` action can now be used to assert against empty strings.
+* Empty `<data>` elements within an `<entity>` now generate correctly.
+* Mapping of the `<magentoCLI>` to the custom command has been fixed.
+
+### GitHub Issues/Pull requests:
+* [#89](https://github.com/magento/magento2-functional-testing-framework/pull/89) -- Add ability to use array entities as arguments.
+* [#68](https://github.com/magento/magento2-functional-testing-framework/issues/68) -- Excessive double quotes are being generated in WaitForElementChange method arguments (fixed in [#103](https://github.com/magento/magento2-functional-testing-framework/pull/103))
+* [#31](https://github.com/magento/magento2-functional-testing-framework/issues/31) -- Can't run tests without a store having "default" store code (fixed in [#86](https://github.com/magento/magento2-functional-testing-framework/pull/86))
+
 2.1.2
 -----
 ### Enhancements

RoboFile.php

@@ -0,0 +1,214 @@
+<?php
+/**
+ * Copyright © Magento, Inc. All rights reserved.
+ * See COPYING.txt for license details.
+ */
+
+/** This is project's console commands configuration for Robo task runner.
+ *
+ * @codingStandardsIgnoreStart
+ * @see http://robo.li/
+ */
+class RoboFile extends \Robo\Tasks
+{
+    use Robo\Task\Base\loadShortcuts;
+
+    /**
+     * Duplicate the Example configuration files used to customize the Project for customization.
+     *
+     * @return void
+     */
+    function cloneFiles()
+    {
+        $this->_exec('cp -vn .env.example .env');
+        $this->_exec('cp -vf codeception.dist.yml codeception.yml');
+        $this->_exec('cp -vf dev' . DIRECTORY_SEPARATOR . 'tests'.  DIRECTORY_SEPARATOR . 'functional' . DIRECTORY_SEPARATOR .'MFTF.suite.dist.yml dev' . DIRECTORY_SEPARATOR . 'tests'.  DIRECTORY_SEPARATOR . 'functional' . DIRECTORY_SEPARATOR .'MFTF.suite.yml');
+    }
+
+    /**
+     * Duplicate the Example configuration files for the Project.
+     * Build the Codeception project.
+     *
+     * @return void
+     */
+    function buildProject()
+    {
+        $this->writeln("<error>This command will be removed in MFTF v3.0.0. Please use bin/mftf build:project instead.</error>\n");
+        $this->cloneFiles();
+        $this->_exec('vendor'. DIRECTORY_SEPARATOR .'bin'. DIRECTORY_SEPARATOR .'codecept build');
+    }
+
+    /**
+     * Generate all Tests in PHP.
+     *
+     * @param array $tests
+     * @param array $opts
+     * @return void
+     */
+    function generateTests(array $tests, $opts = ['config' => null, 'force' => true, 'nodes' => null, 'debug' => false])
+    {
+        require 'dev' . DIRECTORY_SEPARATOR . 'tests'. DIRECTORY_SEPARATOR . 'functional' . DIRECTORY_SEPARATOR . '_bootstrap.php';
+        $GLOBALS['GENERATE_TESTS'] = true;
+        if (!$this->isProjectBuilt()) {
+            $this->say("<info>Please run bin/mftf build:project and configure your environment (.env) first.</info>");
+            exit(\Robo\Result::EXITCODE_ERROR);
+        }
+        $testsObjects = [];
+        foreach ($tests as $test) {
+            $testsObjects[] = Magento\FunctionalTestingFramework\Test\Handlers\TestObjectHandler::getInstance()->getObject($test);
+        }
+        if ($opts['force']) {
+            $GLOBALS['FORCE_PHP_GENERATE'] = true;
+        }
+        $testsReferencedInSuites = \Magento\FunctionalTestingFramework\Suite\SuiteGenerator::getInstance()->generateAllSuites($opts['config']);
+        \Magento\FunctionalTestingFramework\Util\TestGenerator::getInstance(null, $testsObjects, $opts['debug'])->createAllTestFiles($opts['config'], $opts['nodes'], $testsReferencedInSuites);
+        $this->say("<comment>Generate Tests Command Run</comment>");
+    }
+
+    /**
+     * Check if MFTF has been properly configured
+     * @return bool
+     */
+    private function isProjectBuilt()
+    {
+        $actorFile = __DIR__ . DIRECTORY_SEPARATOR . 'src' . DIRECTORY_SEPARATOR . 'Magento' . DIRECTORY_SEPARATOR . 'FunctionalTestingFramework' . DIRECTORY_SEPARATOR . '_generated' . DIRECTORY_SEPARATOR . 'AcceptanceTesterActions.php';
+
+        $login = !empty(getenv('MAGENTO_ADMIN_USERNAME'));
+        $password = !empty(getenv('MAGENTO_ADMIN_PASSWORD'));
+        $baseUrl = !empty(getenv('MAGENTO_BASE_URL'));
+        $backendName = !empty(getenv('MAGENTO_BACKEND_NAME'));
+        $test = (file_exists($actorFile) && $login && $password && $baseUrl && $backendName);
+        return $test;
+    }
+
+    /**
+     * Generate a suite based on name(s) passed in as args.
+     *
+     * @param array $args
+     * @throws Exception
+     * @return void
+     */
+    function generateSuite(array $args)
+    {
+        if (empty($args)) {
+            throw new Exception("Please provide suite name(s) after generate:suite command");
+        }
+
+        require 'dev' . DIRECTORY_SEPARATOR . 'tests'. DIRECTORY_SEPARATOR . 'functional' . DIRECTORY_SEPARATOR . '_bootstrap.php';
+        $sg = \Magento\FunctionalTestingFramework\Suite\SuiteGenerator::getInstance();
+
+        foreach ($args as $arg) {
+            $sg->generateSuite($arg);
+        }
+    }
+
+    /**
+     * Run all MFTF tests.
+     *
+     * @return void
+     */
+    function mftf()
+    {
+        $this->_exec('.' . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'bin' . DIRECTORY_SEPARATOR . 'codecept run MFTF --skip-group skip');
+    }
+
+    /**
+     * Run all Tests with the specified @group tag, excluding @group 'skip'.
+     *
+     * @param string $args
+     * @return void
+     */
+    function group($args = '')
+    {
+        $this->taskExec('.' . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'bin' . DIRECTORY_SEPARATOR . 'codecept run --verbose --steps --skip-group skip --group')->args($args)->run();
+    }
+
+    /**
+     * Run all Functional tests located under the Directory Path provided.
+     *
+     * @param string $args
+     * @return void
+     */
+    function folder($args = '')
+    {
+        $this->taskExec('.' . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'bin' . DIRECTORY_SEPARATOR . 'codecept run ')->args($args)->run();
+    }
+
+    /**
+     * Run all Tests marked with the @group tag 'example'.
+     *
+     * @return void
+     */
+    function example()
+    {
+        $this->_exec('.' . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'bin' . DIRECTORY_SEPARATOR . 'codecept run --group example --skip-group skip');
+    }
+
+    /**
+     * Generate the HTML for the Allure report based on the Test XML output - Allure v1.4.X
+     *
+     * @return \Robo\Result
+     */
+    function allure1Generate()
+    {
+        return $this->_exec('allure generate tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-results'. DIRECTORY_SEPARATOR .' -o tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-report'. DIRECTORY_SEPARATOR .'');
+    }
+
+    /**
+     * Generate the HTML for the Allure report based on the Test XML output - Allure v2.3.X
+     *
+     * @return \Robo\Result
+     */
+    function allure2Generate()
+    {
+        return $this->_exec('allure generate tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-results'. DIRECTORY_SEPARATOR .' --output tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-report'. DIRECTORY_SEPARATOR .' --clean');
+    }
+
+    /**
+     * Open the HTML Allure report - Allure v1.4.X
+     *
+     * @return void
+     */
+    function allure1Open()
+    {
+        $this->_exec('allure report open --report-dir tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-report'. DIRECTORY_SEPARATOR .'');
+    }
+
+    /**
+     * Open the HTML Allure report - Allure v2.3.X
+     *
+     * @return void
+     */
+    function allure2Open()
+    {
+        $this->_exec('allure open --port 0 tests'. DIRECTORY_SEPARATOR .'_output'. DIRECTORY_SEPARATOR .'allure-report'. DIRECTORY_SEPARATOR .'');
+    }
+
+    /**
+     * Generate and open the HTML Allure report - Allure v1.4.X
+     *
+     * @return void
+     */
+    function allure1Report()
+    {
+        $result1 = $this->allure1Generate();
+
+        if ($result1->wasSuccessful()) {
+            $this->allure1Open();
+        }
+    }
+
+    /**
+     * Generate and open the HTML Allure report - Allure v2.3.X
+     *
+     * @return void
+     */
+    function allure2Report()
+    {
+        $result1 = $this->allure2Generate();
+
+        if ($result1->wasSuccessful()) {
+            $this->allure2Open();
+        }
+    }
+}

bin/mftf

@@ -0,0 +1,30 @@
+#!/usr/bin/env php
+
+<?php
+/**
+ * Copyright © Magento, Inc. All rights reserved.
+ * See COPYING.txt for license details.
+ */
+
+if (PHP_SAPI !== 'cli') {
+    echo 'bin/mftf must be run as a CLI application';
+    exit(1);
+}
+
+try {
+    require_once __DIR__ . '/../bootstrap.php';
+    $application = new Symfony\Component\Console\Application();
+    $application->setName('Magento Functional Testing Framework CLI');
+    $application->setVersion('1.0.0');
+    $application->add(new Magento\FunctionalTestingFramework\Console\SetupEnvCommand());
+    $application->add(new Magento\FunctionalTestingFramework\Console\BuildProjectCommand());
+    $application->run();
+} catch (\Exception $e) {
+    while ($e) {
+        echo $e->getMessage();
+        echo $e->getTraceAsString();
+        echo "\n\n";
+        $e = $e->getPrevious();
+    }
+    exit(1);
+}

bin/static-checks

@@ -8,7 +8,7 @@ set -e
 echo "==============================="
 echo "         CODE SNIFFER"
 echo "==============================="
-vendor/bin/phpcs ./src --standard=./dev/tests/static/Magento
+vendor/bin/phpcs ./src --standard=./dev/tests/static/Magento --ignore=src/Magento/FunctionalTestingFramework/Group,src/Magento/FunctionalTestingFramework/AcceptanceTester.php
 vendor/bin/phpcs ./dev/tests/unit --standard=./dev/tests/static/Magento
 vendor/bin/phpcs ./dev/tests/verification --standard=./dev/tests/static/Magento --ignore=dev/tests/verification/_generated
 echo ""
@@ -22,7 +22,7 @@ echo ""
 echo "==============================="
 echo "       MESS DETECTOR"
 echo "==============================="
-vendor/bin/phpmd ./src text /dev/tests/static/Magento/CodeMessDetector/ruleset.xml --exclude _generated
+vendor/bin/phpmd ./src text /dev/tests/static/Magento/CodeMessDetector/ruleset.xml --exclude _generated,src/Magento/FunctionalTestingFramework/Group,src/Magento/FunctionalTestingFramework/AcceptanceTester.php
 echo ""
 
 echo "==============================="

bin/static-checks.bat

@@ -5,15 +5,15 @@
 
 @echo off
 @echo ===============================PHP CODE SNIFFER REPORT===============================
-call vendor\bin\phpcs .\src --standard=.\dev\tests\static\Magento
+call vendor\bin\phpcs .\src --standard=.\dev\tests\static\Magento --ignore=src\Magento\FunctionalTestingFramework\Group,src\Magento\FunctionalTestingFramework\AcceptanceTester.php
 call vendor\bin\phpcs .\dev\tests\unit --standard=.\dev\tests\static\Magento
 call vendor\bin\phpcs .\dev\tests\verification --standard=.\dev\tests\static\Magento --ignore=dev\tests\verification\_generated
 
 @echo ===============================COPY PASTE DETECTOR REPORT===============================
 call vendor\bin\phpcpd .\src
 
 @echo "===============================PHP MESS DETECTOR REPORT===============================
-vendor\bin\phpmd .\src text \dev\tests\static\Magento\CodeMessDetector\ruleset.xml --exclude _generated
+vendor\bin\phpmd .\src text \dev\tests\static\Magento\CodeMessDetector\ruleset.xml --exclude _generated,src\Magento\FunctionalTestingFramework\Group,src\Magento\FunctionalTestingFramework\AcceptanceTester.php
 
 @echo ===============================MAGENTO COPYRIGHT REPORT===============================
 echo msgbox "INFO:Copyright check currently not run as part of .bat implementation" > "%temp%\popup.vbs"

bootstrap.php

@@ -0,0 +1,8 @@
+<?php
+/**
+ * Copyright © Magento, Inc. All rights reserved.
+ * See COPYING.txt for license details.
+ */
+
+require_once 'vendor/autoload.php';
+define('BP', __DIR__);