MQE-2210: Merge DevDoc changes in master back into MFTF release branch (#749)

tomreece · karyna-t · okolesnyk · web-flow · commit d839076051da · 2020-07-08T15:01:52.000-05:00
* Small fix to the Mftf Getting Started doc

The command from the 'Set up an embedded MFTF -&gt; Step 3' has to be executed from project root.

* Update filter documentation with special characters to be able to show correctly in devdocs

* DevDocs: Test merging precedence

* Adding the Selenium server download in the example

* Getting started &gt; Removing code snippet

I removed an unnecessary code snippet which doesn't make sense where it's.

* DevDocs: Update Best Practices

* Add missing information about Backend Domain support.

* DevDocs: Update documentation regarding Versioning

* DevDocs: Update documentation regarding Update process

* Update Introduction for MFTF

* Editorial pass

* Escaped Liquid tag syntax

* Fix an issue with MimeType guesser and latest release of "symfony/http-foundation"

* Fix an issue with MimeType guesser and latest release of "symfony/http-foundation"

* MQE-2052: MFTF uses undeclared dependency

* MQE-2210: Merge DevDoc changes in master back into MFTF release branch

Co-authored-by: Karyna Tsymbal &lt;k.tsymbal@atwix.com&gt;
Co-authored-by: Alex Kolesnyk &lt;kolesnyk@adobe.com&gt;
Co-authored-by: Lukasz Bajsarowicz &lt;lukasz.bajsarowicz@gmail.com&gt;
Co-authored-by: Rafael Corrêa Gomes &lt;rafaelstz@users.noreply.github.com&gt;
Co-authored-by: Kevin Kozan &lt;kkozan@adobe.com&gt;
Co-authored-by: Donald Booth &lt;dobooth@adobe.com&gt;
Co-authored-by: Kevin Kozan &lt;kkozan@magento.com&gt;
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -129,6 +129,9 @@ Magento Functional Testing Framework Changelog
 2.6.3
 -----
 
+### Fixes
+* added dependency to packages MFTF used but never specified in composer.json
+
 ### New Feature
 * `--filter` option was added to `bin/mftf generate:tests` command. For more details please go to https://devdocs.magento.com/mftf/docs/commands/mftf.html#generatetests
 
diff --git a/docs/best-practices.md b/docs/best-practices.md
@@ -2,10 +2,49 @@
 
 Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework.
 
+## Focus on reusability
+
+### Use existing Tests and resources
+
+Magento offers more than **3000** acceptance tests, **2500** [Action group]s, **750** Page declarations with more than **1500** Section definitions.
+It is very probable that behaviour you want to test already exists as a Test or Action Group.
+Instead of writing everything by yourself - use `extends` attribute to refer to existing element and customize it.
+
+**Reusable Resources**
+
+{%raw%}
+
+*  Tests (reusable with `<test extends="...">` argument)
+*  Action Group (reusable with including `<actionGroup ref="...">`, or extending `<actionGroup extends="...">`)
+*  Pages (reusable with reference `{{PageDefinition.url}}`)
+*  Sections (reusable with reference `{{SectionDefinition.elementDefinition}}`)
+*  Data Entities (reusable with reference `<createData entity="...">"` or extending `<entity extends="...">`)
+
+{%endraw%}
+
+<div class="bs-callout bs-callout-warning" markdown="1">
+
+Avoid using resources that are marked as **Deprecated**. Usually there is a replacement provided for a deprecated resource.
+
+</div>
+
+### Extract repetitive Actions
+
+Instead of writing a few of Tests that perform mostly the same actions, you should thing about [Action group] that is a container for repetitive Actions.
+If each run needs different data, use `<arguments>` to inject necessary information.
+
+We recommend to keep Action Groups having single responsibility, for example `AdminLoginActionGroup`, which expected outcome is being logged in as Administrator when [Action group] is executed.
+
+## Contribute
+
+Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly.
+If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute.
+
+You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers.
+
 ## Action group
 
-1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used.
- Add additional explanation in annotations if needed.
+1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed.
 2. Provide default values for the arguments that apply to your most common case scenarios.
 3. One `<actionGroup>` tag is allowed per action group XML file.
 
@@ -95,20 +134,20 @@ Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _cl
 
 Use an upper case first letter for:
 
--  File names. Example: _StorefrontCreateCustomerTest.xml_
--  Test name attributes. Example: `<test name="TestAllTheThingsTest">`
--  Data entity names. Example: `<entity name="OutOfStockProduct">`
--  Page name. Example: `<page name="AdminLoginPage">`
--  Section name. Example: `<section name="AdminCategorySidebarActionSection">`
--  Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`
+*  File names. Example: _StorefrontCreateCustomerTest.xml_
+*  Test name attributes. Example: `<test name="TestAllTheThingsTest">`
+*  Data entity names. Example: `<entity name="OutOfStockProduct">`
+*  Page name. Example: `<page name="AdminLoginPage">`
+*  Section name. Example: `<section name="AdminCategorySidebarActionSection">`
+*  Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`
 
 #### Lower case
 
 Use a lower case first letter for:
 
--  Data keys. Example: `<data key="firstName">`
--  Element names. Examples: `<element name="confirmDeleteButton"/>`
--  Step keys. For example: `<click selector="..." stepKey="clickLogin"/>`
+*  Data keys. Example: `<data key="firstName">`
+*  Element names. Examples: `<element name="confirmDeleteButton"/>`
+*  Step keys. For example: `<click selector="..." stepKey="clickLogin"/>`
 
 ## Page object
 
@@ -147,9 +186,9 @@ Define these three elements and reference them by name in the tests.
 1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution.
  It also helps you to identify the cause of test failure.
 1. Use comments to keep tests readable and maintainable:
-   -  Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
+   *  Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
      It helps to inform the reader of what you are testing and to yield a more descriptive Allure report.
-   -  Explain in comments unclear or tricky test steps.
+   *  Explain in comments unclear or tricky test steps.
 1. Refer to [sections] instead of writing selectors.
 1. One `<test>` tag is allowed per test XML file.
 
@@ -178,3 +217,4 @@ Since the configurable product module could be disabled, this approach is more r
 [merging]: merging.html
 [parameterized selectors]: section/parameterized-selectors.html
 [sections]: section.html
+[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration
diff --git a/docs/commands/mftf.md b/docs/commands/mftf.md
@@ -151,7 +151,7 @@ vendor/bin/mftf generate:tests [option] [<test name>] [<test name>] [--remove]
 | Option | Description|
 | ---| --- |
 | `--config=[<default> or <singleRun> or <parallel>]` | Creates a single manifest file with a list of all tests. The default location is `tests/functional/Magento/FunctionalTest/_generated/testManifest.txt`.<br/> You can split the list into multiple groups using `--config=parallel`; the groups will be generated in `_generated/groups/` like `_generated/groups/group1.txt, group2.txt, ...`.<br/> Available values: `default` (default), `singleRun`(same as `default`), and `parallel`.<br/> Example: `generate:tests --config=parallel`. |
-| `--filter` | Option to filter tests to be generated.<br/>Template: '<filterName>:<filterValue>'.<br/>Existing filter types: severity.<br/>Existing severity values: BLOCKER, CRITICAL, MAJOR, AVERAGE, MINOR.<br/>Example: --filter=severity:CRITICAL|
+| `--filter` | Option to filter tests to be generated.<br/>Template: '&lt;filterName&gt;:&lt;filterValue&gt;'.<br/>Existing filter types: severity.<br/>Existing severity values: BLOCKER, CRITICAL, MAJOR, AVERAGE, MINOR.<br/>Example: --filter=severity:CRITICAL|
 | `--force` | Forces test generation, regardless of the module merge order defined in the Magento instance. Example: `generate:tests --force`. |
 | `-i,--time` | Set time in minutes to determine the group size when `--config=parallel` is used. The __default value__ is `10`. Example: `generate:tests --config=parallel --time=15`|
 | `--tests` | Defines the test configuration as a JSON string.|
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -32,6 +32,16 @@ Example:
 MAGENTO_BACKEND_NAME=admin_12346
 ```
 
+### MAGENTO_BACKEND_BASE_URL
+
+(Optional) If you are running the Admin Panel on separate a domain, specify this value:
+
+Example:
+
+```conf
+MAGENTO_BACKEND_BASE_URL=https://admin.magento2.test
+```
+
 ### MAGENTO_ADMIN_USERNAME
 
 The username that tests can use to access the Magento Admin page
diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -204,7 +204,7 @@ Learn more about environmental settings in [Configuration][].
 
 ### Step 3. Enable the Magento CLI commands
 
-In the `magento2/dev/tests/acceptance` directory, run the following command to enable the MFTF to send Magento CLI commands to your Magento instance.
+In the Magento project root, run the following command to enable the MFTF to send Magento CLI commands to your Magento instance.
 
  ```bash
 cp dev/tests/acceptance/.htaccess.sample dev/tests/acceptance/.htaccess
@@ -216,11 +216,11 @@ To run tests, you need a running Selenium server and [`mftf`][] commands.
 
 #### Run the Selenium server {#selenium-server}
 
-Run the Selenium server in terminal.
-For example, the following commands run the Selenium server for Google Chrome:
+Run the Selenium server in the terminal.
+For example, the following commands download and run the Selenium server for Google Chrome:
 
 ```bash
-cd <path_to_directory_with_selenium_server_and_webdriver>/
+curl -O http://selenium-release.storage.googleapis.com/3.14/selenium-server-standalone-3.14.0.jar
 ```
 
 ```bash
@@ -233,10 +233,6 @@ java -Dwebdriver.chrome.driver=chromedriver -jar selenium-server-standalone-3.14
 vendor/bin/mftf generate:tests
 ```
 
-```bash
-cd dev/tests/acceptance
-```
-
 ```bash
 vendor/bin/codecept run functional -c dev/tests/acceptance/codeception.yml
 ```
diff --git a/docs/img/mftf-extending-precedence.png b/docs/img/mftf-extending-precedence.png
diff --git a/docs/introduction.md b/docs/introduction.md
@@ -7,48 +7,80 @@ To find older documentation, please refer to the [docs folder] of your desired r
 
 [Find your MFTF version][] of the MFTF.
 
-The Magento Functional Testing Framework (MFTF) aims to replace the [Functional Testing Framework] in future releases.
-MFTF improves:
+The Magento Functional Testing Framework (MFTF) is a framework used to perform automated end-to-end functional testing.
+
+## Goals
+
+-  To facilitate functional testing and minimize the effort it takes to perform regression testing.
+-  Enable extension developers to provide the Functional Tests to offered extensions.
+-  Ensuring a common standard of quality between Magento, Extension Developers and System Intergrators.
+
+MFTF also focuses on
 
 -  **Traceability** for clear logging and reporting capabilities.
 -  **Modularity** to run tests based on installed modules and extensions.
 -  **Customizability** for existing tests.
 -  **Readability** using clear and declarative XML test steps.
 -  **Maintainability** based on simple test creation and overall structure.
 
-Because MFTF tests are written in XML, you no longer need to learn PHP to write tests.
+## Audience
 
-<div class="bs-callout bs-callout-info" markdown="1">
-We are actively developing functional tests.
-Refer to `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/` for examples.
-Check out the [MFTF Test Migration][] repo.
-</div>
+-  **Contributors**: Tests build confidence about the results of changes introduced to the platform.
+-  **Extension Developers**: Can adjust expected behaviour according to their customizations.
+-  **System Integrators**: MFTF coverage provided out-of-the-box with Magento is solid base for Acceptance / Regression Tests.
 
-## Audience
+## MFTF tests
 
-This MFTF guide is intended for Magento developers and software engineers, such as QA specialists, PHP developers, and system integrators.
+The MFTF supports two different locations for storing the tests and test artifacts:
 
-## Goals
+-  `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/` is the location of local, customized tests.
+-  `<magento_root>/vendor/<vendor_name>/<module_name>/Test/Mftf/` is location of tests provided by Magento and vendors.
 
-The purpose of MFTF is to:
+If you installed Magento with Composer, please refer to `vendor/magento/<module_dir>/Test/Mftf/` for examples.
 
--  Facilitate functional testing and minimize the effort it takes to perform regression testing.
--  Make it easier to support the extension and customization of tests via XML merging.
+### Directory Structure
 
-## Scope
+The file structure under both of the both path cases is the same:
+
+```tree
+Test
+└── Mftf
+    ├── ActionGroup
+    │   └── ...
+    ├── Data
+    │   └── ...
+    ├── Metadata
+    │   └── ...
+    ├── Page
+    │   └── ...
+    ├── Section
+    │   └── ...
+    └── Test
+        └── ...
+```
+
+<div class="bs-callout bs-callout-info" markdown="1">
 
-MFTF will enable you to:
+We are actively developing functional tests. Check out the [MFTF Test Migration][] repository.
 
--  Test user interactions with web applications in testing.
--  Write functional tests located in `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/`.
--  Cover basic functionality using out-of-the-box tests. You can test extended functionality using custom tests.
--  Automate regression testing.
+</div>
 
 ## Use cases
 
-As a Magento developer, test changes, such as extended search functionality, a new form attribute, or new product tags.
+-  Contributor: changes the core behaviour, fixing the annoing bug.
+   He wants to have automated "supervisor" which is going to verify his work continuously across the stages of bug fixing. Finally, when fix is done - Functional Test is also proof of work done.
+-  Extension Developer: offers extension that changes core behaviour.
+   He can easily write new tests to make sure that after enabling the feature, Magento behaves properly. Everything with just extending existing tests. As a result he don't need to write coverage from scratch.
+-  Integration Agency: maintains Client's e-commerce.
+   They are able to customize tests delivered with Magento core to follow customizations implemented to Magento. After each upgrade they can just run the MFTF tests to know that no regression was introduced.
+
+## MFTF output
 
-As a software engineer, perform regression testing before release to ensure that Magento works as expected with new functionality.
+-  Generated PHP Codeception tests
+-  Codeception results and console logs
+-  Screenshots and HTML failure report
+-  Allure formatted XML results
+-  Allure report dashboard of results
 
 ## Find your MFTF version
 
@@ -57,22 +89,16 @@ There are two options to find out your MFTF version:
 -  using the MFTF CLI
 -  using the Composer CLI
 
-### MFTF CLI
+All the Command Line commands needs to be executed from `<magento_root>`
 
-```bash
-cd <magento_root>/
-```
+### MFTF CLI
 
 ```bash
 vendor/bin/mftf --version
 ```
 
 ### Composer CLI
 
-```bash
-cd <magento_root>/
-```
-
 ```bash
 composer show magento/magento2-functional-testing-framework
 ```
diff --git a/docs/merging.md b/docs/merging.md
@@ -27,6 +27,15 @@ For example:
 Although a file name does not influence merging, we recommend using the same file names in merging updates.
 This makes it easier to search later on.
 
+## Merging precedence
+
+**Magento Functional Testing Framework** uses Module's `<sequence>` to merge all XML configurations into Codeception instructions. If there's no Sequence specified, MFTF would use:
+
+1. Vendor modules (Magento & Vendors) located in `vendor/`
+1. Tests located in `app/code/*/*/Test/Mftf`
+
+![Usual precedence for merging MFTF Tests and resources][mftfExtendingPrecedence image]
+
 ## Add a test
 
 You cannot add another [`<test>`][tests] using merging functionality.
@@ -570,3 +579,4 @@ The `_defaultSample` results corresponds to:
 [`<sections>`]: ./section.md
 [`<tests>`]: ./test.md
 [`<action groups>`]: ./test/action-groups.md
+[mftfExtendingPrecedence image]: img/mftf-extending-precedence.png
diff --git a/docs/update.md b/docs/update.md
@@ -0,0 +1,43 @@
+# Update the Magento Functional Testing Framework
+
+<div class="bs-callout bs-callout-info" markdown="1">
+Both Magento `2.2` and `2.3` supports MFTF `2.5.3` ([Find your version][] of the MFTF).
+</div>
+
+Tests and the Framework itself are stored in different repositories.
+
+*  Tests are stored in Module's directory.
+*  MFTF is installed separately (usually as a Composer dependency)
+
+To understand different types of update - please follow the [Versioning][] page.
+
+## Patch version update
+
+Takes place when **third** digit of version number changes.
+
+1. Make sure that [Security settings][] are set appropriately.
+1. Get latest Framework version with `composer update magento/magento2-functional-testing-framework --with-dependencies`
+1. Generate updated tests with `vendor/bin/mftf generate:tests`
+
+## Minor version update
+
+Takes place when **second** digit of version number changes.
+
+1. Check details about backward incompatible changes in the [Changelog][] and update your new or customized tests.
+1. Perform all the actions provided for [Patch Version Update][]
+1. When updating from versions below `2.5.0`, verify [WYSIWYG settings][]
+1. You may need to run the `upgrade:tests` using `vendor/bin/mftf upgrade:tests app`
+
+## After updating
+
+1. It is a good idea to regenerate your IDE Schema Definition catalog with `vendor/bin/mftf generate:urn-catalog .idea/`
+1. Update your tests, including data, metadata and other resoruces. Check if they contain tags that are unsupported in the newer version.
+1. Remove the references to resources (ActionGroups, Sections, Tests) marked as deprecated.
+
+<!-- Link Definitions -->
+[Changelog]: https://github.com/magento/magento2-functional-testing-framework/blob/master/CHANGELOG.md
+[WYSIWYG settings]: getting-started.md#wysiwyg-settings
+[Security settings]: getting-started.md#security-settings
+[Find your version]: introduction.md#find-your-mftf-version
+[Versioning]: versioning.md#versioning-policy
+[Patch Version Update]: #patch-version-update
diff --git a/docs/versioning.md b/docs/versioning.md
@@ -12,7 +12,6 @@ If a modification to MFTF forces tests to be changed, this is a backward incompa
 To find the version of MFTF that you are using, run the Magento CLI command:
 
 ```bash
-cd <magento_root>/
 vendor/bin/mftf --version
 ```
 
@@ -61,8 +60,10 @@ You must reset the patch and minor version to 0 when you change the major versio
 This table lists the version of the MFTF that was released with a particular version of Magento.
 
 |Magento version| MFTF  version|
-|---|---|
+|---    |---     |
+| 2.3.4 | 2.5.3  |
+| 2.3.3 | 2.4.5  |
 | 2.3.2 | 2.3.14 |
 | 2.3.1 | 2.3.13 |
-| 2.3.0 | 2.3.9 |
+| 2.3.0 | 2.3.9  |
 | 2.2.8 | 2.3.13 |
