diff --git a/docs/best-practices.md b/docs/best-practices.md index 0b95b1c41..c9068a90b 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -2,11 +2,46 @@ Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework. +## Focus on reusability + +### Use existing Tests and resources + +Magento offers more than **3000** acceptance tests, **2500** [Action group]s, **750** Page declarations with more than **1500** Section definitions. +It is very probable that behaviour you want to test already exists as a Test or Action Group. +Instead of writing everything by yourself - use `extends` attribute to refer to existing element and customize it. + +**Reusable Resources** + +* Tests (reusable with `` argument) +* Action Group (reusable with including ``, or extending ``) +* Pages (reusable with reference `{{PageDefinition.url}}`) +* Sections (reusable with reference `{{SectionDefinition.elementDefinition}}`) +* Data Entities (reusable with reference `"` or extending ``) + +
+ +Avoid using resources that are marked as **Deprecated**. Usually there is a replacement provided for a deprecated resource. + +
+ +### Extract repetitive Actions + +Instead of writing a few of Tests that perform mostly the same actions, you should thing about [Action group] that is a container for repetitive Actions. +If each run needs different data, use `` to inject necessary information. + +We recommend to keep Action Groups having single responsibility, for example `AdminLoginActionGroup`, which expected outcome is being logged in as Administrator when [Action group] is executed. + +## Contribute + +Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. +If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute. + +You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers. + ## Action group -1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. - Add additional explanation in annotations if needed. -2. Provide default values for the arguments that apply to your most common case scenarios. +1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed. +1. Provide default values for the arguments that apply to your most common case scenarios. ## `actionGroups` vs `extends` @@ -94,20 +129,20 @@ Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _cl Use an upper case first letter for: -- File names. Example: _StorefrontCreateCustomerTest.xml_ -- Test name attributes. Example: `` -- Data entity names. Example: `` -- Page name. Example: `` -- Section name. Example: `
` -- Action group name. Example: `` +* File names. Example: _StorefrontCreateCustomerTest.xml_ +* Test name attributes. Example: `` +* Data entity names. Example: `` +* Page name. Example: `` +* Section name. Example: `
` +* Action group name. Example: `` #### Lower case Use a lower case first letter for: -- Data keys. Example: `` -- Element names. Examples: `` -- Step keys. For example: `` +* Data keys. Example: `` +* Element names. Examples: `` +* Step keys. For example: `` ## Page object @@ -145,9 +180,9 @@ Define these three elements and reference them by name in the tests. 1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution. It also helps you to identify the cause of test failure. 1. Use comments to keep tests readable and maintainable: - - Keep the inline `` and [``] tags up to date. + * Keep the inline `` and [``] tags up to date. It helps to inform the reader of what you are testing and to yield a more descriptive Allure report. - - Explain in comments unclear or tricky test steps. + * Explain in comments unclear or tricky test steps. 1. Refer to [sections] instead of writing selectors. ## Test step merging order @@ -175,3 +210,4 @@ Since the configurable product module could be disabled, this approach is more r [merging]: merging.html [parameterized selectors]: section/parameterized-selectors.html [sections]: section.html +[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration diff --git a/docs/configuration.md b/docs/configuration.md index 9466f2bcc..a7b81c7c9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,6 +32,16 @@ Example: MAGENTO_BACKEND_NAME=admin_12346 ``` +### MAGENTO_BACKEND_BASE_URL + +(Optional) If you are running the Admin Panel on separate a domain, specify this value: + +Example: + +```conf +MAGENTO_BACKEND_BASE_URL=https://admin.magento2.test +``` + ### MAGENTO_ADMIN_USERNAME The username that tests can use to access the Magento Admin page diff --git a/docs/img/mftf-extending-precedence.png b/docs/img/mftf-extending-precedence.png new file mode 100644 index 000000000..deed5231f Binary files /dev/null and b/docs/img/mftf-extending-precedence.png differ diff --git a/docs/introduction.md b/docs/introduction.md index 61c1eaef3..53b763d8c 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -1,9 +1,14 @@ # Introduction to the Magento Functional Testing Framework -[Find your MFTF version][] of the MFTF. +The Magento Functional Tesitng Framework (MFTF) is a framework used to perform automated end-to-end functional testing. -The Magento Functional Testing Framework (MFTF) aims to replace the [Functional Testing Framework] in future releases. -MFTF improves: +## Goals + +- To facilitate functional testing and minimize the effort it takes to perform regression testing. +- Enable extension developers to provide the Functional Tests to offered extensions. +- Ensuring a common standard of quality between Magento, Extension Developers and System Intergrators. + +MFTF also focuses on - **Traceability** for clear logging and reporting capabilities. - **Modularity** to run tests based on installed modules and extensions. @@ -11,39 +16,64 @@ MFTF improves: - **Readability** using clear and declarative XML test steps. - **Maintainability** based on simple test creation and overall structure. -Because MFTF tests are written in XML, you no longer need to learn PHP to write tests. +## Audience -
-We are actively developing functional tests. -Refer to `/app/code///Test/Mftf/` for examples. -Check out the [MFTF Test Migration][] repo. -
+- **Contributors**: Tests build confidence about the results of changes introduced to the platform. +- **Extension Developers**: Can adjust expected behaviour according to their customizations. +- **System Integrators**: MFTF coverage provided out-of-the-box with Magento is solid base for Acceptance / Regression Tests. -## Audience +## MFTF tests + +The MFTF supports two different locations for storing the tests and test artifacts: -This MFTF guide is intended for Magento developers and software engineers, such as QA specialists, PHP developers, and system integrators. +- `/app/code///Test/Mftf/` is the location of local, customized tests. +- `/vendor///Test/Mftf/` is location of tests provided by Magento and vendors. -## Goals +If you installed Magento with Composer, please refer to `vendor/magento//Test/Mftf/` for examples. + +### Directory Structure -The purpose of MFTF is to: +The file structure under both of the both path cases is the same: -- Facilitate functional testing and minimize the effort it takes to perform regression testing. -- Make it easier to support the extension and customization of tests via XML merging. +```tree +Test +└── Mftf + ├── ActionGroup + │   └── ... + ├── Data + │   └── ... + ├── Metadata + │   └── ... + ├── Page + │   └── ... + ├── Section + │   └── ... + └── Test + └── ... +``` -## Scope +
-MFTF will enable you to: +We are actively developing functional tests. Check out the [MFTF Test Migration][] repository. -- Test user interactions with web applications in testing. -- Write functional tests located in `/app/code///Test/Mftf/`. -- Cover basic functionality using out-of-the-box tests. You can test extended functionality using custom tests. -- Automate regression testing. +
## Use cases -As a Magento developer, test changes, such as extended search functionality, a new form attribute, or new product tags. +- Contributor: changes the core behaviour, fixing the annoing bug. + He wants to have automated "supervisor" which is going to verify his work continuously across the stages of bug fixing. Finally, when fix is done - Functional Test is also proof of work done. +- Extension Developer: offers extension that changes core behaviour. + He can easily write new tests to make sure that after enabling the feature, Magento behaves properly. Everything with just extending existing tests. As a result he don't need to write coverage from scratch. +- Integration Agency: maintains Client's e-commerce. + They are able to customize tests delivered with Magento core to follow customizations implemented to Magento. After each upgrade they can just run the MFTF tests to know that no regression was introduced. + +## MFTF output -As a software engineer, perform regression testing before release to ensure that Magento works as expected with new functionality. +- Generated PHP Codeception tests +- Codeception results and console logs +- Screenshots and HTML failure report +- Allure formatted XML results +- Allure report dashboard of results ## Find your MFTF version @@ -52,11 +82,9 @@ There are two options to find out your MFTF version: - using the MFTF CLI - using the Composer CLI -### MFTF CLI +All the Command Line commands needs to be executed from `` -```bash -cd / -``` +### MFTF CLI ```bash vendor/bin/mftf --version @@ -64,10 +92,6 @@ vendor/bin/mftf --version ### Composer CLI -```bash -cd / -``` - ```bash composer show magento/magento2-functional-testing-framework ``` @@ -89,41 +113,6 @@ utils // The test-running utilities. codeception.dist.yml // Codeception configuration (generated while running 'bin/mftf build:project') ``` -## MFTF output - -- Generated PHP Codeception tests -- Codeception results and console logs -- Screenshots and HTML failure report -- Allure formatted XML results -- Allure report dashboard of results - -## MFTF tests - -The MFTF supports two different locations for storing the tests and test artifacts: - -- `/app/code///Test/Mftf/` is the directory to create new tests. -- `/vendor///Test/Mftf/` is the directory with the out of the box tests (fetched by the Composer). - -All tests and test data from these locations are merged in the order indicated in the above list. - -The file structure under the both path cases is the same: - -```tree - -├── ActionGroup -│   └── ... -├── Data -│   └── ... -├── Metadata -│   └── ... -├── Page -│   └── ... -├── Section -│   └── ... -└── Test - └── ... -``` - ## MFTF on Github Follow the [MFTF project] and [contribute on Github]. diff --git a/docs/merging.md b/docs/merging.md index e4b91fb5f..9cb48c41f 100644 --- a/docs/merging.md +++ b/docs/merging.md @@ -27,6 +27,15 @@ For example: Although a file name does not influence merging, we recommend using the same file names in merging updates. This makes it easier to search later on. +## Merging precedence + +**Magento Functional Testing Framework** uses Module's `` to merge all XML configurations into Codeception instructions. If there's no Sequence specified, MFTF would use: + +1. Vendor modules (Magento & Vendors) located in `vendor/` +1. Tests located in `app/code/*/*/Test/Mftf` + +![Usual precedence for merging MFTF Tests and resources][mftfExtendingPrecedence image] + ## Add a test You cannot add another [``][tests] using merging functionality. @@ -570,3 +579,4 @@ The `_defaultSample` results corresponds to: [``]: ./section.md [``]: ./test.md [``]: ./test/action-groups.md +[mftfExtendingPrecedence image]: img/mftf-extending-precedence.png diff --git a/docs/update.md b/docs/update.md index 129742346..ebd096158 100644 --- a/docs/update.md +++ b/docs/update.md @@ -1,79 +1,43 @@ # Update the Magento Functional Testing Framework
-[Find your version][] of the MFTF. - -The latest Magento 2.3 release supports MFTF 2.3.13. -The latest Magento 2.2 release supports MFTF 2.3.8. +Both Magento `2.2` and `2.3` supports MFTF `2.5.3` ([Find your version][] of the MFTF).
-Magento tests and the framework are stored in different repositories. - -Magento tests are stored in the same repository as the Magento code base. -When you pull changes in the Magento code, you're potentially pulling corresponding tests as well. - -The MFTF is installed separately as a dependency using Composer. -When pulling the latest Magento code, update the corresponding Composer dependencies in the `magento2` root directory. -This ensures that the MFTF is up to date. - -## Update the MFTF from 2.3.x - -To update the MFTF to the latest patch: - -1. Verify that the Magento [WYSIWYG settings][] and [Security settings][] are set appropriately. -1. Check details about backward incompatible changes in the [Changelog][] and update your new or customized tests. -1. Get the latest framework version using Composer: - - ```bash - composer update - ``` +Tests and the Framework itself are stored in different repositories. -1. Generate the updated tests: +* Tests are stored in Module's directory. +* MFTF is installed separately (usually as a Composer dependency) - ```bash - vendor/bin/mftf generate:tests - ``` +To understand different types of update - please follow the [Versioning][] page. -## Update the MFTF from 2.2 +## Patch version update -To update the MFTF from the previous minor version: +Takes place when **third** digit of version number changes. -1. When you update Magento, verify that the Magento [WYSIWYG settings][] and [Security settings][] are set appropriately. -1. Starting at the `magento2/` root directory remove the `vendor/` directory: +1. Make sure that [Security settings][] are set appropriately. +1. Get latest Framework version with `composer update magento/magento2-functional-testing-framework --with-dependencies` +1. Generate updated tests with `vendor/bin/mftf generate:tests` - ```bash - rm -rf vendor/ - ``` +## Minor version update -1. Get the latest framework version from the Composer dependencies: +Takes place when **second** digit of version number changes. - ```bash - composer install - ``` - -1. Run the `upgrade:tests` using the new command line tool: - - ```bash - vendor/bin/mftf upgrade:tests app - ``` - -1. If you are using Phpstorm, update the urn catalog: - - ```bash - vendor/bin/mftf generate:urn-catalog .idea/ - ``` - -1. Update your own tests, including data, metadata, and so on, if they contain tags that are unsupported in the newer version. - - Check details about backward incompatible changes and review new MFTF release documentation in the [Changelog][]. +1. Check details about backward incompatible changes in the [Changelog][] and update your new or customized tests. +1. Perform all the actions provided for [Patch Version Update][] +1. When updating from versions below `2.5.0`, verify [WYSIWYG settings][] +1. You may need to run the `upgrade:tests` using `vendor/bin/mftf upgrade:tests app` -1. Generate newly pulled tests: +## After updating - ```bash - vendor/bin/mftf generate:tests - ``` +1. It is a good idea to regenerate your IDE Schema Definition catalog with `vendor/bin/mftf generate:urn-catalog .idea/` +1. Update your tests, including data, metadata and other resoruces. Check if they contain tags that are unsupported in the newer version. +1. Remove the references to resources (ActionGroups, Sections, Tests) marked as deprecated. [Changelog]: https://github.com/magento/magento2-functional-testing-framework/blob/master/CHANGELOG.md [WYSIWYG settings]: getting-started.md#wysiwyg-settings [Security settings]: getting-started.md#security-settings +[Find your version]: introduction.md#find-your-mftf-version +[Versioning]: versioning.md#versioning-policy +[Patch Version Update]: #patch-version-update diff --git a/docs/versioning.md b/docs/versioning.md index ecb8438c5..15b98f743 100644 --- a/docs/versioning.md +++ b/docs/versioning.md @@ -12,7 +12,6 @@ If a modification to MFTF forces tests to be changed, this is a backward incompa To find the version of MFTF that you are using, run the Magento CLI command: ```bash -cd / vendor/bin/mftf --version ``` @@ -61,8 +60,10 @@ You must reset the patch and minor version to 0 when you change the major versio This table lists the version of the MFTF that was released with a particular version of Magento. |Magento version| MFTF version| -|---|---| +|--- |--- | +| 2.3.4 | 2.5.3 | +| 2.3.3 | 2.4.5 | | 2.3.2 | 2.3.14 | | 2.3.1 | 2.3.13 | -| 2.3.0 | 2.3.9 | +| 2.3.0 | 2.3.9 | | 2.2.8 | 2.3.13 |