-
Notifications
You must be signed in to change notification settings - Fork 132
Moving MFTF docs into this repo. #316
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
7609658
Merge pull request #1 from magento/develop
dshevtsov 2a8c17a
Adding docs folder.
dobooth 7f0207a
Change a lot of code as we move MFTF docs to new repo.
dobooth 0698d92
Updated images to markdown.
dobooth 705236c
Replaced {% raw %} blocks.
dobooth 540a3b5
Removed version note.
dobooth 758de90
Fixed hrefs that pointed to mftf/2.3/ path.
dobooth f6efc8c
<span> to <div> since the latter is block element.
dobooth 5facbb3
Close %raw% tag.
dobooth 70fe5a7
Dupe %raw% tag removed.
dobooth b7dd56b
Fixing links and HTML.
dobooth 5a9b559
Small fixes.
dobooth c7e17b9
Adding CODEOWNERS file for git review control.
dobooth 537ffe8
Move to hidden folder.
dobooth 76a4c58
Add files.
dobooth 26df2fa
Merge branch 'develop' into db_mftfdocs
okolesnyk f901e4f
Adding merge point docs.
dobooth 551e925
Switched .md links to .html
dobooth cb395ff
Merge pull request #2 from magento-devdocs/db_MFTF_extensionpoints
dobooth d952d0c
Fixed codeception paths.
dobooth ac14389
Merge branch 'db_mftfdocs' of github.com:magento-devdocs/magento2-fun…
dobooth cb5a2ba
Added flag per Pangolins
dobooth File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# CODEOWNERS file for /docs/ folder. | ||
# Forces a review from other writers for anything within /docs/. | ||
/docs/ @magento/devdocs-admins |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,166 @@ | ||
# Best practices | ||
|
||
Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework. | ||
|
||
## Action group | ||
|
||
1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. | ||
Add additional explanation in comments if needed. | ||
2. Provide default values for the arguments that apply to your most common case scenarios. | ||
|
||
## `actionGroups` vs `extends` | ||
|
||
Use an action group to wraps a set of actions to reuse them multiple times. | ||
|
||
Use an [extension] when a test or action group needs to be repeated with the exception of a few steps. | ||
|
||
### When to use `extends` | ||
|
||
Use `extends` in your new test or action group when at least one of the following conditions is applicable to your case: | ||
|
||
1. You want to keep the original test without any modifications. | ||
2. You want to create a new test that follows the same path as the original test. | ||
3. You want a new action group that behaves similarly to the existing action group, but you do not want to change the functionality of the original action group. | ||
|
||
### When to avoid `extends` | ||
|
||
Do not use `extends` in the following conditions: | ||
|
||
1. You want to change the functionality of the test or action group and do not need to run the original version. | ||
2. You plan to merge the base test or action group. | ||
|
||
The following pattern is used when merging with `extends`: | ||
|
||
1. The original test is merged. | ||
2. The extended test is created from the merged original test. | ||
3. The extended test is merged. | ||
|
||
## Annotation | ||
|
||
1. Use [annotations] in a test. | ||
2. Update your annotations correspondingly when updating tests. | ||
|
||
## Data entity | ||
|
||
1. Keep your testing instance clean. | ||
Remove data after the test if the test required creating any data. | ||
Use a corresponding [`<deleteData>`] test step in your [`<after>`] block when using a [`<createData>`] action in a [`<before>`] block. | ||
2. Make specific data entries under test to be unique. | ||
Enable data uniqueness where data values are required to be unique in a database by test design. | ||
Use `unique=”suffix”` or `unique=”prefix”` to append or prepend a unique value to the [entity] attribute. | ||
This ensures that tests using the entity can be repeated. | ||
3. Do not modify existing data entity fields or merge additional data fields without complete understanding and verifying the usage of existing data in tests. | ||
Create a new data entity for your test if you are not sure. | ||
|
||
## Naming conventions | ||
|
||
### File names | ||
|
||
Name files according to the following patterns to make searching in future more easy: | ||
|
||
#### Test file name | ||
|
||
Format: {_Admin_ or _Storefront_}{Functionality}_Test.xml_, where Functionality briefly describes the testing functionality. | ||
|
||
Example: _StorefrontCreateCustomerTest.xml_. | ||
|
||
#### Section file name | ||
|
||
Format: {_Admin_ or _Storefront_}{UI Description}_Section.xml_, where UI Description briefly describes the testing UI. | ||
|
||
Example: _AdminNavbarSection.xml_. | ||
|
||
#### Data file name | ||
|
||
Format: {Type}_Data.xml_, where Type represents the entity type. | ||
|
||
Example: _ProductData.xml_. | ||
|
||
### Object names | ||
|
||
Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _classProperties_ in PHP. | ||
|
||
#### Upper case | ||
|
||
Use an upper case first letter for: | ||
|
||
- File names. Example: _StorefrontCreateCustomerTest.xml_ | ||
- Test name attributes. Example: `<test name="TestAllTheThingsTest">`. | ||
- Data entity names. Example: `<entity name="OutOfStockProduct">`. | ||
- Page name. Example: `<page name="AdminLoginPage">`. | ||
- Section name. Example: `<section name="AdminCategorySidebarActionSection">`. | ||
- Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`. | ||
|
||
#### Lower case | ||
|
||
Use a lower case first letter for: | ||
|
||
- Data keys. Example: `<data key="firstName">`. | ||
- Element names. Examples: `<element name="confirmDeleteButton"/>`. | ||
|
||
## Page object | ||
|
||
Use [parameterized selectors] for constructing a selector when test specific or runtime generated information is needed. | ||
Do not use them for static elements. | ||
|
||
<span class="color:red"> | ||
BAD: | ||
</span> | ||
|
||
<!-- {% raw %} --> | ||
|
||
``` xml | ||
<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='{{productType}}']" parameterized="true"/> | ||
``` | ||
|
||
<!-- {% endraw %} --> | ||
|
||
<span class="color:green"> | ||
GOOD: | ||
</span> | ||
|
||
Define these three elements and reference them by name in the tests. | ||
|
||
``` xml | ||
<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='related']"/> | ||
<element name="upSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='upsell']"/> | ||
<element name="crossSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='crosssell']"/> | ||
``` | ||
|
||
## Test | ||
|
||
1. Use actions such as [`<waitForElementVisible>`], [`<waitForLoadingMaskToDisappear>`], and [`<waitForElement>`] to wait the exact time required for the test step. | ||
Try to avoid using the [`<wait>`] action, because it forces the test to wait for the time you specify. You may not need to wait so long to proceed. | ||
1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution. | ||
It also helps you to identify the cause of test failure. | ||
1. Use comments to keep tests readable and maintainable: | ||
- Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date. | ||
It helps to inform the reader of what you are testing and to yield a more descriptive Allure report. | ||
- Explain in comments unclear or tricky test steps. | ||
1. Refer to [sections] instead of writing selectors. | ||
|
||
## Test step merging order | ||
|
||
When setting a [merging] order for a test step, do not depend on steps from Magento modules that could be disabled by an application. | ||
|
||
For example, when you write a test step to create a gift card product, set your test step **after** simple product creation and let the MFTF handle the merge order. | ||
Since the configurable product module could be disabled, this approach is more reliable than setting the test step **before** creating a configurable product. | ||
|
||
<!-- Link definitions --> | ||
|
||
[`<after>`]: test/actions.html#before-and-after | ||
[`<before>`]: test/actions.html#before-and-after | ||
[`<comment>`]: test/actions.html#comment | ||
[`<createData>`]: test/actions.html#createdata | ||
[`<deleteData>`]: test/actions.html#deletedata | ||
[`<wait>`]: test/actions.html#wait | ||
[`<waitForElement>`]: test/actions.html#waitforelement | ||
[`<waitForElementVisible>`]: test/actions.html#waitforelementvisible | ||
[`<waitForLoadingMaskToDisappear>`]: test/actions.html#waitforloadingmasktodisappear | ||
[Action group]: test/action-groups.html | ||
[annotations]: test/annotations.html | ||
[entity]: data.html | ||
[extension]: extending.html | ||
[merging]: merging.html | ||
[parameterized selectors]: section/parameterized-selectors.html | ||
[sections]: section.html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
# CLI commands: vendor/bin/codecept | ||
|
||
<div class="bs-callout bs-callout-warning" markdown="1"> | ||
We do not recommend using Codeception commands directly as they can break the MFTF basic workflow. | ||
All the Codeception commands you need are wrapped using the [mftf tool][]. | ||
|
||
To run the Codeception testing framework commands directly, change your directory to the `<Magento root>`. | ||
</div> | ||
|
||
## Usage examples | ||
|
||
Run all the generated tests: | ||
|
||
```bash | ||
../../../vendor/bin/codecept run functional -c dev/tests/acceptance/codeception.yml | ||
dobooth marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
Run all tests without the `<group value="skip"/>` [annotation][]: | ||
|
||
```bash | ||
../../../vendor/bin/codecept run functional --skip-group skip -c dev/tests/acceptance/codeception.yml | ||
dobooth marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
Run all tests with the `<group value="example"/>` [annotation][] but with no `<group value="skpip"/>`: | ||
|
||
```bash | ||
../../../vendor/bin/codecept run functional --group example --skip-group skip -c dev/tests/acceptance/codeception.yml | ||
dobooth marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
## `codecept run` | ||
|
||
`codecept run` runs the test suites: | ||
|
||
```bash | ||
../../../vendor/bin/codecept run | ||
dobooth marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
<div class="bs-callout bs-callout-info"> | ||
The following documentation corresponds to Codeception 2.3.8. | ||
</div> | ||
|
||
```bash | ||
Full reference: | ||
|
||
Arguments: | ||
suite suite to be tested | ||
test test to be run | ||
|
||
Options: | ||
-o, --override=OVERRIDE Override config values (multiple values allowed) | ||
--config (-c) Use custom path for config | ||
--report Show output in compact style | ||
--html Generate html with results (default: "report.html") | ||
--xml Generate JUnit XML Log (default: "report.xml") | ||
--tap Generate Tap Log (default: "report.tap.log") | ||
--json Generate Json Log (default: "report.json") | ||
--colors Use colors in output | ||
--no-colors Force no colors in output (useful to override config file) | ||
--silent Only outputs suite names and final results | ||
--steps Show steps in output | ||
--debug (-d) Show debug and scenario output | ||
--coverage Run with code coverage (default: "coverage.serialized") | ||
--coverage-html Generate CodeCoverage HTML report in path (default: "coverage") | ||
--coverage-xml Generate CodeCoverage XML report in file (default: "coverage.xml") | ||
--coverage-text Generate CodeCoverage text report in file (default: "coverage.txt") | ||
--coverage-phpunit Generate CodeCoverage PHPUnit report in file (default: "coverage-phpunit") | ||
--no-exit Do not finish with exit code | ||
--group (-g) Groups of tests to be executed (multiple values allowed) | ||
--skip (-s) Skip selected suites (multiple values allowed) | ||
--skip-group (-x) Skip selected groups (multiple values allowed) | ||
--env Run tests in selected environments. (multiple values allowed, environments can be merged with ',') | ||
--fail-fast (-f) Stop after first failure | ||
--help (-h) Display this help message. | ||
--quiet (-q) Do not output any message. | ||
--verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug | ||
--version (-V) Display this application version. | ||
--ansi Force ANSI output. | ||
--no-ansi Disable ANSI output. | ||
--no-interaction (-n) Do not ask any interactive question. | ||
``` | ||
|
||
<!-- Link definitions --> | ||
|
||
[mftf tool]: mftf.html | ||
[annotation]: ../test/annotations.html |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
convert to relative link