Skip to content

Moving MFTF docs into this repo. #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Mar 21, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# CODEOWNERS file for /docs/ folder.
# Forces a review from other writers for anything within /docs/.
/docs/ @magento/devdocs-admins
2 changes: 1 addition & 1 deletion .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ Label| Description
[create issue]: https://help.github.com/articles/creating-an-issue/
[create pr]: https://help.github.com/articles/creating-a-pull-request/
[Definition of Done]: https://devdocs.magento.com/guides/v2.2/contributor-guide/contributing_dod.html
[devdocs]: https://github.com/magento/devdocs/blob/master/.github/CONTRIBUTING.md
[devdocs]: https://github.com/magento/devdocs/blob/master/.github/CONTRIBUTING.html
[existing issues]: https://github.com/magento/magento2-functional-testing-framework/issues?q=is%3Aopen+is%3Aissue
[existing PRs]: https://github.com/magento/magento2-functional-testing-framework/pulls?q=is%3Aopen+is%3Apr
[GitHub documentation]: https://help.github.com/articles/syncing-a-fork
Expand Down
27 changes: 19 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@

## Installation

For the installation guidelines and system requirements, refer to [Getting Started](https://devdocs.magento.com/mftf/2.3/getting-started.html).
For the installation guidelines and system requirements, refer to [Getting Started][].

## Contributing

We would appreciate your contributions to new components or new features, changes to the existing features, tests, documentation, specifications, bug fixes, optimizations, or just good suggestions.
Report about an issue or request features opening a GitHub issue.
Learn more about contributing in our [Contribution Guidelines](.github/CONTRIBUTING.md).
Learn more about contributing in our [Contribution Guidelines][].

If you want to participate in the documentation work, see [DevDocs Contributing](https://github.com/magento/devdocs/blob/master/.github/CONTRIBUTING.md).
If you want to participate in the documentation work, see [DevDocs Contributing][].

### Labels applied by the MFTF team

Expand Down Expand Up @@ -55,15 +55,26 @@ These labels are applied by the MFTF development team to community contributed i

## Reporting security issues

To report security vulnerabilities and other security issues in the Magento software or web sites, send an email with the report at [security@magento.com](mailto:security@magento.com).
To report security vulnerabilities and other security issues in the Magento software or web sites, send an email with the report at [security@magento.com][].
Do not report security issues using GitHub.
Be sure to encrypt your e-mail with our [encryption key](https://info2.magento.com/rs/magentoenterprise/images/security_at_magento.asc) if it includes sensitive information.
Learn more about reporting security issues [here](https://magento.com/security/reporting-magento-security-issue).
Be sure to encrypt your e-mail with our [encryption key][] if it includes sensitive information.
Learn more about reporting security issues [here][].

Stay up-to-date on the latest security news and patches for Magento by signing up for [Security Alert Notifications](https://magento.com/security/sign-up).
Stay up-to-date on the latest security news and patches for Magento by signing up for [Security Alert Notifications][].

## License

Each Magento source file included in this distribution is licensed under AGPL 3.0.

See the license [here](LICENSE_AGPL3.txt) or contact [license@magentocommerce.com](mailto:license@magentocommerce.com) for a copy.
See the license [here][] or contact [license@magentocommerce.com][] for a copy.

<!-- Link Definitions -->
[Getting Started]: https://devdocs.magento.com/mftf/getting-started.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert to relative link

[Contribution Guidelines]: .github/CONTRIBUTING.html
[DevDocs Contributing]: https://github.com/magento/devdocs/blob/master/.github/CONTRIBUTING.md
[security@magento.com]: mailto:security@magento.com
[encryption key]: https://info2.magento.com/rs/magentoenterprise/images/security_at_magento.asc
[here]: https://magento.com/security/reporting-magento-security-issue
[Security Alert Notifications]: https://magento.com/security/sign-up
[here]: LICENSE_AGPL3.txt
[license@magentocommerce.com]: mailto:license@magentocommerce.com
2 changes: 1 addition & 1 deletion composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

166 changes: 166 additions & 0 deletions docs/best-practices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Best practices

Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework.

## Action group

1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used.
Add additional explanation in comments if needed.
2. Provide default values for the arguments that apply to your most common case scenarios.

## `actionGroups` vs `extends`

Use an action group to wraps a set of actions to reuse them multiple times.

Use an [extension] when a test or action group needs to be repeated with the exception of a few steps.

### When to use `extends`

Use `extends` in your new test or action group when at least one of the following conditions is applicable to your case:

1. You want to keep the original test without any modifications.
2. You want to create a new test that follows the same path as the original test.
3. You want a new action group that behaves similarly to the existing action group, but you do not want to change the functionality of the original action group.

### When to avoid `extends`

Do not use `extends` in the following conditions:

1. You want to change the functionality of the test or action group and do not need to run the original version.
2. You plan to merge the base test or action group.

The following pattern is used when merging with `extends`:

1. The original test is merged.
2. The extended test is created from the merged original test.
3. The extended test is merged.

## Annotation

1. Use [annotations] in a test.
2. Update your annotations correspondingly when updating tests.

## Data entity

1. Keep your testing instance clean.
Remove data after the test if the test required creating any data.
Use a corresponding [`<deleteData>`] test step in your [`<after>`] block when using a [`<createData>`] action in a [`<before>`] block.
2. Make specific data entries under test to be unique.
Enable data uniqueness where data values are required to be unique in a database by test design.
Use `unique=”suffix”` or `unique=”prefix”` to append or prepend a unique value to the [entity] attribute.
This ensures that tests using the entity can be repeated.
3. Do not modify existing data entity fields or merge additional data fields without complete understanding and verifying the usage of existing data in tests.
Create a new data entity for your test if you are not sure.

## Naming conventions

### File names

Name files according to the following patterns to make searching in future more easy:

#### Test file name

Format: {_Admin_ or _Storefront_}{Functionality}_Test.xml_, where Functionality briefly describes the testing functionality.

Example: _StorefrontCreateCustomerTest.xml_.

#### Section file name

Format: {_Admin_ or _Storefront_}{UI Description}_Section.xml_, where UI Description briefly describes the testing UI.

Example: _AdminNavbarSection.xml_.

#### Data file name

Format: {Type}_Data.xml_, where Type represents the entity type.

Example: _ProductData.xml_.

### Object names

Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _classProperties_ in PHP.

#### Upper case

Use an upper case first letter for:

- File names. Example: _StorefrontCreateCustomerTest.xml_
- Test name attributes. Example: `<test name="TestAllTheThingsTest">`.
- Data entity names. Example: `<entity name="OutOfStockProduct">`.
- Page name. Example: `<page name="AdminLoginPage">`.
- Section name. Example: `<section name="AdminCategorySidebarActionSection">`.
- Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`.

#### Lower case

Use a lower case first letter for:

- Data keys. Example: `<data key="firstName">`.
- Element names. Examples: `<element name="confirmDeleteButton"/>`.

## Page object

Use [parameterized selectors] for constructing a selector when test specific or runtime generated information is needed.
Do not use them for static elements.

<span class="color:red">
BAD:
</span>

<!-- {% raw %} -->

``` xml
<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='{{productType}}']" parameterized="true"/>
```

<!-- {% endraw %} -->

<span class="color:green">
GOOD:
</span>

Define these three elements and reference them by name in the tests.

``` xml
<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='related']"/>
<element name="upSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='upsell']"/>
<element name="crossSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='crosssell']"/>
```

## Test

1. Use actions such as [`<waitForElementVisible>`], [`<waitForLoadingMaskToDisappear>`], and [`<waitForElement>`] to wait the exact time required for the test step.
Try to avoid using the [`<wait>`] action, because it forces the test to wait for the time you specify. You may not need to wait so long to proceed.
1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution.
It also helps you to identify the cause of test failure.
1. Use comments to keep tests readable and maintainable:
- Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
It helps to inform the reader of what you are testing and to yield a more descriptive Allure report.
- Explain in comments unclear or tricky test steps.
1. Refer to [sections] instead of writing selectors.

## Test step merging order

When setting a [merging] order for a test step, do not depend on steps from Magento modules that could be disabled by an application.

For example, when you write a test step to create a gift card product, set your test step **after** simple product creation and let the MFTF handle the merge order.
Since the configurable product module could be disabled, this approach is more reliable than setting the test step **before** creating a configurable product.

<!-- Link definitions -->

[`<after>`]: test/actions.html#before-and-after
[`<before>`]: test/actions.html#before-and-after
[`<comment>`]: test/actions.html#comment
[`<createData>`]: test/actions.html#createdata
[`<deleteData>`]: test/actions.html#deletedata
[`<wait>`]: test/actions.html#wait
[`<waitForElement>`]: test/actions.html#waitforelement
[`<waitForElementVisible>`]: test/actions.html#waitforelementvisible
[`<waitForLoadingMaskToDisappear>`]: test/actions.html#waitforloadingmasktodisappear
[Action group]: test/action-groups.html
[annotations]: test/annotations.html
[entity]: data.html
[extension]: extending.html
[merging]: merging.html
[parameterized selectors]: section/parameterized-selectors.html
[sections]: section.html
85 changes: 85 additions & 0 deletions docs/commands/codeception.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# CLI commands: vendor/bin/codecept

<div class="bs-callout bs-callout-warning" markdown="1">
We do not recommend using Codeception commands directly as they can break the MFTF basic workflow.
All the Codeception commands you need are wrapped using the [mftf tool][].

To run the Codeception testing framework commands directly, change your directory to the `<Magento root>`.
</div>

## Usage examples

Run all the generated tests:

```bash
../../../vendor/bin/codecept run functional -c dev/tests/acceptance/codeception.yml
```

Run all tests without the `<group value="skip"/>` [annotation][]:

```bash
../../../vendor/bin/codecept run functional --skip-group skip -c dev/tests/acceptance/codeception.yml
```

Run all tests with the `<group value="example"/>` [annotation][] but with no `<group value="skpip"/>`:

```bash
../../../vendor/bin/codecept run functional --group example --skip-group skip -c dev/tests/acceptance/codeception.yml
```

## `codecept run`

`codecept run` runs the test suites:

```bash
../../../vendor/bin/codecept run
```

<div class="bs-callout bs-callout-info">
The following documentation corresponds to Codeception 2.3.8.
</div>

```bash
Full reference:

Arguments:
suite suite to be tested
test test to be run

Options:
-o, --override=OVERRIDE Override config values (multiple values allowed)
--config (-c) Use custom path for config
--report Show output in compact style
--html Generate html with results (default: "report.html")
--xml Generate JUnit XML Log (default: "report.xml")
--tap Generate Tap Log (default: "report.tap.log")
--json Generate Json Log (default: "report.json")
--colors Use colors in output
--no-colors Force no colors in output (useful to override config file)
--silent Only outputs suite names and final results
--steps Show steps in output
--debug (-d) Show debug and scenario output
--coverage Run with code coverage (default: "coverage.serialized")
--coverage-html Generate CodeCoverage HTML report in path (default: "coverage")
--coverage-xml Generate CodeCoverage XML report in file (default: "coverage.xml")
--coverage-text Generate CodeCoverage text report in file (default: "coverage.txt")
--coverage-phpunit Generate CodeCoverage PHPUnit report in file (default: "coverage-phpunit")
--no-exit Do not finish with exit code
--group (-g) Groups of tests to be executed (multiple values allowed)
--skip (-s) Skip selected suites (multiple values allowed)
--skip-group (-x) Skip selected groups (multiple values allowed)
--env Run tests in selected environments. (multiple values allowed, environments can be merged with ',')
--fail-fast (-f) Stop after first failure
--help (-h) Display this help message.
--quiet (-q) Do not output any message.
--verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
--version (-V) Display this application version.
--ansi Force ANSI output.
--no-ansi Disable ANSI output.
--no-interaction (-n) Do not ask any interactive question.
```

<!-- Link definitions -->

[mftf tool]: mftf.html
[annotation]: ../test/annotations.html
Loading