Skip to content

Commit 17082a1

Browse files
authored
Merge pull request #462 from magento/MasterToDevelop
Merge master to develop
2 parents e850699 + e447ac0 commit 17082a1

19 files changed

+327
-93
lines changed

CHANGELOG.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,47 @@
11
Magento Functional Testing Framework Changelog
22
================================================
33

4+
2.5.0
5+
-----
6+
* Traceability
7+
* Allure output has been enhanced to contain new test artifacts created and used per MFTF step:
8+
* `makeScreenshot` will contain an attachment under its Allure step.
9+
* `seeInCurrentUrl` and all other `Url` asserts now contain an attachment with the expects vs actual comparison.
10+
* `createData` and all other `Data` actions now contain attachments with `Request Body` and `Response Body`.
11+
* Modularity
12+
* Added a new `mftf run:manifest` command to run testManifest files generated by `generate:tests`.
13+
* See DevDocs for details
14+
* `mftf generate/run:test` commands now implicitly generates the `suite` the test exists in.
15+
* If a test exists in multiple suites, it will generate it in all suite contexts.
16+
* `mftf run:test <testName>` will now only run the exact test provided, regardless of what is generated.
17+
* Maintainability
18+
* Added an `--allow-skipped` flag that allows MFTF to ignore the `<skip>` annotation. This was added to the following commands:
19+
* `generate:test`
20+
* `run:test`
21+
* `run:group`
22+
* `run:failed`
23+
* Customizability
24+
* `<entity>` defined in data.xml can now reference other `<entity>` directly.
25+
* See DevDocs for details
26+
* Added vault as an alternative credential storage.
27+
* See DevDocs for details
28+
29+
### Fixes
30+
* Fixed an issue where `grab` action variables were not substituting correctly when used as an element parameter.
31+
* Framework will not throw a descriptive error when referencing a `$persisted.field$` that does not exist.
32+
* MFTF test materials that `extends=""` itself will no longer cause infinite recursion.
33+
* Fixed an issue where a test could not reference a `$data.field$` whose casing was modified by the API that it used.
34+
* Fixed an issue with the default `functional.suite.yml` where it was incompatible with `symfony/yaml 4.0.0`.
35+
* Improved test generation performance via class refactors (`~10%` faster).
36+
37+
### GitHub Issues/Pull requests:
38+
* [#377](https://github.com/magento/magento2-functional-testing-framework/pull/377) -- Non-API operations fixes
39+
40+
2.4.5
41+
-----
42+
### Fixes
43+
* Fixed an issue where `.credentials` was required when using `<createData>` actions with field overrides.
44+
445
2.4.4
546
-----
647
### Fixes

bin/mftf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ try {
2929
try {
3030
$application = new Symfony\Component\Console\Application();
3131
$application->setName('Magento Functional Testing Framework CLI');
32-
$application->setVersion('2.4.4');
32+
$application->setVersion('2.5.0');
3333
/** @var \Magento\FunctionalTestingFramework\Console\CommandListInterface $commandList */
3434
$commandList = new \Magento\FunctionalTestingFramework\Console\CommandList;
3535
foreach ($commandList->getCommands() as $command) {

composer.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"name": "magento/magento2-functional-testing-framework",
33
"description": "Magento2 Functional Testing Framework",
44
"type": "library",
5-
"version": "2.4.4",
5+
"version": "2.5.0",
66
"license": "AGPL-3.0",
77
"keywords": ["magento", "automation", "functional", "testing"],
88
"config": {

composer.lock

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/actiongroup-list.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# MFTF action group reference
2+
3+
Action groups are important building blocks for quickly creating tests for the Magento Functional Testing Framework.
4+
This page lists all current action groups so developers can see what is available to them.
5+
6+
{% include mftf/actiongroup_data.md %}

docs/configuration.md

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Configuration
22

33
The `*.env` file provides additional configuration for the Magento Functional Testing Framework (MFTF).
4-
To run the MFTF on your Magento testing instance, specify the basic configuration values.
4+
To run the MFTF on your Magento instance, specify the basic configuration values.
55
Advanced users can create custom configurations based on requirements and environment.
66

77
## Basic configuration
@@ -204,7 +204,7 @@ Example:
204204
FW_BP=~/magento/magento2-functional-testing-framework
205205
```
206206

207-
#### TESTS_MODULE_PATH
207+
### TESTS_MODULE_PATH
208208

209209
The path to where the MFTF modules mirror Magento modules.
210210

@@ -234,8 +234,8 @@ It points to `MAGENTO_BASE_URL` + `dev/tests/acceptance/utils/command.php`
234234

235235
Modify the default value:
236236

237-
- for non-default Magento installation
238-
- when use a subdirectory in the `MAGENTO_BASE_URL`
237+
- for non-default Magento installation
238+
- when use a subdirectory in the `MAGENTO_BASE_URL`
239239

240240
Example: `dev/tests/acceptance/utils/command.php`
241241

@@ -251,9 +251,35 @@ Example:
251251
BROWSER=firefox
252252
```
253253

254+
### CREDENTIAL_VAULT_ADDRESS
255+
256+
The Api address for a vault server.
257+
258+
Default: http://127.0.0.1:8200
259+
260+
Example:
261+
262+
```conf
263+
# Default api address for local vault dev server
264+
CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200
265+
```
266+
267+
### CREDENTIAL_VAULT_SECRET_BASE_PATH
268+
269+
Vault secret engine base path.
270+
271+
Default: secret
272+
273+
Example:
274+
275+
```conf
276+
# Default base path for kv secret engine in local vault dev server
277+
CREDENTIAL_VAULT_SECRET_BASE_PATH=secret
278+
```
279+
254280
<!-- Link definitions -->
255281

256282
[`MAGENTO_CLI_COMMAND_PATH`]: #magento_cli_command_path
257283
[generateDate]: test/actions.md#generatedate
258284
[mftf]: commands/mftf.md
259-
[timezones]: http://php.net/manual/en/timezones.php
285+
[timezones]: http://php.net/manual/en/timezones.php

docs/credentials.md

Lines changed: 108 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,21 @@
11
# Credentials
22

3-
When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys.
3+
When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD,
4+
use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys.
45

5-
## Define sensitive data in `.credentials`
6+
Currently the MFTF supports two types of credential storage:
7+
8+
- **.credentials file**
9+
- **HashiCorp vault**
10+
11+
## Configure File Storage
612

713
The MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`.
814
The file contains an example list of keys for fields that can require credentials.
915

1016
### Create `.credentials`
1117

12-
To make the MFTF process the file with credentials, change directories to `magento2/dev/tests/acceptance/` and copy `.credentials.example` to `.credentials`.
18+
To make the MFTF process the file with credentials, in the command line, navigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`.
1319

1420
```bash
1521
cd dev/tests/acceptance/
@@ -33,49 +39,119 @@ The command outputs the path if the file is excluded:
3339
.credentials
3440
```
3541

36-
### Define sensitive data
42+
### Define sensitive data in the `.credentials` file
3743

38-
Open the `.credentials` file, uncomment the fields you want to use, and add your values:
44+
Open the `.credentials` file and, for Magento core credentials, uncomment the fields you want to use and add your values:
3945

40-
```config
46+
```conf
4147
...
4248
# Credentials for the USPS service
43-
carriers_usps_userid=test_user
44-
carriers_usps_password=Lmgxvrq89uPwECeV
49+
magento/carriers_usps_userid=usps_test_user
50+
magento/carriers_usps_password=Lmgxvrq89uPwECeV
4551
4652
# Credentials for the DHL service
47-
#carriers/dhl/id_us=
48-
#carriers/dhl/password_us=
53+
#magento/carriers_dhl_id_us=dhl_test_user
54+
#magento/carriers_dhl_password_us=Mlgxv3dsagVeG
4955
....
56+
```
57+
58+
Or add new key & value pairs for your own credentials. The keys use the following format:
59+
60+
```conf
61+
<vendor>/<key_name>=<key_value>
5062
```
5163

5264
<div class="bs-callout bs-callout-info" markdown="1">
53-
The `/` symbol is not supported in a key name.
65+
The `/` symbol is not supported in a `key_name` other than the one after your vendor or extension name.
5466
</div>
55-
56-
You are free to use any other keys you like, as they are merely the keys to reference from your tests.
67+
68+
Otherwise you are free to use any other `key_name` you like, as they are merely the keys to reference from your tests.
5769

5870
```conf
5971
# Credentials for the MyAwesome service
60-
my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A
72+
vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A
73+
```
6174

62-
# Credentials for the USPS service
63-
carriers_usps_userid=test_user
64-
carriers_usps_password=Lmgxvrq89uPwECeV
65-
....
75+
## Configure Vault Storage
76+
77+
Hashicorp vault secures, stores, and tightly controls access to data in modern computing.
78+
It provides advanced data protection for your testing credentials.
79+
80+
The MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine.
81+
82+
### Install vault CLI
83+
84+
Download and install vault CLI tool if you want to run or develop MFTF tests locally. [Download Vault][Download Vault]
85+
86+
### Authenticate to vault via vault CLI
87+
88+
Authenticate to vault server via the vault CLI tool: [Login Vault][Login Vault].
89+
90+
```bash
91+
vault login -method -path
92+
```
93+
94+
**Do not** use `-no-store` command option, as the MFTF will rely on the persisted token in the token helper (usually the local filesystem) for future API requests.
95+
96+
### Store secrets in vault
97+
98+
The MFTF uses the `KV Version 2` secret engine for secret storage.
99+
More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2].
100+
101+
#### Secrets path and key convention
102+
103+
The path and key for secret data must follow the format:
104+
105+
```conf
106+
<SECRETS_BASE_PATH>/mftf/<VENDOR>/<SECRET_KEY>
107+
```
108+
109+
```conf
110+
# Secret path and key for carriers_usps_userid
111+
secret/mftf/magento/carriers_usps_userid
112+
113+
# Secret path and key for carriers_usps_password
114+
secret/mftf/magento/carriers_usps_password
115+
```
116+
117+
#### Write secrets to vault
118+
119+
You can use vault CLI or API to write secret data (credentials, etc) to vault. Here is a CLI example:
120+
121+
```bash
122+
vault kv put secret/mftf/magento/carriers_usps_userid carriers_usps_userid=usps_test_user
123+
vault kv put secret/mftf/magento/carriers_usps_password carriers_usps_password=Lmgxvrq89uPwECeV
66124
```
67125

126+
### Setup MFTF to use vault
127+
128+
Add vault configuration environment variables [`CREDENTIAL_VAULT_ADDRESS`][] and [`CREDENTIAL_VAULT_SECRET_BASE_PATH`][]
129+
from `etc/config/.env.example` in `.env`.
130+
Set values according to your vault server configuration.
131+
132+
```conf
133+
# Default vault dev server
134+
CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200
135+
CREDENTIAL_VAULT_SECRET_BASE_PATH=secret
136+
```
137+
138+
## Configure both File Storage and Vault Storage
139+
140+
It is possible and sometimes useful to setup and use both `.credentials` file and vault for secret storage at the same time.
141+
In this case, the MFTF tests are able to read secret data at runtime from both storage options, but the local `.credentials` file will take precedence.
142+
68143
<!-- {% raw %} -->
69144

70145
## Use credentials in a test
71146

72-
Access the data defined in the `.credentials` file using the [`fillField`][] action with the `userInput` attribute.
73-
Define the value as a reference to the corresponding key in the credentials file such as `{{_CREDS.my_data_key}}`:
147+
Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`createData`][].
148+
149+
Define the value as a reference to the corresponding key in the credentials file or vault such as `{{_CREDS.my_data_key}}`:
74150

75151
- `_CREDS` is an environment constant pointing to the `.credentials` file
76-
- `my_data_key` is a key in the the `.credentials` file that contains the value to be used in a test step
152+
- `my_data_key` is a key in the the `.credentials` file or vault that contains the value to be used in a test step
77153

78-
For example:
154+
For example, reference secret data in the [`fillField`][] action with the `userInput` attribute.
79155

80156
```xml
81157
<fillField stepKey="FillApiToken" selector=".api-token" userInput="{{_CREDS.my_data_key}}" />
@@ -88,13 +164,21 @@ For example:
88164
The generated tests do not contain credentials values.
89165
The MFTF dynamically retrieves, encrypts, and decrypts the sensitive data during test execution.
90166
Decrypted credentials do not appear in the console, error logs, or [test reports][].
91-
The decrypted values are only available in the `.credentials` file.
167+
The decrypted values are only available in the `.credentials` file or within vault.
92168

93169
<div class="bs-callout bs-callout-info">
94-
The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data.</div>
170+
The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data.
171+
</div>
95172

96173
<!-- Link definitions -->
97174
[`fillField`]: test/actions.md#fillfield
175+
[`magentoCLI`]: test/actions.md#magentocli
176+
[`createData`]: test/actions.md#createdata
98177
[data]: data.md
99178
[initial setup]: getting-started.md
100179
[test reports]: reporting.md
180+
[Download Vault]: https://www.hashicorp.com/products/vault/
181+
[Login Vault]: https://www.vaultproject.io/docs/commands/login.html
182+
[Vault KV2]: https://www.vaultproject.io/docs/secrets/kv/kv-v2.html
183+
[`CREDENTIAL_VAULT_ADDRESS`]: configuration.md#CREDENTIAL_VAULT_ADDRESS
184+
[`CREDENTIAL_VAULT_SECRET_BASE_PATH`]: configuration.md#CREDENTIAL_VAULT_SECRET_BASE_PATH

docs/data.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,16 @@ The following is an example of a call in test:
173173

174174
This action inputs data from the `name` of the `_defaultCategory` entity (for example, `simpleCategory598742365`) into the field with the locator defined in the selector of the `categoryNameInput` element of the `AdminCategoryBasicFieldSection`.
175175

176+
You can also call data from the xml definition of a `data` tag directly:
177+
178+
```xml
179+
<entity name="NewAdminUser" type="user">
180+
<data key="username" unique="suffix">admin</data>
181+
<data key="current_password">{{AnotherUser.current_password}}</data> <!-- Data from another entity -->
182+
<data key="current_password">{{_ENV.MAGENTO_ADMIN_PASSWORD}}</data> <!-- ENV file reference -->
183+
</entity>
184+
```
185+
176186
## Reference
177187

178188
### entities {#entities-tag}

docs/selectors.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
## Selectors
2+
3+
These guidelines should help you to write high quality selectors.
4+
5+
### Selectors SHOULD be written in CSS instead of XPath whenever possible
6+
7+
CSS is generally easier to read than XPath. For example, `//*[@id="foo"]` in XPath can be expressed as simply as `#foo` in CSS.
8+
See this [XPath Cheatsheet](https://devhints.io/xpath) for more examples.
9+
10+
### XPath selectors SHOULD NOT use `@attribute="foo"`.
11+
12+
This would fail if the attribute was `attribute="foo bar"`.
13+
Instead you SHOULD use `contains(@attribute, "foo")` where `@attribute` is any valid attribute such as `@text` or `@class`.
14+
15+
### CSS and XPath selectors SHOULD be implemented in their most simple form
16+
17+
* <span class="color:green">GOOD:</span> `#foo`
18+
* <span class="color:red">BAD:</span> `button[contains(@id, "foo")]`
19+
20+
### CSS and XPath selectors SHOULD avoid making use of hardcoded indices
21+
22+
Instead you SHOULD parameterize the selector.
23+
24+
* <span class="color:green">GOOD:</span> `.foo:nth-of-type({{index}})`
25+
* <span class="color:red">BAD:</span> `.foo:nth-of-type(1)`
26+
27+
* <span class="color:green">GOOD:</span> `button[contains(@id, "foo")][{{index}}]`
28+
* <span class="color:red">BAD:</span> `button[contains(@id, "foo")][1]`
29+
30+
* <span class="color:green">GOOD:</span> `#actions__{{index}}__aggregator`
31+
* <span class="color:red">BAD:</span> `#actions__1__aggregator`
32+
33+
### CSS and XPath selectors MUST NOT reference the `@data-bind` attribute
34+
35+
The `@data-bind` attribute is used by KnockoutJS, a framework Magento uses to create dynamic Javascript pages. Since this `@data-bind` attribute is tied to a specific framework, it should not be used for selectors. If Magento decides to use a different framework then these `@data-bind` selectors would break.

0 commit comments

Comments
 (0)