@@ -7,29 +7,36 @@ the tests are referred to as "tasks" in the API, but since we primarily use it
7
7
for testing, this document refers to them as "tests".
8
8
9
9
Current statuses for the devicelab are available at
10
- https://flutter-dashboard.appspot.com/#/build . See [ dashboard user guide] ( https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md )
10
+ < https://flutter-dashboard.appspot.com/#/build > . See [ dashboard user
11
+ guide] ( https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md )
11
12
for information on using the dashboards.
12
13
13
14
## Table of Contents
15
+
14
16
* [ How the DeviceLab runs tests] ( #how-the-devicelab-runs-tests )
15
17
* [ Running tests locally] ( #running-tests-locally )
16
18
* [ Writing tests] ( #writing-tests )
17
- * [ Adding tests to continuous integration] ( #adding-tests-to-continuous-integration )
19
+ * [ Adding tests to continuous
20
+ integration] ( #adding-tests-to-continuous-integration )
18
21
* [ Adding tests to presubmit] ( #adding-tests-to-presubmit )
19
22
20
-
21
23
## How the DeviceLab runs tests
22
24
23
- DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
25
+ DeviceLab tests are run against physical devices in Flutter's lab (the
26
+ "DeviceLab").
24
27
25
- Tasks specify the type of device they are to run on (` linux_android ` , ` mac_ios ` , ` mac_android ` , ` windows_android ` , etc).
26
- When a device in the lab is free, it will pickup tasks that need to be completed.
28
+ Tasks specify the type of device they are to run on (` linux_android ` , ` mac_ios ` ,
29
+ ` mac_android ` , ` windows_android ` , etc). When a device in the lab is free, it
30
+ will pickup tasks that need to be completed.
27
31
28
- 1 . If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
29
- all tasks record performance metrics.
30
- 2 . If task fails, an auto rerun happens. Whenever the last run succeeds, the task will be reported as a success. For this case,
31
- a flake will be flagged and populated to the test result.
32
- 3 . If the task fails in all reruns, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
32
+ 1 . If the task succeeds, the test runner reports the success and uploads its
33
+ performance metrics to Flutter's infrastructure. Not all tasks record
34
+ performance metrics.
35
+ 2 . If task fails, an auto rerun happens. Whenever the last run succeeds, the
36
+ task will be reported as a success. For this case, a flake will be flagged and
37
+ populated to the test result.
38
+ 3 . If the task fails in all reruns, the test runner reports the failure to
39
+ Flutter's infrastructure and no performance metrics are collected
33
40
34
41
## Running tests locally
35
42
@@ -63,10 +70,11 @@ To run a test, use option `-t` (`--task`):
63
70
64
71
Where ` NAME_OR_PATH_OF_TEST ` can be either of:
65
72
66
- - the _ name_ of a task, which is a file's basename in ` bin/tasks ` . Example: ` complex_layout__start_up ` .
67
- - the path to a Dart _ file_ corresponding to a task, which resides in ` bin/tasks ` .
68
- Tip: most shells support path auto-completion using the Tab key. Example:
69
- ` bin/tasks/complex_layout__start_up.dart ` .
73
+ * the _ name_ of a task, which is a file's basename in ` bin/tasks ` . Example:
74
+ ` complex_layout__start_up ` .
75
+ * the path to a Dart _ file_ corresponding to a task, which resides in
76
+ ` bin/tasks ` . Tip: most shells support path auto-completion using the Tab key.
77
+ Example: ` bin/tasks/complex_layout__start_up.dart ` .
70
78
71
79
To run multiple tests, repeat option ` -t ` (` --task ` ) multiple times:
72
80
@@ -107,19 +115,19 @@ Example:
107
115
108
116
The ` --ab=10 ` tells the runner to run an A/B test 10 times.
109
117
110
- ` --local-engine=host_debug_unopt ` tells the A/B test to use the ` host_debug_unopt `
111
- engine build. ` --local-engine ` is required for A/B test.
118
+ ` --local-engine=host_debug_unopt ` tells the A/B test to use the
119
+ ` host_debug_unopt ` engine build. ` --local-engine ` is required for A/B test.
112
120
113
- ` --ab-result-file=filename ` can be used to provide an alternate location to output
114
- the JSON results file (defaults to ` ABresults#.json ` ). A single ` # ` character can be
115
- used to indicate where to insert a serial number if a file with that name already
116
- exists, otherwise, the file will be overwritten.
121
+ ` --ab-result-file=filename ` can be used to provide an alternate location to
122
+ output the JSON results file (defaults to ` ABresults#.json ` ). A single ` # `
123
+ character can be used to indicate where to insert a serial number if a file with
124
+ that name already exists, otherwise, the file will be overwritten.
117
125
118
126
A/B can run exactly one task. Multiple tasks are not supported.
119
127
120
128
Example output:
121
129
122
- ```
130
+ ``` text
123
131
Score Average A (noise) Average B (noise) Speed-up
124
132
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
125
133
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
@@ -142,13 +150,14 @@ Summarize tool example:
142
150
ABresults.json ABresults1.json ABresults2.json ...
143
151
```
144
152
145
- ` --[no-]tsv-table ` tells the tool to print the summary in a table with tabs for easy spreadsheet
146
- entry. (defaults to on)
153
+ ` --[no-]tsv-table ` tells the tool to print the summary in a table with tabs for
154
+ easy spreadsheet entry. (defaults to on)
147
155
148
- ` --[no-]raw-summary ` tells the tool to print all per-run data collected by the A/B test formatted
149
- with tabs for easy spreadsheet entry. (defaults to on)
156
+ ` --[no-]raw-summary ` tells the tool to print all per-run data collected by the
157
+ A/B test formatted with tabs for easy spreadsheet entry. (defaults to on)
150
158
151
- Multiple trailing filenames can be specified and each such results file will be processed in turn.
159
+ Multiple trailing filenames can be specified and each such results file will be
160
+ processed in turn.
152
161
153
162
## Reproducing broken builds locally
154
163
@@ -208,7 +217,7 @@ _TASK_- the name of your test that also matches the name of the
208
217
209
218
1 . Add target to
210
219
[ .ci.yaml] ( https://github.com/flutter/flutter/blob/master/.ci.yaml )
211
- - Mirror an existing one that has the recipe ` devicelab_drone `
220
+ * Mirror an existing one that has the recipe ` devicelab_drone `
212
221
213
222
If your test needs to run on multiple operating systems, create a separate
214
223
target for each operating system.
0 commit comments