Add documentation about updating .ci.yaml and flutter-gold. (flutter#154184)

matanlurey · pull[bot] · commit 3033393311ea · 2024-10-23T22:00:57.000Z
Closes flutter#143863. /cc @christopherfujino
diff --git a/docs/contributing/testing/Writing-a-golden-file-test-for-package-flutter.md b/docs/contributing/testing/Writing-a-golden-file-test-for-package-flutter.md
@@ -7,17 +7,20 @@ _(This page is referenced by comments in the Flutter codebase.)_
 Golden file tests for `package:flutter` use [Flutter Gold](https://flutter-gold.skia.org/?query=source_type%3Dflutter) for baseline and version management of golden files. This allows for golden file testing on Linux, Windows, MacOS and Web, which accounts for the occasional subtle rendering differences between these platforms.
 
 ## Index
+
 - [Known Issues](#known-issues)
 - [Build Breakage](#build-breakage)
 - [Creating a New Golden File Test](#creating-a-new-golden-file-test)
-- [Adding a new key in the Skia Client](#Adding-a-new-key-in-the-Skia-Client)
+  - [Add or configure a `.ci.yaml` task](#add-or-configure-a-ciyaml-task)
+  - [Adding a known task name to the `flutter-gold` check](#adding-a-known-task-name-to-the-flutter-gold-check)
+  - [Writing and submitting the test](#writing-and-submitting-the-test)
+- [Adding a new key in the Skia Client](#adding-a-new-key-in-the-skia-client)
 - [Updating a Golden File Test](#updating-a-golden-file-test)
 - [Flutter Gold Login](#flutter-gold-login)
 - [`flutter-gold` Check](#flutter-gold-check)
 - [`reduced-test-set` tag](#reduced-test-set-tag)
 - [Troubleshooting](#troubleshooting)
 
-
 ## Known Issues
 
 ### Negative Images
@@ -31,9 +34,7 @@ If you would like to instantly invalidate all prior renderings, changing the nam
 
 If the Flutter build is broken due to a golden file test failure, this typically means an image change has landed without being triaged. Golden file images should be triaged in pre-submit before a change lands (as described in the steps below). If this process is not followed, a test with an unapproved golden file image will fail in post-submit testing. This will present in the following error message:
 
-<!-- TODO(Piinks): Update this error message in the framework. -->
-
-```
+```txt
   Skia Gold received an unapproved image in post-submit
   testing. Golden file images in flutter/flutter are triaged
   in pre-submit during code review for the given PR.
@@ -50,6 +51,92 @@ Notice, Gold may wrongly attribute blame for image changes on the dashboard. Pos
 
 ## Creating a New Golden File Test
 
+<!--
+  Note: These instructions are also referred to in the flutter/engine repository
+  ensure that when updating them, the workflow is either consistent or the
+  differences are clearly called out.
+-->
+
+Writing a _new_ golden file test if the _test suite_ (sometimes called "task"
+or "shard") is not already configured for golden file testing is an involved
+process, and missing a step can result in a broken build or a failing test.
+
+The process involves:
+
+1. [Adding or configuring a `.ci.yaml` task](#add-or-configure-a-ciyaml-task).
+1. [Writing and submitting the test](#writing-and-submitting-the-test).
+1. [Adding a known task name to the `flutter-gold` check](#adding-a-known-task-name-to-the-flutter-gold-check).
+
+### Add or configure a `.ci.yaml` task
+
+_If you are editing an existing test or test suite that already includes the
+`goldctl` dependency (as seen below), you can skip this step._
+
+Every test suite that runs tests (golden file tests included) needs to be
+configured in the top-level [`.ci.yaml` file](../../../.ci.yaml), which contains
+the configuration for all the CI tasks that run on the Flutter repository. For
+example, the task `Linux framework_tests_widgets` (as of 2024-08-27) looks like
+this:
+
+```yaml
+# Some parts omitted for brevity
+
+- name: Linux framework_tests_widgets
+  properties:
+    dependencies: >-
+      [
+        {"dependency": "goldctl", "version": "git_revision:2387d6fff449587eecbb7e45b2692ca0710b63b9"}
+      ]
+```
+
+What this means is that the task, `Linux framework_tests_widgets`, will install
+the `goldctl` tool (which is used to interface with Skia Gold) at the specified
+revision. If you are adding a new test suite, or adding golden testing to a
+test suite that didn't have it before, you will need to add or modify a task in
+the `.ci.yaml` file:
+
+```diff
++  properties:
++    dependencies: >-
++      [
++        {"dependency": "goldctl", "version": "git_revision:2387d6fff449587eecbb7e45b2692ca0710b63b9"}
++      ]
+```
+
+> [!TIP]
+> The exact git revision of `goldctl` should be copied from an existing task.
+
+If a new dependency was added, see [adding a known task name to the `flutter-gold` check](#adding-a-known-task-name-to-the-flutter-gold-check).
+
+### Adding a known task name to the `flutter-gold` check
+
+The `flutter-gold` check is a cron job that is scheduled to run (every 5
+minutes) through every open (non-draft) PR across `flutter/flutter` and
+`flutter/engine` is checked for tasks that are _known_ to run golden file tests,
+and if they are present, the check asks (via a Skia Gold API) if there were any
+untriaged images generated, showing a :question: if so, or a :white_check_mark:
+if not.
+
+This list is manually curated in the [`flutter/cocoon` codebase](https://github.com/flutter/cocoon/blob/65e89508cad5b6f9622ee072a1178913fed5001e/app_dart/lib/src/request_handlers/push_gold_status_to_github.dart#L123-L144)
+and needs to be updated when new tasks are added or removed.
+
+See [#149544](https://github.com/flutter/flutter/pull/149544) for an example of adding a new task to the list
+and [#143863](https://github.com/flutter/flutter/issues/143863#issuecomment-2310898513) for a discussion on the topic.
+
+Failure to add a new task to this list will result in the `flutter-gold` check
+not running on some PRs (and build breakages as a result).
+
+### Writing and submitting the test
+
+> [!NOTE]
+> These instructions are specific to writing golden file tests for
+> `package:flutter`. If you are writing integration tests, or tests in another
+> repository (such as `flutter/engine`) the specifics will be different; consult
+> relevant documentation and/or team members about the process. Examples:
+>
+> - [`engine > testing > skia_gold_client > README.md`](https://github.com/flutter/engine/blob/main/testing/skia_gold_client/README.md)
+> - [`engine > impeller > golden_tests > README.md`](https://github.com/flutter/engine/blob/main/impeller/golden_tests/README.md)
+
 Write your test as a normal test, using `testWidgets` and `await tester.pumpWidget` and so on.
 
 Put a `RepaintBoundary` widget around the part of the subtree that you want to verify. If you don't, the output will be a 2400x1800 image, since the tests by default use an 800x600 viewport with a device pixel ratio of 3.0. If you would like to further control the image size, put a `SizedBox` around your `RepaintBoundary` to set constraints.
@@ -108,10 +195,9 @@ The updated tests can be triaged from these tryjobs, which will cause the pendin
 
 And that’s it! Your new golden file(s) will be checked in as the baseline(s) for your new test(s), and your PR will be ready to merge. :tada:
 
-
 ## Flutter Gold Login
 
-Triage permission is currently restricted to members of *flutter-hackers*. For more information, see [Contributor Access](../Contributor-access.md).
+Triage permission is currently restricted to members of _flutter-hackers_. For more information, see [Contributor Access](../Contributor-access.md).
 Once you have been added as an authorized user for Flutter Gold, you can log in through the [homepage of the Flutter Gold dashboard](https://flutter-gold.skia.org/) and proceed to triage your image results under [Changelists](https://flutter-gold.skia.org/changelists).
 
 ## `flutter gold` Check
@@ -134,10 +220,11 @@ For more context, see [flutter.dev/go/reduce-ci-tests](https://flutter.dev/go/re
 
 ## Troubleshooting
 
-* Trouble: the `flutter-gold` is stuck at pending status while all other checks are successful (if they are not, see [flutter-gold check](#flutter-gold-check)).
-  * Solution: this may be a side-effect of force-pushed commits (`git push -f`) and a known Skia Gold issue (https://issues.skia.org/issues/40044676), typically after a rebase. If this happens, try rebasing or pushing an empty commit without force pushing. This side-effect is flaky.
-* Trouble: the `flutter-gold` check posted a message saying "Golden file changes have been found..." but the triage page is empty.
-  * Solution: this may be another side-effect of force-pushed commits (see above), but may also be a side-effect of untriaged goldens from already submitted PRs. Try rebasing again, or reach out in #hackers-infra on the Discord.
+- Trouble: the `flutter-gold` is stuck at pending status while all other checks are successful (if they are not, see [flutter-gold check](#flutter-gold-check)).
+  - Solution: this may be a side-effect of force-pushed commits (`git push -f`) and a known Skia Gold issue (https://issues.skia.org/issues/40044676), typically after a rebase. If this happens, try rebasing or pushing an empty commit without force pushing. This side-effect is flaky.
+- Trouble: the `flutter-gold` check posted a message saying "Golden file changes have been found..." but the triage page is empty.
+  - Solution: this may be another side-effect of force-pushed commits (see above), but may also be a side-effect of untriaged goldens from already submitted PRs. Try rebasing again, or reach out in #hackers-infra on the Discord.
 
 ## Additional Resources
-- [Gold APIs used by the Flutter Framework](https://docs.google.com/document/d/1H3CDqT7zBUt4Je2HPQpleYA-drwj2oy0mdPlSdf2d4A/edit?usp=sharing)
+
+- [Gold APIs used by the Flutter Framework](https://docs.google.com/document/d/1H3CDqT7zBUt4Je2HPQpleYA-drwj2oy0mdPlSdf2d4A/edit?usp=sharing)
