PHPC-2261: Revamp Evergreen config #1473

alcaeus · 2023-09-22T11:41:08Z

This PR takes care of PHPC-2260 and PHPC-2261. The following changes have been made, while others are still pending:

Extract build tasks from testing to cache build results
Make test tasks depend on build tasks
Remove matrix build variants in favour of separate ones
Migrate existing special tests (OCSP, etc.) to new format
Automate generation of build tasks and potentially variants
Document structure of evergreen files and how to add/remove tasks/variants in future

The last task is still left to do. However, since generate.tasks does not show generated tasks when configuring a patch build, it is not suitable for us. However, repetitive files (e.g. the version-dependent tasks) will be generated automatically via a script to avoid repetition and errors.

In addition to the changes above, the following semi-related changes have been made:

After compilation, a smoketest is run to ensure the library can be loaded. This prevents running a large number of tasks that won't complete successfully
Add a make target that runs tests without compiling everything. This was done to avoid having to recursively replace directory names in all build output files. We may opt to invoke run-tests.php directly.
Extracted configuration to multiple files to make files easier to understand and use
Removed unused functions/tasks (there may be more left to remove)

For now, tests are only run on Debian systems:

With PHP 8.2, we test all topologies on all MongoDB versions. This uses Debian 11 and Debian 9.2 to cover MongoDB 3.6 up to the latest build
Older PHP versions only test MongoDB 7.0 in a replica set topology with authentication enabled.
Other environments are not tested.

We'll have to discuss what tests we want present, and which ones we want to run for pull requests and commits. We may want to include more tasks but not run them by default to allow for testing special scenarios, although those can be added when necessary.

alcaeus · 2023-09-22T11:41:38Z

.evergreen/compile-unix.sh

 set -o errexit  # Exit the script with error if any of the commands fail

+# Find PHP binary path for the requested version


This was moved over so we can run compile for multiple PHP versions on the same box if we wanted to.

alcaeus · 2023-09-22T11:43:17Z

.evergreen/config/test-variants.yml

+      - name: "test-ocsp-latest"
+        execution_tasks:
+          - ".ocsp .latest"


Display tasks group a number of tasks in the "configure patch" view. While this limits us to have to select/unselect all OCSP tests at once without having a choice of enabling only some of them, I believe this is fine. Each OCSP test is an individual task due to the way these are structured.

SGTM. I think there's also room for improvement here. If we look at libmongoc, I believe they extracted most of their OCSP logic into run-ocsp-test.sh, which makes the Evergreen configuration more readable. That might allow us to do reduce the complexity of _template-ocsp.yml and bring it more in line with other templates where we'd only selecting server versions.

alcaeus · 2023-09-22T11:45:10Z

.evergreen/config/test/ocsp-7.0.yml

@@ -0,0 +1,389 @@
+tasks:
+  - name: "ocsp-test_1-rsa-delegate-7.0"


The OCSP tasks may be candidates for using a task group. This would allow us to only fetch the build once, install the correct MongoDB version, then run all tasks sequentially on the same build host. This would further cut down resource usage. However, we'll want to make some improvements to mongodb-orchestration to make sure we can set up and tear down OCSP responders properly. We might also have to split up the mongo-orchestration files to allow for separate functions for downloading MongoDB, starting/stopping orchestration, and starting/stopping a deployment.

alcaeus · 2023-09-22T11:46:47Z

.evergreen/config.yml

+include:
+  - filename: .evergreen/config/functions.yml


Unfortunately, include is only allowed in top-level files, so we can't nest our includes (and include the various test files in test-tasks.yml).

.evergreen/config.yml

alcaeus · 2023-09-27T07:45:49Z

.evergreen/config.yml

+# Define aliases for patch builds and PR builds. These only apply if no aliases are defined in project settings
+github_pr_aliases:
+  # Run all build tasks, except on Power8 and zSeries due to low number of build hardware
+  - variant: "^build-[^-]+($|-arm64)"
+    task: ".*"
+  # Run tests on debian, only testing local replicasets with authentication enabled
+  - variant: "test-.*"
+    task: "test-mongodb-.*-replicaset-auth"


This configuration only takes effect once the configuration in the project settings has been removed, which I'll do after this pull request is merged.

In addition to this, using variant and task tags for aliases does not work as expected (see EVG-20579), so I've had to come up with regular expressions until that issue is resolved.

.evergreen/config/build-task-groups.yml

.gitattributes

alcaeus · 2023-09-27T07:52:22Z

Makefile.frag

@@ -88,3 +88,27 @@ generate-function-map: all
 	else \
 		echo "ERROR: Cannot generate function maps without CLI sapi."; \
 	fi
+
+test-no-build:


This target was necessary to avoid make trying to recompile the extension due to the different paths in the various .dep files. Instead of replacing paths in more files than just Makefile, I decided to copy the original test target and remove the dependency to all.

Alternatively, we can also extract the call to run-tests.php, but at that point we may as well consider using a different test runner entirely, such as PHPUnit (which can run phpt files as well).

Alternatively, we can also extract the call to run-tests.php

I was originally going to suggest replacing this Makefile target with additional logic in .evergreen/run-tests.sh, but seeing how messy this is (and integrated with the rest of the Makefile) I think this is the right call.

at that point we may as well consider using a different test runner entirely, such as PHPUnit (which can run phpt files as well).

I think this is totally worth exploring. Would you mind creating a ticket so we can revisit the idea?

I would never consider using PHPUnit with PHPC for local development (since it'd likely complicate things like backtraces and leak detection), but it may actually make sense for CI assuming it handles all of the same syntax of run-tests.php and generates comparable test suite output. And we'd also be able to test it alongside run-tests.php (e.g. introduce PHPUnit in just an extra task to start). I assume it will also enable better integration with Evergreen, such that we could start populating the "Tests" tab like we do for PHPLIB.

@alcaeus: ☝️

Created PHPC-2306 to evaluate PHPUnit as a test runner for PHPC. I also created PHPC-2307 to track attaching test results separately.

.evergreen/architecture.md

.evergreen/compile-unix.sh

jmikola · 2023-09-29T17:03:51Z

.evergreen/config.yml

+# Protect ourselves against rogue test case that runs forever
+# Good rule of thumb: the average length a task takes, times 5, which roughly accounts for variable system performance
+# for various build variants
+exec_timeout_secs: 1800


If this is still 6 minutes, does that line up with "the average length a task takes, times 5"? Do our tasks roughly take a bit over a minute to execute?

The 1800 seconds are 30 minutes, meaning that we'll assume a roughly 6-minute task runtime. I'll double-check to make sure the value is still accurate.

@alcaeus: ☝️

Runtime for build tasks is ~2 minutes, while for test tasks it varies between 2 and 5 minutes. Given that this config setting kills tasks that are running too long I don't see an issue with using 30 minutes, but I'm open to reducing it to something more realistic, e.g. 10 minutes.

.evergreen/config.yml

.evergreen/config/test/_template-local.yml

.evergreen/config/test/_template-ocsp.yml

.evergreen/config/test/_template-require-api-version.yml

.evergreen/config/test/_template-skip-crypt-shared.yml

.gitattributes

jmikola

LGTM but there are one or two outstanding questions to follow up on.

.evergreen/architecture.md

jmikola · 2023-10-10T18:24:47Z

.evergreen/architecture.md

+- `local`: All tasks that run a local MongoDB cluster for testing.
+- `<version>`: These tags allow selection based on MongoDB version, e.g. `6.4`.
+- `standalone`, `replicaset`, `sharded`: These tags allow selection based on the MongoDB topology.
+- `loadbalanced`: Allows for selecting tests using a load balancer


I noticed that templates/test/load-balanced.yml mixes this tag with "sharded". Is that necessary since LB should already imply a sharded topology? Would it make sense to consider this a topology and fold it into the previous line?

Removed the sharded tag from load balanced tests, effectively treating loadbalanced like its own topology.

.evergreen/architecture.md

jmikola · 2023-10-10T18:29:08Z

.evergreen/config.yml

+# Protect ourselves against rogue test case that runs forever
+# Good rule of thumb: the average length a task takes, times 5, which roughly accounts for variable system performance
+# for various build variants
+exec_timeout_secs: 1800


@alcaeus: ☝️

This is necessary if we don't want to change all `.dep` files to the new path.

- Artifactory uploads are not used, so we can save on disk space - Evergreen isn't used to test Windows builds, so we don't need to make Windows fixes - Files should already be executable

alcaeus commented Sep 22, 2023

View reviewed changes

alcaeus force-pushed the phpc-2201-2260-revamp-evergreen-config branch 10 times, most recently from c859cd6 to d9bb19b Compare September 27, 2023 07:31

alcaeus commented Sep 27, 2023

View reviewed changes

alcaeus force-pushed the phpc-2201-2260-revamp-evergreen-config branch from b5032b4 to 7da992f Compare September 27, 2023 07:52

alcaeus requested a review from jmikola September 27, 2023 07:52

alcaeus marked this pull request as ready for review September 27, 2023 07:52

alcaeus mentioned this pull request Sep 28, 2023

PHPLIB-1260: Revamp evergreen config mongodb/mongo-php-library#1179

Merged

jmikola reviewed Sep 29, 2023

View reviewed changes

jmikola mentioned this pull request Oct 4, 2023

Hide generated JSON spec test files from displaying in diffs by default mongodb/specifications#1465

Merged

alcaeus force-pushed the phpc-2201-2260-revamp-evergreen-config branch from 7da992f to 3b9f6b0 Compare October 9, 2023 08:33

alcaeus requested a review from jmikola October 9, 2023 08:33

jmikola reviewed Oct 10, 2023

View reviewed changes

alcaeus added 9 commits October 11, 2023 09:24

Remove unused functions and tasks

a67441c

Fix indentation

dc6df70

POC: Use task group to test local deployments

780a9e4

Add make target to test without building

8f99925

This is necessary if we don't want to change all `.dep` files to the new path.

Cache compiled build for later use

16b8e44

Rename tasks for better filtering

7a4b177

Remove working dir upload as it's already uploaded along with the build

038ec06

Update tags in test tasks

d8dbc3d

Create build variants for building driver

a131122

alcaeus added 24 commits October 11, 2023 09:24

Remove obsolete variant

bebb155

Move requireApiVersion tests to new format

6fa2324

Move loadbalanced tests to new format

81614c0

Move OCSP tests to new format

214b8f5

Generate test and build tasks from templates

7443ace

Remove obsolete files

eaee3d8

Refactor generation script

6607c0f

Remove version identifier from libmongoc task names

37c944c

Add variant tags to facilitate patch filters

5b8d711

Define pull request aliases in evergreen config

1a92176

Add architecture documentation

d52d13d

Use regular expression for PR aliases

1033d86

Mark generated files as such

0edd44d

Remove unnecessary functions

7a87c85

- Artifactory uploads are not used, so we can save on disk space - Evergreen isn't used to test Windows builds, so we don't need to make Windows fixes - Files should already be executable

Update CI documentation

d19f8ae

Remove unused YAML tags

305e50d

Document references of libmongoc build tasks

86d8369

Document tags used in tasks

802b0be

Make versions in config generator more readable

10632e3

Split templates and generated configs

2d349d3

Add blank lines to task files for readability

acf1eb7

Add note about libmongoc build tasks in contributing guide

d529aca

Use real server version in architecture docs

308bfc5

Treat loadbalanced as its own topology in tests

a5efc27

alcaeus force-pushed the phpc-2201-2260-revamp-evergreen-config branch from 3b9f6b0 to a5efc27 Compare October 11, 2023 07:42

jmikola approved these changes Oct 11, 2023

View reviewed changes

Kill tasks after 10 minutes

8c0fff7

alcaeus merged commit a204130 into mongodb:master Oct 16, 2023

alcaeus deleted the phpc-2201-2260-revamp-evergreen-config branch October 16, 2023 07:30

alcaeus mentioned this pull request Oct 16, 2023

PHPC-2261: Make OCSP responder file executable #1481

Merged

		set -o errexit # Exit the script with error if any of the commands fail

		# Find PHP binary path for the requested version

		@@ -0,0 +1,389 @@
		tasks:
		- name: "ocsp-test_1-rsa-delegate-7.0"

PHPC-2261: Revamp Evergreen config #1473

PHPC-2261: Revamp Evergreen config #1473

Uh oh!

Conversation

alcaeus commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmikola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alcaeus commented Sep 22, 2023 •

edited

Loading