RUBY-3275 backport CI changes to 2.18 (#2727)

* RUBY-3232 Switch to supported build hosts (#2712)

* Bump to latest toolchain and clean up specs to use consistent (and modern) build hosts

* fix config validation errors

* don't try to run ruby 2.7 on ubuntu2204

* bump submodule to get updated toolchain version

* rhel84 -> rhel80

* keep tweaking

* add "require_solo" helper

* for testing: pointing spec/shared at my own branch

* update spec/shared again to fix typo

* bump spec/shared version again to prevent apt-get install from prompting

* make sure cmake is installed for FLE runs

* don't test on 3.2 yet (broken tests)

* make sure we check for cmake BEFORE the bundle is installed

* use MDB 5.x for testing certain auth mechanisms

this is because run-tests.sh uses `mongo` to configure the MDB instance for
certain auth mechanisms, which isn't shipped in MDB 6+.

* try building the native extension for mongo_kerberos

* remove jruby-9.4 for now to see if things pass any better with 9.3

* bump spec/shared to get updated server registry (for aarch64 versions)

* update serverless to use rhel80

* move the install_cmake function to spec/shared, and make sure it gets called in run-tests-gcp

* don't run zstd tests on jruby (no support)

* bump spec/shared to get updated target arch selection

* replace jruby 9.4 with 9.3 in sample_rubies list

* bump spec/shared again

* make sure kerberos stuff doesn't get confused by the BUNDLE_GEMFILE variable

* don't test jruby on arm for now

* don't assume x86_64 architecture

* fix invalid config

* jruby-9.3 isn't being built for ubuntu/debian :/

* need to build the bundle for the kerberos gem before compiling it

* bump spec/shared to latest from my branch

* jruby-9.3 is being built as a recent ruby now

* don't include ruby-3.2 in supported rubies (yet)

* rebuild config.yml

* bump spec/shared

* leaving out debian11 for now; we have ubuntu2204, which is closely related

* the file is /etc/os-release, not /etc/os_release

* remove jruby from FLE tests

* comment out the failing tests (and reference the corresponding Jira tickets)

this will at least let us get this "build hosts" effort sorted

* forgot to comment out 'latest'

* cmake is only needed if the helper is being installed

Co-authored-by: Dmitry Rybakov <dmitry.rybakov@mongodb.com>

* bump spec/shared

---------

Co-authored-by: Dmitry Rybakov <dmitry.rybakov@mongodb.com>

* more cleanup from the merge

* more merge confusion

* update all of the .evergreen folder

* bump ruby versions in README

* RUBY-3216 Skip failing view spec (#2701)

* remove azure/gcp tests taken from master

* no sdam_unified_spec in 2.18-stable

* RUBY-3275 Update Atlas certificate (#2735)

* RUBY-3275 Update Atlas certificate

* Add a custom error message

* bump drivers-evergreen-tools to try and make GH actions pass

* need to update the GH actions definition, too

* nix the rspec-mocks-diag gem so we can run more recent rspec

all this to fix a ruby 3.2 incompatibility

* disambiguate options hash vs keyword args

* handle expection with kwargs

* RUBY-3117 investigate and fix test failures as a result of removing validating_keys (#2620)

* RUBY-3117 investigate and fix test failures as a result of removing validating_keys

* RUBY-3117 remove all references to BSON::Config.validating_keys?

* Update gemfiles/bson_master.gemfile

Co-authored-by: Oleg Pudeyev <39304720+p-mongo@users.noreply.github.com>

Co-authored-by: Oleg Pudeyev <39304720+p-mongo@users.noreply.github.com>

* trying to get the web identity spec to pass

* no web-identity tests for 2.18-stable

* add an explicit dependency on nokogiri

aws wants an XML parser

* Let's not test 2.18-stable against MDB 7+

* remove unsupported configurations for 2.18-stable

---------

Co-authored-by: Dmitry Rybakov <dmitry.rybakov@mongodb.com>
Co-authored-by: Neil Shweky <neilshweky@gmail.com>
Co-authored-by: Oleg Pudeyev <39304720+p-mongo@users.noreply.github.com>

Author: 4 people

.evergreen/auth_aws

@@ -0,0 +1 @@
+../.mod/drivers-evergreen-tools/.evergreen/auth_aws

.evergreen/aws

@@ -5,36 +5,32 @@ $: << File.join(File.dirname(__FILE__), '../spec')
 require 'support/aws_utils'
 require 'optparse'
 
-options = {}
-OptionParser.new do |opts|
-  opts.banner = "Usage: aws [options] command ..."
+def parse_options
+  options = {}
+  OptionParser.new do |opts|
+    opts.banner = "Usage: aws [options] command ..."
 
-  opts.on("-a", "--access-key-id=ID", "AWS access key ID") do |v|
-    options[:access_key_id] = v
-  end
-
-  opts.on("-s", "--secret-access-key=SECRET", "AWS secret access key") do |v|
-    options[:secret_access_key] = v
-  end
+    opts.on("-a", "--access-key-id=ID", "AWS access key ID") do |v|
+      options[:access_key_id] = v
+    end
 
-  opts.on("-r", "--region=REGION", "AWS region") do |v|
-    options[:region] = v
-  end
+    opts.on("-s", "--secret-access-key=SECRET", "AWS secret access key") do |v|
+      options[:secret_access_key] = v
+    end
 
-  # launch-ecs options
-  opts.on('--ec2', 'Use EC2 launch type instead of Fargate') do |v|
-    options[:ec2] = true
-  end
-end.parse!
+    opts.on("-r", "--region=REGION", "AWS region") do |v|
+      options[:region] = v
+    end
 
-case cmd = ARGV.shift
-when 'setup-resources'
-  AwsUtils::Provisioner.new.setup_aws_auth_resources
-when 'reset-keys'
-  AwsUtils::Provisioner.new.reset_keys
-when 'assume-role'
-  arn = ARGV.shift
+    # launch-ecs options
+    opts.on('--ec2', 'Use EC2 launch type instead of Fargate') do |v|
+      options[:ec2] = true
+    end
+  end.parse!
+  options
+end
 
+def assume_role(arn, options)
   orchestrator = AwsUtils::Orchestrator.new(**options)
   if arn.nil?
     arn = AwsUtils::Inspector.new(**options).assume_role_arn
@@ -43,39 +39,36 @@ when 'assume-role'
   puts "AWS_ACCESS_KEY_ID=#{credentials.access_key_id}"
   puts "AWS_SECRET_ACCESS_KEY=#{credentials.secret_access_key}"
   puts "AWS_SESSION_TOKEN=#{credentials.session_token}"
-  #puts "EXPIRATION=#{credentials.expiration.utc.strftime('%Y%m%dT%H%M%SZ')}"
   puts
-when 'set-instance-profile'
-  instance_id = ARGV.shift
+end
+
+def set_instance_profile(instance_id, options)
   unless instance_id
     raise 'Instance id is required'
   end
-
   orchestrator = AwsUtils::Orchestrator.new(**options)
   orchestrator.set_instance_profile(instance_id)
-when 'clear-instance-profile'
-  instance_id = ARGV.shift
+end
+
+def clear_instance_profile(instance_id, options)
   unless instance_id
     raise 'Instance id is required'
   end
 
   orchestrator = AwsUtils::Orchestrator.new(**options)
   orchestrator.clear_instance_profile(instance_id)
-when 'key-pairs'
-  AwsUtils::Inspector.new(**options).list_key_pairs
-when 'launch-ec2'
-  public_key_path, = ARGV
+end
+
+def launch_ec2(public_key_path, options)
   unless public_key_path
     raise "Public key path must be given"
   end
 
   orchestrator = AwsUtils::Orchestrator.new(**options)
   orchestrator.provision_auth_ec2_instance(public_key_path: public_key_path)
-when 'stop-ec2'
-  orchestrator = AwsUtils::Orchestrator.new(**options)
-  orchestrator.terminate_auth_ec2_instance
-when 'launch-ecs'
-  public_key_path, = ARGV
+end
+
+def launch_ecs(public_key_path, options)
   unless public_key_path
     raise "Public key path must be given"
   end
@@ -84,6 +77,35 @@ when 'launch-ecs'
   orchestrator.provision_auth_ecs_task(
     public_key_path: public_key_path,
   )
+end
+
+options = parse_options
+
+case cmd = ARGV.shift
+when 'setup-resources'
+  AwsUtils::Provisioner.new.setup_aws_auth_resources
+when 'reset-keys'
+  AwsUtils::Provisioner.new.reset_keys
+when 'assume-role'
+  arn = ARGV.shift
+  assume_role(arn, options)
+when 'set-instance-profile'
+  instance_id = ARGV.shift
+  set_instance_profile(instance_id, options)
+when 'clear-instance-profile'
+  instance_id = ARGV.shift
+  clear_instance_profile(instance_id, options)
+when 'key-pairs'
+  AwsUtils::Inspector.new(**options).list_key_pairs
+when 'launch-ec2'
+  public_key_path, = ARGV
+  launch_ec2(public_key_path, options)
+when 'stop-ec2'
+  orchestrator = AwsUtils::Orchestrator.new(**options)
+  orchestrator.terminate_auth_ec2_instance
+when 'launch-ecs'
+  public_key_path, = ARGV
+  launch_ecs(public_key_path, options)
 when 'stop-ecs'
   orchestrator = AwsUtils::Orchestrator.new(**options)
   orchestrator.terminate_auth_ecs_task