CXX-3126 Refactor EVG config to use config_generator (#1244)

* uv: convert Python requirements into pyproject.toml
* generator: import config_generator from C Driver
* Remove obsolete reference to install.sh (6a49989)
* evg: fix tasks which enable extra alignment
* etc: update release instructions with uv commands
* evg: backport root-level config options to .mci.yml
* evg: improve formatting of CMake arguments list output
* Avoid splitting `-D<arg>` in CMake arguments list output
* Add a README documenting the Evergreen config generator

Author: eramongodb

.evergreen/README.md

@@ -0,0 +1,36 @@
+# Evergreen Config Generation
+
+## Generation
+
+Use [Astral uv](https://docs.astral.sh/uv/) to run the `config_generator/generate.py` script from the project root directory:
+
+```bash
+uv run --frozen .evergreen/config_generator/generate.py
+```
+
+Python binary and package requirements are defined in the project root directory's `pyproject.toml` file. When the `pyproject.toml` file is updated, omit `--frozen` to allow `uv.lock` to be updated.
+
+## Layout
+
+The contents of this directory are organized as follows:
+
+- `config.yml`: the root Evergreen config file.
+- `generated_configs`: generated Evergreen config files included by `config.yml`.
+- `config_generator`: Python scripts used to generate config files under `generated_configs`.
+- `scripts`: shell scripts used by the generated Evergreen config.
+
+## Config Generator
+
+Config generator scripts are organized into three subdirectories.
+
+### Components
+
+These scripts define Evergreen functions, tasks, task groups, and build variants. Components which only define Evergreen functions (for reuse by multiple components) are grouped under the `funcs` subdirectory. All other components (which define a task, task group, or build variant) are located outside the `funcs` directory.
+
+### Etc
+
+These scripts define helper utilities used by components, but do not define any Evergreen functions, tasks, task groups, or build variants themselves. These scripts are only imported by scripts under `components`.
+
+### Generators
+
+These scripts are imported by `generate.py` and are each responsible for generating an Evergreen config file under `generated_configs` (`functions.py` generates `functions.yml`, etc.). These scripts only scan the contents of the `components` directory.

.evergreen/build_example_projects.sh

@@ -0,0 +1,38 @@
+# Default timeout for all tasks is 1 hour.
+# May be overridden by individual tasks.
+exec_timeout_secs: 3600
+
+# Default command type is 'system' to indicate failures unrelated to tests or
+# their setup. Most commands should define their type as 'setup' or 'test'.
+#   - setup:  before test (lavendar).
+#   - test:   during test (red).
+#   - system: after test (purple).
+command_type: system
+
+# Ensure Evergreen tests earlier commits when a task fails.
+stepback: true
+
+# Ensure failure in pre commands is also considered task failure.
+pre_error_fails_task: true
+
+# Too many post commands are not designed to handle errors.
+# TODO: set to true once failing post commands have been moved into
+# teardown_group commands of appropriate task groups.
+post_error_fails_task: false
+
+# Commands run after all tasks (excluding those in task groups).
+# Use sparingly and ensure they are error-proof.
+# TODO: move into teardown_group commands of appropriate task groups.
+post:
+    - func: "stop_mongod"
+    - func: "backtrace"
+    # Workaround for CXX-2040
+    # - func: "upload working dir"
+    - func: "upload mongo orchestration artifacts"
+    - func: "upload code coverage"
+
+include:
+  - filename: .evergreen/generated_configs/functions.yml
+  - filename: .evergreen/generated_configs/task_groups.yml
+  - filename: .evergreen/generated_configs/tasks.yml
+  - filename: .evergreen/generated_configs/variants.yml

.evergreen/config.yml

@@ -0,0 +1,188 @@
+from config_generator.components.funcs.install_c_driver import InstallCDriver
+
+from config_generator.etc.distros import find_large_distro
+from config_generator.etc.function import Function, merge_defns
+from config_generator.etc.utils import TaskGroup, bash_exec
+
+from shrub.v3.evg_command import EvgCommandType, git_get_project, s3_put
+from shrub.v3.evg_task import EvgTask, EvgTaskRef
+from shrub.v3.evg_build_variant import BuildVariant, DisplayTask
+
+
+TAG = 'abi-stability'
+
+
+# pylint: disable=line-too-long
+# fmt: off
+MATRIX = [
+    ('polyfill', 11),
+    ('stdlib',   17),
+]
+# fmt: on
+# pylint: enable=line-too-long
+
+
+class AbiComplianceCheck(Function):
+    name = 'abi-compliance-check'
+    commands = [
+        bash_exec(
+            command_type=EvgCommandType.SETUP,
+            script='mongo-cxx-driver/.evergreen/scripts/abi-compliance-check-setup.sh'
+        ),
+        bash_exec(
+            command_type=EvgCommandType.TEST,
+            script='mongo-cxx-driver/.evergreen/scripts/abi-compliance-check-test.sh'
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/html',
+            display_name='ABI Compliance Check (Stable): ',
+            local_files_include_filter='cxx-abi/compat_reports/**/compat_report.html',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abi-compliance-check/abi/',
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/plain',
+            display_name='ABI Compliance Check (Stable): ',
+            local_files_include_filter='cxx-abi/logs/**/log.txt',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abi-compliance-check/abi/',
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/html',
+            display_name='ABI Compliance Check (Unstable): ',
+            local_files_include_filter='cxx-noabi/compat_reports/**/compat_report.html',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abi-compliance-check/noabi/',
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/plain',
+            display_name='ABI Compliance Check (Unstable): ',
+            local_files_include_filter='cxx-noabi/logs/**/log.txt',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abi-compliance-check/noabi/',
+        ),
+    ]
+
+
+class Abidiff(Function):
+    name = 'abidiff'
+    commands = [
+        bash_exec(
+            command_type=EvgCommandType.SETUP,
+            script='mongo-cxx-driver/.evergreen/scripts/abidiff-setup.sh'
+        ),
+        bash_exec(
+            command_type=EvgCommandType.TEST,
+            script='mongo-cxx-driver/.evergreen/scripts/abidiff-test.sh'
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/plain',
+            display_name='abidiff (Stable): ',
+            local_files_include_filter='cxx-abi/*.txt',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abidiff/abi/',
+        ),
+        s3_put(
+            command_type=EvgCommandType.SYSTEM,
+            aws_key='${aws_key}',
+            aws_secret='${aws_secret}',
+            bucket='mciuploads',
+            content_type='text/plain',
+            display_name='abidiff (Unstable): ',
+            local_files_include_filter='cxx-noabi/*.txt',
+            permissions='public-read',
+            remote_file='mongo-cxx-driver/${branch_name}/${revision}/${version_id}/${build_id}/${execution}/abidiff/noabi/',
+        ),
+    ]
+
+
+class AbiProhibitedSymbols(Function):
+    name = 'abi-prohibited-symbols'
+    commands = bash_exec(
+        command_type=EvgCommandType.TEST,
+        script='mongo-cxx-driver/.evergreen/scripts/abi-prohibited-symbols-test.sh'
+    )
+
+
+def functions():
+    return merge_defns(
+        AbiComplianceCheck.defn(),
+        Abidiff.defn(),
+        AbiProhibitedSymbols.defn(),
+    )
+
+
+def tasks():
+    distro_name = 'ubuntu2204'
+    distro = find_large_distro(distro_name)
+
+    return [
+        EvgTask(
+            name=func.name,
+            tags=[TAG, func.name, distro_name],
+            run_on=distro.name,
+            commands=[func.call()],
+        )
+        for func in [AbiComplianceCheck, Abidiff, AbiProhibitedSymbols]
+    ]
+
+
+def task_groups():
+    return [
+        TaskGroup(
+            name=f'tg-{TAG}',
+            max_hosts=-1,
+            setup_group_can_fail_task=True,
+            setup_task=[
+                git_get_project(directory='mongo-cxx-driver'),
+                InstallCDriver.call(),
+                bash_exec(
+                    include_expansions_in_env=['cxx_standard'],
+                    script='mongo-cxx-driver/.evergreen/scripts/abi-stability-setup.sh'
+                ),
+            ],
+            tasks=[f'.{TAG}'],
+            teardown_task_can_fail_task=True,
+            teardown_task=[bash_exec(script='rm -rf *'),],
+        )
+    ]
+
+
+def variants():
+    return [
+        BuildVariant(
+            name=f'abi-stability-{name}',
+            display_name=f'ABI Stability Checks ({name})',
+            expansions={
+                'cxx_standard': f'{cxx_standard}',  # Use a polyfill library.
+            },
+            tasks=[EvgTaskRef(name='tg-abi-stability')],
+            display_tasks=[
+                DisplayTask(
+                    name=f'ABI Stability Checks ({name})',
+                    execution_tasks=[f'.{TAG}'],
+                )
+            ],
+        )
+        for name, cxx_standard in MATRIX
+    ]