CDRIVER-4645 migrate and run load balancer tests #1414

kevinAlbs · 2023-09-27T19:12:22Z

Summary

This PR partially resolves CDRIVER-4645:

Migrate previously unused *-loadbalanced-* tasks to new Evergreen config.
Run simple_http_server.py on port 18000 to avoid conflict with haproxy use of port 8000.
Update load balanced tests to expect the hello monitoring command. CDRIVER-4193 Ensure OP_MSG for handshakes and fix RPC op_egress counters #1256 switched from legacy hello to hello when in Load Balancing mode.

New tasks tested in this patch build: https://spruce.mongodb.com/version/65147abba4cf4765a33cf54d

Background & Motivation

Testing in CI is partially motivated to test a fix specific to load balancers in CI: CDRIVER-4718.

Specifications describe load balanced test requirements

For each server version that supports load balanced clusters, drivers MUST add two Evergreen tasks: one with a sharded cluster with both authentication and TLS enabled and one with a sharded cluster with authentication and TLS disabled.

The Load Balancer Support specification documents Minimum Server Version: 5.0

Use of rhel8.7

The previously unused *-loadbalanced-* tasks used ubuntu1804. https://wiki.corp.mongodb.com/display/DBDEVPROD/Guidelines+around+Evergreen+distros suggest use of newer distros.

ubuntu2204 was tried. The ubuntu2204 distro does not appear to have the necessary dependency haproxy installed. The rhel8.7 distro is instead used. Sanitizers are excluded due to observed failure to configure.

Port 8000 conflicts with HAProxy started by run-load-balancer.sh in DET

Update to commit mongodb/specifications@5af2ab7 to fix test failure when an error is expected.

eramongodb

Minor suggestions; otherwise, LGTM.

.evergreen/config_generator/components/loadbalanced.py

eramongodb · 2023-09-28T16:01:20Z

.evergreen/config_generator/components/loadbalanced.py

+
+def functions():
+    return {
+        'start load balancer': [


Suggested change

'start load balancer': [

'start-load-balancer': [

Recommend using - instead of spaces for consistency in new EVG config generator scripts.

.evergreen/config_generator/components/loadbalanced.py

eramongodb · 2023-09-28T16:11:36Z

.evergreen/config_generator/components/loadbalanced.py

+        name=f"loadbalanced-test-{auth_str}-{ssl_str}-{server_version}",
+        depends_on=[EvgTaskDependency(name="loadbalanced-compile")],
+        # Use `rhel8.7` distro. `rhel8.7` distro includes necessary dependency: `haproxy`.
+        run_on="rhel8.7-small",


Consider including {distro_str} in the task name (using make_distro_str()) for consistency with other new EVG config generator scripts, even if all the (current) tasks are on the same/single distro.

Also recommend specifying the run_on distro using find_small_distro() from config_generator.etc.distros rather than hard-coding the distro string.

Done. Required adding rhel8.7 to the RHEL_DISTROS list.

eramongodb · 2023-09-28T16:13:50Z

.evergreen/config_generator/components/loadbalanced.py

+        depends_on=[EvgTaskDependency(name="loadbalanced-compile")],
+        # Use `rhel8.7` distro. `rhel8.7` distro includes necessary dependency: `haproxy`.
+        run_on="rhel8.7-small",
+        tags=['loadbalanced'],


Recommend including distro_name, compiler, arch, ssl, and auth in task tags for EVG CLI friendliness.

Done. Omitted arch since the no arch is specified in the compile or test task.

Also added the ssl tag to other test tasks. The "nossl" value was omitted due to use on legacy config variants.

eramongodb · 2023-09-28T16:15:43Z

.evergreen/config_generator/components/loadbalanced.py

+    yield Task(
+        name="loadbalanced-compile",
+        run_on="rhel8.7-large",
+        tags=['loadbalanced'],


Recommend including distro_name, compiler, and arch in task tags for EVG CLI friendliness.

Co-authored-by: Ezra Chung <[email protected]>

I initially mistakenly thought `find_large_distro` returned a string.

For consistency with other task naming.

kevinAlbs

Updated. Thank you for the suggested changes.

Latest changes verified with this patch build: https://spruce.mongodb.com/version/6515caaa3627e0fb1f125331

kevinAlbs · 2023-09-28T18:04:37Z

.evergreen/config_generator/components/loadbalanced.py

+        name=f"loadbalanced-test-{auth_str}-{ssl_str}-{server_version}",
+        depends_on=[EvgTaskDependency(name="loadbalanced-compile")],
+        # Use `rhel8.7` distro. `rhel8.7` distro includes necessary dependency: `haproxy`.
+        run_on="rhel8.7-small",


Done. Required adding rhel8.7 to the RHEL_DISTROS list.

kevinAlbs · 2023-09-28T18:31:35Z

.evergreen/config_generator/components/loadbalanced.py

+        depends_on=[EvgTaskDependency(name="loadbalanced-compile")],
+        # Use `rhel8.7` distro. `rhel8.7` distro includes necessary dependency: `haproxy`.
+        run_on="rhel8.7-small",
+        tags=['loadbalanced'],


Done. Omitted arch since the no arch is specified in the compile or test task.

Also added the ssl tag to other test tasks. The "nossl" value was omitted due to use on legacy config variants.

eramongodb · 2023-09-28T19:38:09Z

.evergreen/config_generator/etc/sanitizers/test.py

+            # Do not add `nossl` tag to prevent being selected by legacy config variants.
+            # Remove the `if` when CDRIVER-4571 is resolved.
+            if SSL != 'nossl':
+                test_tags += SSL


I believe this should be test_tags += [SSL] or test_tags.append(SSL) to avoid the following result in the generated config:

[sanitizers-matrix-asan, ..., server, "4.2", o, p, e, n, s, s, l]

Also applies to other instances in this PR.

Oof. Thank you for the catch.

eramongodb · 2023-09-28T19:39:03Z

.evergreen/config_generator/etc/distros.py

+    Distro(name='rhel8.7-large', os='rhel', os_type='linux', os_ver='8.7', size='large'),
+    Distro(name='rhel8.7-small', os='rhel', os_type='linux', os_ver='8.7', size='small'),


Suggested change

Distro(name='rhel8.7-large', os='rhel', os_type='linux', os_ver='8.7', size='large'),

Distro(name='rhel8.7-small', os='rhel', os_type='linux', os_ver='8.7', size='small'),

Distro(name='rhel87-large', os='rhel', os_type='linux', os_ver='8.7', size='large'),

Distro(name='rhel87-small', os='rhel', os_type='linux', os_ver='8.7', size='small'),

Suggest omitting . for consistency with other RHEL distro names.

kevinAlbs · 2023-10-02T13:23:32Z

Latest changes verified with this patch build: https://spruce.mongodb.com/version/6515d9307742ae813127938b/

adriandole

LGTM, minor suggestion to keep the distro string and compiler consistent if the compiler ever changes.

adriandole · 2023-10-10T20:53:40Z

.evergreen/config_generator/components/loadbalanced.py

+            bash_exec(
+                command_type=EvgCommandType.TEST,
+                env={
+                    'CC': 'gcc',


Suggested change

'CC': 'gcc',

'CC': _COMPILER,

adriandole · 2023-10-10T20:54:18Z

.evergreen/config_generator/components/loadbalanced.py

+                'AUTH': auth_str,
+                'SSL': ssl_str,
+                'LOADBALANCED': 'loadbalanced',
+                'CC': 'gcc',


Suggested change

'CC': 'gcc',

'CC': _COMPILER,

kevinAlbs · 2023-10-11T17:52:26Z

Latest changes verified with this patch: https://spruce.mongodb.com/version/6526cff12fbabe871ea125cf

kevinAlbs added 6 commits September 27, 2023 14:54

run simple_http_server.py on port 18000

fd51edf

Port 8000 conflicts with HAProxy started by run-load-balancer.sh in DET

update load_balancers specification tests

be45ce6

Update to commit mongodb/specifications@5af2ab7 to fix test failure when an error is expected.

use hello, not isMaster in test failpoint

10b2e82

add new loadbalanced Evg config

8cfab37

remove legacy load balanced config

7047bf0

regenerate Evergreen config

8005cc1

kevinAlbs marked this pull request as ready for review September 27, 2023 19:14

kevinAlbs requested review from eramongodb and adriandole September 27, 2023 19:14

eramongodb approved these changes Sep 28, 2023

View reviewed changes

eramongodb reviewed Sep 28, 2023

View reviewed changes

kevinAlbs mentioned this pull request Sep 28, 2023

CDRIVER-4718 store recoveryToken for Load Balanced topology #1417

Merged

kevinAlbs and others added 14 commits September 28, 2023 13:34

Sort imports

7076193

Co-authored-by: Ezra Chung <[email protected]>

Use display name "loadbalanced"

7f619d4

Co-authored-by: Ezra Chung <[email protected]>

rename task in order of feature importance

40aef90

Co-authored-by: Ezra Chung <[email protected]>

use dashes, not spaces, in function name

d374a60

use make_distro_str and apply to task names

f3918c9

add missing 'f'

36e12f3

add rhel8.7 distro entries

986fbfc

use find_small_distro and find_large_distro

6abfc32

add more task tags

4f70353

specify return types in distro helpers

c07560f

I initially mistakenly thought `find_large_distro` returned a string.

add SSL tag to other tasks

e3feca9

use openssl, not ssl

0d91d63

For consistency with other task naming.

fix arg to FetchBuild

75558b4

regenerate config

899fafa

kevinAlbs commented Sep 28, 2023

View reviewed changes

eramongodb reviewed Sep 28, 2023

View reviewed changes

fix appending to test_tags

a1da2d9

kevinAlbs added 2 commits September 28, 2023 15:50

use alias of rhel8.7 distro

68f2d97

regenerate config

3d3dc64

kevinAlbs mentioned this pull request Oct 2, 2023

Unskip tests failing to connect to port 8000 #1420

Merged

adriandole approved these changes Oct 10, 2023

View reviewed changes

kevinAlbs added 3 commits October 11, 2023 12:36

Merge branch 'master' into CDRIVER-4645.schedule-lb

5064560

regenerate config

5697708

use _COMPILER

26bc018

kevinAlbs merged commit 82d593a into mongodb:master Oct 11, 2023

		Distro(name='rhel8.7-large', os='rhel', os_type='linux', os_ver='8.7', size='large'),
		Distro(name='rhel8.7-small', os='rhel', os_type='linux', os_ver='8.7', size='small'),

CDRIVER-4645 migrate and run load balancer tests #1414

CDRIVER-4645 migrate and run load balancer tests #1414

Uh oh!

Conversation

kevinAlbs commented Sep 27, 2023

Summary

Background & Motivation

Use of rhel8.7

Uh oh!

eramongodb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinAlbs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinAlbs commented Oct 2, 2023

Uh oh!

adriandole left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinAlbs commented Oct 11, 2023

Uh oh!

Uh oh!