Skip to content

GODRIVER-2114 Fix failing KMS TLS tests #712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 4, 2021
Merged

GODRIVER-2114 Fix failing KMS TLS tests #712

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2035d75
Initial fix
Aug 2, 2021
5feb780
Updates with logging
Aug 3, 2021
95e7fc5
Start separate servers again
Aug 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 10 additions & 14 deletions .evergreen/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -827,20 +827,18 @@ functions:

start-kms-mock-server:
- command: shell.exec
type: test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing type: test seems right here. The default command_type on L13 is setup. If this task fails it will indicate a setup failure, rather than a test failure (https://github.com/evergreen-ci/evergreen/wiki/Project-Configuration-Files#command-failure-colors)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah setup definitely seems like the right type; not sure why I had test before.

params:
working_dir: src
background: true
script: |
${PREPARE_SHELL}

cd ${DRIVERS_TOOLS}/.evergreen/csfle
cat <<EOF > kms_setup.json
{
"kms_ca_file": "${KMS_CA_FILE}",
"kms_cert_file": "${KMS_CERT_FILE}"
}
EOF
mongo --nodb mock_kms.js
. ./activate_venv.sh
- command: shell.exec
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much like the mock OCSP functions, the first command sets up the local environment in the foreground, and the second command starts the Python mock server in the background. These need to be separated for the tests to consistently find the mock KMS server.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. https://github.com/evergreen-ci/evergreen/wiki/Project-Commands#shellexec notes:

background: if set to true, does not wait for the script to exit before running the next commands

My new hypothesis for the cause of the connection refused errors:

  • Evergreen would run start-kms-mock-server and proceed before the script completed.
  • The Go driver tests started before the mock KMS server started.

Starting the virtual environment in a non-background command before helps. But I think this is still hiding a race.

If the mock KMS server does not establish listening sockets before the Go driver tests run, I suspect the same issue will occur. But, given that the OCSP tasks have a similar setup, I bet the likelihood of the KMS server not starting before the Go tests run is slim to none. If we see it failing in the future, we could consider appending a foreground command to loop until it can establish a connection on port 8000. That seems unnecessary for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds exactly right. I think all the current mock servers in testing (KMS, OCSP and maybe load balancer?) have this racey behavior. It seems that if you only have the server-starting call in the background function, the tests pretty much never start before the server. So, if we start to see failures we can consider something like a foreground loop.

params:
background: true
script: |
cd ${DRIVERS_TOOLS}/.evergreen/csfle
./kmstlsvenv/bin/python3 -u kms_http_server.py -v --ca_file ../x509gen/ca.pem --cert_file ../x509gen/${CERT_FILE} --port 8000

run-kms-tls-test:
- command: shell.exec
Expand Down Expand Up @@ -1719,8 +1717,7 @@ tasks:
SSL: "nossl"
- func: start-kms-mock-server
vars:
KMS_CA_FILE: "ca.pem"
KMS_CERT_FILE: "expired.pem"
CERT_FILE: "expired.pem"
- func: run-kms-tls-test
vars:
KMS_TLS_TESTCASE: "INVALID_CERT"
Expand All @@ -1738,8 +1735,7 @@ tasks:
SSL: "nossl"
- func: start-kms-mock-server
vars:
KMS_CA_FILE: "ca.pem"
KMS_CERT_FILE: "wrong-host.pem"
CERT_FILE: "wrong-host.pem"
- func: run-kms-tls-test
vars:
KMS_TLS_TESTCASE: "INVALID_HOSTNAME"
Expand Down
2 changes: 1 addition & 1 deletion mongo/integration/client_side_encryption_prose_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1078,7 +1078,7 @@ func TestClientSideEncryptionProse(t *testing.T) {
bson.D{
{"region", "us-east-1"},
{"key", "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"},
{"endpoint", "mongodb://127.0.0.1:8000"},
{"endpoint", "127.0.0.1:8000"},
},
))
assert.NotNil(mt, err, "expected CreateDataKey error, got nil")
Expand Down