CDRIVER-3620 Audit compile and test scripts #1187

eramongodb · 2023-01-20T20:16:54Z

Description

This PR is part of CDRIVER-3620 and is focused on improving the quality and use of various Evergreen scripts pertaining to compilation and testing.

Verified by this patch.

Do not run tests in `compile` tasks

The most significant change in thiS PR is the removal of tests run after compilation in many debug-compile-* tasks. These "mock tests" are almost always immediately followed by corresponding test-* tasks that have equal if not greater code coverage. Therefore, execution of tests has been completely moved out of compilation-related scripts.

Note, mock server testing is now covered only by the test-asan-memcheck-mock-server task, as regular test tasks set MONGOC_TEST_SKIP_MOCK="on" by default. This can be removed if we want to include execution of mock server tests in most test tasks.

This reduces task duration of most compile tasks by over 50%. Some examples:

debug-compile-nosasl-nossl on ubuntu2004-large: 2m 57s -> 1m 5s.
debug-compile-sasl-openssl on rhel70-large: 3m 15s -> 55s.
release-compile on ubuntu1404-large: 2m 55s -> 1m 16s.
debug-compile-sspi-winssl on windows-64-vs2017-small: 8m 25s -> 5m 17s.
build-and-run-authentication-tests-openssl-1.0.1-fips: 5m 36s -> 3m 2s.

Dependent tasks are therefore able to be executed (in parallel) sooner, reducing the overall runtime of a given Evergreen patch.

Coverage Tasks

Because of the remove of test execution in compile tasks, the debug-compile-coverage task had to be redefined to run and collect coverage data separately. Note: the coverage of tests executed in compile tasks had been very small anyways (MONGOC_TEST_SKIP_LIVE=on and MONGOC_TEST_SKIP_SLOW=on)...

The pattern used for existing test-coverage-* tasks was adopted. A single compile coverage function is used to build the source with COVERAGE=ON, which is then used by subsequent commands in a given test-coverage-* task.

For now, all coverage tasks has been consolidated into a single test-coverage-latest-replica-set-auth-sasl-openssl-cse task run on Ubuntu 18.04 is currently being tested. This set can be expanded as needed by manipulating the required/prohibited conditions in IntegrationTask.

add_expansions_to_env

This very useful parameter to Evergreen functions removes the need for the common export X="${X}" pattern used to translate Evergreen expansions into Bash environment variables. Relevant Evergreen functions and scripts have been updated accordingly to utilize Evergreen expansions (now automatically environment variables) directly within Bash scripts.

OpenSSL

Per CDRIVER-3562, testing with OpenSSL versions older than 1.0.1 (0.9.8 and 1.0.0) have been removed.

Authentication Tests

The silent=True parameter for the run auth tests function was removed to hopefully assist in debugging failures when they occur. This can be reverted if necessary (e.g. if the value of Evergreen expansions such as ${atlas_free} are considered sensitive).

bypass_dlclose

A followup to #1177. Instead of evaluating a target command within a bypass_dlclose function (e.g. bypass_dlclose command with bypass), the function was refactored to allow more flexible use of LD_PRELOAD by returning the paths to libraries to preload as a string (output) instead. This also included the removal of trap 'rm -rf "$tmp"' EXIT, which caused difficult-to-diagnose on-script-exit errexit failures due to the tmp variable being overwritten during execution of the wrapped command. Instead, the bypass_dlclose.so library is now left in a temporary directory without requiring use of complex cleanup routines.

Bash and Hygienic Scripts

Many scripts were refactored to utilize Bash features to improve readability and robustness. Of note, use of "${<varname>:?}" to assert variables that should not be null or unset, arrays to satisfy shellcheck warnings regarding variable expansions, and use of "${OSTYPE}" and "${HOSTTYPE}" instead of $OS+uname to reduce cross-script variable dependencies/leakage.

Bugfixes Galore

As part of auditing the Evergreen scripts, numerous bugs were discovered that were being hidden or ignored.

OpenSSL/LibreSSL Compilation, Installation, and Selection

The install-ssl.sh script masked compilation and installation failure with || true, leading to the subsequent libmongoc compilation linking against system-installed OpenSSL library (usually 1.1.1) that do not match the intended library. After fixing this issue, the build-and-run-authentication-tests-* tasks now actually build and run against the correct SSL libraries.

OpenSSL versions prior to 1.1.0 have trouble being built in parallel (probably what motivated the liberal use of || true). The order of targets built has been explictly enumerated to address these build failures while still maximizing build parallelism (e.g. build-and-run-authentication-tests-openssl-1.0.1-fips: 5m 36s -> 2m 36s).

The OpenSSL 1.1.0 library "letter" release ("exclusively contain bug and security fixes and no new features") had to be bumped from f to l to address Perl incompatibility issues during configuration: "glob" is not exported by the File::Glob module.

Measures were also taken to reduce the verbosity of output by silencing non-important build messages (warnings and errors are still emitted).

Fixing this behavior also triggered/exposed compilation failure for build-and-run-authentication-tests-openssl-1.1.0 due to the following error when attempting to execute CMake:

cmake: /data/mci/eaf46782b3251c2b699378f69145c129/mongoc/install-dir/lib/libssl.so.1.1: version 'OPENSSL_1_1_1' not found (required by /usr/lib/libcurl.so.4)
cmake: /data/mci/eaf46782b3251c2b699378f69145c129/mongoc/install-dir/lib/libcrypto.so.1.1: version 'OPENSSL_1_1_1' not found (required by /usr/lib/libssh2.so.1)

It is unclear why this error only occurs with OpenSSL 1.1.0. I have elected to leave the task in a failing status for now. I would appreciate some assistance with this issue, but I do not consider it a high priority.

`CFLAGS` vs. Maintainer Flags

Some compiler flags such as -Wno-redundant-decls set via CFLAGS were being effectively ignored due to maintainer flags being appended to CMAKE_C_FLAGS rather than prepended. This resulted in the order of flags being <cflags> <maintainer-flags> such that -Wno-redundant-decls was eventually undone by a following -Wredundant-decls. Maintainer flags are now prepended to CMAKE_C_FLAGS so CFLAGS can continue to be used to silence or overwrite default maintainer flags.

`X` and `ENABLE_X` Environment Variables

There were some inconsistencies regarding how flags such as SSL, SASL, and etc. were being defined in Evergreen and forwarded/set within compilation or test scripts, making it difficult to discern if/when a given variable was not being handled properly (i.e. defined as 1 but compared against 'ON'). An attempt to improve their consistency was made, assisted by the introduction of check_var_opt and check_var_req utility functions to print and assert the values of environment variables used by scripts.

Error Handling in BAT Scripts

Errors were being ignored in .bat scripts due to no set -o errexit-like equivalent. || exit /b were added to potentially-failing commands appropriately.

rcsanchez97 · 2023-01-20T22:12:40Z

This are just some comments/responses to your summary writeup. I will follow with any code-specific comments that I might have in a separate review.

Related to CDRIVER-3562 and the dropping of OpenSSL 0.9.8 and 1.0.0, note that the toolchain still contains older SSL implementations, including 0.9.8 and 1.0.0: https://github.com/10gen/mongo-c-toolchain/blob/c6f10761338dc7e94545d7fdc1d1a91fb1818f63/install-ssl.sh#L85-L86

Also, you might take a look at how the paralellism issue is handled in that script.

Note that this is a different install-ssl.sh script. It is located in the toolchain project, not in the C driver.

It might be worth updating that in conjunction with CDRIVER-3562.

Here is the explanation of the build-and-run-authentication-tests-openssl-1.1.0 errors from
CMake.

The OpenSSL version being built is 1.1.0. It is evidently then being placed ahead of the system's libssl in the search path (whether by adding its location to ld.so.conf and executing ldconfig, or by setting LD_LIBRARY_PATH in the environment). However, CMake links to system libraries which are transitively linked to OpenSSL via a versioned symbol dependency. Then when cmake is invoked, its linkages cannot be satisfied. Consider

$ ldd /usr/bin/cmake |egrep 'ssh2|curl'
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007efe999a2000)
        libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007efe98fff000)
$ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4 |grep ssl
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f5f0a09a000)
$ objdump -T /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep OPENSSL_1_1 | cut -c50-63 | sort -u 
OPENSSL_1_1_0
OPENSSL_1_1_1

This happens when CMake was built and linked against a newer OpenSSL library that includes the OPENSSL_1_1_1 symbols and then it instead finds a library without those newer symbols.

In any case, the solution seems to be (and I say this without having looked at the implementation of the Evergreen task itself), to not push the newly build OpenSSL 1.1.0 into the front of the entire environment. Rather, build it and then just tell CMake to go find it only during the execution of the build itself.

If you like, I can tackle this after the PR gets merged.

eramongodb · 2023-01-20T22:36:13Z

you might take a look at how the paralellism issue is handled in that script.

Despite appearances, toolchain's solution to the OpenSSL parallel build problem seems to be to ignore it by setting parallelism to 1, making the use of the -j option redundant.

the solution seems to be [...] to not push the newly build OpenSSL 1.1.0 into the front of the entire environment. Rather, build it and then just tell CMake to go find it only during the execution of the build itself.

I also attempted this workaround, but the goal of permitting CMake to run while dynamically linking against OpenSSL 1.1.1 conflicts with the intent of allowing it to find OpenSSL 1.1.0 during build configuration (it is required during the configuration step, not just during the build step). It is unclear why this only appears to be an issue with OpenSSL 1.1.0 and not with other OpenSSL versions (see linked-to patch, e.g. OpenSSL 1.0.2 seems to work just fine).

vector-of-bool

Not fully reviewed, but I have some prelim comments.

BIG +1 for the build time improvements (my favorite kind of improvements).

.evergreen/compile-unix.sh

build/cmake/MaintainerFlags.cmake

.evergreen/run-ocsp-cache-test.sh

rcsanchez97 · 2023-01-20T23:57:58Z

you might take a look at how the paralellism issue is handled in that script.

Despite appearances, toolchain's solution to the OpenSSL parallel build problem seems to be to ignore it by setting parallelism to 1, making the use of the -j option redundant.

That's a fair point.

the solution seems to be [...] to not push the newly build OpenSSL 1.1.0 into the front of the entire environment. Rather, build it and then just tell CMake to go find it only during the execution of the build itself.

I also attempted this workaround, but the goal of permitting CMake to run while dynamically linking against OpenSSL 1.1.1 conflicts with the intent of allowing it to find OpenSSL 1.1.0 during build configuration (it is required during the configuration step, not just during the build step). It is unclear why this only appears to be an issue with OpenSSL 1.1.0 and not with other OpenSSL versions (see linked-to patch, e.g. OpenSSL 1.0.2 seems to work just fine).

Hmm. That is interesting. I'll try to take a closer look at it.

rcsanchez97 · 2023-01-22T00:59:40Z

the solution seems to be [...] to not push the newly build OpenSSL 1.1.0 into the front of the entire environment. Rather, build it and then just tell CMake to go find it only during the execution of the build itself.

I also attempted this workaround, but the goal of permitting CMake to run while dynamically linking against OpenSSL 1.1.1 conflicts with the intent of allowing it to find OpenSSL 1.1.0 during build configuration (it is required during the configuration step, not just during the build step).

Here is one possible way to deal with the issue:

diff --git a/.evergreen/config.yml b/.evergreen/config.yml
index 366a307a1..d76a67311 100644
--- a/.evergreen/config.yml
+++ b/.evergreen/config.yml
@@ -14367,7 +14367,7 @@ tasks:
       shell: bash
       script: |-
         set -o errexit
-        env DEBUG=ON SASL=OFF SSL=OPENSSL bash .evergreen/compile.sh
+        env OPENSSL_ROOT_DIR=$(pwd)/install-dir-ssl DEBUG=ON SASL=OFF SSL=OPENSSL bash .evergreen/compile.sh
   - func: run auth tests
   - func: upload build
 - name: build-and-run-authentication-tests-libressl-2.5
diff --git a/.evergreen/install-ssl.sh b/.evergreen/install-ssl.sh
index 425e8b056..b58fb5af7 100644
--- a/.evergreen/install-ssl.sh
+++ b/.evergreen/install-ssl.sh
@@ -24,7 +24,7 @@ script_dir="$(to_absolute "$(dirname "${BASH_SOURCE[0]}")")"
 declare mongoc_dir
 mongoc_dir="$(to_absolute "${script_dir}/..")"
 
-declare install_dir="${mongoc_dir}/install-dir"
+declare install_dir="${mongoc_dir}/install-dir-ssl"
 
 declare -a ssl_extra_flags

Here is a patch build showing that it works.

This would actually require changing the logic in build/evergreen_config_lib/tasks.py, however, which my patch didn't deal with as I was going for quick and dirty. The general idea is "install SSL into a directory which isn't put into the LD_LIBRARY_PATH sequence and instead direct CMake's FindOpenSSL module to the correct location with the OPENSSL_ROOT_DIR variable".

It is unclear why this only appears to be an issue with OpenSSL 1.1.0 and not with other OpenSSL versions (see linked-to patch, e.g. OpenSSL 1.0.2 seems to work just fine).

The reason for that is that OpenSSL 1.1.0 and 1.1.1 both generate a library file named libssl.so.1.1. On the platform in question (Arch Linux) you have CMake linked against libssl.so.1.1 that has an actual version of 1.1.1# with symbols marked OPENSSL_1_1_1. Then Evergreen calls a function to build OpenSSL 1.1.0#, which creates another file libssl.so.1.1 that does not have any OPENSSL_1_1_1 symbols and which , thanks to LD_LIBRARY_PATH gets found first by the dynamic linker for anything in the environment that wants a linkage to the library libssl.so.1.1. For the OpenSSL 1.0.2# build this isn't a problem because we are building a library file named libssl.so.1.0, which does not satisfy the linker's search criteria for the linkages of libcurl and libssh2, owing to them wanting libssl.so.1.1 instead (on this platform).

Another potential workaround is to use an ubuntu22 host for the OpenSSL build tasks. On Ubuntu 22 the transitive libssl dependencies of CMake are satisfied by libssl.so.3. Given than all of our SSL-specific tasks use OpenSSL versions < 3, we should not encounter the library collision problem there. Note that I have not yet gotten around to adding a variant for Ubuntu 22 (CDRIVER-4433).

kkloberdanz

Looks good! Just a minor suggestion for what Colby brought up with bounding parallelism in make -j commands to a reasonable upper bound.

eramongodb · 2023-01-23T17:52:39Z

@rcsanchez97 Thank you for the deep dive! Will explore the suggested changes (leaning towards using OPENSSL_ROOT_DIR if able).

eramongodb · 2023-01-23T18:38:34Z

Related to CDRIVER-3562 and the dropping of OpenSSL 0.9.8 and 1.0.0, note that the toolchain still contains older SSL implementations, including 0.9.8 and 1.0.0: https://github.com/10gen/mongo-c-toolchain/blob/c6f10761338dc7e94545d7fdc1d1a91fb1818f63/install-ssl.sh#L85-L86

@rcsanchez97 Regarding CDRIVER-3562, I feel comfortable dropping 0.9.8 and 1.0.0 given the comment in CDRIVER-3562, which refers to libmongoc documentation that effectively mandates OpenSSL 1.0.1 or newer already:

The MongoDB C Driver uses OpenSSL, if available, on Linux and Unix platforms (besides macOS). Industry best practices and some regulations require the use of TLS 1.1 or newer, which requires at least OpenSSL 1.0.1. [...] Ensure your system’s OpenSSL is a recent version (at least 1.0.1), or install a recent version in a non-system path and build against it with [...]

I think this would also justify dropping 0.9.8 and 1.0.0 from the toolchain.

eramongodb · 2023-01-23T22:50:00Z

Latest changes verified by this patch.

Custom OpenSSL libraries (via install-ssl.sh) are now installed under a separate ${mongoc_dir}/openssl-install-dir (LibreSSL is still installed to ${mongoc_dir}/install-dir) which is specified to CMake during compilation via a combination of -DOPENSSL_ROOT_DIR=/path/to/openssl-install-dir and PKG_CONFIG_PATH.

This necessitated the addition of openssl-install-dir/lib to LD_LIBRARY_PATH prior to invoking binaries compiled against a custom OpenSSL library. Calls to ldd in run-auth-tests.sh were added to assist with validating the correct libraries are being linked against:

if command -v ldd >/dev/null; then
  # MacOS does not have ldd.
  LD_LIBRARY_PATH="${openssl_lib_prefix}" ldd "${ping}" | grep "libssl"
  LD_LIBRARY_PATH="${openssl_lib_prefix}" ldd "${test_gssapi}" | grep "libssl"
fi

However, the build-and-run-authentication-tests-openssl-1.1.0 task is continuing to fail with the following error when running the Atlas Free Tier test (the preceeding X.509 test appears to succeed):

TLS handshake failed: certificate verify failed (20): unable to get local issuer certificate calling hello on '<...>.net:27017'

Examining the logs, it appears to be both compiling and linking wih the custom OpenSSL library correctly, but the error seems to suggest there may still be either a mismatch in library versions or misconfiguration of the environment.

vector-of-bool

LGTM

.evergreen/env-var-utils.sh

eramongodb · 2023-01-24T16:23:54Z

Latest changes verified by this patch. Grep failing to match against libssl was causing failures on Windows distros, which has been fixed. Still investigating why build-and-run-authentication-tests-openssl-1.1.0
is failing.

kevinAlbs

Great work.

I fully support the decision to moving the mock server tests to a separate task to reduce build time on the debug-compile tasks.

(e.g. if the value of Evergreen expansions such as ${atlas_free} are considered sensitive).

The URIs for Atlas dev instances for driver testing are considered sensitive. Evergreen is transitioning to be entirely behind authentication (See e-mail with subject "Update: Evergreen Auth Policy"). I expect silent=True is not needed regardless. run auth tests does not appear to log the URIs. It seems OK to remove.

kevinAlbs · 2023-01-24T17:41:51Z

.evergreen/build-and-test-with-toolchain.sh

+  declare new_path
+  new_path="$(readlink -f "/opt/mongo-c-toolchain/${ssl_ver}/bin")"
+  new_path+=":${addl_path}"
+  new_path+=":${old_path}"


Suggested change

new_path+=":${old_path}"

old_path is included in addl_path

kevinAlbs · 2023-01-24T18:04:47Z

.evergreen/compile-unix.sh

+fi
+
+if [[ -n "${ZSTD}" ]]; then
+  # Since zstd inconsitently installed on macos-1014.


Suggested change

# Since zstd inconsitently installed on macos-1014.

# Since zstd is inconsistently installed on macos-1014.

kevinAlbs · 2023-01-24T18:30:06Z

.evergreen/install-ssl.sh

+  tar zxf fips.tar.gz
+  pushd openssl-fips-2.0.16
+  (
+    set -x xtrace


Suggested change

set -x xtrace

set -o xtrace

eramongodb · 2023-01-24T20:02:30Z

Deferring further diagnosis of build-and-run-authentication-tests-openssl-1.1.0 task failure to CDRIVER-4562 to unblock work.

eramongodb · 2023-01-24T20:26:35Z

Latest changes verified by this patch.

Will merge PR to unblock further work once the patch has completed and there are no further feedback/objections.

rcsanchez97 · 2023-01-24T23:26:36Z

if command -v ldd >/dev/null; then
  # MacOS does not have ldd.
  LD_LIBRARY_PATH="${openssl_lib_prefix}" ldd "${ping}" | grep "libssl"
  LD_LIBRARY_PATH="${openssl_lib_prefix}" ldd "${test_gssapi}" | grep "libssl"
fi

FYI, you can use otool -L on macOS. ref.

However, the build-and-run-authentication-tests-openssl-1.1.0 task is continuing to fail with the following error when running the Atlas Free Tier test (the preceeding X.509 test appears to succeed):
TLS handshake failed: certificate verify failed (20): unable to get local issuer certificate calling hello on '<...>.net:27017'
Examining the logs, it appears to be both compiling and linking wih the custom OpenSSL library correctly, but the error seems to suggest there may still be either a mismatch in library versions or misconfiguration of the environment.

It's because the custom-built OpenSSL can't find the root certificate store, which contains the CA that ultimately signed the Atlas server certificate. By default, OpenSSL doesn't include root certificates (TTBOMK) because that is a decision that has wide security implications. The developers probably leave the decision to implementors and integrators because that is the sort of thing that an implementor or integrator would want to decide.

Try this:

diff --git a/.evergreen/run-auth-tests.sh b/.evergreen/run-auth-tests.sh
index 8e0740412..37b2c40f9 100644
--- a/.evergreen/run-auth-tests.sh
+++ b/.evergreen/run-auth-tests.sh
@@ -90,6 +90,8 @@ if command -v ldd >/dev/null; then
   LD_LIBRARY_PATH="${openssl_lib_prefix}" ldd "${test_gssapi}" | grep "libssl" || true
 fi
 
+export SSL_CERT_DIR=/etc/ssl/certs
+
 if [[ "${ssl}" != "OFF" ]]; then
   # FIXME: CDRIVER-2008
   if [[ "${OSTYPE}" != "cygwin" ]]; then

Pathc build

That's the default CA certificate directory on Arch, Debian, and Ubuntu. I'm not sure if it is the same or different on other platforms, but it wouldn't be difficult to add some logic of we are running tasks with custom builds of OpenSSL on other platforms.

rcsanchez97

LGTM, so I am approving. Note that I made a couple of minor comments/suggestions in the review, plus in the main PR conversation I provided a possible solution to the Atlas certificate issue. I will leave it up to you to decide whether to merge as is and defer my suggestions to later, or whether you want to make the tweaks before merging.

rcsanchez97 · 2023-01-25T01:28:41Z

.evergreen/compile-unix.sh

+if [[ "${COMPILE_LIBMONGOCRYPT}" == "ON" ]]; then
+  # Build libmongocrypt, using the previously fetched installed source.
+  # TODO(CDRIVER-4394) update to use libmongocrypt 1.7.0 once there is a stable 1.7.0 release.
+  git clone --depth=1 https://github.com/mongodb/libmongocrypt --branch 1.7.0-alpha1


The 1.7.0 release has been made, so you can probably update this now.

libmongocrypt 1.7.0 requires cmake 3.12 (see mongodb/libmongocrypt#522 (comment)). The C driver builds with older versions of cmake in some distros. I plan to change the libmongocrypt build in CDRIVER-4394 to use libmongocrypt's compile.sh script to obtain the required version of cmake. I suggest leaving 1.7.0-alpha1 in this PR.

Ah, I missed that particular detail.

rcsanchez97 · 2023-01-25T01:35:42Z

.evergreen/compile-unix.sh

+  # Hosts may have libmongocrypt installed from apt/yum. We do not want to pick those up
+  # since those libmongocrypt packages statically link libbson.
+  # Note: may be overwritten by ${EXTRA_CONFIGURE_FLAGS}.


Suggested change

# Hosts may have libmongocrypt installed from apt/yum. We do not want to pick those up

# since those libmongocrypt packages statically link libbson.

# Note: may be overwritten by ${EXTRA_CONFIGURE_FLAGS}.

# Avoid symbol collisions with libmongocrypt installed via apt/yum.

# Note: may be overwritten by ${EXTRA_CONFIGURE_FLAGS}.

If libmongocrypt is installed from apt/yum (from official packages), then the libmongocrypt library will be dynamically linked to the libbson shared library which is a proper dependency:

root@build01:~# ldd /usr/lib/x86_64-linux-gnu/libmongocrypt.so.0.0.0 |grep bson libbson-1.0.so.0 => /lib/x86_64-linux-gnu/libbson-1.0.so.0 (0x00007fdc8b9ef000)

In the case of libmongocrypt installed from our PPA, then it is correct to say that the libbson linkage is static (because we don't ship a separate libbson package in the libmongocrypt PPA).

rcsanchez97 · 2023-01-25T01:40:59Z

.evergreen/compile-windows.sh

+if [ "${COMPILE_LIBMONGOCRYPT}" = "ON" ]; then
+  # Build libmongocrypt, using the previously fetched installed source.
+  # TODO(CDRIVER-4394) update to use libmongocrypt 1.7.0 once there is a stable 1.7.0 release.
+  git clone --depth=1 https://github.com/mongodb/libmongocrypt --branch 1.7.0-alpha1


The 1.7.0 release has been made, so you can probably update this now.

Will refrain given CMake 3.12 dependency mentioned in the other comment.

.evergreen/compile-unix.sh

eramongodb · 2023-01-25T15:41:05Z

It's because the custom-built OpenSSL can't find the root certificate store, which contains the CA that ultimately signed the Atlas server certificate.

Why does this only appear to be an issue for OpenSSL 1.1.0 and not with any other custom-installed OpenSSL version being tested? My understanding was that the copying of tls-ca-bundle.pem into ${openssl_install_dir} and/or ${openssl_install_dir}/ssl was supposed to accomplish certificate "registration".

rcsanchez97 · 2023-01-25T15:58:05Z

It's because the custom-built OpenSSL can't find the root certificate store, which contains the CA that ultimately signed the Atlas server certificate.

Why does this only appear to be an issue for OpenSSL 1.1.0 and not with any other custom-installed OpenSSL version being tested? My understanding was that the copying of tls-ca-bundle.pem into ${openssl_install_dir} and/or ${openssl_install_dir}/ssl was supposed to accomplish certificate "registration".

That is something I don't understand at first glance. It seems like it should work (and I had that thought when reviewing the PR), but it seems that for some reason it isn't working. Figuring it out would likely involve some deep digging. I have some vague ideas, but nothing that I can dive into at the moment given the other things in my queue.

eramongodb added 14 commits January 20, 2023 13:19

Reorder maintainer flags to permit manual disabling of warnings

70f8ac0

Improve behavior of bypass_dlclose()

8976ccd

Add support for add_expansions_to_env to shell_exec

68ac566

Format .codecov.yml

a60e6b7

Format scripts

0d3a655

Sanitize abi-compliance-check.sh

caf03a4

Sanitize add-build-dirs-to-paths.sh

fb21728

Sanitize build-and-test-with-toolchain.sh

8dc2a52

Sanitize compile*.sh scripts

5c0e3a7

Sanitize install-ssl.sh and run*.sh scripts

807df24

CDRIVER-3562 drop OpenSSL versions less than 1.0.1

ef246cc

Add SKIP_MOCK_TESTS=ON to test variants and special variants

374ce98

Do not run tests in compile tasks

0e49426

Move all code coverage test-coverage-* tasks

112ae98

eramongodb requested review from rcsanchez97, kevinAlbs, kkloberdanz and vector-of-bool January 20, 2023 20:16

eramongodb self-assigned this Jan 20, 2023

eramongodb changed the title ~~CDRIVER-3620 Sanitize compile and test scripts~~ CDRIVER-3620 Audit compile and test scripts Jan 20, 2023

vector-of-bool reviewed Jan 20, 2023

View reviewed changes

vector-of-bool self-requested a review January 20, 2023 22:59

kkloberdanz approved these changes Jan 23, 2023

View reviewed changes

Test if value of argument is null, not argument itself

4878c42

eramongodb added 3 commits January 23, 2023 16:30

Use nproc to specify Make job count

527f8aa

Unquote variables being checked to distinguish from values

8791d28

Support linking with custom OpenSSL library without system conflicts

22ea125

vector-of-bool approved these changes Jan 24, 2023

View reviewed changes

.evergreen/env-var-utils.sh Show resolved Hide resolved

eramongodb added 4 commits January 24, 2023 10:21

Use consistent definition of configure_flags_append_if_not_null

ff13e60

Use env-var-utils.sh in run-auth-tests.sh

1eacba0

Do not error on missing libssl in ldd output

f05704a

Remove call to openssl binary in favor of ldd output

e1cd742

kevinAlbs approved these changes Jan 24, 2023

View reviewed changes

eramongodb added 4 commits January 24, 2023 13:09

Remove redundant variable old_path

f6544bf

Fix typo in zstd comment

c1bf56f

Fix set xtrace command

0872b88

Print message when copying certificate to SSL install dir

690e8b5

Do not run tests after mingw compilation either

1d9e2f9

rcsanchez97 approved these changes Jan 25, 2023

View reviewed changes

kevinAlbs reviewed Jan 25, 2023

View reviewed changes

.evergreen/compile-unix.sh Show resolved Hide resolved

eramongodb added 4 commits January 25, 2023 10:39

Update comment documenting libmongocrypt linkage behavior

afd07be

CDRIVER-4562 Add SSL_CERT_DIR to resolve cert verify failure with 1.1.0

c486e20

Fallback to otool on MacOS if available

ca232bf

Fix indentation

ec717f1

eramongodb merged commit f4a83c0 into mongodb:master Jan 25, 2023

eramongodb deleted the cdriver-scripts branch January 25, 2023 16:42

eramongodb mentioned this pull request Jul 24, 2024

CDRIVER-5509 Use Bash and shebangs whenever able #1680

Merged

	# Since zstd inconsitently installed on macos-1014.
	# Since zstd is inconsistently installed on macos-1014.

CDRIVER-3620 Audit compile and test scripts #1187

CDRIVER-3620 Audit compile and test scripts #1187

Uh oh!

Conversation

eramongodb commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Do not run tests in compile tasks

Coverage Tasks

add_expansions_to_env

OpenSSL

Authentication Tests

bypass_dlclose

Bash and Hygienic Scripts

Bugfixes Galore

OpenSSL/LibreSSL Compilation, Installation, and Selection

CFLAGS vs. Maintainer Flags

X and ENABLE_X Environment Variables

Error Handling in BAT Scripts

Uh oh!

rcsanchez97 commented Jan 20, 2023

Uh oh!

eramongodb commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vector-of-bool left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcsanchez97 commented Jan 20, 2023

Uh oh!

rcsanchez97 commented Jan 22, 2023

Uh oh!

kkloberdanz left a comment

Choose a reason for hiding this comment

Uh oh!

eramongodb commented Jan 23, 2023

Uh oh!

eramongodb commented Jan 23, 2023

Uh oh!

eramongodb commented Jan 23, 2023

Uh oh!

vector-of-bool left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eramongodb commented Jan 24, 2023

Uh oh!

kevinAlbs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eramongodb commented Jan 24, 2023

Uh oh!

eramongodb commented Jan 24, 2023

Uh oh!

rcsanchez97 commented Jan 24, 2023

Uh oh!

rcsanchez97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eramongodb commented Jan 20, 2023 •

edited

Loading

Do not run tests in `compile` tasks

`CFLAGS` vs. Maintainer Flags

`X` and `ENABLE_X` Environment Variables

eramongodb commented Jan 20, 2023 •

edited

Loading