PYTHON-2395 Consider connection pool health during server selection

[ShaneHarvey]

An implementation of mongodb/specifications#876

TODO:

• Add test that proves slower servers get fewer operations because of this feature.
• Figure out how to update operation_count after a fork()

[ShaneHarvey]

This is ready for review.

[ShaneHarvey]

Test failures:

• https://evergreen.mongodb.com/task/mongo_python_driver_tests_python_version_rhel62_test_ssl__platform~rhel62_auth~auth_ssl~ssl_python_version~3.6_coverage~coverage_test_4.4_sharded_cluster_patch_d1fd3f7e982f5d1ae3e8b8b8edeba7e146bb02f8_5fbc574cd6d80a2a9efee7e7_20_11_24_00_43_57

 [2020/11/24 03:15:31.596] FAIL [3.289s]: test_load_balancing (test_server_selection_in_window.TestProse)
 [2020/11/24 03:15:31.596] ----------------------------------------------------------------------
 [2020/11/24 03:15:31.596] Traceback (most recent call last):
 [2020/11/24 03:15:31.596]   File "/data/mci/2f407c6b5c7693a603b431e5d6e9bc66/src/test/__init__.py", line 444, in wrap
 [2020/11/24 03:15:31.596]     return f(*args, **kwargs)
 [2020/11/24 03:15:31.596]   File "/data/mci/2f407c6b5c7693a603b431e5d6e9bc66/src/test/__init__.py", line 444, in wrap
 [2020/11/24 03:15:31.596]     return f(*args, **kwargs)
 [2020/11/24 03:15:31.596]   File "/data/mci/2f407c6b5c7693a603b431e5d6e9bc66/src/test/test_server_selection_in_window.py", line 153, in test_load_balancing
 [2020/11/24 03:15:31.596]     self.assertAlmostEqual(freqs[delayed_server], 0.50, delta=0.1)
 [2020/11/24 03:15:31.596] AssertionError: 0.63 != 0.5 within 0.1 delta
 [2020/11/24 03:15:31.629] ----------------------------------------------------------------------

I suppose the tolerance here is too low. Let's increase the delta to 0.15.

https://evergreen.mongodb.com/task/mongo_python_driver_tests_python_version_rhel62_test_ssl__platform~rhel62_auth~auth_ssl~ssl_python_version~pypy3.6_coverage~coverage_test_4.4_replica_set_patch_d1fd3f7e982f5d1ae3e8b8b8edeba7e146bb02f8_5fbc574cd6d80a2a9efee7e7_20_11_24_00_43_57

[2020/11/24 03:14:27.915] root      7650  1.2  0.3 2287872 29072 ?       Sl   03:05   0:06 /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/venv/bin/python3 /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/venv/bin/mongo-orchestration -e default -f /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/orchestration.config --socket-timeout-ms=60000 --bind=127.0.0.1 --enable-majority-read-concern start
[2020/11/24 03:14:27.915] root      8033 11.4 14.0 3356352 1076296 ?     SLl  03:05   0:57 /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/mongodb/bin/mongod --config /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/db/mongo-g8n12d3h --port 27017
[2020/11/24 03:14:28.403] root      8132 12.1 14.3 3266236 1101132 ?     SLl  03:06   1:01 /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/mongodb/bin/mongod --config /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/db/mongo-4vbgcwz2 --port 27018
[2020/11/24 03:14:28.403] root      8278 10.5 11.0 3045604 850520 ?      SLl  03:06   0:51 /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/mongodb/bin/mongod --config /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/drivers-tools/.evergreen/orchestration/db/mongo-9lru297c --port 27019
[2020/11/24 03:14:28.403] root      8452  0.0  0.0   9232   204 ?        S    03:06   0:00 sh
[2020/11/24 03:14:28.403] root      8453  0.0  0.0   9236   244 ?        S    03:06   0:00 sh /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/src/.evergreen/run-tests.sh
[2020/11/24 03:14:28.403] root      8464 49.4 55.6 6955984 4266532 ?     Sl   03:06   3:51 /opt/python/pypy3.6/bin/pypy3 setup.py test --xunit-output=/data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/src/xunit-results
[2020/11/24 03:14:28.403] root     22992  1.0  0.0  13376   752 ?        R    03:14   0:00 ps aux
[2020/11/24 03:15:30.298]   test_gridfs (test_session.TestSession) ... /data/mci/ed7e311d29371e01b48f6f47e6d6b0a4/src/.evergreen/run-tests.sh: line 233:  8464 Killed                  $COVERAGE_OR_PYTHON $EXTRA_ARGS $COVERAGE_ARGS setup.py $C_EXTENSIONS test $OUTPUT
[2020/11/24 03:15:30.307] 10
[2020/11/24 03:15:30.310] Command failed: command encountered problem: error waiting on process 'ee34ca15-acb9-4514-8731-496537dd91f7': exit status 137
[2020/11/24 03:15:30.311] Finished running task commands in 10m12.810387749s.
[2020/11/24 03:15:30.312] Task failed: command failed: command encountered problem: error waiting on process 'ee34ca15-acb9-4514-8731-496537dd91f7': exit status 137
[2020/11/24 03:15:30.313] Task completed - FAILURE.

Not sure what happened here. The test process was killed but there wasn't a EVG timeout. Perhaps the tests used too much memory (OOM killer)?