DRIVERS-1710: Convert retryable reads spec tests to unified test format (#1536)

* Add retryable-reads to unified test format Makefile

Regarding GridFS tests: Drivers may differ in how they query the chunks collection. For instance, PHPLIB queries the chunk's n field with $gte in addition to files_id. Rather than use $$matchAsRoot (schema version 1.13+) to emulate the legacy test behavior of ignoring extra fields, the commandStartedEvent assertion was relaxed and expectResult was introduced to assert that the file is read successfully.

* Remove legacy test files and docs for legacy test format

This also renumbers the prose tests

Author: jmikola

source/retryable-reads/tests/README.rst

@@ -9,192 +9,18 @@ Retryable Reads Tests
 Introduction
 ============
 
-The YAML and JSON files in the ``legacy`` and ``unified`` sub-directories are platform-independent tests
-that drivers can use to prove their conformance to the Retryable Reads spec. Tests in the
-``unified`` directory are written using the `Unified Test Format <../../unified-test-format/unified-test-format.md>`_.
-Tests in the ``legacy`` directory are written using the format described below.
+The YAML and JSON files in this directory are platform-independent tests meant
+to exercise a driver's implementation of retryable reads. These tests utilize
+the [Unified Test Format](../../unified-test-format/unified-test-format.md).
 
-Prose tests, which are not easily expressed in YAML, are also presented
+Several prose tests, which are not easily expressed in YAML, are also presented
 in this file. Those tests will need to be manually implemented by each driver.
 
-Tests will require a MongoClient created with options defined in the tests.
-Integration tests will require a running MongoDB cluster with server versions
-4.0 or later.
+Prose Tests
+===========
 
-N.B. The spec specifies 3.6 as the minimum server version: however,
-``failCommand`` is not supported on 3.6, so for now, testing requires MongoDB
-4.0. Once `DRIVERS-560`_ is resolved, we will attempt to adapt its live failure
-integration tests to test Retryable Reads on MongoDB 3.6.
-
-.. _DRIVERS-560: https://jira.mongodb.org/browse/DRIVERS-560
-
-Server Fail Point
-=================
-
-See: `Server Fail Point`_ in the Transactions spec test suite.
-
-.. _Server Fail Point: ../../transactions/tests#server-fail-point
-
-Disabling Fail Point after Test Execution
------------------------------------------
-
-After each test that configures a fail point, drivers should disable the
-``failCommand`` fail point to avoid spurious failures in
-subsequent tests. The fail point may be disabled like so::
-
-    db.runCommand({
-        configureFailPoint: "failCommand",
-        mode: "off"
-    });
-
-Network Error Tests
-===================
-
-Network error tests are expressed in YAML and should be run against a standalone,
-shard cluster, or single-node replica set.
-
-
-Test Format
------------
-
-Each YAML file has the following keys:
-
-- ``runOn`` (optional): An array of server version and/or topology requirements
-  for which the tests can be run. If the test environment satisfies one or more
-  of these requirements, the tests may be executed; otherwise, this file should
-  be skipped. If this field is omitted, the tests can be assumed to have no
-  particular requirements and should be executed. Each element will have some or
-  all of the following fields:
-
-  - ``minServerVersion`` (optional): The minimum server version (inclusive)
-    required to successfully run the tests. If this field is omitted, it should
-    be assumed that there is no lower bound on the required server version.
-
-  - ``maxServerVersion`` (optional): The maximum server version (inclusive)
-    against which the tests can be run successfully. If this field is omitted,
-    it should be assumed that there is no upper bound on the required server
-    version.
-
-  - ``topology`` (optional): An array of server topologies against which the
-    tests can be run successfully. Valid topologies are "single",
-    "replicaset", "sharded", and "load-balanced". If this field is omitted,
-    the default is all topologies (i.e. ``["single", "replicaset", "sharded",
-    "load-balanced"]``).
-
-  - ``serverless``: (optional): Whether or not the test should be run on Atlas
-    Serverless instances. Valid values are "require", "forbid", and "allow". If
-    "require", the test MUST only be run on Atlas Serverless instances. If
-    "forbid", the test MUST NOT be run on Atlas Serverless instances. If omitted
-    or "allow", this option has no effect.
-
-    The test runner MUST be informed whether or not Atlas Serverless is being
-    used in order to determine if this requirement is met (e.g. through an
-    environment variable or configuration option).
-
-    Note: the Atlas Serverless proxy imitates mongos, so the test runner is not
-    capable of determining if Atlas Serverless is in use by issuing commands
-    such as ``buildInfo`` or ``hello``. Furthermore, connections to Atlas
-    Serverless use a load balancer, so the topology will appear as
-    "load-balanced".
-
-- ``database_name`` and ``collection_name``: Optional. The database and
-  collection to use for testing.
-
-- ``bucket_name``: Optional. The GridFS bucket name to use for testing.
-
-- ``data``: The data that should exist in the collection(s) under test before
-  each test run. This will typically be an array of documents to be inserted
-  into the collection under test (i.e. ``collection_name``); however, this field
-  may also be an object mapping collection names to arrays of documents to be
-  inserted into the specified collection.
-
-- ``tests``: An array of tests that are to be run independently of each other.
-  Each test will have some or all of the following fields:
-
-  - ``description``: The name of the test.
-
-  - ``clientOptions``: Optional, parameters to pass to MongoClient().
-
-  - ``useMultipleMongoses`` (optional): If ``true``, and the topology type is
-    ``Sharded``, the MongoClient for this test should be initialized with multiple
-    mongos seed addresses. If ``false`` or omitted, only a single mongos address
-    should be specified.
-
-    If ``true``, the topology type is ``LoadBalanced``, and Atlas Serverless is
-    not being used, the MongoClient for this test should be initialized with the
-    URI of the load balancer fronting multiple servers. If ``false`` or omitted,
-    the MongoClient for this test should be initialized with the URI of the load
-    balancer fronting a single server.
-
-    ``useMultipleMongoses`` only affects ``Sharded`` and ``LoadBalanced``
-    topologies (excluding Atlas Serverless).
-
-  - ``skipReason``: Optional, string describing why this test should be skipped.
-
-  - ``failPoint``: Optional, a server fail point to enable, expressed as the
-    configureFailPoint command to run on the admin database.
-
-  - ``operations``: An array of documents describing an operation to be
-    executed. Each document has the following fields:
-
-    - ``name``: The name of the operation on ``object``.
-
-    - ``object``: The name of the object to perform the operation on. Can be
-      "database", "collection", "client", or "gridfsbucket."
-
-    - ``arguments``: Optional, the names and values of arguments.
-
-    - ``result``: Optional. The return value from the operation, if any. This
-      field may be a scalar (e.g. in the case of a count), a single document, or
-      an array of documents in the case of a multi-document read.
-
-    - ``error``: Optional. If ``true``, the test should expect an error or
-      exception.
-
-  - ``expectations``: Optional list of command-started events.
-
-GridFS Tests
-------------
-
-GridFS tests are denoted by when the YAML file contains ``bucket_name``.
-The ``data`` field will also be an object, which maps collection names
-(e.g. ``fs.files``) to an array of documents that should be inserted into
-the specified collection.
-
-``fs.files`` and ``fs.chunks`` should be created in the database
-specified by ``database_name``. This could be done via inserts or by
-creating GridFSBuckets—using the GridFS ``bucketName`` (see
-`GridFSBucket spec`_) specified by ``bucket_name`` field in the YAML
-file—and calling ``upload_from_stream_with_id`` with the appropriate
-data.
-
-``Download`` tests should be tested against ``GridFS.download_to_stream``.
-``DownloadByName`` tests should be tested against
-``GridFS.download_to_stream_by_name``.
-
-
-.. _GridFSBucket spec: https://github.com/mongodb/specifications/blob/master../../gridfs/gridfs-spec.md#configurable-gridfsbucket-class
-
-
-Speeding Up Tests
------------------
-
-Drivers can greatly reduce the execution time of tests by setting `heartbeatFrequencyMS`_
-and `minHeartbeatFrequencyMS`_ (internally) to a small value (e.g. 5ms), below what
-is normally permitted in the SDAM spec. If a test specifies an explicit value for
-heartbeatFrequencyMS (e.g. client or URI options), drivers MUST use that value.
-
-.. _minHeartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#minheartbeatfrequencyms
-.. _heartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms
-
-Optional Enumeration Commands
-=============================
-
-A driver only needs to test the optional enumeration commands it has chosen to
-implement (e.g. ``Database.listCollectionNames()``).
-
-PoolClearedError Retryability Test
-==================================
+1. PoolClearedError Retryability Test
+-------------------------------------
 
 This test will be used to ensure drivers properly retry after encountering PoolClearedErrors.
 It MUST be implemented by any driver that implements the CMAP specification.
@@ -232,8 +58,8 @@ This test requires MongoDB 4.2.9+ for ``blockConnection`` support in the failpoi
 
 9. Disable the failpoint.
 
-Retrying Reads in a Sharded Cluster
-===================================
+2. Retrying Reads in a Sharded Cluster
+--------------------------------------
 
 These tests will be used to ensure drivers properly retry reads on a different
 mongos.
@@ -244,8 +70,8 @@ different mongos due to normal SDAM behavior of randomized suitable server
 selection". Verify relevant code paths are correctly executed by the tests using
 external means such as a logging, debugger, code coverage tool, etc.
 
-Retryable Reads Are Retried on a Different mongos When One is Available
------------------------------------------------------------------------
+2.1 Retryable Reads Are Retried on a Different mongos When One is Available
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This test MUST be executed against a sharded cluster that has at least two
 mongos instances, supports ``retryReads=true``, and has enabled the
@@ -278,8 +104,8 @@ mongos instances, supports ``retryReads=true``, and has enabled the
 7. Disable the fail point on both ``s0`` and ``s1``.
 
 
-Retryable Reads Are Retried on the Same mongos When No Others are Available
----------------------------------------------------------------------------
+2.2 Retryable Reads Are Retried on the Same mongos When No Others are Available
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This test MUST be executed against a sharded cluster that supports
 ``retryReads=true`` and has enabled the ``configureFailPoint`` command 
@@ -316,10 +142,12 @@ This test MUST be executed against a sharded cluster that supports
 Changelog
 =========
 
+:2024-03-06: Convert legacy retryable reads tests to unified format.
+
 :2024-02-21: Update mongos redirection prose tests to workaround SDAM behavior
              preventing execution of deprioritization code paths.
 
-:2023-08-26 Add prose tests for retrying in a sharded cluster.
+:2023-08-26: Add prose tests for retrying in a sharded cluster.
 
 :2022-04-22: Clarifications to ``serverless`` and ``useMultipleMongoses``.