DRIVERS-1519: srvMaxHosts option for limiting SRV seeding (#1069)

* DRIVERS-1519: srvMaxHosts option for limiting SRV seeding

Since this commit adds a "MongoClient Configuration" section to the SRV spec, documentation for the srvServiceName option (introduced in c48d7f4) was also added.

This commit also revises existing sections of the SRV spec test README.

* Specify randomization algorithm and require the same for polling

* Require mongodb+srv scheme to use srvMaxHosts

* Update dates for tentative changelog entries

* Do not expect canonical host names for sharded tests

Seeds do not get replaced for non-canonical host names in sharded topologies. That is only done for replica set topologies.

* Positive srvMaxHosts conflicts with loadBalanced=true and replicaSet

Adds URI Validation section to SRV spec. Notes conflict in Load Balancer spec's URI Validation section. Revises sections in URI Options spec (including directConnection validation).

Adds relevant spec tests to SRV and URI Options specs.

* Use correct SRV record for testing srvMaxHosts with loadBalanced=true

The test4 record is intentionally missing and should not have been used in these tests.

The test3 record resolves to a single host, which is suitable for testing loadBalanced=true in the connection string.

Co-authored-by: Benjamin Rewis <[email protected]>
Co-authored-by: Kevin Albertson <[email protected]>

Author: 3 people

source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst

@@ -10,8 +10,8 @@ Initial DNS Seedlist Discovery
 :Authors: Derick Rethans
 :Status: Draft
 :Type: Standards
-:Last Modified: 2021-09-15
-:Version: 1.5.0
+:Last Modified: 2021-10-14
+:Version: 1.6.0
 :Spec Lead: Matt Broadstone
 :Advisory Group: \A. Jesse Jiryu Davis
 :Approver(s): Bernie Hackett, David Golden, Jeff Yemin, Matt Broadstone, A. Jesse Jiryu Davis
@@ -48,6 +48,9 @@ interpreted as described in `RFC 2119 <https://www.ietf.org/rfc/rfc2119.txt>`_.
 Specification
 =============
 
+Connection String Format
+------------------------
+
 The connection string parser in the driver is extended with a new protocol
 ``mongodb+srv`` as a logical pre-processing step before it considers the
 connection string and SDAM specifications. In this protocol, the comma
@@ -61,35 +64,59 @@ format is::
 specification following the ``Host Information``. This includes the ``Auth
 database`` and ``Connection Options``.
 
-Seedlist Discovery
-------------------
 
-In this preprocessing step, the driver will query the DNS server for SRV
-records on ``{hostname}.{domainname}``, prefixed with the SRV service name
-and protocol. The SRV service name is provided in the ``srvServiceName`` URI option and
-defaults to ``mongodb``. The protocol is always ``tcp``. After prefixing, the URI
-should look like: ``_{srvServiceName}._tcp.{hostname}.{domainname}``. This DNS query
-is expected to respond with one or more SRV records. The driver MUST add all the host
-names and port numbers that were returned as part of the DNS SRV query result to the
-seedlist.
+MongoClient Configuration
+-------------------------
 
-The priority and weight fields in returned SRV records MUST be ignored.
+srvMaxHosts
+~~~~~~~~~~~
 
-If ``mongodb+srv`` is used, a driver MUST implicitly also enable TLS. Clients
-can turn this off by passing ``ssl=false`` in either the Connection String,
-or options passed in as parameters in code to the MongoClient constructor (or
-equivalent API for each driver), but not through a TXT record (discussed in
-the next section).
+This option is used to limit the number of mongos connections that may be
+created for sharded topologies. This option limits the number of SRV records
+used to populate the seedlist during initial discovery, as well as the number of
+additional hosts that may be added during
+`SRV polling <../polling-srv-records-for-mongos-discovery/polling-srv-records-for-mongos-discovery.rst>`_.
+This option requires a non-negative integer and defaults to zero (i.e. no
+limit). This option MUST only be configurable at the level of a ``MongoClient``.
 
-A driver MUST verify that in addition to the ``{hostname}``, the
-``{domainname}`` consists of at least two parts: the domain name, and a TLD.
-Drivers MUST raise an error and MUST NOT contact the DNS server to obtain SRV
-(or TXT records) if the full URI does not consists of at least three parts.
 
-A driver MUST verify that the host names returned through SRV records have the
-same parent ``{domainname}``. Drivers MUST raise an error and MUST NOT
-initiate a connection to any returned host name which does not share the same
-``{domainname}``.
+srvServiceName
+~~~~~~~~~~~~~~
+
+This option specifies a valid SRV service name according to
+`RFC 6335 <https://datatracker.ietf.org/doc/html/rfc6335#section-5.1>`_, with
+the exception that it may exceed 15 characters as long as the 63rd (62nd with
+prepended underscore) character DNS query limit is not surpassed. This option
+requires a string value and defaults to "mongodb". This option MUST only be
+configurable at the level of a ``MongoClient``.
+
+
+URI Validation
+~~~~~~~~~~~~~~
+
+The driver MUST report an error if either the ``srvServiceName`` or
+``srvMaxHosts`` URI options are specified with a non-SRV URI (i.e. scheme other
+than ``mongodb+srv``). The driver MUST allow specifying the ``srvServiceName``
+and ``srvMaxHosts`` URI options with an SRV URI (i.e. ``mongodb+srv`` scheme).
+
+If ``srvMaxHosts`` is a positive integer, the driver MUST throw an error in the
+following cases:
+
+- The connection string contains a ``replicaSet`` option.
+- The connection string contains a ``loadBalanced`` option with a value of
+  ``true``.
+
+When validating URI options, the driver MUST first do the SRV and TXT lookup and
+then perform the validation. For drivers that do SRV lookup asynchronously this
+may result in a ``MongoClient`` being instantiated but erroring later during
+operation execution.
+
+
+Seedlist Discovery
+------------------
+
+Validation Before Querying DNS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 It is an error to specify a port in a connection string with the
 ``mongodb+srv`` protocol, and the driver MUST raise a parse error and MUST NOT
@@ -99,14 +126,54 @@ It is an error to specify more than one host name in a connection string with
 the ``mongodb+srv`` protocol, and the driver MUST raise a parse error and MUST
 NOT do DNS resolution or contact hosts.
 
-The driver MUST NOT attempt to connect to any hosts until the DNS query has
-returned its results.
+A driver MUST verify that in addition to the ``{hostname}``, the
+``{domainname}`` consists of at least two parts: the domain name, and a TLD.
+Drivers MUST raise an error and MUST NOT contact the DNS server to obtain SRV
+(or TXT records) if the full URI does not consist of at least three parts.
+
+If ``mongodb+srv`` is used, a driver MUST implicitly also enable TLS. Clients
+can turn this off by passing ``tls=false`` in either the Connection String,
+or options passed in as parameters in code to the MongoClient constructor (or
+equivalent API for each driver), but not through a TXT record (discussed in a
+later section).
+
+
+Querying DNS
+~~~~~~~~~~~~
+
+In this preprocessing step, the driver will query the DNS server for SRV records
+on ``{hostname}.{domainname}``, prefixed with the SRV service name and protocol.
+The SRV service name is provided in the ``srvServiceName`` URI option and
+defaults to ``mongodb``. The protocol is always ``tcp``. After prefixing, the
+URI should look like: ``_{srvServiceName}._tcp.{hostname}.{domainname}``. This
+DNS query is expected to respond with one or more SRV records.
+
+The priority and weight fields in returned SRV records MUST be ignored.
 
 If the DNS result returns no SRV records, or no records at all, or a DNS error
 happens, an error MUST be raised indicating that the URI could not be used to
 find hostnames. The error SHALL include the reason why they could not be
 found.
 
+A driver MUST verify that the host names returned through SRV records have the
+same parent ``{domainname}``. Drivers MUST raise an error and MUST NOT
+initiate a connection to any returned host name which does not share the same
+``{domainname}``.
+
+The driver MUST NOT attempt to connect to any hosts until the DNS query has
+returned its results.
+
+If ``srvMaxHosts`` is zero or greater than or equal to the number of hosts in
+the DNS result, the driver MUST populate the seedlist with all hosts.
+
+If ``srvMaxHosts`` is greater than zero and less than the number of hosts in the
+DNS result, the driver MUST randomly select that many hosts and use them to
+populate the seedlist. Drivers SHOULD use the `Fisher-Yates shuffle`_ for
+randomization.
+
+.. _`Fisher-Yates shuffle`: https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#The_modern_algorithm
+
+
 Default Connection String Options
 ---------------------------------
 
@@ -276,6 +343,11 @@ SRV records.
 ChangeLog
 =========
 
+2021-10-14 - 1.6.0
+    Add ``srvMaxHosts`` MongoClient option and restructure Seedlist Discovery
+    section. Improve documentation for the ``srvServiceName`` MongoClient
+    option and add a new URI Validation section.
+
 2021-09-15 - 1.5.0
     Clarify that service name only defaults to ``mongodb``, and should be
     defined by the ``srvServiceName`` URI option.

source/initial-dns-seedlist-discovery/tests/README.rst

@@ -10,11 +10,20 @@ Test Setup
 
 The tests in the ``replica-set`` directory MUST be executed against a
 three-node replica set on localhost ports 27017, 27018, and 27019 with
-replica set name ``repl0``. The tests in ``load-balanced`` MUST be executed
-against a load-balanced sharded cluster with the mongos servers running on
-localhost ports 27017 and 27018 and load balancers, shard servers, and config
-servers running on any open ports. In both cases, the clusters MUST be
-started with SSL enabled.
+replica set name ``repl0``.
+
+The tests in the ``load-balanced`` directory MUST be executed against a
+load-balanced sharded cluster with the mongos servers running on localhost ports
+27017 and 27018 (corresponding to the script in `drivers-evergreen-tools`_). The
+load balancers, shard servers, and config servers may run on any open ports.
+
+.. _`drivers-evergreen-tools`: https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-load-balancer.sh
+
+The tests in the ``sharded`` directory MUST be executed against a sharded
+cluster with the mongos servers running on localhost ports 27017 and 27018.
+Shard servers and config servers may run on any open ports.
+
+In all cases, the clusters MUST be started with SSL enabled.
 
 To run the tests that accompany this spec, you need to configure the SRV and
 TXT records with a real name server. The following records are required for
@@ -78,28 +87,49 @@ Test Format and Use
 
 These YAML and JSON files contain the following fields:
 
-- ``uri``: a mongodb+srv connection string
+- ``uri``: a ``mongodb+srv`` connection string
 - ``seeds``: the expected set of initial seeds discovered from the SRV record
+- ``numSeeds``: the expected number of initial seeds discovered from the SRV
+  record. This is mainly used to test ``srvMaxHosts``, since randomly selected
+  hosts cannot be deterministically asserted.
 - ``hosts``: the discovered topology's list of hosts once SDAM completes a scan
-- ``options``: the parsed connection string options as discovered from URI and
-  TXT records
-- ``parsed_options``: additional options present in the `Connection String`_
-  URI such as ``Userinfo`` (as ``user`` and ``password``), and ``Auth
-  database`` (as ``auth_database``).
+- ``numHosts``: the expected number of hosts discovered once SDAM completes a
+  scan. This is mainly used to test ``srvMaxHosts``, since randomly selected
+  hosts cannot be deterministically asserted.
+- ``options``: the parsed `URI options`_ as discovered from the
+  `Connection String`_'s "Connection Options" component and SRV resolution
+  (e.g. TXT records, implicit ``tls`` default).
+- ``parsed_options``: additional, parsed options from other `Connection String`_
+  components. This is mainly used for asserting ``UserInfo`` (as ``user`` and
+  ``password``) and ``Auth database`` (as ``auth_database``).
 - ``error``: indicates that the parsing of the URI, or the resolving or
   contents of the SRV or TXT records included errors.
 - ``comment``: a comment to indicate why a test would fail.
 
 .. _`Connection String`: ../../connection-string/connection-string-spec.rst
+.. _`URI options`: ../../uri-options/uri-options.rst
+
+For each file, create a MongoClient initialized with the ``mongodb+srv``
+connection string.
+
+If ``seeds`` is specified, drivers SHOULD verify that the set of hosts in the
+client's initial seedlist matches the list in ``seeds``. If ``numSeeds`` is
+specified, drivers SHOULD verify that the size of that set matches ``numSeeds``.
+
+If ``hosts`` is specified, drivers MUST verify that the set of
+ServerDescriptions in the client's TopologyDescription eventually matches the
+list in ``hosts``. If ``numHosts`` is specified, drivers MUST verify that the
+size of that set matches ``numHosts``.
+
+If ``options`` is specified, drivers MUST verify each of the values under
+``options`` match the MongoClient's parsed value for that option. There may be
+other options parsed by the MongoClient as well, which a test does not verify.
+
+If ``parsed_options`` is specified, drivers MUST verify that each of the values
+under ``parsed_options`` match the MongoClient's parsed value for that option.
+Supported values include, but are not limited to, ``user`` and ``password``
+(parsed from ``UserInfo``) and ``auth_database`` (parsed from
+``Auth database``).
 
-For each file, create MongoClient initialized with the mongodb+srv connection
-string. You SHOULD verify that the client's initial seed list matches the list of
-seeds. You MUST verify that the set of ServerDescriptions in the client's
-TopologyDescription eventually matches the list of hosts. You MUST verify that
-each of the values of the Connection String Options under ``options`` match the
-Client's parsed value for that option. There may be other options parsed by
-the Client as well, which a test does not verify. In ``uri-with-auth`` the URI
-contains a user/password set and additional options are provided in
-``parsed_options`` so that tests can verify authentication is maintained when
-evaluating URIs. You MUST verify that an error has been thrown if ``error`` is
-present.
+If ``error`` is specified and ``true``, drivers MUST verify that an error has
+been thrown.

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-conflicts_with_loadBalanced-true-txt.json

@@ -0,0 +1,7 @@
+{
+  "uri": "mongodb+srv://test20.test.build.10gen.cc/?srvMaxHosts=1",
+  "seeds": [],
+  "hosts": [],
+  "error": true,
+  "comment": "Should fail because positive integer for srvMaxHosts conflicts with loadBalanced=true (TXT)"
+}

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-conflicts_with_loadBalanced-true-txt.yml

@@ -0,0 +1,5 @@
+uri: "mongodb+srv://test20.test.build.10gen.cc/?srvMaxHosts=1"
+seeds: []
+hosts: []
+error: true
+comment: Should fail because positive integer for srvMaxHosts conflicts with loadBalanced=true (TXT)

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-conflicts_with_loadBalanced-true.json

@@ -0,0 +1,7 @@
+{
+  "uri": "mongodb+srv://test3.test.build.10gen.cc/?loadBalanced=true&srvMaxHosts=1",
+  "seeds": [],
+  "hosts": [],
+  "error": true,
+  "comment": "Should fail because positive integer for srvMaxHosts conflicts with loadBalanced=true"
+}

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-conflicts_with_loadBalanced-true.yml

@@ -0,0 +1,5 @@
+uri: "mongodb+srv://test3.test.build.10gen.cc/?loadBalanced=true&srvMaxHosts=1"
+seeds: []
+hosts: []
+error: true
+comment: Should fail because positive integer for srvMaxHosts conflicts with loadBalanced=true

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-zero-txt.json

@@ -0,0 +1,14 @@
+{
+  "uri": "mongodb+srv://test20.test.build.10gen.cc/?srvMaxHosts=0",
+  "seeds": [
+    "localhost.test.build.10gen.cc:27017"
+  ],
+  "hosts": [
+    "localhost.test.build.10gen.cc:27017"
+  ],
+  "options": {
+    "loadBalanced": true,
+    "srvMaxHosts": 0,
+    "ssl": true
+  }
+}

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-zero-txt.yml

@@ -0,0 +1,10 @@
+# loadBalanced=true (TXT) is permitted because srvMaxHosts is non-positive
+uri: "mongodb+srv://test20.test.build.10gen.cc/?srvMaxHosts=0"
+seeds:
+    - localhost.test.build.10gen.cc:27017
+hosts:
+    - localhost.test.build.10gen.cc:27017
+options:
+    loadBalanced: true
+    srvMaxHosts: 0
+    ssl: true

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-zero.json

@@ -0,0 +1,14 @@
+{
+  "uri": "mongodb+srv://test3.test.build.10gen.cc/?loadBalanced=true&srvMaxHosts=0",
+  "seeds": [
+    "localhost.test.build.10gen.cc:27017"
+  ],
+  "hosts": [
+    "localhost.test.build.10gen.cc:27017"
+  ],
+  "options": {
+    "loadBalanced": true,
+    "srvMaxHosts": 0,
+    "ssl": true
+  }
+}

source/initial-dns-seedlist-discovery/tests/load-balanced/srvMaxHosts-zero.yml

@@ -0,0 +1,10 @@
+# loadBalanced=true is permitted because srvMaxHosts is non-positive
+uri: "mongodb+srv://test3.test.build.10gen.cc/?loadBalanced=true&srvMaxHosts=0"
+seeds:
+    - localhost.test.build.10gen.cc:27017
+hosts:
+    - localhost.test.build.10gen.cc:27017
+options:
+    loadBalanced: true
+    srvMaxHosts: 0
+    ssl: true

source/initial-dns-seedlist-discovery/tests/replica-set/srvMaxHosts-conflicts_with_replicaSet-txt.json

@@ -0,0 +1,7 @@
+{
+  "uri": "mongodb+srv://test5.test.build.10gen.cc/?srvMaxHosts=1",
+  "seeds": [],
+  "hosts": [],
+  "error": true,
+  "comment": "Should fail because positive integer for srvMaxHosts conflicts with replicaSet option (TXT)"
+}

source/initial-dns-seedlist-discovery/tests/replica-set/srvMaxHosts-conflicts_with_replicaSet-txt.yml

@@ -0,0 +1,5 @@
+uri: "mongodb+srv://test5.test.build.10gen.cc/?srvMaxHosts=1"
+seeds: []
+hosts: []
+error: true
+comment: Should fail because positive integer for srvMaxHosts conflicts with replicaSet option (TXT)

source/initial-dns-seedlist-discovery/tests/replica-set/srvMaxHosts-conflicts_with_replicaSet.json

@@ -0,0 +1,7 @@
+{
+  "uri": "mongodb+srv://test1.test.build.10gen.cc/?replicaSet=repl0&srvMaxHosts=1",
+  "seeds": [],
+  "hosts": [],
+  "error": true,
+  "comment": "Should fail because positive integer for srvMaxHosts conflicts with replicaSet option"
+}

source/initial-dns-seedlist-discovery/tests/replica-set/srvMaxHosts-conflicts_with_replicaSet.yml

@@ -0,0 +1,5 @@
+uri: "mongodb+srv://test1.test.build.10gen.cc/?replicaSet=repl0&srvMaxHosts=1"
+seeds: []
+hosts: []
+error: true
+comment: Should fail because positive integer for srvMaxHosts conflicts with replicaSet option

source/initial-dns-seedlist-discovery/tests/replica-set/srvMaxHosts-equal_to_srv_records.json

@@ -0,0 +1,17 @@
+{
+  "uri": "mongodb+srv://test1.test.build.10gen.cc/?srvMaxHosts=2",
+  "numSeeds": 2,
+  "seeds": [
+    "localhost.test.build.10gen.cc:27017",
+    "localhost.test.build.10gen.cc:27018"
+  ],
+  "hosts": [
+    "localhost:27017",
+    "localhost:27018",
+    "localhost:27019"
+  ],
+  "options": {
+    "srvMaxHosts": 2,
+    "ssl": true
+  }
+}