(DOCSP-21605) Split-Horizon for multi-clusters (#908)

* (DOCSP-21605) Split-Horizon for multi-clusters

* Build issues

* Build issues

* Build issues

* Addressed initial tech review comments from Mircea, ready for a copy review now

* Ready for a copy review

* Address copy review, with the exception of examples

* Build

* Build

* Fixing a step link

* Fixing a step link

* Finished applying copy review comments, added examples from tech review

* Edits, ready for a final copy review

* Edits, ready for a final copy review

* Address tech review from Raj

* Address one more comment from Raj

* Edit, ready to merge

Author: JuliaMongo

source/includes/prereqs/pem-file-domain-name.rst

@@ -2,8 +2,7 @@ Each certificate should include a valid Domain Name.
 
 For each replica set or sharded cluster member, the Common Name, also
 known as the Domain Name, for that member's certificate must match
-the |fqdn| of the POD on which this cluster member
-is deployed.
+the |fqdn| of the pod this cluster member is deployed on.
 
 The |fqdn| name in each certificate has the following syntax:
 ``pod-name.service-name.namespace.svc.cluster.local``. This name is

source/includes/steps-connect-from-inside-k8s.yaml

@@ -38,4 +38,5 @@ content: |
         .. code-block:: none
             :copyable: false
 
-            mongo --host shardedcluster-mongos-0.shardedcluster-svc.mongodb.svc.cluster.local --port 27017
+            mongosh --host shardedcluster-mongos-0.shardedcluster-svc.mongodb.svc.cluster.local \
+              --port 27017

source/includes/steps-enable-external-sharded.yaml

@@ -119,7 +119,8 @@ content: |
      .. code-block:: sh
         :copyable: false
 
-        mongo --host ec2-54-212-23-143.us-west-2.compute.amazonaws.com --port 30078
+        mongosh --host ec2-54-212-23-143.us-west-2.compute.amazonaws.com \
+          --port 30078
 
   .. include:: /includes/admonitions/fact-get-external-dns.rst
 

source/includes/steps-enable-external-standalone.yaml

@@ -120,7 +120,8 @@ content: |
      .. code-block:: sh
         :copyable: false
 
-        mongo --host ec2-54-212-23-143.us-west-2.compute.amazonaws.com --port 30994
+        mongosh --host ec2-54-212-23-143.us-west-2.compute.amazonaws.com \
+          --port 30994
 
   .. include:: /includes/admonitions/fact-get-external-dns.rst
 

source/includes/steps-enable-split-horizon-openshift.yaml

@@ -25,9 +25,9 @@ content: |
          
         If the `spec.selector` has entries that target headless 
         services or applications, OpenShift may create a software 
-        firewall rule explicitly dropping connectivity.  Review the 
+        firewall rule explicitly dropping connectivity. Review the 
         selectors carefully and consider targeting the stateful set pod 
-        members directly as seen in the example.  Routes in OpenShift 
+        members directly as seen in the example. Routes in OpenShift 
         offer port 80 or port 443. This example service uses 
         port 443.
 
@@ -154,7 +154,7 @@ replacement:
   k8sResourceType: replica-set
 
 ---
-title: Test the connection to the replica set.
+title: "Test the connection to the replica set."
 level: 4
 stepnum: 11
 ref: k8s-ext-rs-test-conn-openshift
@@ -175,24 +175,37 @@ content: |
 
   .. note::
      
-     In the following example, use your replica set names and replace ``{redacted}`` with the domain that you manage.
+     In the following example, for each member of the replica set, use
+     your replica set names and replace ``{redacted}`` with the domain
+     that you manage.
 
   .. code-block:: sh
 
-     mongo --host my-external/my-external-0.{redacted}:443,my-external-1.{redacted}:443,my-external-2.{redacted}:443 \
-           --tls \
+     mongosh --host my-external/my-external-0.{redacted} \
+           --port 443
+           --ssl \
            --tlsAllowInvalidCertificates
 
   .. warning::
 
      Don't use the ``--tlsAllowInvalidCertificates`` flag in production.
-     In production, share the |k8s| |certauth| files with client tools
-     or applications.
+     
+  In production, for each host in a replica set, specify the |tls|
+  certificate and the |certauth| to securely connect to client tools or
+  applications:
+
+  .. code-block:: sh
+
+     mongosh --host my-external/my-external-0.{redacted} \
+       --port 443 \
+       --tls \
+       --tlsCertificateKeyFile server.pem \
+       --tlsCAFile ca-pem
 
   If the connection succeeds, you should see:
 
   .. code-block:: javascript
 
-     MongoDB Enterprise <my-replica-set>:PRIMARY
+     Enterprise <my-replica-set> [primary]
 
 ...    

source/includes/steps-enable-split-horizon.yaml

@@ -10,7 +10,7 @@ content: |
 
   You must enable |tls| for the replica set by providing a value for 
   the :setting:`spec.security.certsSecretPrefix` setting. The replica 
-  set must use a custom certificate stored with 
+  set must use a custom |certauth| certificate stored with 
   :setting:`spec.security.tls.ca`.
 
 ---
@@ -117,8 +117,8 @@ content: |
   :setting:`spec.connectivity.replicaSetHorizons` setting are correct.
 
   External hostnames should match the |dns| names of |k8s| worker nodes.
-  These can be *any* nodes in the |k8s| cluster. |k8s-nodes| do internal
-  routing if the pod is run on another node.
+  These can be *any* nodes in the |k8s| cluster. |k8s| nodes use internal
+  routing if the pod runs on another node.
 
   Set the ports in :setting:`spec.connectivity.replicaSetHorizons` to
   the NodePort values that you discovered.
@@ -152,29 +152,13 @@ source:
 replacement:
   k8sResource: :term:`replica set`
   k8sResourceType: replica-set
+
 ---
-title: Test the connection to the replica set.
 level: 4
 stepnum: 12
-ref: k8s-ext-rs-test-conn
-content: |
-
-  .. code-block:: sh
-
-     mongo --host <my-replica-set>/web1.example.com:30907,web2.example.com:32350,web3.example.com:31185 \
-           --ssl \
-           --sslAllowInvalidCertificates
-
-  .. warning::
-
-     Don't use the ``--sslAllowInvalidCertificates`` flag in production.
-     In production, share the |k8s| |certauth| files with client tools
-     or applications.
-
-  If the connection succeeds, you should see:
-
-  .. code-block:: javascript
-
-     MongoDB Enterprise <my-replica-set>:PRIMARY
+ref: k8s-ext-rs-test-conn-horizon
+inherit:
+  file: steps-source-deploy-k8s-resource.yaml
+  ref: k8s-ext-rs-test-conn
 
 ...

source/includes/steps-multi-cluster-enable-split-horizon.yaml

@@ -0,0 +1,165 @@
+---
+title: ":ref:`Deploy a Multi-Cluster Replica Set <multi-cluster-quick-start-ref>`."
+stepnum: 1
+level: 4
+ref: pre-deploy-replicaset
+content: |
+
+---
+title: ":ref:`Secure the Multi-Cluster with TLS <multi-cluster-secure-tls>`."
+level: 4
+stepnum: 2
+ref: enable-tls--replicaset
+content: |
+
+  Provide values for:
+
+  - The |tls| secret in :setting:`spec.security.certsSecretPrefix`.
+  - The custom |certauth| certificate in :setting:`spec.security.tls.ca`.
+
+---
+title: "Add Subject Alternate Names to your |tls| certificates."
+level: 4
+stepnum: 3
+ref: remove-tls-existing-replicasets
+content: |
+
+  Add each external |dns| name to the certificate |san-dns|.
+
+---
+title: "Create a ``NodePort`` service for each of the Pods in different clusters."
+level: 4
+stepnum: 4
+ref: k8s-ext-rs-create-nodeports
+content: |
+
+  When you create a ``NodePort`` service with ``kubectl``, it assigns a
+  random port in the range from 30000 to 32767, inclusive.
+
+  1. Create a NodePort service.
+
+     - To create a NodePort service that uses a randomly assigned port, run
+       the following command on each Pod in each cluster:
+
+       .. code-block:: sh
+
+          kubectl expose pod/<my-replica-set>-0 --type="NodePort" --port 27017
+
+     - To create a NodePort service that uses a deterministic port, on each
+       Pod in each cluster, create a ``Nodeport`` service definition YAML
+       file similar to the following example. Specify the port you want
+       to use in the :setting:`spec.ports.NodePort` setting. This example
+       configures a NodePort service on port 30007.
+
+       .. code-block:: yaml
+
+          apiVersion: v1
+          kind: Service
+          metadata:
+            name: <my-replica-set>-0
+            labels:
+              controller: mongodb-enterprise-operator
+          spec:
+            type: NodePort
+            selector:
+              controller: mongodb-enterprise-operator
+            ports:
+              port: 27017
+              targetPort: 27017
+              nodePort: 30007
+
+
+       Apply the YAML with ``kubectl apply -f <nodeport-conf>.yaml``.
+
+---
+title: "Verify the NodePort services."
+level: 4
+stepnum: 5
+ref: k8s-ext-rs-discover-nodeports
+content: |
+
+  In each cluster, run this command to verify the NodePort services that
+  you created:
+
+  .. code-block:: sh
+
+     $ kubectl get svc <node_port_service_name>
+
+  The command returns results similar to the following example:
+
+  .. code-block:: sh
+
+     NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
+     <node_port_service_name>   NodePort    10.102.27.116   <none>        27017:30007/TCP   8m30s
+
+---
+title: "Update your replica set resource |yaml| file."
+level: 4
+stepnum: 6
+ref: update-replset-resource-mc-split-horizon
+content: |
+
+  Set the hostnames and ports in :setting:`spec.connectivity.replicaSetHorizons`
+  to the NodePort values that you created in the previous step.
+
+  Confirm that you specified the correct external hostnames. External
+  hostnames should match the |dns| names of |k8s| worker nodes.
+  These can be *any* nodes in the |k8s| cluster. If the Pod runs on another
+  node, |k8s| nodes use internal routing.
+
+  .. code-block:: sh
+
+     apiVersion: mongodb.com/v1
+      kind: MongoDBMulti
+      metadata:
+       name: multi-cluster-replica-set
+       namespace: mongodb
+      spec:
+       clusterSpecList:
+        clusterSpecs:
+        - clusterName: e2e.cluster1.mongokubernetes.com
+          members: 1
+        - clusterName: e2e.cluster2.mongokubernetes.com
+          members: 1
+        - clusterName: e2e.cluster3.mongokubernetes.com
+          members: 1
+       connectivity:
+        replicaSetHorizons:
+        - sample-horizon: web1.example.com:30907
+        - sample-horizon: web2.example.com:30907
+        - sample-horizon: web3.example.com:30907
+       credentials: my-credentials
+       duplicateServiceObjects: false
+       opsManager:
+        configMapRef:
+         name: my-project
+       persistent: true
+       security:
+        certsSecretPrefix: clustercert
+        tls:
+          ca: issuer-ca
+       type: ReplicaSet
+       version: 4.4.0-ent"
+
+---
+title: "Apply the updated replica set file."
+level: 4
+stepnum: 7
+ref: apply-crd
+content: |
+
+  In each cluster, run this command to apply the updated replica set file:
+
+  .. code-block:: sh
+
+     $ Kubectl apply -f <file_name.yaml>
+
+---
+level: 4
+stepnum: 8
+ref: k8s-ext-rs-test-conn-horizon-mc
+inherit:
+  file: steps-source-deploy-k8s-resource.yaml
+  ref: k8s-ext-rs-test-conn
+
+...

source/includes/steps-source-deploy-k8s-resource.yaml

@@ -739,4 +739,43 @@ content: |
 
      kubectl create configmap custom-ca --from-file=ca-pem
 
+---
+title: "Test the connection to the replica set."
+stepnum: 0
+level: 4
+ref: k8s-ext-rs-test-conn
+content: |
+
+  In the development environment, for each host in a replica set, run
+  the following command:
+
+  .. code-block:: sh
+
+     mongosh --host <my-replica-set>/web1.example.com \
+           --port 30907
+           --ssl \
+           --sslAllowInvalidCertificates
+
+  .. note::
+
+     Don't use the ``--sslAllowInvalidCertificates`` flag in production.
+
+  In production, for each host in a replica set, specify the |tls|
+  certificate and the |certauth| to securely connect to client tools or
+  applications:
+
+  .. code-block:: sh
+
+     mongosh --host <my-replica-set>/web1.example.com \
+       --port 30907 \
+       --tls \
+       --tlsCertificateKeyFile server.pem \
+       --tlsCAFile ca-pem
+
+  If the connection succeeds, you should see:
+
+  .. code-block:: javascript
+
+     Enterprise <my-replica-set> [primary]
+
 ...

source/multi-cluster-arch.txt

@@ -27,11 +27,17 @@ clusters are not available in the beta release of the |multi-clusters|:
 - Sharded cluster deployments
 - LDAP authentication
 - |onprem| version earlier than 5.0.7
-- Split Horizon
+
+.. _multi-cluster-capabilities:
 
 Multi-Cluster Capabilities
 --------------------------
 
+This section describes the multi-cluster capabilities that you can
+configure using the same procedures as for single clusters deployed with
+the |k8s-op-short|. Other multi-cluster capabilities have their own
+documentation in this guide.
+
 .. list-table::
    :header-rows: 1
    :widths: 30 70