Skip to content

Commit 113899a

Browse files
JuliaMongojwilliams-mongo
JuliaMongo
authored and
jwilliams-mongo
committed
(DOCSP-23970) (DOCSP-25503) Disaster recovery in multi-Kubernetes-cluster deployments (#1142)
* First batch, to be continued * Minimal reorg to create space for disaster recovery updates * Edits * Edits * copy * Edits * edits * Edits * Edits * Edits * Added a Disaster Recovery topic, have questions and need more info * Edits * Fixed build errors, ready for the initial tech review of the first half.To be continued. * Adding the failover info, multi-cluster CLI options, and renaming the tool to CLI * Fix build warnings and errors * small edit * edits * Fix one build warning, ready for a review * tech review, to be continued * First round of tech review -- have qs for Mircea * Tech review from Mircea, added setup and recover subcommands, added info about identifying failed clusters with healthz pings, other edits * Build warnings * fix one last build warning, ready for a review * Edits from Corry, also added a link to Argo CD example * Fixing links * Trying to fix the tables * trying to fix the table includes * Adding includes for the common params in tables * Fixing the tables with includes * Almost there? tables * Adjusting table column width * Edits * Fixing metadata tags syntax * Fixing keywords * Fixing keywords * Fixing keywords
1 parent 84cb588 commit 113899a

24 files changed

+712
-258
lines changed

‎conf.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@
172172
'.. |k8s-statefulsets| replace:: `StatefulSets <https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/>`__',
173173
'.. |k8s-statefulset| replace:: `StatefulSet <https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/>`__',
174174
'.. |k8s-webhook| replace:: `Webhook <https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks>`__',
175+
'.. |k8s-healthchecks| replace:: `Kubernetes API health endpoints <https://kubernetes.io/docs/reference/using-api/health-checks/>`__',
175176
'.. |k8s| replace:: Kubernetes',
176177
'.. |kdc| replace:: :abbr:`KDC (Key Distribution Center)`',
177178
'.. |kind| replace:: :abbr:`Kind (Kubernetes in Docker)`',
@@ -188,6 +189,9 @@
188189
'.. |multi-cluster| replace:: multi-Kubernetes-cluster deployment',
189190
'.. |Multi-cluster| replace:: Multi-Kubernetes-cluster deployment',
190191
'.. |multi-clusters| replace:: multi-Kubernetes-cluster deployments',
192+
'.. |Multi-clusters| replace:: Multi-Kubernetes-cluster deployments',
193+
'.. |mc-cli| replace:: multi-cluster CLI',
194+
'.. |Mc-cli| replace:: Multi-cluster CLI',
191195
'.. |oc| replace:: :xml:`<mono><link target="https://docs.openshift.com/container-platform/3.11/cli_reference/index.html">oc</link></mono>`',
192196
'.. |onprem| replace:: Ops Manager',
193197
'.. |onprem-link| replace:: :opsmgr:`Ops Manager </>`',
@@ -256,6 +260,7 @@
256260
'tldrl' : ('https://www.tldrlegal.com/l/%s', ''),
257261
'aws': ('http://docs.aws.amazon.com%s', ''),
258262
'aws-blogs': ('http://aws.amazon.com%s', ''),
263+
'argo-cd': ('https://argoproj.github.io/%s', ''),
259264
'gcp': ('https://cloud.google.com%s', ''),
260265
'q-mdb': ('https://quay.io/mongodb%s', ''),
261266
'gatekeeper': ('https://open-policy-agent.github.io/gatekeeper/website/docs%s', ''),

‎source/includes/facts/fact-multi-cluster-cli-actions-setup.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- Creates a single ``mongodb`` namespace in the central cluster and
2+
each member cluster.
3+
- Creates Service Accounts, Roles, and RoleBindings in the central
4+
cluster and each member cluster.
5+
- Applies the correct permissions for service accounts.
6+
- Uses these settings to create your |multi-cluster|.

‎source/includes/list-tables/multi-cluster-cli-central-cluster-ns.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Required. Namespace that the |k8s-op-short|
2+
will be deployed to, such as:
3+
``-central-cluster-namespace="mongodb"``.

‎source/includes/list-tables/multi-cluster-cli-central-cluster.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Central cluster that the |k8s-op-short|
2+
will be deployed in, such as:
3+
``-central-cluster="MDB_CENTRAL_CLUSTER_FULL_NAME"``.
4+

‎source/includes/list-tables/multi-cluster-cli-cleanup.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Optional. Flag that indicates whether to
2+
delete all previously created resources
3+
except for namespaces.
4+
Default value is ``false``.

‎source/includes/list-tables/multi-cluster-cli-cluster-scoped.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Optional. Flag that indicates whether to
2+
create ClusterRole and ClusterRoleBindings
3+
for member clusters. Default value is ``false``.

‎source/includes/steps-multi-cluster-source.yaml

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,24 +24,23 @@ content: |
2424
stepnum: 0
2525
level: 4
2626
ref: run-multi-cluster-tool
27-
title: "Run the ``multi-cluster kubeconfig creator`` tool."
27+
title: "Run the ``multi-cluster CLI``."
2828
content: |
2929
3030
By default, the |k8s-op-short| is scoped to the ``mongodb`` namespace.
3131
The following command creates one central cluster, three member clusters,
3232
and a |k8s-ns| labeled ``mongodb`` in each of the clusters.
3333
3434
a. Change to the directory to which you cloned the |k8s-op-short|
35-
repository, and then to the directory that has the ``multi-cluster kubeconfig creator``
36-
tool.
37-
#. Run the :github:`multi-cluster kubeconfig creator </mongodb/mongodb-enterprise-kubernetes/blob/master/tools/multicluster/main.go>`
38-
tool:
35+
repository, and then to the directory that contains the
36+
:ref:`Multi-cluster CLI <multi-cluster-cli-reference-ref>`.
37+
#. Run the |mc-cli|:
3938
4039
.. code-block:: sh
4140
4241
cd tools/multicluster
4342
go run main.go \
44-
setup
43+
setup \
4544
-central-cluster="${MDB_CENTRAL_CLUSTER_FULL_NAME}" \
4645
-member-clusters="${MDB_CLUSTER_1_FULL_NAME},${MDB_CLUSTER_2_FULL_NAME},${MDB_CLUSTER_3_FULL_NAME}" \
4746
-member-cluster-namespace="mongodb" \
@@ -95,7 +94,7 @@ content: |
9594
9695
If you have not done so already, run the following commands to run
9796
all ``kubectl`` commands on the central cluster in the default
98-
namespace. In the following steps, you will install the |k8s-op-short|
97+
namespace. In the following steps, you install the |k8s-op-short|
9998
into this namespace.
10099
101100
.. code-block:: sh
@@ -126,7 +125,8 @@ content: |
126125
--version <mongodb-kubernetes-operator-version> \
127126
--set operator.name=mongodb-enterprise-operator-multi-cluster \
128127
--set operator.createOperatorServiceAccount=false \
129-
--set "multiCluster.clusters=$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME"
128+
--set "multiCluster.clusters=$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME" \
129+
--set multiCluster.performFailover=false
130130
---
131131
stepnum: 0
132132
title: "Deploy the MongoDB resource."

‎source/index.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ optimal performance.
5454
Install the Operator </installation>
5555
Deploy Ops Manager Resources </om-resources>
5656
Deploy MongoDB Database Resources </mdb-resources>
57-
Deploy MongoDB Resources across Multiple Kubernetes Clusters (Beta) </multi-cluster>
57+
Deploy MongoDB Resources on Multiple Kubernetes Clusters (Beta) </multi-cluster>
5858
/tutorial/modify-resource-image
5959
/reference
6060
Release Notes </release-notes>

‎source/multi-cluster-arch.txt

Lines changed: 23 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
.. _multi-cluster-arch-ref:
22

3-
===========================
4-
Multi-Cluster Architecture
5-
===========================
3+
===========================================
4+
Architecture, Capabilities, and Limitations
5+
===========================================
66

77
.. default-domain:: mongodb
88

@@ -23,23 +23,20 @@ Features Not Available in the Beta Release
2323

2424
.. important::
2525

26-
This feature is a beta release. Use |multi-cluster|
27-
deployments only in development environments.
26+
This feature is a beta release and has the following limiations:
2827

29-
The following features of the |k8s-op-full| and the underlying |k8s|
30-
clusters are not available in the beta release of the |multi-clusters|:
31-
32-
- Sharded cluster deployments
33-
- |onprem| version earlier than 5.0.7
28+
- Deploy |multi-clusters| only in development environments.
29+
- Deploy only replica sets. Sharded cluster deployments aren't supported.
30+
- - Use |onprem| versions later than 5.0.7.
3431

3532
.. _multi-cluster-capabilities:
3633

37-
Multi-Cluster Capabilities
38-
--------------------------
34+
Multi-Kubernetes-Cluster Deployment Capabilities
35+
------------------------------------------------
3936

40-
This section describes the multi-cluster capabilities that you can
37+
This section describes the |multi-cluster| capabilities that you can
4138
configure using the same procedures as for single clusters deployed with
42-
the |k8s-op-short|. Other multi-cluster capabilities have their own
39+
the |k8s-op-short|. Other |multi-cluster| capabilities have their own
4340
documentation in this guide.
4441

4542
.. list-table::
@@ -68,8 +65,8 @@ documentation in this guide.
6865
These procedures are the same as for single clusters
6966
deployed with |k8s-op-short|, with the following exceptions:
7067

71-
- The procedures apply to replica sets only. Multi-cluster
72-
deployments :ref:`do not support creating sharded clusters <multi-cluster-beta-limitations>`.
68+
- The procedures apply to replica sets only. |Multi-clusters|
69+
:ref:`do not support creating sharded clusters <multi-cluster-beta-limitations>`.
7370
- In the ``mongodbResourceRef``, specify the name of the multi-cluster
7471
replica set: ``name: "<my-multi-cluster-replica-set>"``.
7572

@@ -81,16 +78,18 @@ documentation in this guide.
8178

8279
.. _multi-cluster-diagram:
8380

84-
Multi-Cluster Deployment Architecture
85-
-------------------------------------
81+
Deployment Architecture and Diagram
82+
-----------------------------------
8683

87-
The |k8s-op-full| runs in a central |k8s| cluster.
84+
The |k8s-op-full| runs in a central |k8s| cluster of your |multi-cluster|.
8885

8986
The central cluster holds the ``MongoDBMulti`` CustomResource spec for
9087
the MongoDB replica set.
9188
The member |k8s| clusters host the MongoDB replica sets.
9289

93-
If you deploy |onprem| with the |k8s-op-short|, the central cluster may also host |onprem|.
90+
If you deploy |onprem| with the |k8s-op-short|, the central cluster may
91+
also host |onprem|.
92+
9493
|istio| manages the discovery of MongoDB nodes deployed in different
9594
|k8s| member clusters.
9695

@@ -114,17 +113,17 @@ The |k8s-op-full| performs these actions:
114113
confirm that the |multi-cluster| is in the desired state.
115114

116115
- Reconciles resources. Creates the necessary resources, such as
117-
Configmaps, Secrets, Service objects, and Statefulset objects in
118-
each member cluster corresponding to the number of replica set members
119-
in the MongoDB cluster.
116+
Configmaps, Secrets, Service objects, and Statefulset objects in each
117+
member cluster corresponding to the number of replica set members in the
118+
MongoDB cluster.
120119

121120
.. include:: /includes/facts/fact-can-change-secret-storage-tool.rst
122121

123122
The following diagram shows the high-level architecture of a |multi-cluster|
124123
across regions and availability zones:
125124

126125
.. figure:: /images/multi-cluster-arch.svg
127-
:alt: Diagram showing the high-level architecture of a multi-cluster
126+
:alt: Diagram showing the high-level architecture of a multi-cluster-Kubernetes
128127
deployment across regions and availability zones using the
129128
MongoDB Enterprise Kubernetes Operator
130129
:figwidth: 600px

‎source/multi-cluster-central-member-clusters.txt

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
.. _central-and-member-clusters:
2+
3+
===================================
4+
Central Cluster and Member Clusters
5+
===================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. meta::
10+
:keywords: multicluster, central cluster, member clusters, multi-cluster
11+
12+
.. contents:: On this page
13+
:local:
14+
:backlinks: none
15+
:depth: 2
16+
:class: singlecol
17+
18+
MongoDB recommends that you identify one cluster to act as a **central cluster**.
19+
The central cluster hosts the |k8s-op-short| and acts as the control plane
20+
for the multi-cluster deployment. This central cluster can also host
21+
replica set members. This documentation refers to other |k8s| clusters
22+
that host replica set members as **member clusters**.
23+
24+
Communication between replica set members occurs over a service mesh,
25+
which means that your database doesn't rely on the central cluster to function.
26+
Note that if the central cluster fails, you can't use the |k8s-op-short|
27+
to change your deployment until access to this cluster is restored or
28+
until you redeploy the |k8s-op-short| to an available |k8s| cluster.
29+
30+
You can host your application on any |k8s| cluster in the service mesh.
31+
Your application can be co-located on a member cluster with one of the
32+
replica set nodes that you deployed using the |k8s-op-short|, or you can
33+
host your application on a cluster that doesn't host replica set nodes
34+
or the |k8s-op-short|.
35+
36+
To learn more, see the :ref:`multi-cluster-diagram`.

‎source/multi-cluster-cli-reference.txt

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
.. _multi-cluster-cli-reference-ref:
2+
3+
===========================
4+
Multi-Cluster CLI Reference
5+
===========================
6+
7+
.. default-domain:: mongodb
8+
9+
.. meta::
10+
:keywords: multicluster, multi-kubernetes-cluster MongoDB, mongoDBmulti resource, multi-cluster cli setup, multi-cluster cli recover, central cluster, member clusters
11+
:description: Learn how to use the multi-cluster CLI to set up a multi-Kubernetes-cluster MongoDB deployment and automatically recover such a deployment from some cluster failures.
12+
13+
.. contents:: On this page
14+
:local:
15+
:backlinks: none
16+
:depth: 1
17+
:class: singlecol
18+
19+
Use the :github:`multi-cluster CLI
20+
</mongodb/mongodb-enterprise-kubernetes/blob/master/tools/multicluster/>`
21+
to set up |multi-clusters| and run automatic and manual
22+
:ref:`disaster recovery <disaster-recovery-ref>`.
23+
24+
The |mc-cli| has the following subcommands:
25+
26+
- :ref:`multi-cluster-cli-setup`
27+
- :ref:`multi-cluster-cli-recover`
28+
29+
.. _multi-cluster-cli-setup:
30+
31+
The setup Subcommand
32+
--------------------
33+
34+
The ``go run main.go setup`` subcommand sets up the initial |multi-cluster|.
35+
It performs the following actions:
36+
37+
.. include:: /includes/facts/fact-multi-cluster-cli-actions-setup.rst
38+
39+
.. _multi-cluster-cli-setup-options:
40+
41+
The setup Subcommand Options
42+
----------------------------
43+
44+
The ``setup`` subcommand of the |mc-cli| has the following options:
45+
46+
.. list-table::
47+
:header-rows: 1
48+
:widths: 40 20 40
49+
50+
* - Option
51+
- Data Type
52+
- Description
53+
54+
* - ``central-cluster``
55+
- string
56+
- .. include:: /includes/list-tables/multi-cluster-cli-central-cluster.rst
57+
58+
* - ``central-cluster-namespace``
59+
- string
60+
- .. include:: /includes/list-tables/multi-cluster-cli-central-cluster-ns.rst
61+
62+
* - ``cleanup``
63+
- boolean
64+
- .. include:: /includes/list-tables/multi-cluster-cli-cleanup.rst
65+
66+
* - ``cluster-scoped``
67+
- boolean
68+
- .. include:: /includes/list-tables/multi-cluster-cli-cluster-scoped.rst
69+
70+
* - ``install-database-roles``
71+
- boolean
72+
- Optional. Flag that indicates whether to install
73+
the ServiceAccounts and Roles required for running
74+
MongoDB workloads on the member clusters.
75+
Default value is ``false``.
76+
77+
* - ``member-clusters``
78+
- string
79+
- Required. Comma-separated list that contains
80+
member clusters, such as:
81+
``-member-clusters="${MDB_CLUSTER_2_FULL_NAME},
82+
${MDB_CLUSTER_3_FULL_NAME},
83+
${MDB_CLUSTER_4_FULL_NAME}"``.
84+
85+
* - ``member-cluster-namespace``
86+
- string
87+
- Required. Namespace that the member cluster resources
88+
will be deployed to, such as:
89+
``-member-cluster-namespace="mongodb"``.
90+
91+
* - ``service-account``
92+
- string
93+
- Optional. Name of the service account for the
94+
|k8s-op-short| to use to communicate with the
95+
member clusters. Default value is
96+
``mongodb-enterprise-operator-multi-cluster``.
97+
98+
For a full example of the |mc-cli| ``setup`` subcommand's usage,
99+
see the :ref:`multi-Kubernetes-cluster quick start procedure
100+
<multi-cluster-quick-start-procedure>`.
101+
102+
.. _multi-cluster-cli-recover:
103+
104+
The recover Subcommand
105+
----------------------
106+
107+
The ``go run main.go recover`` subcommand can automatically recover a
108+
failed cluster topology in some cases. In other cases, you must
109+
:ref:`manually recover from a failure <disaster-recovery-manual>`. To
110+
learn more, see :ref:`disaster-recovery-ref`.
111+
112+
.. _multi-cluster-cli-recover-options:
113+
114+
The recover Subcommand Options
115+
------------------------------
116+
117+
The ``recover`` subcommand of the |mc-cli| has the following options:
118+
119+
.. list-table::
120+
:header-rows: 1
121+
:widths: 40 20 40
122+
123+
* - Option
124+
- Data Type
125+
- Description
126+
127+
* - ``central-cluster``
128+
- string
129+
- .. include:: /includes/list-tables/multi-cluster-cli-central-cluster.rst
130+
131+
* - ``central-cluster-namespace``
132+
- string
133+
- .. include:: /includes/list-tables/multi-cluster-cli-central-cluster-ns.rst
134+
135+
* - ``cleanup``
136+
- boolean
137+
- .. include:: /includes/list-tables/multi-cluster-cli-cleanup.rst
138+
139+
* - ``cluster-scoped``
140+
- boolean
141+
- .. include:: /includes/list-tables/multi-cluster-cli-cluster-scoped.rst
142+
143+
For a full example of the |mc-cli| ``recover`` subcommand's usage,
144+
see the :ref:`manual disaster recovery procedure <disaster-recovery-manual>`.

0 commit comments

Comments
 (0)