DOCSP-2133: Clarify time required for failover/election

Author: ravindk89

source/core/replica-set-elections.txt

@@ -17,28 +17,31 @@ Replica Set Elections
    :class: singlecol
 
 :term:`Replica sets <replica set>` use elections to determine which
-set member will become :term:`primary`. Elections occur after
-initiating a replica set, and also any time the primary becomes
-unavailable. The primary is the only member in the set that can accept
-write operations. If a primary becomes unavailable, elections allow
-the set to recover normal operations without manual
-intervention.
-In the following three-member replica set, the primary is unavailable.
-One of the remaining secondaries holds an election to elect itself as a
-new primary.
+set member will become :term:`primary`. Replica sets can trigger an 
+election in response to a variety of events, such as:
+
+- Adding a new node to the replica set,
+- :method:`initiating a replica set <rs.initiate()>`,
+- performing replica set maintenance using methods such as :method:`rs.stepDown()` or :method:`rs.reconfig()`, and
+- the :term:`secondary` members losing connectivity to the primary for more than the configured :rsconf:`timeout <settings.electionTimeoutMillis>` (10 seconds by default).
+
+In the following diagram, the primary node was unavailable for longer
+than the :rsconf:`configured timeout <settings.electionTimeoutMillis>`
+and triggers the :ref:`automatic failover <replication-auto-failover>`
+process. One of the remaining secondaries calls for an election to
+select a new primary and automatically resume normal operations.
 
 .. include:: /images/replica-set-trigger-election.rst
 
-Elections are essential for independent operation of a
-replica set; however, elections take time to complete. While an
-election is in process, the replica set has no primary and cannot
-accept writes and all remaining members become read-only.
+The replica set cannot process write operations until the
+election completes successfully. The replica set can continue to serve 
+read queries if such queries are configured to 
+:ref:`run on secondaries <replica-set-read-preference>`.
+
+.. include:: /includes/fact-election-latency.rst
+
+.. include:: /includes/fact-retryable-writes-failover-election.rst
 
-If a majority of the replica set is inaccessible or unavailable to the
-current primary, the primary will step down and become a secondary. The
-replica set cannot accept writes after this occurs, but remaining
-members can continue to serve read queries if such queries are
-configured to run on secondaries.
 
 Factors and Conditions that Affect Elections
 --------------------------------------------

source/core/replica-set-high-availability.txt

@@ -15,29 +15,8 @@ Replica Set High Availability
    :depth: 1
    :class: singlecol
 
-:term:`Replica sets <replica set>` provide high availability using
-automatic :term:`failover`. Failover allows a :term:`secondary` member
-to become :term:`primary` if the current primary becomes unavailable.
+Replica sets use elections to support high availability. 
 
-.. versionchanged:: 3.2
+.. include:: /includes/toc/dfn-list-replica-set-high-availability.rst
 
-   MongoDB introduces a version 1 of the replication protocol
-   (:rsconf:`protocolVersion: 1 <protocolVersion>`) to reduce replica set
-   failover time and accelerates the detection of multiple simultaneous
-   primaries. New replica sets will, by default, use
-   :rsconf:`protocolVersion: 1 <protocolVersion>`. Previous versions of
-   MongoDB use version 0 of the protocol.
-
-Replica set members keep the same data set but are otherwise
-independent. If the primary becomes unavailable, an eligible secondary
-holds an :doc:`election </core/replica-set-elections>` to elect itself
-as a new primary. In some situations, the failover process may undertake
-a :doc:`rollback </core/replica-set-rollbacks>`. [#rollback-automatic]_
-
-.. class:: hidden
-
-   .. include:: /includes/toc/replica-set-high-availability.rst
-
-.. [#rollback-automatic] Replica sets remove "rollback" data when
-   needed without intervention. Administrators must apply or discard
-   rollback data manually.
+.. include:: /includes/toc/replica-set-high-availability.rst

source/includes/fact-election-latency.rst

@@ -0,0 +1,11 @@
+The median time before a cluster elects a new primary should not
+typically exceed 12 seconds, assuming default :rsconf:`replica
+configuration settings <settings>`. This includes time required to
+mark the primary as :ref:`unavailable <replication-auto-failover>` and
+call and complete an :ref:`election <replica-set-elections>`.
+You can tune this time period by modifying the 
+:rsconf:`settings.electionTimeoutMillis` replication configuration 
+option. Factors such as network latency may extend the time required
+for replica set elections to complete, which in turn affects the amount 
+of time your cluster may operate without a primary. These factors are 
+dependent on your particular cluster architecture.

source/includes/fact-replica-set-protocolVersion1.rst

@@ -2,7 +2,8 @@
 
    MongoDB introduces a version 1 of the replication protocol
    (:rsconf:`protocolVersion: 1 <protocolVersion>`) to reduce replica
-   set failover time and accelerates the detection of multiple
-   simultaneous primaries. New replica sets will, by default, use
+   set failover time and accelerate the detection of multiple
+   simultaneous primaries. New replica sets, by default, use
    :rsconf:`protocolVersion: 1 <protocolVersion>`. Previous versions of
-   MongoDB use version 0 of the protocol.
+   MongoDB use version 0 of the protocol. See :ref:`replication election
+   enhancements <3.2-rel-notes-rs-enhancements>` for details.

source/includes/fact-retryable-writes-failover-election.rst

@@ -0,0 +1,9 @@
+Your application connection logic should include tolerance for 
+automatic failovers and the subsequent elections.
+
+.. versionadded:: 3.6
+
+   MongoDB 3.6+ drivers can detect the loss of the primary and 
+   automatically :ref:`retry certain write operations 
+   <retryable-writes>` a single time, providing additional built-in 
+   handling of automatic failovers and elections. 

source/includes/steps-perform-maintenance-task-on-replica-set-members.yaml

@@ -90,8 +90,8 @@ action:
   language: javascript
   code: rs.stepDown(300)
 post: |
-  After the primary steps down, the replica set elects a new
-  primary. See :doc:`/core/replica-set-elections` for more
+  After the primary steps down, the replica set will elect a new
+  primary. See :ref:`replica-set-elections` for more
   information about replica set elections.
 
   Restart :binary:`~bin.mongod` as a standalone instance, making

source/reference/command/replSetGetStatus.txt

@@ -343,14 +343,14 @@ following fields:
 
       For the current primary, information regarding the election
       :ref:`Timestamp <document-bson-type-timestamp>` from the
-      operation log. See :doc:`/core/replica-set-elections` for more
+      operation log. See :doc:`/core/replica-set-high-availability` for more
       information about elections.
 
    .. data:: replSetGetStatus.members[n].electionDate
 
       For the current primary, an :term:`ISODate` formatted date string
       that reflects the election date. See
-      :doc:`/core/replica-set-elections` for more information about
+      :doc:`/core/replica-set-high-availability` for more information about
       elections.
 
    .. data:: replSetGetStatus.members[n].self

source/reference/command/replSetReconfig.txt

@@ -53,15 +53,21 @@ considerations:
 - A majority of the set's members must be operational for the
   changes to propagate properly.
 
-- This command can cause downtime as the set renegotiates
-  primary-status. Typically this is 10-20 seconds, but could
-  be as long as a minute or more. Therefore, you should attempt
-  to reconfigure only during scheduled maintenance periods.
-
-- In some cases, :dbcommand:`replSetReconfig` forces the current
-  primary to step down, initiating an election for primary among the
-  members of the replica set. When this happens, the primary node will drop
-  all current connections.
+- :dbcommand:`replSetReconfig` can trigger the current 
+  primary to step down in some situations. When the primary steps down, 
+  it forcibly closes all client connections. Primary step-down triggers 
+  an :ref:`election <replica-set-elections>` to select a new 
+  :term:`primary`. 
+  
+  .. include:: /includes/fact-election-latency.rst
+  
+  During the election process, the cluster cannot
+  accept write operations until it elects the new primary.
+
+  .. include:: /includes/fact-retryable-writes-failover-election.rst
+
+  To further reduce potential impact to a production cluster,
+  reconfigure only during scheduled maintenance periods.
 
 .. versionchanged:: 3.2
 

source/reference/glossary.txt

@@ -304,7 +304,7 @@ Glossary
    failover
       The process that allows a :term:`secondary` member of a
       :term:`replica set` to become :term:`primary` in the event of a
-      failure. See :ref:`replica-set-failover`.
+      failure. See :ref:`replication-auto-failover`.
 
    field
       A name-value pair in a :term:`document <document>`. A document has

source/reference/method/rs.reconfig.txt

@@ -46,10 +46,19 @@ Availability
 ~~~~~~~~~~~~
 
 The :method:`rs.reconfig()` shell method can trigger the current primary
-to step down in some situations. When the
-primary steps down, it forcibly closes all client connections. This is
-by design. Since it may take a period of time to elect a new primary, 
-schedule reconfiguration changes during maintenance periods to minimize loss of write availability.
+to step down in some situations. When the primary steps down, it 
+forcibly closes all client connections. Primary step-down triggers an 
+:ref:`election <replica-set-elections>` to select a new :term:`primary`.
+  
+.. include:: /includes/fact-election-latency.rst
+
+During the election process, the cluster cannot
+accept write operations until it elects the new primary.
+
+.. include:: /includes/fact-retryable-writes-failover-election.rst
+
+To further reduce potential impact to a production cluster,
+reconfigure only during scheduled maintenance periods.
 
 ``{ force: true }``
 ~~~~~~~~~~~~~~~~~~~

source/replication.txt

@@ -104,30 +104,43 @@ Automatic Failover
 ------------------
 
 When a primary does not communicate with the other members of the set
-for more than 10 seconds, an eligible secondary will hold an election
-to elect itself the new primary. The first secondary to hold an
-election and receive a majority of the members' votes becomes primary.
-
-.. include:: /includes/fact-replica-set-protocolVersion1.rst
+for more than the configured :rsconf:`electionTimeoutMillis` period 
+(10 seconds by default), an eligible secondary calls for an election
+to nominate itself as the new primary. The cluster attempts to 
+complete the election of a new primary and eesume normal operations.
 
 .. include:: /images/replica-set-trigger-election.rst
 
-Although the timing varies, the failover process generally completes
-within a minute. For instance, it may take 10-30 seconds for the
-members of a :term:`replica set` to declare a :term:`primary`
-inaccessible (see :rsconf:`~settings.electionTimeoutMillis`). One of
-the remaining secondaries holds an :term:`election` to elect itself as
-a new primary. The election itself may take another 10-30 seconds.
+The replica set cannot process write operations 
+until the election completes successfully.  The replica set can continue
+to serve read queries if such queries are configured to 
+:ref:`run on secondaries <replica-set-read-preference>` while the
+primary is offline.
+
+.. include:: /includes/fact-election-latency.rst
+
+Lowering the :rsconf:`~settings.electionTimeoutMillis` 
+replication configuration option from the default ``10000`` (10 seconds)
+can result in faster detection of primary failure. However,
+the cluster may call elections more frequently due to factors such as
+temporary network latency even if the primary is otherwise healthy.
+This can result in increased :ref:`rollbacks <replica-set-rollback>` for 
+:ref:`w : 1 <wc-w>` write operations.
+
+.. include:: /includes/fact-retryable-writes-failover-election.rst 
+
+See :ref:`replica-set-elections` for complete documentation on
+replica set elections.  
 
-.. versionchanged:: 3.2
+To learn more about MongoDB's failover process, see:
 
-   Starting in MongoDB 3.2, with the :ref:`replication election
-   enhancements <3.2-rel-notes-rs-enhancements>`, MongoDB reduces
-   replica set failover time. See :ref:`replication election
-   enhancements <3.2-rel-notes-rs-enhancements>` for details.
+- :ref:`replica-set-elections`
+- :ref:`retryable-writes`
+- :ref:`replica-set-rollback`
 
-See :ref:`replica-set-elections` and
-:ref:`replica-set-rollbacks` for more information.
+.. seealso:
+   
+   :ref:`write-concern`
 
 Read Operations
 ---------------
@@ -174,7 +187,7 @@ See :ref:`replica-set-secondary-only-members`,
     /core/replica-set-oplog
     /core/replica-set-sync
     /core/replica-set-architectures
-    /core/replica-set-high-availability
+    Replica Set High Availability </core/replica-set-high-availability>
     /applications/replication
     /administration/replica-set-deployment
     /administration/replica-set-member-configuration