documentation: adding new section for smdataparallel 1.1.0

Author: Talia Chopra

doc/api/training/sdp_versions/v1.0.0/smd_data_parallel_pytorch.rst

@@ -8,7 +8,7 @@ PyTorch Guide to SageMaker's distributed data parallel library
    - :ref:`pytorch-sdp-api`
 
 .. _pytorch-sdp-modify:
-	:noindex:
+   :noindex:
 
 Modify a PyTorch training script to use SageMaker data parallel
 ======================================================================
@@ -150,7 +150,7 @@ you will have for distributed training with the distributed data parallel librar
 
 
 .. _pytorch-sdp-api:
-	:noindex:
+   :noindex:
 
 PyTorch API
 ===========
@@ -161,7 +161,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.is_available()
-	:noindex:
+   :noindex:
 
    Check if script started as a distributed job. For local runs user can
    check that is_available returns False and run the training script
@@ -177,7 +177,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.init_process_group(*args, **kwargs)
-	:noindex:
+   :noindex:
 
    Initialize ``smdistributed.dataparallel``. Must be called at the
    beginning of the training script, before calling any other methods.
@@ -202,7 +202,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.is_initialized()
-	:noindex:
+   :noindex:
 
    Checks if the default process group has been initialized.
 
@@ -216,7 +216,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.get_world_size(group=smdistributed.dataparallel.torch.distributed.group.WORLD)
-	:noindex:
+   :noindex:
 
    The total number of GPUs across all the nodes in the cluster. For
    example, in a 8 node cluster with 8 GPU each, size will be equal to 64.
@@ -236,7 +236,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.get_rank(group=smdistributed.dataparallel.torch.distributed.group.WORLD)
-	:noindex:
+   :noindex:
 
    The rank of the node in the cluster. The rank ranges from 0 to number of
    nodes - 1. This is similar to MPI's World Rank.
@@ -256,7 +256,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.get_local_rank()
-	:noindex:
+   :noindex:
 
    Local rank refers to the relative rank of
    the ``smdistributed.dataparallel`` process within the node the current
@@ -275,7 +275,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.all_reduce(tensor, op=smdistributed.dataparallel.torch.distributed.ReduceOp.SUM, group=smdistributed.dataparallel.torch.distributed.group.WORLD, async_op=False)
-	:noindex:
+   :noindex:
 
    Performs an all-reduce operation on a tensor (torch.tensor) across
    all ``smdistributed.dataparallel`` workers
@@ -320,7 +320,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.broadcast(tensor, src=0, group=smdistributed.dataparallel.torch.distributed.group.WORLD, async_op=False)
-	:noindex:
+   :noindex:
 
    Broadcasts the tensor (torch.tensor) to the whole group.
 
@@ -345,7 +345,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.all_gather(tensor_list, tensor, group=smdistributed.dataparallel.torch.distributed.group.WORLD, async_op=False)
-	:noindex:
+   :noindex:
 
    Gathers tensors from the whole group in a list.
 
@@ -372,7 +372,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.all_to_all_single(output_t, input_t, output_split_sizes=None, input_split_sizes=None, group=group.WORLD, async_op=False)
-	:noindex:
+   :noindex:
 
    Each process scatters input tensor to all processes in a group and return gathered tensor in output.
 
@@ -397,7 +397,7 @@ PyTorch API
 
 
 .. function:: smdistributed.dataparallel.torch.distributed.barrier(group=smdistributed.dataparallel.torch.distributed.group.WORLD, async_op=False)
-	:noindex:
+   :noindex:
 
    Synchronizes all ``smdistributed.dataparallel`` processes.
 
@@ -423,7 +423,7 @@ PyTorch API
 
 
 .. class:: smdistributed.dataparallel.torch.parallel.DistributedDataParallel(module, device_ids=None, output_device=None, broadcast_buffers=True, process_group=None, bucket_cap_mb=None)
-	:noindex:
+   :noindex:
 
    ``smdistributed.dataparallel's`` implementation of distributed data
    parallelism for PyTorch. In most cases, wrapping your PyTorch Module
@@ -517,7 +517,7 @@ PyTorch API
 
 
 .. class:: smdistributed.dataparallel.torch.distributed.ReduceOp
-	:noindex:
+   :noindex:
 
    An enum-like class for supported reduction operations
    in ``smdistributed.dataparallel``.

doc/api/training/sdp_versions/v1.0.0/smd_data_parallel_tensorflow.rst

@@ -8,7 +8,7 @@ TensorFlow Guide to SageMaker's distributed data parallel library
    - :ref:`tensorflow-sdp-api`
 
 .. _tensorflow-sdp-modify:
-	:noindex:
+   :noindex:
 
 Modify a TensorFlow 2.x training script to use SageMaker data parallel
 ======================================================================
@@ -151,7 +151,7 @@ script you will have for distributed training with the library.
 
 
 .. _tensorflow-sdp-api:
-	:noindex:
+   :noindex:
 
 TensorFlow API
 ==============
@@ -162,7 +162,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.init()
-	:noindex:
+   :noindex:
 
    Initialize ``smdistributed.dataparallel``. Must be called at the
    beginning of the training script.
@@ -186,7 +186,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.size()
-	:noindex:
+   :noindex:
 
    The total number of GPUs across all the nodes in the cluster. For
    example, in a 8 node cluster with 8 GPUs each, ``size`` will be equal
@@ -204,7 +204,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.local_size()
-	:noindex:
+   :noindex:
 
    The total number of GPUs on a node. For example, on a node with 8
    GPUs, ``local_size`` will be equal to 8.
@@ -219,7 +219,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.rank()
-	:noindex:
+   :noindex:
 
    The rank of the node in the cluster. The rank ranges from 0 to number of
    nodes - 1. This is similar to MPI's World Rank.
@@ -234,7 +234,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.local_rank()
-	:noindex:
+   :noindex:
 
    Local rank refers to the relative rank of the
    GPUs’ ``smdistributed.dataparallel`` processes within the node. For
@@ -253,7 +253,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.allreduce(tensor, param_index, num_params, compression=Compression.none, op=ReduceOp.AVERAGE)
-	:noindex:
+   :noindex:
 
    Performs an all-reduce operation on a tensor (``tf.Tensor``).
 
@@ -281,7 +281,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.broadcast_global_variables(root_rank)
-	:noindex:
+   :noindex:
 
    Broadcasts all global variables from root rank to all other processes.
 
@@ -296,7 +296,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.broadcast_variables(variables, root_rank)
-	:noindex:
+   :noindex:
 
    Applicable for TensorFlow 2.x only.
    ​
@@ -319,7 +319,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.oob_allreduce(tensor, compression=Compression.none, op=ReduceOp.AVERAGE)
-	:noindex:
+   :noindex:
 
    OutOfBand (oob) AllReduce is simplified AllReduce function for use cases
    such as calculating total loss across all the GPUs in the training.
@@ -353,7 +353,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.overlap(tensor)
-	:noindex:
+   :noindex:
 
    This function is applicable only for models compiled with XLA. Use this
    function to enable ``smdistributed.dataparallel`` to efficiently
@@ -391,7 +391,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.broadcast(tensor, root_rank)
-	:noindex:
+   :noindex:
 
    Broadcasts the input tensor on root rank to the same input tensor on all
    other ``smdistributed.dataparallel`` processes.
@@ -412,7 +412,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.shutdown()
-	:noindex:
+   :noindex:
 
    Shuts down ``smdistributed.dataparallel``. Optional to call at the end
    of the training script.
@@ -427,7 +427,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.DistributedOptimizer
-	:noindex:
+   :noindex:
 
    Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3.1).
    ​
@@ -468,7 +468,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.DistributedGradientTape
-	:noindex:
+   :noindex:
 
    Applicable to TensorFlow 2.x only.
 
@@ -504,7 +504,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.BroadcastGlobalVariablesHook
-	:noindex:
+   :noindex:
 
    Applicable if you use the ``tf.estimator`` API in TensorFlow 2.x (2.3.1).
 
@@ -533,7 +533,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.Compression
-	:noindex:
+   :noindex:
 
    Optional Gradient Compression algorithm that can be used in AllReduce
    operation.
@@ -545,7 +545,7 @@ TensorFlow API
 
 
 .. function:: smdistributed.dataparallel.tensorflow.ReduceOp
-	:noindex:
+   :noindex:
 
    Supported reduction operations in ``smdistributed.dataparallel``.