documentation: add SageMaker distributed libraries documentation (#549)

Author: aaronmarkham

doc/_static/theme_overrides.css

@@ -0,0 +1,10 @@
+/* override table width restrictions */
+.wy-table-responsive table td, .wy-table-responsive table th {
+    white-space: normal;
+}
+
+.wy-table-responsive {
+    margin-bottom: 24px;
+    max-width: 100%;
+    overflow: visible;
+}

doc/api/index.rst

@@ -9,5 +9,6 @@ The SageMaker Python SDK consists of a variety classes for preparing data, train
 
     prep_data/feature_store
     training/index
+    training/distributed
     inference/index
     utility/index

doc/api/training/distributed.rst

@@ -0,0 +1,11 @@
+Distributed Training APIs
+-------------------------
+SageMaker distributed training libraries offer both data parallel and model parallel training strategies.
+They combine software and hardware technologies to improve inter-GPU and inter-node communications.
+They extend SageMaker’s training capabilities with built-in options that require only small code changes to your training scripts.
+
+.. toctree::
+   :maxdepth: 3
+
+   smd_data_parallel
+   smd_model_parallel

doc/api/training/index.rst

@@ -3,7 +3,13 @@ Training APIs
 #############
 
 .. toctree::
-   :maxdepth: 1
-   :glob:
+   :maxdepth: 4
 
-   *
+   analytics
+   automl
+   debugger
+   estimators
+   algorithm
+   tuner
+   parameter
+   processing

doc/api/training/smd_data_parallel.rst

@@ -0,0 +1,64 @@
+###################################
+Distributed data parallel
+###################################
+
+SageMaker distributed data parallel (SDP) extends SageMaker’s training
+capabilities on deep learning models with near-linear scaling efficiency,
+achieving fast time-to-train with minimal code changes.
+
+- SDP optimizes your training job for AWS network infrastructure and EC2 instance topology.
+- SDP takes advantage of gradient update to communicate between nodes with a custom AllReduce algorithm.
+
+When training a model on a large amount of data, machine learning practitioners
+will often turn to distributed training to reduce the time to train.
+In some cases, where time is of the essence,
+the business requirement is to finish training as quickly as possible or at
+least within a constrained time period.
+Then, distributed training is scaled to use a cluster of multiple nodes,
+meaning not just multiple GPUs in a computing instance, but multiple instances
+with multiple GPUs. As the cluster size increases, so does the significant drop
+in performance. This drop in performance is primarily caused the communications
+overhead between nodes in a cluster.
+
+
+.. rubric:: Customize your training script
+
+To customize your own training script, you will need the following:
+
+.. raw:: html
+
+   <div data-section-style="5" style="">
+
+-  You must provide TensorFlow / PyTorch training scripts that are
+   adapted to use SDP.
+-  Your input data must be in an S3 bucket or in FSx in the AWS region
+   that you will use to launch your training job. If you use the Jupyter
+   notebooks provided, create a SageMaker notebook instance in the same
+   region as the bucket that contains your input data. For more
+   information about storing your training data, refer to
+   the `SageMaker Python SDK data
+   inputs <https://sagemaker.readthedocs.io/en/stable/overview.html#use-file-systems-as-training-inputs>`__ documentation.
+
+.. raw:: html
+
+   </div>
+
+Use the API guides for each framework to see
+examples of training scripts that can be used to convert your training scripts.
+Then, use one of the example notebooks as your template to launch a training job.
+You’ll need to swap your training script with the one that came with the
+notebook and modify any input functions as necessary.
+Once you have launched a training job, you can monitor it using CloudWatch.
+
+Then you can see how to deploy your trained model to an endpoint by
+following one of the example notebooks for deploying a model. Finally,
+you can follow an example notebook to test inference on your deployed
+model.
+
+
+
+.. toctree::
+   :maxdepth: 2
+
+   smd_data_parallel_pytorch
+   smd_data_parallel_tensorflow