additional small typo fixes

Author: Talia Chopra

training/distributed_training/pytorch/model_parallel/bert/smp_bert_tutorial.ipynb

@@ -13,7 +13,7 @@
     "In this notebook, you will use a BERT example training script with SMP.\n",
     "The example script is based on [Nvidia Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/BERT) and requires you to download the datasets and upload them to Amazon Simple Storage Service (Amazon S3) as explained in the instructions below. This is a large dataset, and so depending on your connection speed, this process can take hours to complete. \n",
     "\n",
-    "This notebook depends on the following files:\n",
+    "This notebook depends on the following files. You can find all files in the [bert directory](https://github.com/aws/amazon-sagemaker-examples/tree/master/training/distributed_training/pytorch/model_parallel/bert) in the model parllel section of the Amazon SageMaker Examples notebooks repo.\n",
     "\n",
     "* `bert_example/sagemaker_smp_pretrain.py`: This is an entrypoint script that is passed to the Pytorch estimator in the notebook instructions. This script is responsible for end to end training of the BERT model with SMP. The script has additional comments at places where the SMP API is used.\n",
     "\n",
@@ -25,9 +25,7 @@
     "\n",
     "* `bert_example/utils.py`: This contains different helper utility functions used in end to end training of the BERT model (`bert_example/sagemaker_smp_pretrain.py`).\n",
     "\n",
-    "* `bert_example/file_utils.py`: Contains different file utility functions used in model definition (*bert_example/modeling.py*).\n",
-    "\n",
-    "*Getting Started*: The bert directory needs to be zipped and uploaded to a Sagemaker notebook instance. Unzip on the notebook instance and follow the instructions in the notebook.\n"
+    "* `bert_example/file_utils.py`: Contains different file utility functions used in model definition (`bert_example/modeling.py`).\n"
    ]
   },
   {
@@ -92,13 +90,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Prepare/Identify your Training Data in Amazon S3"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
+    "## Prepare/Identify your Training Data in Amazon S3\n",
+    "\n",
     "If you don't already have the BERT dataset in an S3 bucket, please see the instructions in [Nvidia BERT Example](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT/README.md) to download the dataset and upload it to a s3 bucket. See the prerequisites at the beginning of this notebook for more information.\n",
     "\n",
     "Uncomment and use the following cell to specify the Amazon S3 bucket and prefix that contains your training data. For example, if your training data is in s3://your-bucket/training, enter `'your-bucket'` for s3_bucket and `'training'` for prefix. Note that your output data will be stored in the same bucket, under the `output/` prefix."
@@ -193,9 +186,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "mpioptions = \"-verbose --mca orte_base_help_aggregate 0 \"\n",
-    "mpioptions += \"--mca btl_vader_single_copy_mechanism none\"\n",
-    "parameters = {\"optimize\": \"speed\", \"microbatches\": 12, \"partitions\": 2, \"ddp\": True, \"pipeline\": \"interleaved\", \"overlapping_allreduce\": True, \"placement_strategy\": \"cluster\", \"memory_weight\": 0.3}\n",
+    "mpi_options = \"-verbose --mca orte_base_help_aggregate 0 \"\n",
+    "mpi_options += \"--mca btl_vader_single_copy_mechanism none\"\n",
+    "smp_parameters = {\"optimize\": \"speed\", \"microbatches\": 12, \"partitions\": 2, \"ddp\": True, \"pipeline\": \"interleaved\", \"overlapping_allreduce\": True, \"placement_strategy\": \"cluster\", \"memory_weight\": 0.3}\n",
     "timeout = 60 * 60\n",
     "metric_definitions = [{\"Name\": \"base_metric\", \"Regex\": \"<><><><><><>\"}]\n",
     "\n",
@@ -235,7 +228,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "pytorch_estimator = PyTorch(\"sagemaker_smp_pretrain.py\",\n",
+    "pytorch_estimator = PyTorch(\"sagemaker_rbk_pretrain.py\",\n",
     "                            role=role,\n",
     "                            instance_type=\"ml.p3.16xlarge\",\n",
     "                            volume_size=200,\n",
@@ -247,13 +240,13 @@
     "                                \"smdistributed\": {\n",
     "                                    \"modelparallel\": {\n",
     "                                        \"enabled\": True,\n",
-    "                                        \"parameters\": parameters\n",
+    "                                        \"parameters\": smp_parameters\n",
     "                                    }\n",
     "                                },\n",
     "                                \"mpi\": {\n",
     "                                    \"enabled\": True,\n",
     "                                    \"process_per_host\": 8,\n",
-    "                                    \"custom_mpi_options\": mpioptions,\n",
+    "                                    \"custom_mpi_options\": mpi_options,\n",
     "                                }\n",
     "                            },\n",
     "                            source_dir='bert_example',\n",

training/distributed_training/pytorch/model_parallel/mnist/pytorch_smmodelparallel_mnist.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "SageMaker Distributed Model Parallel (SMP) is a model parallelism library for training large deep learning models that were previously difficult to train due to GPU memory limitations. SageMaker Distributed Model Parallel automatically and efficiently splits a model across multiple GPUs and instances and coordinates model training, allowing you to increase prediction accuracy by creating larger models with more parameters.\n",
     "\n",
-    "Use this notebook to configure Sagemaker Distributed Model Parallel to train a model using an example PyTorch training script, `pt_mnist.py` and [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#train-a-model-with-the-sagemaker-python-sdk). \n",
+    "Use this notebook to configure Sagemaker Distributed Model Parallel to train a model using an example PyTorch training script, `utils/pt_mnist.py` and [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#train-a-model-with-the-sagemaker-python-sdk). \n",
     "\n",
     "\n",
     "### Additional Resources\n",
@@ -17,7 +17,7 @@
     "\n",
     "* To learn more about the SageMaker model parallelism library, see [Model Parallel Distributed Training with SageMaker Distributed](http://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel.html).\n",
     "\n",
-    "* To learn more about using the SageMaker Python SDK with Pytorch, see [Using PyTorch with the SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#using-third-party-libraries).\n",
+    "* To learn more about using the SageMaker Python SDK with Pytorch, see [Using PyTorch with the SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html).\n",
     "\n",
     "* To learn more about launching a training job in Amazon SageMaker with your own training image, see [Use Your Own Training Algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html)."
    ]
@@ -166,7 +166,7 @@
     "                          \"pipeline\": \"interleaved\",\n",
     "                          \"optimize\": \"speed\",\n",
     "                          \"partitions\": 2,\n",
-    "                          \"ddp\": 1,\n",
+    "                          \"ddp\": True,\n",
     "                      }\n",
     "                  }\n",
     "              },\n",