Modified license year, classify to cluster, and warning on running times

Author: JonathanTaws

sagemaker-spark/pyspark_mnist/pyspark_mnist_custom_estimator.ipynb

@@ -15,7 +15,7 @@
     "7. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n",
     "\n",
     "## Introduction\n",
-    "This notebook will show how to classify handwritten digits through the SageMaker PySpark library. \n",
+    "This notebook will show how to cluster handwritten digits through the SageMaker PySpark library. \n",
     "\n",
     "We will manipulate data through Spark using a SparkSession, and then use the SageMaker Spark library to interact with SageMaker for training and inference. \n",
     "We will use a custom estimator to perform the classification task, and train and infer using that custom estimator.\n",
@@ -31,7 +31,7 @@
    "source": [
     "## Setup\n",
     "\n",
-    "First, we import the necessary modules and create the SparkSession and `SparkSession` with the SageMaker-Spark dependencies attached. "
+    "First, we import the necessary modules and create the `SparkSession` with the SageMaker-Spark dependencies attached. "
    ]
   },
   {
@@ -194,7 +194,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let's train this estimator by calling fit on it with the training data."
+    "Let's train this estimator by calling fit on it with the training data. Please note the below code will take several minutes to run and create all the resources needed for this model. "
    ]
   },
   {
@@ -335,7 +335,7 @@
    "pygments_lexer": "ipython3",
    "version": "3.6.4"
   },
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
+  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2

sagemaker-spark/pyspark_mnist/pyspark_mnist_kmeans.ipynb

@@ -16,7 +16,7 @@
     "10. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n",
     "\n",
     "## Introduction\n",
-    "This notebook will show how to classify handwritten digits through the SageMaker PySpark library. \n",
+    "This notebook will show how to cluster handwritten digits through the SageMaker PySpark library. \n",
     "\n",
     "We will manipulate data through Spark using a SparkSession, and then use the SageMaker Spark library to interact with SageMaker for training and inference. \n",
     "We will first train on SageMaker using K-Means clustering on the MNIST dataset. Then, we will see how to re-use models from existing endpoints and from a model stored on S3 in order to only run inference. \n",
@@ -32,7 +32,7 @@
    "source": [
     "## Setup\n",
     "\n",
-    "First, we import the necessary modules and create the SparkSession and `SparkSession` with the SageMaker-Spark dependencies attached. "
+    "First, we import the necessary modules and create the `SparkSession` with the SageMaker-Spark dependencies attached. "
    ]
   },
   {
@@ -540,7 +540,7 @@
    "pygments_lexer": "ipython3",
    "version": "3.6.4"
   },
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
+  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2

sagemaker-spark/pyspark_mnist/pyspark_mnist_pca_kmeans.ipynb

@@ -15,7 +15,7 @@
     "7. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n",
     "\n",
     "## Introduction\n",
-    "This notebook will show how to classify handwritten digits through the SageMaker PySpark library. \n",
+    "This notebook will show how to cluster handwritten digits through the SageMaker PySpark library. \n",
     "\n",
     "We will manipulate data through Spark using a SparkSession, and then use the SageMaker Spark library to interact with SageMaker for training and inference. \n",
     "We will create a pipeline consisting of a first step to reduce the dimensionality using SageMaker's PCA algorithm, followed by the final K-Means clustering step on SageMaker. \n",
@@ -31,7 +31,7 @@
    "source": [
     "## Setup\n",
     "\n",
-    "First, we import the necessary modules and create the SparkSession and `SparkSession` with the SageMaker-Spark dependencies attached. "
+    "First, we import the necessary modules and create the `SparkSession` with the SageMaker-Spark dependencies attached. "
    ]
   },
   {
@@ -180,16 +180,11 @@
     "pipelineSM = Pipeline(stages=[pcaSageMakerEstimator, kMeansSageMakerEstimator])"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now that we've defined the `Pipeline`, we can call fit on the training data. "
+    "Now that we've defined the `Pipeline`, we can call fit on the training data. Please note the below code will take several minutes to run and create all the resources needed for this pipeline. "
    ]
   },
   {
@@ -231,6 +226,13 @@
     "![PCA and KMeans on SageMaker](img/sagemaker-spark-pca-kmeans-architecture.png)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please note the below code will take several minutes to run and create the final K-Means endpoint needed for this pipeline. "
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -354,7 +356,7 @@
    "pygments_lexer": "ipython3",
    "version": "3.6.4"
   },
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
+  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2

sagemaker-spark/pyspark_mnist/pyspark_mnist_pca_mllib_kmeans.ipynb

@@ -15,14 +15,18 @@
     "7. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n",
     "\n",
     "## Introduction\n",
-    "This notebook will show how to classify handwritten digits through the SageMaker PySpark library. \n",
+    "This notebook will show how to cluster handwritten digits through the SageMaker PySpark library. \n",
     "\n",
     "We will manipulate data through Spark using a SparkSession, and then use the SageMaker Spark library to interact with SageMaker for training and inference. \n",
     "We will create a pipeline consisting of a first step to reduce the dimensionality using Spark MLLib PCA algorithm, followed by the final K-Means clustering step on SageMaker. \n",
     "\n",
     "You can visit SageMaker Spark's GitHub repository at https://github.com/aws/sagemaker-spark to learn more about SageMaker Spark.\n",
     "\n",
-    "This notebook was created and tested on an ml.m4.xlarge notebook instance."
+    "This notebook was created and tested on an ml.m4.xlarge notebook instance.\n",
+    "\n",
+    "## Why use Spark MLLib algorithms? \n",
+    "\n",
+    "The use of Spark MLLib PCA in this notebook is meant to showcase how you can use different pre-processting steps, ranging from data transformers to algorithms, with tools such as Spark MLLib that are well suited for data pre-processing. You can then use SageMaker algorithms and features through the SageMaker-Spark SDK. Here in our case, PCA is in charge of reducing the feature vector as a pre-processing step, and K-Means responsible for clustering the data. "
    ]
   },
   {
@@ -31,7 +35,7 @@
    "source": [
     "## Setup\n",
     "\n",
-    "First, we import the necessary modules and create the SparkSession and `SparkSession` with the SageMaker-Spark dependencies attached. "
+    "First, we import the necessary modules and create the `SparkSession` with the SageMaker-Spark dependencies attached. "
    ]
   },
   {
@@ -191,7 +195,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now that we've defined the `Pipeline`, we can call fit on the training data. "
+    "Now that we've defined the `Pipeline`, we can call fit on the training data. Please note the below code will take several minutes to run and create all the resources needed for this pipeline. "
    ]
   },
   {
@@ -219,7 +223,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Inference"
+    "## Inference\n",
+    "\n",
+    "Let's use our test data on our pipeline by calling `transform`. Please note the below code will take several minutes to run and create the endpoints needed in order to serve this pipeline.  "
    ]
   },
   {
@@ -350,7 +356,7 @@
    "pygments_lexer": "ipython3",
    "version": "3.6.4"
   },
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
+  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2