Skip to content

Commit 71242a2

Browse files
djarpindjarpin
authored
Merge branch 'master' into arpin_blazingtext_readme
2 parents 901bb84 + e2f6338 commit 71242a2

File tree

6 files changed

+568
-3
lines changed

6 files changed

+568
-3
lines changed

‎advanced_functionality/working_with_redshift_data/working_with_redshift_data.ipynb

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@
2727
"1. Preload that cluster with data from the [iris data set](https://archive.ics.uci.edu/ml/datasets/iris) in a table named public.irisdata.\n",
2828
"1. Update the credential file (`redshift_creds_template.json.nogit`) file with the appropriate information.\n",
2929
"\n",
30+
"Also, note that this Notebook instance needs to resolve to a private IP when connecting to the Redshift instance. There are two ways to resolve the Redshift DNS name to a private IP:\n",
31+
"1. The Redshift cluster is not publicly accessible so by default it will resolve to private IP.\n",
32+
"1. The Redshift cluster is publicly accessible and has an EIP associated with it but when accessed from within a VPC, it should resolve to private IP of the Redshift cluster. This is possible by setting following two VPC attributes to yes: DNS resolution and DNS hostnames. For instructions on setting that up, see Redshift public docs on [Managing Clusters in an Amazon Virtual Private Cloud (VPC)](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-clusters-vpc.html).\n",
33+
"\n",
3034
"### Notebook Setup\n",
3135
"Let's start by installing `psycopg2`, a PostgreSQL database adapter for the Python, adding a few imports and specifying a few configs. "
3236
]

‎introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,7 @@
298298
"\n",
299299
"> `Training job ended with status: Completed`\n",
300300
"\n",
301-
"then that means training sucessfully completed and the output model was stored in the output path specified by `training_params['OutputDataConfig']`.\n",
301+
"then that means training successfully completed and the output model was stored in the output path specified by `training_params['OutputDataConfig']`.\n",
302302
"\n",
303303
"You can also view information about and the status of a training job using the AWS SageMaker console. Just click on the \"Jobs\" tab."
304304
]

‎sagemaker-python-sdk/tensorflow_distributed_mnist/tensorflow_distributed_mnist.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@
120120
"When distributed training happens, the same neural network will be sent to the multiple training instances. Each instance will predict a batch of the dataset, calculate loss and minimize the optimizer. One entire loop of this process is called **training step**.\n",
121121
"\n",
122122
"### Syncronizing training steps\n",
123-
"A [global step](https://www.tensorflow.org/api_docs/python/tf/train/global_step) is a global variable shared between the instances. It necessary for distributed training, so the optimizer will keep track of the number of **training steps** between runs: \n",
123+
"A [global step](https://www.tensorflow.org/api_docs/python/tf/train/global_step) is a global variable shared between the instances. It's necessary for distributed training, so the optimizer will keep track of the number of **training steps** between runs: \n",
124124
"\n",
125125
"```python\n",
126126
"train_op = optimizer.minimize(loss, tf.train.get_or_create_global_step())\n",

‎sagemaker-spark/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@
44

55
These examples show how to use Amazon SageMaker for model training, hosting, and inference through Apache Spark using [SageMaker Spark](https://github.com/aws/sagemaker-spark). SageMaker Spark allows you to interleave Spark Pipeline stages with Pipeline stages that interact with Amazon SageMaker.
66

7-
- [MNIST with SageMaker PySpark](pyspark_mnist)
7+
- [MNIST with SageMaker PySpark](pyspark_mnist)

‎sagemaker-spark/pyspark_mnist/pyspark_mnist_kmeans.ipynb

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# SageMaker PySpark K-Means Clustering MNIST Example\n",
8+
"\n",
9+
"1. [Introduction](#Introduction)\n",
10+
"2. [Setup](#Setup)\n",
11+
"3. [Loading the Data](#Loading-the-Data)\n",
12+
"4. [Training and Hosting a Model](#Training-and-Hosting-a-Model)\n",
13+
"5. [Inference](#Inference)\n",
14+
"6. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n"
15+
]
16+
},
17+
{
18+
"cell_type": "markdown",
19+
"metadata": {},
20+
"source": [
21+
"## Introduction\n",
22+
"This notebook will show how to classify handwritten digits using the K-Means clustering algorithm through the SageMaker PySpark library. We will train on Amazon SageMaker using K-Means clustering on the MNIST dataset, host the trained model on Amazon SageMaker, and then make predictions against that hosted model.\n",
23+
"\n",
24+
"Unlike the other notebooks that demonstrate K-Means clustering on Amazon SageMaker, this notebook uses a SparkSession to manipulate data, and uses the SageMaker Spark library to interact with SageMaker with Spark Estimators and Transformers.\n",
25+
"\n",
26+
"You can visit SageMaker Spark's GitHub repository at https://github.com/aws/sagemaker-spark to learn more about SageMaker Spark.\n",
27+
"\n",
28+
"This notebook was created and tested on an ml.m4.xlarge notebook instance."
29+
]
30+
},
31+
{
32+
"cell_type": "markdown",
33+
"metadata": {},
34+
"source": [
35+
"## Setup\n",
36+
"\n",
37+
"First, we import the necessary modules and create the SparkSession with the SageMaker Spark dependencies."
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": null,
43+
"metadata": {
44+
"collapsed": true
45+
},
46+
"outputs": [],
47+
"source": [
48+
"import os\n",
49+
"\n",
50+
"from pyspark import SparkContext, SparkConf\n",
51+
"from pyspark.sql import SparkSession\n",
52+
"\n",
53+
"import sagemaker\n",
54+
"from sagemaker import get_execution_role\n",
55+
"import sagemaker_pyspark\n",
56+
"\n",
57+
"role = get_execution_role()\n",
58+
"\n",
59+
"# Configure Spark to use the SageMaker Spark dependency jars\n",
60+
"jars = sagemaker_pyspark.classpath_jars()\n",
61+
"\n",
62+
"classpath = \":\".join(sagemaker_pyspark.classpath_jars())\n",
63+
"\n",
64+
"# See the SageMaker Spark Github repo under sagemaker-pyspark-sdk\n",
65+
"# to learn how to connect to a remote EMR cluster running Spark from a Notebook Instance.\n",
66+
"spark = SparkSession.builder.config(\"spark.driver.extraClassPath\", classpath)\\\n",
67+
" .master(\"local[*]\").getOrCreate()"
68+
]
69+
},
70+
{
71+
"cell_type": "markdown",
72+
"metadata": {},
73+
"source": [
74+
"## Loading the Data\n",
75+
"\n",
76+
"Now, we load the MNIST dataset into a Spark Dataframe, which dataset is available in LibSVM format at\n",
77+
"\n",
78+
"`s3://sagemaker-sample-data-[region]/spark/mnist/train/`\n",
79+
"\n",
80+
"where `[region]` is replaced with a supported AWS region, such as us-east-1.\n",
81+
"\n",
82+
"In order to train and make inferences our input DataFrame must have a column of Doubles (named \"label\" by default) and a column of Vectors of Doubles (named \"features\" by default).\n",
83+
"\n",
84+
"Spark's LibSVM DataFrameReader loads a DataFrame already suitable for training and inference.\n",
85+
"\n",
86+
"Here, we load into a DataFrame in the SparkSession running on the local Notebook Instance, but you can connect your Notebook Instance to a remote Spark cluster for heavier workloads. Starting from EMR 5.11.0, SageMaker Spark is pre-installed on EMR Spark clusters. For more on connecting your SageMaker Notebook Instance to a remote EMR cluster, please see [this blog post](https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/)."
87+
]
88+
},
89+
{
90+
"cell_type": "code",
91+
"execution_count": null,
92+
"metadata": {
93+
"collapsed": true
94+
},
95+
"outputs": [],
96+
"source": [
97+
"import boto3\n",
98+
"\n",
99+
"region = boto3.Session().region_name\n",
100+
"\n",
101+
"trainingData = spark.read.format('libsvm')\\\n",
102+
" .option('numFeatures', '784')\\\n",
103+
" .load('s3a://sagemaker-sample-data-{}/spark/mnist/train/'.format(region))\n",
104+
"\n",
105+
"testData = spark.read.format('libsvm')\\\n",
106+
" .option('numFeatures', '784')\\\n",
107+
" .load('s3a://sagemaker-sample-data-{}/spark/mnist/test/'.format(region))\n",
108+
"\n",
109+
"trainingData.show()"
110+
]
111+
},
112+
{
113+
"cell_type": "markdown",
114+
"metadata": {},
115+
"source": [
116+
"## Training and Hosting a Model\n",
117+
"Now we create a KMeansSageMakerEstimator, which uses the KMeans Amazon SageMaker Algorithm to train on our input data, and uses the KMeans Amazon SageMaker model image to host our model.\n",
118+
"\n",
119+
"Calling fit() on this estimator will train our model on Amazon SageMaker, and then create an Amazon SageMaker Endpoint to host our model.\n",
120+
"\n",
121+
"We can then use the SageMakerModel returned by this call to fit() to transform Dataframes using our hosted model.\n",
122+
"\n",
123+
"The following cell runs a training job and creates an endpoint to host the resulting model, so this cell can take up to twenty minutes to complete."
124+
]
125+
},
126+
{
127+
"cell_type": "code",
128+
"execution_count": null,
129+
"metadata": {
130+
"collapsed": true
131+
},
132+
"outputs": [],
133+
"source": [
134+
"import random\n",
135+
"\n",
136+
"from sagemaker_pyspark import IAMRole, S3DataPath\n",
137+
"from sagemaker_pyspark.algorithms import KMeansSageMakerEstimator\n",
138+
"\n",
139+
"kmeans_estimator = KMeansSageMakerEstimator(\n",
140+
" sagemakerRole=IAMRole(role),\n",
141+
" trainingInstanceType='ml.m4.xlarge',\n",
142+
" trainingInstanceCount=1,\n",
143+
" endpointInstanceType='ml.m4.xlarge',\n",
144+
" endpointInitialInstanceCount=1)\n",
145+
"\n",
146+
"kmeans_estimator.setK(10)\n",
147+
"kmeans_estimator.setFeatureDim(784)\n",
148+
"\n",
149+
"# train\n",
150+
"model = kmeans_estimator.fit(trainingData)"
151+
]
152+
},
153+
{
154+
"cell_type": "markdown",
155+
"metadata": {},
156+
"source": [
157+
"## Inference\n",
158+
"\n",
159+
"Now we transform our DataFrame.\n",
160+
"To do this, we serialize each row's \"features\" Vector of Doubles into a Protobuf format for inference against the Amazon SageMaker Endpoint. We deserialize the Protobuf responses back into our DataFrame. This serialization and deserialization is handled automatically by the `transform()` method:"
161+
]
162+
},
163+
{
164+
"cell_type": "code",
165+
"execution_count": null,
166+
"metadata": {
167+
"collapsed": true
168+
},
169+
"outputs": [],
170+
"source": [
171+
"transformedData = model.transform(testData)\n",
172+
"\n",
173+
"transformedData.show()"
174+
]
175+
},
176+
{
177+
"cell_type": "markdown",
178+
"metadata": {},
179+
"source": [
180+
"How well did the algorithm perform? Let us display the digits from each of the clusters and manually inspect the results:"
181+
]
182+
},
183+
{
184+
"cell_type": "code",
185+
"execution_count": null,
186+
"metadata": {
187+
"collapsed": true
188+
},
189+
"outputs": [],
190+
"source": [
191+
"from pyspark.sql.types import DoubleType\n",
192+
"import matplotlib.pyplot as plt\n",
193+
"import numpy as np\n",
194+
"\n",
195+
"# helper function to display a digit\n",
196+
"def show_digit(img, caption='', xlabel='', subplot=None):\n",
197+
" if subplot==None:\n",
198+
" _,(subplot)=plt.subplots(1,1)\n",
199+
" imgr=img.reshape((28,28))\n",
200+
" subplot.axes.get_xaxis().set_ticks([])\n",
201+
" subplot.axes.get_yaxis().set_ticks([])\n",
202+
" plt.title(caption)\n",
203+
" plt.xlabel(xlabel)\n",
204+
" subplot.imshow(imgr, cmap='gray')\n",
205+
"\n",
206+
"images = np.array(transformedData.select(\"features\").cache().take(250))\n",
207+
"clusters = transformedData.select(\"closest_cluster\").cache().take(250)\n",
208+
"\n",
209+
"for cluster in range(10):\n",
210+
" print('\\n\\n\\nCluster {}:'.format(int(cluster)))\n",
211+
" digits = [ img for l, img in zip(clusters, images) if int(l.closest_cluster) == cluster ]\n",
212+
" height=((len(digits)-1)//5)+1\n",
213+
" width=5\n",
214+
" plt.rcParams[\"figure.figsize\"] = (width,height)\n",
215+
" _, subplots = plt.subplots(height, width)\n",
216+
" subplots=np.ndarray.flatten(subplots)\n",
217+
" for subplot, image in zip(subplots, digits):\n",
218+
" show_digit(image, subplot=subplot)\n",
219+
" for subplot in subplots[len(digits):]:\n",
220+
" subplot.axis('off')\n",
221+
"\n",
222+
" plt.show()"
223+
]
224+
},
225+
{
226+
"cell_type": "markdown",
227+
"metadata": {},
228+
"source": [
229+
"Since we don't need to make any more inferences, now we delete the endpoint:"
230+
]
231+
},
232+
{
233+
"cell_type": "code",
234+
"execution_count": null,
235+
"metadata": {
236+
"collapsed": true
237+
},
238+
"outputs": [],
239+
"source": [
240+
"# Delete the endpoint\n",
241+
"\n",
242+
"from sagemaker_pyspark import SageMakerResourceCleanup\n",
243+
"\n",
244+
"resource_cleanup = SageMakerResourceCleanup(model.sagemakerClient)\n",
245+
"resource_cleanup.deleteResources(model.getCreatedResources())"
246+
]
247+
},
248+
{
249+
"cell_type": "markdown",
250+
"metadata": {},
251+
"source": [
252+
"## More on SageMaker Spark\n",
253+
"\n",
254+
"The SageMaker Spark Github repository has more about SageMaker Spark, including how to use SageMaker Spark with your own algorithms on Amazon SageMaker: https://github.com/aws/sagemaker-spark\n"
255+
]
256+
}
257+
],
258+
"metadata": {
259+
"kernelspec": {
260+
"display_name": "conda_python3",
261+
"language": "python",
262+
"name": "conda_python3"
263+
},
264+
"language_info": {
265+
"codemirror_mode": {
266+
"name": "ipython",
267+
"version": 3
268+
},
269+
"file_extension": ".py",
270+
"mimetype": "text/x-python",
271+
"name": "python",
272+
"nbconvert_exporter": "python",
273+
"pygments_lexer": "ipython3",
274+
"version": "3.6.2"
275+
},
276+
"notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
277+
},
278+
"nbformat": 4,
279+
"nbformat_minor": 2
280+
}

0 commit comments

Comments
 (0)