|
6 | 6 | "source": [ |
7 | 7 | "# Compile and Deploy a TensorFlow model on Inf1 instances\n", |
8 | 8 | "\n", |
| 9 | + "Amazon SageMaker supports Inf1 instances for high performance and cost-effective inferences. Inf1 instances are ideal for large scale machine learning inference applications like image recognition, speech recognition, natural language processing, personalization, and fraud detection. In this example, train a classification model on the MNIST dataset using TensorFlow, compile it using Amazon SageMaker Neo, deploy the model on Inf1 instances on a SageMaker endpoint, and use the Neo Deep Learning Runtime to make inferences in real-time and with low latency. \n", |
9 | 10 | "\n", |
10 | | - "Amazon SageMaker now supports Inf1 instances for high performance and cost-effective inferences. Inf1 instances are ideal for large scale machine learning inference applications like image recognition, speech recognition, natural language processing, personalization, and fraud detection. In this example, we train a classification model on the MNIST dataset using TensorFlow, compile it using Amazon SageMaker Neo, and deploy the model on Inf1 instances on a SageMaker endpoint and use the Neo Deep Learning Runtime to make inferences in real-time and with low latency. \n", |
| 11 | + "## Inf 1 instances \n", |
| 12 | + "Inf1 instances are built from the ground up to support machine learning inference applications and feature up to 16 AWS Inferentia chips, which are high-performance machine learning inference chips designed and built by AWS. The Inferentia chips are coupled with the latest custom 2nd generation Intel® Xeon® Scalable processors and up to 100 Gbps networking to enable high throughput inference. With 1 to 16 AWS Inferentia chips per instance, Inf1 instances can scale in performance to up to 2000 Tera Operations per Second (TOPS) and deliver extremely low latency for real-time inference applications. The large on-chip memory on AWS Inferentia chips used in Inf1 instances allows caching of machine learning models directly on the chip. This eliminates the need to access outside memory resources during inference, enabling low latency without impacting bandwidth. \n", |
11 | 13 | "\n", |
12 | | - "### Inf 1 instances \n", |
13 | | - "Inf1 instances are built from the ground up to support machine learning inference applications and feature up to 16 AWS Inferentia chips, high-performance machine learning inference chips designed and built by AWS. The Inferentia chips are coupled with the latest custom 2nd generation Intel® Xeon® Scalable processors and up to 100 Gbps networking to enable high throughput inference. With 1 to 16 AWS Inferentia chips per instance, Inf1 instances can scale in performance to up to 2000 Tera Operations per Second (TOPS) and deliver extremely low latency for real-time inference applications. The large on-chip memory on AWS Inferentia chips used in Inf1 instances allows caching of machine learning models directly on the chip. This eliminates the need to access outside memory resources during inference, enabling low latency without impacting bandwidth. \n" |
| 14 | + "## Prerequisites\n", |
| 15 | + "\n", |
| 16 | + "* SageMaker Studio with Python 3 (Data Science) kernel\n", |
| 17 | + "* SageMaker SDK version 1.x" |
14 | 18 | ] |
15 | 19 | }, |
16 | 20 | { |
17 | 21 | "cell_type": "markdown", |
18 | 22 | "metadata": {}, |
19 | 23 | "source": [ |
20 | | - "Make sure you selected `Python 3 (Data Science)` kernel." |
| 24 | + "## Setup\n", |
| 25 | + "\n", |
| 26 | + "Install the required version of SageMaker and TensorFlow." |
21 | 27 | ] |
22 | 28 | }, |
23 | 29 | { |
|
26 | 32 | "metadata": {}, |
27 | 33 | "outputs": [], |
28 | 34 | "source": [ |
29 | | - "%cd /root/amazon-sagemaker-examples/aws_sagemaker_studio/sagemaker_neo_compilation_jobs/deploy_tensorflow_model_on_Inf1_instance" |
| 35 | + "import sagemaker\n", |
| 36 | + "if sagemaker.__version__ >= '2':\n", |
| 37 | + " orig_sm_version = sagemaker.__version__\n", |
| 38 | + " with open('orig_sm_version.txt', \"w\") as f:\n", |
| 39 | + " f.write(orig_sm_version)\n", |
| 40 | + " %pip install \"sagemaker>=1.14.2,<2\"\n", |
| 41 | + "\n", |
| 42 | + "if sagemaker.__version__ >= '2':\n", |
| 43 | + " print(f\"WARNING: The current running version of the SageMaker SDK is {sagemaker.__version__}, which will cause this notebook to fail. \"\n", |
| 44 | + " f\"Restart the kernel to run the required version of the SDK.\")" |
30 | 45 | ] |
31 | 46 | }, |
32 | 47 | { |
|
35 | 50 | "metadata": {}, |
36 | 51 | "outputs": [], |
37 | 52 | "source": [ |
38 | | - "import sys\n", |
39 | | - "!{sys.executable} -m pip install tensorflow==1.13.1" |
| 53 | + "%pip install tensorflow==1.15.4" |
40 | 54 | ] |
41 | 55 | }, |
42 | 56 | { |
43 | 57 | "cell_type": "markdown", |
44 | 58 | "metadata": {}, |
45 | 59 | "source": [ |
46 | | - "### Set up the environment" |
| 60 | + "Start a SageMaker session and get the excecution role." |
47 | 61 | ] |
48 | 62 | }, |
49 | 63 | { |
|
90 | 104 | "cell_type": "markdown", |
91 | 105 | "metadata": {}, |
92 | 106 | "source": [ |
93 | | - "### Upload the data\n", |
94 | | - "We use the ```sagemaker.Session.upload_data``` function to upload our datasets to an S3 location. The return value inputs identifies the location -- we will use this later when we start the training job." |
| 107 | + "### Upload the data to Amazon Simple Storage Service (Amazon S3)\n", |
| 108 | + "Use the `sagemaker.Session.upload_data` function to upload datasets to an S3 location. The return value is the location, which is used when the training job is started." |
95 | 109 | ] |
96 | 110 | }, |
97 | 111 | { |
|
107 | 121 | "cell_type": "markdown", |
108 | 122 | "metadata": {}, |
109 | 123 | "source": [ |
110 | | - "# Construct a script for distributed training \n", |
111 | | - "Here is the full code for the network model:" |
| 124 | + "## Construct a script for distributed training \n", |
| 125 | + "\n", |
| 126 | + "To see the code for the network model, either browse to `mnist.py` in the File Browser or run the following command to show it here." |
112 | 127 | ] |
113 | 128 | }, |
114 | 129 | { |
|
124 | 139 | "cell_type": "markdown", |
125 | 140 | "metadata": {}, |
126 | 141 | "source": [ |
127 | | - "The script here is and adaptation of the [TensorFlow MNIST example](https://github.com/tensorflow/models/tree/master/official/vision/image_classification). It provides a ```model_fn(features, labels, mode)```, which is used for training, evaluation and inference. See [TensorFlow MNIST distributed training notebook](The script here is and adaptation of the [TensorFlow MNIST example](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/mnist). It provides a ```model_fn(features, labels, mode)```, which is used for training, evaluation and inference. See [TensorFlow MNIST distributed training notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_script_mode_training_and_serving/tensorflow_script_mode_training_and_serving.ipynb) for more details about the training script.\n", |
| 142 | + "This script is an adaptation of the [TensorFlow MNIST example](https://github.com/tensorflow/models/tree/master/official/vision/image_classification). It provides a `model_fn(features, labels, mode)` function that is used for training, evaluation and inference. For more details, see [TensorFlow MNIST distributed training notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_script_mode_training_and_serving/tensorflow_script_mode_training_and_serving.ipynb).\n", |
128 | 143 | "\n", |
| 144 | + "At the end of the training script, there are two additional functions that are used with Neo Deep Learning Runtime:\n", |
129 | 145 | "\n", |
130 | | - "At the end of the training script, there are two additional functions, to be used with Neo Deep Learning Runtime:\n", |
131 | | - "* `neo_preprocess(payload, content_type)`: Function that takes in the payload and Content-Type of each incoming request and returns a NumPy array\n", |
132 | | - "* `neo_postprocess(result)`: Function that takes the prediction results produced by Deep Learining Runtime and returns the response body\n", |
| 146 | + "* `neo_preprocess(payload, content_type)`: takes the payload and Content-Type of each incoming request and returns a NumPy array.\n", |
| 147 | + "* `neo_postprocess(result)`: takes the prediction results produced by Deep Learning Runtime and returns the response body.\n", |
133 | 148 | "\n", |
134 | 149 | "LeCun, Y., Cortes, C., & Burges, C. (2010). MNIST handwritten digit databaseATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2." |
135 | 150 | ] |
|
138 | 153 | "cell_type": "markdown", |
139 | 154 | "metadata": {}, |
140 | 155 | "source": [ |
141 | | - "## Create a training job using the sagemaker.TensorFlow estimator" |
| 156 | + "## Create a training job\n", |
| 157 | + "\n", |
| 158 | + "Use the `sagemaker.TensorFlow` estimator to create a training job." |
142 | 159 | ] |
143 | 160 | }, |
144 | 161 | { |
|
167 | 184 | "cell_type": "markdown", |
168 | 185 | "metadata": {}, |
169 | 186 | "source": [ |
170 | | - "The **```fit```** method will create a training job in two **ml.c5.xlarge** instances. The logs above will show the instances doing training, evaluation, and incrementing the number of **training steps**. \n", |
| 187 | + "The `fit` method creates a training job in two **ml.c5.xlarge** instances. The logs from `fit` show the instances training, evaluating, and incrementing the number of **training steps**. \n", |
171 | 188 | "\n", |
172 | | - "In the end of the training, the training job will generate a saved model for compilation." |
| 189 | + "At the end of the training, the training job generates a saved model for compilation." |
173 | 190 | ] |
174 | 191 | }, |
175 | 192 | { |
176 | 193 | "cell_type": "markdown", |
177 | 194 | "metadata": {}, |
178 | 195 | "source": [ |
179 | | - "# Deploy the trained model on Inf1 instance for real-time inferences\n", |
| 196 | + "## Deploy the trained model\n", |
| 197 | + "\n", |
| 198 | + "Deploy the model to an Inf1 instance for real-time inferences. Once the training is complete, compile the model using SageMaker Neo to optimize performance for the desired deployment target. SageMaker Neo enables you to train machine learning models once and run them anywhere in the cloud and at the edge. To compile the trained model for deployment to Inf1 instances, use the `TensorFlowEstimator.compile_model` method and select `ml_inf1` as the deployment target. The compiled model is deployed on an endpoint that uses Inf1 instances in SageMaker.\n", |
180 | 199 | "\n", |
181 | | - "Once the training is complete, we compile the model using Amazon SageMaker Neo to optize performance for our desired deployment target. Amazon SageMaker Neo enables you to train machine learning models once and run them anywhere in the cloud and at the edge. To compile our trained model for deploying on Inf1 instances, we are using the ``TensorFlowEstimator.compile_model`` method and select ``'ml_inf1'`` as our deployment target. The compiled model will then be deployed on an endpoint using Inf1 instances in Amazon SageMaker. \n", |
| 200 | + "### Compile the model \n", |
182 | 201 | "\n", |
183 | | - "## Compile the model \n", |
| 202 | + "The `input_shape` is the definition for the model's input tensor and `output_path` is where the compiled model is stored in S3.\n", |
184 | 203 | "\n", |
185 | | - "The ``input_shape`` is the definition for the model's input tensor and ``output_path`` is where the compiled model will be stored in S3. **Important. If the following command result in a permission error, scroll up and locate the value of execution role returned by `get_execution_role()`. The role must have access to the S3 bucket specified in ``output_path``.**" |
| 204 | + "> Note: If `compile_model` results in a permission error, verify that the execution role returned previously by `get_execution_role()` has access to the Amazon S3 bucket specified in `output_path`." |
186 | 205 | ] |
187 | 206 | }, |
188 | 207 | { |
|
204 | 223 | "cell_type": "markdown", |
205 | 224 | "metadata": {}, |
206 | 225 | "source": [ |
207 | | - "## Deploy the compiled model on a SageMaker endpoint\n", |
| 226 | + "### Deploy to a SageMaker endpoint\n", |
208 | 227 | "\n", |
209 | | - "Now that we have the compiled model, we will deploy it on an Amazon SageMaker endpoint. Inf1 instances in Amazon SageMaker are available in four sizes: ml.inf1.xlarge, ml.inf1.2xlarge, ml.inf1.6xlarge, and ml.inf1.24xlarge. In this example, we are using ``'ml.inf1.xlarge'`` for deploying our model.\n" |
| 228 | + "Deploy the compiled model to an Amazon SageMaker endpoint. This example uses the Inf1 `ml.inf1.xlarge` instance type." |
210 | 229 | ] |
211 | 230 | }, |
212 | 231 | { |
|
219 | 238 | " instance_type = 'ml.inf1.xlarge')" |
220 | 239 | ] |
221 | 240 | }, |
| 241 | + { |
| 242 | + "cell_type": "markdown", |
| 243 | + "metadata": {}, |
| 244 | + "source": [ |
| 245 | + "Configure a serializer for `application/vnd+python.numpy+binary` Content-Type." |
| 246 | + ] |
| 247 | + }, |
222 | 248 | { |
223 | 249 | "cell_type": "code", |
224 | 250 | "execution_count": null, |
|
242 | 268 | "source": [ |
243 | 269 | "## Invoking the endpoint\n", |
244 | 270 | "\n", |
245 | | - "Once the endpoint is ready, you can send requests to it and receive inference results in real-time with low latency. " |
| 271 | + "When the endpoint is ready, send requests to it and receive inference results in real time with low latency. " |
246 | 272 | ] |
247 | 273 | }, |
248 | 274 | { |
|
277 | 303 | "cell_type": "markdown", |
278 | 304 | "metadata": {}, |
279 | 305 | "source": [ |
280 | | - "## Deleting endpoint\n", |
| 306 | + "## Cleanup\n", |
281 | 307 | "\n", |
282 | | - "Delete the endpoint if you no longer need it. " |
| 308 | + "Delete the endpoint. " |
283 | 309 | ] |
284 | 310 | }, |
285 | 311 | { |
|
290 | 316 | "source": [ |
291 | 317 | "sagemaker_session.delete_endpoint(optimized_predictor.endpoint)" |
292 | 318 | ] |
| 319 | + }, |
| 320 | + { |
| 321 | + "cell_type": "markdown", |
| 322 | + "metadata": {}, |
| 323 | + "source": [ |
| 324 | + "Rollback the SageMaker Python SDK version" |
| 325 | + ] |
| 326 | + }, |
| 327 | + { |
| 328 | + "cell_type": "code", |
| 329 | + "execution_count": null, |
| 330 | + "metadata": {}, |
| 331 | + "outputs": [], |
| 332 | + "source": [ |
| 333 | + "# rollback the SageMaker Python SDK to the kernel's original version\n", |
| 334 | + "if os.path.exists('orig_sm_version.txt'):\n", |
| 335 | + " with open('orig_sm_version.txt', 'r') as f:\n", |
| 336 | + " orig_sm_version = f.read()\n", |
| 337 | + " print(f\"Original version: {orig_sm_version}\")\n", |
| 338 | + " print(f\"Current version: {sagemaker.__version__}\")\n", |
| 339 | + " %pip install sagemaker=={orig_sm_version}\n", |
| 340 | + " os.remove('orig_sm_version.txt')" |
| 341 | + ] |
| 342 | + }, |
| 343 | + { |
| 344 | + "cell_type": "markdown", |
| 345 | + "metadata": {}, |
| 346 | + "source": [ |
| 347 | + "Restart the kernel to run the updated version of the SDK." |
| 348 | + ] |
293 | 349 | } |
294 | 350 | ], |
295 | 351 | "metadata": { |
|
0 commit comments