Skip to content

Update TFS EI example. #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 29, 2019
Merged

Update TFS EI example. #660

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
52b8e07
Update tf ei example to reflect changes in tfs container.
Mar 5, 2019
aa17e3d
Rename notebook.
Mar 5, 2019
39426e8
Remove TF container reference.
Mar 6, 2019
08817f3
Add TFS container reference.
Mar 7, 2019
083bd14
Change TFS refenrence.
Mar 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 ...ic_inference_with_your_own_model/entry.py → ...ic_inference_with_your_own_model/entry.py
View file
Open in desktop
File renamed without changes.
36 changes: 16 additions & 20 deletions ..._pretrained_model_elastic_inference.ipynb → ..._pretrained_model_elastic_inference.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using Amazon Elastic Inference with a pre-trained TensorFlow model on SageMaker\n",
"# Using Amazon Elastic Inference with a pre-trained TensorFlow Serving model on SageMaker\n",
"\n",
"This notebook demonstrates how to enable and use Amazon Elastic Inference with our predefined SageMaker TensorFlow containers.\n",
"This notebook demonstrates how to enable and use Amazon Elastic Inference with our predefined SageMaker TensorFlow Serving containers.\n",
"\n",
"Amazon Elastic Inference (EI) is a resource you can attach to your Amazon EC2 instances to accelerate your deep learning (DL) inference workloads. EI allows you to add inference acceleration to an Amazon SageMaker hosted endpoint or Jupyter notebook for a fraction of the cost of using a full GPU instance. For more information please visit: https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html\n",
"\n",
"This notebook's main objective is to show how to create an endpoint, backed by an Elastic Inference, to serve our pre-trained TensorFlow model for predictions. With a more efficient cost per performance, Amazon Elastic Inference can prove to be useful for those looking to use GPUs for higher inference performance at a lower cost.\n",
"This notebook's main objective is to show how to create an endpoint, backed by an Elastic Inference, to serve our pre-trained TensorFlow Serving model for predictions. With a more efficient cost per performance, Amazon Elastic Inference can prove to be useful for those looking to use GPUs for higher inference performance at a lower cost.\n",
"\n",
"1. [The model](#The-model)\n",
"1. [Setup role for SageMaker](#Setup-role-for-SageMaker)\n",
"1. [Load the TensorFlow Model on Amazon SageMaker using Python SDK](#Load-the-TensorFlow-Model-on-Amazon-SageMaker-using-Python-SDK)\n",
"1. [Load the TensorFlow Serving Model on Amazon SageMaker using Python SDK](#Load-the-TensorFlow-Serving-Model-on-Amazon-SageMaker-using-Python-SDK)\n",
"1. [Deploy the trained Model to an Endpoint with EI](#Deploy-the-trained-Model-to-an-Endpoint-with-EI)\n",
" 1. [Using EI with a SageMaker notebook instance](#Using-EI-with-a-SageMaker-notebook-instance)\n",
" 1. [Invoke the Endpoint to get inferences](#Invoke-the-Endpoint-to-get-inferences)\n",
Expand All @@ -38,7 +38,7 @@
"\n",
"The pre-trained model we will be using for this example is a NCHW ResNet-50 model from the [official Tensorflow model Github repository](https://github.com/tensorflow/models/tree/master/official/resnet#pre-trained-model). For more information in regards to deep residual networks, please check [here](https://github.com/tensorflow/models/tree/master/official/resnet). It isn't a requirement to train our model on SageMaker to use SageMaker for serving our model.\n",
"\n",
"SageMaker expects our models to be compressed in a tar.gz format in S3. Thankfully, our model already comes in that format. The predefined SageMaker TensorFlow containers utilize TensorFlow serving for loading and handling inferences, so it is expecting a SavedModel. The [predefined SageMaker Tensorflow](https://github.com/aws/sagemaker-tensorflow-container/blob/master/src/tf_container/serve.py#L108) container expects the base folder name for the saved model to be `export/Servo` followed by a verison and your SavedModel artifacts.\n",
"SageMaker expects our models to be compressed in a tar.gz format in S3. Thankfully, our model already comes in that format. The predefined TensorFlow Serving containers use REST API for handling inferences, for more informationm, please see [Deploying to TensorFlow Serving Endpoints](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst#making-predictions-against-a-sagemaker-endpoint).\n",
"\n",
"To host our model for inferences in SageMaker, we need to first upload the SavedModel to S3. This can be done through the AWS console or AWS command line.\n",
"\n",
Expand Down Expand Up @@ -85,15 +85,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load the TensorFlow Model on Amazon SageMaker using Python SDK\n",
"## Load the TensorFlow Serving Model on Amazon SageMaker using Python SDK\n",
"\n",
"We can use the SageMaker Python SDK to load our pre-trained TensorFlow model for hosting in SageMaker for predictions.\n",
"We can use the SageMaker Python SDK to load our pre-trained TensorFlow Serving model for hosting in SageMaker for predictions.\n",
"\n",
"There are a few parameters that our TensorFlowModel is expecting.\n",
"There are a few parameters that our TensorFlow Serving Model is expecting.\n",
"1. `model_data` - The S3 location of a model tar.gz file to load in SageMaker\n",
"2. `entry_point` - Path (absolute or relative) to the Python source file which should be executed as the entry point to model hosting. If a blank file is provided, default configurations will be used for model hosting.\n",
"3. `role` - An IAM role name or ARN for SageMaker to access AWS resources on your behalf.\n",
"4. `framework_version` - TensorFlow serving version you want to use for handling your inference request .\n"
"2. `role` - An IAM role name or ARN for SageMaker to access AWS resources on your behalf.\n",
"3. `framework_version` - TensorFlow Serving version you want to use for handling your inference request .\n"
]
},
{
Expand All @@ -102,12 +101,11 @@
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.tensorflow import TensorFlowModel\n",
"from sagemaker.tensorflow.serving import Model\n",
"\n",
"tensorflow_model = TensorFlowModel(model_data=saved_model,\n",
" entry_point='entry.py',\n",
" role=role,\n",
" framework_version='1.12')"
"tensorflow_model = Model(model_data=saved_model,\n",
" role=role,\n",
" framework_version='1.12')"
]
},
{
Expand All @@ -118,9 +116,7 @@
"\n",
"The `deploy()` method creates an endpoint which serves prediction requests in real-time.\n",
"\n",
"The only change required for utilizing EI with our SageMaker TensorFlow containers only requires providing an `accelerator_type` parameter, which determines which type of EI accelerator to attach to your endpoint. The supported types of accelerators can be found here: https://aws.amazon.com/sagemaker/pricing/instance-types/\n",
"\n",
"No code changes are necessary for your model, as our predefined TensorFlow containers utilizes TensorFlow serving, which has been modified to utilize EI for inference, as long as an EI accelerator is attached to the endpoint."
"The only change required for utilizing EI with our SageMaker TensorFlow Serving containers only requires providing an `accelerator_type` parameter, which determines which type of EI accelerator to attach to your endpoint. The supported types of accelerators can be found here: https://aws.amazon.com/sagemaker/pricing/instance-types/\n"
]
},
{
Expand Down Expand Up @@ -167,7 +163,7 @@
"import numpy as np\n",
"random_input = np.random.rand(1, 1, 3, 3)\n",
"\n",
"prediction = predictor.predict({'input': random_input.tolist()})\n",
"prediction = predictor.predict({'inputs': random_input.tolist()})\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why the 'inputs' keyword is provided here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFS container checks if input data is labeled as 'instances/inputs/examples' (https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/tensorflow-serving.js#L131). If yes, it will pass the data as it is. Otherwise, for example, if the label is 'input', it prepends an 'instances' to the data (https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/tensorflow-serving.js#L139).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should mention that in the notebook, as it seems this functionality is TFS container specific.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it is a functionality of TFS: https://www.tensorflow.org/tfx/serving/api_rest#specifying_input_tensors_in_row_format

"\n",
"print(prediction)"
]
Expand Down