|
4 | 4 | "cell_type": "markdown", |
5 | 5 | "metadata": {}, |
6 | 6 | "source": [ |
7 | | - "# Taking Fully Advantage of Parallelism With Data Distribution\n", |
| 7 | + "# Taking Full Advantage of Parallelism With Data Distribution\n", |
8 | 8 | "_**Using Amazon SageMaker's Managed, Distributed Training with Different Data Distribution Methods**_\n", |
9 | 9 | "\n", |
10 | 10 | "---\n", |
|
26 | 26 | "\n", |
27 | 27 | "## Background\n", |
28 | 28 | "\n", |
29 | | - "Amazon SageMaker makes it easy to train machine learning models across a large number of machines. This a non-trivial process, but Amazon SageMaker Algorithms and pre-bruilt MXNet and TensorFlow containers hide most of the complexity from you. Nevertheless, there are decisions on how a user structures their data which will have an implication on how the distributed training is carried out. This notebook will walk through details on setting up your data to take full advantage of distributed processing.\n", |
| 29 | + "Amazon SageMaker makes it easy to train machine learning models across a large number of machines. This a non-trivial process, but Amazon SageMaker Algorithms and pre-built MXNet and TensorFlow containers hide most of the complexity from you. Nevertheless, there are decisions on how a user structures their data which will have an implication on how the distributed training is carried out. This notebook will walk through details on setting up your data to take full advantage of distributed processing.\n", |
30 | 30 | "\n", |
31 | 31 | "---\n", |
32 | 32 | "# Setup\n", |
33 | 33 | "\n", |
34 | | - "_This notebook was created and tested on an ml.m4xlarge notebook instance._\n", |
| 34 | + "_This notebook was created and tested on an ml.m4.xlarge notebook instance._\n", |
35 | 35 | "\n", |
36 | 36 | "Let's start by specifying:\n", |
37 | 37 | "\n", |
|
78 | 78 | "import matplotlib.pyplot as plt\n", |
79 | 79 | "from IPython.display import display\n", |
80 | 80 | "import io\n", |
81 | | - "import convert_data\n", |
82 | 81 | "import time\n", |
83 | 82 | "import copy\n", |
84 | 83 | "import json\n", |
85 | | - "import sys" |
| 84 | + "import sys\n", |
| 85 | + "import sagemaker.amazon.common as smac\n", |
| 86 | + "import os" |
86 | 87 | ] |
87 | 88 | }, |
88 | 89 | { |
|
160 | 161 | "metadata": {}, |
161 | 162 | "source": [ |
162 | 163 | "We can see:\n", |
163 | | - "- `EventCode` is pretty unevently distributed, with some events making up 7%+ of the observations and others being a thousandth of a percent.\n", |
| 164 | + "- `EventCode` is pretty unevenly distributed, with some events making up 7%+ of the observations and others being a thousandth of a percent.\n", |
164 | 165 | "- `AvgTone` seems to be reasonably smoothly distributed, while `NumArticles` has a long tail, and `Actor` geo features have suspiciously large spikes near 0.\n", |
165 | 166 | "\n", |
166 | 167 | "Let's remove the (0, 0) lat-longs, one hot encode `EventCode`, and prepare our data for a machine learning model. For this example we'll keep things straightforward and try to predict `AvgTone`, using the other variables in our dataset as features.\n", |
|
193 | 194 | "outputs": [], |
194 | 195 | "source": [ |
195 | 196 | "def write_to_s3(bucket, prefix, channel, file_prefix, X, y):\n", |
196 | | - " f = io.BytesIO()\n", |
197 | | - " feature_size = X.shape[1]\n", |
198 | | - " for features, target in zip(X, y):\n", |
199 | | - " convert_data.write_recordio(f, convert_data.list_to_record_bytes(features, label=target, feature_size=feature_size))\n", |
200 | | - " f.seek(0)\n", |
201 | | - " boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, channel, file_prefix + '.data')).upload_fileobj(f)\n", |
| 197 | + " buf = io.BytesIO()\n", |
| 198 | + " smac.write_numpy_to_dense_tensor(buf, X.astype('float32'), y.astype('float32'))\n", |
| 199 | + " buf.seek(0)\n", |
| 200 | + " boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, channel, file_prefix + '.data')).upload_fileobj(buf)\n", |
202 | 201 | "\n", |
203 | 202 | "def transform_gdelt(df, events=None):\n", |
204 | 203 | " df = df[['AvgTone', 'EventCode', 'NumArticles', 'Actor1Geo_Lat', 'Actor1Geo_Long', 'Actor2Geo_Lat', 'Actor2Geo_Long']]\n", |
|
649 | 648 | "cell_type": "markdown", |
650 | 649 | "metadata": {}, |
651 | 650 | "source": [ |
652 | | - "Next, because POST requests to our endpoint are limited to ~6MB, we'll setup a small function to split our test data up into mini-batches, loop through and invoke our endpoint to get predictions, and gather them into a single array." |
| 651 | + "Next, because POST requests to our endpoint are limited to ~6MB, we'll setup a small function to split our test data up into mini-batches that are each about 5MB, loop through and invoke our endpoint to get predictions for those mini-batches, and gather them into a single array." |
653 | 652 | ] |
654 | 653 | }, |
655 | 654 | { |
|
714 | 713 | "\n", |
715 | 714 | "Different algorithms can be expected to show variation in which distribution mechanism is most effective at achieving optimal compute spend per point of model accuracy. The message remains the same though, that the process of finding the right distribution type is another experiment in optimizing model training times." |
716 | 715 | ] |
| 716 | + }, |
| 717 | + { |
| 718 | + "cell_type": "markdown", |
| 719 | + "metadata": {}, |
| 720 | + "source": [ |
| 721 | + "### (Optional) Clean-up\n", |
| 722 | + "\n", |
| 723 | + "If you're ready to be done with this notebook, please uncomment and run the cell below. This will remove the hosted endpoints you created and avoid any charges from a stray instance being left on." |
| 724 | + ] |
| 725 | + }, |
| 726 | + { |
| 727 | + "cell_type": "code", |
| 728 | + "execution_count": null, |
| 729 | + "metadata": {}, |
| 730 | + "outputs": [], |
| 731 | + "source": [ |
| 732 | + "#sm.delete_endpoint(EndpointName=sharded_endpoint)\n", |
| 733 | + "#sm.delete_endpoint(EndpointName=replicated_endpoint)" |
| 734 | + ] |
717 | 735 | } |
718 | 736 | ], |
719 | 737 | "metadata": { |
|
733 | 751 | "nbconvert_exporter": "python", |
734 | 752 | "pygments_lexer": "ipython3", |
735 | 753 | "version": "3.6.3" |
736 | | - } |
| 754 | + }, |
| 755 | + "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License." |
737 | 756 | }, |
738 | 757 | "nbformat": 4, |
739 | 758 | "nbformat_minor": 2 |
|
0 commit comments