Merge branch 'master' into custom_image_name

Author: iquintero

CHANGELOG.rst

@@ -2,15 +2,30 @@
 CHANGELOG
 =========
 
+1.5.1dev
+========
+
+* enhancement: Let Framework models reuse code uploaded by Framework estimators
+* enhancement: Unify generation of model uploaded code location
+* feature: Change minimum required scipy from 1.0.0 to 0.19.0
+* feature: Allow Chainer, Tensorflow and MXNet estimators to use a custom docker image.
+
+1.5.0
+=====
+
+* feature: Add Support for PyTorch Framework
+* feature: Estimators: add support for TensorFlow 1.7.0
+* feature: Estimators: add support for TensorFlow 1.8.0
+* feature: Allow Local Serving of Models in S3
+* enhancement: Allow option for ``HyperparameterTuner`` to not include estimator metadata in job
+* bug-fix: Estimators: Join tensorboard thread after fitting
+
 1.4.2
 =====
 
 * bug-fix: Estimators: Fix attach for LDA
 * bug-fix: Estimators: allow code_location to have no key prefix
 * bug-fix: Local Mode: Fix s3 training data download when there is a trailing slash
-* feature: Allow Local Serving of Models in S3
-* feature: Allow Chainer, Tensorflow and MXNet estimators to use a custom docker image.
-
 
 1.4.1
 =====

README.rst

@@ -28,10 +28,11 @@ Table of Contents
 3. `MXNet SageMaker Estimators <#mxnet-sagemaker-estimators>`__
 4. `TensorFlow SageMaker Estimators <#tensorflow-sagemaker-estimators>`__
 5. `Chainer SageMaker Estimators <#chainer-sagemaker-estimators>`__
-6. `AWS SageMaker Estimators <#aws-sagemaker-estimators>`__
-7. `BYO Docker Containers with SageMaker Estimators <#byo-docker-containers-with-sagemaker-estimators>`__
-8. `SageMaker Automatic Model Tuning <#sagemaker-automatic-model-tuning>`__
-9. `BYO Model <#byo-model>`__
+6. `PyTorch SageMaker Estimators <#pytorch-sagemaker-estimators>`__
+7. `AWS SageMaker Estimators <#aws-sagemaker-estimators>`__
+8. `BYO Docker Containers with SageMaker Estimators <#byo-docker-containers-with-sagemaker-estimators>`__
+9. `SageMaker Automatic Model Tuning <#sagemaker-automatic-model-tuning>`__
+10. `BYO Model <#byo-model>`__
 
 
 Getting SageMaker Python SDK
@@ -49,7 +50,7 @@ You can install from source by cloning this repository and issuing a pip install
 
     git clone https://github.com/aws/sagemaker-python-sdk.git
     python setup.py sdist
-    pip install dist/sagemaker-1.4.2.tar.gz
+    pip install dist/sagemaker-1.5.0.tar.gz
 
 Supported Python versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -64,19 +65,26 @@ http://aws.amazon.com/apache2.0/
 Running tests
 ~~~~~~~~~~~~~
 
-SageMaker Python SDK uses tox for running Python tests. You can run the tests by running tox:
+SageMaker Python SDK has unit tests and integration tests.
 
-::
-
-    tox
+**Unit tests**
 
-Tests are defined in ``tests/`` and includes unit and integ tests. If you just want to run unit tests, then you can issue:
+tox is a prerequisite for running unit tests so you need to make sure you have it installed. To run the unit tests:
 
 ::
 
     tox tests/unit
 
-To just run integ tests, issue the following command:
+**Integrations tests**
+
+To be able to run the integration tests, the following prerequisites must be met
+
+1. Access to an AWS account to run the tests on
+2. Make the AWS account credentials available to boto3 clients used in the tests
+3. Ensure the AWS account has an IAM role named :code:`SageMakerRole`
+4. Ensure the libraries mentioned in setup.py extra_require for test are installed which can be achieved using :code:`pip install --upgrade .[test]`
+
+You can run integ tests by issuing the following command:
 
 ::
 
@@ -117,6 +125,7 @@ Later sections of this document explain how to use the different Estimators and
 * `MXNet SageMaker Estimators and Models <#mxnet-sagemaker-estimators>`__
 * `TensorFlow SageMaker Estimators and Models <#tensorflow-sagemaker-estimators>`__
 * `Chainer SageMaker Estimators and Models <#chainer-sagemaker-estimators>`__
+* `PyTorch SageMaker Estimators <#pytorch-sagemaker-estimators>`__
 * `AWS SageMaker Estimators and Models <#aws-sagemaker-estimators>`__
 * `Custom SageMaker Estimators and Models <#byo-docker-containers-with-sagemaker-estimators>`__
 
@@ -215,7 +224,7 @@ TensorFlow SageMaker Estimators allow you to run your own TensorFlow
 training algorithms on SageMaker Learner, and to host your own TensorFlow
 models on SageMaker Hosting.
 
-Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``.
+Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``.
 
 More details at `TensorFlow SageMaker Estimators and Models`_.
 
@@ -236,6 +245,20 @@ More details at `Chainer SageMaker Estimators and Models`_.
 .. _Chainer SageMaker Estimators and Models: src/sagemaker/chainer/README.rst
 
 
+PyTorch SageMaker Estimators
+-------------------------------
+
+With PyTorch Estimators, you can train and host PyTorch models on Amazon SageMaker.
+
+Supported versions of PyTorch: ``0.4.0``
+
+You can visit the PyTorch repository at https://github.com/pytorch/pytorch.
+
+More details at `PyTorch SageMaker Estimators and Models`_.
+
+.. _PyTorch SageMaker Estimators and Models: src/sagemaker/pytorch/README.rst
+
+
 AWS SageMaker Estimators
 ------------------------
 Amazon SageMaker provides several built-in machine learning algorithms that you can use for a variety of problem types.
@@ -321,6 +344,14 @@ In addition, the ``fit()`` call uses a list of ``RecordSet`` objects instead of
     # Start hyperparameter tuning job
     my_tuner.fit([train_records, test_records])
 
+To aid with attaching a previously-started hyperparameter tuning job with a ``HyperparameterTuner`` instance, ``fit()`` injects metadata in the hyperparameters by default.
+If the algorithm you are using cannot handle unknown hyperparameters (e.g. an Amazon ML algorithm that does not have a custom estimator in the Python SDK), then you can set ``include_cls_metadata`` to ``False`` when calling fit:
+
+.. code:: python
+
+    my_tuner.fit({'train': 's3://my_bucket/my_training_data', 'test': 's3://my_bucket_my_testing_data'},
+                 include_cls_metadata=False)
+
 There is also an analytics object associated with each ``HyperparameterTuner`` instance that presents useful information about the hyperparameter tuning job.
 For example, the ``dataframe`` method gets a pandas dataframe summarizing the associated training jobs:
 

setup.py

@@ -23,7 +23,7 @@ def read(fname):
 
 
 setup(name="sagemaker",
-      version="1.4.2",
+      version="1.5.0",
       description="Open source library for training and deploying models on Amazon SageMaker.",
       packages=find_packages('src'),
       package_dir={'': 'src'},
@@ -44,7 +44,7 @@ def read(fname):
       ],
 
       # Declare minimal set for installation
-      install_requires=['boto3>=1.4.8', 'numpy>=1.9.0', 'protobuf>=3.1', 'scipy>=1.0.0', 'urllib3>=1.2',
+      install_requires=['boto3>=1.4.8', 'numpy>=1.9.0', 'protobuf>=3.1', 'scipy>=0.19.0', 'urllib3>=1.2',
                         'PyYAML>=3.2'],
 
       extras_require={

src/sagemaker/chainer/README.rst

@@ -243,10 +243,11 @@ Saving models
 In order to save your trained Chainer model for deployment on SageMaker, your training script should save your model
 to a certain filesystem path called `model_dir`. This value is accessible through the environment variable
 ``SM_MODEL_DIR``. The following code demonstrates how to save a trained Chainer model named ``model`` as
-``model.npz`` at the :
+``model.npz`` at the end of training:
 
 .. code:: python
-import chainer
+
+    import chainer
     import argparse
     import os
 
@@ -255,7 +256,7 @@ import chainer
         parser.add_argument('--model-dir', type=str, default=os.environ['SM_MODEL_DIR'])
         args, _ = parser.parse_known_args()
 
-        # ... train `model`, then save it to `model_dir`
+        # ... train `model`, then save it to `model_dir` as file 'model.npz'
         chainer.serializers.save_npz(os.path.join(args.model_dir, 'model.npz'), model)
 
 After your training job is complete, SageMaker will compress and upload the serialized model to S3, and your model data

src/sagemaker/chainer/estimator.py

@@ -109,7 +109,7 @@ def create_model(self, model_server_workers=None):
             sagemaker.chainer.model.ChainerModel: A SageMaker ``ChainerModel`` object.
                 See :func:`~sagemaker.chainer.model.ChainerModel` for full details.
         """
-        return ChainerModel(self.model_data, self.role, self.entry_point, source_dir=self.source_dir,
+        return ChainerModel(self.model_data, self.role, self.entry_point, source_dir=self._model_source_dir(),
                             enable_cloudwatch_metrics=self.enable_cloudwatch_metrics, name=self._current_job_name,
                             container_log_level=self.container_log_level, code_location=self.code_location,
                             py_version=self.py_version, framework_version=self.framework_version,

src/sagemaker/chainer/model.py

@@ -13,11 +13,10 @@
 from __future__ import absolute_import
 
 import sagemaker
-from sagemaker.fw_utils import create_image_uri
+from sagemaker.fw_utils import create_image_uri, model_code_key_prefix
 from sagemaker.model import FrameworkModel, MODEL_SERVER_WORKERS_PARAM_NAME
 from sagemaker.chainer.defaults import CHAINER_VERSION
 from sagemaker.predictor import RealTimePredictor, npy_serializer, numpy_deserializer
-from sagemaker.utils import name_from_image
 
 
 class ChainerPredictor(RealTimePredictor):
@@ -85,7 +84,8 @@ def prepare_container_def(self, instance_type):
             region_name = self.sagemaker_session.boto_session.region_name
             deploy_image = create_image_uri(region_name, self.__framework_name__, instance_type,
                                             self.framework_version, self.py_version)
-        deploy_key_prefix = self.key_prefix or self.name or name_from_image(deploy_image)
+
+        deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
         self._upload_code(deploy_key_prefix)
         deploy_env = dict(self.env)
         deploy_env.update(self._framework_env_vars())

src/sagemaker/estimator.py

@@ -566,7 +566,7 @@ def _prepare_for_training(self, job_name=None):
         self._hyperparameters[SAGEMAKER_REGION_PARAM_NAME] = self.sagemaker_session.boto_region_name
 
     def _stage_user_code_in_s3(self):
-        """ Upload the user training script to s3 and return the location.
+        """Upload the user training script to s3 and return the location.
 
         Returns: s3 uri
 
@@ -584,6 +584,14 @@ def _stage_user_code_in_s3(self):
                                   script=self.entry_point,
                                   directory=self.source_dir)
 
+    def _model_source_dir(self):
+        """Get the appropriate value to pass as source_dir to model constructor on deploying
+
+        Returns:
+            str: Either a local or an S3 path pointing to the source_dir to be used for code by the model to be deployed
+        """
+        return self.source_dir if self.sagemaker_session.local_mode else self.uploaded_code.s3_prefix
+
     def hyperparameters(self):
         """Return the hyperparameters as a dictionary to use for training.
 

src/sagemaker/fw_utils.py

@@ -19,6 +19,8 @@
 from collections import namedtuple
 from six.moves.urllib.parse import urlparse
 
+from sagemaker.utils import name_from_image
+
 """This module contains utility functions shared across ``Framework`` components."""
 
 
@@ -156,7 +158,7 @@ def framework_name_from_image(image_name):
     else:
         # extract framework, python version and image tag
         # We must support both the legacy and current image name format.
-        name_pattern = re.compile('^sagemaker-(tensorflow|mxnet|chainer):(.*?)-(.*?)-(py2|py3)$')
+        name_pattern = re.compile('^sagemaker-(tensorflow|mxnet|chainer|pytorch):(.*?)-(.*?)-(py2|py3)$')
         legacy_name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')
         name_match = name_pattern.match(sagemaker_match.group(8))
         legacy_match = legacy_name_pattern.match(sagemaker_match.group(8))
@@ -199,3 +201,21 @@ def parse_s3_url(url):
     if parsed_url.scheme != "s3":
         raise ValueError("Expecting 's3' scheme, got: {} in {}".format(parsed_url.scheme, url))
     return parsed_url.netloc, parsed_url.path.lstrip('/')
+
+
+def model_code_key_prefix(code_location_key_prefix, model_name, image):
+    """Returns the s3 key prefix for uploading code during model deployment
+
+    The location returned is a potential concatenation of 2 parts
+        1. code_location_key_prefix if it exists
+        2. model_name or a name derived from the image
+
+    Args:
+        code_location_key_prefix (str): the s3 key prefix from code_location
+        model_name (str): the name of the model
+        image (str): the image from which a default name can be extracted
+
+    Returns:
+        str: the key prefix to be used in uploading code
+    """
+    return '/'.join(filter(None, [code_location_key_prefix, model_name or name_from_image(image)]))

src/sagemaker/model.py

@@ -17,7 +17,7 @@
 import sagemaker
 
 from sagemaker.local import LocalSession
-from sagemaker.fw_utils import tar_and_upload_dir, parse_s3_url
+from sagemaker.fw_utils import tar_and_upload_dir, parse_s3_url, model_code_key_prefix
 from sagemaker.session import Session
 from sagemaker.utils import name_from_image, get_config_value
 
@@ -132,6 +132,7 @@ def __init__(self, model_data, image, role, entry_point, source_dir=None, predic
             source_dir (str): Path (absolute or relative) to a directory with any other training
                 source code dependencies aside from tne entry point file (default: None). Structure within this
                 directory will be preserved when training on SageMaker.
+                If the directory points to S3, no code will be uploaded and the S3 location will be used instead.
             predictor_cls (callable[string, sagemaker.session.Session]): A function to call to create
                a predictor (default: None). If not None, ``deploy`` will return the result of invoking
                this function on the created endpoint name.
@@ -169,7 +170,8 @@ def prepare_container_def(self, instance_type):
         Returns:
             dict[str, str]: A container definition object usable with the CreateModel API.
         """
-        self._upload_code(self.key_prefix or self.name or name_from_image(self.image))
+        deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, self.image)
+        self._upload_code(deploy_key_prefix)
         deploy_env = dict(self.env)
         deploy_env.update(self._framework_env_vars())
         return sagemaker.container_def(self.image, self.model_data, deploy_env)

src/sagemaker/mxnet/estimator.py

@@ -81,9 +81,9 @@ def create_model(self, model_server_workers=None):
         if self.image_name:
             kwargs['image'] = self.image_name
 
-        return MXNetModel(self.model_data, self.role, self.entry_point, source_dir=self.source_dir,
+        return MXNetModel(self.model_data, self.role, self.entry_point, source_dir=self._model_source_dir(),
                           enable_cloudwatch_metrics=self.enable_cloudwatch_metrics, name=self._current_job_name,
-                          container_log_level=self.container_log_level, code_location=self.code_location,
+                          container_log_level=self.container_log_level, code_location=self.code_location, image=self.image_name,
                           py_version=self.py_version, framework_version=self.framework_version,
                           model_server_workers=model_server_workers, sagemaker_session=self.sagemaker_session, **kwargs)
 

src/sagemaker/mxnet/model.py

@@ -13,11 +13,10 @@
 from __future__ import absolute_import
 
 import sagemaker
-from sagemaker.fw_utils import create_image_uri
+from sagemaker.fw_utils import create_image_uri, model_code_key_prefix
 from sagemaker.model import FrameworkModel, MODEL_SERVER_WORKERS_PARAM_NAME
 from sagemaker.mxnet.defaults import MXNET_VERSION
 from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
-from sagemaker.utils import name_from_image
 
 
 class MXNetPredictor(RealTimePredictor):
@@ -85,7 +84,8 @@ def prepare_container_def(self, instance_type):
             region_name = self.sagemaker_session.boto_session.region_name
             deploy_image = create_image_uri(region_name, self.__framework_name__, instance_type,
                                             self.framework_version, self.py_version)
-        deploy_key_prefix = self.key_prefix or self.name or name_from_image(deploy_image)
+
+        deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
         self._upload_code(deploy_key_prefix)
         deploy_env = dict(self.env)
         deploy_env.update(self._framework_env_vars())