Merge branch 'zwei' into add-serde-params

Author: bveeramani

CHANGELOG.md

@@ -1,5 +1,79 @@
 # Changelog
 
+## v1.72.0 (2020-07-29)
+
+### Features
+
+ * Neo: Add Granular Target Description support for compilation
+
+### Documentation Changes
+
+ * Add xgboost doc on bring your own model
+ * fix typos on processing docs
+
+## v1.71.1 (2020-07-27)
+
+### Bug Fixes and Other Changes
+
+ * remove redundant information from the user_agent string.
+
+### Testing and Release Infrastructure
+
+ * use unique model name in TFS integ tests
+ * use pytest-cov instead of coverage
+
+## v1.71.0 (2020-07-23)
+
+### Features
+
+ * Add mpi support for mxnet estimator api
+
+### Bug Fixes and Other Changes
+
+ * use 'sagemaker' logger instead of root logger
+ * account for "py36" and "py37" in image tag parsing
+
+## v1.70.2 (2020-07-22)
+
+### Bug Fixes and Other Changes
+
+ * convert network_config in processing_config to dict
+
+### Documentation Changes
+
+ * Add ECR URI Estimator example
+
+## v1.70.1 (2020-07-21)
+
+### Bug Fixes and Other Changes
+
+ * Nullable fields in processing_config
+
+## v1.70.0 (2020-07-20)
+
+### Features
+
+ * Add model monitor support for us-gov-west-1
+ * support TFS 2.2
+
+### Bug Fixes and Other Changes
+
+ * reshape Artifacts into data frame in ExperimentsAnalytics
+
+### Documentation Changes
+
+ * fix MXNet version info for requirements.txt support
+
+## v1.69.0 (2020-07-09)
+
+### Features
+
+ * Add ModelClientConfig Fields for Batch Transform
+
+### Documentation Changes
+
+ * add KFP Processing component
+
 ## v2.0.0.rc1 (2020-07-08)
 
 ### Breaking Changes

buildspec-unittests.yml

@@ -7,11 +7,11 @@ phases:
       - TOX_PARALLEL_NO_SPINNER=1
       - PY_COLORS=0
       - start_time=`date +%s`
-      - tox -e flake8,pylint,twine,black-check
+      - tox -e flake8,pylint,twine,black-check --parallel all
       - ./ci-scripts/displaytime.sh 'flake8,pylint,twine,black-check' $start_time
 
       - start_time=`date +%s`
-      - tox -e sphinx,doc8
+      - tox -e sphinx,doc8 --parallel all
       - ./ci-scripts/displaytime.sh 'sphinx,doc8' $start_time
 
       # run unit tests

doc/amazon_sagemaker_processing.rst

@@ -10,14 +10,14 @@ Amazon SageMaker Processing allows you to run steps for data pre- or post-proces
 Background
 ==========
 
-Amazon SageMaker lets developers and data scientists train and deploy machine learning models. With Amazon SageMaker Processing, you can run processing jobs on for data processing steps in your machine learning pipeline, which accept data from Amazon S3 as input, and put data into Amazon S3 as output.
+Amazon SageMaker lets developers and data scientists train and deploy machine learning models. With Amazon SageMaker Processing, you can run processing jobs for data processing steps in your machine learning pipeline. Processing jobs accept data from Amazon S3 as input and store data into Amazon S3 as output.
 
 .. image:: ./amazon_sagemaker_processing_image1.png
 
 Setup
 =====
 
-The fastest way to run get started with Amazon SageMaker Processing is by running a Jupyter notebook. You can follow the `Getting Started with Amazon SageMaker`_ guide to start running notebooks on Amazon SageMaker.
+The fastest way to get started with Amazon SageMaker Processing is by running a Jupyter notebook. You can follow the `Getting Started with Amazon SageMaker`_ guide to start running notebooks on Amazon SageMaker.
 
 .. _Getting Started with Amazon SageMaker: https://docs.aws.amazon.com/sagemaker/latest/dg/gs.html
 

doc/api/utility/image_uris.rst

@@ -0,0 +1,7 @@
+Image URIs
+----------
+
+.. automodule:: sagemaker.image_uris
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/frameworks/mxnet/using_mxnet.rst

@@ -159,13 +159,14 @@ If there are other packages you want to use with your script, you can include a
 Both ``requirements.txt`` and your training script should be put in the same folder.
 You must specify this folder in ``source_dir`` argument when creating an MXNet estimator.
 
-The function of installing packages using ``requirements.txt`` is supported for all MXNet versions during training.
+The function of installing packages using ``requirements.txt`` is supported for MXNet versions 1.3.0 and higher during training.
+
 When serving an MXNet model, support for this function varies with MXNet versions.
 For MXNet 1.6.0 or newer, ``requirements.txt`` must be under folder ``code``.
 The SageMaker MXNet Estimator automatically saves ``code`` in ``model.tar.gz`` after training (assuming you set up your script and ``requirements.txt`` correctly as stipulated in the previous paragraph).
 In the case of bringing your own trained model for deployment, you must save ``requirements.txt`` under folder ``code`` in ``model.tar.gz`` yourself or specify it through ``dependencies``.
-For MXNet 1.4.1, ``requirements.txt`` is not supported for inference.
-For MXNet 0.12.1-1.3.0, ``requirements.txt`` must be in ``source_dir``.
+For MXNet 0.12.1-1.2.1, 1.4.0-1.4.1, ``requirements.txt`` is not supported for inference.
+For MXNet 1.3.0, ``requirements.txt`` must be in ``source_dir``.
 
 A ``requirements.txt`` file is a text file that contains a list of items that are installed by using ``pip install``.
 You can also specify the version of an item to install.

doc/frameworks/tensorflow/using_tf.rst

@@ -178,6 +178,28 @@ To use Python 3.7, please specify both of the args:
 Where the S3 url is a path to your training data within Amazon S3.
 The constructor keyword arguments define how SageMaker runs your training script.
 
+Specify a Docker image using an Estimator
+-----------------------------------------
+
+There are use cases, such as extending an existing pre-built Amazon SageMaker images, that require specifing a Docker image when creating an Estimator by directly specifying the ECR URI instead of the Python and framework version. For a full list of available container URIs, see `Available Deep Learning Containers Images <https://github.com/aws/deep-learning-containers/blob/master/available_images.md>`__ For more information on using Docker containers, see `Use Your Own Algorithms or Models with Amazon SageMaker <https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html>`__.
+
+When specifying the image, you must use the ``image_name=''`` arg to replace the following arg:
+
+- ``py_version=''``
+
+You should still specify the ``framework_version=''`` arg because the SageMaker Python SDK accomodates for differences in the images based on the version.
+
+The following example uses the ``image_name=''`` arg to specify the container image, Python version, and framework version.
+
+.. code:: python
+
+   tf_estimator = TensorFlow(entry_point='tf-train.py',
+                             role='SageMakerRole',
+                             train_instance_count=1,
+                             train_instance_type='ml.p2.xlarge',
+                             image_name='763104351884.dkr.ecr.<region>.amazonaws.com/<framework>-<job type>:<framework version>-<cpu/gpu>-<python version>-ubuntu18.04',
+                             script_mode=True)
+
 For more information about the sagemaker.tensorflow.TensorFlow estimator, see `SageMaker TensorFlow Classes`_.
 
 Call the fit Method

doc/frameworks/xgboost/using_xgboost.rst

@@ -392,6 +392,56 @@ The function should return a byte array of data serialized to ``content_type``.
 The default implementation expects ``prediction`` to be a NumPy array and can serialize the result to JSON, CSV, or NPY.
 It accepts response content types of "application/json", "text/csv", and "application/x-npy".
 
+Bring Your Own Model
+--------------------
+
+You can deploy an XGBoost model that you trained outside of SageMaker by using the Amazon SageMaker XGBoost container.
+Typically, you save an XGBoost model by pickling the ``Booster`` object or calling ``booster.save_model``.
+The XGBoost `built-in algorithm mode <https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html#xgboost-modes>`_
+supports both a pickled ``Booster`` object and a model produced by ``booster.save_model``.
+You can also deploy an XGBoost model by using XGBoost as a framework.
+By using XGBoost as a framework, you have more flexibility.
+To deploy an XGBoost model by using XGBoost as a framework, you need to:
+
+- Write an inference script.
+- Create the XGBoostModel object.
+
+Write an Inference Script
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You must create an inference script that implements (at least) the ``model_fn`` function that calls the loaded model to get a prediction.
+
+Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output,
+and ``predict_fn`` to customize how the model server gets predictions from the loaded model.
+For information about how to write an inference script, see `SageMaker XGBoost Model Server <#sagemaker-xgboost-model-server>`_.
+Pass the filename of the inference script as the ``entry_point`` parameter when you create the `XGBoostModel` object.
+
+Create an XGBoostModel Object
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To create a model object, call the ``sagemaker.xgboost.model.XGBoostModel`` constructor,
+and then call its ``deploy()`` method to deploy your model for inference.
+
+.. code:: python
+
+    xgboost_model = XGBoostModel(
+        model_data="s3://my-bucket/my-path/model.tar.gz",
+        role="my-role",
+        entry_point="inference.py",
+        framework_version="1.0-1"
+    )
+
+    predictor = xgboost_model.deploy(
+        instance_type='ml.c4.xlarge',
+        initial_instance_count=1
+    )
+
+    # If payload is a string in LIBSVM format, we need to change serializer.
+    predictor.serializer = str
+    predictor.predict("<label> <index1>:<value1> <index2>:<value2>")
+
+To get predictions from your deployed model, you can call the ``predict()`` method.
+
 Host Multiple Models with Multi-Model Endpoints
 -----------------------------------------------
 
@@ -403,7 +453,6 @@ in the AWS documentation.
 For a sample notebook that uses Amazon SageMaker to deploy multiple XGBoost models to an endpoint, see the
 `Multi-Model Endpoint XGBoost Sample Notebook <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.ipynb>`_.
 
-
 *************************
 SageMaker XGBoost Classes
 *************************

doc/index.rst

@@ -1,7 +1,3 @@
-.. important::
-    We are working on v2.0.0. See https://github.com/aws/sagemaker-python-sdk/issues/1459
-    for more info on our plans and to leave feedback!
-
 ###########################
 Amazon SageMaker Python SDK
 ###########################

doc/v2.rst

@@ -1,42 +1,29 @@
-##########################################################
-Use Version 2.0 of the SageMaker Python SDK (experimental)
-##########################################################
-
-Development on v2.0.0 of the SageMaker Python SDK is underway.
-For more info on our plans, see https://github.com/aws/sagemaker-python-sdk/issues/1459.
+###########################################
+Use Version 2.x of the SageMaker Python SDK
+###########################################
 
 .. contents::
     :local:
+    :depth: 2
 
 ************
 Installation
 ************
 
-.. warning::
-    Version 2.0.0 is currently experimental, so proceed with caution. If you do run into issues or have any other feedback,
-    please let us know by `opening an issue <https://github.com/aws/sagemaker-python-sdk/issues/new/choose>`_
-    or `commenting on our planning issue <https://github.com/aws/sagemaker-python-sdk/issues/1459>`_.
-
-To install the latest release candidate:
-
-.. code:: bash
-
-    pip install git+git://github.com/aws/[email protected]
-
-To install the latest version of v2:
+To install the latest version:
 
 .. code:: bash
 
-    pip install git+git://github.com/aws/sagemaker-python-sdk.git@zwei
+    pip install --upgrade sagemaker
 
 If you are executing this pip install command in a notebook, make sure to restart your kernel.
 
-*******
-Changes
-*******
+****************
+Breaking Changes
+****************
 
 This section is for major changes that may require updates to your SageMaker Python SDK code.
-You can also see what changes have been made in the `CHANGELOG <https://github.com/aws/sagemaker-python-sdk/blob/zwei/CHANGELOG.md>`_.
+For the full list of changes, see the `CHANGELOG <https://github.com/aws/sagemaker-python-sdk/blob/master/CHANGELOG.md>`_.
 
 Deprecations
 ============
@@ -134,8 +121,46 @@ write MIME types as a string,
 | ``CONTENT_TYPE_NPY``          | ``"application/x-npy"``        |
 +-------------------------------+--------------------------------+
 
+Image URI Functions (e.g. ``get_image_uri``)
+--------------------------------------------
+
+The following functions have been deprecated in favor of :func:`sagemaker.image_uris.retrieve`:
+
+- ``sagemaker.amazon_estimator.get_image_uri()``
+- ``sagemaker.fw_utils.create_image_uri()``
+- ``sagemaker.fw_registry.registry()``
+- ``sagemaker.utils.get_ecr_image_uri_prefix()``
+
+For more information about usage, see :func:`sagemaker.image_uris.retrieve`.
+
+SageMaker Python SDK CLI
+------------------------
+
+The SageMaker Python SDK CLI has been deprecated.
+(This is different from the AWS CLI.)
+
+``enable_cloudwatch_metrics`` for Estimators and Models
+-------------------------------------------------------
+
+The parameter ``enable_cloudwatch_metrics`` has been deprecated.
+CloudWatch metrics are already emitted for all Training Jobs, etc.
+
+``sagemaker.fw_utils.parse_s3_url``
+-----------------------------------
+
+The ``sagemaker.fw_utils.parse_s3_url`` function has been deprecated.
+Please use :func:`sagemaker.s3.parse_s3_url` instead.
+
+``sagemaker.session.ModelContainer``
+------------------------------------
+
+The class ``sagemaker.session.ModelContainer`` has been deprecated, as it is not needed for creating inference pipelines.
+
+Changes in Default Behavior
+===========================
+
 Require ``framework_version`` and ``py_version`` for Frameworks
-===============================================================
+---------------------------------------------------------------
 
 Framework estimator and model classes now require ``framework_version`` and ``py_version`` instead of supplying defaults,
 unless an image URI is explicitly supplied.
@@ -165,6 +190,19 @@ For example:
         py_version="py3",  # now required
     )
 
+Log Display Behavior with ``attach()``
+--------------------------------------
+
+Logs are no longer printed when using ``attach()`` with an estimator.
+To view logs after attaching a training job to an estimator, use :func:`sagemaker.estimator.EstimatorBase.logs`.
+
+``HyperparameterTuner.fit()`` and ``Transformer.transform()``
+-------------------------------------------------------------
+
+:func:`sagemaker.tuner.HyperparameterTuner.fit` and :func:`sagemaker.transformer.Transformer.transform` now wait
+until the completion of the Hyperparameter Tuning Job or Batch Transform Job, respectively.
+To make the function non-blocking, use ``wait=False``.
+
 Parameter and Class Name Changes
 ================================
 
@@ -288,6 +326,19 @@ TensorFlow Serving Predictor
 (For the previous implementation of that class, see `Deprecate Legacy TensorFlow <#deprecate-legacy-tensorflow>`_).
 
 
+Inputs
+------
+
+``s3_input``
+~~~~~~~~~~~~
+
+``sagemaker.session.s3_input`` has been renamed to :class:`sagemaker.inputs.TrainingInput`.
+
+``ShuffleConfig``
+~~~~~~~~~~~~~~~~~
+
+``sagemaker.session.ShuffleConfig`` has been renamed to :class:`sagemaker.inputs.ShuffleConfig`.
+
 Airflow
 -------
 

doc/workflows/kubernetes/amazon_sagemaker_components_for_kubeflow_pipelines.rst

@@ -89,6 +89,10 @@ Pipelines workflow. For more information, see \ `SageMaker
 hyperparameter optimization Kubeflow Pipeline
 component <https://github.com/kubeflow/pipelines/tree/master/components/aws/sagemaker/hyperparameter_tuning>`__.
 
+**Processing**
+
+The Processing component enables you to submit processing jobs to Amazon SageMaker directly from a Kubeflow Pipelines workflow. For more information, see \ `SageMaker Processing Kubeflow Pipeline component <https://github.com/kubeflow/pipelines/tree/master/components/aws/sagemaker/process>`__.
+
 Inference components
 ^^^^^^^^^^^^^^^^^^^^
 

setup.py

@@ -33,7 +33,7 @@ def read_version():
 
 # Declare minimal set for installation
 required_packages = [
-    "boto3>=1.13.24",
+    "boto3>=1.14.12",
     "google-pasta",
     "numpy>=1.9.0",
     "protobuf>=3.1",