Merge branch 'master' into network-config-keyword

Author: ajaykarpur

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## v2.14.0 (2020-10-05)
+
+### Features
+
+ * upgrade Neo MxNet to 1.7
+
+### Bug Fixes and Other Changes
+
+ * add a condition to retrieve correct image URI for xgboost
+
+## v2.13.0 (2020-09-30)
+
+### Features
+
+ * add xgboost framework version 1.2-1
+
+### Bug Fixes and Other Changes
+
+ * revert "feature: upgrade Neo MxNet to 1.7 (#1928)"
+
+## v2.12.0 (2020-09-29)
+
+### Features
+
+ * upgrade Neo MxNet to 1.7
+
+## v2.11.0 (2020-09-28)
+
+### Features
+
+ * Add SDK support for SparkML Serving Container version 2.4
+
+### Bug Fixes and Other Changes
+
+ * pin pytest version <6.1.0 to avoid pytest-rerunfailures breaking changes
+ * temporarily skip the MxNet Neo test until we fix them
+
+### Documentation Changes
+
+ * fix conda setup for docs
+
 ## v2.10.0 (2020-09-23)
 
 ### Features

README.rst

@@ -164,7 +164,7 @@ Setup a Python environment, and install the dependencies listed in ``doc/require
     # conda
     conda create -n sagemaker python=3.7
     conda activate sagemaker
-    conda install --file doc/requirements.txt
+    conda install sphinx=3.1.1 sphinx_rtd_theme=0.5.0
 
     # pip
     pip install -r doc/requirements.txt
@@ -201,7 +201,7 @@ In order to host a SparkML model in SageMaker, it should be serialized with ``ML
 
 For more information on MLeap, see https://github.com/combust/mleap .
 
-Supported major version of Spark: 2.2 (MLeap version - 0.9.6)
+Supported major version of Spark: 2.4 (MLeap version - 0.9.6)
 
 Here is an example on how to create an instance of  ``SparkMLModel`` class and use ``deploy()`` method to create an
 endpoint which can be used to perform prediction against your trained SparkML Model.

VERSION

@@ -1 +1 @@
-2.10.1.dev0
+2.14.1.dev0

doc/amazon_sagemaker_processing.rst

@@ -69,15 +69,15 @@ For an in-depth look, please see the `Scikit-learn Data Processing and Model Eva
 
 Data Processing with Spark
 ============================================
-SageMaker provides two classes for customers to run Spark applications: :class:`sagemaker.processing.PySparkProcessor` and :class:`sagemaker.processing.SparkJarProcessor`
+SageMaker provides two classes for customers to run Spark applications: :class:`sagemaker.spark.processing.PySparkProcessor` and :class:`sagemaker.spark.processing.SparkJarProcessor`
 
 
 PySparkProcessor
 ---------------------
 
-You can use the :class:`sagemaker.processing.PySparkProcessor` class to run PySpark scripts as processing jobs.
+You can use the :class:`sagemaker.spark.processing.PySparkProcessor` class to run PySpark scripts as processing jobs.
 
-This example shows how you can take an existing PySpark script and run a processing job with the :class:`sagemaker.processing.PySparkProcessor` class and the pre-built SageMaker Spark container.
+This example shows how you can take an existing PySpark script and run a processing job with the :class:`sagemaker.spark.processing.PySparkProcessor` class and the pre-built SageMaker Spark container.
 
 First you need to create a :class:`PySparkProcessor` object
 
@@ -230,8 +230,8 @@ Processing class documentation
 - :class:`sagemaker.processing.Processor`
 - :class:`sagemaker.processing.ScriptProcessor`
 - :class:`sagemaker.sklearn.processing.SKLearnProcessor`
-- :class:`sagemaker.sklearn.processing.PySparkProcessor`
-- :class:`sagemaker.sklearn.processing.SparkJarProcessor`
+- :class:`sagemaker.spark.processing.PySparkProcessor`
+- :class:`sagemaker.spark.processing.SparkJarProcessor`
 - :class:`sagemaker.processing.ProcessingInput`
 - :class:`sagemaker.processing.ProcessingOutput`
 - :class:`sagemaker.processing.ProcessingJob`

doc/api/training/processing.rst

@@ -5,3 +5,8 @@ Processing
     :members:
     :undoc-members:
     :show-inheritance:
+
+.. automodule:: sagemaker.spark.processing
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/overview.rst

@@ -672,7 +672,7 @@ You can install all necessary for this feature dependencies using pip:
 For more detailed examples of running hyperparameter tuning jobs, see:
 
 - `Using the TensorFlow estimator with hyperparameter tuning <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/tensorflow_mnist/hpo_tensorflow_mnist.ipynb>`__
-- `Bringing your own estimator for hyperparameter tuning <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/r_bring_your_own/hpo_r_bring_your_own.ipynb>`__
+- `Bringing your own estimator for hyperparameter tuning <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/r_bring_your_own/tune_r_bring_your_own.ipynb>`__
 - `Analyzing results <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/analyze_results/HPO_Analyze_TuningJob_Results.ipynb>`__
 
 You can also find these notebooks in the **Hyperprameter Tuning** section of the **SageMaker Examples** section in a notebook instance.

doc/v2.rst

@@ -231,7 +231,7 @@ The following estimator parameters have been renamed:
 +------------------------------+------------------------+
 | ``train_use_spot_instances`` | ``use_spot_instances`` |
 +------------------------------+------------------------+
-| ``train_max_run_wait``       | ``max_wait``           |
+| ``train_max_wait``           | ``max_wait``           |
 +------------------------------+------------------------+
 | ``train_volume_size``        | ``volume_size``        |
 +------------------------------+------------------------+

setup.py

@@ -61,7 +61,7 @@ def read_version():
         extras["all"],
         "tox",
         "flake8",
-        "pytest",
+        "pytest<6.1.0",
         "pytest-cov",
         "pytest-rerunfailures",
         "pytest-xdist",

src/sagemaker/algorithm.py

@@ -17,6 +17,7 @@
 import sagemaker.parameter
 from sagemaker import vpc_utils
 from sagemaker.deserializers import BytesDeserializer
+from sagemaker.deprecations import removed_kwargs
 from sagemaker.estimator import EstimatorBase
 from sagemaker.serializers import IdentitySerializer
 from sagemaker.transformer import Transformer
@@ -291,6 +292,9 @@ def create_model(
         Returns:
             a Model ready for deployment.
         """
+        removed_kwargs("content_type", kwargs)
+        removed_kwargs("accept", kwargs)
+
         if predictor_cls is None:
 
             def predict_wrapper(endpoint, session):

src/sagemaker/amazon/amazon_estimator.py

@@ -23,6 +23,7 @@
 from sagemaker.amazon import validation
 from sagemaker.amazon.hyperparameter import Hyperparameter as hp  # noqa
 from sagemaker.amazon.common import write_numpy_to_dense_tensor
+from sagemaker.deprecations import renamed_warning
 from sagemaker.estimator import EstimatorBase, _TrainingJob
 from sagemaker.inputs import FileSystemInput, TrainingInput
 from sagemaker.utils import sagemaker_timestamp
@@ -454,3 +455,22 @@ def upload_numpy_to_s3_shards(
                 s3.Object(bucket, key_prefix + file).delete()
         finally:
             raise ex
+
+
+def get_image_uri(region_name, repo_name, repo_version=1):
+    """Deprecated method. Please use sagemaker.image_uris.retrieve().
+
+    Args:
+        region_name: name of the region
+        repo_name: name of the repo (e.g. xgboost)
+        repo_version: version of the repo
+
+    Returns:
+        the image uri
+    """
+    renamed_warning("The method get_image_uri")
+    return image_uris.retrieve(
+        framework=repo_name,
+        region=region_name,
+        version=repo_version,
+    )

src/sagemaker/amazon/common.py

@@ -21,6 +21,7 @@
 import numpy as np
 
 from sagemaker.amazon.record_pb2 import Record
+from sagemaker.deprecations import deprecated_class
 from sagemaker.deserializers import BaseDeserializer
 from sagemaker.serializers import BaseSerializer
 from sagemaker.utils import DeferredError
@@ -298,3 +299,7 @@ def _resolve_type(dtype):
     if dtype == np.dtype("float32"):
         return "Float32"
     raise ValueError("Unsupported dtype {} on array".format(dtype))
+
+
+numpy_to_record_serializer = deprecated_class(RecordSerializer, "numpy_to_record_serializer")
+record_deserializer = deprecated_class(RecordDeserializer, "record_deserializer")

src/sagemaker/cli/compatibility/v2/modifiers/training_params.py

@@ -45,7 +45,7 @@
     "train_instance_count",
     "train_instance_type",
     "train_max_run",
-    "train_max_run_wait",
+    "train_max_wait",
     "train_use_spot_instances",
     "train_volume_size",
     "train_volume_kms_key",
@@ -63,7 +63,7 @@ def node_should_be_modified(self, node):
         - ``train_instance_count``
         - ``train_instance_type``
         - ``train_max_run``
-        - ``train_max_run_wait``
+        - ``train_max_wait``
         - ``train_use_spot_instances``
         - ``train_volume_kms_key``
         - ``train_volume_size``

src/sagemaker/content_types.py

@@ -0,0 +1,23 @@
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""Deprecated content type constants. Just use the mime type strings."""
+from __future__ import absolute_import
+
+import deprecations
+
+deprecations.removed_warning("The sagemaker.content_types module")
+
+CONTENT_TYPE_JSON = "application/json"
+CONTENT_TYPE_CSV = "text/csv"
+CONTENT_TYPE_OCTET_STREAM = "application/octet-stream"
+CONTENT_TYPE_NPY = "application/x-npy"