Merge branch 'master' into new-regions

Author: ajaykarpur

CHANGELOG.md

@@ -1,10 +1,108 @@
 # Changelog
 
+## v2.0.1 (2020-08-05)
+
+### Bug Fixes and Other Changes
+
+ * use pathlib.PurePosixPath for S3 URLs and Unix paths
+ * fix regions for updated RL images
+
+### Documentation Changes
+
+ * update CHANGELOG to reflect v2.0.0 changes
+
+### Testing and Release Infrastructure
+
+ * remove v2-incompatible notebooks from notebook build
+
 ## v2.0.0 (2020-08-04)
 
 ### Breaking Changes
 
- * merge v2 changes into master
+ * rename s3_input to TrainingInput
+ * Move _NumpyDeserializer to sagemaker.deserializers.NumpyDeserializer
+ * rename numpy_to_record_serializer to RecordSerializer
+ * Move _CsvDeserializer to sagemaker.deserializers and rename to CSVDeserializer
+ * Move _JsonSerializer to sagemaker.serializers.JSONSerializer
+ * Move _NPYSerializer to sagemaker.serializers and rename to NumpySerializer
+ * Move _JsonDeserializer to sagemaker.deserializers.JSONDeserializer
+ * Move _CsvSerializer to sagemaker.serializers.CSVSerializer
+ * preserve script path when S3 source_dir is provided
+ * use image_uris.retrieve() for XGBoost URIs
+ * deprecate sagemaker.amazon.amazon_estimator.get_image_uri()
+ * deprecate fw_registry module and use image_uris.retrieve() for SparkML
+ * deprecate Python SDK CLI
+ * Remove the content_types module
+ * deprecate unused parameters
+ * deprecate fw_utils.create_image_uri()
+ * use images_uris.retrieve() for Debugger
+ * deprecate fw_utils.parse_s3_url in favor of s3.parse_s3_url
+ * deprecate unused functions from utils and fw_utils
+ * Remove content_type and accept parameters from Predictor
+ * Add parameters to deploy and remove parameters from create_model
+ * Add LibSVM serializer for XGBoost predictor
+ * move ShuffleConfig from sagemaker.session to sagemaker.inputs
+ * deprecate get_ecr_image_uri_prefix
+ * rename estimator.train_image() to estimator.training_image_uri()
+ * deprecate is_version_equal_or_higher and is_version_equal_or_lower
+ * default wait=True for HyperparameterTuner.fit() and Transformer.transform()
+ * remove unused bin/sagemaker-submit file
+
+### Features
+
+ * start new module for retrieving prebuilt SageMaker image URIs
+ * handle separate training/inference images and EI in image_uris.retrieve
+ * add support for Amazon algorithms in image_uris.retrieve()
+ * Add pandas deserializer
+ * Remove LegacySerializer and LegacyDeserializer
+ * Add sparse matrix serializer
+ * Add v2 SerDe compatability
+ * Add JSON Lines serializer
+ * add framework upgrade tool
+ * add 1p algorithm image_uris migration tool
+ * Update migration tool to support breaking changes to create_model
+ * support PyTorch 1.6 training
+
+### Bug Fixes and Other Changes
+
+ * handle named variables in v2 migration tool
+ * add modifier for s3_input class
+ * add XGBoost support to image_uris.retrieve()
+ * add MXNet configuration to image_uris.retrieve()
+ * add remaining Amazon algorithms for image_uris.retrieve()
+ * add PyTorch configuration for image_uris.retrieve()
+ * make image_scope optional for some images in image_uris.retrieve()
+ * separate logs() from attach()
+ * use image_uris.retrieve instead of fw_utils.create_image_uri for DLC frameworks
+ * use images_uris.retrieve() for scikit-learn classes
+ * use image_uris.retrieve() for RL images
+ * Rename BaseDeserializer.deserialize data parameter
+ * Add allow_pickle parameter to NumpyDeserializer
+ * Fix scipy.sparse imports
+ * Improve code style of SerDe compatibility
+ * use image_uris.retrieve for Neo and Inferentia images
+ * use generated RL version fixtures and update Ray version
+ * use image_uris.retrieve() for ModelMonitor default image
+ * use _framework_name for 'protected' attribute
+ * Fix JSONLinesDeserializer
+ * upgrade TFS version and fix py_versions KeyError
+ * Fix PandasDeserializer tests to more accurately mock response
+ * don't require instance_type for image_uris.retrieve() if only one option
+ * ignore code cells with shell commands in v2 migration tool
+ * Support multiple Accept types
+
+### Documentation Changes
+
+ * fix pip install command
+ * document name changes for TFS classes
+ * document v2.0.0 changes
+ * update KFP full pipeline
+
+### Testing and Release Infrastructure
+
+ * generate Chainer latest version fixtures from config
+ * use generated TensorFlow version fixtures
+ * use generated MXNet version fixtures
 
 ## v1.72.0 (2020-07-29)
 

README.rst

@@ -1,8 +1,3 @@
-+-------------------------------------------------------------------------------------------------+
-| **NOTE**: We are working on v2.0.0. See https://github.com/aws/sagemaker-python-sdk/issues/1459 |
-| for more info on our plans and to leave feedback!                                               |
-+-------------------------------------------------------------------------------------------------+
-
 .. image:: https://github.com/aws/sagemaker-python-sdk/raw/master/branding/icon/sagemaker-banner.png
     :height: 100px
     :alt: SageMaker

VERSION

@@ -1 +1 @@
-2.0.1.dev0
+2.0.2.dev0

src/sagemaker/image_uri_config/coach-tensorflow.json

@@ -167,11 +167,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/ray-pytorch.json

@@ -12,11 +12,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/ray-tensorflow.json

@@ -136,11 +136,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",
@@ -159,11 +156,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/vw.json

@@ -11,11 +11,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/local/utils.py

@@ -19,6 +19,8 @@
 from distutils.dir_util import copy_tree
 from six.moves.urllib.parse import urlparse
 
+from sagemaker import s3
+
 
 def copy_directory_structure(destination_directory, relative_path):
     """Create all the intermediate directories required for relative_path to
@@ -62,8 +64,8 @@ def move_to_destination(source, destination, job_name, sagemaker_session):
         final_uri = destination
     elif parsed_uri.scheme == "s3":
         bucket = parsed_uri.netloc
-        path = _create_s3_prefix(parsed_uri.path, job_name)
-        final_uri = "s3://%s/%s" % (bucket, path)
+        path = s3.s3_path_join(parsed_uri.path, job_name)
+        final_uri = s3.s3_path_join("s3://", bucket, path)
         sagemaker_session.upload_data(source, bucket, path)
     else:
         raise ValueError("Invalid destination URI, must be s3:// or file://, got: %s" % destination)
@@ -72,22 +74,6 @@ def move_to_destination(source, destination, job_name, sagemaker_session):
     return final_uri
 
 
-def _create_s3_prefix(path, job_name):
-    """Constructs a path out of the given path and job name to be
-    used as an S3 prefix.
-
-    Args:
-        path (str): the original path. If the path is only ``"/"``,
-            then it is ignored.
-        job_name (str): the job name to be appended to the path.
-
-    Returns:
-        str: an S3 prefix of the form ``"path/job_name"``
-    """
-    path = path.strip("/")
-    return job_name if path == "" else "/".join((path, job_name))
-
-
 def recursive_copy(source, destination):
     """A wrapper around distutils.dir_util.copy_tree but won't throw any
     exception when the source directory does not exist.

src/sagemaker/model_monitor/data_capture_config.py

@@ -16,8 +16,7 @@
 """
 from __future__ import print_function, absolute_import
 
-import os
-
+from sagemaker import s3
 from sagemaker.session import Session
 
 _MODEL_MONITOR_S3_PATH = "model-monitor"
@@ -67,7 +66,7 @@ def __init__(
         self.destination_s3_uri = destination_s3_uri
         if self.destination_s3_uri is None:
             sagemaker_session = sagemaker_session or Session()
-            self.destination_s3_uri = os.path.join(
+            self.destination_s3_uri = s3.s3_path_join(
                 "s3://",
                 sagemaker_session.default_bucket(),
                 _MODEL_MONITOR_S3_PATH,