Merge branch 'master' into model-dir-slashes

Author: laurenyu

CHANGELOG.md

@@ -4,7 +4,90 @@
 
 ### Breaking Changes
 
- * merge v2 changes into master
+ * rename s3_input to TrainingInput
+ * Move _NumpyDeserializer to sagemaker.deserializers.NumpyDeserializer
+ * rename numpy_to_record_serializer to RecordSerializer
+ * Move _CsvDeserializer to sagemaker.deserializers and rename to CSVDeserializer
+ * Move _JsonSerializer to sagemaker.serializers.JSONSerializer
+ * Move _NPYSerializer to sagemaker.serializers and rename to NumpySerializer
+ * Move _JsonDeserializer to sagemaker.deserializers.JSONDeserializer
+ * Move _CsvSerializer to sagemaker.serializers.CSVSerializer
+ * preserve script path when S3 source_dir is provided
+ * use image_uris.retrieve() for XGBoost URIs
+ * deprecate sagemaker.amazon.amazon_estimator.get_image_uri()
+ * deprecate fw_registry module and use image_uris.retrieve() for SparkML
+ * deprecate Python SDK CLI
+ * Remove the content_types module
+ * deprecate unused parameters
+ * deprecate fw_utils.create_image_uri()
+ * use images_uris.retrieve() for Debugger
+ * deprecate fw_utils.parse_s3_url in favor of s3.parse_s3_url
+ * deprecate unused functions from utils and fw_utils
+ * Remove content_type and accept parameters from Predictor
+ * Add parameters to deploy and remove parameters from create_model
+ * Add LibSVM serializer for XGBoost predictor
+ * move ShuffleConfig from sagemaker.session to sagemaker.inputs
+ * deprecate get_ecr_image_uri_prefix
+ * rename estimator.train_image() to estimator.training_image_uri()
+ * deprecate is_version_equal_or_higher and is_version_equal_or_lower
+ * default wait=True for HyperparameterTuner.fit() and Transformer.transform()
+ * remove unused bin/sagemaker-submit file
+
+### Features
+
+ * start new module for retrieving prebuilt SageMaker image URIs
+ * handle separate training/inference images and EI in image_uris.retrieve
+ * add support for Amazon algorithms in image_uris.retrieve()
+ * Add pandas deserializer
+ * Remove LegacySerializer and LegacyDeserializer
+ * Add sparse matrix serializer
+ * Add v2 SerDe compatability
+ * Add JSON Lines serializer
+ * add framework upgrade tool
+ * add 1p algorithm image_uris migration tool
+ * Update migration tool to support breaking changes to create_model
+ * support PyTorch 1.6 training
+
+### Bug Fixes and Other Changes
+
+ * handle named variables in v2 migration tool
+ * add modifier for s3_input class
+ * add XGBoost support to image_uris.retrieve()
+ * add MXNet configuration to image_uris.retrieve()
+ * add remaining Amazon algorithms for image_uris.retrieve()
+ * add PyTorch configuration for image_uris.retrieve()
+ * make image_scope optional for some images in image_uris.retrieve()
+ * separate logs() from attach()
+ * use image_uris.retrieve instead of fw_utils.create_image_uri for DLC frameworks
+ * use images_uris.retrieve() for scikit-learn classes
+ * use image_uris.retrieve() for RL images
+ * Rename BaseDeserializer.deserialize data parameter
+ * Add allow_pickle parameter to NumpyDeserializer
+ * Fix scipy.sparse imports
+ * Improve code style of SerDe compatibility
+ * use image_uris.retrieve for Neo and Inferentia images
+ * use generated RL version fixtures and update Ray version
+ * use image_uris.retrieve() for ModelMonitor default image
+ * use _framework_name for 'protected' attribute
+ * Fix JSONLinesDeserializer
+ * upgrade TFS version and fix py_versions KeyError
+ * Fix PandasDeserializer tests to more accurately mock response
+ * don't require instance_type for image_uris.retrieve() if only one option
+ * ignore code cells with shell commands in v2 migration tool
+ * Support multiple Accept types
+
+### Documentation Changes
+
+ * fix pip install command
+ * document name changes for TFS classes
+ * document v2.0.0 changes
+ * update KFP full pipeline
+
+### Testing and Release Infrastructure
+
+ * generate Chainer latest version fixtures from config
+ * use generated TensorFlow version fixtures
+ * use generated MXNet version fixtures
 
 ## v1.72.0 (2020-07-29)
 

README.rst

@@ -1,8 +1,3 @@
-+-------------------------------------------------------------------------------------------------+
-| **NOTE**: We are working on v2.0.0. See https://github.com/aws/sagemaker-python-sdk/issues/1459 |
-| for more info on our plans and to leave feedback!                                               |
-+-------------------------------------------------------------------------------------------------+
-
 .. image:: https://github.com/aws/sagemaker-python-sdk/raw/master/branding/icon/sagemaker-banner.png
     :height: 100px
     :alt: SageMaker

src/sagemaker/image_uri_config/coach-tensorflow.json

@@ -157,11 +157,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/ray-pytorch.json

@@ -12,11 +12,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/ray-tensorflow.json

@@ -128,11 +128,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",
@@ -151,11 +148,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

src/sagemaker/image_uri_config/vw.json

@@ -11,11 +11,8 @@
                 "ap-southeast-2": "462105765813",
                 "ca-central-1": "462105765813",
                 "eu-central-1": "462105765813",
-                "eu-north-1": "462105765813",
                 "eu-west-1": "462105765813",
                 "eu-west-2": "462105765813",
-                "eu-west-3": "462105765813",
-                "sa-east-1": "462105765813",
                 "us-east-1": "462105765813",
                 "us-east-2": "462105765813",
                 "us-west-1": "462105765813",

tests/integ/__init__.py

@@ -72,11 +72,28 @@
     "us-west-2",
 ]
 
+RL_SUPPORTED_REGIONS = (
+    "ap-northeast-1",
+    "ap-northeast-2",
+    "ap-south-1",
+    "ap-southeast-1",
+    "ap-southeast-2",
+    "ca-central-1",
+    "eu-central-1",
+    "eu-west-1",
+    "eu-west-2",
+    "us-east-1",
+    "us-east-2",
+    "us-west-1",
+    "us-west-2",
+)
+
 NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1", "me-south-1"]
 NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1", "me-south-1"]
 NO_AUTO_ML_REGIONS = ["sa-east-1", "me-south-1", "ap-east-1", "eu-west-3"]
 NO_MODEL_MONITORING_REGIONS = ["me-south-1"]
 
+
 EFS_TEST_ENABLED_REGION = []
 
 logging.getLogger("boto3").setLevel(logging.INFO)

tests/integ/test_rl.py

@@ -19,7 +19,7 @@
 
 from sagemaker.rl import RLEstimator, RLFramework, RLToolkit
 from sagemaker.utils import sagemaker_timestamp, unique_name_from_base
-from tests.integ import DATA_DIR
+from tests.integ import DATA_DIR, RL_SUPPORTED_REGIONS, test_region
 from tests.integ.timeout import timeout, timeout_and_delete_endpoint_by_name
 
 
@@ -51,6 +51,10 @@ def test_coach_mxnet(sagemaker_session, coach_mxnet_latest_version, cpu_instance
     assert 0 < action[0][1] < 1
 
 
+@pytest.mark.skipif(
+    test_region() not in RL_SUPPORTED_REGIONS,
+    reason="Updated RL images aren't in {}".format(test_region()),
+)
 def test_coach_tf(sagemaker_session, coach_tensorflow_latest_version, cpu_instance_type):
     estimator = _test_coach(
         sagemaker_session,
@@ -98,6 +102,10 @@ def _test_coach(sagemaker_session, rl_framework, rl_coach_version, cpu_instance_
     )
 
 
+@pytest.mark.skipif(
+    test_region() not in RL_SUPPORTED_REGIONS,
+    reason="Updated RL images aren't in {}".format(test_region()),
+)
 @pytest.mark.canary_quick
 def test_ray_tf(sagemaker_session, ray_tensorflow_latest_version, cpu_instance_type):
     source_dir = os.path.join(DATA_DIR, "ray_cartpole")

tests/scripts/run-notebook-test.sh

@@ -22,8 +22,6 @@ echo "set SAGEMAKER_ROLE_ARN=$SAGEMAKER_ROLE_ARN"
 ./amazon-sagemaker-examples/advanced_functionality/kmeans_bring_your_own_model/kmeans_bring_your_own_model.ipynb \
 ./amazon-sagemaker-examples/advanced_functionality/tensorflow_iris_byom/tensorflow_BYOM_iris.ipynb \
 ./amazon-sagemaker-examples/sagemaker-python-sdk/1P_kmeans_highlevel/kmeans_mnist.ipynb \
-./amazon-sagemaker-examples/sagemaker-python-sdk/managed_spot_training_mxnet/managed_spot_training_mxnet.ipynb \
-./amazon-sagemaker-examples/sagemaker-python-sdk/managed_spot_training_tensorflow_estimator/managed_spot_training_tensorflow_estimator.ipynb \
 ./amazon-sagemaker-examples/sagemaker-python-sdk/mxnet_onnx_eia/mxnet_onnx_eia.ipynb \
 ./amazon-sagemaker-examples/sagemaker-python-sdk/mxnet_onnx_export/mxnet_onnx_export.ipynb \
 ./amazon-sagemaker-examples/sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb \