Merge branch 'master' into master

Author: aaronmarkham

.flake8

@@ -1,3 +1,5 @@
 [flake8]
 application_import_names = sagemaker, tests
 import-order-style = google
+per-file-ignores =
+    tests/unit/test_tuner.py: F405

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,13 +2,23 @@
 
 *Description of changes:*
 
+*Testing done:*
+
 ## Merge Checklist
 
 _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request._
 
+#### General
+
 - [ ] I have read the [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md) doc
 - [ ] I used the commit message format described in [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md#committing-your-change)
+- [ ] I have used the regional endpoint when creating S3 and/or STS clients (if appropriate)
+- [ ] I have updated any necessary documentation, including [READMEs](https://github.com/aws/sagemaker-python-sdk/blob/master/README.rst) and [API docs](https://github.com/aws/sagemaker-python-sdk/tree/master/doc) (if appropriate)
+
+#### Tests
+
 - [ ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
-- [ ] I have updated any necessary [documentation](https://github.com/aws/sagemaker-python-sdk/blob/master/README.rst) (if appropriate)
+- [ ] I have checked that my tests are not configured for a specific region or account (if appropriate)
+- [ ] I have used [`unique_name_from_base`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/utils.py#L77) to create resource names in integ tests (if appropriate)
 
 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

CHANGELOG.md

@@ -1,5 +1,263 @@
 # Changelog
 
+## v1.50.3 (2020-01-15)
+
+### Bug fixes and other changes
+
+ * ignore private Automatic Model Tuning hyperparameter when attaching AlgorithmEstimator
+
+### Documentation changes
+
+ * add Debugger API docs
+
+## v1.50.2 (2020-01-14)
+
+### Bug fixes and other changes
+
+ * add tests to quick canary
+ * honor 'wait' flag when updating endpoint
+ * add default framework version warning message in Model classes
+ * Adding role arn explanation for sagemaker role
+ * allow predictor to be returned from AutoML.deploy()
+ * add PR checklist item about unique_name_from_base()
+ * use unique_name_from_base for multi-algo tuning test
+ * update copyright year in license header
+
+### Documentation changes
+
+ * add version requirement for using "requirement.txt" when serving a PyTorch model
+ * add SageMaker Debugger overview
+ * clarify requirements.txt usage for Chainer, MXNet, and Scikit-learn
+ * change "associate" to "create" for OpenID connector
+ * fix typo and improve clarity on installing packages via "requirements.txt"
+
+## v1.50.1 (2020-01-07)
+
+### Bug fixes and other changes
+
+ * fix PyTorchModel deployment crash on Windows
+ * make PyTorch empty framework_version warning include the latest PyTorch version
+
+## v1.50.0 (2020-01-06)
+
+### Features
+
+ * allow disabling debugger_hook_config
+
+### Bug fixes and other changes
+
+ * relax urllib3 and requests restrictions.
+ * Add uri as return statement for upload_string_as_file_body
+ * refactor logic in fw_utils and fill in docstrings
+ * increase poll from 5 to 30 for DescribeEndpoint lambda.
+ * fix test_auto_ml tests for regions without ml.c4.xlarge hosts.
+ * fix test_processing for regions without m4.xlarge instances.
+ * reduce test's describe frequency to eliminate throttling error.
+ * Increase number of retries when describing an endpoint since tf-2.0 has larger images and takes longer to start.
+
+### Documentation changes
+
+ * generalize Model Monitor documentation from SageMaker Studio tutorial
+
+## v1.49.0 (2019-12-23)
+
+### Features
+
+ * Add support for TF-2.0.0.
+ * create ProcessingJob from ARN and from name
+
+### Bug fixes and other changes
+
+ * Make tf tests tf-1.15 and tf-2.0 compatible.
+
+### Documentation changes
+
+ * add Model Monitor documentation
+ * add link to Amazon algorithm estimator parent class to clarify **kwargs
+
+## v1.48.1 (2019-12-18)
+
+### Bug fixes and other changes
+
+ * use name_from_base in auto_ml.py but unique_name_from_base in tests.
+ * make test's custom bucket include region and account name.
+ * add Keras to the list of Neo-supported frameworks
+
+### Documentation changes
+
+ * add link to parent classes to clarify **kwargs
+ * add link to framework-related parent classes to clarify **kwargs
+
+## v1.48.0 (2019-12-17)
+
+### Features
+
+ * allow setting the default bucket in Session
+
+### Bug fixes and other changes
+
+ * set integration test parallelization to 512
+ * shorten base job name to avoid collision
+ * multi model integration test to create ECR repo with unique names to allow independent parallel executions
+
+## v1.47.1 (2019-12-16)
+
+### Bug fixes and other changes
+
+ * Revert "feature: allow setting the default bucket in Session (#1168)"
+
+### Documentation changes
+
+ * add AutoML README
+ * add missing classes to API docs
+
+## v1.47.0 (2019-12-13)
+
+### Features
+
+ * allow setting the default bucket in Session
+
+### Bug fixes and other changes
+
+ * allow processing users to run code in s3
+
+## v1.46.0 (2019-12-12)
+
+### Features
+
+ * support Multi-Model endpoints
+
+### Bug fixes and other changes
+
+ * update PR template with items about tests, regional endpoints, and API docs
+
+## v1.45.2 (2019-12-10)
+
+### Bug fixes and other changes
+
+ * modify schedule cleanup to abide by latest validations
+ * lower log level when getting execution role from a SageMaker Notebook
+ * Fix "ValueError: too many values to unpack (expected 2)" is occurred in windows local mode
+ * allow ModelMonitor and Processor to take IAM role names (in addition to ARNs)
+
+### Documentation changes
+
+ * mention that the entry_point needs to be named inference.py for tfs
+
+## v1.45.1 (2019-12-06)
+
+### Bug fixes and other changes
+
+ * create auto ml job for tests that based on existing job
+ * fixing py2 support for latest TF version
+ * fix tags in deploy call for generic estimators
+ * make multi algo integration test assertion less specific
+
+## v1.45.0 (2019-12-04)
+
+### Features
+
+ * add support for TF 1.15.0, PyTorch 1.3.1 and MXNet 1.6rc0.
+ * add S3Downloader.list(s3_uri) functionality
+ * introduce SageMaker AutoML
+ * wrap up Processing feature
+ * add a few minor features to Model Monitoring
+ * add enable_sagemaker_metrics flag
+ * Amazon SageMaker Model Monitoring
+ * add utils.generate_tensorboard_url function
+ * Add jobs list to Estimator
+
+### Bug fixes and other changes
+
+ * remove unnecessary boto model files
+ * update boto version to >=1.10.32
+ * correct Debugger tests
+ * fix bug in monitor.attach() for empty network_config
+ * Import smdebug_rulesconfig from PyPI
+ * bump the version to 1.45.0 (publishes 1.46.0) for re:Invent-2019
+ * correct AutoML imports and expose current_job_name
+ * correct Model Monitor eu-west-3 image name.
+ * use DLC prod images
+ * remove unused env variable for Model Monitoring
+ * aws model update
+ * rename get_debugger_artifacts to latest_job_debugger_artifacts
+ * remove retain flag from update_endpoint
+ * correct S3Downloader behavior
+ * consume smdebug_ruleconfig .whl for ITs
+ * disable DebuggerHook and Rules for TF distributions
+ * incorporate smdebug_ruleconfigs pkg until availability in PyPI
+ * remove pre/post scripts per latest validations
+ * update rules_config .whl
+ * remove py_version from SKLearnProcessor
+ * AutoML improvements
+ * stop overwriting custom rules volume and type
+ * fix tests due to latest server-side validations
+ * Minor processing changes
+ * minor processing changes (instance_count + docs)
+ * update api to latest
+ * Eureka master
+ * Add support for xgboost version 0.90-2
+ * SageMaker Debugger revision
+ * Add support for SageMaker Debugger [WIP]
+ * Fix linear learner crash when num_class is string and predict type is `multiclass_classifier`
+ * Additional Processing Jobs integration tests
+ * Migrate to updated Processing Jobs API
+ * Processing Jobs revision round 2
+ * Processing Jobs revision
+ * remove instance_pools parameter from tuner
+ * Multi-Algorithm Hyperparameter Tuning Support
+ * Import Processors in init files
+ * Remove SparkML Processors and corresponding unit tests
+ * Processing Jobs Python SDK support
+
+## v1.44.4 (2019-12-02)
+
+### Bug fixes and other changes
+
+ * Documentation for Amazon Sagemaker Operators
+
+## v1.44.3 (2019-11-26)
+
+### Bug fixes and other changes
+
+ * move sagemaker config loading to LocalSession since it is only used for local code support.
+
+### Documentation changes
+
+ * fix docstring wording.
+
+## v1.44.2 (2019-11-25)
+
+### Bug fixes and other changes
+
+ * add pyyaml dependencies to the required list.
+
+### Documentation changes
+
+ * Correct info on code_location parameter
+
+## v1.44.1 (2019-11-21)
+
+### Bug fixes and other changes
+
+ * Remove local mode dependencies from required.
+
+## v1.44.0 (2019-11-21)
+
+### Features
+
+ * separating sagemaker dependencies into more use case specific installable components.
+
+### Bug fixes and other changes
+
+ * remove docker-compose as a required dependency.
+
+## v1.43.5 (2019-11-18)
+
+### Bug fixes and other changes
+
+ * remove red from possible colors when streaming logs
+
 ## v1.43.4.post1 (2019-10-29)
 
 ### Documentation changes

README.rst

@@ -61,7 +61,11 @@ Table of Contents
 17. `Secure Training and Inference with VPC <https://sagemaker.readthedocs.io/en/stable/overview.html#secure-training-and-inference-with-vpc>`__
 18. `BYO Model <https://sagemaker.readthedocs.io/en/stable/overview.html#byo-model>`__
 19. `Inference Pipelines <https://sagemaker.readthedocs.io/en/stable/overview.html#inference-pipelines>`__
-20. `SageMaker Workflow <#sagemaker-workflow>`__
+20. `Amazon SageMaker Operators for Kubernetes <#amazon-sagemaker-operators-for-kubernetes>`__
+21. `SageMaker Workflow <#sagemaker-workflow>`__
+22. `SageMaker Autopilot <#sagemaker-autopilot>`__
+23. `Model Monitoring <#amazon-sagemaker-model-monitoring>`__
+24. `SageMaker Debugger <#amazon-sagemaker-debugger>`__
 
 
 Installing the SageMaker Python SDK
@@ -190,7 +194,7 @@ TensorFlow SageMaker Estimators
 
 By using TensorFlow SageMaker Estimators, you can train and host TensorFlow models on Amazon SageMaker.
 
-Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14``.
+Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.``, ``1.15.0``, ``2.0.0``.
 
 Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14``.
 
@@ -222,7 +226,7 @@ PyTorch SageMaker Estimators
 
 With PyTorch SageMaker Estimators, you can train and host PyTorch models on Amazon SageMaker.
 
-Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``.
+Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``.
 
 We recommend that you use the latest supported version, because that's where we focus most of our development efforts.
 
@@ -326,6 +330,15 @@ For more information, see `AWS SageMaker Estimators and Models`_.
 
 .. _AWS SageMaker Estimators and Models: src/sagemaker/amazon/README.rst
 
+Amazon SageMaker Operators for Kubernetes
+-----------------------------------------
+
+You can use Amazon SageMaker Operators for Kubernetes to optimize hyperparameters for a given model, run batch transform jobs over existing models, and set up inference endpoints.
+
+For more information, see `Amazon SageMaker Operators for Kubernetes`_.
+
+.. _Amazon SageMaker Operators for Kubernetes: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_operators_for_kubernetes.html
+
 SageMaker Workflow
 ------------------
 
@@ -334,3 +347,33 @@ You can use Apache Airflow to author, schedule and monitor SageMaker workflow.
 For more information, see `SageMaker Workflow in Apache Airflow`_.
 
 .. _SageMaker Workflow in Apache Airflow: https://sagemaker.readthedocs.io/en/stable/using_workflow.html
+
+SageMaker Autopilot
+-------------------
+
+Amazon SageMaker Autopilot is an automated machine learning solution (commonly referred to as "AutoML") for tabular
+datasets. It automatically trains and tunes the best machine learning models for classification or regression based
+on your data, and hosts a series of models on an Inference Pipeline.
+
+For more information about SageMaker Autopilot, see `SageMaker Autopilot`_.
+
+.. _SageMaker Autopilot: src/sagemaker/automl/README.rst
+
+Amazon SageMaker Model Monitoring
+---------------------------------
+
+You can use Amazon SageMaker Model Monitoring to automatically detect concept drift by monitoring your machine learning models.
+
+For more information, see `Amazon SageMaker Model Monitoring`_.
+
+.. _Amazon SageMaker Model Monitoring: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html
+
+Amazon SageMaker Debugger
+-------------------------
+
+You can use Amazon SageMaker Debugger to automatically detect anomalies while training your machine learning models.
+
+For more information, see `Amazon SageMaker Debugger`_.
+
+.. _Amazon SageMaker Debugger: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_debugger.html
+

VERSION

@@ -1 +1 @@
-1.43.5.dev0
+1.50.4.dev0

buildspec.yml

@@ -1,6 +1,10 @@
 version: 0.2
 
 phases:
+  pre_build:
+    commands:
+      - start-dockerd
+
   build:
     commands:
       - IGNORE_COVERAGE=-
@@ -16,7 +20,7 @@ phases:
       - start_time=`date +%s`
       - |
         if has-matching-changes "tests/" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"; then
-          tox -e py36 -- tests/integ -m "not local_mode" -n 48  --reruns 3 --reruns-delay 5 --durations 50 --boto-config '{"region_name": "us-east-1"}'
+          tox -e py36 -- tests/integ -m "not local_mode" -n 512 --reruns 3 --reruns-delay 5 --durations 50 --boto-config '{"region_name": "us-east-2"}'
         fi
       - ./ci-scripts/displaytime.sh 'py36 tests/integ' $start_time