Merge branch 'master' into PipelineModel

Author: jesterhazy

.travis.yml

@@ -14,6 +14,8 @@ CHANGELOG
 * doc-fix: move overview content in main README into sphynx project
 * bug-fix: pass accelerator_type in ``deploy`` for REST API TFS ``Model``
 * feature: ``PipelineModel``: Create a Transformer from a PipelineModel
+* doc-fix: move content from tf/README.rst into sphynx project
+* doc-fix: Improve new developer experience in README
 
 1.18.3.post1
 ============

CHANGELOG.rst

@@ -75,7 +75,7 @@ Supported Python Versions
 SageMaker Python SDK is tested on:
 
 - Python 2.7
-- Python 3.5
+- Python 3.6
 
 AWS Permissions
 ~~~~~~~~~~~~~~~
@@ -97,9 +97,16 @@ Running tests
 
 SageMaker Python SDK has unit tests and integration tests.
 
+You can install the libraries needed to run the tests by running :code:`pip install --upgrade .[test]` or, for Zsh users: :code:`pip install --upgrade .\[test\]`
+
 **Unit tests**
 
-tox is a prerequisite for running unit tests so you need to make sure you have it installed. To run the unit tests:
+
+We run unit tests with tox, which is a program that lets you run unit tests for multiple Python versions, and also make sure the
+code fits our style guidelines. We run tox with Python 2.7 and 3.6, so to run unit tests
+with the same configuration we do, you'll need to have interpreters for Python 2.7 and Python 3.6 installed.
+
+To run the unit tests with tox, run:
 
 ::
 
@@ -109,23 +116,29 @@ tox is a prerequisite for running unit tests so you need to make sure you have i
 
 To run the integration tests, the following prerequisites must be met
 
-1. Access to an AWS account to run the tests on
-2. AWS account credentials available to boto3 clients used in the tests
-3. The AWS account has an IAM role named :code:`SageMakerRole`
-4. The libraries listed in the ``extras_require`` object in ``setup.py`` for ``test`` are installed.
-   You can do this by running the following command: :code:`pip install --upgrade .[test]`
+1. AWS account credentials are available in the environment for the boto3 client to use.
+2. The AWS account has an IAM role named :code:`SageMakerRole` with the AmazonSageMakerFullAccess policy attached.
+
+We recommend selectively running just those integration tests you'd like to run. You can filter by individual test function names with:
+
+::
+
+    pytest -k 'test_i_care_about'
+
 
-You can run integ tests by issuing the following command:
+You can also run all of the integration tests by running the following command, which runs them in sequence, which may take a while:
 
 ::
 
     pytest tests/integ
 
-You can also filter by individual test function names (usable with any of the previous commands):
+
+You can also run them in parallel:
 
 ::
 
-    pytest -k 'test_i_care_about'
+    pytest -n auto tests/integ
+
 
 Building Sphinx docs
 ~~~~~~~~~~~~~~~~~~~~

README.rst

@@ -0,0 +1,51 @@
+version: 0.2
+
+phases:
+  pre_build:
+    commands:
+      - start-dockerd
+
+  build:
+    commands:
+      # run linters
+      - tox -e flake8,pylint
+
+      # run package and docbuild checks
+      - tox -e twine
+      - tox -e sphinx
+
+      # run unit tests
+      - AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_SESSION_TOKEN=
+        AWS_CONTAINER_CREDENTIALS_RELATIVE_URI= AWS_DEFAULT_REGION=
+        tox -e py27,py36 -- tests/unit
+
+      # run notebook test
+      - |
+        if has-matching-changes "src/*.py" "setup.py" "setup.cfg"; then
+          echo "running notebook test"
+          python setup.py sdist
+          aws s3 --region us-west-2 cp ./dist/sagemaker-*.tar.gz s3://sagemaker-python-sdk-pr/
+          aws s3 cp s3://sagemaker-mead-cli/mead-nb-test.tar.gz mead-nb-test.tar.gz
+          tar -xzf mead-nb-test.tar.gz
+          git clone --depth 1 https://github.com/awslabs/amazon-sagemaker-examples.git
+          JAVA_HOME=$(get-java-home)
+          echo "set JAVA_HOME=$JAVA_HOME"
+          SAGEMAKER_ROLE_ARN=$(get-sagemaker-role-arn)
+          echo "set SAGEMAKER_ROLE_ARN=$SAGEMAKER_ROLE_ARN"
+          ./runtime/bin/mead-run-nb-test \
+            --instance-type ml.c4.8xlarge \
+            --region us-west-2 \
+            --lifecycle-config-name install-python-sdk \
+            --notebook-instance-role-arn $SAGEMAKER_ROLE_ARN \
+            ./amazon-sagemaker-examples/sagemaker-python-sdk/tensorflow_distributed_mnist/tensorflow_batch_transform_mnist.ipynb
+         else
+           echo "skipping notebook test"
+         fi
+
+      # run integration tests
+      - |
+        if has-matching-changes "tests/" "src/*.py" "setup.py" "setup.cfg"; then
+          IGNORE_COVERAGE=- tox -e py27,py36 -- -n 24 --boxed tests/integ
+        else
+          echo "skipping integration tests"
+        fi

buildspec.yml

@@ -50,6 +50,11 @@ TensorFlow
 **********
 A managed environment for TensorFlow training and hosting on Amazon SageMaker
 
+.. toctree::
+    :maxdepth: 1
+
+    using_tf
+
 .. toctree::
     :maxdepth: 2