feature: add tensorflow training 1.15.2 py37 support #1458

chuyang-deng · 2020-05-05T20:31:10Z

Issue #, if available:

Description of changes:
Add Python 3.7 support for TensorFlow script mode 1.15.2 version.
tensorflow-training:1.15.2-py37 prod containers are not released yet.

Testing done:
unit test and integ test

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to any/all clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-05-05T20:42:54Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T20:44:40Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T20:52:19Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T20:53:11Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:04:28Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:06:44Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:16:04Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:27:06Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:37:42Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:55:20Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:01:46Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:03:41Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:14:17Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:17:35Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:27:11Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:36:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T22:41:05Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

yuanzhua · 2020-05-05T22:50:23Z

src/sagemaker/fw_utils.py

@@ -71,6 +71,7 @@
    "pytorch-serving",
 ]
 PY2_RESTRICTED_EIA_FRAMEWORKS = ["pytorch-serving"]
+PY37_SUPPORTED_FRAMEWORKS = ["tensorflow-scriptmode"]


why do we need this? should we just return image not found if the given combination (Framework, Framework version, Python version) is not support?

If we strip this line (L74) Python SDK will have to hit ECR service to find out if the combination exists.

If we do nothing to check for the py37 support before constructing the image uri, here will throw KeyError (key "py37" not found).

Updating the MERGED_FRAMEWORKS_LOWEST_VERSIONS dict to let each entry have a "py_version" key is also doable to prevent Python SDK hit the ECR service, but I think this dict is for checking whether the image is DLC merged frameworks instead of checking Python version support.

src/sagemaker/tensorflow/README.rst

laurenyu · 2020-05-05T23:14:39Z

tests/integ/test_tf_script_mode.py

@@ -131,7 +131,9 @@ def test_mnist_distributed(sagemaker_session, instance_type, tf_full_version):
        train_instance_count=2,
        train_instance_type=instance_type,
        sagemaker_session=sagemaker_session,
-        py_version=tests.integ.PYTHON_VERSION,
+        py_version="py37"


what about other places in this file where py_veresion=tests.integ.PYTHON_VERSION?

I was going to update them all once the container is released. But probably it's better to update them all at once.

hmm...maybe it would be cleaner to put this into a fixture? i.e.

@pytest.fixture def py_version(): return "py37" if tf_full_version == TensorFlow._LATEST_1X_VERSION else tests.integ.PYTHON_VERSION

and then just use py_version in all the tests

👍 Yes. This would be much better.

ah, I think you can leave it in the TF script mode test file, since that's the only file that uses it, and the if condition is rather TF-specific

laurenyu · 2020-05-05T23:15:22Z

src/sagemaker/fw_utils.py

@@ -257,6 +258,9 @@ def create_image_uri(
    if py_version and py_version not in VALID_PY_VERSIONS:
        raise ValueError("invalid py_version argument: {}".format(py_version))

+    if py_version == "py37" and framework not in PY37_SUPPORTED_FRAMEWORKS:
+        raise ValueError("{} does not support Python 3.7 yet.".format(framework))


maybe s/yet/at this time. There might some some frameworks that never receive Python 3.7 support.

sagemaker-bot · 2020-05-06T00:08:15Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T00:19:09Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T00:29:12Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T17:18:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T17:28:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T17:38:54Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T17:40:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:04:14Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:15:40Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:19:43Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:26:33Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:30:09Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:37:06Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T18:42:22Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T21:33:46Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T21:45:04Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T21:51:56Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T21:56:05Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T21:56:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…o only test the latest version

sagemaker-bot · 2020-05-06T22:41:05Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T22:51:47Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T23:03:52Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-06T23:04:14Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Chuyang Deng and others added 2 commits May 5, 2020 13:27

feature: tensorflow training 1.15.2 py37 support

56635c9

Merge branch 'master' into master

ecb3e76

Chuyang Deng added 2 commits May 5, 2020 14:41

update documentation

3f404df

Merge branch 'master' of github.com:ChuyangDeng/sagemaker-python-sdk

e82886b

laurenyu changed the title ~~feature: tensorflow training 1.15.2 py37 support~~ feature: add tensorflow training 1.15.2 py37 support May 5, 2020

Merge branch 'master' into master

5e8f01e

yuanzhua reviewed May 5, 2020

View reviewed changes

laurenyu reviewed May 5, 2020

View reviewed changes

Chuyang Deng added 2 commits May 5, 2020 16:56

add more docs and update error message

80ef94b

Merge branch 'master' of github.com:ChuyangDeng/sagemaker-python-sdk

b998482

Merge branch 'master' into master

4150361

Chuyang Deng added 2 commits May 6, 2020 11:07

move py_version fixture to test_tf_script_mode

921f709

Merge branch 'master' of github.com:ChuyangDeng/sagemaker-python-sdk

b87ffdd

laurenyu previously approved these changes May 6, 2020

View reviewed changes

Merge branch 'master' into master

475ff82

Chuyang Deng added 2 commits May 6, 2020 15:29

not testing py37 with test_mnist_async because the test is intended t…

d978b78

…o only test the latest version

Merge branch 'master' of github.com:ChuyangDeng/sagemaker-python-sdk

5c8dab0

chuyang-deng dismissed laurenyu’s stale review via 5c8dab0 May 6, 2020 22:29

laurenyu approved these changes May 6, 2020

View reviewed changes

chuyang-deng merged commit d53b957 into aws:master May 6, 2020

feature: add tensorflow training 1.15.2 py37 support #1458

feature: add tensorflow training 1.15.2 py37 support #1458

Uh oh!

Conversation

chuyang-deng commented May 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 5, 2020

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chuyang-deng commented May 5, 2020 •

edited

Loading