feature: add framework upgrade tool #1762

chuyang-deng · 2020-07-28T17:05:23Z

Issue #, if available:

Description of changes:
Add a script for updating framework image uris json file.
[x] allow for DLC-like framework version upgrade
[x] allow for Algo-like framework version upgrade (json files with "scope")
[x] support for region update

Testing done:
unit test
manual test of command line to update a framework version and checked corresponding json file

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-07-28T17:13:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4028d21
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T17:13:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d9c8e5c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T17:33:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4028d21
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T17:36:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d9c8e5c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T17:48:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4028d21
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T17:49:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d9c8e5c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-28T18:28:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: d9c8e5c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu · 2020-07-28T21:54:31Z

src/sagemaker/image_uri_config/framework-region-registry.json

@@ -0,0 +1,749 @@
+{


how about instead of duplicating all of this here just using whatever is in the most recent version (for a given framework/algo)?

Removed this json file as well the algorithm -> registry json file.

laurenyu · 2020-07-28T21:57:46Z

src/sagemaker/image_uri_config/framework_upgrade.py

@@ -0,0 +1,146 @@
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.


maybe let's put this in the CLI folder? what I'm thinking (and tell me if this is unrealistic) is that for future version/region releases, a person could run the script and then commit the changes.

Moved the CLI folder.

For the order of running script and commit changes. Running the script only changes the content of json files, thus affecting the result of image_uris.retrieve(). But those changes won't be effective until merged.

laurenyu · 2020-07-28T22:00:11Z

src/sagemaker/image_uri_config/framework_upgrade.py

+    py_versions = py_versions.split(",")
+    processors = processors.split(",")
+    registries = _read_framework_region_regestries(framework)["registries"]
+    add_dlc_framework_version(


could we basically default to the most recent version for anything that's not provided to the user?

Removed registries related files and take latest region account mappings, py_versions and repository values.

sagemaker-bot · 2020-07-29T18:06:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 058d6d2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T18:25:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 058d6d2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T18:34:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2de4bd2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T18:39:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 058d6d2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T18:52:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2de4bd2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T19:07:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2de4bd2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-29T19:46:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 2de4bd2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…-python-sdk into json-update-script

sagemaker-bot · 2020-07-29T21:34:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6d18122
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu

just a couple, small stylistic suggestions. otherwise lgtm.

laurenyu · 2020-07-30T19:44:52Z

src/sagemaker/cli/framework_upgrade.py

+            existing_content["versions"][version]["registries"][region] = account
+
+
+def update_json(


maybe rename this to add_version to be more specific about what it's doing?

Yes, it should be add_version. The original name was created without having add_region in mind.

laurenyu · 2020-07-30T19:45:52Z

src/sagemaker/cli/framework_upgrade.py

+    framework = args.framework
+    short_version = args.short_version
+    full_version = args.full_version
+    processors = args.processors
+    py_versions = args.py_versions
+    scope = args.scope
+    tag_prefix = args.tag_prefix
+    region = args.region
+    account = args.account


(optional/nitpick) I think I'd probably just use the args.<name> directly rather than assigning new variables - I don't feel like this is providing much value here.

make sense!

sagemaker-bot · 2020-07-30T21:16:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 348646d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:16:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1e7799f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:30:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e8e4f4e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:35:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 1e7799f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:36:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 348646d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:48:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e8e4f4e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:50:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1e7799f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T21:51:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 348646d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T22:03:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e8e4f4e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-30T22:15:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: e8e4f4e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

metrizable · 2020-07-30T22:23:48Z

src/sagemaker/cli/framework_upgrade.py

+    latest_version = list(existing_content[scope]["versions"].keys())[-1]
+    registries = existing_content[scope]["versions"][latest_version]["registries"]
+    py_versions = existing_content[scope]["versions"][latest_version]["py_versions"]
+    repository = existing_content[scope]["versions"][latest_version]["repository"]


I think there is something wrong with this if you've already gotten existing_content from scope above.

You are right, the last 3 lines should not reference "scope" anymore. I've updated the method and created unit tests for it.

…-python-sdk into json-update-script

sagemaker-bot · 2020-07-31T00:16:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5e68e01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T00:35:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5e68e01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T00:49:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5e68e01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T01:27:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 5e68e01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

metrizable

looks good to me. one small comment.

metrizable · 2020-07-31T04:01:59Z

src/sagemaker/cli/framework_upgrade.py

+    )
+    parser.add_argument("--short-version", help="Abbreviated framework version (e.g. 2.0)")
+    parser.add_argument("--full-version", help="Full framework version (e.g. 2.0.1)")
+    parser.add_argument("--processors", help="Suppoted processors (e.g. cpu, gpu)")


Suggested change

parser.add_argument("--processors", help="Suppoted processors (e.g. cpu, gpu)")

parser.add_argument("--processors", help="Supported processors (e.g. cpu, gpu)")

sagemaker-bot · 2020-07-31T16:16:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 433e608
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T16:35:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 433e608
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T16:49:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 433e608
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-31T17:21:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 433e608
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Chuyang Deng and others added 3 commits July 28, 2020 10:02

feature: add framework upgrade tool

7a61365

add framework upgrade test

4028d21

Merge branch 'zwei' into json-update-script

d9c8e5c

chuyang-deng changed the title ~~feature: add framework upgrade tool~~ [WIP] feature: add framework upgrade tool Jul 28, 2020

laurenyu reviewed Jul 28, 2020

View reviewed changes

add algo image upgrade logic

058d6d2

Merge branch 'zwei' into json-update-script

2de4bd2

Chuyang Deng added 2 commits July 29, 2020 14:25

support for region update

13b9f1a

Merge branch 'json-update-script' of github.com:ChuyangDeng/sagemaker…

6d18122

…-python-sdk into json-update-script

Chuyang Deng and others added 2 commits July 29, 2020 14:38

move framework upgrade tool to cli folder

da8efc8

Merge branch 'zwei' into json-update-script

77cae0c

chuyang-deng changed the title ~~[WIP] feature: add framework upgrade tool~~ feature: add framework upgrade tool Jul 29, 2020

chuyang-deng added 3 commits July 29, 2020 14:45

fix typo

194df26

fix auto changes

9b22fad

fix auto changes

d946684

laurenyu reviewed Jul 30, 2020

View reviewed changes

Chuyang Deng and others added 2 commits July 30, 2020 14:08

simplify and rename method

1e7799f

Merge branch 'zwei' into json-update-script

348646d

laurenyu previously approved these changes Jul 30, 2020

View reviewed changes

Merge branch 'zwei' into json-update-script

e8e4f4e

metrizable suggested changes Jul 30, 2020

View reviewed changes

Chuyang Deng added 2 commits July 30, 2020 17:07

fix dict KeyError and and unit tests

e0cc4e1

Merge branch 'json-update-script' of github.com:ChuyangDeng/sagemaker…

5e68e01

…-python-sdk into json-update-script

chuyang-deng dismissed laurenyu’s stale review via 5e68e01 July 31, 2020 00:07

metrizable approved these changes Jul 31, 2020

View reviewed changes

Merge branch 'zwei' into json-update-script

433e608

chuyang-deng merged commit a6f51b7 into aws:zwei Jul 31, 2020

		@@ -0,0 +1,146 @@
		# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

		existing_content["versions"][version]["registries"][region] = account


		def update_json(

	parser.add_argument("--processors", help="Suppoted processors (e.g. cpu, gpu)")
	parser.add_argument("--processors", help="Supported processors (e.g. cpu, gpu)")

feature: add framework upgrade tool #1762

feature: add framework upgrade tool #1762

Uh oh!

Conversation

chuyang-deng commented Jul 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 28, 2020

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 29, 2020

AWS CodeBuild CI Report

Uh oh!

laurenyu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jul 30, 2020

AWS CodeBuild CI Report

chuyang-deng commented Jul 28, 2020 •

edited

Loading