change: Implement override solution for pipeline variables #2995

qidewenwhen · 2022-03-14T17:43:30Z

Description of changes: This PR implement the overridden solution to make the SM Pipeline variables to have similar behaviors as Python primitives to resolve some possible customer issues.

Testing done: unit tests and integ tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-03-14T18:07:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 221eb8d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T18:38:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 221eb8d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-03-14T18:47:59Z

Codecov Report

Merging #2995 (898ae35) into master (1c3b9a4) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2995      +/-   ##
==========================================
- Coverage   89.97%   89.91%   -0.06%     
==========================================
  Files        1028      196     -832     
  Lines       83918    16633   -67285     
==========================================
- Hits        75503    14956   -60547     
+ Misses       8415     1677    -6738

Impacted Files	Coverage Δ
src/sagemaker/estimator.py	`91.30% <100.00%> (-0.03%)`	⬇️
src/sagemaker/parameter.py	`97.29% <100.00%> (-0.14%)`	⬇️
src/sagemaker/tuner.py	`97.91% <100.00%> (+<0.01%)`	⬆️
src/sagemaker/workflow/entities.py	`97.50% <100.00%> (+2.26%)`	⬆️
src/sagemaker/workflow/execution_variables.py	`100.00% <100.00%> (ø)`
src/sagemaker/workflow/functions.py	`100.00% <100.00%> (ø)`
src/sagemaker/workflow/parameters.py	`98.61% <100.00%> (+0.03%)`	⬆️
src/sagemaker/workflow/properties.py	`89.61% <100.00%> (+0.13%)`	⬆️
...on3.9/site-packages/sagemaker/workflow/emr_step.py
...agemaker/cli/compatibility/v2/modifiers/airflow.py
... and 831 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c3b9a4...898ae35. Read the comment docs.

sagemaker-bot · 2022-03-14T19:13:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 70d0adc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T19:45:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 70d0adc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T19:53:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 221eb8d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/workflow/entities.py

src/sagemaker/workflow/functions.py

src/sagemaker/workflow/entities.py

src/sagemaker/tuner.py

sagemaker-bot · 2022-03-16T02:17:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b76ac7b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T02:44:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7ae479f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T03:12:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b76ac7b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T03:12:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7ae479f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T04:05:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b76ac7b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T04:05:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7ae479f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T19:01:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 66e165b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T19:39:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 66e165b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-16T19:52:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 66e165b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/workflow/entities.py

tests/unit/sagemaker/workflow/test_parameters.py

tests/unit/sagemaker/workflow/test_properties.py

src/sagemaker/estimator.py

tests/integ/sagemaker/workflow/test_tuning_steps.py

sagemaker-bot · 2022-03-18T01:01:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 51627d0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-18T01:48:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 51627d0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-18T01:53:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 51627d0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-21T20:38:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 898ae35
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-21T21:45:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 898ae35
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen force-pushed the improve branch from 221eb8d to 70d0adc Compare March 14, 2022 18:47

qidewenwhen requested review from jerrypeng7773 and navaj0 March 14, 2022 18:49

navaj0 reviewed Mar 15, 2022

View reviewed changes

qidewenwhen force-pushed the improve branch from 70d0adc to b76ac7b Compare March 16, 2022 01:52

qidewenwhen force-pushed the improve branch from b76ac7b to 7ae479f Compare March 16, 2022 02:19

qidewenwhen force-pushed the improve branch from 7ae479f to 66e165b Compare March 16, 2022 17:56

qidewenwhen changed the title ~~change: Implement overridden solution for pipeline variables~~ change: Implement overiden solution for pipeline variables Mar 16, 2022

qidewenwhen changed the title ~~change: Implement overiden solution for pipeline variables~~ change: Implement overide solution for pipeline variables Mar 16, 2022

qidewenwhen changed the title ~~change: Implement overide solution for pipeline variables~~ change: Implement override solution for pipeline variables Mar 16, 2022

jerrypeng7773 reviewed Mar 16, 2022

View reviewed changes

src/sagemaker/workflow/entities.py Show resolved Hide resolved

jerrypeng7773 reviewed Mar 16, 2022

View reviewed changes

tests/unit/sagemaker/workflow/test_parameters.py Show resolved Hide resolved

jerrypeng7773 reviewed Mar 16, 2022

View reviewed changes

tests/unit/sagemaker/workflow/test_properties.py Show resolved Hide resolved

navaj0 reviewed Mar 17, 2022

View reviewed changes

src/sagemaker/estimator.py Show resolved Hide resolved

tests/integ/sagemaker/workflow/test_tuning_steps.py Outdated Show resolved Hide resolved

Dewen Qi added 2 commits March 21, 2022 10:22

change: Implement overide solution for pipeline varialbes

4af1e4e

fix: Add more unit tests

898ae35

qidewenwhen force-pushed the improve branch from 51627d0 to 898ae35 Compare March 21, 2022 20:10

jerrypeng7773 approved these changes Mar 23, 2022

View reviewed changes

navinsoni approved these changes Mar 25, 2022

View reviewed changes

shreyapandit approved these changes Mar 26, 2022

View reviewed changes

navinsoni merged commit 850972c into aws:master Mar 26, 2022

shreyapandit pushed a commit to shreyapandit/sagemaker-python-sdk that referenced this pull request Mar 26, 2022

change: Implement override solution for pipeline variables (aws#2995)

019618f

Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen mentioned this pull request Mar 30, 2022

"error: argument *: invalid int value" for arguments that come from ParameterInteger or ParameterFloat #2978

Closed

jerrypeng7773 pushed a commit to jerrypeng7773/sagemaker-python-sdk that referenced this pull request May 13, 2022

change: Implement override solution for pipeline variables (aws#2995)

d602db5

Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen deleted the improve branch January 25, 2024 18:34

change: Implement override solution for pipeline variables #2995

change: Implement override solution for pipeline variables #2995

Uh oh!

Conversation

qidewenwhen commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 16, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Mar 18, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 18, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 18, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 21, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 21, 2022

AWS CodeBuild CI Report

qidewenwhen commented Mar 14, 2022 •

edited

Loading

codecov-commenter commented Mar 14, 2022 •

edited

Loading