fix: Correctly interpolate Callback output parameters #2467

staubhp · 2021-06-17T19:34:30Z

…eference

Issue #, if available:
N/A

Description of changes:
There are two ways to reference a callback output parameter in another pipeline step.

By the step's properties:

...Defining a step...
    inputs={
      'arg1': step_callback1.properties.Outputs['out_int']
    },
...

Or by direct reference to the callback output parameter:

callback1_output=CallbackOutput(output_name="out_int", output_type=CallbackOutputTypeEnum.Integer)
...Defining a step...
    inputs={
      'arg1': callback1_output
    },
...

The latter is not working in all cases today because the output's name is used both as the dict key as well as the step name when the JSON Get expression is created.

This change creates a map of callback output names to step names during pipeline definition interpolation to fix this issue.

Testing done:
Unit, manual

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…eference

sagemaker-bot · 2021-06-17T19:45:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6deec71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-17T20:07:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6deec71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-17T20:11:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6deec71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-17T20:19:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6deec71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-17T20:30:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6deec71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

icywang86rui

Is there change already covered by existing integration tests? If not can we add one.

src/sagemaker/workflow/callback_step.py

icywang86rui · 2021-06-21T15:56:29Z

src/sagemaker/workflow/pipeline.py

@@ -263,7 +266,7 @@ def format_start_parameters(parameters: Dict[str, Any]) -> List[Dict[str, Any]]:
    return [{"Name": name, "Value": str(value)} for name, value in parameters.items()]


-def interpolate(request_obj: RequestType) -> RequestType:
+def interpolate(request_obj: RequestType, **kwargs) -> RequestType:


add kwargs to api doc bellow.

Removed kwargs

icywang86rui · 2021-06-21T15:56:39Z

src/sagemaker/workflow/pipeline.py



-def _interpolate(obj: Union[RequestType, Any]):
+def _interpolate(obj: Union[RequestType, Any], **kwargs):


icywang86rui · 2021-06-21T15:58:12Z

src/sagemaker/workflow/pipeline.py

@@ -263,7 +266,7 @@ def format_start_parameters(parameters: Dict[str, Any]) -> List[Dict[str, Any]]:
    return [{"Name": name, "Value": str(value)} for name, value in parameters.items()]


-def interpolate(request_obj: RequestType) -> RequestType:
+def interpolate(request_obj: RequestType, **kwargs) -> RequestType:


is there a reason to use kwargs here? Seems like you are only adding one parameter.

It was my way of making the arg optional since it won't be included in every interpolate call, but I can get rid of the kwargs

sagemaker-bot · 2021-06-21T16:10:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2507d19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T16:35:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2507d19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T16:38:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2507d19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T16:38:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2507d19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T16:44:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2507d19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…agemaker-python-sdk into callback-param-bug-fix

sagemaker-bot · 2021-06-21T18:37:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: accac70
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T18:48:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8b7b81b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:09:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: accac70
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:11:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: accac70
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:12:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 8b7b81b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:13:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8b7b81b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:15:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8b7b81b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:18:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: accac70
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T19:21:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8b7b81b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T22:49:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b3bd0bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T23:13:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b3bd0bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T23:14:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b3bd0bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T23:16:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b3bd0bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-21T23:24:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b3bd0bb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T16:20:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2b8727e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T16:49:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2b8727e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T16:53:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2b8727e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T16:54:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2b8727e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T16:56:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2b8727e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T17:28:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4bb77f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T17:51:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4bb77f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T17:53:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4bb77f1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T18:00:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4bb77f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T18:31:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4bb77f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-06-22T21:13:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4bb77f1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Payton Staub added 3 commits June 17, 2021 12:27

Correctly interpolate Callback output parameters that are passed by r…

7ff222f

…eference

Correctly interpolate Callback output parameters that are passed by r…

428e028

…eference

Remove debug statements

6deec71

staubhp changed the title ~~[Bug] Correctly interpolate Callback output parameters that are passed by r…~~ fix: Correctly interpolate Callback output parameters that are passed by r… Jun 17, 2021

jerrypeng7773 previously approved these changes Jun 18, 2021

View reviewed changes

icywang86rui reviewed Jun 21, 2021

View reviewed changes

Merge branch 'master' into callback-param-bug-fix

2507d19

Payton Staub added 2 commits June 21, 2021 11:33

Address PR comments

3a9fa41

Merge branch 'callback-param-bug-fix' of https://github.com/staubhp/s…

accac70

…agemaker-python-sdk into callback-param-bug-fix

staubhp dismissed jerrypeng7773’s stale review via accac70 June 21, 2021 18:34

Update api docs

8b7b81b

Add missing positional arg to integ test

b3bd0bb

Merge branch 'master' into callback-param-bug-fix

2b8727e

icywang86rui approved these changes Jun 22, 2021

View reviewed changes

Merge branch 'master' into callback-param-bug-fix

4bb77f1

icywang86rui changed the title ~~fix: Correctly interpolate Callback output parameters that are passed by r…~~ fix: Correctly interpolate Callback output parameters Jun 22, 2021

icywang86rui merged commit a05b10b into aws:master Jun 22, 2021



		def _interpolate(obj: Union[RequestType, Any]):
		def _interpolate(obj: Union[RequestType, Any], **kwargs):

fix: Correctly interpolate Callback output parameters #2467

fix: Correctly interpolate Callback output parameters #2467

Uh oh!

Conversation

staubhp commented Jun 17, 2021

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jun 17, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 17, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 17, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 17, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 17, 2021

AWS CodeBuild CI Report

Uh oh!

icywang86rui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

icywang86rui Jun 21, 2021

Choose a reason for hiding this comment

Uh oh!

staubhp Jun 21, 2021

Choose a reason for hiding this comment

Uh oh!

icywang86rui Jun 21, 2021

Choose a reason for hiding this comment

Uh oh!

icywang86rui Jun 21, 2021

Choose a reason for hiding this comment

Uh oh!

staubhp Jun 21, 2021

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 21, 2021