feature: support cluster lifecycle management for Sagemaker EMR step #3604

Shegufta · 2023-01-20T07:54:09Z

Issue #, if available:
Sagemaker Pipeline (SMP) supports EMRStep where users provide an already created EMR cluster id. In the current approach users are responsible for managing the EMR cluster's lifecycle. Sometimes they might forget to terminate the cluster in time, which incurs unexpected cost.

We are working on a feature request where instead of creating and maintaining a cluster, user would provide a ClusterConfig. SMP will create a cluster based on the user provided config. Once the step finishes, SMP will terminate the cluster. This will free users from managing the EMR cluster lifecycle.

So far we've made necessary backend changes that allows to add the ClusterConfig in DSL, and directly use it to create the Pipeline. However, we still need to offer the same feature through the Sagemaker Python SDK.

Description of changes:
This CR introduces ClusterConfig in the SM Python SDK. It would allow user to create and use ClusterConfig, during the EMR Step creation process.

Testing done:
We've added tests that ensures that the python SDK is properly converting the user-provided ClusterConfig into an appropriate JSON format (DSL).

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

vikas0203

/bot run all

knikure

/bot run all

sagemaker-bot · 2023-01-20T17:14:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 26d7fd8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T17:46:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 26d7fd8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T17:52:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 26d7fd8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

danRubiks · 2023-01-20T17:51:20Z

src/sagemaker/workflow/emr_step.py

@@ -61,6 +61,59 @@ def to_request(self) -> RequestType:
        return config


+def validate_cluster_config(cluster_config, name):


could you specify name as step name?

replaced with step_name

sagemaker-bot · 2023-01-20T18:08:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 26d7fd8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T18:23:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 26d7fd8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run all

sagemaker-bot · 2023-01-20T18:36:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T19:16:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a4c299f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T19:20:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T19:35:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T19:36:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a4c299f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run notebook-tests

vikas0203

/bot run slow-tests

vikas0203

/bot run unit-tests

sagemaker-bot · 2023-01-20T20:41:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T20:42:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T21:08:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a4c299f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run notebook-tests

vikas0203

/bot run slow-tests

vikas0203

/bot run slow-tests

sagemaker-bot · 2023-02-06T22:09:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4946c86
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-06T22:11:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4946c86
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run notebook-tests

vikas0203

/bot run slow-tests

vikas0203

/bot run all

sagemaker-bot · 2023-02-06T22:45:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 16f4c1d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run slow-tests

sagemaker-bot · 2023-02-06T23:21:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 16f4c1d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-06T23:21:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 16f4c1d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-06T23:32:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 16f4c1d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run pr

sagemaker-bot · 2023-02-06T23:46:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 16f4c1d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T00:27:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 16f4c1d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run all

sagemaker-bot · 2023-02-07T01:43:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4b2b6f7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T01:43:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4b2b6f7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T02:10:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4b2b6f7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T02:29:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4b2b6f7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-07T03:54:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4b2b6f7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run slow-tests

sagemaker-bot · 2023-02-07T06:23:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4b2b6f7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ws#3604) Co-authored-by: Shegufta Ahsan <[email protected]> Co-authored-by: vikas0203 <[email protected]>

Shegufta requested a review from a team as a code owner January 20, 2023 07:54

Shegufta requested review from navinsoni and removed request for a team January 20, 2023 07:54

knikure assigned vikas0203 Jan 20, 2023

vikas0203 reviewed Jan 20, 2023

View reviewed changes

feature: support cluster lifecycle management for Sagemaker EMR step

26d7fd8

knikure force-pushed the master branch from 9defc87 to 26d7fd8 Compare January 20, 2023 17:09

knikure reviewed Jan 20, 2023

View reviewed changes

danRubiks reviewed Jan 20, 2023

View reviewed changes

refactoring name as step_name

a4c299f

vikas0203 reviewed Jan 20, 2023

View reviewed changes

danRubiks approved these changes Jan 20, 2023

View reviewed changes

vikas0203 reviewed Jan 20, 2023

View reviewed changes

vikas0203 reviewed Feb 6, 2023

View reviewed changes

updating the cluster_config type

16f4c1d

aoguo64 approved these changes Feb 6, 2023

View reviewed changes

vikas0203 reviewed Feb 6, 2023

View reviewed changes

updating the documentations

4b2b6f7

vikas0203 reviewed Feb 7, 2023

View reviewed changes

aoguo64 approved these changes Feb 7, 2023

View reviewed changes

vikas0203 reviewed Feb 7, 2023

View reviewed changes

vikas0203 approved these changes Feb 7, 2023

View reviewed changes

vikas0203 merged commit f09f64b into aws:master Feb 7, 2023

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: support cluster lifecycle management for Sagemaker EMR step (a…

9ca63ff

…ws#3604) Co-authored-by: Shegufta Ahsan <[email protected]> Co-authored-by: vikas0203 <[email protected]>

		@@ -61,6 +61,59 @@ def to_request(self) -> RequestType:
		return config


		def validate_cluster_config(cluster_config, name):

feature: support cluster lifecycle management for Sagemaker EMR step #3604

feature: support cluster lifecycle management for Sagemaker EMR step #3604

Uh oh!

Conversation

Shegufta commented Jan 20, 2023

Merge Checklist

General

Tests

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

knikure left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

danRubiks Jan 20, 2023

Choose a reason for hiding this comment

Uh oh!

Shegufta Jan 20, 2023

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 20, 2023

AWS CodeBuild CI Report

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment

Uh oh!

vikas0203 left a comment

Choose a reason for hiding this comment