add data wrangler processor #2306

EthanShouhanCheng · 2021-04-27T20:19:22Z

Issue #, if available:

Description of changes: Add data wrangler processor for SageMaker pipeline

Testing done:

test_data_wrangler_processor_with_required_parameters passed
test_data_wrangler_processor_with_mock_input passed

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-04-27T20:22:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 642c958
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-27T20:55:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 642c958
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-27T20:55:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 642c958
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-27T21:03:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 642c958
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-27T21:03:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 642c958
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jerrypeng7773 · 2021-04-28T21:26:59Z

src/sagemaker/workflow/processing.py

+            max_runtime_in_seconds=max_runtime_in_seconds,
+            base_job_name=base_job_name,
+            sagemaker_session=sagemaker_session,
+            tags=tags,


I assume we are not including env here since DW container will never use it?

that is correct, there is not reason for user to pass their env

Can we just add env? It looks like a common parameter to have

sagemaker-python-sdk/src/sagemaker/clarify.py

Line 336 in d3d07b4

env=None,

. We might enable some user provided env in the future, since we support custom code in DW

jerrypeng7773 · 2021-04-28T21:28:17Z

tests/unit/sagemaker/workflow/test_processing.py

+        s3_input_mode="File",
+        s3_data_distribution_type="FullyReplicated",
+    )
+    processor.run(inputs=[mock_input])


consider adding a test for output?

will do, thx!

actually nothing changes for outputs, so I will not add unit test for it.
but output will be covered in intg tests, I will update this PR later.

sagemaker-bot · 2021-04-29T14:17:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 58baed4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-29T14:51:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 58baed4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-29T14:56:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 58baed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-29T14:57:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 58baed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-29T14:59:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 58baed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-29T19:47:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 58baed4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T00:34:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ca55485
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T00:36:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d8cceb7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T00:37:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ca55485
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:08:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ca55485
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:09:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d8cceb7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:12:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d8cceb7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:15:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ca55485
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:19:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d8cceb7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T01:21:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d8cceb7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T15:41:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d8cceb7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-30T16:14:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d8cceb7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T22:45:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: cc4598f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:10:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: cc4598f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:14:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: cc4598f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:18:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: cc4598f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:19:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: cc4598f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:34:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 511e4c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-12T23:59:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 511e4c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T00:06:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 511e4c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T00:11:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 511e4c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T00:24:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 511e4c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T03:58:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fdd66c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T04:21:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fdd66c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T04:26:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: fdd66c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T04:26:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fdd66c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T04:31:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fdd66c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T17:57:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fdd66c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T19:43:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c35c8b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T20:05:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fdd66c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T20:11:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c35c8b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T20:16:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c35c8b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T20:17:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c35c8b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-05-13T20:23:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c35c8b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

EthanShouhanCheng marked this pull request as draft April 27, 2021 20:21

EthanShouhanCheng marked this pull request as ready for review April 27, 2021 20:22

jerrypeng7773 reviewed Apr 28, 2021

View reviewed changes

EthanShouhanCheng force-pushed the change/add_data_wrangler_processor branch from 58baed4 to ca55485 Compare April 30, 2021 00:31

EthanShouhanCheng force-pushed the change/add_data_wrangler_processor branch from ca55485 to d8cceb7 Compare April 30, 2021 00:34

EthanShouhanCheng requested review from staubhp and rohangujarathi April 30, 2021 18:59

EthanShouhanCheng dismissed jerrypeng7773’s stale review via cc4598f May 12, 2021 22:34

EthanShouhanCheng force-pushed the change/add_data_wrangler_processor branch from 2f4a3a8 to cc4598f Compare May 12, 2021 22:34

EthanShouhanCheng force-pushed the change/add_data_wrangler_processor branch from cc4598f to 511e4c6 Compare May 12, 2021 23:23

add data wrangler processor

fdd66c2

EthanShouhanCheng force-pushed the change/add_data_wrangler_processor branch from 511e4c6 to fdd66c2 Compare May 13, 2021 03:47

icywang86rui approved these changes May 13, 2021

View reviewed changes

Merge branch 'master' into change/add_data_wrangler_processor

c35c8b2

ajaykarpur merged commit 2a61c41 into aws:master May 13, 2021

add data wrangler processor #2306

add data wrangler processor #2306

Uh oh!

Conversation

EthanShouhanCheng commented Apr 27, 2021

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Apr 27, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2021

AWS CodeBuild CI Report

Uh oh!

jerrypeng7773 Apr 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EthanShouhanCheng Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

chenliu0831 Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

jerrypeng7773 Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

EthanShouhanCheng Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

EthanShouhanCheng Apr 29, 2021

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 29, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 30, 2021

jerrypeng7773 Apr 28, 2021 •

edited

Loading