feature: add dataset definition support for processing jobs #2031

guoqiao1992 · 2020-12-11T23:23:29Z

Issue #, if available:
Today Processor does not allow customers to pass DatasetDefinition

Description of changes:

Add BaseConfig that uses attr metedata to serialize/deserialize object.
Add DatasetDefinition and S3Input
Update ProcessingInput

Testing done:

Add unit test for dataset definition
Add integration test for dataset definition(not testing functionality, but only verify the the values are passed to service)

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

[x ] I have read the CONTRIBUTING doc
[x ] I used the commit message format described in CONTRIBUTING
[x ] I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
[x ] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

[x ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
x[ ] I have checked that my tests are not configured for a specific region or account (if appropriate)
[x ] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-12-11T23:32:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9c60705
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/dataset_definition/__init__.py

src/sagemaker/dataset_definition/inputs.py

src/sagemaker/processing.py

tests/integ/test_processing.py

src/sagemaker/dataset_definition/inputs.py

sagemaker-bot · 2020-12-12T00:04:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9c60705
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-12T00:05:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9c60705
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-12T01:04:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9c60705
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-14T22:37:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 01254b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

metrizable

please add an integ test in SageMaker Pipelines to ensure that service's support for datasets.

src/sagemaker/apiutils/_base_types.py

src/sagemaker/processing.py

sagemaker-bot · 2020-12-14T23:07:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 01254b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-14T23:08:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 01254b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-14T23:15:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fd29b54
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-14T23:42:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fd29b54
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-14T23:45:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fd29b54
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T01:50:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e501593
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T02:19:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e501593
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T02:23:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e501593
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T02:28:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e501593
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T05:01:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c3d1b33
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T05:32:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c3d1b33
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T05:34:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c3d1b33
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T05:40:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c3d1b33
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T21:17:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 24518f5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T21:51:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 24518f5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T21:51:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 24518f5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T22:45:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: aae8efa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T23:15:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: aae8efa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T23:16:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: aae8efa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-15T23:21:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: aae8efa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T16:47:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4619671
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/apiutils/_base_types.py

src/sagemaker/processing.py

sagemaker-bot · 2020-12-16T17:19:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4619671
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T17:20:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4619671
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T17:25:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4619671
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Eric Johnson <[email protected]>

sagemaker-bot · 2020-12-16T18:30:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b754bf9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T18:31:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ec65f6c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T19:02:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b754bf9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T19:03:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ec65f6c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T19:11:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ec65f6c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T19:16:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b754bf9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-12-16T19:21:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ec65f6c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

brightsparc · 2020-12-17T07:17:42Z

Please note this issue has broken support for using Processing Jobs in StepFunctions with error: Invalid State Machine Definition: ‘SCHEMA_VALIDATION_FAILED: The field “AppManaged” is not supported by Step Functions at /States/SageMaker Jobs/Branches[0]/States/Baseline Job/Parameters

Example JSON payload created for SFN:

 {
    "InputName": "baseline_dataset_input",
    "AppManaged": false,
    "S3Input": {
      "S3Uri": "s3://sagemaker-us-east-1-540347945558/nyc-tlc/v1/data/baseline/baseline.csv",
      "LocalPath": "/opt/ml/processing/input/baseline_dataset_input",
      "S3DataType": "S3Prefix",
      "S3InputMode": "File",
      "S3DataDistributionType": "FullyReplicated",
      "S3CompressionType": "None"
    }
  }

Tested with the stepfunctions==2.0.0rc1 version of SFN that supports SM v2

guoqiao1992 and others added 5 commits October 26, 2020 22:25

reset command for each run invocation

f81097b

Merge remote-tracking branch 'upstream/master'

945c769

remove response assertion on history server subpage

736cd50

Merge branch 'master' into master

75b127c

Merge remote-tracking branch 'upstream/master'

cbf9b58

icywang86rui reviewed Dec 11, 2020

View reviewed changes

add datasetdefinition

01254b2

guoqiao1992 force-pushed the master branch from 9c60705 to 01254b2 Compare December 14, 2020 22:28

metrizable suggested changes Dec 14, 2020

View reviewed changes

src/sagemaker/apiutils/_base_types.py Outdated Show resolved Hide resolved

src/sagemaker/processing.py Outdated Show resolved Hide resolved

small fix

fd29b54

add DatasetDefinition input in pipeline integ test

e501593

print reponse to debug integ test failures

c3d1b33

clean up code

24518f5

change dd default value

aae8efa

guoqiao1992 requested a review from metrizable December 16, 2020 01:20

Merge branch 'master' into master

4619671

metrizable reviewed Dec 16, 2020

View reviewed changes

src/sagemaker/apiutils/_base_types.py Outdated Show resolved Hide resolved

src/sagemaker/processing.py Outdated Show resolved Hide resolved

guoqiao1992 and others added 3 commits December 16, 2020 10:21

Update src/sagemaker/apiutils/_base_types.py

b754bf9

Co-authored-by: Eric Johnson <[email protected]>

update docstring to be consistent with aws document

6677cb7

default AppManaged to False

ec65f6c

metrizable approved these changes Dec 16, 2020

View reviewed changes

icywang86rui approved these changes Dec 16, 2020

View reviewed changes

metrizable merged commit 64371d3 into aws:master Dec 16, 2020

feature: add dataset definition support for processing jobs #2031

feature: add dataset definition support for processing jobs #2031

Uh oh!

Conversation

guoqiao1992 commented Dec 11, 2020

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Dec 11, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Dec 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

metrizable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 15, 2020

AWS CodeBuild CI Report

brightsparc commented Dec 17, 2020 •

edited

Loading