feature: support JSON for input dataset and model output #3687

spoorn · 2023-02-28T19:08:13Z

Issue #, if available: https://issues.amazon.com/issues/RAI-4252, https://issues.amazon.com/issues/RAI-4256

Description of changes: Add support for JSON for input dataset and model output for Clarify jobs. This includes enabling application/json type, updating documentation for fields' support for JSON, and removing some checks that blocked application/json or some other types for functionality (predicted_label and excluded_columns support).

A follow up PR will be done to support JSON for model inputs.

Testing done: Ran the unit tests locally, manual testing using a notebook

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-02-28T19:49:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 30dc6c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T19:55:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 30dc6c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T19:58:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 30dc6c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T20:04:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 30dc6c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-02-28T20:29:46Z

Codecov Report

Merging #3687 (3488d5d) into master (ad4a92a) will decrease coverage by 0.77%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3687      +/-   ##
==========================================
- Coverage   89.79%   89.02%   -0.77%     
==========================================
  Files         980      230     -750     
  Lines       92208    22811   -69397     
==========================================
- Hits        82796    20307   -62489     
+ Misses       9412     2504    -6908

Impacted Files	Coverage Δ
src/sagemaker/clarify.py	`95.88% <100.00%> (ø)`

... and 1209 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-02-28T20:29:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 30dc6c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xiaoyi-cheng

lgtm

navinsoni · 2023-03-02T18:37:47Z

src/sagemaker/clarify.py

@@ -311,7 +312,7 @@ def __init__(
        s3_analysis_config_output_path: Optional[str] = None,
        label: Optional[str] = None,
        headers: Optional[List[str]] = None,
-        features: Optional[List[str]] = None,
+        features: Optional[str] = None,


This is not a backward compatible change

The mypy type was incorrect. When schema validation is done, it's required to be a str, so it would have failed anyways: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/clarify.py#L62

Chasing back the changes: 853e4d8#diff-5742d4f1aec0dedeca53accab9a77e0a63cb45e6044419975002be003357df82L68
It was accidentally being changed to List[str], which should be str.

Better re-examine all these types since it's not being added by our team. @spoorn

navinsoni · 2023-03-02T18:55:53Z

src/sagemaker/clarify.py

        if dataset_type not in [
            "text/csv",
            "application/jsonlines",
+            "application/json",
            "application/x-parquet",
            "application/x-image",
        ]:
            raise ValueError(
                f"Invalid dataset_type '{dataset_type}'."
                f" Please check the API documentation for the supported dataset types."
            )
-        # parameters for analysis on datasets without facets are only supported for CSV datasets
-        if dataset_type != "text/csv":
+        if dataset_type not in [
+            "text/csv",
+            "application/jsonlines",
+            "application/json",
+            "application/x-parquet",
+        ]:
            if predicted_label:
                raise ValueError(
                    f"The parameter 'predicted_label' is not supported"


Check on line 394 do not make sense to me. Can you please explain it.

predicted_label is the model predicted labels that the user can optionally provide. excluded_columns are the column(s) that are excluded from model inference and the Clarify report/computations. These are only supported for tabular datasets, thus we use an inclusive allowlist here and exclude other datasets such as images.

I added a comment to the code

sagemaker-bot · 2023-03-02T21:18:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: cf15c0c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T21:48:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: cf15c0c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T21:59:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: cf15c0c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T22:06:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: cf15c0c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T22:09:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: cf15c0c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T23:39:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a5f5a47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T23:49:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a5f5a47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T23:57:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a5f5a47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-02T23:59:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a5f5a47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T00:40:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a5f5a47
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T01:57:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 031e463
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T02:06:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 031e463
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T02:14:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 031e463
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T02:15:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 031e463
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T02:21:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 031e463
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T03:45:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b2f536b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T04:16:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b2f536b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T04:25:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b2f536b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T04:29:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b2f536b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T04:35:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b2f536b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T23:13:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: aa1dfff
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T23:20:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a9e058a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-03T23:51:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a9e058a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-04T00:01:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a9e058a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-04T00:05:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a9e058a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-04T00:22:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a9e058a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

/bot run slow-tests

sagemaker-bot · 2023-03-08T23:45:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a9e058a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

spoorn · 2023-03-08T23:54:10Z

/bot run slow-tests

sagemaker-bot · 2023-03-09T00:42:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3488d5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-09T00:47:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 3488d5d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-09T00:48:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3488d5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-09T00:50:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3488d5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-09T02:47:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3488d5d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

/bot run pr,slow-tests

sagemaker-bot · 2023-03-09T18:19:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 3488d5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-09T20:41:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3488d5d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Michael Trinh <[email protected]>

spoorn requested a review from a team as a code owner February 28, 2023 19:08

spoorn requested review from navinsoni and removed request for a team February 28, 2023 19:08

xiaoyi-cheng approved these changes Mar 1, 2023

View reviewed changes

larroy approved these changes Mar 2, 2023

View reviewed changes

navinsoni reviewed Mar 2, 2023

View reviewed changes

spoorn force-pushed the jsonio branch from 30dc6c7 to cf15c0c Compare March 2, 2023 21:12

spoorn force-pushed the jsonio branch from cf15c0c to a5f5a47 Compare March 2, 2023 23:03

feature: support JSON for input dataset and model output

aa1dfff

spoorn force-pushed the jsonio branch from b2f536b to aa1dfff Compare March 3, 2023 23:07

Merge branch 'master' into jsonio

a9e058a

navinsoni approved these changes Mar 4, 2023

View reviewed changes

navinsoni reviewed Mar 8, 2023

View reviewed changes

Merge branch 'master' into jsonio

3488d5d

navinsoni reviewed Mar 9, 2023

View reviewed changes

navinsoni merged commit 7f823e1 into aws:master Mar 9, 2023

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: support JSON for input dataset and model output (aws#3687)

36287c1

Co-authored-by: Michael Trinh <[email protected]>

feature: support JSON for input dataset and model output #3687

feature: support JSON for input dataset and model output #3687

Uh oh!

Conversation

spoorn commented Feb 28, 2023

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Feb 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

Uh oh!

xiaoyi-cheng left a comment

Choose a reason for hiding this comment

Uh oh!

navinsoni Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

spoorn Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

xiaoyi-cheng Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinsoni Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

spoorn Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2023

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Feb 28, 2023 •

edited

Loading

xiaoyi-cheng Mar 2, 2023 •

edited

Loading