feature: added _AnalysisConfigGenerator for clarify #3271

dosatos · 2022-08-01T09:25:33Z

Issue #, if available:

Description of changes:

Added _AnalysisConfigGenerator class to simplify maintainability of the config creation.
- Also, with this generator, customers may reproduce the state of the config locally during debugging (i.e. without a need to opening it in the S3 bucket)
Simplified job_name creation part
Added a private method _last_analysis_config) to store last analysis_config used for a run (to be useful in case of debugging)
Extended the std out with the analysis config used for a run. Currently it already prints other values, like Job Name, Inputs, and Outputs:

Job Name:  Clarify-Bias-2022-07-22-09-00-31-294
Inputs:  [{...}]
Outputs:  [...}]

Testing done:
Unit Tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

larroy

Two comments:

Can you please add types?
Can you please test a config including pre-post-explainability all at once?

Can the generator synthetise the config for all the methods enabled + explainability?

dosatos · 2022-08-03T09:28:53Z

Can you please add types?

Done, also created a backlog item to address this for the other/existing methods.

Can you please test a config including pre-post-explainability all at once?
Can the generator synthetise the config for all the methods enabled + explainability?

So far there is no API to call for both Bias and Explainability at once. So I cannot.
(Maybe I did not get the question?)

larroy · 2022-08-04T16:12:31Z

weird, but analysis config supports having a job with explainability and bias at once, how is it that we can't do it with the SDK? or am I missing something?

larroy

Changes look fine, one of my question is not addressed, but it should not block your PR. Thank you for your changes, LGTM.

src/sagemaker/clarify.py

keerthanvasist

Changes LGTM

sagemaker-bot · 2022-08-05T19:18:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ea9ea51
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-05T20:08:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ea9ea51
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-05T20:11:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ea9ea51
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-05T20:16:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ea9ea51
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-05T20:20:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ea9ea51
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T17:17:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 51d42e9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T17:58:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 51d42e9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:21:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 51d42e9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:22:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 51d42e9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-11T18:22:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 51d42e9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T10:23:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 24bd02e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T11:10:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 24bd02e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T11:22:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 24bd02e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T11:26:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 24bd02e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T11:32:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 24bd02e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:01:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b50a44b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-08-12T13:02:38Z

Codecov Report

Merging #3271 (b50a44b) into master (284ddbe) will decrease coverage by 0.79%.
The diff coverage is 97.67%.

@@            Coverage Diff             @@
##           master    #3271      +/-   ##
==========================================
- Coverage   89.82%   89.03%   -0.80%     
==========================================
  Files         645      203     -442     
  Lines       55518    18280   -37238     
==========================================
- Hits        49871    16276   -33595     
+ Misses       5647     2004    -3643

Impacted Files	Coverage Δ
src/sagemaker/clarify.py	`95.52% <97.67%> (ø)`
...python3.9/site-packages/sagemaker/xgboost/utils.py
...e-packages/sagemaker/automl/candidate_estimator.py
...n3.9/site-packages/sagemaker/wrangler/ingestion.py
...python3.7/site-packages/sagemaker/pytorch/model.py
...on3.8/site-packages/sagemaker/amazon/validation.py
...ite-packages/sagemaker/jumpstart/notebook_utils.py
.../sagemaker/huggingface/training_compiler/config.py
...thon3.7/site-packages/sagemaker/utilities/cache.py
.../cli/compatibility/v2/modifiers/training_params.py
... and 839 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-08-12T13:17:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b50a44b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:21:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b50a44b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:36:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b50a44b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-12T13:38:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b50a44b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

larroy

lgtm

shreyapandit · 2022-08-15T23:54:10Z

PR check for serverless inference failing because of known backend issue, will override and merge this to unblock team.

sagemaker-bot · 2022-08-16T01:03:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b50a44b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* feature: extracted analysis config generation for explainability * feature: extracted analysis config generation for bias pre_training * feature: extracted analysis config generation for bias post_training * feature: extracted analysis config generation for bias * feature: simplified job_name creation * feature: extended analysis config generator methods with common logic * feature: refactored _AnalysisConfigGenerator methods * feature: added _last_analysis_config in SageMakerClarifyProcessor * added data types in _AnalysisConfigGenerator methods * applied style formatting to fix build issues Co-authored-by: Yeldos Balgabekov <[email protected]>

dosatos force-pushed the clarify/analyis-config-locally branch 5 times, most recently from e90a3ea to 7f00235 Compare August 1, 2022 11:53

dosatos changed the title ~~feature: refactorred clarify analysis config genreation for SageMaker…~~ feature: added _AnalysisConfigGenerator for clarify Aug 1, 2022

larroy reviewed Aug 1, 2022

View reviewed changes

dosatos force-pushed the clarify/analyis-config-locally branch from e502748 to 3cefaef Compare August 3, 2022 09:19

larroy approved these changes Aug 4, 2022

View reviewed changes

navinsoni reviewed Aug 4, 2022

View reviewed changes

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

dosatos force-pushed the clarify/analyis-config-locally branch from 3cefaef to 1a51f96 Compare August 4, 2022 17:45

keerthanvasist approved these changes Aug 4, 2022

View reviewed changes

navinsoni approved these changes Aug 4, 2022

View reviewed changes

navinsoni force-pushed the clarify/analyis-config-locally branch from 1a51f96 to ea9ea51 Compare August 4, 2022 18:52

dosatos force-pushed the clarify/analyis-config-locally branch from fb68b4a to 1ecdbd4 Compare August 8, 2022 12:43

dosatos mentioned this pull request Aug 8, 2022

Added bias_and_explainability to SageMakerClarifyProcessor #3282

Closed

9 tasks

aws-byeldos added 7 commits August 8, 2022 14:53

feature: extracted analysis config generation for explainability

c0f1d79

feature: extracted analysis config generation for bias pre_training

4fa12c1

feature: extracted analysis config generation for bias post_training

cf4b08e

feature: extracted analysis config generation for bias

3b2df62

feature: simplified job_name creation

32650ee

feature: extended analysis config generator methods with common logic

88b1f4d

feature: refactored _AnalysisConfigGenerator methods

71989f9

aws-byeldos added 2 commits August 8, 2022 14:53

feature: added _last_analysis_config in SageMakerClarifyProcessor

e1fb7a2

added data types in _AnalysisConfigGenerator methods

ee04c97

navinsoni force-pushed the clarify/analyis-config-locally branch from 1ecdbd4 to a5cbe0e Compare August 8, 2022 21:53

dosatos force-pushed the clarify/analyis-config-locally branch from 51d42e9 to 24bd02e Compare August 12, 2022 10:18

applied style formatting to fix build issues

b50a44b

dosatos force-pushed the clarify/analyis-config-locally branch from 24bd02e to b50a44b Compare August 12, 2022 12:31

larroy approved these changes Aug 15, 2022

View reviewed changes

shreyapandit merged commit 8fa7650 into aws:master Aug 15, 2022

feature: added _AnalysisConfigGenerator for clarify #3271

feature: added _AnalysisConfigGenerator for clarify #3271

Uh oh!

Conversation

dosatos commented Aug 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

larroy left a comment

Choose a reason for hiding this comment

Uh oh!

dosatos commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larroy commented Aug 4, 2022

Uh oh!

larroy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

keerthanvasist left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Aug 5, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 5, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 5, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 5, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 5, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 11, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Aug 12, 2022

Codecov Report

Uh oh!

sagemaker-bot commented Aug 12, 2022

AWS CodeBuild CI Report

Uh oh!

dosatos commented Aug 1, 2022 •

edited

Loading

dosatos commented Aug 3, 2022 •

edited

Loading