feature: support clarify bias detection when facets not included #3221

samsaranc · 2022-07-08T23:28:16Z

Issue #, if available:
https://sim.amazon.com/issues/RAI-1197

Summary of changes

feature: support running bias detection with Clarify when facets are not included in the input dataset (and passed through as other parameters).

documentation: add clarify DataConfig params for facet not included in input dataset
documentation: add clarify DataConfig param for analysis with predicted labels
documentation: add clarify DataConfig param for analysis with excluded columns
documentation: correct DataConfig label parameter description
documentation: add details about running SHAP and PDP to run_explainability

Testing done:

unit tests passed
successfully ran tox commands for sphinx, twine, docstyle, flake8, pylint, black-check
created integration tests for new parameters & re-ran test_clarify.py integration tests
created unit tests validating the new parameters

Integration tests added

Pre-training bias analysis
- separate facet dataset (in 1 files), label, joinsource
Post-training bias analysis
- run inference with facets not included in input dataset & facet datasets in multiple files (with joinsource)
- run inference with excluded columns (but facets in input dataset), no joinsource

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-08T23:32:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d037928
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-08T23:45:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c224bd2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/clarify.py

keerthanvasist · 2022-07-08T23:59:59Z

tests/integ/test_clarify.py

+                    "value"
+                ]
+            )
+            <= 1.0


If you set the seed in SHAPConfig, you might be able to assert exact values.

We're not running SHAP or PDP, and the parameters of run_pretraining_bias in clarify.py don't currently support an input SHAPConfig, so I don't think that solution would work here.

sagemaker-bot · 2022-07-09T00:16:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d037928
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:19:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c224bd2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:20:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d037928
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:22:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c224bd2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:36:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d037928
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:45:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c224bd2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T00:45:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a0cdf69
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-09T00:46:09Z

Codecov Report

Merging #3221 (e973bef) into master (011539a) will decrease coverage by 0.80%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3221      +/-   ##
==========================================
- Coverage   89.61%   88.80%   -0.81%     
==========================================
  Files         645      203     -442     
  Lines       54585    17999   -36586     
==========================================
- Hits        48914    15984   -32930     
+ Misses       5671     2015    -3656

Impacted Files	Coverage Δ
src/sagemaker/clarify.py	`95.43% <100.00%> (ø)`
...thon3.7/site-packages/sagemaker/mxnet/estimator.py
...maker/cli/compatibility/v2/modifiers/image_uris.py
...b/python3.7/site-packages/sagemaker/transformer.py
...ython3.7/site-packages/sagemaker/mxnet/defaults.py
...ython3.7/site-packages/sagemaker/local/__init__.py
...on3.8/site-packages/sagemaker/amazon/object2vec.py
.../sagemaker/huggingface/training_compiler/config.py
.../lib/python3.8/site-packages/sagemaker/pipeline.py
...3.7/site-packages/sagemaker/workflow/model_step.py
... and 839 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 011539a...e973bef. Read the comment docs.

sagemaker-bot · 2022-07-09T01:07:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a0cdf69
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T01:19:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a0cdf69
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T01:26:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a0cdf69
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-09T01:34:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a0cdf69
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-10T20:25:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 424d90e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-10T20:48:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 424d90e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-10T21:04:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 424d90e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-10T21:24:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 424d90e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-10T22:51:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 424d90e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-14T23:47:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 424d90e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T00:52:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6fc0517
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:14:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 76a09dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:23:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d9a7cb0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:28:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d9a7cb0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:31:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 76a09dd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:35:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 76a09dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:45:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d9a7cb0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:52:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6fc0517
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:53:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 76a09dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T03:36:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6fc0517
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T03:40:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 76a09dd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-18T19:18:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 76a09dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-18T20:31:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 76a09dd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T06:25:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 76a09dd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

goswamig · 2022-07-26T20:45:21Z

@navinsoni Can we get this merge ?

* feature: clarify bias detection when facets are not included in the input dataset (and passed through as other parameters) * feature: clarify model inference with excluded columns in the input dataset * feature: clarify bias analysis with input predicted label dataset (supports post-training bias analysis without model inference API calls) * documentation: add clarify DataConfig params for facet not included in input dataset * documentation: add clarify DataConfig param for analysis with predicted labels * documentation: add clarify DataConfig param for analysis with excluded columns * documentation: correct DataConfig label parameter description * documentation: add details about running SHAP and PDP to run_explainability

sagemaker-bot · 2022-07-26T21:58:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e973bef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni · 2022-07-26T22:09:39Z

@navinsoni Can we get this merge ?

@goswamig we are ready for the merge. we just have to wait for the tests to pass

sagemaker-bot · 2022-07-26T22:19:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e973bef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T22:21:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e973bef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T22:41:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e973bef
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T23:01:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e973bef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T00:19:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e973bef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…#3221)

samsaranc force-pushed the bias_without_facets branch from d037928 to c224bd2 Compare July 8, 2022 23:36

keerthanvasist suggested changes Jul 8, 2022

View reviewed changes

src/sagemaker/clarify.py Show resolved Hide resolved

src/sagemaker/clarify.py Show resolved Hide resolved

keerthanvasist reviewed Jul 8, 2022

View reviewed changes

samsaranc force-pushed the bias_without_facets branch from c224bd2 to a0cdf69 Compare July 9, 2022 00:18

samsaranc force-pushed the bias_without_facets branch from a0cdf69 to 424d90e Compare July 10, 2022 19:58

keerthanvasist approved these changes Jul 14, 2022

View reviewed changes

samsaranc force-pushed the bias_without_facets branch 3 times, most recently from 6fc0517 to 76a09dd Compare July 16, 2022 00:47

navinsoni approved these changes Jul 18, 2022

View reviewed changes

navinsoni force-pushed the bias_without_facets branch from 76a09dd to 7ec1510 Compare July 26, 2022 16:57

mufaddal-rohawala closed this Jul 26, 2022

mufaddal-rohawala reopened this Jul 26, 2022

samsaranc force-pushed the bias_without_facets branch from 7ec1510 to e973bef Compare July 26, 2022 21:31

navinsoni merged commit 58dd338 into aws:master Jul 27, 2022

knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request Aug 4, 2022

feature: support clarify bias detection when facets not included (aws…

bf9d11a

…#3221)

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

feature: support clarify bias detection when facets not included (aws…

fb5344c

…#3221)

feature: support clarify bias detection when facets not included #3221

feature: support clarify bias detection when facets not included #3221

Uh oh!

Conversation

samsaranc commented Jul 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Testing done:

Integration tests added

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jul 8, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 8, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

keerthanvasist Jul 8, 2022

Choose a reason for hiding this comment

Uh oh!

samsaranc Jul 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Jul 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 10, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 10, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 10, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 10, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 10, 2022

AWS CodeBuild CI Report

Uh oh!

samsaranc commented Jul 8, 2022 •

edited

Loading

samsaranc Jul 10, 2022 •

edited

Loading

codecov-commenter commented Jul 9, 2022 •

edited

Loading