change: Update BiasConfig to accept multiple facet params #2243

georschi · 2021-03-25T11:34:35Z

BiasConfig will now accept a list of feature/attribute names to perform the bias analysis. This is already supported by the service and with this update the SDK will be able to make use of it.

Issue #, if available:

NA

Description of changes:

This change allows the user to pass in a list of attributes against which they wish to run a bias analysis. It extends the current implementation (remaining backwards compatible) by checking for the variable type and branching to write behaviour / way of parsing the information.

Prior to this, to achieve the same, the user would need to do:

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name='Sex',
                                facet_values_or_threshold=[0],
                                group_name='Age')
example.analysis_config['facet'].append({'name_or_index':'Ethnic group'})

The equivalent to the above now is:

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name=['Sex','Ethnic group'],
                                facet_values_or_threshold=[[0], None],
                                group_name='Age')

Testing done:

Tested for usecases:
original to check for compatibility

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name='Sex',
                                facet_values_or_threshold=[0],
                                group_name='Age')
print(example.get_config())

multiple facet names

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name=['Sex','Ethnic group'],
                                facet_values_or_threshold=[[0], [1,2]],
                                group_name='Age')
print(example.get_config())

multiple facet names with subset of those defining a facet value

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name=['Sex','Ethnic group'],
                                facet_values_or_threshold=[[0], None],
                                group_name='Age')
print(example.get_config())

multiple facet names with no facet value provided

example = BiasConfig(label_values_or_threshold=[1],
                                facet_name=['Sex','Ethnic group', 'age'],
                                group_name='Age')
print(example.get_config())

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

BiasConfig will now accept a list of feature/attribute names to perform the bias analysis. This is already supported by the service and with this update the SDK will be able to make use of it.

sagemaker-bot · 2021-03-25T11:36:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7c8beec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:05:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 026c9ac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:08:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7c8beec
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:17:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 7c8beec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:21:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7c8beec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:36:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 026c9ac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:42:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 026c9ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

fixed long lines and trailing whitespace

sagemaker-bot · 2021-03-25T12:48:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 026c9ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T12:50:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 026c9ac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T13:08:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3ee71c7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T13:41:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 3ee71c7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T13:42:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3ee71c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T13:44:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3ee71c7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-25T13:53:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3ee71c7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ajaykarpur

Thanks for contributing @georschi. When you submit a pull request, you will also need to add and update tests for your change. Please read the CONTRIBUTING file for details: https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md#run-the-unit-tests

ajaykarpur · 2021-04-02T19:14:39Z

src/sagemaker/clarify.py

@@ -88,21 +88,34 @@ def __init__(
        Args:
            label_values_or_threshold (Any): List of label values or threshold to indicate positive
                outcome used for bias metrics.
-            facet_name (str): Sensitive attribute in the input data for which we like to compare
-                metrics.
+            facet_name (Any): String or List of strings of sensitive attribute(s) in the input data


Isn't the type of this parameter now str or [str], not Any?

You are right about this, but I changed to "Any" instead of "str or [str]" to be consistent with the way that this is defined for other parameters. For example on the line just above, label_values_or_threshold is of type Any when it could take a limited number of types.
I will be updating as suggested shortly

sagemaker-bot · 2021-04-06T13:27:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 918fc1b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

georschi · 2021-04-06T13:30:21Z

Thanks for contributing @georschi. When you submit a pull request, you will also need to add and update tests for your change. Please read the CONTRIBUTING file for details: https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md#run-the-unit-tests

Thank you for looking into this! I have now added additional unittests to cover the extra cases that are now supported. Please let me know if any other changes would be required.

sagemaker-bot · 2021-04-06T13:31:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d11246d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:00:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 918fc1b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:01:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 918fc1b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:05:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d11246d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:07:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d11246d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:09:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 918fc1b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-06T14:12:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d11246d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-08T14:37:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1695db6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-08T15:13:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1695db6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-08T15:16:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 1695db6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-08T15:19:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1695db6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-08T15:31:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 1695db6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

larroy

Approach LGTM, is not strongly typed but in similar style to numpy and pandas seems easy to use and understand.

sagemaker-bot · 2021-07-09T09:50:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 18e322a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T10:17:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 18e322a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T10:17:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 18e322a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T10:25:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 18e322a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T10:25:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 18e322a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T16:03:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0498077
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T16:26:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0498077
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T16:29:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0498077
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T16:34:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0498077
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-09T16:36:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0498077
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-15T18:01:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 93f9ba9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-15T18:24:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 93f9ba9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-15T18:27:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 93f9ba9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-15T18:32:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 93f9ba9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-15T18:34:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 93f9ba9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Update BiasConfig to accept multiple facet params

7c8beec

BiasConfig will now accept a list of feature/attribute names to perform the bias analysis. This is already supported by the service and with this update the SDK will be able to make use of it.

fix: typo in the file change

026c9ac

fix: flake8 format enforcement

3ee71c7

fixed long lines and trailing whitespace

ajaykarpur suggested changes Apr 2, 2021

View reviewed changes

ajaykarpur reviewed Apr 2, 2021

View reviewed changes

added unit tests for change in biasConfig

918fc1b

fix: update param type in docstring

d11246d

Merge branch 'master' into patch-1

1695db6

larroy previously approved these changes Jul 8, 2021

View reviewed changes

xiaoyi-cheng mentioned this pull request Jul 8, 2021

feature: support multiple facets for Clarify #2494

Closed

7 tasks

pylint fix to docstring length

18e322a

georschi dismissed larroy’s stale review via 18e322a July 9, 2021 09:40

Merge branch 'master' into patch-1

0498077

ahsan-z-khan approved these changes Jul 9, 2021

View reviewed changes

ajaykarpur approved these changes Jul 13, 2021

View reviewed changes

Merge branch 'master' into patch-1

93f9ba9

ahsan-z-khan merged commit 87b634c into aws:master Jul 15, 2021

change: Update BiasConfig to accept multiple facet params #2243

change: Update BiasConfig to accept multiple facet params #2243

Uh oh!

Conversation

georschi commented Mar 25, 2021

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 25, 2021

AWS CodeBuild CI Report

Uh oh!

ajaykarpur left a comment

Choose a reason for hiding this comment

Uh oh!

ajaykarpur Apr 2, 2021

Choose a reason for hiding this comment

Uh oh!

georschi Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!

georschi commented Apr 6, 2021

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 6, 2021

AWS CodeBuild CI Report

Uh oh!