feature: feature store with_feature_group functionality changes #3630

patrickmcarlos · 2023-02-03T01:14:04Z

Issue #, if available:

Description of changes:
Adds three new parameters to the with_feature_group Feature Store dataset_builder class.

These three new parameters are:

        feature_name_in_target (str): A string representing the feature in the target feature group
            that will be compared to the target feature in the base feature group. If None is
            provided, the record identifier will be used in the join statement. (default: None).
        join_comparator (JoinComparatorEnum): A JoinComparatorEnum representing the comparator used
            when joining the target feature in the base feature group and the feature in the target
            feature group (default: None).
        join_type (JoinTypeEnum): A JoinTypeEnum representing the type of join between the base and
            target feature groups. (default: None).

feature_name_in_target

Allows users to provide the feature name in the target feature group that will be used as a join key alongside the currently existing target_feature_name_in_base feature in the SQL JOIN statement. Before, only the record identifier feature in the target feature group could be used when creating the query.

join_comparator

Allows user to provide a Join comparator, which will be used to compare the two join keys. For example, we can now join a row together if feature_1 (target_feature_name_in_base) in feature group A is greater than feature_1 (feature_name_in_target). Before, only = (Equals) was able to be used.

join_type

Allows user to provide a Join type, which will specify the type of Join in the SQL statement. Before, users could only specify an (INNER) JOIN.
Enum keys are based on the join types available in the Athena docs https://docs.aws.amazon.com/athena/latest/ug/select.html

Usage

Say we have two feature groups, feature_group_a, and feature_group_b.

feature_group_a has three features, rec-id-a, event-time-id-a and feature-a.
feature_group_b has three features, rec-id-b, event-time-id-b and feature-b.

To join these feature groups together, a user might write the code:

dataset_builder = feature_store.create_dataset(
    base=feature_group_a,
    output_path=f"s3://{s3_bucket_name}",
)

result_df, query = dataset_builder.with_feature_group(
    feature_group=feature_group_b,
    target_feature_name_in_base="feature-a",
).to_dataframe()

However, a user may want to join feature-a from feature_group_a (base) and feature-b from feature_group_b (target). Furthermore, the user wants to perform a RIGHT_JOIN instead of a regular JOIN (inner) and use the LESS_THAN comparator instead of equals

Our code could look like this:

result_df, query = dataset_builder.with_feature_group(
    feature_group=feature_group_b,
    target_feature_name_in_base="feature-a",
    feature_name_in_target="feature-b"
    join_comparator=JoinComparatorEnum.LESS_THAN,
    join_type=JoinTypeEnum.RIGHT_JOIN).to_dataframe()

or without named arguments

result_df, query = dataset_builder.with_feature_group(
    feature_group_b,
    "feature-a",
    None,
    "feature-b"
    JoinComparatorEnum.LESS_THAN,
    JoinTypeEnum.RIGHT_JOIN).to_dataframe()

Testing done:

Unit & integration test to ensure properties of dataset_builder class are correctly updated when the new parameters are used in with_feature_group
Unit & integration test for the SQL query generated when using the with_feature_group with the new parameters.
Currently existing unit tests are unchanged.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

patrickmcarlos · 2023-02-08T07:20:35Z

/bot run all

trajanikant

/bot run all

sagemaker-bot · 2023-02-08T20:23:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0a9fbec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-08T21:07:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0a9fbec
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-08T21:27:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0a9fbec
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-08T21:33:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0a9fbec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-09T00:15:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0a9fbec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run all

sagemaker-bot · 2023-02-09T01:48:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 41cde1e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-09T01:55:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 41cde1e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-02-09T01:56:50Z

Codecov Report

Merging #3630 (41cde1e) into master (58fe72a) will decrease coverage by 0.78%.
The diff coverage is 95.23%.

@@            Coverage Diff             @@
##           master    #3630      +/-   ##
==========================================
- Coverage   89.56%   88.78%   -0.78%     
==========================================
  Files         968      227     -741     
  Lines       89844    22237   -67607     
==========================================
- Hits        80470    19744   -60726     
+ Misses       9374     2493    -6881

Impacted Files	Coverage Δ
src/sagemaker/feature_store/dataset_builder.py	`87.80% <95.23%> (ø)`
...on3.7/site-packages/sagemaker/apiutils/__init__.py
...on3.7/site-packages/sagemaker/debugger/debugger.py
...n3.7/site-packages/sagemaker/workflow/functions.py
...3.9/site-packages/sagemaker/wrangler/processing.py
...n3.10/site-packages/sagemaker/workflow/pipeline.py
...ython3.9/site-packages/sagemaker/lineage/_utils.py
...hon3.9/site-packages/sagemaker/sklearn/defaults.py
...-packages/sagemaker/workflow/parallelism_config.py
.../site-packages/sagemaker/model_monitor/__init__.py
... and 1186 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-02-09T01:57:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-09T02:13:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 41cde1e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-09T03:52:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

patrickmcarlos · 2023-02-09T17:23:19Z

Build Logs

Looks the failing Feature Store related sagemaker-python-sdk-pr tests are due to hitting the max # of feature groups in the AWS account. This causes a failed DeleteFeatureGroup request during the cleanup stage as well. This will require cleaning up these feature groups and rerunning the sagemaker-python-sdk-pr and sagemaker-python-sdk-slow-tests

trajanikant

/bot run pr, slow-tests

sagemaker-bot · 2023-02-09T17:37:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-09T20:24:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run pr, slow-tests

sagemaker-bot · 2023-02-11T00:25:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-11T05:52:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run pr, slow-tests

sagemaker-bot · 2023-02-14T11:37:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-14T11:55:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 41cde1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…es (#3630)" This reverts commit 5dba70c.

…3630)

patrickmcarlos added 5 commits February 2, 2023 22:42

updates with_feature-group def and adds test

50b8010

applies lint

247de3b

adds integration test

aacd12b

adds additional unit test for dataset builder object

e6e6ad5

adds docstrings, lints

f34ac66

patrickmcarlos changed the title ~~dnm: join functionality changes~~ feature: join functionality changes Feb 8, 2023

patrickmcarlos marked this pull request as ready for review February 8, 2023 01:16

patrickmcarlos requested a review from a team as a code owner February 8, 2023 01:16

patrickmcarlos requested review from mufaddal-rohawala and removed request for a team February 8, 2023 01:16

patrickmcarlos changed the title ~~feature: join functionality changes~~ feature: feature store with_feature_group functionality changes Feb 8, 2023

runs lint for test_dataset_builder

0a9fbec

patrickmcarlos force-pushed the patmcar-improve-join-functionality branch from 56fc682 to 0a9fbec Compare February 8, 2023 07:13

trajanikant reviewed Feb 8, 2023

View reviewed changes

additional docstring changes

282366f

use enum types for join type and comparator params

41cde1e

trajanikant reviewed Feb 9, 2023

View reviewed changes

trajanikant reviewed Feb 10, 2023

View reviewed changes

trajanikant reviewed Feb 14, 2023

View reviewed changes

trajanikant merged commit 5dba70c into aws:master Feb 14, 2023

trajanikant added a commit that referenced this pull request Feb 14, 2023

Revert "feature: feature store with_feature_group functionality chang…

e027f8b

…es (#3630)" This reverts commit 5dba70c.

trajanikant mentioned this pull request Feb 14, 2023

Revert "feature: feature store with_feature_group functionality changes" #3657

Merged

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

feature: feature store with_feature_group functionality changes (aws#…

cc9093c

…3630)

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: feature store with_feature_group functionality changes (aws#…

850fc98

…3630)

feature: feature store with_feature_group functionality changes #3630

feature: feature store with_feature_group functionality changes #3630

Uh oh!

Conversation

patrickmcarlos commented Feb 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Merge Checklist

General

Tests

Uh oh!

patrickmcarlos commented Feb 8, 2023

Uh oh!

trajanikant left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 8, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

trajanikant left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

patrickmcarlos commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trajanikant left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2023

AWS CodeBuild CI Report

Uh oh!

trajanikant left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 11, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 11, 2023

AWS CodeBuild CI Report

Uh oh!

trajanikant left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 14, 2023

AWS CodeBuild CI Report

Uh oh!

patrickmcarlos commented Feb 3, 2023 •

edited

Loading

codecov-commenter commented Feb 9, 2023 •

edited

Loading

patrickmcarlos commented Feb 9, 2023 •

edited

Loading