feat: Adding scan and tagging utility #4499

chrstfu · 2024-03-13T22:43:02Z

Issue #, if available:

Description of changes:

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

src/sagemaker/jumpstart/curated_hub/curated_hub.py

mufaddal-rohawala · 2024-03-14T00:15:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 5f8119b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/jumpstart/curated_hub/curated_hub.py

src/sagemaker/jumpstart/curated_hub/utils.py

tests/unit/sagemaker/jumpstart/curated_hub/test_utils.py

src/sagemaker/jumpstart/curated_hub/types.py

bencrabtree

Nice revision Chris! Few questions but I think this looks good overall. Can you please:

Make the tagging logic specific for unsupported models
Move the logging into the add_tags function

src/sagemaker/jumpstart/curated_hub/curated_hub.py

bencrabtree · 2024-03-15T17:25:16Z

src/sagemaker/jumpstart/curated_hub/curated_hub.py

-
-            # Model does not exist in Hub, sync
-            if not matched_model:
+            if not matched_model or Version(matched_model.hub_content_version) < Version(model.version):


Sorry, can you please bring this logic back? Is there a reason you removed it?

bencrabtree · 2024-03-15T17:27:26Z

src/sagemaker/jumpstart/curated_hub/types.py

@@ -19,6 +19,16 @@

 from sagemaker.jumpstart.types import JumpStartDataHolderType, JumpStartModelSpecs

+class CuratedHubTagName(str, Enum):


Hmm I still disagree. This enum should only be for UnsupportedFlags since it's trying to perform a specific job. If you want to make enums for all tags for CuratedHub models, you can create a new enum, then have a aggregator type

class CuratedHubUnsupportedFlag(str, Enum): ... class CuratedHubBlahBlah(str, Enum): ... CuratedHubTag = CuratedHubUnsupportedFlag | CuratedHubBlahBlah

I'd prefer if we didn't overthink this to be generalizable, and cross that bridge when we get there. This is a private beta, and we need to keep the logic rigid.

bencrabtree · 2024-03-15T17:30:19Z

src/sagemaker/jumpstart/curated_hub/types.py

+  key: CuratedHubTagName
+  value: str


Okay, I'll disagree and commit :). But if you're going to keep this, why not make the key/value be capitalized so it's compatible with sagemaker:AddTags?

bencrabtree · 2024-03-15T17:32:24Z

src/sagemaker/jumpstart/curated_hub/utils.py

+    )
+    hub_content_versions = list_versions_response["HubContentSummaries"]
+
+    tag_name_to_versions_map: Dict[CuratedHubTagName, List[str]] = {}


I still disagree here Chris. If you make everything super generalized it loses it's context and makes it really difficult for people to read and understand. Have you heard of any use cases where we want to add additional tags?

src/sagemaker/jumpstart/curated_hub/utils.py

tests/unit/sagemaker/jumpstart/curated_hub/test_curated_hub.py

src/sagemaker/jumpstart/curated_hub/utils.py

mufaddal-rohawala · 2024-03-15T19:00:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 554dd20
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/jumpstart/curated_hub/utils.py

mufaddal-rohawala · 2024-03-15T21:58:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6d5f599
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov · 2024-03-18T14:57:14Z

Codecov Report

Attention: Patch coverage is 79.13043% with 24 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@a632795). Click here to learn what that means.

Files	Patch %	Lines
src/sagemaker/jumpstart/curated_hub/curated_hub.py	34.37%	21 Missing ⚠️
src/sagemaker/jumpstart/cache.py	72.72%	3 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##             master-jumpstart-curated-hub    #4499   +/-   ##
===============================================================
  Coverage                                ?   87.15%           
===============================================================
  Files                                   ?      397           
  Lines                                   ?    37340           
  Branches                                ?        0           
===============================================================
  Hits                                    ?    32545           
  Misses                                  ?     4795           
  Partials                                ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mufaddal-rohawala · 2024-03-18T15:46:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 77d7b50
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/jumpstart/curated_hub/utils.py

mufaddal-rohawala · 2024-03-18T19:09:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 42efbaa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-03-18T21:56:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0deb20e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-03-18T23:37:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d21fd41
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-03-19T00:39:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6b107a2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-03-19T04:50:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: bd02739
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-03-19T16:35:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4c829c9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chrstfu added 9 commits March 12, 2024 17:35

fix: Adding new utils

73ce401

Merge branch 'curated_hub_tagris' into curated_hub_tagris_copy

fbd233f

feat: Adding Curated Hub scanning feature

6d43cbe

fix: Refactoring

2ec0ca5

fix: Removing test values

c1519b8

fix: Refactoring

c2d4a17

fix: removing lru cache temporarily

48d6325

fix: Adding unit tests

0947840

fix: renaming

5f8119b

jinyoung-lim reviewed Mar 13, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/curated_hub.py Outdated Show resolved Hide resolved

jinyoung-lim reviewed Mar 13, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/curated_hub.py Outdated Show resolved Hide resolved

bencrabtree suggested changes Mar 14, 2024

View reviewed changes

chrstfu added 6 commits March 14, 2024 16:57

fix: Initial refactorings

395e99c

fix: Adding more alterations

c957e71

fix: Addressing unit tests

453bed4

fix: Adding more unittests

9e703a9

fix: Add tests

7cda304

fix: Adding list to scan input

9dbe3d6

bencrabtree suggested changes Mar 15, 2024

View reviewed changes

fix: typo

554dd20

fix: Addressing naming comments

9255849

bencrabtree reviewed Mar 15, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/utils.py Outdated Show resolved Hide resolved

fix: changing from string

6d5f599

Merge branch 'master-jumpstart-curated-hub' into curated_hub_tagris_copy

77d7b50

fix: formatter

0e5d5b4

evakravi reviewed Mar 18, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/utils.py Outdated Show resolved Hide resolved

evakravi reviewed Mar 18, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/utils.py Outdated Show resolved Hide resolved

chrstfu added 2 commits March 18, 2024 16:56

fix: linter

4b82389

fix: linter

42efbaa

chrstfu added 3 commits March 18, 2024 19:57

fix: linters

a73b5b2

fix: linter

4252d86

fix: linting

0deb20e

fix: more linting :/

d21fd41

fix: linting

6b107a2

chrstfu and others added 4 commits March 19, 2024 02:03

fix: more linting :/

29a0740

Merge branch 'master-jumpstart-curated-hub' into curated_hub_tagris_copy

054a8cf

fix: linting

bddf5e3

fix: tests

bd02739

bencrabtree self-requested a review March 19, 2024 13:54

chrstfu added 3 commits March 19, 2024 14:26

fix: Adding __init__.py

b33a63b

fix: linting

e1da9fc

fix: linting

4c829c9

bencrabtree approved these changes Mar 19, 2024

View reviewed changes

chrstfu marked this pull request as ready for review March 19, 2024 17:32

chrstfu requested a review from a team as a code owner March 19, 2024 17:32

chrstfu requested review from knikure and removed request for a team March 19, 2024 17:32

benieric approved these changes Mar 19, 2024

View reviewed changes

benieric merged commit 74bbb09 into aws:master-jumpstart-curated-hub Mar 19, 2024

		@@ -19,6 +19,16 @@

		from sagemaker.jumpstart.types import JumpStartDataHolderType, JumpStartModelSpecs

		class CuratedHubTagName(str, Enum):

feat: Adding scan and tagging utility #4499

feat: Adding scan and tagging utility #4499

Uh oh!

Conversation

chrstfu commented Mar 13, 2024

Merge Checklist

General

Tests

Uh oh!

Uh oh!

Uh oh!

mufaddal-rohawala commented Mar 14, 2024

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bencrabtree left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bencrabtree Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

bencrabtree Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

bencrabtree Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

bencrabtree Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mufaddal-rohawala commented Mar 15, 2024

AWS CodeBuild CI Report

Uh oh!

Uh oh!

mufaddal-rohawala commented Mar 15, 2024

AWS CodeBuild CI Report

Uh oh!

codecov bot commented Mar 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mufaddal-rohawala commented Mar 18, 2024

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

mufaddal-rohawala commented Mar 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Mar 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Mar 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Mar 19, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Mar 19, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Mar 19, 2024

AWS CodeBuild CI Report

Uh oh!

Uh oh!

codecov bot commented Mar 18, 2024 •

edited

Loading