feat: Support selective pipeline execution for function step #4372

qidewenwhen · 2024-01-12T00:16:20Z

Issue #, if available:
Currently configuring selective pipeline execution w.r.t the step decorated functions would fail as such functions' output are uploaded to the s3 path: <root>/<pipeline_name>/<ExecutionVariables.PIPELINE_EXECUTION_ID>/<step_name>/results

The ExecutionVariables.PIPELINE_EXECUTION_ID is always pointing to the current execution, but in selective pipeline execution, we may need to retrieve step output from the previous execution, which leads to the issue of unable to retrieve the dependent step output.

Description of changes: Support selective pipeline execution for function step.
As in backend, the selective pipeline execution logics would replace the step properties with the source execution steps' output, in SDK we can replace the entire s3 path <root>/<pipeline_name>/<ExecutionVariables.PIPELINE_EXECUTION_ID>/<step_name>/results with the properties "Steps.{function_step.name}.OutputDataConfig.S3OutputPath".
Accordingly, two additional updates to the S3OutputPath properties are also required:

it should be a Join function:

Join(
                on="/",
                values=[
                    s3_base_uri,
                    ExecutionVariables.PIPELINE_EXECUTION_ID,
                    step_compilation_context.step_name,
                    "results",
                ],
            )

it should be added to the container args' --property_references field.

Additionally, some extra changes are made to keep it backward compatible:

add the -U flag behind pip install command so that if users specify dependency versions to be xxx >= yyy, < zzz, the more up to date version would be installed in the job container. As a result, if users bump up the SDK version locally, the SDK version in remote job would be able to catch up

Note: this won’t help to upgrade the SDK version in job container if users lock the version in req.txt. However, in this case, users have the responsibility to make sure the versions to be sync between local and remote job.
- I’ll submit a separate PR to add a warning message for such case

keep the old delayed return s3 uri resolver behavior though it's obsoleted to keep backward compatible.
Note: ideally, the new SDK deserializer/resolver should be able to parse the old serializer schema

Testing done: Integ tests
Tested two cases:

function step1 -> [select: function step2]
regular step -> [select: function step]

Note: For the third case:
3. function step -> [select: regular step]
it's not implemented nor tested because the backend currently has a strict validation on JsonGet s3_uri, which allows Parameters and ExecutionVariables and primitive types only for Join function.

Next steps:

loosen the backend validation logic on JsonGet s3_uri
add support for the third case function step -> [select: regular step] in SDK in a separate PR.
update the doc to remove selective execution from limitations section

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

mufaddal-rohawala · 2024-01-12T01:15:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 76119f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-12T01:21:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 76119f9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-12T01:22:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 76119f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-12T01:22:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 76119f9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-12T02:19:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 76119f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2024-01-12T02:20:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (086c946) 86.89% compared to head (e7e59d7) 86.83%.
Report is 9 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4372      +/-   ##
==========================================
- Coverage   86.89%   86.83%   -0.07%     
==========================================
  Files        1197      380     -817     
  Lines      106830    35234   -71596     
==========================================
- Hits        92830    30594   -62236     
+ Misses      14000     4640    -9360

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/sagemaker/remote_function/core/pipeline_variables.py

mufaddal-rohawala · 2024-01-13T03:05:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0e1758d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-13T03:55:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0e1758d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-13T03:58:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0e1758d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-13T04:32:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0e1758d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-13T04:49:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0e1758d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-17T22:08:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 260a13c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-17T22:10:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 260a13c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-17T22:30:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 260a13c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-17T23:08:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 260a13c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-17T23:12:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 260a13c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-18T02:37:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 83f090b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-18T02:39:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 83f090b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-18T02:41:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 83f090b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-19T22:03:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1a05d6b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure

/bot run local-mode-tests

mufaddal-rohawala · 2024-01-22T19:43:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1a05d6b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

qidewenwhen · 2024-01-22T22:18:19Z

Pending @navaj0 's review, which will be done by EOD PT 01/22

src/sagemaker/remote_function/core/pipeline_variables.py

src/sagemaker/workflow/pipeline.py

mufaddal-rohawala · 2024-01-23T19:46:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 20852a5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T19:50:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 20852a5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T19:53:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 20852a5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T21:13:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e7e59d7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T21:16:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e7e59d7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T21:16:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e7e59d7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T21:30:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e7e59d7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-01-23T22:13:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e7e59d7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure

/bot run pr

mufaddal-rohawala · 2024-01-24T00:31:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e7e59d7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ngestion. (#4413) * change: update image_uri_configs 12-13-2023 12:23:06 PST * change: update image_uri_configs 12-13-2023 14:04:54 PST * prepare release v2.200.1 * update development version to v2.200.2.dev0 * fix: Move func and args serialization of function step to step level (#4312) * fix: Add write permission to job output dirs for remote and step decorator running on non-root job user (#4325) * feat: Added update for model package (#4309) Co-authored-by: Keshav Chandak <[email protected]> * documentation: fix ModelBuilder sample notebook links (#4319) * feat: Use specific images for SMP v2 jobs (#4333) * Add check for smp lib * update syntax * Remove unused images * Update repo name and regions * Update account number * Update framework name and check for None distribution * Add unit tests for smp v2 uri * Check enabled * Remove logging * Add cuda version in uri * Update cu121 * Update syntax * Fix black check * Fix black --------- Co-authored-by: huilgolr <[email protected]> * Fix: Updated js mb compression logic - ModelBuilder (#4294) Co-authored-by: EC2 Default User <[email protected]> * documentation: SMP v2 doc updates (#1423) (#4336) * doc update for estimator distribution art * add note to the SMP doc and minor fixes * remove subnodes * rm all v1 content as documenting everything in aws docs * fix build errors * fix white spaces * rm smdistributed from TF estimator distribution * rm white spaces * add notes to TF estimator distribution * fix links * incorporate feedback * update example values * fix version numbers in the notes Co-authored-by: Miyoung <[email protected]> * prepare release v2.201.0 * update development version to v2.201.1.dev0 * Fix: Add additional model builder telemetry (#4334) * move telemetry code to public * add additional test --------- Co-authored-by: EC2 Default User <[email protected]> * feature: support remote debug for sagemaker training job (#4315) * feature: support remote debug for sagemaker training job * change: Replace update_remote_config with 2 helper methods for enable and disable respectively * change: add new argument enable_remote_debug to skip set of test_jumpstart_estimator_kwargs_match_parent_class * chore: add jumpstart support for remote debug --------- Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Evan Kravitz <[email protected]> * Update tblib constraint (#4317) * Fix: Fix job_objective type (#4303) * change: update image_uri_configs 12-21-2023 08:32:41 PST * prepare release v2.202.0 * update development version to v2.202.1.dev0 * Using logging instead of prints (#4133) * documentation: update issue template. (#4337) * change: update model path in local mode (#4296) * Update model path in local mode * Add test * change: update image_uri_configs 12-22-2023 06:17:35 PST * prepare release v2.202.1 * update development version to v2.202.2.dev0 * change: create role if needed in `get_execution_role` (#4323) * Create role if needed in get_execution_role * Add tests * Change: More pythonic tags (#4327) * Change: More pythonic tags * Fix broken tags * More tags formatting and add a test * Fix tests * Raise Exception for debug (#4344) Co-authored-by: Ruilian Gao <[email protected]> * Change: Allow extra_args to be passed to uploader (#4338) * Change: Allow extra_args to be passed to uploader * Fix tests * Black * Fix test * Change: Drop py2 tag from the wheel as we don't support Python 2 (#4343) * Disable failed test in IR (#4345) * Disable failed test in IR * Fix format --------- Co-authored-by: Ruilian Gao <[email protected]> * change: update image_uri_configs 12-25-2023 06:17:33 PST * feat: Supporting tbac in load_run (#4039) * feature: support local mode in SageMaker Studio (#1300) (#4347) * feature: support local mode in SageMaker Studio * chore: fix typo * chore: fix formatting * chore: revert changes for docker compose logs * chore: black-format * change: Use predtermined dns-allow-listed-hostname for Studio Local Support * add support for CodeEditor and JupyterLabs --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> * prepare release v2.203.0 * update development version to v2.203.1.dev0 * change: update image_uri_configs 12-29-2023 06:17:34 PST * query hf api for model md (#4346) Co-authored-by: EC2 Default User <[email protected]> * fix: skip failing integs (#4348) Co-authored-by: Mufaddal Rohawala <[email protected]> * change: TGI 1.3.3 (#4335) * prepare release v2.203.1 * update development version to v2.203.2.dev0 * feat: parallelize notebook search utils, add new operators (#4342) * feat: parallelize notebook search utils * chore: raise exception in notebook utils if thread has error * chore: improve variable name * fix: not passing region to get jumpstart bucket * chore: add sagemaker session to notebook utils * chore: address PR comments * feat: add support for includes, begins with, ends with * fix: pylint * feat: private util for model eula key * fix: unit tests, use verify_model_region_and_return_specs in notebook utils * Revert "feat: private util for model eula key" This reverts commit e2daefc. * chore: add search keywords to header * fix: change ConditionNot incorrect property Expression to Condition (#4351) * fix: Huggingface glue failing tests (#4367) * fix: Huggingface glue failing tests * fix: Sphinx doc build failure * fix: Huggingface glue failing tests * fix: failing sphinx tests * fix: failing sphinx tests * fix: failing black check * fix: sphinx doc errors * fix: sphinx doc errors * sphinx * black-format * sphinx * sphinx * sphinx --------- Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> * fix: Add PyTorch 2.1.0 SM Training DLC to UNSUPPORTED_DLC_IMAGE_FOR_SM_PARALLELISM list (#4356) * add 2.1 unsupported smddp * formatting * feat: Support custom repack model settings (#4328) * change: update sphinx version (#4377) * change: update sphinx version * Update sphinx * change: Updates for DJL 0.26.0 release (#4366) * change: TGI NeuronX (#4375) * TGI NeuronX * Update * Update * fix: add warning message for job-prefixed pipeline steps when no job name is provided (#4371) Co-authored-by: svia3 <[email protected]> * change: JumpStart - TLV region launch (#4379) * feat: add throughput management support for feature group (#4359) * feat: add throughput management support for feature group * documentation: add doc for feature group throughput config --------- Co-authored-by: Nilesh PS <[email protected]> * change: Enable galactus integ tests (#4376) * feat: Enable galactus integ tests * fix flake8 * fix doc8 * trying to see if it works with slow tests * small fixes in import error * fix missing import * try to remove some dependencies from requirement to see if pr test can be fixed * fix flake8 * Enable more tests * Add rerun annotation and further remove dependencies * comment out 2 integ tests * Remove local mode test for now * fix flake8 * prepare release v2.204.0 * update development version to v2.204.1.dev0 * fix: Add validation for empty ParameterString value in start local pipeline (#4354) * feat: Support selective pipeline execution for function step (#4372) * change: update image_uri_configs 01-24-2024 06:17:33 PST * fix: update get_execution_role_arn from metadata file if present (#4388) * fix: Support using PipelineDefinitionConfig in local mode (#4352) * fix: remove fastapi and uvicorn dependencies (#4365) They are not used in the codebase. Closes #4361 #4295 * prepare release v2.205.0 * update development version to v2.205.1.dev0 * change: TGI NeuronX 0.0.17 (#4390) * fix: Support PipelineVariable for ModelQualityCheckConfig attributes (#4353) * feat: Logic to detect hardware GPU count and aggregate GPU memory size in MiB (#4389) * Add logic to detect hardware GPU count and aggregate GPU memory size in MiB * Fix all formatting * Addressed PR review comments * Addressed PR Review messages * Addressed PR Review Messages * Addressed PR Review comments * Addressed PR Review Comments * Add integration tests * Add config * Fix integration tests * Include Instance Types GPU infor Config files * Addressed PR review comments * Fix unit tests * Fix unit test: 'Mock' object is not subscriptable --------- Co-authored-by: Jonathan Makunga <[email protected]> * fix: fixed create monitoring schedule failing after validation error (#4385) Co-authored-by: Keshav Chandak <[email protected]> * Add collection type support for Feaure Group Ingestion. Add TargetStores support for PutRecord and Ingestion. * Remove merge conflicts. * Update the feature definition type * Black formatting * Fix Flake8 formatting * Fix Pylint * Fix Formatting. --------- Co-authored-by: sagemaker-bot <[email protected]> Co-authored-by: ci <ci> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Co-authored-by: stacicho <[email protected]> Co-authored-by: Teng-xu <[email protected]> Co-authored-by: huilgolr <[email protected]> Co-authored-by: Gary Wang <[email protected]> Co-authored-by: EC2 Default User <[email protected]> Co-authored-by: akrishna1995 <[email protected]> Co-authored-by: Miyoung <[email protected]> Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Evan Kravitz <[email protected]> Co-authored-by: martinRenou <[email protected]> Co-authored-by: Duc Trung Le <[email protected]> Co-authored-by: ruiliann666 <[email protected]> Co-authored-by: Ruilian Gao <[email protected]> Co-authored-by: ananth102 <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: amzn-choeric <[email protected]> Co-authored-by: evakravi <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Sirut Buasai <[email protected]> Co-authored-by: Sindhu Somasundaram <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> Co-authored-by: Haixin Wang <[email protected]> Co-authored-by: Nilesh PS <[email protected]> Co-authored-by: Nilesh PS <[email protected]> Co-authored-by: jiapinw <[email protected]> Co-authored-by: Jay Goyani <[email protected]> Co-authored-by: Justin <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]>

…ngestion. (aws#4413) * change: update image_uri_configs 12-13-2023 12:23:06 PST * change: update image_uri_configs 12-13-2023 14:04:54 PST * prepare release v2.200.1 * update development version to v2.200.2.dev0 * fix: Move func and args serialization of function step to step level (aws#4312) * fix: Add write permission to job output dirs for remote and step decorator running on non-root job user (aws#4325) * feat: Added update for model package (aws#4309) Co-authored-by: Keshav Chandak <[email protected]> * documentation: fix ModelBuilder sample notebook links (aws#4319) * feat: Use specific images for SMP v2 jobs (aws#4333) * Add check for smp lib * update syntax * Remove unused images * Update repo name and regions * Update account number * Update framework name and check for None distribution * Add unit tests for smp v2 uri * Check enabled * Remove logging * Add cuda version in uri * Update cu121 * Update syntax * Fix black check * Fix black --------- Co-authored-by: huilgolr <[email protected]> * Fix: Updated js mb compression logic - ModelBuilder (aws#4294) Co-authored-by: EC2 Default User <[email protected]> * documentation: SMP v2 doc updates (aws#1423) (aws#4336) * doc update for estimator distribution art * add note to the SMP doc and minor fixes * remove subnodes * rm all v1 content as documenting everything in aws docs * fix build errors * fix white spaces * rm smdistributed from TF estimator distribution * rm white spaces * add notes to TF estimator distribution * fix links * incorporate feedback * update example values * fix version numbers in the notes Co-authored-by: Miyoung <[email protected]> * prepare release v2.201.0 * update development version to v2.201.1.dev0 * Fix: Add additional model builder telemetry (aws#4334) * move telemetry code to public * add additional test --------- Co-authored-by: EC2 Default User <[email protected]> * feature: support remote debug for sagemaker training job (aws#4315) * feature: support remote debug for sagemaker training job * change: Replace update_remote_config with 2 helper methods for enable and disable respectively * change: add new argument enable_remote_debug to skip set of test_jumpstart_estimator_kwargs_match_parent_class * chore: add jumpstart support for remote debug --------- Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Evan Kravitz <[email protected]> * Update tblib constraint (aws#4317) * Fix: Fix job_objective type (aws#4303) * change: update image_uri_configs 12-21-2023 08:32:41 PST * prepare release v2.202.0 * update development version to v2.202.1.dev0 * Using logging instead of prints (aws#4133) * documentation: update issue template. (aws#4337) * change: update model path in local mode (aws#4296) * Update model path in local mode * Add test * change: update image_uri_configs 12-22-2023 06:17:35 PST * prepare release v2.202.1 * update development version to v2.202.2.dev0 * change: create role if needed in `get_execution_role` (aws#4323) * Create role if needed in get_execution_role * Add tests * Change: More pythonic tags (aws#4327) * Change: More pythonic tags * Fix broken tags * More tags formatting and add a test * Fix tests * Raise Exception for debug (aws#4344) Co-authored-by: Ruilian Gao <[email protected]> * Change: Allow extra_args to be passed to uploader (aws#4338) * Change: Allow extra_args to be passed to uploader * Fix tests * Black * Fix test * Change: Drop py2 tag from the wheel as we don't support Python 2 (aws#4343) * Disable failed test in IR (aws#4345) * Disable failed test in IR * Fix format --------- Co-authored-by: Ruilian Gao <[email protected]> * change: update image_uri_configs 12-25-2023 06:17:33 PST * feat: Supporting tbac in load_run (aws#4039) * feature: support local mode in SageMaker Studio (aws#1300) (aws#4347) * feature: support local mode in SageMaker Studio * chore: fix typo * chore: fix formatting * chore: revert changes for docker compose logs * chore: black-format * change: Use predtermined dns-allow-listed-hostname for Studio Local Support * add support for CodeEditor and JupyterLabs --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> * prepare release v2.203.0 * update development version to v2.203.1.dev0 * change: update image_uri_configs 12-29-2023 06:17:34 PST * query hf api for model md (aws#4346) Co-authored-by: EC2 Default User <[email protected]> * fix: skip failing integs (aws#4348) Co-authored-by: Mufaddal Rohawala <[email protected]> * change: TGI 1.3.3 (aws#4335) * prepare release v2.203.1 * update development version to v2.203.2.dev0 * feat: parallelize notebook search utils, add new operators (aws#4342) * feat: parallelize notebook search utils * chore: raise exception in notebook utils if thread has error * chore: improve variable name * fix: not passing region to get jumpstart bucket * chore: add sagemaker session to notebook utils * chore: address PR comments * feat: add support for includes, begins with, ends with * fix: pylint * feat: private util for model eula key * fix: unit tests, use verify_model_region_and_return_specs in notebook utils * Revert "feat: private util for model eula key" This reverts commit e2daefc. * chore: add search keywords to header * fix: change ConditionNot incorrect property Expression to Condition (aws#4351) * fix: Huggingface glue failing tests (aws#4367) * fix: Huggingface glue failing tests * fix: Sphinx doc build failure * fix: Huggingface glue failing tests * fix: failing sphinx tests * fix: failing sphinx tests * fix: failing black check * fix: sphinx doc errors * fix: sphinx doc errors * sphinx * black-format * sphinx * sphinx * sphinx --------- Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> * fix: Add PyTorch 2.1.0 SM Training DLC to UNSUPPORTED_DLC_IMAGE_FOR_SM_PARALLELISM list (aws#4356) * add 2.1 unsupported smddp * formatting * feat: Support custom repack model settings (aws#4328) * change: update sphinx version (aws#4377) * change: update sphinx version * Update sphinx * change: Updates for DJL 0.26.0 release (aws#4366) * change: TGI NeuronX (aws#4375) * TGI NeuronX * Update * Update * fix: add warning message for job-prefixed pipeline steps when no job name is provided (aws#4371) Co-authored-by: svia3 <[email protected]> * change: JumpStart - TLV region launch (aws#4379) * feat: add throughput management support for feature group (aws#4359) * feat: add throughput management support for feature group * documentation: add doc for feature group throughput config --------- Co-authored-by: Nilesh PS <[email protected]> * change: Enable galactus integ tests (aws#4376) * feat: Enable galactus integ tests * fix flake8 * fix doc8 * trying to see if it works with slow tests * small fixes in import error * fix missing import * try to remove some dependencies from requirement to see if pr test can be fixed * fix flake8 * Enable more tests * Add rerun annotation and further remove dependencies * comment out 2 integ tests * Remove local mode test for now * fix flake8 * prepare release v2.204.0 * update development version to v2.204.1.dev0 * fix: Add validation for empty ParameterString value in start local pipeline (aws#4354) * feat: Support selective pipeline execution for function step (aws#4372) * change: update image_uri_configs 01-24-2024 06:17:33 PST * fix: update get_execution_role_arn from metadata file if present (aws#4388) * fix: Support using PipelineDefinitionConfig in local mode (aws#4352) * fix: remove fastapi and uvicorn dependencies (aws#4365) They are not used in the codebase. Closes aws#4361 aws#4295 * prepare release v2.205.0 * update development version to v2.205.1.dev0 * change: TGI NeuronX 0.0.17 (aws#4390) * fix: Support PipelineVariable for ModelQualityCheckConfig attributes (aws#4353) * feat: Logic to detect hardware GPU count and aggregate GPU memory size in MiB (aws#4389) * Add logic to detect hardware GPU count and aggregate GPU memory size in MiB * Fix all formatting * Addressed PR review comments * Addressed PR Review messages * Addressed PR Review Messages * Addressed PR Review comments * Addressed PR Review Comments * Add integration tests * Add config * Fix integration tests * Include Instance Types GPU infor Config files * Addressed PR review comments * Fix unit tests * Fix unit test: 'Mock' object is not subscriptable --------- Co-authored-by: Jonathan Makunga <[email protected]> * fix: fixed create monitoring schedule failing after validation error (aws#4385) Co-authored-by: Keshav Chandak <[email protected]> * Add collection type support for Feaure Group Ingestion. Add TargetStores support for PutRecord and Ingestion. * Remove merge conflicts. * Update the feature definition type * Black formatting * Fix Flake8 formatting * Fix Pylint * Fix Formatting. --------- Co-authored-by: sagemaker-bot <[email protected]> Co-authored-by: ci <ci> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Co-authored-by: Keshav Chandak <[email protected]> Co-authored-by: stacicho <[email protected]> Co-authored-by: Teng-xu <[email protected]> Co-authored-by: huilgolr <[email protected]> Co-authored-by: Gary Wang <[email protected]> Co-authored-by: EC2 Default User <[email protected]> Co-authored-by: akrishna1995 <[email protected]> Co-authored-by: Miyoung <[email protected]> Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Xinyu Xie <[email protected]> Co-authored-by: Evan Kravitz <[email protected]> Co-authored-by: martinRenou <[email protected]> Co-authored-by: Duc Trung Le <[email protected]> Co-authored-by: ruiliann666 <[email protected]> Co-authored-by: Ruilian Gao <[email protected]> Co-authored-by: ananth102 <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: amzn-choeric <[email protected]> Co-authored-by: evakravi <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Sirut Buasai <[email protected]> Co-authored-by: Sindhu Somasundaram <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> Co-authored-by: Haixin Wang <[email protected]> Co-authored-by: Nilesh PS <[email protected]> Co-authored-by: Nilesh PS <[email protected]> Co-authored-by: jiapinw <[email protected]> Co-authored-by: Jay Goyani <[email protected]> Co-authored-by: Justin <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]>

qidewenwhen requested a review from a team as a code owner January 12, 2024 00:16

qidewenwhen requested review from jgoyani1 and removed request for a team January 12, 2024 00:16

navaj0 reviewed Jan 12, 2024

View reviewed changes

src/sagemaker/remote_function/core/pipeline_variables.py Outdated Show resolved Hide resolved

qidewenwhen force-pushed the selective branch from 76119f9 to 0e1758d Compare January 13, 2024 02:59

qidewenwhen force-pushed the selective branch 2 times, most recently from c50e3b8 to 260a13c Compare January 17, 2024 20:51

qidewenwhen requested review from nmadan, rohangujarathi and svia3 January 17, 2024 20:54

qidewenwhen force-pushed the selective branch from 260a13c to 83f090b Compare January 18, 2024 01:42

knikure self-assigned this Jan 22, 2024

knikure reviewed Jan 22, 2024

View reviewed changes

knikure approved these changes Jan 22, 2024

View reviewed changes

rohangujarathi approved these changes Jan 22, 2024

View reviewed changes

navaj0 reviewed Jan 23, 2024

View reviewed changes

src/sagemaker/remote_function/core/pipeline_variables.py Show resolved Hide resolved

navaj0 reviewed Jan 23, 2024

View reviewed changes

src/sagemaker/workflow/pipeline.py Outdated Show resolved Hide resolved

qidewenwhen force-pushed the selective branch from 1a05d6b to 20852a5 Compare January 23, 2024 18:51

navaj0 approved these changes Jan 23, 2024

View reviewed changes

feat: Support selective pipeline execution for function step

e7e59d7

qidewenwhen force-pushed the selective branch from 20852a5 to e7e59d7 Compare January 23, 2024 20:16

knikure reviewed Jan 23, 2024

View reviewed changes

knikure merged commit 9828101 into aws:master Jan 24, 2024

qidewenwhen mentioned this pull request Jan 25, 2024

feat: Support selective pipeline execution between function step and regular step #4392

Merged

9 tasks

samruds pushed a commit to samruds/sagemaker-python-sdk that referenced this pull request Jan 30, 2024

feat: Support selective pipeline execution for function step (aws#4372)

90cdf0d

suryans-commit pushed a commit to suryans-commit/sagemaker-python-sdk that referenced this pull request Feb 5, 2024

feat: Support selective pipeline execution for function step (aws#4372)

f7bba26

feat: Support selective pipeline execution for function step #4372

feat: Support selective pipeline execution for function step #4372

Uh oh!

Conversation

qidewenwhen commented Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

mufaddal-rohawala commented Jan 12, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 12, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 12, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 12, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 12, 2024

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Jan 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

mufaddal-rohawala commented Jan 13, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 13, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 13, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 13, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 13, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 17, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 17, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 17, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 17, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 17, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 18, 2024

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jan 19, 2024

AWS CodeBuild CI Report

Uh oh!

knikure left a comment

Choose a reason for hiding this comment

Uh oh!

mufaddal-rohawala commented Jan 22, 2024

AWS CodeBuild CI Report

Uh oh!

qidewenwhen commented Jan 22, 2024

Uh oh!

qidewenwhen commented Jan 12, 2024 •

edited

Loading

codecov-commenter commented Jan 12, 2024 •

edited

Loading