documentation: Add Remote Function updates #3805

dpatro · 2023-04-22T17:57:04Z

Issue #, if available:

Description of changes:

Add remote function related classes and method specifications
Update Configuration File specification to include remote function function as supported SageMaker capability.

Testing done:

Tested locally to see the changes on HTML.
tox -e twine,sphinx pass locally.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-04-22T18:40:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 939cdf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-22T18:48:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 939cdf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-22T18:53:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 939cdf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-22T19:04:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 939cdf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-04-22T19:05:56Z

Codecov Report

Merging #3805 (83095a3) into master (b334d4e) will decrease coverage by 0.75%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3805      +/-   ##
==========================================
- Coverage   89.87%   89.13%   -0.75%     
==========================================
  Files        1056      246     -810     
  Lines      100912    24973   -75939     
==========================================
- Hits        90693    22259   -68434     
+ Misses      10219     2714    -7505

Impacted Files	Coverage Δ
src/sagemaker/remote_function/client.py	`96.86% <ø> (ø)`
src/sagemaker/workflow/functions.py	`94.73% <ø> (ø)`

... and 1300 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-04-22T20:51:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 939cdf4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

rohangujarathi · 2023-04-22T21:54:58Z

doc/overview.rst

+            Tags:
+              - {'Key': 'yourTagKey', 'Value': 'yourTagValue'}


should this tags key and value be in a similar format as line 2021 to keep the consistency?

agree, done

rohangujarathi · 2023-04-22T21:57:13Z

doc/overview.rst

+              - 'command_1'
+              - 'command_2'
+            PreExecutionScript: 'path/to/script.sh'
+            RoleArn: 'arn:aws:iam::366666666666:role/MyRole'


just for the sake of consistency, can we change account Id here to 555555555555 as line 2020, same with imageUri field

rohangujarathi · 2023-04-22T21:58:53Z

doc/overview.rst

-   'context': {...}}
+   'context': {...}}cd


is this change on purpose? what does cd do here?

good catch.

rohangujarathi · 2023-04-22T22:02:46Z

src/sagemaker/remote_function/client.py

-    """Function that starts a new SageMaker job synchronously with overridden runtime settings.
+    """Decorator for running the annotated function as a SageMaker training job.
+
+    This decorator wraps the annotated code and runs it is a new SageMaker job synchronously


nit: runs it is a --> runs it as a

rohangujarathi · 2023-04-22T22:09:55Z

src/sagemaker/remote_function/client.py

+        instance_count (int): The number of instance to use. Defaults to 1.
+
+        instance_type (str): The Amazon Elastic Compute Cloud (EC2) instance type to use to run
+          the SageMaker job. Valid values include ml.c4.xlarge. If not provided, ValueError is


Valid values include ml.c4.xlarge.
It can be anything right?

changed the statement to "e.g. ml..."

rohangujarathi · 2023-04-22T22:11:41Z

src/sagemaker/remote_function/client.py

+        max_retry_attempts (int): The max number of times the job is retried after an on
+          ```InternalServerFailure``` Error. Defaults to 1.


after an on?

rohangujarathi · 2023-04-22T22:13:21Z

src/sagemaker/remote_function/client.py

+
+        max_runtime_in_seconds (int): The upper limit in seconds to be used for training. After
+          this specified amount of time, SageMaker terminates the job regardless of its current
+          status. The max_run_duration time defaults to 1 day or (86400 seconds).


Should this be The max_runtime_in_seconds time defaults to 1 day or (86400 seconds).

rohangujarathi · 2023-04-22T22:14:49Z

src/sagemaker/remote_function/client.py

+        s3_root_uri (str): The root S3 folder to which where the code archives and data are
+          uploaded to. Defaults to ``s3://<sagemaker-default-bucket>``.


to which where --> to which

rohangujarathi · 2023-04-22T22:21:38Z

src/sagemaker/remote_function/client.py

+            timeout (int): Timeout in seconds to wait for until the job is to completed before it is
+              stopped. Defaults to ``None``.


job is to completed sounds grammatically incorrect

rohangujarathi · 2023-04-22T22:26:27Z

src/sagemaker/remote_function/client.py

+        enable_network_isolation (bool): A flag that specifies whether container will run in
+          network isolation mode. Defaults to ``False``. Network isolation mode restricts the
+          container access to outside networks (such as the Internet). The container does not
+          make any inbound or outbound network calls. Also known as Internet-free mode.
    """

    def _remote(func):


all the comments as previous def remote function might apply here, please verify

sagemaker-bot · 2023-04-23T00:14:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e83ee34
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-23T00:30:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e83ee34
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-23T00:31:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e83ee34
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-23T00:39:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e83ee34
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-23T00:43:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e83ee34
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

p-nanimal · 2023-04-24T16:18:58Z

src/sagemaker/remote_function/client.py

+              in the system path.
+
+          * The parameter dependencies is set to auto_capture. SageMaker will automatically
+            generate a env_snapshot.yml corresponding to the current active conda environment’s


generate (an) env_snapshot.yml (file)

p-nanimal · 2023-04-24T16:19:27Z

doc/overview.rst

+Run Machine Learning code on SageMaker using remote function
+************************************************************
+
+You can seamlessly integrate your local machine language (ML) code to run in a Amazon SageMaker Training job by wrapping


->remove "seamlessly"

p-nanimal · 2023-04-24T16:21:07Z

src/sagemaker/remote_function/client.py

+          Amazon Elastic Container Registry (ECR). Defaults to the following based on where the SDK
+          is running:
+
+            * For SageMaker Studio notebook cases, the image used as the kernel image for the


For SageMaker Studio notebook (instances),...

fixed to align with next comment.

p-nanimal · 2023-04-24T16:21:26Z

src/sagemaker/remote_function/client.py

+
+            * For SageMaker Studio notebook cases, the image used as the kernel image for the
+              notebook is used.
+            * For other cases, it is resolved to base python image with the same python version


For other (users),...

p-nanimal · 2023-04-24T16:21:45Z

src/sagemaker/remote_function/client.py

+          local directories. Set to ``True`` if the remote function code imports local modules and
+          methods that are not available via PyPI or conda. Default value is ``False``.
+
+        instance_count (int): The number of instance to use. Defaults to 1.


instance(s)

p-nanimal · 2023-04-24T16:22:22Z

src/sagemaker/remote_function/client.py

+          warm pools. The use of warmpools reduces the latency time spent to provision new
+          resources. The default value for ``keep_alive_period_in_seconds`` is 0.
+          NOTE: Additional charges associated with warm pools may apply. Using this parameter will
+          also activate a new persistent cache feature, which will further reduce job start up


Persistent Cache (capitalized, according to Tritin)

p-nanimal · 2023-04-24T16:27:50Z

src/sagemaker/remote_function/client.py


        Returns:
-            The Python object returned by the function
+            The Python object returned by the remote function execution.


remove (execution)

p-nanimal · 2023-04-24T16:28:58Z

src/sagemaker/remote_function/client.py

-        """Wait for the underlying sagemaker job to complete.
+        """Wait for the underlying SageMaker job to complete.
+
+        This method waits for the SageMaker job created as a result of the remote function execution


execution -> run

p-nanimal · 2023-04-24T16:29:51Z

src/sagemaker/remote_function/client.py

        Args:
-            timeout (int): Timeout in seconds to wait until the job is completed. ``None`` by
-                default.
+            timeout (int): Timeout in seconds to wait for until the job is completed before it is


remove (for)-> Timeout in seconds to wait until the job is completed...

p-nanimal · 2023-04-24T16:30:25Z

src/sagemaker/remote_function/client.py

-             manages interactions with Amazon SageMaker APIs and any other
-             AWS services needed.
+        job_name_prefix (str): A prefix used to identify the SageMaker jobs associated with remote
+          function execution.


execution -> run

p-nanimal · 2023-04-24T16:30:52Z

src/sagemaker/remote_function/client.py

-             AWS services needed.
+        job_name_prefix (str): A prefix used to identify the SageMaker jobs associated with remote
+          function execution.
+        sagemaker_session (sagemaker.session.Session): A session object which manages interactions


A session object (that) manages

sagemaker-bot · 2023-04-24T17:08:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0ea96a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T17:11:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0ea96a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T17:18:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0ea96a0
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T17:18:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0ea96a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T17:21:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0ea96a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T20:17:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f1c4704
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T20:32:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f1c4704
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T20:39:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f1c4704
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T20:46:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f1c4704
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* Add remote function related classes and method specifications * Update Configuration File specification to include remote function function as supported SageMaker capability.

sagemaker-bot · 2023-04-24T22:13:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T22:18:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T22:19:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T22:19:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T22:29:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T23:06:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T23:06:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T23:06:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-24T23:07:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-25T00:09:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 83095a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

dpatro requested a review from a team as a code owner April 22, 2023 17:57

dpatro requested review from mufaddal-rohawala and removed request for a team April 22, 2023 17:57

rohangujarathi reviewed Apr 22, 2023

View reviewed changes

rohangujarathi approved these changes Apr 23, 2023

View reviewed changes

trajanikant force-pushed the master branch from e83ee34 to 0ea96a0 Compare April 24, 2023 16:14

p-nanimal reviewed Apr 24, 2023

View reviewed changes

dpatro force-pushed the master branch from 0ea96a0 to f1c4704 Compare April 24, 2023 19:41

p-nanimal approved these changes Apr 24, 2023

View reviewed changes

dpatro force-pushed the master branch 2 times, most recently from c385f93 to 9990b3b Compare April 24, 2023 21:23

Dipankar Patro added 6 commits April 24, 2023 14:24

documentation: Adding Remote Function updates

f3329c7

* Add remote function related classes and method specifications * Update Configuration File specification to include remote function function as supported SageMaker capability.

Add missed remote_function/client.py

ef8a9ec

Address PR feedback

11cfd6f

Address PR feedback

6fcef34

persistent cache reference updates

34959f8

addressing some more feedback

83095a3

dpatro force-pushed the master branch from 9990b3b to 83095a3 Compare April 24, 2023 21:24

knikure approved these changes Apr 24, 2023

View reviewed changes

knikure changed the title ~~documentation: Adding Remote Function updates~~ documentation: Add Remote Function updates Apr 24, 2023

knikure merged commit 97eaed3 into aws:master Apr 24, 2023

		max_retry_attempts (int): The max number of times the job is retried after an on
		```InternalServerFailure``` Error. Defaults to 1.

		s3_root_uri (str): The root S3 folder to which where the code archives and data are
		uploaded to. Defaults to ``s3://<sagemaker-default-bucket>``.

		timeout (int): Timeout in seconds to wait for until the job is to completed before it is
		stopped. Defaults to ``None``.

documentation: Add Remote Function updates #3805

documentation: Add Remote Function updates #3805

Uh oh!

Conversation

dpatro commented Apr 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Apr 22, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2023

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Apr 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Apr 22, 2023

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpatro Apr 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 23, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 23, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 23, 2023

AWS CodeBuild CI Report

Uh oh!

dpatro commented Apr 22, 2023 •

edited

Loading

codecov-commenter commented Apr 22, 2023 •

edited

Loading

dpatro Apr 22, 2023 •

edited

Loading

dpatro Apr 24, 2023 •

edited

Loading