feature: enable smdataparallel custom mpi options support #2255

karan6181 · 2021-03-30T00:01:50Z

Description of changes:

Support of custom MPI options for Sagemaker Distibuted Data Parallel. This feature will enable the user to pass the custom MPI flags during the training.

E.g.:

mpi_options = '-verbose -x NCCL_DEBUG=INFO'
distribution = {'smdistributed':{'dataparallel':{
                                    'enabled': True,
                                    'custom_mpi_options': mpi_options,
                                    }
                                }
                }

NOTE:
The SageMaker Distributed Data Parallel cutom MPI support require changes in sagemaker-training-toolkit and sagemaker-python-sdk. The front-end custom MPI flag support require changes in sagemaker-python-sdk and backend feature processing require changes in sagemaker-training-toolkit. For user to use this feature, both the repository changes has to be present in the production. But to make things easier with respect to merging the code changes and availability of those features in production, we will divide this feature into two phases. The first phase is to make changes in sagemaker-training-toolkit which is PR #99 and this will merge first and be available in production via a DLC. This PR #99 will not break the existing feature, meaning, user can still run the SageMaker Distributed Data Parallel without custom MPI support. The second phase is to merge the changes in sagemaker-python-sdk which is PR #2255. The integration test present in sagemaker-python-sdk depends on the availability of feature #99 in production. The sagemaker-python-sdk pypi wheel will be updated in DLC and Sagemaker Jupyter notebook conda environments. Once the PR #99 and PR #2255 available in production, user can use the custom mpi options in SageMaker Distributed Data Parallel.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-03-30T00:04:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 54b7200
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T00:25:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: cc9cf32
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T00:38:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 54b7200
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T00:40:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 54b7200
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T00:43:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 54b7200
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T00:58:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: cc9cf32
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T01:01:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: cc9cf32
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T01:04:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: cc9cf32
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T02:02:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: cc9cf32
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T16:16:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2f5aed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T16:40:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2f5aed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T16:43:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2f5aed4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T16:47:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2f5aed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-30T17:09:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2f5aed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T20:53:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b464716
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T21:16:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b464716
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T21:18:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b464716
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T21:25:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b464716
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T21:27:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b464716
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T16:39:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 165052a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ChaiBapchya

Thanks!

Discussed offline - Let's make sure sm-training-toolkit change gets merged & released. Also hope the CI is running these integration tests.

sagemaker-bot · 2021-04-02T17:02:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 165052a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T17:04:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 165052a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T17:08:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 165052a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T17:12:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 165052a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T16:16:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9668a65
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

… DLC TF 2.4.1

sagemaker-bot · 2021-04-13T16:43:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6343e19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T16:44:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9668a65
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T16:45:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9668a65
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T16:50:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9668a65
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T17:08:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6343e19
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T17:09:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6343e19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T17:14:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6343e19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-13T17:15:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6343e19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

karan6181 · 2021-04-13T17:27:46Z

The test tests/integ/test_edge.py::test_edge_packaging_job failed in CI sagemaker-python-sdk-pr which is irrelevant to the changes in this PR.

sagemaker-bot · 2021-04-14T18:02:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1b03c5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-14T18:26:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 1b03c5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-14T18:28:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1b03c5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-14T18:35:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 1b03c5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-14T18:36:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1b03c5d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Ahsan Khan <[email protected]>

karan6181 mentioned this pull request Mar 30, 2021

feature: smdataparallel custom mpi options support aws/sagemaker-training-toolkit#99

Merged

6 tasks

karan6181 changed the title ~~[WIP] feature: smdataparallel custom mpi options support~~ feature: smdataparallel custom mpi options support Apr 2, 2021

ChaiBapchya previously approved these changes Apr 2, 2021

View reviewed changes

karan6181 changed the title ~~feature: smdataparallel custom mpi options support~~ [DO-NOT_MERGE] feature: smdataparallel custom mpi options support Apr 2, 2021

karan6181 changed the title ~~[DO-NOT_MERGE] feature: smdataparallel custom mpi options support~~ feature: smdataparallel custom mpi options support Apr 13, 2021

feature: smdataparallel custom mpi options support

9881bb1

karan6181 added 3 commits April 13, 2021 09:05

Fixed pylint

e51db9e

Fixed black-check

a5e0822

Added unit and integration test

9668a65

karan6181 dismissed ChaiBapchya’s stale review via 9668a65 April 13, 2021 16:06

karan6181 force-pushed the smddp_custom_mpi_support branch from 165052a to 9668a65 Compare April 13, 2021 16:06

Enable the TF2 smdataparallel test since the issue is fixed in latest…

6343e19

… DLC TF 2.4.1

Merge branch 'master' into smddp_custom_mpi_support

1b03c5d

ahsan-z-khan approved these changes Apr 14, 2021

View reviewed changes

ahsan-z-khan changed the title ~~feature: smdataparallel custom mpi options support~~ feature: enable smdataparallel custom mpi options support Apr 14, 2021

ahsan-z-khan merged commit 44fbcc9 into aws:master Apr 14, 2021

TEChopra1000 pushed a commit to TEChopra1000/sagemaker-python-sdk that referenced this pull request Apr 28, 2021

feature: enable smdataparallel custom mpi options support (aws#2255)

12030d9

Co-authored-by: Ahsan Khan <[email protected]>

feature: enable smdataparallel custom mpi options support #2255

feature: enable smdataparallel custom mpi options support #2255

Uh oh!

Conversation

karan6181 commented Mar 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 30, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

ChaiBapchya left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

karan6181 commented Mar 30, 2021 •

edited

Loading