[huggingface] Add `torch.distributed` support for Trainium and `torchrun` #3759

philschmid · 2023-03-29T12:57:32Z

Description of changes:

This PR adds the torch_distrubted support to the HuggingFace estimator. The code is mostly copied from the already existing implementation for it in the PyTorch estimator.

Testing done:

local tests and added a new test test_huggingface_torch_distributed.py

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

claytonparnell

/bot run slow-tests

sagemaker-bot · 2023-03-31T19:17:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6bc5835
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-31T19:17:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6bc5835
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-31T19:48:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6bc5835
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-31T20:09:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6bc5835
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-31T20:20:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6bc5835
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid · 2023-04-01T09:00:20Z

/bot run all

sagemaker-bot · 2023-04-03T16:49:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c4f3397
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T17:38:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c4f3397
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T17:39:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c4f3397
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T17:42:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c4f3397
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T19:46:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c4f3397
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

claytonparnell · 2023-04-03T22:03:08Z

Hi @philschmid - I see there are some formatting errors, could you please run tox -e black-format and tox -e flake8 to resolve?

philschmid · 2023-04-04T06:58:21Z

Hi @philschmid - I see there are some formatting errors, could you please run tox -e black-format and tox -e flake8 to resolve?

@claytonparnell Fixed! Sorry for that.

claytonparnell

/bot run all

claytonparnell · 2023-04-12T15:14:18Z

tests/integ/test_huggingface_torch_distributed.py

+from tests.integ import DATA_DIR, TRAINING_DEFAULT_TIMEOUT_MINUTES
+
+
+@pytest.mark.release


Could we remove the mark as release? This makes test run in all regions, and I worry that instance type may not be available in all regions. Test will still run in us-west-2.

Sorry I didn't catch this earlier, once this is removed I can make sure it's merged this week

sagemaker-bot · 2023-04-12T15:44:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0b4d65d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-12T15:45:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0b4d65d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-12T15:48:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0b4d65d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-12T15:54:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0b4d65d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-04-12T15:58:19Z

Codecov Report

Merging #3759 (35f9a40) into master (162e922) will decrease coverage by 0.77%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master    #3759      +/-   ##
==========================================
- Coverage   89.86%   89.09%   -0.77%     
==========================================
  Files        1016      237     -779     
  Lines       96428    23875   -72553     
==========================================
- Hits        86656    21272   -65384     
+ Misses       9772     2603    -7169

Impacted Files	Coverage Δ
src/sagemaker/huggingface/estimator.py	`88.18% <71.42%> (ø)`

... and 1252 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

claytonparnell

/bot run all

sagemaker-bot · 2023-04-12T16:38:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 335d1c4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid · 2023-04-12T16:49:06Z

@claytonparnell fixed the flake8 error. can you run the tests again

sagemaker-bot · 2023-04-12T17:11:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 335d1c4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-12T17:27:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 335d1c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-12T17:30:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 335d1c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

claytonparnell

/bot run all

sagemaker-bot · 2023-04-17T19:01:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6aee919
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-17T19:06:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6aee919
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-17T19:15:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6aee919
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-17T19:16:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6aee919
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-17T19:22:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6aee919
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/integ/test_huggingface_torch_distributed.py

claytonparnell

/bot run pr, local-mode-tests

sagemaker-bot · 2023-04-18T21:24:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 35f9a40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-18T21:26:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 35f9a40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-18T22:10:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 35f9a40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-18T22:16:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 35f9a40
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-18T22:30:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 35f9a40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-18T22:43:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 35f9a40
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-19T00:34:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 35f9a40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-19T00:42:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 35f9a40
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-19T01:05:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 35f9a40
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-19T15:22:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 35f9a40
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid · 2023-04-19T15:26:51Z

Errors are not related to the PR

philschmid requested a review from a team as a code owner March 29, 2023 12:57

philschmid requested review from vikas0203 and removed request for a team March 29, 2023 12:57

claytonparnell approved these changes Mar 31, 2023

View reviewed changes

claytonparnell reviewed Apr 12, 2023

View reviewed changes

philschmid and others added 8 commits April 17, 2023 14:13

make tox happy

1d9d2e2

add HF test

d92f9e6

change test name

6d8ece0

Update test_huggingface_torch_distributed.py

68edb38

fix tox and black

e9c5818

removed mark

2220dc2

fix flake8

eb0f740

fix test

6aee919

claytonparnell force-pushed the add-torchrun-hf branch from a81e10e to 6aee919 Compare April 17, 2023 18:13

claytonparnell reviewed Apr 17, 2023

View reviewed changes

tests/integ/test_huggingface_torch_distributed.py Outdated Show resolved Hide resolved

fix instance type

35f9a40

claytonparnell reviewed Apr 18, 2023

View reviewed changes

claytonparnell merged commit ebd48c9 into aws:master Apr 19, 2023

		from tests.integ import DATA_DIR, TRAINING_DEFAULT_TIMEOUT_MINUTES


		@pytest.mark.release

[huggingface] Add torch.distributed support for Trainium and torchrun #3759

[huggingface] Add torch.distributed support for Trainium and torchrun #3759

Uh oh!

Conversation

philschmid commented Mar 29, 2023

Merge Checklist

General

Tests

Uh oh!

claytonparnell left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 31, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2023

AWS CodeBuild CI Report

Uh oh!

philschmid commented Apr 1, 2023

Uh oh!

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

Uh oh!

claytonparnell commented Apr 3, 2023

Uh oh!

philschmid commented Apr 4, 2023

Uh oh!

claytonparnell left a comment

Choose a reason for hiding this comment

Uh oh!

claytonparnell Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

claytonparnell Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

philschmid Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 12, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 12, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 12, 2023

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 12, 2023

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

claytonparnell left a comment

Choose a reason for hiding this comment

Uh oh!

[huggingface] Add `torch.distributed` support for Trainium and `torchrun` #3759

[huggingface] Add `torch.distributed` support for Trainium and `torchrun` #3759

claytonparnell left a comment •

edited

Loading

codecov-commenter commented Apr 12, 2023 •

edited

Loading

claytonparnell left a comment •

edited

Loading