breaking: require framework_version, py_version for pytorch #1568

metrizable · 2020-06-10T17:11:47Z

require framework_version and py_version for PyTorch

Issue #, if available: #1465

Description of changes:

changes include:

framework_version, py_version required for framework PyTorch
framework_version, py_version required for framework PyTorchModel
re-order of args: convention to follow entry_point, push args with defaults below
testing updates
doc updates
ignore coverage results for py27 env due to v2 migration scripts

Testing done:

unit:

% tox --parallel 3 tests/unit
✔ OK black-format in 8.908 seconds
✔ OK twine in 13.317 seconds
✔ OK flake8 in 22.256 seconds
✔ OK pylint in 33.459 seconds
✔ OK doc8 in 18.439 seconds
✔ OK py36 in 41.879 seconds
✔ OK sphinx in 1 minute, 19.063 seconds
✔ OK py37 in 41.58 seconds
✔ OK py27 in 1 minute, 47.14 seconds

integ:

% export IGNORE_COVERAGE=- ; tox -e py37 -- -s -vv tests/integ/test_pytorch_train.py; unset IGNORE_COVERAGE
...
tests/integ/test_pytorch_train.py::test_region <- tests/integ/__init__.py PASSED
tests/integ/test_pytorch_train.py::test_sync_fit_deploy 2020-06-10 16:37:12 Starting - Starting the training job...
2020-06-10 16:37:15 Starting - Launching requested ML instances......
...
...
2020-06-10 16:49:30,774 [INFO ] pool-1-thread-2 ACCESS_LOG - /10.32.0.2:47254 "GET /ping HTTP/1.1" 200 0
PASSED
tests/integ/test_pytorch_train.py::test_fit_deploy
...
algo-1-ny85y_1  | 2020-06-10 16:58:01,649 [INFO ] W-9002-model ACCESS_LOG - /172.17.0.1:54396 "POST /invocations HTTP/1.1" 200 23
Gracefully stopping... (press Ctrl+C again to force)
PASSED
tests/integ/test_pytorch_train.py::test_deploy_model
...
2020-06-10 17:05:40,357 [INFO ] pool-1-thread-2 ACCESS_LOG - /10.32.0.2:60302 "GET /ping HTTP/1.1" 200 1
PASSED
tests/integ/test_pytorch_train.py::test_deploy_packed_model_with_entry_point_name
...

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to any/all clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

* framework_version, py_version required for framework PyTorch * framework_version, py_version required for framework PyTorchModel * re-order of args: convention to follow entry_point, push args with defaults below * testing updates * doc updates * ignore coverage results for py27 env due to v2 migration scripts

sagemaker-bot · 2020-06-10T17:15:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f2a13ef
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T17:15:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f2a13ef
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/fw_utils.py

src/sagemaker/pytorch/estimator.py

tests/integ/test_pytorch_train.py

tests/integ/test_source_dirs.py

tests/unit/test_pytorch.py

sagemaker-bot · 2020-06-10T17:22:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d850210
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T17:40:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f2a13ef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T17:49:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: d850210
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T17:58:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d850210
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T18:12:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d850210
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T20:52:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5f4926e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T20:54:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a1d4192
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T21:13:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5f4926e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T21:14:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a1d4192
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu

one small comment. otherwise looks good.

tests/integ/test_tuner.py

sagemaker-bot · 2020-06-10T21:32:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: a1d4192
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T21:40:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5f4926e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T21:41:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a1d4192
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T22:02:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c55492e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T22:22:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c55492e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T22:44:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c55492e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T22:58:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8015936
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T23:16:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8015936
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-10T23:40:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8015936
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T01:02:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 8015936
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T14:56:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9b92c02
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T15:17:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9b92c02
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T15:20:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 9b92c02
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T15:43:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9b92c02
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/pytorch/model.py

tests/integ/test_airflow_config.py

tests/integ/test_pytorch_train.py

laurenyu

you can hardcode the PyTorch versions to whatever the default used to be for the ones that aren't easily upgradable to PyTorch 1.4

tests/integ/test_git.py

tests/integ/test_pytorch_train.py

sagemaker-bot · 2020-06-11T16:46:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a9c51b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T17:05:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a9c51b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T17:10:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: a9c51b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T17:33:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a9c51b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…n-pytorch

sagemaker-bot · 2020-06-11T17:57:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8527ed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T18:15:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-zwei-pr
Commit ID: 8527ed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T18:16:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8527ed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-11T18:43:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8527ed4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Change to make Model Trainer return a Model Object * Fix * Cleanup * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * Updates * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Add Distributed Training Support Model Trainer (#1536) * Add path to set Additional Settings in ModelTrainer (#1555) * Updates * Mask Sensitive Env Logs in Container (#1568) * Cleanup PR * Codestyle fixes * Update logic to use model parameter instead of model_path * Fixes * Fixes * Tests * Codestyle Fixes * Codestyle Fixes * Codestyle Fixes * Codestyle Fixes --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: pintaoz-aws <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]>

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * Single container local mode training * Add wait argument * Implement helper funtions * Add helper functions * Fix bugs * Fix codestyle * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Fix test and codestyle * Add Distributed Training Support Model Trainer (#1536) * Add tests * Add path to set Additional Settings in ModelTrainer (#1555) * Added example notebook * Fix codestyle * Address comments * resolve merge conflict * Support multi container local training (#1576) * Fix codestyle * Mask Sensitive Env Logs in Container (#1568) * Fix bug in script mode setup ModelTrainer (#1575) * Support multi container local training * Merge branch 'single_container_local_training' into multi_container_local_training * Update unit tests --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> * Remove LocalTrainingJob class * Bypass pydantic check * Add example --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Gokul Anantha Narayanan <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]>

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Add Distributed Training Support Model Trainer (#1536) * Add path to set Additional Settings in ModelTrainer (#1555) * feature: support HuggingFace models with JumpStart configs * Update bucket name for the model mapping * Mask Sensitive Env Logs in Container (#1568) * Fix unit test * Fix bug in script mode setup ModelTrainer (#1575) * Save mapping as attribute * Fix style issues * Fix style issues * Fix: bypass jumpstart mapping when not in endpoint mode * Skip JS model mapping with env vars or image URI provided * Revert "Merge branch 'aws:master' into dev-morpheus" This reverts commit 26a0b0bb37e0343b3287f5c5c484df22726fc858, reversing changes made to d19d4e178442be4b6e1d07d55498dd76dfac50f0. * Merge branch 'aws:master' into dev-morpheus This reverts commit 076442bd83e5ca977bf5b6ce1b716474d2794feb. * Rebase on master-morpheus * Fix unit test description * Fix TEI integ test * Fix style issue * Fix style issues * Fix schema builder integ tests * Fix TEI integ test * Fix code style issue --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Gokul Anantha Narayanan <[email protected]> Co-authored-by: pintaoz-aws <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]> Co-authored-by: Xiong Zeng <[email protected]> Co-authored-by: Gary Wang <[email protected]>

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Change to make Model Trainer return a Model Object * Fix * Cleanup * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * Updates * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Add Distributed Training Support Model Trainer (#1536) * Add path to set Additional Settings in ModelTrainer (#1555) * Updates * Mask Sensitive Env Logs in Container (#1568) * Cleanup PR * Codestyle fixes * Update logic to use model parameter instead of model_path * Fixes * Fixes * Tests * Codestyle Fixes * Codestyle Fixes * Codestyle Fixes * Codestyle Fixes --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: pintaoz-aws <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]>

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * Single container local mode training * Add wait argument * Implement helper funtions * Add helper functions * Fix bugs * Fix codestyle * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Fix test and codestyle * Add Distributed Training Support Model Trainer (#1536) * Add tests * Add path to set Additional Settings in ModelTrainer (#1555) * Added example notebook * Fix codestyle * Address comments * resolve merge conflict * Support multi container local training (#1576) * Fix codestyle * Mask Sensitive Env Logs in Container (#1568) * Fix bug in script mode setup ModelTrainer (#1575) * Support multi container local training * Merge branch 'single_container_local_training' into multi_container_local_training * Update unit tests --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> * Remove LocalTrainingJob class * Bypass pydantic check * Add example --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Gokul Anantha Narayanan <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]>

* Base model trainer (#1521) * Base model trainer * flake8 * add testing notebook * add param validation & set defaults * Implement simple train method * feature: support script mode with local train.sh (#1523) * feature: support script mode with local train.sh * Stop tracking train.sh and add it to .gitignore * update message * make dir if not exist * fix docs * fix: docstyle * Address comments * fix hyperparams * Revert pydantic custom error * pylint * Image Spec refactoring and updates (#1525) * Image Spec refactoring and updates * Unit tests and update function for Image Spec * Fix hugging face test * Fix Tests * Add unit tests for ModelTrainer (#1527) * Add unit tests for ModelTrainer * Flake8 * format * Add example notebook (#1528) * Add testing notebook * format * use smaller data * remove large dataset * update * pylint * flake8 * ignore docstyle in directories with test * format * format * Add enviornment variable bootstrapping script (#1530) * Add enviornment variables scripts * format * fix comment * add docstrings * fix comment * feature: add utility function to capture local snapshot (#1524) * local snapshot * Update pip list command * Remove function calls * Address comments * Address comments * Support intelligent parameters (#1540) * Support intelligent parameters * fix codestyle * Revert Image Spec (#1541) * Cleanup ModelTrainer (#1542) * General image builder (#1546) * General image builder * General image builder * Fix codestyle * Fix codestyle * Move location * Add warnings * Add integ tests * Fix integ test * Fix integ test * Fix region error * Add region * Latest Container Image (#1545) * Latest Container Image * Test Fixes * Parameterized tests and some logic updates * Test fixes * Move to Image URI * Fixes for unit test * Fixes for unit test * Fix codestyle error checks * Cleanup ModelTrainer code (#1552) * feat: add pre-processing and post-processing logic to inference_spec (#1560) * add pre-processing and post-processing logic to inference_spec * fix format * make accept_type and content_type optional * remove accept_type and content_type from pre/post processing * correct typo * Add Distributed Training Support Model Trainer (#1536) * Add path to set Additional Settings in ModelTrainer (#1555) * feature: support HuggingFace models with JumpStart configs * Update bucket name for the model mapping * Mask Sensitive Env Logs in Container (#1568) * Fix unit test * Fix bug in script mode setup ModelTrainer (#1575) * Save mapping as attribute * Fix style issues * Fix style issues * Fix: bypass jumpstart mapping when not in endpoint mode * Skip JS model mapping with env vars or image URI provided * Revert "Merge branch 'aws:master' into dev-morpheus" This reverts commit 26a0b0bb37e0343b3287f5c5c484df22726fc858, reversing changes made to d19d4e178442be4b6e1d07d55498dd76dfac50f0. * Merge branch 'aws:master' into dev-morpheus This reverts commit 076442bd83e5ca977bf5b6ce1b716474d2794feb. * Rebase on master-morpheus * Fix unit test description * Fix TEI integ test * Fix style issue * Fix style issues * Fix schema builder integ tests * Fix TEI integ test * Fix code style issue --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Gokul Anantha Narayanan <[email protected]> Co-authored-by: pintaoz-aws <[email protected]> Co-authored-by: Pravali Uppugunduri <[email protected]> Co-authored-by: Xiong Zeng <[email protected]> Co-authored-by: Gary Wang <[email protected]>

metrizable force-pushed the require-framework-version-pytorch branch from f2a13ef to d850210 Compare June 10, 2020 17:14

laurenyu reviewed Jun 10, 2020

View reviewed changes

metrizable added 2 commits June 10, 2020 13:43

Merge branch 'zwei' into require-framework-version-pytorch

5f4926e

breaking: addressing comments in PR feedback

a1d4192

laurenyu reviewed Jun 10, 2020

View reviewed changes

tests/integ/test_tuner.py Outdated Show resolved Hide resolved

breaking: small updates based on PR feedback

c55492e

Merge branch 'zwei' into require-framework-version-pytorch

8015936

laurenyu previously approved these changes Jun 11, 2020

View reviewed changes

fix: use py3 when using pytorch_full_version

9b92c02

metrizable dismissed laurenyu’s stale review via 9b92c02 June 11, 2020 14:47

chuyang-deng reviewed Jun 11, 2020

View reviewed changes

src/sagemaker/pytorch/model.py Outdated Show resolved Hide resolved

tests/integ/test_airflow_config.py Outdated Show resolved Hide resolved

tests/integ/test_pytorch_train.py Show resolved Hide resolved

laurenyu reviewed Jun 11, 2020

View reviewed changes

tests/integ/test_git.py Show resolved Hide resolved

tests/integ/test_pytorch_train.py Show resolved Hide resolved

metrizable added 2 commits June 11, 2020 10:43

breaking: updates based on PR feedback

839be3c

Merge remote-tracking branch 'aws/zwei' into require-framework-versio…

8527ed4

…n-pytorch

metrizable force-pushed the require-framework-version-pytorch branch from a9c51b5 to 8527ed4 Compare June 11, 2020 17:48

laurenyu approved these changes Jun 11, 2020

View reviewed changes

metrizable merged commit 7919331 into aws:zwei Jun 11, 2020

metrizable deleted the require-framework-version-pytorch branch June 11, 2020 19:29

pintaoz-aws pushed a commit that referenced this pull request Dec 4, 2024

Mask Sensitive Env Logs in Container (#1568)

8cc19a3

pintaoz-aws pushed a commit that referenced this pull request Dec 4, 2024

Mask Sensitive Env Logs in Container (#1568)

18d3cda

breaking: require framework_version, py_version for pytorch #1568

breaking: require framework_version, py_version for pytorch #1568

Uh oh!

Conversation

metrizable commented Jun 10, 2020

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

laurenyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 11, 2020