Skip to content

Commit abea136

Browse files
akrishna1995qidewenwhenlopezfelipeDewen Qiphilipskokoh
authored
Reinvent keynote3 (#4490)
* feat: Add notebooks for step decorator (#272) * UCI heart example for step decorator with EMR step for preprocessing (#266) * UCI heart example for step decorator with EMR step for preprocessing * UCI heart example for step decorator with EMR step, after linter * UCI heart example for step decorator with EMR step, after linter * removed content_types * using prod S3 bucket * using XGBClassifier * fix code format --------- Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> * add notebook example on basic pipeline for batch inference using step decorator (#264) * add basic pipeline for batch inference using step decorator * change Booster to XGBClassifier; incorporating feedback from aws/amazon-sagemaker-examples-staging#264 * fix minor typos * incorporate comments from PR aws/amazon-sagemaker-examples-staging#264 * incorporate feedback from aws/amazon-sagemaker-examples-staging#264 --------- Co-authored-by: pprasety <[email protected]> * Add remote-function notebook fixes (#265) Co-authored-by: svia3 <[email protected]> * Add pipeline step decorator quick start notebook (#267) add pipeline scheduler examples Address comments and refine Add pipeline step decorator ablone notebook Address review meeting comments Update clean up sections Add rate-based schedules back Udpate notebooks for Public Beta Fix the colliding endpoint name across different executions Upgrading pandas to fix ImportError in Studio DataScience 3.0 image add scheduler-light additions to quick_start notebook add scheduler-light additions to quick_start notebook fix invalid notebook json Update notebooks for GA Add modular package for lightsaber Add a simple notebook to demonstrate mix use of training step and step deco Address comments fix pipeline delete resouce leak issue in using_step_decorator notebook dummy commit Co-authored-by: Dewen Qi <[email protected]> * remove local SDK tar and retrieve SDK from public --------- Co-authored-by: Felipe Lopez <[email protected]> Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Philips Kokoh <[email protected]> Co-authored-by: pprasety <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> * Notebook Job Step Example (#274) * Create README.md * Adding notebooks * Delete sagemaker-pipelines/notebook-job-step/README.md * Adding example for inference components and managed instance scaling for SageMaker real time hosting and inference (#275) * Cleaned up notebooks Cleaned up for initial push to staging * removed references to goldfinch * Updated readme * moved to proper directory * fixed session object reference * Updated session variable * Updated with logic to check store vars. Need to remove internal only code * linted notebooks and added test header and footers * Fixed prompt and parameters for codegen25 * Updated descriptions, added handling * Removed custom model shapes * Making Jumpstart notebooks Python 3.10 compatible (#269) * removed roleARN which is not needed * smart sifting notebooks (#281) * smart sifting notebooks * smart sifting notebooks updated description * Added new flow diagram --------- Co-authored-by: Arun Lokanatha <[email protected]> --------- Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Felipe Lopez <[email protected]> Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Philips Kokoh <[email protected]> Co-authored-by: pprasety <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> Co-authored-by: Ram Vegiraju <[email protected]> Co-authored-by: James Park <[email protected]> Co-authored-by: Pooja Karadgi <[email protected]> Co-authored-by: Arun Lokanatha <[email protected]> Co-authored-by: Arun Lokanatha <[email protected]>
1 parent ef36606 commit abea136

File tree

62 files changed

+17917
-66
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+17917
-66
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,6 @@
77
**/_build
88
*.iml
99
tox.ini
10+
11+
**/sagemaker-pipelines/step-decorator/**/emr-examples/code
12+
**/sagemaker-pipelines/step-decorator/**/quick-start/**/dummy_train.py

‎README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,13 @@ More examples for models such as BERT and YOLOv5 can be found in [distributed_tr
189189
- [Train EleutherAI GPT-J with Model Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/11_train_gptj_smp_tensor_parallel_notebook.ipynb) shows how to train EleutherAI GPT-J with PyTorch and Tensor Parallelism technique in the SageMaker Model Parallelism Library.
190190
- [Train MaskRCNN with Data Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb) shows how to train MaskRCNN with PyTorch and SageMaker Data Parallelism Library.
191191

192+
### Amazon SageMaker Smart Sifting
193+
194+
These examples provide an Introduction to Smart Sifting library. Smart Sifting is a framework to speed up training of PyTorch models. The framework implements a set of algorithms that filter out inconsequential training examples during training, reducing the computational cost and accelerating the training process. It is configuration-driven and extensible, allowing users to add custom logic to transform their training examples into a filterable format. Smart sifting provides a generic utility for any DNN model, and can reduce the training cost by up to 35% in infrastructure cost.
195+
196+
- [Train Image Classification using Vision Transformer with Smart Sifting](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/smart_sifting/Image_Classification_VIT/Train_Image_classification.ipynb): This Example shows how to use Smart sifting to fine tune Vision Transformers for Image Classification.
197+
- [Train Text Classification using BERT with Smart Sifting](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/smart_sifting/Text_Classification_BERT/Train_text_classification.ipynb): This Example shows how to use Smart Sifting to fine tune BERT for Text Classification.
198+
192199
### Amazon SageMaker Clarify
193200

194201
These examples provide an introduction to SageMaker Clarify which provides machine learning developers with greater visibility into their training data and models so they can identify and limit bias and explain predictions.

‎inference/generativeai/llm-workshop/lab-inference-components-with-scaling/1_create_endpoint.ipynb

Lines changed: 949 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)