Fix: Accelerate packaging in ModelBuilder #4549

samruds · 2024-03-29T20:49:05Z

Issue #, if available:

Since accelerate is an extra dependencies for Model size calculations fix logic such that we log error if module not present. The flow continues as usual in the absence of accelerate making it an opt in feature for model size calculations and placement.

Ensure dependency manager for torch servers installs dependencies.

Also enabled the pt test and re-ran it.

Happy case logic:
Module is installed and we pass memory_size_mb which is checked in ModelBuilder._can_fit_on_single_gpu to return True or False

Module not installed

Catch import error in _total_inference_model_size_mib
Throw ValueError to ModelBuilder._can_fit_on_single_gpu and return False

Description of changes:

Testing done:

Integ test:

py310 run-test: commands[5] | pytest -v -s tests/integ/sagemaker/serve/test_serve_pt_happy.py::test_happy_pytorch_sagemaker_endpoint
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/samruds/.config/sagemaker/config.yaml
======================================================================================================================================================= test session starts =======================================================================================================================================================
platform linux -- Python 3.10.13, pytest-6.2.5, py-1.11.0, pluggy-1.4.0 -- /local/home/samruds/sagemaker-python-sdk/.tox/py310/bin/python
cachedir: .tox/py310/.pytest_cache
rootdir: /local/home/samruds/sagemaker-python-sdk, configfile: tox.ini
plugins: time-machine-2.14.0, anyio-4.3.0, rerunfailures-10.2, forked-1.6.0, cov-3.0.0, xdist-2.4.0, timeout-2.1.0
collected 1 item

tests/integ/sagemaker/serve/test_serve_pt_happy.py::test_happy_pytorch_sagemaker_endpoint[model_builder_inference_spec_schema_builder] ModelBuilder: WARNING:     model_path provided with no image_uri. Attempting to autodetect the image                    by loading the model using inference_spec.load()...
ModelBuilder: INFO:     Autodetecting image since image_uri was not provided in ModelBuilder()
ModelBuilder: WARNING:     Auto detection is only supported for single models DLCs with a framework backend.
ModelBuilder: INFO:     Autodetected framework is pytorch
ModelBuilder: INFO:     Autodetected framework version is 2.0.1
ModelBuilder: INFO:     Auto detected 763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:2.0.1-cpu-py310. Proceeding with the the deployment.
ModelBuilder: DEBUG:     Uploading the model resources to bucket=sagemaker-us-west-2-480675430175, key_prefix=session-default-prefix/pytorch-inference-2024-03-29-20-35-59-270.
Uploading model artifacts: 100%|█████████████████| 7901680/7901680 [00:00<00:00, 12174675.61bytes/s]
ModelBuilder: DEBUG:     Model resources uploaded to: s3://sagemaker-us-west-2-480675430175/session-default-prefix/pytorch-inference-2024-03-29-20-35-59-270/serve.tar.gz
ModelBuilder: INFO:     ModelBuilder will collect telemetry to help us better understand our user's needs, diagnose issues, and deliver additional features. To opt out of telemetry, please disable via TelemetryOptOut in intelligent defaults. See https://sagemaker.readthedocs.io/en/stable/overview.html#configuring-and-using-defaults-with-the-sagemaker-python-sdk for more info.
ModelBuilder: DEBUG:     ModelBuilder metrics emitted.
PASSED

======================================================================================================================================================== warnings summary =========================================================================================================================================================
.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3
  /local/home/samruds/sagemaker-python-sdk/.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================================================================================================ 1 passed, 1 warning in 220.53s (0:03:40) =============================================================================================================================================
_____________________________________________________________________________________________________________________________________________________________ summary _____________________________________________________________________________________________________________________________________________________________
  py310: commands succeeded
  congratulations :)
(pysdk-dev-310) [samruds@dev-dsk-samruds-2c-e20b7667 sagemaker-python-sdk]$

Notebook: Happy path

from sagemaker.serve.builder.model_builder import ModelBuilder
from sagemaker.serve.builder.schema_builder import SchemaBuilder
from sagemaker.serve import Mode
import json

import requests
import json

# Run inference
url = "http://localhost:8080/invocations"

sample_input = {
 "inputs": "The man worked as a [MASK].",
}

headers = {'Content-Type': 'application/json', "Accept":"application/json"}
res = requests.post(url, data=json.dumps(sample_input), headers=headers, timeout=60)
loaded_response = json.loads(res.text)

model_builder = ModelBuilder(model="bert-base-uncased",
                             mode=Mode.LOCAL_CONTAINER,
                             schema_builder=SchemaBuilder(sample_input, loaded_response),
                             instance_type='ml.g4dn.xlarge')
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml

%%time
builder = model_builder.build()
INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole
ModelBuilder: WARNING:     HuggingFace JumpStart Model ID not detected. Building for HuggingFace Model ID.
ModelBuilder: INFO:     GPU Info [g4dn.xlarge]: (1, 16384)
ModelBuilder: INFO:     GPU info (1, 16384) for instance ml.g4dn.xlarge
/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/cuda/__init__.py:628: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
Loading pretrained config for `bert-base-uncased` from `transformers`...
ModelBuilder: INFO:     Total memory size MIB: 501.179296875
ModelBuilder: INFO:     Total memory size MIB: 501.179296875
ModelBuilder: INFO:     Total inference model size MIB 501.179296875, single GPU size for instance MIB 16384.0
ModelBuilder: WARNING:     CUDA is not enabled on your device. Command '['nvidia-smi', '--query-gpu=name,memory.free', '--format=csv']' returned non-zero exit status 9.. Please run ModelBuilder on CUDA enabled hardware to deploy locally.
ModelBuilder: WARNING:     87.62856471008337 percent of disk space used. Please consider freeing up disk space or increasing the EBS volume if you are on a SageMaker Notebook.
ModelBuilder: WARNING:     87.62856471008337 percent of docker disk space at /var/lib/docker is used. Please consider freeing up disk space or increasing the EBS volume if you are on a SageMaker Notebook.
ModelBuilder: INFO:     {'HF_MODEL_ID': 'bert-base-uncased'}
Loading pretrained config for `bert-base-uncased` from `transformers`...
ModelBuilder: INFO:     Detected 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.7.1-transformers4.6.1-cpu-py36-ubuntu18.04. Proceeding with the the deployment.
CPU times: user 2.25 s, sys: 403 ms, total: 2.66 s
Wall time: 4.79 s

Notebook: Force uninstalled accelerate

from sagemaker.serve.builder.model_builder import ModelBuilder
from sagemaker.serve.builder.schema_builder import SchemaBuilder
from sagemaker.serve import Mode
import json

import requests
import json

# Run inference
url = "http://localhost:8080/invocations"

sample_input = {
 "inputs": "The man worked as a [MASK].",
}

headers = {'Content-Type': 'application/json', "Accept":"application/json"}
res = requests.post(url, data=json.dumps(sample_input), headers=headers, timeout=60)
loaded_response = json.loads(res.text)

model_builder = ModelBuilder(model="bert-base-uncased",
                             mode=Mode.LOCAL_CONTAINER,
                             schema_builder=SchemaBuilder(sample_input, loaded_response),
                             instance_type='ml.g4dn.xlarge')
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml
%%time
builder = model_builder.build()
INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole
ModelBuilder: WARNING:     HuggingFace JumpStart Model ID not detected. Building for HuggingFace Model ID.
ModelBuilder: INFO:     GPU Info [g4dn.xlarge]: (1, 16384)
ModelBuilder: INFO:     GPU info (1, 16384) for instance ml.g4dn.xlarge
ModelBuilder: ERROR:     To enable Model size calculations: Install HuggingFace extras dependencies using pip install 'sagemaker[huggingface]>=2.212.0'
ModelBuilder: INFO:     Unable to determine single GPU size for instance ml.g4dn.xlarge
ModelBuilder: WARNING:     CUDA is not enabled on your device. Command '['nvidia-smi', '--query-gpu=name,memory.free', '--format=csv']' returned non-zero exit status 9.. Please run ModelBuilder on CUDA enabled hardware to deploy locally.
ModelBuilder: WARNING:     88.19689381471613 percent of disk space used. Please consider freeing up disk space or increasing the EBS volume if you are on a SageMaker Notebook.
ModelBuilder: WARNING:     88.19689381471613 percent of docker disk space at /var/lib/docker is used. Please consider freeing up disk space or increasing the EBS volume if you are on a SageMaker Notebook.
ModelBuilder: INFO:     {'HF_MODEL_ID': 'bert-base-uncased'}
ModelBuilder: INFO:     Detected 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.7.1-transformers4.6.1-cpu-py36-ubuntu18.04. Proceeding with the the deployment.
CPU times: user 776 ms, sys: 33.5 ms, total: 809 ms
Wall time: 1.92 s

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
[] I have checked that my tests are not configured for a specific region or account (if appropriate)
[] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2024-03-29T21:15:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.44%. Comparing base (cb9aedb) to head (05dddc6).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4549   +/-   ##
=======================================
  Coverage   87.44%   87.44%           
=======================================
  Files         389      389           
  Lines       36885    36888    +3     
=======================================
+ Hits        32253    32256    +3     
  Misses       4632     4632

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/sagemaker/serve/utils/hardware_detector.py

samruds · 2024-03-29T22:39:57Z

py310 run-test: commands[5] | pytest tests/unit/sagemaker/serve/utils/test_hardware_detector.py
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.13, pytest-6.2.5, py-1.11.0, pluggy-1.4.0
cachedir: .tox/py310/.pytest_cache
rootdir: /local/home/samruds/sagemaker-python-sdk, configfile: tox.ini
plugins: time-machine-2.14.0, anyio-4.3.0, rerunfailures-10.2, forked-1.6.0, cov-3.0.0, xdist-2.4.0, timeout-2.1.0
collected 8 items

tests/unit/sagemaker/serve/utils/test_hardware_detector.py ........                                                                                     [100%]

====================================================================== warnings summary =======================================================================
.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3
  /local/home/samruds/sagemaker-python-sdk/.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================ 8 passed, 1 warning in 1.78s =================================================================
___________________________________________________________________________ summary ___________________________________________________________________________
  py310: commands succeeded
  congratulations :)

samruds · 2024-03-29T22:40:44Z

py310 run-test: commands[5] | pytest tests/unit/sagemaker/serve/builder/test_model_builder.py
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.13, pytest-6.2.5, py-1.11.0, pluggy-1.4.0
cachedir: .tox/py310/.pytest_cache
rootdir: /local/home/samruds/sagemaker-python-sdk, configfile: tox.ini
plugins: time-machine-2.14.0, anyio-4.3.0, rerunfailures-10.2, forked-1.6.0, cov-3.0.0, xdist-2.4.0, timeout-2.1.0
collected 29 items

tests/unit/sagemaker/serve/builder/test_model_builder.py .............................                                                                  [100%]

====================================================================== warnings summary =======================================================================
.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3
  /local/home/samruds/sagemaker-python-sdk/.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================ 29 passed, 1 warning in 1.60s ================================================================
___________________________________________________________________________ summary ___________________________________________________________________________
  py310: commands succeeded
  congratulations :)

samruds · 2024-03-29T22:47:40Z

py310 run-test: commands[5] | pytest tests/unit/sagemaker/serve/detector/test_dependency_manager.py
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.13, pytest-6.2.5, py-1.11.0, pluggy-1.4.0
cachedir: .tox/py310/.pytest_cache
rootdir: /local/home/samruds/sagemaker-python-sdk, configfile: tox.ini
plugins: time-machine-2.14.0, anyio-4.3.0, rerunfailures-10.2, forked-1.6.0, cov-3.0.0, xdist-2.4.0, timeout-2.1.0
collected 3 items

tests/unit/sagemaker/serve/detector/test_dependency_manager.py ...                                                                                      [100%]

====================================================================== warnings summary =======================================================================
.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3
  /local/home/samruds/sagemaker-python-sdk/.tox/py310/lib/python3.10/site-packages/invoke/loader.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================ 3 passed, 1 warning in 0.28s =================================================================
___________________________________________________________________________ summary ___________________________________________________________________________
  py310: commands succeeded
  congratulations :)

sagemaker-bot · 2024-03-29T23:37:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 05dddc6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* Move module level accelerate import to be function level * Ensure Pt test runs * Fix formatting * Fix flake8 * Fix local import patch path * Add coverage for import error * Fix dependency manager ut

samruds added 3 commits March 29, 2024 12:39

Move module level accelerate import to be function level

cb387bc

Ensure Pt test runs

d004f3c

Fix formatting

2e9dd0a

samruds requested a review from a team as a code owner March 29, 2024 20:49

samruds requested review from liujiaorr and jiapinw and removed request for a team and liujiaorr March 29, 2024 20:49

Fix flake8

d8b599b

samruds force-pushed the fixTest branch from 0389c5b to f35b373 Compare March 29, 2024 21:32

Fix local import patch path

abd6852

samruds force-pushed the fixTest branch from f35b373 to abd6852 Compare March 29, 2024 21:34

Add coverage for import error

31eed78

jiapinw suggested changes Mar 29, 2024

View reviewed changes

src/sagemaker/serve/utils/hardware_detector.py Show resolved Hide resolved

jiapinw approved these changes Mar 29, 2024

View reviewed changes

samruds requested a review from akrishna1995 March 29, 2024 22:41

Fix dependency manager ut

10a5b8f

Merge branch 'master' into fixTest

05dddc6

akrishna1995 approved these changes Mar 29, 2024

View reviewed changes

akrishna1995 merged commit 2caf5df into aws:master Mar 29, 2024

samruds deleted the fixTest branch March 29, 2024 23:26

This was referenced Mar 30, 2024

Model cannot be loaded in the SageMaker endpoint after update of SageMaker SDK to 2.212 #4488

Open

feature: Add and use sagemaker_schema_inference_artifacts dependency for huggingface in schema builder (question-answering only) #4554

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix: Accelerate packaging in ModelBuilder #4549

Fix: Accelerate packaging in ModelBuilder #4549

Uh oh!

samruds commented Mar 29, 2024

Uh oh!

codecov bot commented Mar 29, 2024 •

edited

Loading

Uh oh!

Uh oh!

samruds commented Mar 29, 2024 •

edited

Loading

Uh oh!

samruds commented Mar 29, 2024 •

edited

Loading

Uh oh!

samruds commented Mar 29, 2024

Uh oh!

sagemaker-bot commented Mar 29, 2024

Uh oh!

Uh oh!

Fix: Accelerate packaging in ModelBuilder #4549

Fix: Accelerate packaging in ModelBuilder #4549

Uh oh!

Conversation

samruds commented Mar 29, 2024

Merge Checklist

General

Tests

Uh oh!

codecov bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

samruds commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samruds commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samruds commented Mar 29, 2024

Uh oh!

sagemaker-bot commented Mar 29, 2024

AWS CodeBuild CI Report

Uh oh!

Uh oh!

codecov bot commented Mar 29, 2024 •

edited

Loading

samruds commented Mar 29, 2024 •

edited

Loading

samruds commented Mar 29, 2024 •

edited

Loading