Skip to content

documentation: Jumpstart doc strings and added new sections #2893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 54 commits into from
Feb 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
b09793a
feature: client cache for jumpstart models
evakravi Dec 7, 2021
1da6b98
feature: allow conditional parellel builds (#2727)
mufaddal-rohawala Nov 4, 2021
1489363
fix endpoint bug (#2772)
BasilBeirouti Dec 6, 2021
ba55962
fix: local mode - support relative file structure (#2768)
mufaddal-rohawala Dec 7, 2021
221411c
prepare release v2.72.0
Dec 13, 2021
395bc24
update development version to v2.72.1.dev0
Dec 13, 2021
b689689
fix: Set ProcessingStep upload locations deterministically to avoid c…
staubhp Dec 8, 2021
5cd25f3
fix: Prevent repack_model script from referencing nonexistent directo…
staubhp Dec 9, 2021
2e8710d
fix: S3Input - add support for instance attributes (#2754)
mufaddal-rohawala Dec 15, 2021
05478a0
fix: typos and broken link (#2765)
mohamed-ali Dec 16, 2021
7da8254
prepare release v2.72.1
Dec 20, 2021
51404ba
update development version to v2.72.2.dev0
Dec 20, 2021
5e83d5c
fix: Model Registration with BYO scripts (#2797)
sreedes Dec 17, 2021
7def439
fix: Add ContentType in test_auto_ml_describe
navinns Dec 27, 2021
cd870ec
fix: Re-deploy static integ test endpoint if it is not found
Dec 27, 2021
116ce8a
documentation :SageMaker model parallel library 1.6.0 API doc (#2814)
mchoi8739 Dec 30, 2021
9ebfaf1
fix: fix kmeans test deletion sequence, increment lineage statics (#2…
mufaddal-rohawala Dec 31, 2021
071228a
fix: Increment static lineage pipeline (#2817)
mufaddal-rohawala Jan 3, 2022
d36d9aa
fix: Update CHANGELOG.md (#2832)
ahsan-z-khan Jan 6, 2022
02e0b15
prepare release v2.72.2
Jan 6, 2022
ee247c0
update development version to v2.72.3.dev0
Jan 6, 2022
20740cb
change: update master from dev (#2836)
ahsan-z-khan Jan 10, 2022
3d491ac
prepare release v2.72.3
Jan 10, 2022
068ad64
update development version to v2.72.4.dev0
Jan 10, 2022
3859a94
fix: fixes unnecessary session call while generating pipeline definit…
xchen909 Jan 10, 2022
77bf04f
feature: Add models_v2 under lineage context (#2800)
yzhu0 Jan 10, 2022
5da5364
feature: enable python 3.9 (#2802)
mufaddal-rohawala Jan 10, 2022
b105b19
change: Update CHANGELOG.md (#2842)
shreyapandit Jan 11, 2022
ac57772
fix: update pricing link (#2805)
ahsan-z-khan Jan 11, 2022
b691d3d
feature: Adding Jumpstart retrieval functions (#2789)
evakravi Jan 12, 2022
00f23e6
feature: jumpstart hyperparameters and environment variables (#2850)
evakravi Jan 14, 2022
167b723
feature: script mode for model class (#2841)
evakravi Jan 19, 2022
d9d8c68
feat: Script mode support for Estimator class (#2834)
evakravi Jan 21, 2022
c03efb2
feature: jumpstart vulnerability and deprecated check (#2855)
evakravi Jan 23, 2022
63b0372
Feat: tagging jumpstart models (#2860)
evakravi Jan 24, 2022
c9aa29b
feat: Integ tests for jumpstart model and estimator (#2865)
evakravi Jan 25, 2022
423f389
feat: Hyperparameter validation (#2856)
evakravi Jan 25, 2022
39b524b
edited doc strings and added new sections
Jan 28, 2022
239099e
restructuring fixes
Jan 28, 2022
74593be
fixed formatting issues
Jan 28, 2022
0c2eab3
fixed whitespace errors
Jan 28, 2022
69aa401
fixed formatting issues
Feb 2, 2022
6bd1585
Merge branch 'dev' into jumpstart
IvyBazan Feb 3, 2022
b4ddac3
Update VERSION
IvyBazan Feb 3, 2022
3f90257
Update .coveragerc
IvyBazan Feb 3, 2022
5fb9c0e
Update test_steps.py
IvyBazan Feb 3, 2022
d73ab26
Update environment_variables.rst
IvyBazan Feb 3, 2022
cac1b17
Update hyperparameters.rst
IvyBazan Feb 3, 2022
74c41cc
Update model_uris.rst
IvyBazan Feb 3, 2022
0939ed3
Update script_uris.rst
IvyBazan Feb 3, 2022
65c203b
Update environment_variables.rst
IvyBazan Feb 3, 2022
b4f08e0
Update hyperparameters.rst
IvyBazan Feb 3, 2022
da85582
Update model_uris.rst
IvyBazan Feb 3, 2022
172c68f
Update script_uris.rst
IvyBazan Feb 3, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions doc/api/utility/environment_variables.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Environment Variables
---------------------

.. automodule:: sagemaker.environment_variables
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions doc/api/utility/hyperparameters.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Hyperparameters
---------------

.. automodule:: sagemaker.hyperparameters
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions doc/api/utility/model_uris.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Model URIs
----------

.. automodule:: sagemaker.model_uris
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions doc/api/utility/script_uris.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Script URIs
-----------

.. automodule:: sagemaker.script_uris
:members:
:undoc-members:
:show-inheritance:
10 changes: 5 additions & 5 deletions src/sagemaker/environment_variables.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ def retrieve_default(
"""Retrieves the default container environment variables for the model matching the arguments.

Args:
region (str): Optional. Region for which to retrieve default environment variables.
(Default: None).
model_id (str): Optional. Model ID of the model for which to
region (str): Optional. The AWS Region for which to retrieve the default environment
variables. (Default: None).
model_id (str): Optional. The model ID of the model for which to
retrieve the default environment variables. (Default: None).
model_version (str): Optional. Version of the model for which to retrieve the
model_version (str): Optional. The version of the model for which to retrieve the
default environment variables. (Default: None).
Returns:
dict: the variables to use for the model.
dict: The variables to use for the model.

Raises:
ValueError: If the combination of arguments specified is not supported.
Expand Down
214 changes: 108 additions & 106 deletions src/sagemaker/estimator.py

Large diffs are not rendered by default.

33 changes: 17 additions & 16 deletions src/sagemaker/hyperparameters.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,20 +34,21 @@ def retrieve_default(
"""Retrieves the default training hyperparameters for the model matching the given arguments.

Args:
region (str): Region for which to retrieve default hyperparameters. (Default: None).
model_id (str): Model ID of the model for which to
region (str): The AWS Region for which to retrieve the default hyperparameters.
Defaults to ``None``.
model_id (str): The model ID of the model for which to
retrieve the default hyperparameters. (Default: None).
model_version (str): Version of the model for which to retrieve the
model_version (str): The version of the model for which to retrieve the
default hyperparameters. (Default: None).
include_container_hyperparameters (bool): True if container hyperparameters
should be returned as well. Container hyperparameters are not used to tune
the specific algorithm, but rather by SageMaker Training to setup
include_container_hyperparameters (bool): ``True`` if the container hyperparameters
should be returned. Container hyperparameters are not used to tune
the specific algorithm. They are used by SageMaker Training jobs to set up
the training container environment. For example, there is a container hyperparameter
that indicates the entrypoint script to use. These hyperparameters may be required
when creating a training job with boto3, however the ``Estimator`` classes
should take care of adding container hyperparameters to the job. (Default: False).
add required container hyperparameters to the job. (Default: False).
Returns:
dict: the hyperparameters to use for the model.
dict: The hyperparameters to use for the model.

Raises:
ValueError: If the combination of arguments specified is not supported.
Expand All @@ -69,22 +70,22 @@ def validate(
hyperparameters: Optional[dict] = None,
validation_mode: Optional[HyperparameterValidationMode] = None,
) -> None:
"""Validate hyperparameters for models.
"""Validates hyperparameters for models.

Args:
region (str): Region for which to validate hyperparameters. (Default: None).
model_id (str): Model ID of the model for which to validate hyperparameters.
(Default: None)
model_version (str): Version of the model for which to validate hyperparameters.
(Default: None)
region (str): The AWS Region for which to validate hyperparameters. (Default: None).
model_id (str): The model ID of the model for which to validate hyperparameters.
(Default: None).
model_version (str): The version of the model for which to validate hyperparameters.
(Default: None).
hyperparameters (dict): Hyperparameters to validate.
(Default: None)
(Default: None).
validation_mode (HyperparameterValidationMode): Method of validation to use with
hyperparameters. If set to ``VALIDATE_PROVIDED``, only hyperparameters provided
to this function will be validated, the missing hyperparameters will be ignored.
If set to``VALIDATE_ALGORITHM``, all algorithm hyperparameters will be validated.
If set to ``VALIDATE_ALL``, all hyperparameters for the model will be validated.
(Default: None)
(Default: None).

Raises:
JumpStartHyperparametersError: If the hyperparameter is not formatted correctly,
Expand Down
26 changes: 13 additions & 13 deletions src/sagemaker/image_uris.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,20 +77,20 @@ def retrieve(
training_compiler_config (:class:`~sagemaker.training_compiler.TrainingCompilerConfig`):
A configuration class for the SageMaker Training Compiler
(default: None).
model_id (str): JumpStart model ID for which to retrieve image URI
model_id (str): The JumpStart model ID for which to retrieve the image URI
(default: None).
model_version (str): Version of the JumpStart model for which to retrieve the
model_version (str): The version of the JumpStart model for which to retrieve the
image URI (default: None).
tolerate_vulnerable_model (bool): True if vulnerable versions of model specifications
should be tolerated (exception not raised). If False, raises an exception if
tolerate_vulnerable_model (bool): ``True`` if vulnerable versions of model specifications
should be tolerated without an exception raised. If ``False``, raises an exception if
the script used by this version of the model has dependencies with known security
vulnerabilities. (Default: False).
tolerate_deprecated_model (bool): True if deprecated versions of model specifications
should be tolerated (exception not raised). If False, raises an exception
should be tolerated without an exception raised. If False, raises an exception
if the version of the model is deprecated. (Default: False).

Returns:
str: the ECR URI for the corresponding SageMaker Docker image.
str: The ECR URI for the corresponding SageMaker Docker image.

Raises:
NotImplementedError: If the scope is not supported.
Expand Down Expand Up @@ -404,26 +404,26 @@ def get_training_image_uri(
pytorch_version=None,
instance_type=None,
) -> str:
"""Retrieve image uri for training.
"""Retrieves the image URI for training.

Args:
region (str): AWS region to use for image URI.
region (str): The AWS region to use for image URI.
framework (str): The framework for which to retrieve an image URI.
framework_version (str): The framework version for which to retrieve an
image URI (default: None).
py_version (str): The python version to use for the image (default: None).
image_uri (str): If an image URI is supplied, it will be returned (default: None).
image_uri (str): If an image URI is supplied, it is returned (default: None).
distribution (dict): A dictionary with information on how to run distributed
training (default: None).
compiler_config (:class:`~sagemaker.training_compiler.TrainingCompilerConfig`):
A configuration class for the SageMaker Training Compiler
(default: None).
tensorflow_version (str): Version of tensorflow to use. (default: None)
pytorch_version (str): Version of pytorch to use. (default: None)
instance_type (str): Instance type fo use. (default: None)
tensorflow_version (str): The version of TensorFlow to use. (default: None)
pytorch_version (str): The version of PyTorch to use. (default: None)
instance_type (str): The instance type to use. (default: None)

Returns:
str: the image URI string.
str: The image URI string.
"""

if image_uri:
Expand Down
73 changes: 38 additions & 35 deletions src/sagemaker/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,14 +135,14 @@ def __init__(
model container is pulled from ECR, or private registry in your
VPC. By default it is set to pull model container image from
ECR. (default: None).
source_dir (str): Path (absolute, relative or an S3 URI) to a directory
source_dir (str): The absolute, relative, or S3 URI Path to a directory
with any other training source code dependencies aside from the entry
point file (default: None). If ``source_dir`` is an S3 URI, it must
point to a tar.gz file. Structure within this directory are preserved
point to a tar.gz file. Structure within this directory is preserved
when training on Amazon SageMaker. If 'git_config' is provided,
'source_dir' should be a relative location to a directory in the Git repo.
If the directory points to S3, no code will be uploaded and the S3 location
will be used instead.
If the directory points to S3, no code is uploaded and the S3 location
is used instead.

.. admonition:: Example

Expand All @@ -155,14 +155,14 @@ def __init__(

You can assign entry_point='inference.py', source_dir='src'.
code_location (str): Name of the S3 bucket where custom code is
uploaded (default: None). If not specified, default bucket
uploaded (default: None). If not specified, the default bucket
created by ``sagemaker.session.Session`` is used.
entry_point (str): Path (absolute or relative) to the Python source
file which should be executed as the entry point to model
hosting (default: None). If ``source_dir`` is specified,
then ``entry_point`` must point to a file located at the root of
``source_dir``. If 'git_config' is provided, 'entry_point' should
be a relative location to the Python source file in the Git repo.
entry_point (str): The absolute or relative path to the local Python
source file that should be executed as the entry point to
model hosting. (Default: None). If ``source_dir`` is specified, then ``entry_point``
must point to a file located at the root of ``source_dir``.
If 'git_config' is provided, 'entry_point' should be
a relative location to the Python source file in the Git repo.

Example:
With the following GitHub repo directory structure:
Expand All @@ -176,9 +176,9 @@ def __init__(
container_log_level (int): Log level to use within the container
(default: logging.INFO). Valid values are defined in the Python
logging module.
dependencies (list[str]): A list of paths to directories (absolute
or relative) with any additional libraries that will be exported
to the container (default: []). The library folders will be
dependencies (list[str]): A list of absolute or relative paths to directories
with any additional libraries that should be exported
to the container (default: []). The library folders are
copied to SageMaker in the same folder where the entrypoint is
copied. If 'git_config' is provided, 'dependencies' should be a
list of relative locations to directories with any additional
Expand All @@ -193,7 +193,7 @@ def __init__(
>>> Model(entry_point='inference.py',
... dependencies=['my/libs/common', 'virtual-env'])

results in the following inside the container:
results in the following structure inside the container:

>>> $ ls

Expand All @@ -210,7 +210,9 @@ def __init__(
``repo`` specifies the Git repository where your training script
is stored. If you don't provide ``branch``, the default value
'master' is used. If you don't provide ``commit``, the latest
commit in the specified branch is used. .. admonition:: Example
commit in the specified branch is used.

.. admonition:: Example

The following config:

Expand All @@ -219,7 +221,7 @@ def __init__(
>>> 'commit': '329bfcf884482002c05ff7f44f62599ebc9f445a'}

results in cloning the repo specified in 'repo', then
checkout the 'master' branch, and checkout the specified
checking out the 'master' branch, and checking out the specified
commit.

``2FA_enabled``, ``username``, ``password`` and ``token`` are
Expand All @@ -232,26 +234,25 @@ def __init__(
repositories.

For GitHub and other Git repos, when SSH URLs are provided, it
doesn't matter whether 2FA is enabled or disabled; you should
either have no passphrase for the SSH key pairs, or have the
ssh-agent configured so that you will not be prompted for SSH
passphrase when you do 'git clone' command with SSH URLs. When
HTTPS URLs are provided: if 2FA is disabled, then either token
or username+password will be used for authentication if provided
(token prioritized); if 2FA is enabled, only token will be used
doesn't matter whether 2FA is enabled or disabled. You should
either have no passphrase for the SSH key pairs or have the
ssh-agent configured so that you will not be prompted for the SSH
passphrase when you run the 'git clone' command with SSH URLs. When
HTTPS URLs are provided, if 2FA is disabled, then either ``token``
or ``username`` and ``password`` are be used for authentication if provided.
``Token`` is prioritized. If 2FA is enabled, only ``token`` is used
for authentication if provided. If required authentication info
is not provided, python SDK will try to use local credentials
storage to authenticate. If that fails either, an error message
will be thrown.
is not provided, the SageMaker Python SDK attempts to use local credentials
to authenticate. If that fails, an error message is thrown.

For CodeCommit repos, 2FA is not supported, so '2FA_enabled'
For CodeCommit repos, 2FA is not supported, so ``2FA_enabled``
should not be provided. There is no token in CodeCommit, so
'token' should not be provided too. When 'repo' is an SSH URL,
the requirements are the same as GitHub-like repos. When 'repo'
is an HTTPS URL, username+password will be used for
authentication if they are provided; otherwise, python SDK will
try to use either CodeCommit credential helper or local
credential storage for authentication.
``token`` should also not be provided. When ``repo`` is an SSH URL,
the requirements are the same as GitHub repos. When ``repo``
is an HTTPS URL, ``username`` and ``password`` are used for
authentication if they are provided. If they are not provided,
the SageMaker Python SDK attempts to use either the CodeCommit
credential helper or local credential storage for authentication.

"""
self.model_data = model_data
Expand Down Expand Up @@ -1313,7 +1314,9 @@ def __init__(
``repo`` specifies the Git repository where your training script
is stored. If you don't provide ``branch``, the default value
'master' is used. If you don't provide ``commit``, the latest
commit in the specified branch is used. .. admonition:: Example
commit in the specified branch is used.

.. admonition:: Example

The following config:

Expand Down
22 changes: 11 additions & 11 deletions src/sagemaker/model_uris.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
"""Accessors to retrieve the model artifact S3 URI of pretrained ML models."""
"""Accessors to retrieve the model artifact S3 URI of pretrained machine learning models."""
from __future__ import absolute_import

import logging
Expand All @@ -31,25 +31,25 @@ def retrieve(
tolerate_vulnerable_model: bool = False,
tolerate_deprecated_model: bool = False,
) -> str:
"""Retrieves the model artifact S3 URI for the model matching the given arguments.
"""Retrieves the model artifact Amazon S3 URI for the model matching the given arguments.

Args:
region (str): Region for which to retrieve model S3 URI.
model_id (str): JumpStart model ID of the JumpStart model for which to retrieve
region (str): The AWS Region for which to retrieve the Jumpstart model S3 URI.
model_id (str): The model ID of the JumpStart model for which to retrieve
the model artifact S3 URI.
model_version (str): Version of the JumpStart model for which to retrieve
model_version (str): The version of the JumpStart model for which to retrieve
the model artifact S3 URI.
model_scope (str): The model type, i.e. what it is used for.
model_scope (str): The model type.
Valid values: "training" and "inference".
tolerate_vulnerable_model (bool): True if vulnerable versions of model
specifications should be tolerated (exception not raised). If False, raises an
tolerate_vulnerable_model (bool): ``True`` if vulnerable versions of model
specifications should be tolerated without raising an exception. If ``False``, raises an
exception if the script used by this version of the model has dependencies with known
security vulnerabilities. (Default: False).
tolerate_deprecated_model (bool): True if deprecated versions of model
specifications should be tolerated (exception not raised). If False, raises
tolerate_deprecated_model (bool): ``True`` if deprecated versions of model
specifications should be tolerated without raising an exception. If ``False``, raises
an exception if the version of the model is deprecated. (Default: False).
Returns:
str: the model artifact S3 URI for the corresponding model.
str: The model artifact S3 URI for the corresponding model.

Raises:
NotImplementedError: If the scope is not supported.
Expand Down
Loading