Skip to content

doc: clean up MXNet and TF documentation #865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 19, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ As a managed service, Amazon SageMaker performs operations on your behalf on the
Amazon SageMaker can perform only operations that the user permits.
You can read more about which permissions are necessary in the `AWS Documentation <https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html>`__.

The SageMaker Python SDK should not require any additional permissions.
The SageMaker Python SDK should not require any additional permissions aside from what is required for using SageMaker.
However, if you are using an IAM role with a path in it, you should grant permission for ``iam:GetRole``.

Licensing
Expand Down Expand Up @@ -117,27 +117,28 @@ To run the unit tests with tox, run:
To run the integration tests, the following prerequisites must be met

1. AWS account credentials are available in the environment for the boto3 client to use.
2. The AWS account has an IAM role named :code:`SageMakerRole` with the AmazonSageMakerFullAccess policy attached.
2. The AWS account has an IAM role named :code:`SageMakerRole`.
It should have the AmazonSageMakerFullAccess policy attached as well as a policy with `the necessary permissions to use Elastic Inference <https://docs.aws.amazon.com/sagemaker/latest/dg/ei-setup.html>`__.

We recommend selectively running just those integration tests you'd like to run. You can filter by individual test function names with:

::

pytest -k 'test_i_care_about'
tox -- -k 'test_i_care_about'


You can also run all of the integration tests by running the following command, which runs them in sequence, which may take a while:

::

pytest tests/integ
tox -- tests/integ


You can also run them in parallel:

::

pytest -n auto tests/integ
tox -- -n auto tests/integ


Building Sphinx docs
Expand All @@ -149,16 +150,16 @@ Building Sphinx docs

make html

You can edit the templates for any of the pages in the docs by editing the .rst files in the "doc" directory and then running "``make html``" again.
You can edit the templates for any of the pages in the docs by editing the .rst files in the ``doc`` directory and then running ``make html`` again.

MXNet SageMaker Estimators
--------------------------

By using MXNet SageMaker ``Estimators``, you can train and host MXNet models on Amazon SageMaker.
By using MXNet SageMaker Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.3.0``, ``1.2.1``, ``1.1.0``, ``1.0.0``, ``0.12.1``.
Supported versions of MXNet: ``0.12.1``. ``1.0.0``, ``1.1.0``, ``1.2.1``, ``1.3.0``, ``1.4.0``

Supported versions of MXNet for Elastic Inference: ``1.3.0``
Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

Expand All @@ -170,7 +171,7 @@ For more information, see `Using MXNet with the SageMaker Python SDK`_.
TensorFlow SageMaker Estimators
-------------------------------

By using TensorFlow SageMaker ``Estimators``, you can train and host TensorFlow models on Amazon SageMaker.
By using TensorFlow SageMaker Estimators, you can train and host TensorFlow models on Amazon SageMaker.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``.

Expand All @@ -186,47 +187,47 @@ For more information, see `Using TensorFlow with the SageMaker Python SDK`_.
Chainer SageMaker Estimators
----------------------------

By using Chainer SageMaker ``Estimators``, you can train and host Chainer models on Amazon SageMaker.
By using Chainer SageMaker Estimators, you can train and host Chainer models on Amazon SageMaker.

Supported versions of Chainer: ``4.0.0``, ``4.1.0``, ``5.0.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about Chainer, see https://github.com/chainer/chainer.

For more information about Chainer SageMaker ``Estimators``, see `Using Chainer with the SageMaker Python SDK`_.
For more information about Chainer SageMaker Estimators, see `Using Chainer with the SageMaker Python SDK`_.

.. _Using Chainer with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_chainer.html


PyTorch SageMaker Estimators
----------------------------

With PyTorch SageMaker ``Estimators``, you can train and host PyTorch models on Amazon SageMaker.
With PyTorch SageMaker Estimators, you can train and host PyTorch models on Amazon SageMaker.

Supported versions of PyTorch: ``0.4.0``, ``1.0.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about PyTorch, see https://github.com/pytorch/pytorch.

For more information about PyTorch SageMaker ``Estimators``, see `Using PyTorch with the SageMaker Python SDK`_.
For more information about PyTorch SageMaker Estimators, see `Using PyTorch with the SageMaker Python SDK`_.

.. _Using PyTorch with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_pytorch.html


Scikit-learn SageMaker Estimators
---------------------------------

With Scikit-learn SageMaker ``Estimators``, you can train and host Scikit-learn models on Amazon SageMaker.
With Scikit-learn SageMaker Estimators, you can train and host Scikit-learn models on Amazon SageMaker.

Supported versions of Scikit-learn: ``0.20.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about Scikit-learn, see https://scikit-learn.org/stable/

For more information about Scikit-learn SageMaker ``Estimators``, see `Using Scikit-learn with the SageMaker Python SDK`_.
For more information about Scikit-learn SageMaker Estimators, see `Using Scikit-learn with the SageMaker Python SDK`_.

.. _Using Scikit-learn with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_sklearn.html

Expand All @@ -236,13 +237,13 @@ SageMaker Reinforcement Learning Estimators

With Reinforcement Learning (RL) Estimators, you can use reinforcement learning to train models on Amazon SageMaker.

Supported versions of Coach: ``0.10.1`` with TensorFlow, ``0.11.0`` with TensorFlow or MXNet.
Supported versions of Coach: ``0.10.1``, ``0.11.1`` with TensorFlow, ``0.11.0`` with TensorFlow or MXNet.
For more information about Coach, see https://github.com/NervanaSystems/coach

Supported versions of Ray: ``0.5.3`` with TensorFlow.
Supported versions of Ray: ``0.5.3``, ``0.6.5`` with TensorFlow.
For more information about Ray, see https://github.com/ray-project/ray

For more information about SageMaker RL ``Estimators``, see `SageMaker Reinforcement Learning Estimators`_.
For more information about SageMaker RL Estimators, see `SageMaker Reinforcement Learning Estimators`_.

.. _SageMaker Reinforcement Learning Estimators: src/sagemaker/rl/README.rst

Expand Down
25 changes: 12 additions & 13 deletions doc/using_mxnet.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,10 @@ In the following sections, we'll discuss how to prepare a training script for ex
Preparing the MXNet training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| WARNING |
+==========================================================================================================================================================+
| The structure for training scripts changed starting at MXNet version 1.3. |
| Make sure you refer to the correct section of this README when you prepare your script. |
| For information on how to upgrade an old script to the new format, see `"Updating your MXNet training script" <#updating-your-mxnet-training-script>`__. |
+----------------------------------------------------------------------------------------------------------------------------------------------------------+
.. warning::
The structure for training scripts changed starting at MXNet version 1.3.
Make sure you refer to the correct section of this README when you prepare your script.
For information on how to upgrade an old script to the new format, see `"Updating your MXNet training script" <#updating-your-mxnet-training-script>`__.

For versions 1.3 and higher
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -142,7 +139,7 @@ You don't have to use all the arguments, arguments you don't care about can be i
def train(hyperparameters, num_gpus, **kwargs):
pass

**Note: Writing a training script that imports correctly**
**Note: Writing a training script that imports correctly:**
When SageMaker runs your training script, it imports it as a Python module and then invokes ``train`` on the imported module. Consequently, you should not include any statements that won't execute successfully in SageMaker when your module is imported. For example, don't attempt to open any local files in top-level statements in your training script.

If you want to run your training script locally via the Python interpreter, look at using a ``___name__ == '__main__'`` guard, discussed in more detail here: https://stackoverflow.com/questions/419163/what-does-if-name-main-do .
Expand Down Expand Up @@ -701,12 +698,12 @@ This one function should handle processing the input, performing a prediction, a
The return object should be one of the following:

For versions 1.4 and higher:
----------------------------

- a tuple with two items: the response data and ``accept_type`` (the content type of the response data), or
- the response data: (the content type of the response will be set to either the accept header in the initial request or default to application/json)
- the response data: (the content type of the response will be set to either the accept header in the initial request or default to "application/json")

For versions 1.3 and lower:
---------------------------

- a tuple with two items: the response data and ``accept_type`` (the content type of the response data), or
- a Flask response object: http://flask.pocoo.org/docs/1.0/api/#response-objects

Expand Down Expand Up @@ -828,5 +825,7 @@ The Docker images extend Ubuntu 16.04.
You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are listed in the above table. You can also set ``framework_version`` to only specify major and minor version, e.g ``1.2``, which will cause your training script to be run on the latest supported patch version of that minor version, which in this example would be 1.2.1.
Alternatively, you can build your own image by following the instructions in the SageMaker MXNet containers repository, and passing ``image_name`` to the MXNet Estimator constructor.

You can visit the SageMaker MXNet training containers repository here: https://github.com/aws/sagemaker-mxnet-container
You can visit the SageMaker MXNet serving containers repository here: https://github.com/aws/sagemaker-mxnet-serving-container
You can visit the SageMaker MXNet container repositories here:

- training: https://github.com/aws/sagemaker-mxnet-container
- serving: https://github.com/aws/sagemaker-mxnet-serving-container
27 changes: 14 additions & 13 deletions doc/using_tf.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ models on SageMaker Hosting.

**Note:** This topic describes how to use script mode for TensorFlow versions 1.11 and later.
For Documentation of the previous Legacy Mode versions, see:

* `1.4.1 <https://github.com/aws/sagemaker-python-sdk/tree/v1.0.0#tensorflow-sagemaker-estimators>`_
* `1.5.0 <https://github.com/aws/sagemaker-python-sdk/tree/v1.1.0#tensorflow-sagemaker-estimators>`_
* `1.6.0 <https://github.com/aws/sagemaker-python-sdk/blob/v1.5.0/src/sagemaker/tensorflow/README.rst#tensorflow-sagemaker-estimators-and-models>`_
Expand All @@ -16,18 +17,15 @@ For Documentation of the previous Legacy Mode versions, see:
* `1.9.0 <https://github.com/aws/sagemaker-python-sdk/blob/v1.9.2/src/sagemaker/tensorflow/README.rst#tensorflow-sagemaker-estimators-and-models>`_
* `1.10.0 <https://github.com/aws/sagemaker-python-sdk/blob/v1.10.0/src/sagemaker/tensorflow/README.rst#tensorflow-sagemaker-estimators-and-models>`_

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| WARNING |
+=============================================================================================================================================================================+
| We have added a new format of your TensorFlow training script with TensorFlow version 1.11. |
| This new way gives the user script more flexibility. |
| This new format is called Script Mode, as opposed to Legacy Mode, which is what we support with TensorFlow 1.11 and older versions. |
| In addition we are adding Python 3 support with Script Mode. |
| Last supported version of Legacy Mode will be TensorFlow 1.12. |
| Script Mode is available with TensorFlow version 1.11 and newer. |
| Make sure you refer to the correct version of this README when you prepare your script. |
| You can find the Legacy Mode README `here <https://github.com/aws/sagemaker-python-sdk/tree/v1.12.0/src/sagemaker/tensorflow#tensorflow-sagemaker-estimators-and-models>`_. |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
.. warning::
We have added a new format of your TensorFlow training script with TensorFlow version 1.11.
This new way gives the user script more flexibility.
This new format is called Script Mode, as opposed to Legacy Mode, which is what we support with TensorFlow 1.11 and older versions.
In addition we are adding Python 3 support with Script Mode.
Last supported version of Legacy Mode will be TensorFlow 1.12.
Script Mode is available with TensorFlow version 1.11 and newer.
Make sure you refer to the correct version of this README when you prepare your script.
You can find the Legacy Mode README `here <https://github.com/aws/sagemaker-python-sdk/tree/v1.12.0/src/sagemaker/tensorflow#tensorflow-sagemaker-estimators-and-models>`_.

.. contents::

Expand Down Expand Up @@ -507,4 +505,7 @@ You can select version of TensorFlow by passing a ``framework_version`` keyword
Alternatively, you can build your own image by following the instructions in the SageMaker TensorFlow containers
repository, and passing ``image_name`` to the TensorFlow Estimator constructor.

For more information on the contents of the images, see the SageMaker TensorFlow containers repository here: https://github.com/aws/sagemaker-tensorflow-containers/
For more information on the contents of the images, see the SageMaker TensorFlow containers repositories here:

- training: https://github.com/aws/sagemaker-tensorflow-container
- serving: https://github.com/aws/sagemaker-tensorflow-serving-container
5 changes: 4 additions & 1 deletion src/sagemaker/tensorflow/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,7 @@ You can select version of TensorFlow by passing a ``framework_version`` keyword
Alternatively, you can build your own image by following the instructions in the SageMaker TensorFlow containers
repository, and passing ``image_name`` to the TensorFlow Estimator constructor.

For more information on the contents of the images, see the SageMaker TensorFlow containers repository here: https://github.com/aws/sagemaker-tensorflow-containers/
For more information on the contents of the images, see the SageMaker TensorFlow containers repositories here:

- training: https://github.com/aws/sagemaker-tensorflow-container
- serving: https://github.com/aws/sagemaker-tensorflow-serving-container
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,7 @@ You can also bring in external dependencies to help with your data
processing. There are 2 ways to do this:

1. If you included ``requirements.txt`` in your ``source_dir`` or in
your dependencies, the container installs the Python dependencies at runtime using ``pip install -r``:
your dependencies, the container installs the Python dependencies at runtime using ``pip install -r``:

.. code::

Expand Down