Update to latest upstream (#6)

* Add data_type to hyperparameters (aws#54)

When we describe a training job the data type of the hyper parameters is
lost because we use a dict[str, str]. This adds a new field to
Hyperparameter so that we can convert the datatypes at runtime.

instead of validating with isinstance(), we cast the hp value to the type it
is meant to be. This enforces a "strongly typed" value. When we
deserialize from the API string responses it becomes easier to deal with
too.

* Add wrapper for LDA. (aws#56)

Update CHANGELOG and bump the version number.

* Add support for async fit() (aws#59)

when calling fit(wait=False) it will return immediately. The training
job will carry on even if the process exits. by using attach() the
estimator can be retrieved by providing the training job name.

_prepare_init_params_from_job_description() is now a classmethod instead
of being a static method. Each class is responsible to implement their
specific logic to convert a training job description into arguments that
can be passed to its own __init__()

* Fix Estimator role expansion (aws#68)

Instead of manually constructing the role ARN, use the IAM boto client
to do it. This properly expands service-roles and regular roles.

* Add FM and LDA to the documentation. (aws#66)

* Fix description of an argument of sagemaker.session.train (aws#69)

* Fix description of an argument of sagemaker.session.train

'input_config' should be an array which has channel objects.

* Add a link to the botocore docs

* Use 'list' instead of 'array' in the description

* Add ntm algorithm with doc, unit tests, integ tests (aws#73)

* JSON serializer: predictor.predict accepts dictionaries (aws#62)

Add support for serializing python dictionaries to json
Add prediction with dictionary in tf iris integ test

* Fixing timeouts for PCA async integration test. (aws#78)

Execute tf_cifar test without logs to eliminate delay to detect that job has finished.

* Fixes in LinearLearner and unit tests addition. (aws#77)

* Print out billable seconds after training completes (aws#30)

* Added: print out billable seconds after training completes

* Fixed: test_session.py to pass unit tests

* Fixed: removed offending tzlocal()

* Use sagemaker_timestamp when creating endpoint names in integration tests. (aws#81)

* Support TensorFlow-1.5.0 and MXNet-1.0.0  (aws#82)

* Update .gitignore to ignore pytest_cache.

* Support TensorFlow-1.5.0 and MXNet-1.0.0

* Update and refactor tests. Add tests for fw_utils.

* Fix typo.

* Update changelog for 1.1.0 (aws#85)

Author: jalabort

.gitignore

@@ -21,3 +21,4 @@ doc/_build
 **/.DS_Store
 venv/
 *~
+.pytest_cache/

CHANGELOG.rst

@@ -2,11 +2,41 @@
 CHANGELOG
 =========
 
+1.1.0
+=====
+
+* feature: Estimators: add support for TensorFlow-1.5.0
+* feature: Estimators: add support for MXNet-1.0.0
+* feature: Tests: use ``sagemaker_timestamp`` when creating endpoint names in integration tests
+* feature: Session: print out billable seconds after training completes
+* bug-fix: Estimators: fix LinearLearner and add unit tests
+* bug-fix: Tests: fix timeouts for PCA async integration test
+* feature: Predictors: allow ``predictor.predict()`` in the JSON serializer to accept dictionaries
+
+1.0.4
+=====
+
+* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
+* feature: Documentation: fix description of an argument of sagemaker.session.train
+* feature: Documentation: add FM and LDA to the documentation
+* feature: Estimators: add support for async fit
+* bug-fix: Estimators: fix estimator role expansion
+
+1.0.3
+=====
+
+* feature: Estimators: add support for Amazon LDA algorithm
+* feature: Hyperparameters: add data_type to hyperparameters
+* feature: Documentation: update TensorFlow examples following API change
+* feature: Session: support multi-part uploads
+* feature: add new SageMaker CLI
+
+
 1.0.2
 =====
 
 * feature: Estimators: add support for Amazon FactorizationMachines algorithm
-* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
+* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
 * feature: Tests: add training failure tests for TF and MXNet
 * feature: Documentation: show how to make predictions against existing endpoint
 * feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
@@ -27,4 +57,3 @@ CHANGELOG
 =====
 
 * Initial commit
-

README.rst

@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install
 
     git clone https://github.com/aws/sagemaker-python-sdk.git
     python setup.py sdist
-    pip install dist/sagemaker-1.0.0.tar.gz
+    pip install dist/sagemaker-1.1.0.tar.gz
 
 Supported Python versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -97,7 +97,7 @@ SageMaker Python SDK provides several high-level abstractions for working with A
 - **Estimators**: Encapsulate training on SageMaker. Can be ``fit()`` to run training, then the resulting model ``deploy()`` ed to a SageMaker Endpoint.
 - **Models**: Encapsulate built ML models. Can be ``deploy()`` ed to a SageMaker Endpoint.
 - **Predictors**: Provide real-time inference and transformation using Python data-types against a SageMaker Endpoint.
-- **Session**: Provides a collection of convience methods for working with SageMaker resources.
+- **Session**: Provides a collection of convenience methods for working with SageMaker resources.
 
 Estimator and Model implementations for MXNet, TensorFlow, and Amazon ML algorithms are included. There's also an Estimator that runs SageMaker compatible custom Docker containers, allowing you to run your own ML algorithms via SageMaker Python SDK.
 
@@ -114,6 +114,8 @@ MXNet SageMaker Estimators
 
 With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.
 
+Supported versions of MXNet: ``1.0.0``, ``0.12.1``.
+
 Training with MXNet
 ~~~~~~~~~~~~~~~~~~~
 
@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
 Using MXNet and numpy
 ^^^^^^^^^^^^^^^^^^^^^
 
-You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
+You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
 
 Running an MXNet training script in SageMaker
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script
 
 SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.
 
-Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
+Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
 
-Each Docker container has the following dependencies installed:
+The Docker images have the following dependencies installed:
 
--  Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
-   the MXNet constructor.
--  MXNet 0.12, built for either GPU or CPU, depending on the instance
-   type for training or deploying.
--  CUDA 9.0
--  numpy 1.12
++-------------------------+--------------+-------------+
+| Dependencies            | MXNet 0.12.1 | MXNet 1.0.0 |
++-------------------------+--------------+-------------+
+| Python                  |   2.7 or 3.5 |   2.7 or 3.5|
++-------------------------+--------------+-------------+
+| CUDA                    |          9.0 |         9.0 |
++-------------------------+--------------+-------------+
+| numpy                   |       1.13.3 |      1.13.3 |
++-------------------------+--------------+-------------+
 
 The Docker images extend Ubuntu 16.04.
 
+You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.
+
 TensorFlow SageMaker Estimators
 -------------------------------
 
 TensorFlow SageMaker Estimators allow you to run your own TensorFlow
 training algorithms on SageMaker Learner, and to host your own TensorFlow
 models on SageMaker Hosting.
 
+Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.
+
 Training with TensorFlow
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
-version is **1.4.0**. This training script **must contain** the following functions:
+versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
 
 - ``model_fn``: defines the model that will be trained.
 - ``train_input_fn``: preprocess and load training data.
@@ -1150,6 +1159,7 @@ Optional arguments
 
 -  ``wait (bool)``: Defaults to True, whether to block and wait for the
    training script to complete before returning.
+   If set to False, it will return immediately, and can later be attached to.
 -  ``logs (bool)``: Defaults to True, whether to show logs produced by training
    job in the Python session. Only meaningful when wait is True.
 - ``run_tensorboard_locally (bool)``: Defaults to False. Executes TensorBoard in a different
@@ -1178,9 +1188,25 @@ the ``TensorFlow`` estimator parameter ``training_steps`` is finished or when th
 job execution time reaches the ``TensorFlow`` estimator parameter ``train_max_run``.
 
 When the training job finishes, a `TensorFlow serving <https://www.tensorflow.org/serving/serving_basic>`_
-with the result of the training is generated and saved to the S3 location define by
+with the result of the training is generated and saved to the S3 location defined by
 the ``TensorFlow`` estimator parameter ``output_path``.
 
+
+If the ``wait=False`` flag is passed to ``fit``, then it will return immediately. The training job will continue running
+asynchronously. At a later time, a Tensorflow Estimator can be obtained by attaching to the existing training job. If
+the training job is not finished it will start showing the standard output of training and wait until it completes.
+After attaching, the estimator can be deployed as usual.
+
+.. code:: python
+
+    tf_estimator.fit(your_input_data, wait=False)
+    training_job_name = tf_estimator.latest_training_job.name
+
+    # after some time, or in a separate python notebook, we can attach to it again.
+
+    tf_estimator = TensorFlow.attach(training_job_name=training_job_name)
+
+
 The evaluation process
 """"""""""""""""""""""
 
@@ -1244,6 +1270,8 @@ You can access TensorBoard locally at http://localhost:6006 or using your SakeMa
 `https*workspace_base_url*proxy/6006/ <proxy/6006/>`_ (TensorBoard will not work if you forget to put the slash,
 '/', in end of the url). If TensorBoard started on a different port, adjust these URLs to match.
 
+Note that TensorBoard is not supported when passing wait=False to ``fit``.
+
 
 Deploying TensorFlow Serving models
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1400,39 +1428,61 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
 SageMaker TensorFlow Docker containers
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
-- awscli 1.12.1
-- boto3 1.4.7
-- botocore 1.5.92
-- futures 2.2.0
-- gevent 1.2.2
-- grpcio 1.7.0
-- numpy 1.13.3
-- pandas 0.21.0
-- protobuf 3.4.0
-- requests 2.14.2
-- scikit-learn 0.19.1
-- scipy 1.0.0
-- six 1.10.0
-- sklearn 0.0
-- tensorflow 1.4.0
-- tensorflow-serving-api 1.4.0
-- tensorflow-tensorboard 0.4.0rc2
+The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:
+
++------------------------+------------------+------------------+
+| Dependencies           | tensorflow 1.4.1 | tensorflow 1.5.0 |
++------------------------+------------------+------------------+
+| awscli                 |           1.12.1 |          1.14.35 |
++------------------------+------------------+------------------+
+| boto3                  |            1.4.7 |           1.5.22 |
++------------------------+------------------+------------------+
+| botocore               |           1.5.92 |           1.8.36 |
++------------------------+------------------+------------------+
+| futures                |            2.2.0 |            2.2.0 |
++------------------------+------------------+------------------+
+| gevent                 |            1.2.2 |            1.2.2 |
++------------------------+------------------+------------------+
+| grpcio                 |            1.7.0 |            1.9.0 |
++------------------------+------------------+------------------+
+| numpy                  |           1.13.3 |           1.14.0 |
++------------------------+------------------+------------------+
+| pandas                 |           0.21.0 |           0.22.0 |
++------------------------+------------------+------------------+
+| protobuf               |            3.4.0 |            3.5.1 |
++------------------------+------------------+------------------+
+| requests               |           2.14.2 |           2.18.4 |
++------------------------+------------------+------------------+
+| scikit-learn           |           0.19.1 |           0.19.1 |
++------------------------+------------------+------------------+
+| scipy                  |            1.0.0 |            1.0.0 |
++------------------------+------------------+------------------+
+| six                    |           1.10.0 |           1.10.0 |
++------------------------+------------------+------------------+
+| sklearn                |              0.0 |              0.0 |
++------------------------+------------------+------------------+
+| tensorflow             |            1.4.1 |            1.5.0 |
++------------------------+------------------+------------------+
+| tensorflow-serving-api |            1.4.0 |            1.5.0 |
++------------------------+------------------+------------------+
+| tensorflow-tensorboard |            0.4.0 |            1.5.1 |
++------------------------+------------------+------------------+
 
 The Docker images extend Ubuntu 16.04.
 
+You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.
 
 AWS SageMaker Estimators
 ------------------------
 Amazon SageMaker provides several built-in machine learning algorithms that you can use for a variety of problem types.
 
 The full list of algorithms is available on the AWS website: https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html
 
-SageMaker Python SDK includes Estimator wrappers for the AWS K-means, Principal Components Analysis, and Linear Learner algorithms.
+SageMaker Python SDK includes Estimator wrappers for the AWS K-means, Principal Components Analysis(PCA), Linear Learner, Factorization Machines, Latent Dirichlet Allocation(LDA) and Neural Topic Model(NTM) algorithms.
 
 Definition and usage
 ~~~~~~~~~~~~~~~~~~~~
-Estimators that wrap Amazon's built-in algorithms define algorithm's hyperparameters with defaults. When a default is not possible you need to provide the value during construction:
+Estimators that wrap Amazon's built-in algorithms define algorithm's hyperparameters with defaults. When a default is not possible you need to provide the value during construction, e.g.:
 
 - ``KMeans`` Estimator requires parameter ``k`` to define number of clusters
 - ``PCA`` Estimator requires parameter ``num_components`` to define number of principal components

doc/conf.py

@@ -18,7 +18,7 @@ def __getattr__(cls, name):
                 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
 sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
 
-version = '1.0'
+version = '1.1.0'
 project = u'sagemaker'
 
 # Add any Sphinx extension module names here, as strings. They can be extensions

doc/factorization_machines.rst

@@ -0,0 +1,22 @@
+FactorizationMachines
+-------------------------
+
+The Amazon SageMaker Factorization Machines algorithm.
+
+.. autoclass:: sagemaker.FactorizationMachines
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:
+    :exclude-members: image, num_factors, predictor_type, epochs, clip_gradient, mini_batch_size, feature_dim, eps, rescale_grad, bias_lr, linear_lr, factors_lr, bias_wd, linear_wd, factors_wd, bias_init_method, bias_init_scale, bias_init_sigma, bias_init_value, linear_init_method, linear_init_scale, linear_init_sigma, linear_init_value, factors_init_method, factors_init_scale, factors_init_sigma, factors_init_value
+
+
+.. autoclass:: sagemaker.FactorizationMachinesModel
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.FactorizationMachinesPredictor
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/index.rst

@@ -38,11 +38,15 @@ A managed environment for TensorFlow training and hosting on Amazon SageMaker
 
 SageMaker First-Party Algorithms
 --------------------------------
-Amazon provides implementations of some common machine learning algortithms optimized for GPU archicture and massive datasets.
+Amazon provides implementations of some common machine learning algortithms optimized for GPU architecture and massive datasets.
 
 .. toctree::
     :maxdepth: 2
 
     kmeans
     pca
     linear_learner
+    sagemaker.amazon.amazon_estimator
+    factorization_machines
+    lda
+    ntm

doc/lda.rst

@@ -0,0 +1,22 @@
+LDA
+--------------------
+
+The Amazon SageMaker LDA algorithm.
+
+.. autoclass:: sagemaker.LDA
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:
+    :exclude-members: image, num_topics, alpha0, max_restarts, max_iterations, mini_batch_size, feature_dim, tol
+
+
+.. autoclass:: sagemaker.LDAModel
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.LDAPredictor
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/ntm.rst

@@ -0,0 +1,23 @@
+NTM
+--------------------
+
+The Amazon SageMaker NTM algorithm.
+
+.. autoclass:: sagemaker.NTM
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:
+    :exclude-members: image, num_topics, encoder_layers, epochs, encoder_layers_activation, optimizer, tolerance,
+                 num_patience_epochs, batch_norm, rescale_gradient, clip_gradient, weight_decay, learning_rate
+
+
+.. autoclass:: sagemaker.NTMModel
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.NTMPredictor
+    :members:
+    :undoc-members:
+    :show-inheritance:

setup.py

@@ -11,7 +11,7 @@ def read(fname):
 
 
 setup(name="sagemaker",
-      version="1.0.2",
+      version="1.1.0",
       description="Open source library for training and deploying models on Amazon SageMaker.",
       packages=find_packages('src'),
       package_dir={'': 'src'},

src/sagemaker/__init__.py

@@ -15,9 +15,11 @@
 from sagemaker import estimator
 from sagemaker.amazon.kmeans import KMeans, KMeansModel, KMeansPredictor
 from sagemaker.amazon.pca import PCA, PCAModel, PCAPredictor
+from sagemaker.amazon.lda import LDA, LDAModel, LDAPredictor
 from sagemaker.amazon.linear_learner import LinearLearner, LinearLearnerModel, LinearLearnerPredictor
 from sagemaker.amazon.factorization_machines import FactorizationMachines, FactorizationMachinesModel
 from sagemaker.amazon.factorization_machines import FactorizationMachinesPredictor
+from sagemaker.amazon.ntm import NTM, NTMModel, NTMPredictor
 
 from sagemaker.model import Model
 from sagemaker.predictor import RealTimePredictor
@@ -30,6 +32,7 @@
 
 __all__ = [estimator, KMeans, KMeansModel, KMeansPredictor, PCA, PCAModel, PCAPredictor, LinearLearner,
            LinearLearnerModel, LinearLearnerPredictor,
+           LDA, LDAModel, LDAPredictor,
            FactorizationMachines, FactorizationMachinesModel, FactorizationMachinesPredictor,
-           Model, RealTimePredictor, Session,
+           Model, NTM, NTMModel, NTMPredictor, RealTimePredictor, Session,
            container_def, s3_input, production_variant, get_execution_role]