Skip to content

Commit 041af1b

Browse files
committed
Merge branch 'master' of https://github.com/aws/sagemaker-python-sdk into refactor-pytorch
2 parents b88babd + cfaa2c6 commit 041af1b

32 files changed

+641
-104
lines changed

.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request._
88

99
- [ ] I have read the [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md) doc
10-
- [ ] I used the commit message format described in [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md#commit-message-guidlines)
10+
- [ ] I used the commit message format described in [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md#committing-your-change)
1111
- [ ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
1212
- [ ] I have updated any necessary [documentation](https://github.com/aws/sagemaker-python-sdk/blob/master/README.rst) (if appropriate)
1313

CHANGELOG.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,50 @@
11
# Changelog
22

3+
## v1.36.3 (2019-08-13)
4+
5+
### Bug fixes and other changes
6+
7+
* fix integration test failures masked by timeout bug
8+
* prevent multiple values error in sklearn.transformer()
9+
* model.transformer() passes tags to create_model()
10+
11+
## v1.36.2 (2019-08-12)
12+
13+
### Bug fixes and other changes
14+
15+
* rework CONTRIBUTING.md to include a development workflow
16+
17+
## v1.36.1 (2019-08-08)
18+
19+
### Bug fixes and other changes
20+
21+
* prevent integration test's timeout functions from hiding failures
22+
23+
### Documentation changes
24+
25+
* correct typo in using_sklearn.rst
26+
27+
## v1.36.0 (2019-08-07)
28+
29+
### Features
30+
31+
* support for TensorFlow 1.14
32+
33+
### Bug fixes and other changes
34+
35+
* ignore FI18 flake8 rule
36+
* allow Airflow enabled estimators to use absolute path entry_point
37+
38+
## v1.35.1 (2019-08-01)
39+
40+
### Bug fixes and other changes
41+
42+
* update sklearn document to include 3p dependency installation
43+
44+
### Documentation changes
45+
46+
* refactor and edit using_mxnet topic
47+
348
## v1.35.0 (2019-07-31)
449

550
### Features

CONTRIBUTING.md

Lines changed: 105 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -7,68 +7,108 @@ Please read through this document before submitting any issues or pull requests
77
information to effectively respond to your bug report or contribution.
88

99

10+
## Table of Contents
11+
12+
* [Table of Contents](#table-of-contents)
13+
* [Reporting Bugs/Feature Requests](#reporting-bugsfeature-requests)
14+
* [Contributing via Pull Requests (PRs)](#contributing-via-pull-requests-prs)
15+
* [Setting up Your Development Environment *[Optional, but Recommended]*](#setting-up-your-development-environment-optional-but-recommended)
16+
* [Pulling Down the Code](#pulling-down-the-code)
17+
* [Running the Unit Tests](#running-the-unit-tests)
18+
* [Running the Integration Tests](#running-the-integration-tests)
19+
* [Making and Testing Your Change](#making-and-testing-your-change)
20+
* [Committing Your Change](#committing-your-change)
21+
* [Sending a Pull Request](#sending-a-pull-request)
22+
* [Finding Contributions to Work On](#finding-contributions-to-work-on)
23+
* [Code of Conduct](#code-of-conduct)
24+
* [Security Issue Notifications](#security-issue-notifications)
25+
* [Licensing](#licensing)
26+
1027
## Reporting Bugs/Feature Requests
1128

1229
We welcome you to use the GitHub issue tracker to report bugs or suggest features.
1330

14-
When filing an issue, please check [existing open](https://github.com/aws/sagemaker-python-sdk/issues), or [recently closed](https://github.com/aws/sagemaker-python-sdk/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
31+
When filing an issue, please check [existing open](https://github.com/aws/sagemaker-python-sdk/issues) and [recently closed](https://github.com/aws/sagemaker-python-sdk/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20) issues to make sure somebody else hasn't already
1532
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
1633

17-
* A reproducible test case or series of steps
18-
* The version of our code being used
19-
* Any modifications you've made relevant to the bug
20-
* A description of your environment or deployment
21-
22-
## Setting up your development environment [optional, but recommended]
23-
24-
* Set up the Cloud9 environment:
25-
* Instance type: You'll need at least 4 GB of RAM to avoid running into memory issues. We recommend at least a t3.medium to run the unit tests. Larger hosts will reduce the chance of encountering resource limits.
26-
* Follow the instructions at [Creating a Cloud9 EC2 Environment](https://docs.aws.amazon.com/cloud9/latest/user-guide/create-environment.html#create-environment-main) to set up a Cloud9 EC2 environment
27-
* Expand the storage of the EC2 instance from 10GB to 20GB
28-
* Because you'll need a minimum of 11GB of disk storage on the EC2 instance to run the repository's unit tests, you'll need to expand your EC2 volume size. We recommend at least 20GB. A larger volume will reduce the chance of encountering resource limits.
29-
* Follow the instructions at [Modifying an EBS Volume Using Elastic Volumes (Console)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html#modify-ebs-volume) to increase the EBS volume size associated with the newly created EC2 instance.
30-
* Wait 5-10min for the new EBS volume increase to take effect.
31-
* Allow EC2 to claim the additional space by stopping and then starting your EC2 host.
32-
* Create a fork of this repository on GitHub. You should end up with a fork at `https://github.com/<username>/sagemaker-python-sdk`
33-
* Follow the instructions at [Fork a repo](https://help.github.com/en/articles/fork-a-repo) to fork a GitHub repository.
34-
* In the Cloud9 UI, pull down this repository by clicking on "Clone from Github" or running the following command in the Cloud9 terminal: `git clone https://github.com/<username>/sagemaker-python-sdk` where <username> is your github username.
35-
* Install tox using `pip install tox`
36-
* Install coverage using `pip install .[test]`
37-
* cd into the sagemaker-python-sdk folder: `cd sagemaker-python-sdk` or `cd /environment/sagemaker-python-sdk`
38-
* Run the following tox command and verify that all unit tests pass: `tox tests/unit`
39-
40-
## Contributing via Pull Requests
41-
Contributions via pull requests are much appreciated.
42-
43-
You can use the following commands to setup your developing and testing environment after you fork sagemaker-python-sdk repository:
44-
45-
```bash
46-
git clone [email protected]:<your-github-username>/sagemaker-python-sdk.git
47-
cd sagemaker-python-sdk
48-
pip install -U .
49-
pip install -U .[test]
50-
```
34+
* A reproducible test case or series of steps.
35+
* The version of our code being used.
36+
* Any modifications you've made relevant to the bug.
37+
* A description of your environment or deployment.
38+
39+
40+
## Contributing via Pull Requests (PRs)
41+
42+
Contributions via pull requests are much appreciated.
5143

5244
Before sending us a pull request, please ensure that:
5345

54-
1. You are working against the latest source on the *master* branch.
55-
2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
56-
3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
46+
* You are working against the latest source on the *master* branch.
47+
* You check the existing open and recently merged pull requests to make sure someone else hasn't already addressed the problem.
48+
* You open an issue to discuss any significant work - we would hate for your time to be wasted.
49+
50+
51+
### Setting up Your Development Environment *[Optional, but Recommended]*
52+
53+
1. Set up the Cloud9 environment:
54+
1. Instance type: You'll need at least 4 GB of RAM to avoid running into memory issues. We recommend at least a t3.medium to run the unit tests. A larger host will reduce the chance of encountering resource limits.
55+
1. Follow the instructions at [Creating a Cloud9 EC2 Environment](https://docs.aws.amazon.com/cloud9/latest/user-guide/create-environment.html#create-environment-main) to set up a Cloud9 EC2 environment.
56+
1. Expand the storage of the EC2 instance from 10GB to 20GB:
57+
1. Because you'll need a minimum of 11GB of disk storage on the EC2 instance to run the repository's unit tests, you'll need to expand your EC2 volume size. We recommend at least 20GB. A larger volume will reduce the chance of encountering resource limits.
58+
1. Follow the instructions at [Modifying an EBS Volume Using Elastic Volumes (Console)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html#modify-ebs-volume) to increase the EBS volume size associated with the newly created EC2 instance.
59+
1. Wait 5-10min for the new EBS volume increase to finalize.
60+
1. Allow EC2 to claim the additional space by stopping and then starting your EC2 host.
61+
62+
63+
### Pulling Down the Code
64+
65+
1. If you do not already have one, create a GitHub account by following the prompts at [Join Github](https://github.com/join).
66+
1. Create a fork of this repository on GitHub. You should end up with a fork at `https://github.com/<username>/sagemaker-python-sdk`.
67+
1. Follow the instructions at [Fork a Repo](https://help.github.com/en/articles/fork-a-repo) to fork a GitHub repository.
68+
1. Clone your fork of the repository: `git clone https://github.com/<username>/sagemaker-python-sdk` where `<username>` is your github username.
69+
70+
71+
### Running the Unit Tests
72+
73+
1. Install tox using `pip install tox`
74+
1. Install coverage using `pip install .[test]`
75+
1. cd into the sagemaker-python-sdk folder: `cd sagemaker-python-sdk` or `cd /environment/sagemaker-python-sdk`
76+
1. Run the following tox command and verify that all code checks and unit tests pass: `tox tests/unit`
77+
78+
You can also run a single test with the following command: `tox -e py36 -- -s -vv <path_to_file><file_name>::<test_function_name>`
79+
* Note that the coverage test will fail if you only run a single test, so make sure to surround the command with `export IGNORE_COVERAGE=-` and `unset IGNORE_COVERAGE`
80+
* Example: `export IGNORE_COVERAGE=- ; tox -e py36 -- -s -vv tests/unit/test_estimator.py::test_sagemaker_model_s3_uri_invalid ; unset IGNORE_COVERAGE`
5781

58-
To send us a pull request, please:
5982

60-
1. Fork the repository.
61-
2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
62-
3. Include unit tests when you contribute new features or make bug fixes, as they help to a) prove that your code works correctly, and b) guard against future breaking changes to lower the maintenance cost.
63-
4. Ensure local tests pass.
64-
5. Use commit messages (and PR titles) that follow [these guidelines](#commit-message-guidelines).
65-
6. Send us a pull request, answering any default questions in the pull request interface.
66-
7. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
83+
### Running the Integration Tests
6784

68-
GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
69-
[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
85+
Our CI system runs integration tests (the ones in the `tests/integ` directory), in parallel, for every Pull Request.
86+
You should only worry about manually running any new integration tests that you write, or integration tests that test an area of code that you've modified.
7087

71-
### Commit message guidelines
88+
1. Follow the instructions at [Set Up the AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/polly/latest/dg/setup-aws-cli.html).
89+
1. To run a test, specify the test file and method you want to run per the following command: `tox -e py36 -- -s -vv <path_to_file><file_name>::<test_function_name>`
90+
* Note that the coverage test will fail if you only run a single test, so make sure to surround the command with `export IGNORE_COVERAGE=-` and `unset IGNORE_COVERAGE`
91+
* Example: `export IGNORE_COVERAGE=- ; tox -e py36 -- -s -vv tests/integ/test_tf_script_mode.py::test_mnist ; unset IGNORE_COVERAGE`
92+
93+
If you are writing or modifying a test that creates a SageMaker job (training, tuner, or transform) or endpoint, it's important to assign a concurrency-friendly `job_name` (or `endpoint_name`), or your tests may fail randomly due to name collisions. We have a helper method `sagemaker.utils.unique_name_from_base(base, max_length)` that makes test-friendly names. You can find examples of how to use it [here](https://github.com/aws/sagemaker-python-sdk/blob/3816a5658d3737c9767e01bc8d37fc3ed5551593/tests/integ/test_tfs.py#L37) and
94+
[here](https://github.com/aws/sagemaker-python-sdk/blob/3816a5658d3737c9767e01bc8d37fc3ed5551593/tests/integ/test_tuner.py#L616), or by searching for "unique\_name\_from\_base" in our test code.
95+
96+
97+
### Making and Testing Your Change
98+
99+
1. Create a new git branch:
100+
```shell
101+
git checkout -b my-fix-branch master
102+
```
103+
1. Make your changes, **including unit tests** and, if appropriate, integration tests.
104+
1. Include unit tests when you contribute new features or make bug fixes, as they help to:
105+
1. Prove that your code works correctly.
106+
1. Guard against future breaking changes to lower the maintenance cost.
107+
1. Please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
108+
1. Run all the unit tests as per [Running the Unit Tests](#running-the-unit-tests), and verify that all checks and tests pass.
109+
110+
111+
### Committing Your Change
72112

73113
We use commit messages to update the project version number and generate changelog entries, so it's important for them to follow the right format. Valid commit messages include a prefix, separated from the rest of the message by a colon and a space. Here are a few examples:
74114
@@ -81,34 +121,43 @@ documentation: add MXNet documentation
81121
Valid prefixes are listed in the table below.
82122
83123
| Prefix | Use for... |
84-
|-----------------|------------------------------------------------------------------------------------------------|
124+
|----------------:|:-----------------------------------------------------------------------------------------------|
85125
| `breaking` | Incompatible API changes. |
86126
| `deprecation` | Deprecating an existing API or feature, or removing something that was previously deprecated. |
87127
| `feature` | Adding a new feature. |
88128
| `fix` | Bug fixes. |
89129
| `change` | Any other code change. |
90130
| `documentation` | Documentation changes. |
91131
92-
Some of the prefixes allow abbreviation -- `break`, `feat`, `depr`, and `doc` are all valid. If you omit a prefix, the commit will be treated as a `change`.
132+
Some of the prefixes allow abbreviation ; `break`, `feat`, `depr`, and `doc` are all valid. If you omit a prefix, the commit will be treated as a `change`.
93133
94134
For the rest of the message, use imperative style and keep things concise but informative. See [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/) for guidance.
95135
96-
### Integration tests
97136
98-
Our CI system runs integration tests (the ones in the `tests/integ` directory) in parallel. If you are writing or modifying a test that creates a SageMaker job (training, tuner, or transform) or an endpoint, it's important to assign a concurrency-friendly `job_name` (or `endpoint_name`), or your tests may fail randomly due to name collisions. We have a helper method `sagemaker.utils.unique_name_from_base(base, max_length)` that makes test-friendly names. You can find examples of how to use it [here](https://github.com/aws/sagemaker-python-sdk/blob/3816a5658d3737c9767e01bc8d37fc3ed5551593/tests/integ/test_tfs.py#L37) and
99-
[here](https://github.com/aws/sagemaker-python-sdk/blob/3816a5658d3737c9767e01bc8d37fc3ed5551593/tests/integ/test_tuner.py#L616), or by searching for "unique\_name\_from\_base" in our test code.
137+
### Sending a Pull Request
138+
139+
GitHub provides additional document on [Creating a Pull Request](https://help.github.com/articles/creating-a-pull-request/).
140+
141+
Please remember to:
142+
* Use commit messages (and PR titles) that follow the guidelines under [Committing Your Change](#committing-your-change).
143+
* Send us a pull request, answering any default questions in the pull request interface.
144+
* Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
145+
146+
147+
## Finding Contributions to Work On
100148
101-
## Finding contributions to work on
102149
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws/sagemaker-python-sdk/labels/help%20wanted) issues is a great place to start.
103150
104151
105152
## Code of Conduct
153+
106154
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
107155
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
108156
[email protected] with any additional questions or comments.
109157
110158
111-
## Security issue notifications
159+
## Security Issue Notifications
160+
112161
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
113162
114163

README.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -189,9 +189,9 @@ TensorFlow SageMaker Estimators
189189

190190
By using TensorFlow SageMaker Estimators, you can train and host TensorFlow models on Amazon SageMaker.
191191

192-
Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``.
192+
Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14``.
193193

194-
Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.0``
194+
Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.1``
195195

196196
We recommend that you use the latest supported version, because that's where we focus most of our development efforts.
197197

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.35.1.dev0
1+
1.36.4.dev0

doc/using_sklearn.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ Using third-party libraries
103103
When running your training script on SageMaker, it will have access to some pre-installed third-party libraries including ``scikit-learn``, ``numpy``, and ``pandas``.
104104
For more information on the runtime environment, including specific package versions, see `SageMaker Scikit-learn Docker containers <https://github.com/aws/sagemaker-scikit-learn-container>`__.
105105

106-
If there are other packages you want to use with your script, you can include a ``requirements.txte` file in the same directory as your training script to install other dependencies at runtime.
106+
If there are other packages you want to use with your script, you can include a ``requirements.txt`` file in the same directory as your training script to install other dependencies at runtime.
107107
A ``requirements.txt`` file is a text file that contains a list of items that are installed by using ``pip install``. You can also specify the version of an item to install.
108108
For information about the format of a ``requirements.txt`` file, see `Requirements Files <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`__ in the pip documentation.
109109

doc/using_tf.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@ models on SageMaker Hosting.
88

99
For general information about using the SageMaker Python SDK, see :ref:`overview:Using the SageMaker Python SDK`.
1010

11+
.. warning::
12+
The TensorFlow estimator is available only for Python 3, starting by the TensorFlow version 1.14.
13+
1114
.. warning::
1215
We have added a new format of your TensorFlow training script with TensorFlow version 1.11.
1316
This new way gives the user script more flexibility.

0 commit comments

Comments
 (0)