Skip to content

Add airflow tuning config export API #486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 15, 2018
Merged

Conversation

yangaws
Copy link
Contributor

@yangaws yangaws commented Nov 15, 2018

Issue #, if available:

Description of changes:

  1. Improve airflow_name_from_base, add limits for length and auto trim naming to fit length.
  2. Fix missing hyperparameter mini_batch_size from training config.
  3. Add tuning_config API to export tuning config from a tuner.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have updated the changelog with a description of my changes (if appropriate)
  • I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@codecov-io
Copy link

codecov-io commented Nov 15, 2018

Codecov Report

Merging #486 into master will increase coverage by 0.02%.
The diff coverage is 97.5%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #486      +/-   ##
==========================================
+ Coverage   93.82%   93.84%   +0.02%     
==========================================
  Files          58       58              
  Lines        4323     4355      +32     
==========================================
+ Hits         4056     4087      +31     
- Misses        267      268       +1
Impacted Files Coverage Δ
src/sagemaker/utils.py 92.62% <100%> (+0.38%) ⬆️
src/sagemaker/workflow/airflow.py 90.24% <96.87%> (+2.74%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11d3fcf...e2bf31c. Read the comment docs.

@@ -16,7 +16,7 @@
import pytest
import mock

from sagemaker import estimator, tensorflow
from sagemaker import estimator, tensorflow, mxnet, tuner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alphabetize the imported modules

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

mxnet_estimator = mxnet.MXNet(
entry_point="{{ entry_point }}",
source_dir="{{ source_dir }}",
py_version='py2',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's just a unit test, but it does seem a bit strange to me we're still adding new code that "uses" Python 2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

"""Export Airflow training config from an estimator

Args:
estimator (sagemaker.estimator.EstimatroBase):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Estimatro/Estimator

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

Amazon algorithm. For other estimators, batch size should be specified in the estimator.

Returns:
A dict of training config that can be directly used by SageMakerTrainingOperator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change this to "dict: training config..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@yangaws yangaws merged commit 501eced into aws:master Nov 15, 2018
@yangaws yangaws deleted the airflow_model branch November 15, 2018 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants