Skip to content

Commit 8fd3854

Browse files
jesterhazyjesterhazy
authored
add hyperparameter tuning examples (#270)
1 parent 2412ef5 commit 8fd3854

21 files changed

+3713
-0
lines changed

‎hyperparameter_tuning/analyze_results/HPO_Analyze_TuningJob_Results.ipynb

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Analyze Results of a Hyperparameter Tuning job\n",
8+
"\n",
9+
"Once you have completed a tuning job, (or even while the job is still running) you can use this notebook to analyze the results to understand how each hyperparameter effects the quality of the model.\n",
10+
"\n",
11+
"---\n",
12+
"## Set up the environment\n",
13+
"To start the analysis, you must pick the name of the hyperparameter tuning job."
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"metadata": {},
20+
"outputs": [],
21+
"source": [
22+
"import boto3\n",
23+
"import sagemaker\n",
24+
"import os\n",
25+
"\n",
26+
"region = boto3.Session().region_name\n",
27+
"sage_client = boto3.Session().client('sagemaker')\n",
28+
"\n",
29+
"tuning_job_name = 'YOUR-HYPERPARAMETER-TUNING-JOB-NAME'"
30+
]
31+
},
32+
{
33+
"cell_type": "markdown",
34+
"metadata": {},
35+
"source": [
36+
"## Track hyperparameter tuning job progress\n",
37+
"After you launch a tuning job, you can see its progress by calling describe_tuning_job API. The output from describe-tuning-job is a JSON object that contains information about the current state of the tuning job. You can call list_training_jobs_for_tuning_job to see a detailed list of the training jobs that the tuning job launched."
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": null,
43+
"metadata": {
44+
"scrolled": true
45+
},
46+
"outputs": [],
47+
"source": [
48+
"# run this cell to check current status of hyperparameter tuning job\n",
49+
"tuning_job_result = sage_client.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)\n",
50+
"\n",
51+
"status = tuning_job_result['HyperParameterTuningJobStatus']\n",
52+
"if status != 'Completed':\n",
53+
" print('Reminder: the tuning job has not been completed.')\n",
54+
" \n",
55+
"job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']\n",
56+
"print(\"%d training jobs have completed\" % job_count)\n",
57+
" \n",
58+
"is_minimize = (tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['Type'] != 'Maximize')\n",
59+
"objective_name = tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['MetricName']"
60+
]
61+
},
62+
{
63+
"cell_type": "code",
64+
"execution_count": null,
65+
"metadata": {},
66+
"outputs": [],
67+
"source": [
68+
"from pprint import pprint\n",
69+
"print(\"Best model found so far:\")\n",
70+
"pprint(tuning_job_result['BestTrainingJob'])"
71+
]
72+
},
73+
{
74+
"cell_type": "markdown",
75+
"metadata": {},
76+
"source": [
77+
"## Fetch all results as DataFrame\n",
78+
"We can list hyperparameters and objective metrics of all training jobs and pick up the training job with the best objective metric."
79+
]
80+
},
81+
{
82+
"cell_type": "code",
83+
"execution_count": null,
84+
"metadata": {
85+
"scrolled": false
86+
},
87+
"outputs": [],
88+
"source": [
89+
"import pandas as pd\n",
90+
"\n",
91+
"tuner = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
92+
"\n",
93+
"full_df = tuner.dataframe()\n",
94+
"\n",
95+
"if len(full_df) > 0:\n",
96+
" df = full_df[full_df['FinalObjectiveValue'] > -float('inf')]\n",
97+
" if len(df) > 0:\n",
98+
" df = df.sort_values('FinalObjectiveValue', ascending=is_minimize)\n",
99+
" print(\"Number of training jobs with valid objective: %d\" % len(df))\n",
100+
" print({\"lowest\":min(df['FinalObjectiveValue']),\"highest\": max(df['FinalObjectiveValue'])})\n",
101+
" pd.set_option('display.max_colwidth', -1) # Don't truncate TrainingJobName \n",
102+
" else:\n",
103+
" print(\"No training jobs have reported valid results yet.\")\n",
104+
" \n",
105+
"df"
106+
]
107+
},
108+
{
109+
"cell_type": "markdown",
110+
"metadata": {},
111+
"source": [
112+
"## See TuningJob results vs time\n",
113+
"Next we will show how the objective metric changes over time, as the tuning job progresses. For Bayesian strategy, you should expect to see a general trend towards better results, but this progress will not be steady as the algorithm needs to balance _exploration_ of new areas of parameter space against _exploitation_ of known good areas. This can give you a sense of whether or not the number of training jobs is sufficient for the complexity of your search space."
114+
]
115+
},
116+
{
117+
"cell_type": "code",
118+
"execution_count": null,
119+
"metadata": {
120+
"scrolled": false
121+
},
122+
"outputs": [],
123+
"source": [
124+
"import bokeh\n",
125+
"import bokeh.io\n",
126+
"bokeh.io.output_notebook()\n",
127+
"from bokeh.plotting import figure, show\n",
128+
"from bokeh.models import HoverTool\n",
129+
"\n",
130+
"class HoverHelper():\n",
131+
"\n",
132+
" def __init__(self, tuning_analytics):\n",
133+
" self.tuner = tuning_analytics\n",
134+
"\n",
135+
" def hovertool(self):\n",
136+
" tooltips = [\n",
137+
" (\"FinalObjectiveValue\", \"@FinalObjectiveValue\"),\n",
138+
" (\"TrainingJobName\", \"@TrainingJobName\"),\n",
139+
" ]\n",
140+
" for k in self.tuner.tuning_ranges.keys():\n",
141+
" tooltips.append( (k, \"@{%s}\" % k) )\n",
142+
"\n",
143+
" ht = HoverTool(tooltips=tooltips)\n",
144+
" return ht\n",
145+
"\n",
146+
" def tools(self, standard_tools='pan,crosshair,wheel_zoom,zoom_in,zoom_out,undo,reset'):\n",
147+
" return [self.hovertool(), standard_tools]\n",
148+
"\n",
149+
"hover = HoverHelper(tuner)\n",
150+
"\n",
151+
"p = figure(plot_width=900, plot_height=400, tools=hover.tools(), x_axis_type='datetime')\n",
152+
"p.circle(source=df, x='TrainingStartTime', y='FinalObjectiveValue')\n",
153+
"show(p)"
154+
]
155+
},
156+
{
157+
"cell_type": "markdown",
158+
"metadata": {},
159+
"source": [
160+
"## Analyze the correlation between objective metric and individual hyperparameters \n",
161+
"Now you have finished a tuning job, you may want to know the correlation between your objective metric and individual hyperparameters you've selected to tune. Having that insight will help you decide whether it makes sense to adjust search ranges for certain hyperparameters and start another tuning job. For exmaple, if you see a positive trend between objective metric and a numerical hyperparameter, you probably want to set a higher tuning range for that hyperparameter in your next tuning job.\n",
162+
"\n",
163+
"The following cell draws a graph for each hyperparameter to show its correlation with your objective metric."
164+
]
165+
},
166+
{
167+
"cell_type": "code",
168+
"execution_count": null,
169+
"metadata": {
170+
"scrolled": false
171+
},
172+
"outputs": [],
173+
"source": [
174+
"ranges = tuner.tuning_ranges\n",
175+
"figures = []\n",
176+
"for hp_name, hp_range in ranges.items():\n",
177+
" categorical_args = {}\n",
178+
" if hp_range.get('Values'):\n",
179+
" # This is marked as categorical. Check if all options are actually numbers.\n",
180+
" def is_num(x):\n",
181+
" try:\n",
182+
" float(x)\n",
183+
" return 1\n",
184+
" except:\n",
185+
" return 0 \n",
186+
" vals = hp_range['Values']\n",
187+
" if sum([is_num(x) for x in vals]) == len(vals):\n",
188+
" # Bokeh has issues plotting a \"categorical\" range that's actually numeric, so plot as numeric\n",
189+
" print(\"Hyperparameter %s is tuned as categorical, but all values are numeric\" % hp_name)\n",
190+
" else:\n",
191+
" # Set up extra options for plotting categoricals. A bit tricky when they're actually numbers.\n",
192+
" categorical_args['x_range'] = vals\n",
193+
"\n",
194+
" # Now plot it\n",
195+
" p = figure(plot_width=500, plot_height=500, \n",
196+
" title=\"Objective vs %s\" % hp_name,\n",
197+
" tools=hover.tools(),\n",
198+
" x_axis_label=hp_name, y_axis_label=objective_name,\n",
199+
" **categorical_args)\n",
200+
" p.circle(source=df, x=hp_name, y='FinalObjectiveValue')\n",
201+
" figures.append(p)\n",
202+
"show(bokeh.layouts.Column(*figures))"
203+
]
204+
}
205+
],
206+
"metadata": {
207+
"kernelspec": {
208+
"display_name": "conda_python3",
209+
"language": "python",
210+
"name": "conda_python3"
211+
},
212+
"language_info": {
213+
"codemirror_mode": {
214+
"name": "ipython",
215+
"version": 3
216+
},
217+
"file_extension": ".py",
218+
"mimetype": "text/x-python",
219+
"name": "python",
220+
"nbconvert_exporter": "python",
221+
"pygments_lexer": "ipython3",
222+
"version": "3.6.4"
223+
},
224+
"notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
225+
},
226+
"nbformat": 4,
227+
"nbformat_minor": 2
228+
}

‎hyperparameter_tuning/keras_bring_your_own/Dockerfile

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
ARG VERSION
2+
FROM tensorflow/tensorflow:$VERSION
3+
WORKDIR /
4+
5+
RUN pip install keras h5py
6+
7+
COPY /trainer /trainer
8+
9+
ENTRYPOINT ["python", "-m", "trainer.start"]

0 commit comments

Comments
 (0)