Merge pull request aws#62 from awslabs/scikit_bring_your_own

Scikit bring your own

Merging this in preparation for Monday's meeting.

Author: djarpin

advanced_functionality/scikit_bring_your_own/container/Dockerfile

@@ -0,0 +1,43 @@
+# Build an image that can do training and inference in SageMaker
+# This is a Python 2 image that uses the nginx, gunicorn, flask stack
+# for serving inferences in a stable way.
+
+FROM ubuntu:16.04
+
+MAINTAINER Amazon AI <[email protected]>
+
+
+RUN apt-get -y update && apt-get install -y --no-install-recommends \
+         wget \
+         python \
+         nginx \
+         ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# Here we get all python packages.
+# There's substantial overlap between scipy and numpy that we eliminate by
+# linking them together. Likewise, pip leaves the install caches populated which uses
+# a significant amount of space. These optimizations save a fair amount of space in the
+# image, which reduces start up time.
+RUN wget https://bootstrap.pypa.io/get-pip.py && python get-pip.py && \
+    pip install numpy scipy scikit-learn pandas flask gevent gunicorn && \
+        (cd /usr/local/lib/python2.7/dist-packages/scipy/.libs; rm *; ln ../../numpy/.libs/* .) && \
+        rm -rf /root/.cache
+
+# Set some environment variables. PYTHONUNBUFFERED keeps Python from buffering our standard
+# output stream, which means that logs can be delivered to the user quickly. PYTHONDONTWRITEBYTECODE
+# keeps Python from writing the .pyc files which are unnecessary in this case. We also update
+# PATH so that the train and serve programs are found when the container is invoked.
+
+ENV PYTHONUNBUFFERED=TRUE
+ENV PYTHONDONTWRITEBYTECODE=TRUE
+ENV PATH="/opt/program:${PATH}"
+
+# Make nginx log to stdout/err so that the log messages will be picked up by the
+# Docker logger
+RUN ln -s /dev/stdout /tmp/nginx.access.log && ln -s /dev/stderr /tmp/nginx.error.log
+
+# Set up the program in the image
+COPY decision_trees /opt/program
+WORKDIR /opt/program
+

advanced_functionality/scikit_bring_your_own/container/ReadMe.md

@@ -0,0 +1,72 @@
+# Bring-your-own Algorithm Sample
+
+This example shows how to package an algorithm for use with IM. We have chosen a simple [scikit-learn][skl] implementation of decision trees to illustrate the procedure.
+
+IM supports two execution modes: _training_ where the algorithm uses input data to train a new model and _serving_ where the algorithm accepts HTTP requests and uses the previously trained model to do an inference (also called "scoring", "prediction", or "transformation").
+
+The algorithm that we have built here supports both training and scoring in IM with the same container image. It is perfectly reasonable to build an algorithm that supports only training _or_ scoring as well as to build an algorithm that has separate container images for training and scoring.v
+
+In order to build a production grade inference server into the container, we use the following stack to make the implementer's job simple:
+
+1. __[nginx][nginx]__ is a light-weight layer that handles the incoming HTTP requests and manages the I/O in and out of the container efficiently.
+2. __[gunicorn][gunicorn]__ is a WSGI pre-forking worker server that runs multiple copies of your application and load balances between them.
+3. __[flask][flask]__ is a simple web framework used in the inference app that you write. It lets you respond to call on the `/ping` and `/invocations` endpoints without having to write much code.
+
+## The Structure of the Sample Code
+
+The components are as follows:
+
+* __Dockerfile__: The _Dockerfile_ describes how the image is built and what it contains. It is a recipe for your container and gives you tremendous flexibility to construct almost any execution environment you can imagine. Here. we use the Dockerfile to describe a pretty standard python science stack and the simple scripts that we're going to add to it. See the [Dockerfile reference][dockerfile] for what's possible here.
+
+* __build\_and\_push.sh__: The script to build the Docker image (using the Dockerfile above) and push it to the [Amazon EC2 Container Registry (ECR)][ecr] so that it can be deployed to IM. Specify the name of the image as the argument to this script. The script will generate a full name for the repository in your account and your configured AWS region. If this ECR repository doesn't exist, the script will create it.
+
+* __im-decision-trees__: The directory that contains the application to run in the container. See the next session for details about each of the files.
+
+* __local-test__: A directory containing scripts and a setup for running a simple training and inference jobs locally so that you can test that everything is set up correctly. See below for details.
+
+### The application run inside the container
+
+When IM starts a container, it will invoke the container with an argument of either __train__ or __serve__. We have set this container up so that the argument in treated as the command that the container executes. When training, it will run the __train__ program included and, when serving, it will run the __serve__ program.
+
+* __train__: The main program for training the model. When you build your own algorithm, you'll edit this to include your training code.
+* __serve__: The wrapper that starts the inference server. In most cases, you can use this file as-is.
+* __wsgi.py__: The start up shell for the individual server workers. This only needs to be changed if you changed where predictor.py is located or is named.
+* __predictor.py__: The algorithm-specific inference server. This is the file that you modify with your own algorithm's code.
+* __nginx.conf__: The configuration for the nginx master server that manages the multiple workers.
+
+### Setup for local testing
+
+The subdirectory local-test contains scripts and sample data for testing the built container image on the local machine. When building your own algorithm, you'll want to modify it appropriately.
+
+* __train-local.sh__: Instantiate the container configured for training.
+* __serve-local.sh__: Instantiate the container configured for serving.
+* __predict.sh__: Run predictions against a locally instantiated server.
+* __test-dir__: The directory that gets mounted into the container with test data mounted in all the places that match the container schema.
+* __payload.csv__: Sample data for used by predict.sh for testing the server.
+
+#### The directory tree mounted into the container
+
+The tree under test-dir is mounted into the container and mimics the directory structure that IM would create for the running container during training or hosting.
+
+* __input/config/hyperparameters.json__: The hyperparameters for the training job.
+* __input/data/training/leaf_train.csv__: The training data.
+* __model__: The directory where the algorithm writes the model file.
+* __output__: The directory where the algorithm can write its success or failure file.
+
+## Environment variables
+
+When you create an inference server, you can control some of Gunicorn's options via environment variables. These
+can be supplied as part of the CreateModel API call.
+
+    Parameter                Environment Variable              Default Value
+    ---------                --------------------              -------------
+    number of workers        MODEL_SERVER_WORKERS              the number of CPU cores
+    timeout                  MODEL_SERVER_TIMEOUT              60 seconds
+
+
+[skl]: http://scikit-learn.org "scikit-learn Home Page"
+[dockerfile]: https://docs.docker.com/engine/reference/builder/ "The official Dockerfile reference guide"
+[ecr]: https://aws.amazon.com/ecr/ "ECR Home Page"
+[nginx]: http://nginx.org/
+[gunicorn]: http://gunicorn.org/
+[flask]: http://flask.pocoo.org/

advanced_functionality/scikit_bring_your_own/container/build_and_push.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+
+# This script shows how to build the Docker image and push it to ECR to be ready for use
+# by IM.
+
+# The argument to this script is the image name. This will be used as the image on the local
+# machine and combined with the account and region to form the repository name for ECR.
+image=$1
+
+if [ "$image" == "" ]
+then
+    echo "Usage: $0 <image-name>"
+    exit 1
+fi
+
+# Get the account number associated with the current IAM credentials
+account=$(aws sts get-caller-identity --query Account --output text)
+
+if [ $? -ne 0 ]
+then
+    exit 255
+fi
+
+
+# Get the region defined in the current configuration (default to us-west-2 if none defined)
+region=$(aws configure get region)
+region=${region:-us-west-2}
+
+
+fullname="${account}.dkr.ecr.${region}.amazonaws.com/${image}:latest"
+
+# If the repository doesn't exist in ECR, create it.
+
+aws ecr describe-repositories --repository-names "${image}" > /dev/null 2>&1
+
+if [ $? -ne 0 ]
+then
+    aws ecr create-repository --repository-name "${image}" > /dev/null
+fi
+
+# Get the login command from ECR and execute it directly
+$(aws ecr get-login --region ${region} --no-include-email)
+
+# Build the docker image locally with the image name and then push it to ECR
+# with the full name.
+docker build  -t ${image} .
+docker tag ${image} ${fullname}
+
+docker push ${fullname}

advanced_functionality/scikit_bring_your_own/container/decision_trees/nginx.conf

@@ -0,0 +1,38 @@
+worker_processes 1;
+daemon off; # Prevent forking
+
+
+pid /tmp/nginx.pid;
+error_log /tmp/nginx.error.log;
+
+events {
+  # defaults
+}
+
+http {
+  include /etc/nginx/mime.types;
+  default_type application/octet-stream;
+  access_log /tmp/nginx.access.log combined;
+
+  upstream gunicorn {
+    server unix:/tmp/gunicorn.sock;
+  }
+
+  server {
+    listen 8080 deferred;
+    client_max_body_size 5m;
+
+    keepalive_timeout 5;
+
+    location ~ ^/(ping|invocations) {
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      proxy_set_header Host $http_host;
+      proxy_redirect off;
+      proxy_pass http://gunicorn;
+    }
+
+    location / {
+      return 404 "{}";
+    }
+  }
+}

advanced_functionality/scikit_bring_your_own/container/decision_trees/predictor.py

@@ -0,0 +1,83 @@
+# This is the file that implements a flask server to do inferences. It's the file that you will modify to
+# implement the scoring for your own algorithm.
+
+from __future__ import print_function
+
+import os
+import json
+import pickle
+import StringIO
+import sys
+import signal
+import traceback
+
+import flask
+
+import pandas as pd
+
+prefix = '/opt/ml/'
+model_path = os.path.join(prefix, 'model')
+
+# A singleton for holding the model. This simply loads the model and holds it.
+# It has a predict function that does a prediction based on the model and the input data.
+
+class ScoringService(object):
+    model = None                # Where we keep the model when it's loaded
+
+    @classmethod
+    def get_model(cls):
+        """Get the model object for this instance, loading it if it's not already loaded."""
+        if cls.model == None:
+            with open(os.path.join(model_path, 'decision-tree-model.pkl'), 'r') as inp:
+                cls.model = pickle.load(inp)
+        return cls.model
+
+    @classmethod
+    def predict(cls, input):
+        """For the input, do the predictions and return them.
+
+        Args:
+            input (a pandas dataframe): The data on which to do the predictions. There will be
+                one prediction per row in the dataframe"""
+        clf = cls.get_model()
+        return clf.predict(input)
+
+# The flask app for serving predictions
+app = flask.Flask(__name__)
+
+@app.route('/ping', methods=['GET'])
+def ping():
+    """Determine if the container is working and healthy. In this sample container, we declare
+    it healthy if we can load the model successfully."""
+    health = ScoringService.get_model() is not None  # You can insert a health check here
+
+    status = 200 if health else 404
+    return flask.Response(response='\n', status=status, mimetype='application/json')
+
+@app.route('/invocations', methods=['POST'])
+def transformation():
+    """Do an inference on a single batch of data. In this sample server, we take data as CSV, convert
+    it to a pandas data frame for internal use and then convert the predictions back to CSV (which really
+    just means one prediction per line, since there's a single column.
+    """
+    data = None
+
+    # Convert from CSV to pandas
+    if flask.request.content_type == 'text/csv':
+        data = flask.request.data.decode('utf-8')
+        s = StringIO.StringIO(data)
+        data = pd.read_csv(s, header=None)
+    else:
+        return flask.Response(response='This predictor only supports CSV data', status=415, mimetype='text/plain')
+
+    print('Invoked with {} records'.format(data.shape[0]))
+
+    # Do the prediction
+    predictions = ScoringService.predict(data)
+
+    # Convert from numpy back to CSV
+    out = StringIO.StringIO()
+    pd.DataFrame({'results':predictions}).to_csv(out, header=False, index=False)
+    result = out.getvalue()
+
+    return flask.Response(response=result, status=200, mimetype='text/csv')

advanced_functionality/scikit_bring_your_own/container/decision_trees/serve

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+
+# This file implements the scoring service shell. You don't necessarily need to modify it for various
+# algorithms. It starts nginx and gunicorn with the correct configurations and then simply waits until
+# gunicorn exits.
+#
+# The flask server is specified to be the app object in wsgi.py
+#
+# We set the following parameters:
+#
+# Parameter                Environment Variable              Default Value
+# ---------                --------------------              -------------
+# number of workers        MODEL_SERVER_WORKERS              the number of CPU cores
+# timeout                  MODEL_SERVER_TIMEOUT              60 seconds
+
+from __future__ import print_function
+import multiprocessing
+import os
+import signal
+import subprocess
+import sys
+
+cpu_count = multiprocessing.cpu_count()
+
+model_server_timeout = os.environ.get('MODEL_SERVER_TIMEOUT', 60)
+model_server_workers = int(os.environ.get('MODEL_SERVER_WORKERS', cpu_count))
+
+def sigterm_handler(nginx_pid, gunicorn_pid):
+    try:
+        os.kill(nginx_pid, signal.SIGQUIT)
+    except OSError:
+        pass
+    try:
+        os.kill(gunicorn_pid, signal.SIGTERM)
+    except OSError:
+        pass
+
+    sys.exit(0)
+
+def start_server():
+    print('Starting the inference server with {} workers.'.format(model_server_workers))
+
+
+    nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])
+    gunicorn = subprocess.Popen(['gunicorn',
+                                 '--timeout', str(model_server_timeout),
+                                 '-k', 'gevent',
+                                 '-b', 'unix:/tmp/gunicorn.sock',
+                                 '-w', str(model_server_workers),
+                                 'wsgi:app'])
+
+    signal.signal(signal.SIGTERM, lambda a, b: sigterm_handler(nginx.pid, gunicorn.pid))
+
+    # If either subprocess exits, so do we.
+    pids = set([nginx.pid, gunicorn.pid])
+    while True:
+        pid, _ = os.wait()
+        if pid in pids:
+            break
+
+    sigterm_handler(nginx.pid, gunicorn.pid)
+    print('Inference server exiting')
+
+# The main routine just invokes the start function.
+
+if __name__ == '__main__':
+    start_server()