Merge branch 'master' into min_df_fix

Author: aaronmarkham

advanced_functionality/multi_model_bring_your_own/multi_model_endpoint_bring_your_own.ipynb

@@ -11,6 +11,10 @@
     "\n",
     "For the inference container to serve multiple models in a multi-model endpoint, it must implement [additional APIs](https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html) in order to load, list, get, unload and invoke specific models. This notebook demonstrates how to build your own inference container that implements these APIs.\n",
     "\n",
+    "**Note**: Because this notebook builds a Docker container, it does not run in Amazon SageMaker Studio.\n",
+    "\n",
+    "This notebook was tested with the `conda_mxnet_p36` kernel running SageMaker Python SDK version 2.15.3 on an Amazon SageMaker notebook instance.\n",
+    "\n",
     "---\n",
     "\n",
     "### Contents\n",
@@ -553,7 +557,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.7.6"
   }
  },
  "nbformat": 4,

autopilot/autopilot_customer_churn_high_level_with_evaluation.ipynb

@@ -22,7 +22,7 @@
     "\n",
     "---\n",
     "\n",
-    "This notebook works with sagemaker python sdk >= 1.65.1.\n",
+    "This notebook works with sagemaker python sdk 2.x\n",
     "\n",
     "## Contents\n",
     "\n",
@@ -794,17 +794,19 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from sagemaker.predictor import RealTimePredictor\n",
-    "from sagemaker.content_types import CONTENT_TYPE_CSV\n",
+    "from sagemaker.predictor import Predictor\n",
+    "from sagemaker.serializers import CSVSerializer\n",
+    "from sagemaker.deserializers import CSVDeserializer\n",
     "\n",
     "predictor = automl.deploy(initial_instance_count=1,\n",
     "                          instance_type='ml.m5.2xlarge',\n",
     "                          candidate=candidates[best_candidate_idx],\n",
     "                          inference_response_keys=inference_response_keys,\n",
-    "                          predictor_cls=RealTimePredictor)\n",
-    "predictor.content_type = CONTENT_TYPE_CSV\n",
+    "                          predictor_cls=Predictor,\n",
+    "                          serializer=CSVSerializer(),\n",
+    "                          deserializer=CSVDeserializer())\n",
     "\n",
-    "print(\"Created endpoint: {}\".format(predictor.endpoint))\n"
+    "print(\"Created endpoint: {}\".format(predictor.endpoint_name))\n"
    ]
   },
   {
@@ -829,11 +831,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from io import StringIO\n",
-    "\n",
-    "prediction = predictor.predict(test_data_no_target.to_csv(sep=',', header=False, index=False)).decode('utf-8')\n",
-    "prediction_df = pd.read_csv(StringIO(prediction), header=None, names=inference_response_keys)\n",
-    "custom_predicted_labels = prediction_df.iloc[:,1].values >= best_candidate_threshold\n",
+    "prediction = predictor.predict(test_data_no_target.to_csv(sep=',', header=False, index=False))\n",
+    "prediction_df = pd.DataFrame(prediction, columns=inference_response_keys)\n",
+    "custom_predicted_labels = prediction_df.iloc[:,1].astype(float).values >= best_candidate_threshold\n",
     "prediction_df['custom_predicted_label'] = custom_predicted_labels\n",
     "prediction_df['custom_predicted_label'] = prediction_df['custom_predicted_label'].map({False: target_attribute_values[0], True: target_attribute_values[1]})\n",
     "prediction_df"
@@ -906,4 +906,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}

autopilot/model-explainability/explaining_customer_churn_model.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Kernel `Python 3 (Data Science)` works well with this notebook.\n",
     "\n",
-    "_This notebook was created and tested on an ml.m5.large notebook instance._\n",
+    "_This notebook was created and tested on an ml.m5.xlarge notebook instance._\n",
     "\n",
     "## Table of Contents\n",
     "\n",
@@ -101,9 +101,8 @@
    "source": [
     "import shap\n",
     "\n",
-    "from kernel_explainer_wrapper import KernelExplainerWrapper\n",
+    "from shap import KernelExplainer\n",
     "from shap import sample\n",
-    "from shap.common import LogitLink, IdentityLink\n",
     "from scipy.special import expit\n",
     "\n",
     "# Initialize plugin to make plots interactive.\n",
@@ -235,7 +234,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "churn_data = pd.read_csv('./Data sets/churn.txt')\n",
+    "churn_data = pd.read_csv('../Data sets/churn.txt')\n",
     "data_without_target = churn_data.drop(columns=['Churn?'])\n",
     "\n",
     "background_data = sample(data_without_target, 50)"
@@ -252,7 +251,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next, we create the `KernelExplainer`. Note that since it's a black box explainer, `KernelExplainer` only requires a handle to the predict (or predict_proba) function and does not require any other information about the model. For classification it is recommended to derive feature importance scores in the log-odds space since additivity is a more natural assumption there thus we use `LogitLink`. For regression `IdentityLink` should be used."
+    "Next, we create the `KernelExplainer`. Note that since it's a black box explainer, `KernelExplainer` only requires a handle to the\n",
+    "predict (or predict_proba) function and does not require any other information about the model. For classification it is recommended to\n",
+    "derive feature importance scores in the log-odds space since additivity is a more natural assumption there thus we use `logit`. For\n",
+    "regression `identity` should be used."
    ]
   },
   {
@@ -263,17 +265,16 @@
    "source": [
     "# Derive link function \n",
     "problem_type = automl_job.describe_auto_ml_job(job_name=automl_job_name)['ResolvedAttributes']['ProblemType'] \n",
-    "link_fn = IdentityLink if problem_type == 'Regression' else LogitLink \n",
+    "link = \"identity\" if problem_type == 'Regression' else \"logit\"\n",
     "\n",
-    "# the handle to predict_proba is passed to KernelExplainerWrapper since KernelSHAP requires the class probability\n",
-    "explainer = KernelExplainerWrapper(automl_estimator.predict_proba, background_data, link=link_fn())"
+    "# the handle to predict_proba is passed to KernelExplainer since KernelSHAP requires the class probability\n",
+    "explainer = KernelExplainer(automl_estimator.predict_proba, background_data, link=link)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Currently, `shap.KernelExplainer` only supports numeric data. A version of SHAP that supports text will become available soon. A workaround is provided by our wrapper `KernelExplainerWrapper`. Once a new version of SHAP is released, `shap.KernelExplainer` should be used instead of `KernelExplainerWrapper`.\n",
     "\n",
     "By analyzing the background data `KernelExplainer` provides us with `explainer.expected_value` which is the model prediction with all features missing. Considering a customer for which we have no data at all (i.e. all features are missing) this should theoretically be the model prediction."
    ]
@@ -326,7 +327,7 @@
    "outputs": [],
    "source": [
     "# Since shap_values are provided in the log-odds space, we convert them back to the probability space by using LogitLink\n",
-    "shap.force_plot(explainer.expected_value, shap_values, x, link=link_fn())"
+    "shap.force_plot(explainer.expected_value, shap_values, x, link=link)"
    ]
   },
   {
@@ -348,7 +349,7 @@
    "source": [
     "with ManagedEndpoint(ep_name) as mep:\n",
     "    shap_values = explainer.shap_values(x, nsamples='auto', l1_reg='num_features(5)')\n",
-    "shap.force_plot(explainer.expected_value, shap_values, x, link=link_fn())"
+    "shap.force_plot(explainer.expected_value, shap_values, x, link=link)"
    ]
   },
   {
@@ -396,7 +397,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "shap.force_plot(explainer.expected_value, shap_values, X, link=link_fn())"
+    "shap.force_plot(explainer.expected_value, shap_values, X, link=link)"
    ]
   },
   {

autopilot/model-explainability/kernel_explainer_wrapper.py

@@ -258,8 +258,6 @@
     "\n",
     "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n",
     "region=$(aws configure get region)\n",
-    "# specifically setting to us-east-2 since during the pre-release period, we support only that region.\n",
-    "region=${region:-us-east-2}\n",
     "\n",
     "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n",
     "\n",
@@ -595,14 +593,6 @@
     "Now that you have verified that the algorithm code works for training, live inference and batch inference in the above sections, you can start packaging it up as an Amazon SageMaker Algorithm."
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### Region Limitation\n",
-    "Seller onboarding is limited to us-east-2 region (CMH) only. The client we are creating below will be hard-coded to talk to our us-east-2 endpoint only."
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -611,7 +601,7 @@
    "source": [
     "import boto3\n",
     "\n",
-    "smmp = boto3.client('sagemaker', region_name='us-east-2', endpoint_url=\"https://sagemaker.us-east-2.amazonaws.com\")"
+    "smmp = boto3.client('sagemaker')"
    ]
   },
   {
@@ -807,21 +797,13 @@
     "A Model Package is a reusable model artifacts abstraction that packages all ingredients necessary for inference. It consists of an inference specification that defines the inference image to use along with an optional model weights location.\n"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### Region Limitation\n",
-    "Seller onboarding is limited to us-east-2 region (CMH) only. The client we are creating below will be hard-coded to talk to our us-east-2 endpoint only. (Note: You may have previous done this step in Part 3. Repeating here to keep Part 4 self contained.)"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "smmp = boto3.client('sagemaker', region_name='us-east-2', endpoint_url=\"https://sagemaker.us-east-2.amazonaws.com\")"
+    "smmp = boto3.client('sagemaker')"
    ]
   },
   {
@@ -982,7 +964,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.6.10"
   }
  },
  "nbformat": 4,

aws_marketplace/creating_marketplace_products/Bring_Your_Own-Creating_Algorithm_and_Model_Package.ipynb

@@ -13,6 +13,7 @@
     "\n",
     "This sample notebook shows you how to train a custom ML model using <font color='red'> For Seller to update:[Title_of_your_Algorithm](Provide link to your marketplace listing of your product)</font> from AWS Marketplace.\n",
     "\n",
+    "> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.\n",
     "\n",
     "#### Pre-requisites:\n",
     "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
@@ -844,9 +845,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "conda_python3",
    "language": "python",
-   "name": "python3"
+   "name": "conda_python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -858,7 +859,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.4"
+   "version": "3.6.10"
   }
  },
  "nbformat": 4,

aws_marketplace/curating_aws_marketplace_listing_and_sample_notebook/Algorithm/Sample_Notebook_Template/title_of_your_product-Algorithm.ipynb

@@ -11,6 +11,7 @@
     "\n",
     "This sample notebook shows you how to deploy <font color='red'> For Seller to update:[Title_of_your_ML Model](Provide link to your marketplace listing of your product)</font> using Amazon SageMaker.\n",
     "\n",
+    "> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.\n",
     "\n",
     "#### Pre-requisites:\n",
     "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
@@ -416,9 +417,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "conda_python3",
    "language": "python",
-   "name": "python3"
+   "name": "conda_python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -430,7 +431,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.4"
+   "version": "3.6.10"
   }
  },
  "nbformat": 4,

aws_marketplace/curating_aws_marketplace_listing_and_sample_notebook/ModelPackage/Sample_Notebook_Template/title_of_your_product-Model.ipynb

@@ -27,6 +27,8 @@ def get_vehicle_damage_detection_model_package_arn(current_region):
     def get_vehicle_recognition_model_package_arn(current_region):
         mapping = {
           "us-east-1" : "arn:aws:sagemaker:us-east-1:865070037744:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
+          "us-east-2" : "arn:aws:sagemaker:us-east-2:057799348421:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
+
           "ap-northeast-1" : "arn:aws:sagemaker:ap-northeast-1:977537786026:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
          "ap-northeast-2" : "arn:aws:sagemaker:ap-northeast-2:745090734665:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
           "ap-southeast-1" : "arn:aws:sagemaker:ap-southeast-1:192199979996:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
@@ -35,6 +37,9 @@ def get_vehicle_recognition_model_package_arn(current_region):
         "ap-south-1":    "arn:aws:sagemaker:ap-south-1:077584701553:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
          "ca-central-1":"arn:aws:sagemaker:ca-central-1:470592106596:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",   
           "eu-west-1" : "arn:aws:sagemaker:eu-west-1:985815980388:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
-          "eu-west-2" : "arn:aws:sagemaker:eu-west-2:856760150666:model-package/vehicle-5bbb43353155de115c9fabdde5167c06"
+          "eu-west-2" : "arn:aws:sagemaker:eu-west-2:856760150666:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
+          "us-west-2" : "arn:aws:sagemaker:us-west-2:594846645681:model-package/vehicle-5bbb43353155de115c9fabdde5167c06",
+          "us-west-1" : "arn:aws:sagemaker:us-west-1:382657785993:model-package/vehicle-5bbb43353155de115c9fabdde5167c06"
+
         }
         return mapping[current_region]

aws_marketplace/using_model_packages/auto_insurance/src/model_package_arns.py

@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -10,6 +11,7 @@
     "\n",
     "If such a sample notebook does not exist and you want to deploy and try an ML model package via code written in python language, this generic notebook can guide you on how to deploy and perform inference on an ML model package from AWS Marketplace.\n",
     "\n",
+    "> **Note**: This is a reference notebook and it cannot run unless you make changes suggested in the notebook.\n",
     "\n",
     "> **Note**:If  you are facing technical issues while trying an ML model package from AWS Marketplace and need help, please open a support ticket or write to the team on [email protected] for additional assistance.\n",
     "\n",
@@ -935,7 +937,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.6.10"
   }
  },
  "nbformat": 4,