Merge branch 'master' into feature/gcp

Author: RobertLucian

.github/ISSUE_TEMPLATE/bug-report.md

@@ -7,48 +7,48 @@ assignees: ''
 
 ---
 
-#### Version
+### Version
 
 (use `cortex version` to determine your version)
 
-#### Description
+### Description
 
 (describe the bug)
 
-#### Configuration
+### Configuration
 
 (paste relevant `cortex.yaml` or `cluster.yaml` configuration)
 
-#### Steps to reproduce
+### Steps to reproduce
 
 1. ...
 2. ...
 3. ...
 
-#### Expected behavior
+### Expected behavior
 
 (describe the behavior you expected)
 
-#### Actual behavior
+### Actual behavior
 
 (describe the behavior you experienced)
 
-#### Screenshots
+### Screenshots
 
 (optional)
 
-#### Stack traces
+### Stack traces
 
 (error output from `cortex logs <api name>`)
 
 ```text
 <paste stack traces here>
 ```
 
-#### Additional context
+### Additional context
 
 (optional)
 
-#### Suggested solution
+### Suggested solution
 
 (optional)

.github/ISSUE_TEMPLATE/feature-request.md

@@ -7,14 +7,14 @@ assignees: ''
 
 ---
 
-#### Description
+### Description
 
 (describe the feature)
 
-#### Motivation
+### Motivation
 
 (how this will improve the product / what problem will this solve / what is the use case)
 
-#### Additional context
+### Additional context
 
 (optional)

README.md

@@ -5,8 +5,7 @@
 
 <!-- Delete on release branches -->
 <!-- CORTEX_VERSION_README_MINOR -->
-
-[install](https://docs.cortex.dev/install) • [documentation](https://docs.cortex.dev) • [examples](https://github.com/cortexlabs/cortex/tree/0.22/examples) • [support](https://gitter.im/cortexlabs/cortex)
+[install](https://docs.cortex.dev/install) • [documentation](https://docs.cortex.dev) • [examples](https://github.com/cortexlabs/cortex/tree/0.23/examples) • [community](https://gitter.im/cortexlabs/cortex)
 
 # Deploy machine learning models to production
 
@@ -16,49 +15,45 @@ Cortex is an open source platform for deploying, managing, and scaling machine l
 
 ## Model serving infrastructure
 
-* Supports deploying TensorFlow, PyTorch, sklearn and other models as realtime or batch APIs
-* Ensures high availability with availability zones and automated instance restarts
-* Scales to handle production workloads with request-based autoscaling
-* Runs inference on spot instances with on-demand backups
-* Manages traffic splitting for A/B testing
+* Supports deploying TensorFlow, PyTorch, sklearn and other models as realtime or batch APIs.
+* Ensures high availability with availability zones and automated instance restarts.
+* Runs inference on spot instances with on-demand backups.
+* Autoscales to handle production workloads.
 
-#### Configure your cluster:
+#### Configure Cortex
 
 ```yaml
 # cluster.yaml
 
 region: us-east-1
-availability_zones: [us-east-1a, us-east-1b]
-api_gateway: public
 instance_type: g4dn.xlarge
 min_instances: 10
 max_instances: 100
 spot: true
 ```
 
-#### Spin up your cluster on your AWS account:
+#### Spin up Cortex on your AWS account
 
 ```text
 $ cortex cluster up --config cluster.yaml
 
 ￮ configuring autoscaling ✓
 ￮ configuring networking ✓
 ￮ configuring logging ✓
-￮ configuring metrics dashboard ✓
 
 cortex is ready!
 ```
 
 <br>
 
-## Reproducible model deployments
+## Reproducible deployments
 
-* Implement request handling in Python
-* Customize compute, autoscaling, and networking for each API
-* Package dependencies, code, and configuration for reproducible deployments
-* Test locally before deploying to your cluster
+* Package dependencies, code, and configuration for reproducible deployments.
+* Configure compute, autoscaling, and networking for each API.
+* Integrate with your data science platform or CI/CD system.
+* Test locally before deploying to your cluster.
 
-#### Implement a predictor:
+#### Implement a predictor
 
 ```python
 # predictor.py
@@ -73,70 +68,67 @@ class PythonPredictor:
     return self.model(payload["text"])[0]
 ```
 
-#### Configure an API:
-
-```yaml
-# cortex.yaml
-
-name: text-generator
-kind: RealtimeAPI
-predictor:
-  path: predictor.py
-compute:
-  gpu: 1
-  mem: 4Gi
-autoscaling:
-  min_replicas: 1
-  max_replicas: 10
-networking:
-  api_gateway: public
-```
-
-#### Deploy to production:
-
-```text
-$ cortex deploy cortex.yaml
-
-creating https://example.com/text-generator
+#### Configure an API
 
-$ curl https://example.com/text-generator \
-    -X POST -H "Content-Type: application/json" \
-    -d '{"text": "deploy machine learning models to"}'
-
-"deploy machine learning models to production"
+```python
+api_spec = {
+  "name": "text-generator",
+  "kind": "RealtimeAPI",
+  "predictor": {
+    "type": "python",
+    "path": "predictor.py"
+  },
+  "compute": {
+    "gpu": 1,
+    "mem": "8Gi",
+  },
+  "autoscaling": {
+    "min_replicas": 1,
+    "max_replicas": 10
+  },
+  "networking": {
+    "api_gateway": "public"
+  }
+}
 ```
 
 <br>
 
-## API management
+## Scalable machine learning APIs
 
-* Monitor API performance
-* Aggregate and stream logs
-* Customize prediction tracking
-* Update APIs without downtime
+* Scale to handle production workloads with request-based autoscaling.
+* Stream performance metrics and logs to any monitoring tool.
+* Serve many models efficiently with multi model caching.
+* Configure traffic splitting for A/B testing.
+* Update APIs without downtime.
 
-#### Manage your APIs:
+#### Deploy to your cluster
 
-```text
-$ cortex get
+```python
+import cortex
 
-realtime api       status     replicas   last update   latency   requests
+cx = cortex.client("aws")
+cx.deploy(api_spec, project_dir=".")
 
-text-generator     live       34         9h            247ms     71828
-object-detector    live       13         15h           23ms      828459
+# creating https://example.com/text-generator
+```
 
+#### Consume your API
 
-batch api          running jobs   last update
+```python
+import requests
 
-image-classifier   5              10h
+endpoint = "https://example.com/text-generator"
+payload = {"text": "hello world"}
+prediction = requests.post(endpoint, payload)
 ```
 
 <br>
 
 ## Get started
 
-```text
-$ pip install cortex
+```bash
+pip install cortex
 ```
 
 See the [installation guide](https://docs.cortex.dev/install) for next steps.

docs/guides/docker-hub-rate-limiting.md

@@ -2,6 +2,8 @@
 
 _WARNING: you are on the master branch, please refer to the docs on the branch that matches your `cortex version`_
 
+*Note: This guide is only relevant for Cortex version `0.22.1` and earlier. Starting in version `0.23.0`, we've migrated from Docker Hub to Quay (which allows for unlimited image pulls for unauthenticated users). If you upgrade to version >= `0.23.0`, you can disregard this guide.*
+
 Docker Hub's [newly enforced rate-limiting policy](https://www.docker.com/increase-rate-limits) can negatively impact your cluster. This is much likelier to be an issue if you've set `subnet_visibility: private` in your cluster configuration file, since with private subnets, all requests from all nodes are routed through the NAT Gateway, and will therefore have the same IP address (docker imposes the rate limit per IP address). If you haven't specified `subnet_visibility` or have set `subnet_visibility: public`, this is less likely to be an issue for you, since each instance will have its own IP address.
 
 If you are affected by Docker Hub's rate limiting, your may encounter issues such as:
@@ -56,7 +58,7 @@ image_istio_galley: quay.io/cortexlabs/istio-galley:<VERSION>
 image_istio_citadel: quay.io/cortexlabs/istio-citadel:<VERSION>
 ```
 
-For Cortex cluster version < `0.16.0`, please upgrade your cluster to the latest version.
+For Cortex cluster version < `0.16.0`, please upgrade your cluster to the latest version. If you upgrade to version >= `0.23.0`, you can disregard this guide.
 
 Once you've updated your cluster configuration file, you can spin up your cluster (e.g. `cortex cluster up --config cluster.yaml`).
 

docs/guides/self-hosted-images.md

@@ -24,14 +24,15 @@ Also, take note of your AWS account ID. The account ID can be found in the _My A
 
 You can use the script below to push the Cortex images to your ECR registry. Make sure to update the `ecr_region`, `aws_account_id`, and `cortex_version` variables at the top of the file. Copy-paste the contents into a new file (e.g. `ecr.sh`), and then run `chmod +x ecr.sh`, followed by `./ecr.sh`. It is recommended to run this from an EC2 instance in the same region as your ECR repository, since it will be much faster.
 
+<!-- CORTEX_VERSION_README -->
 ```bash
 #!/bin/bash
 set -euo pipefail
 
 # user set variables
 ecr_region="us-west-2"
 aws_account_id="620970939130"  # example account ID
-cortex_version="0.22.1"
+cortex_version="0.23.0"
 
 source_registry="quay.io/cortexlabs"
 destination_ecr_prefix="cortexlabs"

examples/batch/image-classifier/README.md

@@ -1,6 +1,6 @@
 # Deploy models as Batch APIs
 
-_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_
+_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_
 
 This example shows how to deploy a batch image classification api that accepts a list of image urls as input, downloads the images, classifies them, and writes the results to S3.
 

examples/batch/image-classifier/cortex.yaml

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 - name: image-classifier
   kind: BatchAPI

examples/batch/image-classifier/predictor.py

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 import os
 import requests

examples/batch/onnx/README.md

@@ -1,6 +1,6 @@
 # Batch Image Classifier in ONNX
 
-_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_
+_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_
 
 <!-- CORTEX_VERSION_MINOR -->
 Please refer to the [tutorial](https://docs.cortex.dev/v/master/batch-api/image-classifier#deploy-your-batch-api) to see how to deploy a Batch API with Cortex.

examples/batch/onnx/cortex.yaml

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 - name: image-classifier
   kind: BatchAPI

examples/batch/onnx/predictor.py

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 import requests
 import numpy as np

examples/batch/tensorflow/README.md

@@ -1,6 +1,6 @@
 # Batch Image Classifier in TensorFlow
 
-_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_
+_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_
 
 <!-- CORTEX_VERSION_MINOR -->
 Please refer to the [tutorial](https://docs.cortex.dev/v/master/batch-api/image-classifier#deploy-your-batch-api) to see how to deploy a Batch API with Cortex.

examples/batch/tensorflow/cortex.yaml

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 - name: image-classifier
   kind: BatchAPI

examples/batch/tensorflow/predictor.py

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 import requests
 import numpy as np

examples/keras/document-denoiser/README.md

@@ -1,6 +1,6 @@
 # Clean Dirty Documents w/ Autoencoders
 
-_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_
+_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_
 
 This example model cleans text documents of anything that isn't text (aka noise): coffee stains, old wear artifacts, etc. You can inspect the notebook that has been used to train the model [here](trainer.ipynb).
 

examples/keras/document-denoiser/cortex.yaml

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 - name: document-denoiser
   kind: RealtimeAPI

examples/keras/document-denoiser/predictor.py

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 import boto3, base64, cv2, re, os, requests
 from botocore import UNSIGNED

examples/keras/document-denoiser/trainer.ipynb

@@ -13,7 +13,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# _WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_\n",
+    "# _WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_\n",
     "\n",
     "\n",
     "import keras\n",

examples/live-reloading/onnx/README.md

@@ -1,6 +1,6 @@
 ## Live-reloading model APIs
 
-_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)_
+_WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)_
 
 The model live-reloading feature is automatically enabled for the ONNX predictors. This means that any ONNX examples found in the [examples](../..) directory will already have this running.
 

examples/live-reloading/python/mpg-estimator/cortex.yaml

@@ -1,4 +1,4 @@
-# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.22.*, run `git checkout -b 0.22` or switch to the `0.22` branch on GitHub)
+# WARNING: you are on the master branch; please refer to examples on the branch corresponding to your `cortex version` (e.g. for version 0.23.*, run `git checkout -b 0.23` or switch to the `0.23` branch on GitHub)
 
 - name: mpg-estimator
   kind: RealtimeAPI