|
| 1 | +# [SMP](https://github.com/StreamingMicroservicesPlatform) using Kubernetes and Kafka |
1 | 2 |
|
| 3 | +Streaming Platform for start-ups and small DevOps teams. |
| 4 | +A self-hosted PaaS, if you will, for using Kafka as backend for Your Microservices. |
2 | 5 |
|
3 |
| -# Kafka on Kubernetes |
| 6 | +We do what [Confluent's quickstart](https://docs.confluent.io/current/quickstart.html) does, |
| 7 | +but in Kubernetes as that's where services live in production. |
| 8 | +The setup includes Kafka, Schema Regitstry and REST Proxy. |
| 9 | + |
| 10 | +## Scope |
| 11 | + |
| 12 | + * Starts nicely on Minikube, with resources left for your services. |
| 13 | + * Can scale up to production workloads. |
| 14 | + - Example: `kubectl apply -f ./scale-3/` (TODO; we haven't really solved how to make scale a simple parameter yet) |
| 15 | + |
| 16 | +## Decisions up front |
| 17 | + |
| 18 | +Before you `kubectl` anything, you need to decide on: |
| 19 | + |
| 20 | + * Storage classes for Kafka and Zookeeper volumes. Select one of: |
| 21 | + - `kubectl apply -f configure-minikube/` |
| 22 | + - `kubectl apply -f configure-gke-pd/` |
| 23 | + * Run self-tests or not. They do generate some load, but indicate if the platform is working or not. |
| 24 | + - To include tests, replace `apply -f` with `apply -fR` in your `kubectl`s below. |
| 25 | + - Anything that isn't READY in `kubectl get pods -l test-target=kafka,test-type=readiness -w --all-namespaces` is a failed test. |
| 26 | + |
| 27 | +## Create |
| 28 | + |
| 29 | +Prerequisites: |
| 30 | + * Kubernets 1.8 (the [v2.0.0](https://github.com/Yolean/kubernetes-kafka/releases/tag/v2.0.0) release supports earlier versions) |
| 31 | + |
| 32 | +Required: |
| 33 | + * `kubectl apply -f ./zookeeper/` |
| 34 | + * `kubectl apply -f ./kafka/` |
| 35 | + |
| 36 | +Optional: |
| 37 | + * `kubectl apply -f ./confluent-platform/` |
| 38 | + * `kubectl apply -f ./prometheus/` |
| 39 | + * `kubectl apply -f ./kube-events/` |
| 40 | + * `kubectl apply -f ./ksql/` (coming in v3.1.0) |
| 41 | + * `kubectl apply -f ./log-processing-pipeline/` (coming in v3.2.0) |
| 42 | + |
| 43 | + |
| 44 | + |
| 45 | + |
| 46 | + |
| 47 | +# (previous readme) Kafka on Kubernetes |
4 | 48 |
|
5 | 49 | Transparent Kafka setup that you can grow with.
|
6 | 50 | Good for both experiments and production.
|
|
0 commit comments