Use a lighter readme with more links, as readers' background will vary widely

Author: solsson

README.md

@@ -1,72 +1,53 @@
 
-# Kafka as Kubernetes StatefulSet
 
-Example of three Kafka brokers depending on five Zookeeper instances.
+# Kafka on Kubernetes
 
-To get consistent service DNS names `kafka-N.broker.kafka`(`.svc.cluster.local`), run everything in a [namespace](http://kubernetes.io/docs/admin/namespaces/walkthrough/):
-```
-kubectl create -f 00namespace.yml
-```
+Transparent Kafka setup that you can grow with.
+Good for both experiments and production.
 
-## Set up Zookeeper
+How to use:
+ * Run a Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) or real.
+ * To quickly get a small Kafka cluster running, use the `kubectl apply`s below.
+ * To start using Kafka for real, fork and have a look at [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s.
+ * Join the discussion here in issues and PRs.
 
-The Kafka book (Definitive Guide, O'Reilly 2016) recommends that Kafka has its own Zookeeper cluster with at least 5 instances.
-We use the zookeeper build that comes with the Kafka distribution, and tweak the startup command to support StatefulSet.
+Why?
+See for yourself. No single readable readme can properly introduce both Kafka and Kubernets.
+Back when we read [Newman](http://samnewman.io/books/building_microservices/) we were beginners with both.
+Now we read [Kleppmann](http://dataintensive.net/), [Confluent's blog](https://www.confluent.io/blog/) and [SRE](https://landing.google.com/sre/book.html) and enjoy this "Streaming Platform" lock-in :smile:.
 
-```
-kubectl create -f ./zookeeper/
-```
+## What you get
 
-## Start Kafka
+Keep an eye on `kubectl --namespace kafka get pods -w`.
 
-Assuming you have your PVCs `Bound`, or enabled automatic provisioning (see above), go ahead and:
+[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092`
+`
 
-```
-kubectl create -f ./
-```
+Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`.
 
-You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:
-```
-kubectl -n kafka logs kafka-0 | grep "Registered broker"
-# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)
-```
+## Start Zookeeper
 
-## Testing manually
+The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances.
 
-There's a Kafka pod that doesn't start the server, so you can invoke the various shell scripts.
 ```
-kubectl create -f test/99testclient.yml
+kubectl create -f ./zookeeper/
 ```
 
-See `./test/test.sh` for some sample commands.
-
-## Automated test, while going chaosmonkey on the cluster
+To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage.
 
-This is WIP, but topic creation has been automated. Note that as a [Job](http://kubernetes.io/docs/user-guide/jobs/), it will restart if the command fails, including if the topic exists :(
-```
-kubectl create -f test/11topic-create-test1.yml
-```
+## Start Kafka
 
-Pods that keep consuming messages (but they won't exit on cluster failures)
 ```
-kubectl create -f test/21consumer-test1.yml
+kubectl create -f ./
 ```
 
-## Teardown & cleanup
-
-Testing and retesting... delete the namespace. PVs are outside namespaces so delete them too.
+You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:
 ```
-kubectl delete namespace kafka
-rm -R ./data/ && kubectl -n kafka delete pv datadir-kafka-0 datadir-kafka-1 datadir-kafka-2
+kubectl -n kafka logs kafka-0 | grep "Registered broker"
+# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)
 ```
 
-## Metrics, Prometheus style
-
-Is the metrics system up and running?
-```
-kubectl logs -c metrics kafka-0
-kubectl exec -c broker kafka-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics'
-kubectl logs -c metrics zoo-0
-kubectl exec -c zookeeper zoo-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics'
-```
-Metrics containers can't be used for the curl because they're too short on memory.
+That's it. Just add business value :wink:.
+For clients we tend to use [librdkafka](https://github.com/edenhill/librdkafka)-based drivers like [node-rdkafka](https://github.com/Blizzard/node-rdkafka).
+To use [Kafka Connect](http://kafka.apache.org/documentation/#connect) and [Kafka Streams](http://kafka.apache.org/documentation/streams/) you may want to take a look at our [sample](https://github.com/solsson/dockerfiles/tree/master/connect-files) [Dockerfile](https://github.com/solsson/dockerfiles/tree/master/streams-logfilter)s.
+Don't forget the [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s.