Merge branch 'master' into test-producer-acks-config

Author: solsson

kafka/00namespace.yml renamed to 00-namespace.yml

@@ -1,78 +1,69 @@
-_Manifests here require Kubernetes 1.8 now.
-On earlier versions use [v2.1.0](https://github.com/Yolean/kubernetes-kafka/tree/v2.1.0)._
+# Kafka for Kubernetes
 
-# Kafka on Kubernetes
+This community seeks to provide:
+ * Production-worthy Kafka setup for persistent (domain- and ops-) data at small scale.
+ * Operational knowledge, biased towards resilience over throughput, as Kubernetes manifest.
+ * A platform for event-driven (streaming!) microservices design using Kubernetes.
 
-Transparent Kafka setup that you can grow with.
-Good for both experiments and production.
+To quote [@arthurk](https://github.com/Yolean/kubernetes-kafka/issues/82#issuecomment-337532548):
 
-How to use:
- * Good to know: you'll likely want to fork this repo. It prioritizes clarity over configurability, using plain manifests and .propeties files; no client side logic.
- * Run a Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) or real.
- * Quickstart: use the `kubectl apply`s below.
- * Have a look at [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s, or the official forks:
-   - [kubernetes-kafka-small](https://github.com/Reposoft/kubernetes-kafka-small) for single-node clusters like Minikube.
-   - [StreamingMicroservicesPlatform](https://github.com/StreamingMicroservicesPlatform/kubernetes-kafka) Like Confluent's [platform quickstart](https://docs.confluent.io/current/connect/quickstart.html) but for Kubernetes.
- * Join the discussion in issues and PRs.
+> thanks for creating and maintaining this Kubernetes files, they're up-to-date (unlike the kubernetes contrib files, don't require helm and work great!
 
-No readable readme can properly introduce both [Kafka](http://kafka.apache.org/) and [Kubernetes](https://kubernetes.io/),
-but we think the combination of the two is a great backbone for microservices.
-Back when we read [Newman](http://samnewman.io/books/building_microservices/) we were beginners with both.
-Now we've read [Kleppmann](http://dataintensive.net/), [Confluent](https://www.confluent.io/blog/) and [SRE](https://landing.google.com/sre/book.html) and enjoy this "Streaming Platform" lock-in :smile:.
+## Getting started
 
-We also think the plain-yaml approach of this project is easier to understand and evolve than [helm](https://github.com/kubernetes/helm) [chart](https://github.com/kubernetes/charts/tree/master/incubator/kafka)s.
+We suggest you `apply -f` manifests in the following order:
+ * Your choice of storage classes from [./configure](./configure/)
+ * [namespace](./00-namespace.yml)
+ * [./rbac-namespace-default](./rbac-namespace-default/)
+ * [./zookeeper](./zookeeper/)
+ * [./kafka](./kafka/)
 
-## What you get
+That'll give you client "bootstrap" `bootstrap.kafka.svc.cluster.local:9092`.
 
-Keep an eye on `kubectl --namespace kafka get pods -w`.
+## Fork
 
-The goal is to provide [Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092`
-`
+Our only dependency is `kubectl`. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate.
+If you begin to rely on this kafka setup we recommend you fork, for example to edit [broker config](https://github.com/Yolean/kubernetes-kafka/blob/master/kafka/10broker-config.yml#L47).
 
-Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`.
+## Version history
 
-## Prepare storage classes
+| tag   | k8s ≥ | highlights |
+| ----- | ------ | ---------- |
+| 4.x  | 1.9+    | Kafka 1.1 dynamic config |
+| v4.1 | 1.9+    | Kafka 1.0.1 new [default](#148) [config](#170) |
+| v3.2 | 1.9.4, 1.8.9, 1.7.14 | Required for read-only ConfigMaps [#162](https://github.com/Yolean/kubernetes-kafka/issues/162) [#163](https://github.com/Yolean/kubernetes-kafka/pull/163) [k8s #58720](https://github.com/kubernetes/kubernetes/pull/58720) |
+| v3.1  | 1.8    | The painstaking path to `min.insync.replicas`=2 |
+| v3.0  | 1.8    | [Outside access](#78), [modern manifests](#84), [bootstrap.kafka](#52) |
+| v2.1  | 1.5    | Kafka 1.0, the init script concept |
+| v2.0  | 1.5    | [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s |
+| v1.0  | 1      | Stateful? In Kubernetes? In 2016? Yes. |
 
-For Minikube run `kubectl apply -f configure/minikube-storageclass-broker.yml; kubectl apply -f configure/minikube-storageclass-zookeeper.yml`.
+All available as [releases](https://github.com/Yolean/kubernetes-kafka/releases).
 
-There's a similar setup for GKE, `configure/gke-*`. You might want to tweak it before creating.
+## Monitoring
 
-## Start Zookeeper
+Have a look at:
+ * [./prometheus](./prometheus/)
+ * [./linkedin-burrow](./linkedin-burrow/)
+ * [or plain JMX](https://github.com/Yolean/kubernetes-kafka/pull/96)
+ * what's happening in the [monitoring](https://github.com/Yolean/kubernetes-kafka/labels/monitoring) label.
+ * Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first.
 
-The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances.
+## Outside (out-of-cluster) access
 
-```
-kubectl apply -f ./zookeeper/
-```
+Available for:
 
-To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage.
+ * [Brokers](./outside-services/)
 
-## Start Kafka
+## Fewer than three nodes?
 
-```
-kubectl apply -f ./kafka/
-```
+For [minikube](https://github.com/kubernetes/minikube/), [youkube](https://github.com/Yolean/youkube) etc:
 
-You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:
-```
-kubectl -n kafka logs kafka-0 | grep "Registered broker"
-# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)
-```
+ * [Scale 1](https://github.com/Yolean/kubernetes-kafka/pull/44)
+ * [Scale 2](https://github.com/Yolean/kubernetes-kafka/pull/118)
 
-That's it. Just add business value :wink:.
+## Stream...
 
-## RBAC
-
-For clusters that enfoce [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/) there's a minimal set of policies in
-```
-kubectl apply -f rbac-namespace-default/
-```
-
-## Tests
-
-Tests are based on the [kube-test](https://github.com/Yolean/kube-test) concept.
-Like the rest of this repo they have `kubectl` as the only local dependency.
-
-Run self-tests or not. They do generate some load, but indicate if the platform is working or not.
- * To include tests, replace `apply -f` with `apply -R -f` in your `kubectl`s above.
- * Anything that isn't READY in `kubectl get pods -l test-type=readiness --namespace=test-kafka` is a failed test.
+ * [Kubernetes events to Kafka](./events-kube/)
+ * [Container logs to Kafka](https://github.com/Yolean/kubernetes-kafka/pull/131)
+ * [Heapster metrics to Kafka](https://github.com/Yolean/kubernetes-kafka/pull/120)

kafka/test/00namespace.yml renamed to 01-test-namespace.yml

@@ -0,0 +1,43 @@
+kind: ConfigMap
+metadata:
+  name: avro-tools-config
+  namespace: kafka
+apiVersion: v1
+data:
+  schema-registry.properties: |-
+    port=80
+    listeners=http://0.0.0.0:80
+    kafkastore.bootstrap.servers=PLAINTEXT://bootstrap.kafka:9092
+    kafkastore.topic=_schemas
+    debug=false
+
+    # https://github.com/Landoop/schema-registry-ui#prerequisites
+    access.control.allow.methods=GET,POST,PUT,OPTIONS
+    access.control.allow.origin=*
+
+  kafka-rest.properties: |-
+    #id=kafka-rest-test-server
+    listeners=http://0.0.0.0:80
+    bootstrap.servers=PLAINTEXT://bootstrap.kafka:9092
+    schema.registry.url=http://avro-schemas.kafka:80
+
+    # https://github.com/Landoop/kafka-topics-ui#common-issues
+    access.control.allow.methods=GET,POST,PUT,DELETE,OPTIONS
+    access.control.allow.origin=*
+
+  log4j.properties: |-
+    log4j.rootLogger=INFO, stdout
+
+    log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+    log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+    log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
+
+    log4j.logger.kafka=WARN, stdout
+    log4j.logger.org.apache.zookeeper=WARN, stdout
+    log4j.logger.org.apache.kafka=WARN, stdout
+    log4j.logger.org.I0Itec.zkclient=WARN, stdout
+    log4j.additivity.kafka.server=false
+    log4j.additivity.kafka.consumer.ZookeeperConsumerConnector=false
+
+    log4j.logger.org.apache.kafka.clients.Metadata=DEBUG, stdout
+    log4j.logger.org.apache.kafka.clients.consumer.internals.AbstractCoordinator=INFO, stdout

README.md

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: avro-rest
+  namespace: kafka
+spec:
+  ports:
+  - port: 80
+  selector:
+    app: rest-proxy

avro-tools/avro-tools-config.yml

@@ -0,0 +1,46 @@
+apiVersion: apps/v1beta2
+kind: Deployment
+metadata:
+  name: avro-rest
+  namespace: kafka
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: rest-proxy
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app: rest-proxy
+    spec:
+      containers:
+      - name: cp
+        image: solsson/kafka-cp@sha256:2797da107f477ede2e826c29b2589f99f22d9efa2ba6916b63e07c7045e15044
+        env:
+        - name: KAFKAREST_LOG4J_OPTS
+          value: -Dlog4j.configuration=file:/etc/kafka-rest/log4j.properties
+        command:
+        - kafka-rest-start
+        - /etc/kafka-rest/kafka-rest.properties
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 80
+        livenessProbe:
+          httpGet:
+            path: /
+            port: 80
+        ports:
+        - containerPort: 80
+        volumeMounts:
+        - name: config
+          mountPath: /etc/kafka-rest
+      volumes:
+      - name: config
+        configMap:
+          name: avro-tools-config

avro-tools/rest-service.yml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: avro-schemas
+  namespace: kafka
+spec:
+  ports:
+  - port: 80
+  selector:
+    app: schema-registry

avro-tools/rest.yml

@@ -0,0 +1,47 @@
+apiVersion: apps/v1beta2
+kind: Deployment
+metadata:
+  name: avro-schemas
+  namespace: kafka
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: schema-registry
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app: schema-registry
+    spec:
+      containers:
+      - name: cp
+        image: solsson/kafka-cp@sha256:2797da107f477ede2e826c29b2589f99f22d9efa2ba6916b63e07c7045e15044
+        env:
+        - name: SCHEMA_REGISTRY_LOG4J_OPTS
+          value: -Dlog4j.configuration=file:/etc/schema-registry/log4j.properties
+        command:
+        - schema-registry-start
+        - /etc/schema-registry/schema-registry.properties
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 80
+        livenessProbe:
+          httpGet:
+            path: /
+            port: 80
+          initialDelaySeconds: 60
+        ports:
+        - containerPort: 80
+        volumeMounts:
+        - name: config
+          mountPath: /etc/schema-registry
+      volumes:
+      - name: config
+        configMap:
+          name: avro-tools-config

avro-tools/schemas-service.yml

@@ -0,0 +1,43 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: rest-test1
+  namespace: kafka
+spec:
+  backoffLimit: 1
+  template:
+    metadata:
+      name: rest-test1
+    spec:
+      containers:
+      - name: curl
+        image: solsson/curl@sha256:523319afd39573746e8f5a7c98d4a6cd4b8cbec18b41eb30c8baa13ede120ce3
+        env:
+        - name: REST
+          value: http://rest.kafka.svc.cluster.local
+        - name: TOPIC
+          value: test1
+        command:
+        - /bin/bash
+        - -ce
+        - >
+          curl --retry 10 --retry-delay 30 --retry-connrefused -I $REST;
+
+          curl -H 'Accept: application/vnd.kafka.v2+json' $REST/topics;
+
+          curl --retry 10 -H 'Accept: application/vnd.kafka.v2+json' $REST/topics/test1;
+          curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H "Accept: application/vnd.kafka.v2+json" --data "{\"records\":[{\"value\":\"Test from $HOSTNAME at $(date)\"}]}" $REST/topics/$TOPIC -v;
+          curl --retry 10 -H 'Accept: application/vnd.kafka.v2+json' $REST/topics/test2;
+
+          curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H "Accept: application/vnd.kafka.v2+json" --data '{"records":[{"value":{"foo":"bar"}}]}' $REST/topics/$TOPIC -v;
+
+          curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"name": "my_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}' $REST/consumers/my_json_consumer -v;
+
+          curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data "{\"topics\":[\"$TOPIC\"]}" $REST/consumers/my_json_consumer/instances/my_consumer_instance/subscription -v;
+
+          curl -X GET -H "Accept: application/vnd.kafka.json.v2+json" $REST/consumers/my_json_consumer/instances/my_consumer_instance/records -v;
+
+          curl -X DELETE -H "Content-Type: application/vnd.kafka.v2+json" $REST/consumers/my_json_consumer/instances/my_consumer_instance -v;
+
+          sleep 300
+      restartPolicy: Never