• Last Updated: 16 Oct 2023
• Category
• Arvind Das

kafka-kraft cluster in kubernetes

So lets get started, we are talking about using the kafka controller without the zookeeper controller.

The alternative that helps us here is the KRaft controller which now comes part of the kafka and removes the dependency on the zookeeper service. 

So, what is raft, as per the documentation from Raft,

Raft is a consensus algorithm that is designed to be easy to understand. It's equivalent to Paxos in fault-tolerance and performance. The difference is that it's decomposed into relatively independent subproblems, and it cleanly addresses all major pieces needed for practical systems. We hope Raft will make consensus available to a wider audience, and that this wider audience will be able to develop a variety of higher quality consensus-based systems than are available today.

 What is zookeeper:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

As mentioned at: https://www.confluent.io/blog/how-to-prepare-for-kip-500-kafka-zookeeper-removal-guide/

Current solution uses zookeeper for cluter info and metadata where as the proposed one removed dependency on zookeeper and uses raft to get consensus.

Below is the configuration to setup kafka with raft in kubernetes

  apiVersion: v1
  kind: Service
  metadata:
    name: kafka-svc
    labels:
      app: kafka-app
  spec:
    clusterIP: None
    ports:
      - name: "9092"
        port: 9092
        protocol: TCP
        targetPort: 9092
    selector:
      app: kafka-app
  ---
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: kafka
    labels:
      app: kafka-app
  spec:
    serviceName: kafka-svc
    replicas: 3
    selector:
      matchLabels:
        app: kafka-app
    template:
      metadata:
        labels:
          app: kafka-app
      spec:
        containers:
          - name: kafka-container
            image: doughgle/kafka-kraft
            ports:
              - containerPort: 9092
              - containerPort: 9093
            env:
              - name: REPLICAS
                value: "3"
              - name: SERVICE
                value: kafka-svc
              - name: NAMESPACE
                value: kafka
              - name: SHARE_DIR
                value: /mnt/kafka
              - name: CLUSTER_ID
                value: oh-sxaDRTcyAr6pFRbXyzA    
              - name: DEFAULT_REPLICATION_FACTOR
                value: "3"
              - name: DEFAULT_MIN_INSYNC_REPLICAS
                value: "2"
            volumeMounts:
              - name: data
                mountPath: /mnt/kafka
    volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes:
            - "ReadWriteOnce"
          resources:
            requests:
              storage: "1Gi"