So lets get started, we are talking about using the kafka controller without the zookeeper controller.
The alternative that helps us here is the KRaft controller which now comes part of the kafka and removes the dependency on the zookeeper service.
So, what is raft, as per the documentation from Raft,
Raft is a consensus algorithm that is designed to be easy to understand. It's equivalent to Paxos in fault-tolerance and performance. The difference is that it's decomposed into relatively independent subproblems, and it cleanly addresses all major pieces needed for practical systems. We hope Raft will make consensus available to a wider audience, and that this wider audience will be able to develop a variety of higher quality consensus-based systems than are available today.
What is zookeeper:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.
As mentioned at: https://www.confluent.io/blog/how-to-prepare-for-kip-500-kafka-zookeeper-removal-guide/
Current solution uses zookeeper for cluter info and metadata where as the proposed one removed dependency on zookeeper and uses raft to get consensus.
Below is the configuration to setup kafka with raft in kubernetes