This guide is for setting up a local Kafka environment in a single machine quickly. A properly configured Kafka cluster is laborious to set up and has a lot of dependencies like Apache ZooKeeper etc. and that can be disconcerting if you just want to develop locally or put together a proof of concept prototype.
This is obviously not the way to set up a production Kafka cluster as there is no security, redundancy, fault tolerance, scaling and a host of other requirements for a production grade environment. This will set up all the components required for a Kafka cluster (which would be individually ditrributed across several servers in a production environment) in a single machine. Which is handy to set up in a laptop, desktop or even a virtual machine you can use to represent your Kafka cluster.
The Setup uses the community edition of the Confluent Platform and is installed natively (not on Docker) on a Debian/Ubuntu based Linux Distribution. If you are using a different operating system for your workstation I suggest installing Virtualbox and set up a Debian/Ubuntu based virtual machine to install your Kafka cluster on.
Install JDK
Apache Kafka and it’s dependencies (like ZooKeeper) run on Java so you will need a working Java runtime. If you already have a working Java runtime you can skip this step.
sudo apt install openjdk-11-jdk
Add Confluent package repository
These commands will add a new repository to your apt sources and its associated key
wget -qO - https://packages.confluent.io/deb/5.5/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/5.5 stable main"
Install Confluent Platform
Now that the dependencies are in place and the repository is set up you can just install the Confluent Platform, in the example below version 2.12 is installed
sudo apt-get update && sudo apt-get install confluent-platform-2.12
To get the current version refer to this page. Also if you only want to install the Confluent Platform using only Confluent Community components use this command instead
sudo apt-get update && sudo apt-get install confluent-community-2.12
It is also a good idea to install Confluent’s CLI tool to make it easy to manage your cluster. In the example below I am installing it under bin in my home directory
curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /home/haddad/bin/
Start your Cluster
Now you should be able to start your cluster using the Confluent CLI tool
~/bin/confluent local start
You should see an output similar to this
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.ZMh9oLJH
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]
Starting control-center
control-center is [UP]
You can verify that all the components are up and running by using this command
sudo jps
You should see an output similar to this
22628 ConnectDistributed
21285 ConduktorLauncher
24518 Jps
22182 SchemaRegistryMain
23351 KsqlServerMain
21707 QuorumPeerMain
22444 KafkaRestMain
21836 SupportedKafka
23631 ControlCenter
You now have a working Kafka cluster running locally. You will have all the Kafka and ZooKeeper shell utilities as well as Confluent’s Control Center which you can access locally on http://localhost:9021
To stop the cluster just run this command
~/bin/confluent local stop