Jek’s Kafka Get Started Notes

Terminology

Message →Batch

Topic → Partition

Producer → Consumer

Broker → Cluster

Zookeeper

Monitoring

Metrics

Broker-level Metrics

Topic-level Metrics

Partition-level Metrics

Install Apache Kafka

$  sudo apt update$  sudo apt install openjdk-11-jdk$ java -version
$ curl <download url of kafka...tgz> -o kafka.tgz
$ mkdir kafka$ cd kafka$ tar -xvzf ~/Downloads/kafka.tgz --strip 1

Start Apache Zookeeper server

$ bin/zookeeper-server-start.sh configookeeper.properties

Start Apache Kafka server

$ bin/kafka-server-start.sh config/server.properties

Create Kafka Topic

$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic testtopic --partitions 3 --replication-factor 1
$ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic testtopic
$ bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Push messages using Kafka Console Producer

$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic testtopic> abc
> xyz
>

Pull messages using Kafka Console Consumer

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testtopic
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testtopic --from-beginning

Message Structure

Start Zookeeper Server and Kafka Server in MacOS (installed via Homebrew)

$ /usr/local/bin/zkServer start$ /usr/local/bin/kafka-server-start /usr/local/etc/kafka/server.properties$ zookeeper-shell localhost:2181 ls /brokers/ids

Stop Zookeeper Server in MacOS (installed via Homebrew)

$ /usr/local/bin/zkServer stop

Java Example

Node.js Example

Python Example

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
CHOO Jek Bao

Love writing my thoughts, reading biographies, and meeting like-minded friends to talk on B2B software sales, engineering & cloud solution architecture.