Apache Kafka Notes

A quick console demonstration; here are the steps:

1. Go to the Kafka download...

   $ cd /home/russ/Downloads/ApacheKafka
   

2. Start Zookeeper...

   $ bin/zookeeper-server-start.sh config/zookeeper.properties
   

3. Start Kafka Broker...

   $ bin/kafka-server-start.sh config/server.properties
   

4. Create topic...

   $ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic Demo
   

5. List created topics...

   $ bin/kafka-topics.sh --list --bootstrap-server localhost:9092
   Demo
   Test
   __consumer_offsets
   

6. Start Producer...

   $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Demo
   >This is a test of the Emergency Broadcast System.
   >This is only a test.    (Brought down consumer after second message received, then sent third message.)
   >This is the third message.
   

7. Start Consumer...

   $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic Demo
   This is a test of the Emergency Broadcast System.
   This is only a test.
   ^C                       (Bring down consumer after second message received, then bring consumer back up, still with option from-beginning.)
   $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic Demo
   This is a test of the Emergency Broadcast System.
   This is only a test.
   This is the third message
   

1. Start Zookeeper server. This is essential for Kafka operation:

   $ sudo /opt/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start
   /usr/bin/java
   ZooKeeper JMX enabled by default
   Using config: /opt/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
   Starting zookeeper ... STARTED
   

   Note: you can use the convenient Zookeeper instance bundled with Kafka to get a quick-n-dirty, single node Zookeeper instance if you like. To launch this instance, after installing Kafka, from Kafka's root subdirectory:

   $ bin/zookeeper-server-start.sh config/zookeeper.properties
   

2. Download Kafka binaries from Kafka: Download. Choose a mirror.

3. Copy the tarball, as root, to /opt and explode it.

   ~/Downloads/kafka $ sudo bash
   kafka # cp kafka_2.12-2.3.1.tgz /opt
   kafka # cd /opt
   /opt # tar -zxf kafka_2.12-2.3.1.tgz
   

4. Since Kafka can handle requests over a network, you should create a dedicated user for it to minimize damage to your host in case Kafka's compromised.

   kafka_2.12-2.3.1
   

Why? See bike-shedding.

message-type.dataset-name.data-name

Legal punctuation in Kafka...

...is limited to the set:

• dot/period: ..
• hyphen: -
• underscore: _
• less encouragingly: any alphanumeric

Legal characters in Kafka topic names...

...include, of course, ASCII alphanumerics plus dot, hyphen and underscore.

Message-type values

• logging —logging data
• queueing —classical queueing use cases
• tracking —tracking events like user clicks, page views, ad views, etc.
• etl/db —for ETL or data-capture uses
• streaming —intermediate topics created by stream-processing pipelines
• push —data push from off-line (batch) into on-line environments
• user —user-specific data such as scratch and test topics

Dataset values

The dataset name is analogous to the database name in an RDBMS. Think of it as a "category" for grouping topics. It's a sort of namespace.

Message-type values

Think of the data name as a table name in RDBMS. It's a good idea to include further dotted notation if more hierarchy would be useful within the dataset namespace.

Enforce rules?

This can be done by disabling auto.create.topics.enable and limiting who can create or how anyone can create topics.

Warning

Once a topic name is created, it cannot be changed.