Notes on Installing Kafka

There are some good posts about setting up Kafka like here. But they are missing a few things I had to figure out for myself, so I will go through the steps here.
Let's start with some kafka configuration:
  1. Install Java.
  2. Download kafka: wget http://mirror.cogentco.com/pub/apache/kafka/0.8.1/kafka_2.9.2-0.8.1.tgz to both your kafka and zookeeper nodes.
  3. tar zxvf kafka_2.9.2-0.8.1.tgz
  4. In config/server.properties set broker.id to a unique integer for each broker node.
  5. Also set: zookeeper.connect=zoo-host01:2181,zoo-host02:2181,zoo-host03:2181/kafka
  6. mkdir /tmp/kafka-logs with the correct ownership.
If Kafka is your first experience with Zookeeper, then you will also need to properly setup a multi-node zookeeper cluster.
  1. The zookeeper configuration on the zookeeper nodes from config/zookeeper.properties should have the following:
    • server.1=zoo-host01:2888:3888
    • server.2=zoo-host02:2888:3888
    • server.3=zoo-host03:2888:3888
    • initLimit=5
    • syncLimit=2
    • dataDir=/var/lib/zookeeper
  2. Now in the data dir create a file called myid which contains a unique integer per host name. So 1 for zoo-host01, 2 for zoo-host02 etc.
Starting things up:
  1. start up zookeeper on all your zookeeper nodes: su - myuser -c "nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2> /var/log/zookeeper/err.log > /var/log/zookeeper/out.log"
  2. Enter the zookeeper shell: bin/zookeeper-shell.sh zoo-host01:2181,zoo-host02:2181,zoo-host03:2181
  3. Create the /kafka chroot like this: create /kafka ""
  4. Now start up kafka on all the brokers: su - myuser -c "nohup bin/kafka-server-start.sh config/server.properties 2> /var/log/kafka/err.log 1> /var/log/kafka/out.log" &
Now create a topic with: bin/kafka-topics.sh --create --zookeeper zoo-host01:2181/kafka --replication-factor 3 --partitions 3 --topic test
If you have made it this far, you should be able to start sinking messages to the test topic above. The excellent documentation on the main Kafka site is probably where you need to head next.

Comments

Post a Comment