Kafka Python Demo

Orignally for PyCon Fi 2015

Kafka setup

Start Kafka in a docker container. Using spotify/kafka we get a single container running zookeeper and a single node of kafka (here we are running a docker-machine named dev for docker.

docker run -p 2181:2181 -p 9092:9092 \
  --env ADVERTISED_HOST=`docker-machine ip dev` \
  --env ADVERTISED_PORT=9092 spotify/kafka

Kafka demo

Start the consumer

export ZOOKEEPER=`docker-machine ip dev`:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test

Start the producer:

export KAFKA=`docker-machine ip dev`:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test

Punch keyboard, hit enter!

Consuming everything in a topic

Start the consumer with --from-beginning

export ZOOKEEPER=`docker-machine ip dev`:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test \
    --from-beginning

Kafka with Python

First, let's install kafka-python

pip install kafka-python

Then the jupyter:

pip install jupyter

Now you are good to go and can launch the attached notebook!

jupyter notebook Kafka\ Demo.ipynb