Connecting Apache Kafka to Azure Event Hubs

Recently, I worked on an integration with Azure Event Hubs. A colleague of mine faced challenges while trying to export messages from an existing Kafka topic and import them into Event Hubs. To assist, I’ve documented the steps below, which you may find useful.

Step 1: Download and Extract Apache Kafka

Apache Kafka is an open-source, distributed event streaming platform. It facilitates the construction of distributed systems and ensures high throughput. You can download Apache Kafka from the following link: Apache Kafka Download

tar -xzf kafka_2.13-3.1.0.tgz
cd kafka_2.13-3.1.0

Step 2: Start the Kafka Environment

Ensure that Java 8 or higher is already installed in your local environment. If not, download and install it from Oracle’s website.

To start all services, execute the following commands:

Start the ZooKeeper service:

bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka broker:

bin/kafka-server-start.sh config/server.properties

Step 3: Create and Set Up Configuration Files

Create a new file named connector.properties with the values below:

... (The content is mostly fine and technical, no changes)

Replace the placeholder values with those from your Azure endpoint. If you haven’t already, create a new namespace and deploy Event Hubs resources from the Azure portal. Note that you might need to select the Standard pricing tier or higher to successfully create Kafka topics in the next step.

The required password can be found in the Shared access policies settings of the Event Hub namespace, under the SAS Policy labeled RootManageSharedAccessKey.

Step 4: Create Three Kafka Topics

To create the topics manually, use the kafka-topics commands:

Create the configs topic:

... (Commands are mostly fine and technical, no changes)

Create the offsets topic:

... (Commands are mostly fine and technical, no changes)

Create the status topic:

... (Commands are mostly fine and technical, no changes)

Step 5: Run Kafka Connect

Kafka Connect is a tool for reliably and scalably streaming data between Apache Kafka and Azure Event Hubs. To continuously import and export your data, start the worker locally in distributed mode.

bin/connect-distributed.sh path/to/connect-distributed.properties

With everything set up, you can proceed to test import and export functions.

Step 6: Create Input and Output Files

Create a directory and two files: one for seed data to be read by the FileStreamSource connector and another to be written to by the FileStreamSink connector.

mkdir ~/connect-demo
seq 1000 > ~/connect-demo/input.txt
touch ~/connect-demo/output.txt

Step 7: Create FileStreamSource Connector

Next, let me guide you through launching the FileStreamSource connector:

... (Commands are mostly fine and technical, no changes)

Step 8: Create FileStreamSink Connector

Similarly, let’s proceed to launch the FileStreamSink connector:

... (Commands are mostly fine and technical, no changes)

Finally, confirm that the data has been replicated between files and is identical.

cat ~/connect-demo/output.txt

You should see that the output.txt file contains numbers from 1 to 1000, just like the input.txt file. That’s it! If you update input.txt, output.txt will sync accordingly.

Please note that Azure Event Hubs’ support for the Kafka Connect API is still in public preview. The FileStreamSource and FileStreamSink connectors deployed are not intended for production use and should only be used for demonstration purposes.