MongoDB Kafka Connector

July 03, 2022

Apache Kafka is an open-source publish/subscribe messaging system. Kafka Connect, a component of Apache Kafka, addresses the challenge of linking Apache Kafka with various datastores, including MongoDB. Kafka Connect offers:

2022 07 03

  • A fault-tolerant runtime for transferring data to and from datastores
  • A framework that enables the Apache Kafka community to share solutions for connecting Apache Kafka to different datastores

In this post, we'll focus on using MongoDB as a data lake. The MongoDB Kafka sink connector is a Kafka Connect connector that reads data from Apache Kafka and writes it to MongoDB. The official MongoDB Kafka Connector can be found here.


Start the Kafka Environment

Download the latest Kafka version from here.

$ curl -o kafka_2.13-3.2.0.tgz
$ tar -xzf kafka_2.13-3.2.0.tgz
$ cd kafka_2.13-3.2.0

Run the following commands to start all the services in the correct order. Begin with the ZooKeeper service.

$ bin/ config/

In another terminal session, start the Kafka broker service:

$ bin/ config/

Once all the services have successfully launched, you will have a basic Kafka environment up and running.

Install the Plugin

Download the JAR file from here and navigate to the /libs directory.

curl -L -o plugin/mongo-kafka-connect-1.7.0-all.jar

Edit config/ and update the plugin.path to point to the downloaded JAR file.


Create Configuration Properties

In the /config folder, create a file named


Message Types


Specific MongoDB Sink Connector Configuration


In the /config folder, create a file named


Connection and Source Configuration


Install MongoDB

Import the MongoDB public GPG Key by running the following command:

wget -qO - | sudo apt-key add -

Create the MongoDB Source List

echo "deb [ arch=amd64,arm64 ] focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list

Update the Local Package Database

sudo apt-get update

Install MongoDB Packages

sudo apt-get install -y mongodb-org

If you encounter any errors related to unmet dependencies, fix them with the following commands:

echo "deb impish-security main" | sudo tee /etc/apt/sources.list.d/impish-security.list
sudo apt-get update
sudo apt-get install libssl1.1

Verify MongoDB Status

Check that MongoDB has started successfully:

sudo systemctl status mongod

If it's inactive and needs to restart, run:

sudo systemctl restart mongod

Start Kafka Connect

To start Kafka Connect, execute the following command:

bin/ config/ config/ config/

Write Data to the Topic

Run the console producer client to write a few events into your topic. Each line you enter will result in a separate event being written to the topic.

$ bin/ --topic connect-test --bootstrap-server localhost:9092
This is my first event
This is my second event

Send Document Contents Through Your Connectors

To send the contents of a document through your connectors, insert a document into the MongoDB collection from which your source connector reads data. Use the following MongoDB shell commands:

use quickstart

After inserting the document, verify that your connectors have processed the change by checking the topicData collection.


You should see output similar to the following:

    "_id": ObjectId(...),
    "hello": "world",
    "travel": "MongoDB Kafka Connector"


For more information, visit the MongoDB Kafka Connector documentation.

Profile picture

Victor Leung, who blog about business, technology and personal development. Happy to connect on LinkedIn