Victor Leung
Victor Leung
BlogFlower shop

Azure EventHub, LogicApp and DataVerse

May 19, 2022

Kafka messages can be exported and imported to Microsoft Cloud for Financial Services (FSI). It provides different components, such as a unified customer profile to manage customer data, and it could be used to store personally identifiable information (PII) data. The connection could be made; where data flow from Kafka to Azure EventHub, it uses LogicApp to further sync the data to DataVerse, which FSI could consume as shown in the diagram below:

screenshot 2022 05 18 at 9 47 25 pm 1

Follow the steps below to setup this connection:

  1. Send event to Azure EventHub.

For example, the python script below could use to send three simple event messages to Azure EventHub.

import time
import os
from azure.eventhub import EventHubProducerClient, EventData
from azure.eventhub.exceptions import EventHubError
import json

CONNECTION_STR = "Endpoint=sb://;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REPLACE_WITH_SHARED_ACCESS_KEY"

body = json.dumps({
    "id": "something"

def send_event_data_batch(producer, i):
    # Without specifying partition_id or partition_key
    # the events will be distributed to available partitions via round-robin.
    event_data_batch = producer.create_batch()

producer = EventHubProducerClient.from_connection_string(

start_time = time.time()
with producer:
    for i in range(3):
        send_event_data_batch(producer, i)

print("Send messages in {} seconds.".format(time.time() - start_time))

Replace the placeholder of the event hub name and connection string. If the message was sent successfully, you would see the message, like Send messages in 1.730254888534546 seconds.

If you got an error “Authentication Put-Token failed. Retries exhausted“, double-check the placeholder values and see if it’s correct.

  1. Azure Eventhub to LogicApp

Navigate to the Azure portal, search for LogicApp and create a new one. It can be used as automated workflows, where EventHub events as the trigger and DataVerse as the output. Fill in the webform to create a Logic App. You can select plan type as Consumption, which is best for entry-level development purposes.

screenshot 2022 05 18 at 9 50 53 pm

Once the LogicApp is created, navigate to Development Tools for the Logic app designer. We would make three steps in the process.

2.1 EventHub Trigger In the first step of the workflow, we would connect to EventHub as the trigger. For development purposes, shorten the check items time to 3 seconds. Follow the UI to select and connect your existing EventHub name with an example shown below:

screenshot 2022 05 18 at 9 52 05 pm

2.2. Initialize variables Next step, we would parse the events hubs message. The example message sent above is:

    "id": "something"

Where we want to get the value using the key “id”, this could be done via expression:


screenshot 2022 05 18 at 9 53 56 pm

2.3 Add a row to Data Verse Finally, the third step is to use the database connector. We can link to the corresponding Dataverse Tables to add a new row. If the table doesn’t exist yet, go to , select Dataverse, Tables to create a Table. The corresponding field can be filled in using step 2 initialized variables.

screenshot 2022 05 18 at 9 54 26 pm

Once it’s done, click Save of the workflow.


Dataverse is a database for storing data with Tables. If the LogicApp is triggered successfully when a new event is added, then you should see the new row is added to the DataVerse table

screenshot 2022 05 18 at 9 55 06 pm

Finally, once all the Vault data are synced to Azure FSI, the component can consume it, such as the Unified customer profile to manage customer data.

Navigate to Microsoft Cloud Solution Center with the account credentials to select the component.

screenshot 2022 05 18 at 9 55 51 pm

To launch the Dynamic 365 sandbox, navigate the Solution centre and click the “Launch” button.

screenshot 2022 05 18 at 9 56 38 pm

There are populated sample data for the Unified customer profile app like:

screenshot 2022 05 18 at 9 56 58 pm

Let me know if you got any questions setting this up. Cheers.

About Victor Leung

Software development professional with expertise in application architecture, cloud solutions deployment, and financial products development. Possess a Master's degree in Computer Science and an MBA in Finance. Highly skilled in AWS (Certified Solutions Architect Professional, Developer and SysOps Administrator), GCP (Professional Cloud Architect), Microsoft Azure, Kubernetes(CKA, CKAD, CKS, KCNA), and Scrum(PSM, PSPO) methodologies.

Happy to connect

Continuous improvement

Copyright © 2023.