Fine-grained fan-out and replication of Kafka event firehose between clusters

Matias Prados - January 8, 2021

Fine-grained fan-out and replication of Kafka event firehose between clusters/sites.

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka.

A simple projects illustrating real-time replication and fan-out of foreign exchange (fx) event streams from Kafka cluster A to Kafka cluster B, through Diffusion Cloud instance via the use of our Kafka Adapter.

These JavaScript code examples will help you publish fx events on real-time from a front-end app to a Kafka cluster A, consume from it and transform data on-the-fly via our powerful Topic Views feature.

Although we provide a fx data simulation client using JavaScript, to populate Kafka cluster A, this tutorial purely focus on the no-code solution to deliver event data between remote Kafka sites/clusters where not all the firehose data from one site needs to be replicated/fanned-out to the other.


Fine-grained distribution of Kafka event firehose with Topic Views

kafka-app-L1 introduces the concept of Topic Views, a dynamic mechanism to map part of a server’s Topic Tree to another. This enables real-time data transformation before replicating to a remote cluster as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data). This lesson also shows how to use our Kafka adapter to ingest and broadcast fx data using Diffusion Topic Views in order to consume what you need, not all the Kafka stream.


Features used in this lesson

Step 1: Configure Kafka Adapter in Cloud to ingest from Kafka Cluster A
Go to: Diffusion Cloud > Manage Service > Adapters > Kafka Adapter > Ingest from Kafka

Adapters > Kafka Adapter > Ingest from Kafka Config: - Bootstrap Server > connect to you Kafka cluster A (eg: "kafka-sasl.preprod-demo.pushtechnology") - Diffusion service credentials > admin, password (use the "Security" tab to create a user or admin account) - Kafka Topic subscription > the source topic from your Kafka cluster (eg: "FXPairs") - Kafka Topic value type > we are using JSON, but can be string, integer, byte, etc. - Kafka Topic key type > use string type for this code example.

Step 2: Check the Kafka stream FXPairs is ingested
Go to: Diffusion Cloud > Manage Service > Console > Topics
We can see the events from FXPairs Kafka topic (from cluster A) is now being published to Diffusion topic path: FXPairs. If there are no new events, it might be because the FXPairs topic has not received any updates from Kafka yet.


Step 3: Create Topic Views using Source value directives

Source value directives use the keyword expand() value directive to create multiple reference topics from a single JSON source topic, or value() directive to create a new JSON value with a subset of a JSON source topic.

Go to: Diffusion Cloud > Manage Service > Console > Topics > Topic Views

We are going to map the topic FXPairs stream (we get from Kafka cluster A) to a new Diffusion Topic View with path: pairs/<expand(/value/pairs,/pairName)>/ where /value/pairs,/pairName is the Kafka payload currency pairName (part of the JSON structure in the FXPairs Kafka topic).

This Topic View will take care of dynamic branching and routing of event streams in real-time, by only sending the specific currency pair from Kafka cluster A, to Kafka cluster B, and not the whole stream.

Topic View Specification for Currency Breakout:

map ?FXPairs// to pairs/<expand(/value/pairs,/pairName)>/

Topic View Specification for Tiers Expansion:

map ?pairs// to tiers/<expand(/tiers)>/<path(1)>


Step 4: Dynamic branching and routing of Kafka events firehose

As new events are coming in from the Kafka cluster A firehose, Diffusion is dynamically branching and routing the currency pairs, on-the-fly when replicating and fan-out to Kafka cluster B.

Note: The topic path will dynamically change as new currency pair values come in.

Go to: Diffusion Cloud > Manage Service > Console > Topics

The following image shows a Topic View for the following specification:

+ pairs

map ?FXPairs// to pairs/<expand(/value/pairs,/pairName)>/

+ tiers

map ?pairs// to tiers/<expand(/tiers)>/<path(1)>

When clicking on the “+” for “tiers” topic tree, the following image shows a Topic View for the following specification:

map ?pairs// to tiers/<expand(/tiers)>/<path(1)>


Suggested: 6 Lessons Using Topic Views

Lesson 1: Mapping Topics

Lesson 2: Mapping Topic Branches

Lesson 3: Extracting Source Topic Values

Lesson 4: Throttling Reference Topics

Lesson 5: Naming Reference Topic With Topic Content

Lesson 6: Changing Topic Properties Of Reference Topics

 


Step 5: Configure Kafka Adapter in Cloud to broadcast to Kafka cluster B

Go to: Diffusion Cloud > Manage Service > Adapters > Kafka Adapter > Broadcast to Kafka

Adapters > Kafka Adapter > Broadcast to Kafka Config: - Bootstrap Server > connect to you Kafka cluster A (eg: "kafka-plain.preprod-demo.pushtechnology") - Diffusion service credentials > admin, password (use the "Security" tab to create a user or admin account) - Topic subscription > the source topic from Diffusion to be broadcasted to Kafka cluster B (eg: from "tiers/2/GBP-USD" to "FXPairs.tier2.GBP.USD") - Topic value type > we are using JSON, but can be string, integer, byte, etc. - Kafka Topic key type > use string type for this code example.

Pre-requisites

  • Download our code examples or clone them to your local environment:
          ‘git clone https://github.com/diffusion-playground/kafka-integration-no-code
    • A Diffusion service (Cloud or On-Premise), version 6.5.0 or greater. Create a service here.

FX Data Generator

Really easy, just open the ‘index.html‘ file locally and off you go!


 

Download the Code Now

Get familiar with our SDK, our user friendly dashboard and all the power of our platform. Start Now!


Watch our Webinar

Tuned our webinars introducing concepts of your Kafka Adapter, Diffusion dashboard, console and Data Wrangling.

 

Thank you!

 


The Diffusion® Intelligent Event-Data Platform makes it easy to consume, enrich and deliver event-data in real-time across all network conditions. Push Technology pioneered and is the sole provider of real-time delta data streaming™ technology that powers mission-critical business applications worldwide. Leading brands use Push Technology to bring innovative products to market faster by reducing the software development efforts using the low-code features. The Diffusion® Intelligent Event-Data Platform is available on-premise, in-the-cloud, or in a hybrid configuration. Learn how Push Technology can reduce infrastructure costs, and increase speed, efficiency, and reliability, of your web, mobile, and IoT application.

Learn More: View Blog Posts and Developer Resources

Share:

Check out these other resources

Blog

Extend Kafka Topic Modeling with Diffusion

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka. A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our Kafka Adapter.

Blog

New Kafka Adapter: Connect Diffusion 6.6 with Apache Kafka

With the release of Diffusion 6.6 Preview 1, a beta version of the new Kafka adapter is now available for on-premise Diffusion. The Kafka adapter translates data between Diffusion topics and Apache Kafka events. We’ve designed it to make it quick and easy to integrate Kafka with Diffusion.

Blog

Diffusion® Extends Wrangling of Event Data

Event-driven architecture is at the core of modernization and digital transformation initiatives of IT organizations worldwide. As IT teams become more business and product aligned, they seek more autonomy to build software to be agile in addressing customer needs and market opportunities..