Extend Kafka Topic Modeling with Diffusion

Matias Prados - November 30, 2020


Diffusion Real-Time Event Stream

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka.
A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our Kafka Adapter.
These JavaScript code examples will help you publish fx data on real-time from a front-end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful Topic Views feature. You can also use other programming languages from our SDKs, including iOS, Android, C, .NET, and more.


Lesson 1: Publish event streams to Kafka cluster

Lesson 1 code example introduces the concept of our Kafka Adapter and its configuration, to consume fx topic data from your Diffusion server and publish it to Kafka cluster in specific Kafka topic.

Kafka Broadcast Diagram

Lesson 2: Fine-grained distribution of Kafka event firehose with Topic Views

Lesson 2 code example introduces the concept of Topic Views, a dynamic mechanism to map part of a server’s Topic Tree (data structure) to another. This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data).

Kafka Ingest Diagram

Lesson 3: Security with Topic-level access control

Lesson 3 code example introduces the concept of Security and topic path permissions for fine-grained security management of your data structure. It also shows how to Subscribe to fx data using Diffusion Topic Views in order to consume what you need, not all the Kafka stream.


Pre-requisites

  • Download our code examples or clone them to your local environment:
git clone https://github.com/pushtechnology/tutorials/
  • A Diffusion service (Cloud or On-Premise), version 6.5.0 or greater. Create a service here.

Create Service

Setup

Make sure to add Diffusion library to your code. For JavaScript, we have added the following line in our ‘public/kafka2diffusion.html‘ and ‘public/diffusion2kafka.html‘:

src=’https://download.pushtechnology.com/clients/6.5.1/js/diffusion-6.5.1.js’

Update ‘public/js/producerApp.js‘ and ‘public/js/subscriberApp.js‘ to the hostname of your Diffusion Cloud service, which you can find in your service dashboard. You can also leave the default values and connect to our sandbox service:

  • host: host (“kafkagateway.us.diffusion.cloud” by default)
  • user: ‘user’
  • password: ‘password’

Execution

Really easy, just open the ‘index.html‘ file locally and off you go!


The code in action

(Click to watch the video)

Video Tutorial


Download the Code Now

Get familiar with our SDK, our user friendly dashboard and all the power of our platform.
Start Now!


Watch our Webinar

Tuned our webinar introducing the basic concept of your Diffusion dashboard, console and Data Wrangling.

 

Thank you!

 


The Diffusion® Intelligent Event-Data Platform makes it easy to consume, enrich and deliver event-data in real-time across all network conditions. Push Technology pioneered and is the sole provider of real-time delta data streaming™ technology that powers mission-critical business applications worldwide. Leading brands use Push Technology to bring innovative products to market faster by reducing the software development efforts using the low-code features. The Diffusion® Intelligent Event-Data Platform is available on-premise, in-the-cloud, or in a hybrid configuration. Learn how Push Technology can reduce infrastructure costs, and increase speed, efficiency, and reliability, of your web, mobile, and IoT application.

Learn More: View Blog Posts and Developer Resources

Share:

Check out these other resources

Info

Info Center

Case Studies, White Papers, and Tech Talk notes relating to Diffusion and the customers benefiting from using it.

Info

Quick Start Guide

Step-by-step guide to getting started fast with Diffusion or Diffusion Cloud.

Info

Diffusion Cloud

Get the facts and the details about our Diffusion Cloud platform.