New Kafka Adapter: Connect Diffusion 6.6 with Apache Kafka

Naveena Dhougoda-Hamal - November 17, 2020

With the release of Diffusion 6.6 Preview 1, a beta version of the new Kafka adapter is now available for on-premise Diffusion.

The Kafka adapter translates data between Diffusion topics and Apache Kafka events. We’ve designed it to make it quick and easy to integrate Kafka with Diffusion.

Kafka is a widely used event streaming tool with high performance, solid scalability and resilience within the data center. Diffusion is efficient in providing the last mile delivery of data across unreliable networks, using delta streaming and reliable session reconnection. The new Kafka adapter acts as a bridge between these two robust systems and enables you to build a complete real-time data distribution solution.

Two-way translation between Kafka and Diffusion

The earlier standalone Kafka adapter for Diffusion used the Kafka Connect framework and was available via GitHub. The new adapter does not use Kafka Connect, and is instead based on Kafka Producers and Consumers.

A version of this new Kafka adapter has been available on Diffusion Cloud since August.

For the on-prem preview release, the adapter is more refined with additional features and improved configuration options. The adapter is included within the Diffusion server installation and can be stopped and started independently from the main server.

Features:

Pub-sub

With the publisher function of the adapter, messages from different Kafka topics can be aggregated in Diffusion and extracted into separate reference topics using topic views or modified using the new topic view insert clause. The newly transformed data can be published to another Kafka topic (using the subscriber feature of the adapter) or to Diffusion clients.

You can publish updates from a specific Diffusion topic path to Kafka. If you prefer, you can map updates from multiple Diffusion topic paths to a single Kafka topic. This is all configurable in the adapter.

Multiple Kafka cluster support

In this version of adapter, users can connect Diffusion with multiple Kafka clusters and transfer messages across them. With this feature, data from different Kafka clusters can be published to Diffusion topics. The adapter can be configured to consume messages from one Kafka cluster and publish messages to others. Here’s a diagram of an example use case:

Example configuration of Kafka adapter

Kafka security support

You can configure the adapter to connect securely to a Kafka cluster. SSL connection and SASL authentication mechanisms are supported. In this version, for SASL authentication, SASL/GSSAPI (Kerberos) and SASL/PLAIN are supported.

Configurable Kafka clients

Apache Kafka provides various configuration options to configure Kafka consumer and producers. These can be freely added as key-value pairs in the adapter configuration, so you can fine-tune the capability of consumers and producers.

Time series topics

Publishing and subscribing to Diffusion time series topics are supported by the adapter. To publish to a time series topic, use the “mapToTimeSeriesTopic” flag in the configuration.

Monitoring

The adapter uses Prometheus to expose its metrics. In addition to the default JVM level metrics provided by Prometheus, publisher and subscriber specific counters are added, giving you more insight into the messages processed by adapter.

More information

The 6.6 Preview 1 manual has full documentation about how to configure and use the Kafka adapter.

Remember, you can download and install 6.6 Preview 1 now or create a 6.6 service on Diffusion Cloud.

Please let us know what you think about the new adapter. We plan to expand its capabilities in the full 6.6 release and subsequent releases, so please get in touch with your feedback and feature requests.


The Diffusion® Intelligent Event-Data Platform makes it easy to consume, enrich and deliver event-data in real-time across all network conditions. Push Technology pioneered and is the sole provider of real-time delta data streaming™ technology that powers mission-critical business applications worldwide. Leading brands use Push Technology to bring innovative products to market faster by reducing the software development efforts using the low-code features. The Diffusion® Intelligent Event-Data Platform is available on-premise, in-the-cloud, or in a hybrid configuration. Learn how Push Technology can reduce infrastructure costs, and increase speed, efficiency, and reliability, of your web, mobile, and IoT application.

Learn More: View Blog Posts and Developer Resources

Share:

Check out these other resources

Info

Diffusion Kafka adapter documentation

Full docs in the 6.6 Preview 1 manual

Info

Apache Kafka site

Kafka downloads and documentation