November 17, 2020 Sanjai Marimadaiah

Diffusion® Extends Wrangling of Event Data in v6.6 Preview 1

Event-driven architecture is at the core of modernization and digital transformation initiatives of IT organizations worldwide. As IT teams become more business and product aligned, they seek more autonomy to build software to be agile in addressing customer needs and market opportunities. Hence decoupling of the software modules to create microservices that interact via events has become an integral part of the modernization strategy. Creation and distribution of event-data is core to the event-driven architecture.

Push Technology is a pioneer in the distribution of event data over the last mile through our flagship product Diffusion. Over the past decade our customers have demanded more capabilities to reduce the software development efforts at various stages in the lifecycle of event-data. As a result, Diffusion has transformed into a holistic platform for event-driven applications which we now call Diffusion® Intelligent Event Data Platform.

 

Diffusion can consume, enrich and deliver event-data

 

Introducing Diffusion 6.6 Preview 1

To meet the agility needs of our customers, we are introducing preview releases to provide early access to the innovation at Push Technology. These preview releases adhere to the high quality and stability standards of Diffusion and are fully supported in production. However, a few capabilities of a preview releases will evolve further before the full release: for example, Python SDK and Kafka adapter in Preview 1. Please refer to the release notes for more details.

Diffusion 6.6 Preview 1 is available now for download or use on Diffusion Cloud.

The following is a brief overview of the motivation behind the capabilities introduced in Preview 1.


Data Distribution

Efficient distribution of event data is a key requirement in an event-driven architecture. Diffusion is well known for its patented capabilities (including delta streaming, auto-scaling, load balancing etc.) to facilitate efficient event-data distribution via a pub-sub or request-response model. The simple-to-use APIs and SDKs are available in many popular programming languages and provide both WebSocket and REST Interfaces.

Python SDK

Growing adoption of Python has prompted us to introduce the Python SDK for interacting with Diffusion. In this preview release, the Python SDK supports subscribing to topics, and request-response messaging. We plan to expand its capabilities in the full 6.6 release.


Data Wrangling

Diffusion is the only platform to offer comprehensive features to manipulate and transform event-data through topic trees, topic views and access-control assignment at a granular level. Topic trees facilitate organization of topics into a multi-level hierarchy model. Topic views make it easy to re-organize the topic hierarchy by creating reference topics.

Topic view inserts

We are introducing capabilities to augment event-data associated with a topic via topic view inserts. With topic view insert, you can now merge data from topics other than the selected source topic into JSON reference topics. You can insert whole values, or partial data specified with JSON pointers. Advanced features include the ability to derive paths from values, and chaining multiple inserts.

Time series topic enhancements

Preview 1 brings two big enhancements to time series topics. You can now update a time series topic via the standard topic update API, treating a time series topic as if it were a single topic with the same event type as the time series. This means that when you update time series topics, you can now use features like update constraints, update streams and the addAndSet operation.

We now enable loading persisted or historical data into a time series topic and also allow setting a custom timestamp on historical data, instead of one based on the current time.


Data Gateway

Diffusion’s Data Gateway makes it easy to consume both static and streaming data from a wide array of sources and provides capabilities to prepare the data for wrangling before an event-data is created and made ready for distribution.

Kafka adapter

Many IT organizations are adopting Apache Kafka to streamline the movement of event-data between systems. However, they are hitting the limitations of Kafka very soon. Diffusion can seamlessly overcome the limitations of Kafka. Hence to address the fine-grained distribution and topic-level access control needs of our customers, we are introducing a new integrated adapter to interface with Apache Kafka. The adapter is bidirectional. It can translate data from Kafka topics to Diffusion topics, and from Diffusion to Kafka.


Conclusion

As we continue to innovate along the three dimensions of consumption, enrichment and delivery of event-data, we seek early adopters to provide feedback and collaborate with us to push the boundaries of event-driven architecture. Please contact us to deploy Diffusion® and collaborate with us.

Diffusion 6.6 Preview 1 is available now for download or use on Diffusion Cloud.

diffusion

Enjoy the rich functionality of Diffusion 6.7 as part of your event-driven application.

diffusion
Quick Start Guide

Step-by-Step guides to get started

diffusion
Diffusion Cloud

SaaS offering to focus on business

diffusion
Diffusion On-Premise

A holistic pub-sub platform for real-time apps