Diffusion® Extends Wrangling of Event Data in v6.6 Preview 1

Sanjai Marimadaiah - November 17, 2020

Event-driven architecture is at the core of modernization and digital transformation initiatives of IT organizations worldwide. As IT teams become more business and product aligned, they seek more autonomy to build software to be agile in addressing customer needs and market opportunities. Hence decoupling of the software modules to create microservices that interact via events has become an integral part of the modernization strategy. Creation and distribution of event-data is core to the event-driven architecture.

Push Technology is a pioneer in the distribution of event data over the last mile through our flagship product Diffusion. Over the past decade our customers have demanded more capabilities to reduce the software development efforts at various stages in the lifecycle of event-data. As a result, Diffusion has transformed into a holistic platform for event-driven applications which we now call Diffusion® Intelligent Event Data Platform.

 

Diffusion can consume, enrich and deliver event-data

 

Introducing Diffusion 6.6 Preview 1

To meet the agility needs of our customers, we are introducing preview releases to provide early access to the innovation at Push Technology. These preview releases adhere to the high quality and stability standards of Diffusion and are fully supported in production. However, a few capabilities of a preview releases will evolve further before the full release: for example, Python SDK and Kafka adapter in Preview 1. Please refer to the release notes for more details.

Diffusion 6.6 Preview 1 is available now for download or use on Diffusion Cloud.

The following is a brief overview of the motivation behind the capabilities introduced in Preview 1.


Data Distribution

Efficient distribution of event data is a key requirement in an event-driven architecture. Diffusion is well known for its patented capabilities (including delta streaming, auto-scaling, load balancing etc.) to facilitate efficient event-data distribution via a pub-sub or request-response model. The simple-to-use APIs and SDKs are available in many popular programming languages and provide both WebSocket and REST Interfaces.

Python SDK

Growing adoption of Python has prompted us to introduce the Python SDK for interacting with Diffusion. In this preview release, the Python SDK supports subscribing to topics, and request-response messaging. We plan to expand its capabilities in the full 6.6 release.


Data Wrangling

Diffusion is the only platform to offer comprehensive features to manipulate and transform event-data through topic trees, topic views and access-control assignment at a granular level. Topic trees facilitate organization of topics into a multi-level hierarchy model. Topic views make it easy to re-organize the topic hierarchy by creating reference topics.

Topic view inserts

We are introducing capabilities to augment event-data associated with a topic via topic view inserts. With topic view insert, you can now merge data from topics other than the selected source topic into JSON reference topics. You can insert whole values, or partial data specified with JSON pointers. Advanced features include the ability to derive paths from values, and chaining multiple inserts.

Time series topic enhancements

Preview 1 brings two big enhancements to time series topics. You can now update a time series topic via the standard topic update API, treating a time series topic as if it were a single topic with the same event type as the time series. This means that when you update time series topics, you can now use features like update constraints, update streams and the addAndSet operation.

We now enable loading persisted or historical data into a time series topic and also allow setting a custom timestamp on historical data, instead of one based on the current time.


Data Gateway

Diffusion’s Data Gateway makes it easy to consume both static and streaming data from a wide array of sources and provides capabilities to prepare the data for wrangling before an event-data is created and made ready for distribution.

Kafka adapter

Many IT organizations are adopting Apache Kafka to streamline the movement of event-data between systems. However, they are hitting the limitations of Kafka very soon. Diffusion can seamlessly overcome the limitations of Kafka. Hence to address the fine-grained distribution and topic-level access control needs of our customers, we are introducing a new integrated adapter to interface with Apache Kafka. The adapter is bidirectional. It can translate data from Kafka topics to Diffusion topics, and from Diffusion to Kafka.


Conclusion

As we continue to innovate along the three dimensions of consumption, enrichment and delivery of event-data, we seek early adopters to provide feedback and collaborate with us to push the boundaries of event-driven architecture. Please contact us to deploy Diffusion® and collaborate with us.

Diffusion 6.6 Preview 1 is available now for download or use on Diffusion Cloud.


The Diffusion® Intelligent Event-Data Platform makes it easy to consume, enrich and deliver event-data in real-time across all network conditions. Push Technology pioneered and is the sole provider of real-time delta data streaming™ technology that powers mission-critical business applications worldwide. Leading brands use Push Technology to bring innovative products to market faster by reducing the software development efforts using the low-code features. The Diffusion® Intelligent Event-Data Platform is available on-premise, in-the-cloud, or in a hybrid configuration. Learn how Push Technology can reduce infrastructure costs, and increase speed, efficiency, and reliability, of your web, mobile, and IoT application.

Learn More: View Blog Posts and Developer Resources

Share:

Check out these other resources

Info

Info Center

Case Studies, White Papers, and Tech Talk notes relating to Diffusion and the customers benefiting from using it.

Info

Quick Start Guide

Step-by-step guide to getting started fast with Diffusion or Diffusion Cloud.

Info

Diffusion Cloud

Get the facts and the details about our Diffusion Cloud platform.