Extending Kafka

The Diffusion Real-Time API Management Platform enhances the benefits of Kafka by easily extending it over the Internet, for real-time web, mobile, and IoT applications. Diffusion synchronizes, manages, & distributes data among applications, devices, & systems via web, mobile, & satellite networks, extending the reliability, scalability, & functionalities of Kafka to the network’s edge.

Diffusion + Kafka Design

Diffusion is designed to offer pub/sub functionality with focus on data management and optimization over the internet and last-mile network connections, while retaining high-throughput and Quality-of-Service at scale. As a result, Diffusion reduces application code complexity and speeds time-to-market. By combining Kafka and Diffusion, developers get the benefits of a well-integrated back-end data pipeline, plus the critical reliability, resilience, and high-performance front-end data delivery.

When Kafka is used within an Enterprise, edge communication is less reliable; and scale, data types, and complex operations are big issues -- Diffusion simply works. - Brenock

Where Kafka meets Diffusion

While Kafka has a proven capability for high- throughput data handling within the data center, it is not designed for last-mile streaming. Real-time data delivery at scale over the public internet to Web, Mobile, or IoT devices is where Kafka meets Diffusion. When dealing with potentially hundreds of thousands of concurrent connections, the qualities that make Kafka extremely powerful in the back-end do not scale to address the unique challenges of edge networks.

Kafka Diffusion adapter diagram

The Diffusion Kafka Adapter

The Diffusion Kafka Adapter seamlessly integrates with Kafka Connect and Confluent’s Control Center. By simply placing the adapter in your Kafka deployment’s plugin directory, you can use the Control Center UI to easily add a new Source or Sink connector for ingestion or broadcast of events. The Diffusion Kafka Adapter automatically maps between Kafka message types and JSON, allowing for web, mobile, or IoT clients to consume data that is stored as rich data structures within Kafka.

The Diffusion Kafka adapter includes a source connector for publishing real-time Diffusion topic updates to Kafka topics, as well as a sink connector that broadcasts messages from one or more Kafka topics to Diffusion topics.

Using Diffusion adapter, Kafka data can be easily published:

  • To large numbers of web or mobile clients,
  • Over the Internet,
  • At high throughput and low latency.

The Diffusion Kafka Adapter supports:

  • primative values,
  • arrays, maps and structs,
  • dynamic mapping between Diffusion and Kafka topic paths.

The Diffusion Kafka Adapter Benefits

Diffusion radically simplifies the complexities of building Internet-based applications. By integrating Diffusion with Kafka, developers can:

  • Easily and quickly extend existing data pipelines across the web, without incurring the costs of solving the challenges of data delivery at scale.
  • Use Diffusion to act as the edge-broker for dispersing data to large numbers of end- user applications & devices, or efficiently and reliably ingest data from disparate sources including IoT devices, without any changes to the pre-existing Kafka architecture.
  • Increase reliability and resiliency of web & mobile applications, especially when coping with congested or unreliable networks
  • Expand and scale operations on-premise, in-the-cloud, or in hybrid environments to match rapidly increasing or fluctuating demand, with minimal investment in infrastructure.