Why Streaming Still Isn't Mainstream

Let's Solve the Triumvirate of Big Issues

Prologue

Confluent’s most recent stock price slump last week prompted a fair number on discussions on social media, and I couldn’t hold myself back from also taking part in some of them. In one of the threads on LinkedIn a fairly popular figure in the streaming scene with nearly 20k followers on LinkedIn, Tim Berglund, finally convinced me to stop just writing short comments, but to start writing down something a little bit more elaborate. So witness the start of my blog.

The Elusive Hockey Stick

I vividly remember the time when I found out about Kafka some 10+ years ago, and my enthusiasm about Kafka Streams when I found out about it in 2016. At the time I was working at SAP. I could witness the first delegation of Confluent (including Jay) coming over to the SAP headquarters (WDF01), and I was fortunate to host one of the first meetups featuring a brand new Confluent employee who just recently had come over from DataStax in 2017. Guess who.

At the time, I thought that streaming, and with it, Confluent, would take off in no time. We would see a hockey stick-like growth of both the streaming space in general, and Confluent in particular.

And both did and do grow. But the hockey stick? No.

But why? Why has a technology so promising to many not really delivered what many of us streaming enthusiasts would have predicted? Why is the streaming space and its accompanying market still so tiny compared to the database, lakehouse, and data lake space? Why hasn’t streaming yet arrived in the mainstream?

Of course there is no silver bullet to shoot down this question. That would be too easy. We are in software still, not in AI (sigh). But I am convinced that there are three big issues with streaming that can be solved right now. And I am convinced that by putting our focus on these three issues, the streaming space would be at a much more advantageous position to get closer to finding the holy grail of hockey stick growth. Oh, and AI has almost nothing to do with it.

On with the issues, and with how to solve them directly afterwards.

The Triumvirate of Big Issues

Issue 1: Leaky Abstractions

The first issue for streaming still not being mainstream is one of leaky abstractions, in streaming itself, but primarily in stream processing. What do I mean by this?

Imagine you are a software developer. Now picture that there would be a new programming language that’s cool and fast and lends itself perfectly for real-time scenarios. But, alas, it does require you to understand how your code is actually compiled into machine code and how it works at run-time. And for real-world use, you sometimes have to go down to directly writing some machine code here and there to work around some intentionally inbred limitations of the programming language (hey, that’s why it’s so cool and fast and lends itself perfectly for real-time scenarios in the first place).

But seriously, would you seriously consider switching to this new programming language?

Or, maybe a bit more fitting, imagine you are a database engineer. There is a new database-like technology called “stream processing” that’s also cool and fast and lends itself perfectly for real-time scenarios. But, unfortunately, it does require you oftentimes to understand how your SQL is actually compiled into a stream processing job, and what’s worse, how it is actually executed at run-time. And, you sometimes have also do to go down to directly writing some additional lower level API code here and there to work around some intentionally inbred design decisions of the new technology.

Sounds familiar? And would you actually consider switching to this new technology if you were a database engineer?

Up to here, my point should be a bit more obvious. I think that one of the fundamental issues that are standing in the way of the mainstream adoption of streaming is that most (popular) stream processing systems require you to know too much about too many things that should be more properly abstracted away. As it stands, the most popular stream processing systems, Kafka Streams and Flink, and many others as well, are teeming with leaky abstractions.

Contrast that with the database world, where you just write tons of SQL without ever thinking about how the SQL is transformed into an execution plan and how it is eventually executed.

The leaky abstractions in most stream processing systems today are not in itself a problem. In fact, for us streaming enthusiasts, they are a boon. We are needed. And it is fun. And intellectually challenging. And stream processing systems like Kafka Streams and Flink power many of the most important corporations world-wide - many that couldn’t do what they are doing now without stream processing.

But there is a but. With this state-of-affairs. We will not be able to convert too many database engineers to consider stream processing, and it is just plain logical that the streaming space cannot really grow too much outside its own rather little bubble.

And my beloved hockey stick - it will keep being next to impossible to reach.

Issue 2: Data Integration

We come to the second big issue of that still plagues streaming today: data integration. Like the leaky abstraction issue, it is a near direct consequence of Kafka coming from and first being adopted by tech companies, who boast, in stark contrast to most other firms on this planet, an abundance of tech talent.

If you have, like tech companies or at least tech-heavy companies, at least a critical mass of good developers, you can actually do stream processing using tools like Kafka Streams and Flink, and you can go down to the Processor API (Kafka Streams) or the DataStream API (Flink) in one of the (too many) cases where you have to. It is still hard and cumbersome, and debugging is a nightmare, but it works.

However, if we go back to Confluent and its recent stock slump, you cannot expect to grow the market significantly. You can only farm (=convert from Open Source Kafka/Flink to Confluent Cloud) those companies that are already in the streaming space, and leave all the the rest of the crop to the likes of Databricks, Snowflake et al.

With data integration, the story is in many ways similar and at times overlapping. So what is it?

Data integration with Kafka boils down to two questions:

1. How do I bring data from my source system to Kafka?

2. How do I bring data from Kafka to my target system?

You might be tempted to answer the following:

“Hey, that’s why we have Kafka Connect with its 100s of source and sink connectors, many of them even available in fully managed form (e.g. on Confluent Cloud)!”

I wholeheartedly disagree. Why? Two main reasons:

Firstly, what happens if none the available connectors fulfills your requirements, maybe in just one tiny little detail? Would you have someone who could write a new connector? And if so, would you be willing to suddenly run and operate your very own Kafka Connect cluster (your new connector would surely not be available fully-managed)?

Secondly, and even more important, I deem Kafka Connect a dinosaur. Sorry for the bluntness. I mean what is Kafka Connect? It is a JVM-based runtime for connectors written for the JVM that offers a number of Kafka-specific features like Single Message Transforms (SMTs) for stateless stream processing. Honestly, for me, it boils down to being a pluggable monolith from the pre-cloud and pre-microservices era. We have Kubernetes for 10+ years now.

To get a bit fairer, Kafka Connect does shine for straightforward connections from/to standard systems, e.g. for CDC scenarios (Debezium!). But how would you do Kafka-based data integration for all the other use cases? What are the alternatives?

There are alternatives. You could go for Benthos. But Benthos still is a Kafka Connect-like runtime for connectors with a limited set of transformations. Or point-and-click integration frameworks like Striim or StreamSets. Or Flink-based tools like Decodable. Or many others.

But what if, as ever so often in real-life, you need that tiny bit of custom coding to integrate your systems?

Then get ready to hit the abyss.

I did write that we have Kubernetes now. But now imagine you are leading the Kafka efforts of a non-tech company. Where you typically don’t have is an abundance of skills in cloud development and DevOps. Now you have a team that wants to read some data off Kafka and push that into their target system. And you tell the team to do this with a microservice to be deployed on Kubernetes, with a CI/CD pipeline in GitHub, GitLab or Azure DevOps. Plus they shall learn their bit of Kafka, deep down to the likes of “auto.offset.reset” and “session.timeout.ms” etc., not to forget the rebalancing. Oh, and for joins, please don’t forget to read up about the semantics of Stream-Stream and Stream-Table and Table-Table joins in Kafka Streams.

Can you see the leaky abstractions lurking out at us again? In the end, the team in question will, most likely, try to avoid having to use Kafka at all costs and secretly ask your still existing middleware team for help (or just keep sharing their data via sFTP).

In short, the problem with Kafka-based data integration is that there is Kafka Connect and a bunch of other systems available, but none of them is flexible enough to cover a significant range of real-world use cases. There is always that tiny little bit missing. And then, probably because Kafka and the streaming space were born out of the big tech companies, the only alternative that remains is deep down coding and DevOps.

There is no tangible middle ground. But there should be. As Maxime Beauchemin put it aptly in his blog “The Rise of the Data Engineer”:

ultimately code is the best abstraction there is for software

…and once you need code, in the streaming ecosystem, you are currently almost guaranteed to fall into the abyss of having to have a DevOps team to integrate your system with Kafka. Which you simply don’t have in many non-tech companies. As a consequence, these systems are more often than not just not integrated with Kafka.

And this is the second big issue holding streaming, and also Confluent, back from, well, you know, the hockey stick.

Issue 3: (Real-time) Use Case Thinking

The last big issue is maybe more a sales issue than anything else, but I find it important to mention nonetheless.

Frame yourself at a streaming conference or meetup. You stand at a table, have a drink, and you talk to someone else you don’t yet know. What question is definitely among the first few questions that you get asked?

What use case do you use Kafka for at your company?

Now think about it - would you get the same question on a Postgres meetup? I bet at least a lot less often.

We streaming enthusiasts are, in my opinion, too obsessed with isolated use cases rather than the whole picture. We think in terms of singular use cases such as logging, fraud detection etc., but we seldom think about what we could achieve with streaming in general. We also tend to narrow down our vision to real-time use cases only. Which are, optimistically, about 5% of the use cases in a typical non-tech company.

And this obsession with (real-time) use cases doesn’t only hold for us streaming practicioners. It percolates through towards the streaming vendor sales reps as well. As a consequence, they typically try to dig down deep into every pocket of the companies that they speak to to find just these (real-time) use cases in order to sell there. This must be a frustrating exercise. I wouldn’t like to walk in their shoes, to be honest. Maybe they also sometimes ask themselves: “Hockey stick, where art thou?”

How to Solve The Three Big Issues

Ok, I have written a lot about the issues. But what about the solutions?

So let’s switch from pessimist to optimist. Or, as Rutger Bregman will also tell you if you read his books (please do!), realist. Here you are. Three free solutions for the three big issues potentially worth billions of dollars ;-)

Solution 1: Differential Dataflow or DBSP

What can we do about the notorious leaky abstractions which make stream processing so hard to adopt for mere mortals?

First of all, a little marketing. Hubert Dulay and me have written an entire O’Reilly book called “Streaming Databases” also often touching this topic. TL;DR: After a lot of research and interviews (see Hubert’s substack), we came to the conclusion that there are stream processing engines that come without the leaky abstractions.

It all started with Differential Dataflow (DD), a stream processing engine implemented in Rust (GitHub) and now powering the likes of Materialize.com, pathway.com and bytewax.io. DD was the first stream processing engine allowing to build a Postgres-compatible streaming database (=Materialize) that can execute database SQL on streams, with the same semantics and consistency. One of the chapters in our book, chapter 6, goes a bit deeper into how it works, by heavily borrowing from a blogpost from former Materialize employee Jamie Brandon called “Internal Consistency in Streaming Systems”.

The next generation of these kinds of stream processing engines that are able to fully blur the distinction between databases and streaming is DBSP, implemented in Rust (GitHub) and also (pure!) Python (GitHub). This is the engine underneath feldera.com. With it, Feldera is able to convert customers e.g. running hundreds of lines of batch SQL on Databricks to their incremental stream processing engine.

Hence, what we could immediately build would be:

• a new version of Kafka Streams, possibly also for Python (I don’t write about the still rather unhealthy Java-centricity of streaming here, that would be another blog post), based on DBSP

• a fully managed version of DBSP/SQL a la Feldera, including Materialized Views, but maybe with a user experience like Materialize.com or RisingWave.com.

So the pieces are there. Out in the open. But sadly, the streaming scene and also Confluent do not seem to take notice. With these stream processing engines, the large leap from batch to streaming becomes a mere sidestep, and the huge market of databases, lakehouses and data lakes could finally be encroached upon.

Solution 2: A Low-Code Platform For Data Integration

I wrote a lot about data integration and how problematic it still is in the streaming ecosystem. We need to create a platform bridging the huge gap between the Kafka Connects of this world and the “abyss” of deep down cloud development just to connect your bespoke system to Kafka. How would it look like?

I would, in a way, follow in the footsteps of Databricks. What they have done right is to focus on the most low-code-ish programming language there is - Python. Which of course also comes in handy in the current world of AI.

What if we built a low-code platform for (Kafka-based) data integration based on Python?

You would have a full programming language at your disposal, giving you all the flexibility that you need in real-life. You have all the libraries in the world at your disposal (possibly even libraries akin to Kafka Connect connectors for e.g. CDC…), for all kinds of APIs and protocols. Compatible to all sources and sinks in the world. And you can use Confluent’s Python client library to interact with Kafka, or make your life even easier with my library on top, Kafi.

But stop. I cannot write a blog without a bit of AI-washing. You can of course also use LLMs to give you a first approximation of your coding, which you can then happily debug and perfect using on a Jupyter notebook. Once you are happy, you should be able to click on some “Deploy” button to turn your code into a container on Kubernetes, of course with all the logs and metrics for monitoring and alerting coming to your favorite observability platform out of the box. Quix has already built a platform kind of like this, albeit not centered on Data Integration.

Such a platform would still have cloud development and Kubernetes under the covers, and it would still have CI/CD pipelines, but easy as pie also for non-DevOps cracks. No excuse any longer for anybody not wanting to integrate their systems with Kafka in your average non-tech company.

Solution 3: Ubiquitous Streaming

Last but not least, what would I propose to fight our bias of “(real-time) use case thinking”?

Kafka is made for scale. Kafka only really shines at scale. But my point is that we shouldn’t fall prey to the thinking that we have to find this scale in individual, isolated (real-time) use cases. Plus I stand firmly to state that we should not restrict the use of Kafka just to real-time use cases. Especially in times where we can store limitless amounts of data on Kafka much more cheaply than ever before - think e.g. Warpstream, Redpanda, StreamNative, and Aiven’s diskless Kafka - and also avoid the large replication costs of the past.

In fact, we should aim high. Extremely high. We should aim at making Kafka the default for data, and, if you consider the latest Kafka/Iceberg integrations (Confluent’s Tableflow, Redpanda Iceberg Topics, Streambased.io, again Aiven’s diskless Kafka), also for lakehouses and data lakes. Regardless of whether the data is used for real-time use cases or not.

Only then can we benefit from the fact that the adoption of Kafka can secretly drive your entire organization towards a more decentralized, data mesh-like mindset. Adopting domain ownership is an emergent property of organizations focusing on Kafka for data integration. Streaming data products are as well a byproduct. And federated governance is at least eased by data contracts/schema registry on a truly self-service infra which is Kafka.

And now imagine that we had a database-like stream processing engine (“Solution 1”) and an easy as pie low-code data integration (“Solution 2”) - maybe even all wrapped in one easy-to-use streaming-based data platform.

We would be able to make streaming truly ubiquitous.

Epilogue

So there you have it.

Honestly, what I’d just like to achieve with this blog is that some fellow streaming enthusiasts wake up and begin to start peering out of the box. The situation we are all in, and Confluent with us, is ok. We can all live a good fun professional life (ok, some Kafka Connect administrators might argue with that).

But I cannot shake off the feeling that streaming could be so much bigger. So much more ubiquitous. And so much more mainstream. And that, if you do start peering out of the box, all the ingredients to cook up the eventual streaming revolution are actually already in front of us.

And the hockey stick.