Prologue
Two years ago, Hubert Dulay and me released our book “Streaming Databases” for O’Reilly. Our research for the book also involved a number of discussions with the leading figures of the space (https://hubertdulay.substack.com/podcast).
From then on, the more I delved deeper into the ocean of stream processing. the more intrigued I got. Something was off. Something was fundamentally wrong.
But what?
One year ago, Tim Berglund prompted me to write my very first Substack blog about what I had found out until then. I called the blog “Why Streaming Still Isn’t Mainstream”. It closes with this sentence:
“If you do start peering out of the box, all the ingredients to cook up the eventual streaming revolution are actually already in front of us.”
And they were. And as nearly everybody else was and still is in the process of being swallowed by the AI abyss, I took the liberty to, over the last year, start cooking up the stream processing revolution myself.
The result is called Kafi Streams.
“Kafi Streams” does sound a lot like “Kafka Streams” - on purpose. I have been a big fan of Kafka Streams since its first release in 2016, and on the surface, Kafi Streams offers a developer experience akin to Kafka Streams.
But under the surface, Kafi Streams makes stream processing strongly consistent and 10x easier.
How Can Stream Processing Be Made 10x Easier?
In my blog post a year ago I lamented the leaky abstractions of streaming and especially stream processing. I argued that the leaky abstractions of classical stream processing are effectively the main reason for why streaming as a whole has not yet become mainstream.
You could build super fancy real-time RAG with stream processing. Or you could feed your AI agents with always fresh data. You could save millions on your Snowflake, Databricks, Redshift or BigQuery bill.
Could, could, could. Too many conjunctives. Until today, the reality is that stream processing has been just too complicated and expensive to fulfill these promises. Because of the leaky abstractions.
And they led to too many contradictions, too. For example:
While stream proceesing could, in theory, make your systems more consistent because you can update them in real-time, the lack of strong consistency would immediately undermine that promise.
While streaming could, in theory, bring down your costs because you can replace batch re-computation with streaming incremental computation, millions of intermediate results from classical, eventually consistent stream processing would flood your downstream systems and immediately eat up that cost advantage as well.
Not speaking of the expenses to build, maintain and extend the stream processing pipelines in the first place.
Kafi Streams is stream processing without the leaky abstractions. With strong consistency. And 10x easier.
Kafi Streams deviates from classical stream processing by taking six design decisions that, taken together, are hopefully going to help to propel stream processing into another dimension altogether.
1 DBSP Instead of Ad-hoc
The first design decision is a change of perspective. To go for a relational stream processing engine instead of the append-only or ad-hoc way of classical stream processing.
Kafi Streams is based on pydbsp by Bruno Rucy (https://github.com/lucidarium-systems/pydbsp), a pure Python implementation of the DataBase Stream Processing (DBSP) theory by Mihai Budiu, Leonid Rhyzhyk et al. of Feldera (https://arxiv.org/abs/2203.16684).
Without DBSP and pydbsp, Kafi Streams would not exist. I cannot thank Mihai, Leonid and Bruno enough for their work.
DBSP enables fully incremental and fully deterministic stream processing:
Incremental as in true streaming. Every new message only triggers the re-computation of the data really affected by it.
Deterministic as in the same input streams always result in the same output streams. Message by message.
It is DBSP that gives Kafi Streams its clean, relational semantics where your joins do what your joins would do in a database, fully without any of the side effects that you need to cater for in classical stream processing. No surprises. No special stream processing expertise required.
2 Strong Consistency instead of Exactly-once Delivery
About 10 years ago, classical stream processors and also Apache Kafka itself had a big goal: exactly-once delivery. To achieve this, every possible stone was turned. Successfully. We got transactions in Kafka. We got exactly-once delivery in Kafka Streams based on them. And we got exactly-once delivery in Flink and many other stream processors.
Now isn’t it great that we have exactly-once delivery? On the one hand, yes. On the other, the implementation of it is so involved that it has further exacerbated the leaky abstractions. If you implement exactly-once delivery in a stream processor, you need to turn stones inside the entire system, on virtually all levels. And you get more complexity. And more leaky abstractions.
Reality check. 90% of consumers are either idempotent by design (e.g. they upsert the consumed keyed Kafka messages into their database), or, when they implement use cases such as email or text message sending, they de-duplicate the consumed messages anyway, e.g. using Redis - also because there is no guarantee that the messages that have come into Kafka in the first place do not contain duplicates caused by transient network errors etc..
And data lake ingestion? One of the main reasons for the rush towards exactly-once delivery was that data lakes at the time were simple, non-idempotent blob storages (Hadoop- or S3-based). We all know that today, this has fundamentally changed with the widespread adoption of table formats such as Iceberg that do support upserts.
Thus, the quest for exactly-once delivery looks a bit like chasing ghosts if you look at it today. And, worse, it completely overshadowed the efforts towards a much bigger issue: consistency. Somehow, we all quietly accepted that stream processing has to stay only eventually consistent - and vendor marketing made us believe that exactly-once delivery would even help with that somehow, even though the two are completely unrelated concepts.
Kafi Streams takes a different design decision. It does not implement exactly-once delivery because that would inherently make it much more complex and prone to leaky abstractions. And it provides strong consistency instead of eventual consistency.
3 Code Instead of SQL
Materialize has strongly consistent, relational stream processing since 2019 (based on Frank McSherry‘s Differential Dataflow). Feldera has it since 2023 (based on DBSP). But both tools decided to use SQL as their API.
In the real world of stream processing, SQL is often more a hindrance than a boon. Streaming data is messy. Much messier than data in a data warehouse. It’s operational data, typically coming from multiple disparate systems. Very often, you need that tiny bit of additional flexibility to sort out those messy real world issues.
In last year’s blog, I already quoted Maxime Beauchemin’s blog “The Rise of the Data Engineer” (https://www.freecodecamp.org/news/the-rise-of-the-data-engineer-91be18f1e603/) with:
“Ultimately, code is the best abstraction there is for software.”
Just code can provide you with the flexibility that is actually required in stream processing. This is why we have tools like Flink with its code-based Datastream or Table APIs, or Kafka Streams with its code-based DSL and Processor API. And this is why, IMHO, especially Materialize with its SQL-only approach has not found the adoption that I should it would get.
BTW, same story with Flink SQL and Confluent. We all know how this story has lately unfolded.
So - as a Kafka Streams fan from its inception, the next design decision was clear. Kafi Streams uses a code-based API. One where Kafka Streams developers will feel at home and hygge from day one. And where every map, filter, join or aggregation can be customized by code and lambdas.
All the flexibility to clean up the messy bits of the real world. And all the freedom.
4 Python Instead of Java
Kafi Streams is built for code-based stream processing. Now which programming language is by far the most accessible and has the by far most advanced ecosystem, especially in the context of data, and, I almost dare not include these two ubiquitous letters, also AI?
The next design decision is to base Kafi Streams on Python. And, by being code-based instead of SQL-based, Kafi Streams provides you the entire massive Python ecosystem at your fingertips.
Think Pandas, Polars, DuckDB. Or Pytorch, LangChain, Pydantic AI, smolagents, LlamaIndex, all the HuggingFace stuff, even inside your stream processing pipeline. And think about conveniently prototyping your stream processing code in a Jupyter notebook.
5 Single Node instead of Distributed Systems
I think that classical stream processing has, from the beginning, fallen into a trap of premature optimization - for horizontal scaling. Driven by the “NoSQL” hype around Hadoop and distributed systems, and the few use cases of the few big tech use cases where horizontal scaling (at the time) couldn’t be avoided, we were all made to think that horizontal scaling was a prerequisite for stream processing and, worse, more important than consistency and simplicity.
Horizontal scaling was the origin of most of those heinous leaky abstractions.
Don’t get me wrong, it makes total sense that a central streaming engine such as Kafka is distributed. But does that automatically mean that the stream processing also has to be a distributed system?
Reality check. The vast majority of stream processing use cases even in bigger companies do not need horizontal scaling at all. And even for those where it does not suffice - the world in 2026 is significantly different than 10-15 years ago.
Today, actually also pushed by the ever increasing hunger for vertical scaling for LLM inference that has slighly nudged also the CPU industry into that direction, we can have up to 288 cores in a single node.
There is another exciting new stream processing library out now (StoatFlow by Hartmut Armbruster,
https://stoatflow.io/
) that is, in a similar vein as Kafi Streams, built for single node. And, by the way, also recently solved Jamie Brandon’s “internal consistency” problem (https://stoatflow.io/blog/internal-consistency) (remember chapter 6 of Hubert Dulay’s and my book?).
So in short, today:
Single node is enough for most use cases, and
if you need more performance, multicore will probably bring you there.
And this is not just about stream processing. There is a growing trend for single node instead of distributed systems everywhere you look. Think DuckDB, new versions of SQLite, or the ever growing list of databases/services based on Postgres.
It looks like horizontal scaling is just not needed for 90% percent of the use cases. 90% of the use cases in most companies, including mine, a large retailer in Switzerland with $30B revenue. It is needed for telemetry, IoT, AdTech or log observability. And of course fraud detection. That’s the 10%. But the remaining 90% are use cases are mostly based on business data. SAP data. Checkout data. Products. Prices. Much much smaller volumes. At my company, the largest throughput Kafka topics transport 5M messages per day. That’s less than 60 messages per second on average. Why on earth would you employ a distributed system to process that type of workload and fall foul to all its leaky abstractions and just the sheer complexity?
BTW - the DuckDB folks seem to have made similar experiences: https://motherduck.com/blog/big-data-is-dead/, https://medium.com/towards-data-engineering/duckdb-the-death-of-small-scale-spark-e86ac6aedf23. I’d like to cite the latter blog by Nevenka Lukic:
“The right question is “does this job actually need a cluster?” For most pipelines processing under 50 GB on a single developer machine, the honest answer is no. The cluster adds JVM overhead, executor spin-up latency, shuffle staging cost, and configuration complexity to a problem that a single binary could finish in seconds. The engineer pays the distributed tax without receiving the distributed benefit.”
So the next design decision is to embrace single node instead of horizontal scaling.
And the advantages of dropping horizontal scaling are immense:
There is no need to accept eventual consistency. With Kafi Streams, you get strong consistency as you are used to from the database world.
There is no need for individual state stores and all the DevOps hassle caused by them. In Kafi Streams, there is only one global state per stream processing topology.
Ensuring fault tolerance by checkpointing the global state and the source stream offsets becomes almost trivial. Consequently, in Kafi Streams, you can either checkpoint to Kafka itself or to disk, S3 or Azure Blob Storage.
Dropping horizontal scaling does not come without trade-offs. Kafi Streams will definitely not replace the large Flink clusters at Uber.
But the goal of Kafi Streams is to open up the world of stream processing for everyone. Including everyone who has not been able to use stream processing before. And I assume that most of the use cases where stream processing could now be adopted are in fact so low in data volume that Kafi Streams on a single node easily suffices.
6 Native Time Windows and Expiration instead of Bolted On
What about processing ever growing amounts of transactional data? Stream processing needs to cater for that. You need message expiry and time windows. Is that even possible with a relational backbone such as DBSP?
Oh yes. Kafi Streams offers you the full set of time windows from classical stream processing, including sliding, tumbling, hopping, cumulative and session windows.
Message expiry and time windows in Kafi Streams are native to the engine by design. pydbsp-native. Fully relational. This is the last design decision.
In fact, a time window in Kafi Streams boils down to message expiry plus a simple group by/aggregate operation.
This also means that Kafi Streams lets you define new types of (time) windows effortlessly. Think threshold windows. Or dynamic windows based on a combination of timestamps and other data points. Whatever you do - the garbage collection of Kafi Streams/pydbsp makes room for more transactional data coming in.
And, alas, one more thing. In Kafi Streams, you only define the message expiry once at the beginning of your stream processing pipeline. All the following stateful operators further downstream are automatically cleaned up. All the joins. All the aggregations. Kafi Streams frees you from having to define a GC strategy for each and every stateful operator any longer.
Epilogue
I love streaming. I love all you colleagues from the streaming scene. I love my three wonderful daughters on a completely different level, and my family, and my new family, granted. But I know that this blog is slighly megalomaniac and, at times, could even be considered offensive to readers from the streaming space. I am sorry. But I am convinced that a dose of hyperbole is necessary to help us bounce back.
And rest assured that Kafi Streams is not about replacing Kafka Streams or Flink for the 10% of high throughput use cases where Kafi Streams, as a pure Python/single node library just cannot compete.
Kafi Streams is for the 90% of use cases, and for all those who just couldn’t use stream processing before just because of its sheer complexity. It’s for that long neglected sweet spot, not just of the streaming, but the overall IT and data market.
Today, every me and every you - any developer with a bit of Python expertise - can do complex stream processing. Having all the Python ecosystem at your fingertips. Pandas, Polars, DuckDB. Or Pytorch, LangChain, Pydantic AI, smolagents, LlamaIndex, all the HuggingFace stuff.
For instance, shifting left and saving millions on your Snowflake, Databricks, RedShift or BigQuery bill by substituting batch processing with incremental stream processing *does* become feasible for the first time in history.
Suddenly, building real-time RAG does become easy as cake.
And yes. Of course, if you really want to, you can use Kafi Streams to update your AI agents with the freshest possible data as well. Strongly consistent and always correct.
I think stream processing had to become 10x easier to be able to finally lift off.
Now it is 10x easier.
Now, streaming is going to become mainstream.
-
Kafi Streams 1.0.0 is out here:
Please start trying it out, performance-testing it, battle-testing it, and helping to make it better now :-)
And please use something like Visual Studio Code to view the documentation (after you have cloned the repo). Viewing it on GitHub in your browser does not allow you to run the examples + see the diagrams.