...as for first-class schemas - yes, that's another point. But at least Confluent does focus a lot on extensions of Schema Registry nowadays, albeit only in their proprietary CCloud/CPlatform offerings.
As for microbatching - I don't care whether you use that or streaming - that's for me an implementation issue. If you use some form of diskless Kafka, you don't have to care. The good thing to using the stream/log as the underlying logical primitive is still that you get the "Flip the ownership" benefit + you can always, if you need, gear up to real-time. And you have a uniform underlying data integration primitive.
Woa thanks a lot Stanislav - also for the pointers! When I look at [1] - the most important reason of those why Kafka came into existence in the first place is the first: "Flip the ownership". Aka Data Mesh in fact. That's what you immediately get into your organization once you adopt Kafka at large scale. Not just for these few real-time use cases, but for everything. But you *can* also build all kinds of real-time use cases if you need to. I think that the focus of Confluent should be firmly on this aspect. Not latency. Not throughput. But "Flip the ownership".
Plus making it easier to actually execute the flip.
1. Abstractions - wdyt of kSQL and now FlinkSQL? They're meant to deal with that - but evidently didn't (or aren't) panning out.
2. Connect - spot on. I was also mulling over an old blog post and came to the conclusion Data Integration is nowhere near as good as it needs to be with Kafka. Connect and Schemas (both hand in hand IMO) did not get developed enough. [1]
3. Schemas. Have you thought about how difficult it is to evolve schemas with stream processing? How can a stream processing system (Kafka) not support first-class schemas?
I've heard from Kafka Streams engineers that the system was never super-well integrated with Schema Registry, and that the two teams were sort of siloed leading to gaps in certain design choices (or, at least, outcomes) like schema evolution.
4. Is it worth it?
I really want to challenge the need of real time streaming (<1000ms) vs something more micro-batchy (60s). WarpStream and the Diskless Kafka movement is beginning to prove that we probably extremely over-estimated the need for low latency for a lot of use cases [3]
Micro-batching is way, way simpler. It doesn't require you to learn a new paradigm. It doesn't require you to build a new super-complex system (that currently has leaky abstractions).
I think stream processing will remain niche simply because the effort-reward ratio isn't there for a majority of organizations to adopt. And even if vendors reduced the effort required to adopt the technology (eg eliminate leaky abstractions as you say) -- would the reward justify the migration costs and new paradigm learning?
In my blunt opinion, the Stream Processing industry has largely been a result of large-scale nerd sniping[4] - an industry that asks "can we" more often rather than "should we". Without a shadow of a doubt, there are use cases that cannot work without stream processing - but history so far has proven these are pretty niche.
This is somewhat similar to the Big Data industry, where a lot of overengineering has gone into distributed systems necessary to handle really large scale workloads, whereas most organizations have small data.[5] It explains the rise of DuckDB, SQLite and the like.
One theory I have is that both things are remnants of the ZIRP environment, where organizations were keen to spend on the new and shiny promises (data is the new oil) without being too critical or practical about it.
...as for first-class schemas - yes, that's another point. But at least Confluent does focus a lot on extensions of Schema Registry nowadays, albeit only in their proprietary CCloud/CPlatform offerings.
As for microbatching - I don't care whether you use that or streaming - that's for me an implementation issue. If you use some form of diskless Kafka, you don't have to care. The good thing to using the stream/log as the underlying logical primitive is still that you get the "Flip the ownership" benefit + you can always, if you need, gear up to real-time. And you have a uniform underlying data integration primitive.
Spark Structured Streaming + Diskless Kafka, interesting rabbit hole potentially
Woa thanks a lot Stanislav - also for the pointers! When I look at [1] - the most important reason of those why Kafka came into existence in the first place is the first: "Flip the ownership". Aka Data Mesh in fact. That's what you immediately get into your organization once you adopt Kafka at large scale. Not just for these few real-time use cases, but for everything. But you *can* also build all kinds of real-time use cases if you need to. I think that the focus of Confluent should be firmly on this aspect. Not latency. Not throughput. But "Flip the ownership".
Plus making it easier to actually execute the flip.
Great post!
1. Abstractions - wdyt of kSQL and now FlinkSQL? They're meant to deal with that - but evidently didn't (or aren't) panning out.
2. Connect - spot on. I was also mulling over an old blog post and came to the conclusion Data Integration is nowhere near as good as it needs to be with Kafka. Connect and Schemas (both hand in hand IMO) did not get developed enough. [1]
3. Schemas. Have you thought about how difficult it is to evolve schemas with stream processing? How can a stream processing system (Kafka) not support first-class schemas?
I've heard from Kafka Streams engineers that the system was never super-well integrated with Schema Registry, and that the two teams were sort of siloed leading to gaps in certain design choices (or, at least, outcomes) like schema evolution.
4. Is it worth it?
I really want to challenge the need of real time streaming (<1000ms) vs something more micro-batchy (60s). WarpStream and the Diskless Kafka movement is beginning to prove that we probably extremely over-estimated the need for low latency for a lot of use cases [3]
Micro-batching is way, way simpler. It doesn't require you to learn a new paradigm. It doesn't require you to build a new super-complex system (that currently has leaky abstractions).
I think stream processing will remain niche simply because the effort-reward ratio isn't there for a majority of organizations to adopt. And even if vendors reduced the effort required to adopt the technology (eg eliminate leaky abstractions as you say) -- would the reward justify the migration costs and new paradigm learning?
In my blunt opinion, the Stream Processing industry has largely been a result of large-scale nerd sniping[4] - an industry that asks "can we" more often rather than "should we". Without a shadow of a doubt, there are use cases that cannot work without stream processing - but history so far has proven these are pretty niche.
This is somewhat similar to the Big Data industry, where a lot of overengineering has gone into distributed systems necessary to handle really large scale workloads, whereas most organizations have small data.[5] It explains the rise of DuckDB, SQLite and the like.
One theory I have is that both things are remnants of the ZIRP environment, where organizations were keen to spend on the new and shiny promises (data is the new oil) without being too critical or practical about it.
[1] - https://www.reddit.com/r/apachekafka/comments/1mjz2xk/did_we_forget_the_primary_use_case_for_kafka/
[2] - https://www.reddit.com/r/apachekafka/comments/1h80if5/why_doesnt_kafka_have_firstclass_schema_support/
[3] - https://www.linkedin.com/posts/stanislavkozlovski_an-inconvenient-truth-a-lot-of-kafka-use-activity-7333110731833274368-KJ_o
[4] - https://xkcd.com/356/
[5] - https://motherduck.com/blog/small-data-manifesto/