So much that we presume in the modern cloud wasn't a given when Apache Kafka was first released in 2011.
kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights.
As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020.
Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on.
Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda.
It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better price-performance. Faster writes. Lower p99 latencies. It's just... better.
Now, I am biased. I work at Redpanda now. But I've been a big fan of Kafka since 2015. I am still bullish on data streaming. I just think that Apache Kafka, as a Java-based platform, needs some serious rearchitecture,
Even Confluent doesn't use vanilla Kafka. They rewrote their own engine, Kora. They claim it is 10x faster. Or 30x faster. Depending on what you're measuring.
There have been annual layoffs at RedHat since 2023. This year they just laid off more. The layoffs this year are expected to be "a low single digit percentage of our global workforce." Which will likely include hundreds of folks at Red Hat.
There haven't been layoffs at Red Hat after 2023, whereas according to your statement there should have been two more rounds. The layoffs from your articles are at IBM, and did not affect Red Hat.
I have thought quite a bit today about the news from Confluent and IBM. I have friends and colleagues at both companies. When I was an undergrad at Carnegie Mellon University in the 1980s I used to wear a big brown and tan IBM button that said "THINK."
And here is a picture of Ben Lorica 罗瑞卡 interviewing Jay Kreps and other industry leaders at The Hive back on the evening of 25 February 2015. I believe they were talking about strategies for implementing Lambda Architecture.
All of which is to say: I have been a big fan of both companies for a long, long time. While today I am at employed at Redpanda Data, a direct competitor of Confluent, I hope to set aside any "team"-based bias to provide a sober and honest appraisal.
First, IBM has been shrinking. They were at 345,000 employees as of their 2020 Annual Report. But the COVID-19 pandemic was only one of many setbacks the company faced when Arvind Krishna took the helm as CEO. By December 2024 the employee base shrank to 270,000 — a drop of nearly 22%.
IBM revenue in 2020: $73.6B.
IBM revenue in 2024: $62.75B — a less-precipitous drop of 15%.
Revenue per employee over that period rose from $213k to $232k.
Confluent on its own? $400k.
And to compare: Amazon earns $580k per employee. Microsoft generates over $1M per. Nvidia? $4M-$5M.
And now, in November, they announced thousands of more layoffs. No one seems safe, regardless of job title. Those cut include positions in "artificial intelligence, marketing, software engineering and cloud technology."
Next, IBM has had a mixed record as a steward of acquisitions. Red Hat has doubled in revenues since their 2019 acquisition. For a while its headcount continued to grow, as much as 19,000 by 2023. But then it was forced into layoffs by parent IBM in April of that year, and then each year since, even while it remains one of the highest margin businesses in their portfolio.
SoftLayer — "IBM Cloud Classic" — also suffered significant layoffs in early 2025, with offshoring sending jobs to India.
DataStax had layoffs in 2023-2024, even before its acquisition was announced. Maybe they were "trimming the fat" to get into a shape to be acquired.
As a person with a long career in marketing, I know that many of the first roles to be jettisoned at a newly-acquired company tend to be in go-to-market organizations. Sales, Marketing, Developer Relations, Documentation, Training, Community, Customer Service. These tend to be seen as "nice to haves" by upper management. But their loss guts organizations and hollows out user-facing teams and open source communities.
My hope is that Confluent is spared as much of the pain and turmoil as possible. That, like Red Hat, it is run autonomously as much as possible.
A note on your "IBM has been shrinking" take: IBM spun out Kyndryl in 2021, including 90,000 employees [0]. That leaves the remaining company with 255,000 employees after the spin-off in 2021, which means it has actually grown by 15,000 people to reach the 270,000 in 2024.
The spin-off also explains the drop in revenue. In 2021, the first results reported excluding Kyndryl, IBM had revenues of $57.4B [1].
Since 2021, revenue grew by ~9% to reach $62.75B in 2024.
Fair. A good callout. And maybe the right move. However, a healthy IBM would not have needed to calve off its entire Global Technology Services business.
You can have DeepWiki literally scan the source code and tell you:
> 2. Delayed Sync Mode (Default)
> In the default mode, writes are batched and marked with needSync = true for later synchronization filestore.go:7093-7097 . The actual sync happens during the next syncBlocks() execution.
However, if you read DeepWiki's conclusion, it is far more optimistic than what Aphyr uncovered in real-world testing.
> Durability Guarantees
> Even with delayed fsyncs, NATS provides protection against data loss through:
> 1. Write-Ahead Logging: Messages are written to log files before being acknowledged
> 2. Periodic Sync: The sync timer ensures data is eventually flushed to disk
> 3. State Snapshots: Full state is periodically written to index.db files filestore.go:9834-9850
> 4. Error Handling: If sync operations fail, NATS attempts to rebuild state from existing data filestore.go:7066-7072"
Also, one could argue French itself is an agglomeration of Vulgar Latin (87%) as well as its own Frankish Germanic roots (10%), and a few of Gaulish and Breton Celtic origin.
It is not straightforward to define a metric like this. What counts as an "English word" ? There are books full of the scientific names of plant and animal species, which usually come from Latin; do these count as "English words" ? What's the cutoff?
IMO, a much better metric is frequency-weighted; that is, taking some corpus of real English and counting the words in it, rather than weighting "every English word" with the value 1.
If you do this frequency-weighted analysis, Old English is far ahead of French and Latin combined (especially in colloquial speech; they're closer in formal writing).
> • Latin (including modern scientific and technical Latin): 28.24%;
This skews the numbers because there are thousands (probably tens of thousands) of scientific terms, names of species, etc., that probably shouldn't be counted here if you're looking at the origin of the language and that is because we continue to use Latin for new scientific terms or names of species today. In other words, they're not Latin words that have been absorbed into English, they're just Latin words period.
> 25% old Norse? That is almost certainly not correct.
But in this case, it is like saying "You don't need a fuel truck. You can transport 9,000 gallons of gasoline between cities by gathering 9,000 1-gallon milk jugs and filling each, then getting 4,500 volunteers to each carry 2 gallons and walk the entire distance on foot."
In this case, you do just need a single fuel truck. That's what it was built for. Avoiding using a design-for-purpose tool to achieve the same result actually is wasteful. You don't need 288 cores to achieve 243,000 messages/second. You can do that kind of throughput with a Kafka-compatible service on a laptop.
I'll push the metaphor a bit: I think the point is that if you have a fleet of vehicles you want to fuel, go ahead and get a fuel truck and bite off on that expense. However, if you only have 1 or 2, a couple of jerry cans you probably already have + a pickup truck is probably sufficient.
Getting a 288-core machine might be easier than setting up Kafka; I'm guessing that it would be a couple of weeks of work to learn enough to install Kafka the first time. Installing Postgres is trivial.
"Lots of the team knows Postgres really well, nobody knows Kafka at all yet" is also an underrated factor in making choices. "Kafka was the ideal technical choice but we screwed up the implementation through well-intentioned inexperience" being an all too plausible outcome.
Indeed, I've seen this happen first hand where there was really only one guy who really "knew" Kafka, and it was too big of a job for just him. In that case it was fine until he left the company, and then it became a massive albatross and a major pain point. In another case, the eng team didn't really have anyone who really "knew" Kafka but used a managed service thinking it would be fine. It was until it wasn't, and switching away is not a light lift, nor is mass educating the dev team.
Kafka et al definitely have their place, but I think most people would be much better off reaching for a simpler queue system (or for some things, just using Postgres) unless you really need the advanced features.
Postgres is the solution in question of the article because I simply assume the majority of companies will start with Postgres as their first piece of infra. And it is often the case. If not - MySQL, SQLite, whatever. Just optimize for the thing you know, and see if it can handle your use case (often you'll be surprised)
The only thing that might take "weeks" is procrastination. Presuming absolutely no background other than general data engineering, a decent beginner online course in Kafka (or Redpanda) will run about 1-2 hours.
Right, I was talking about installing Kafka, not installing Redpanda. Redpanda may be perfectly fine software, but bringing it up in that context is a bit apples-and-oranges since it's not open-source: https://news.ycombinator.com/item?id=45748426
Just use Strimzi if you're in a K8s world (disclosure used to work on Strimzi for RH, but I still think it's far better than Helm charts or fully self-managed, and far cheaper than fully managed).
Even though I'm a few years on from Red Hat, I still really recommend Strimzi. I think the best way to describe it is "a sorta managed Kafka". It'll make things that are hard in self-managed Kafka (like rolling upgrades) easy as.
reply