Peter (the author) is a really, really cool guy. We recorded a 3hr 30m podcast[0] with him a month ago. For anyone interested in the Kafka space, performance optimization in Rust and the general "why yet another Kafka", I'd shamelessly recommend the video:
Great link. I've always been drawn to sqlite3 just from a simplicity and operational point of view. And with tools like "make it easy to replcate" Litestream and "make it easy to use" sqlite-utils, it just becomes easier.
And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.
I love the idea of SQLite, but I actually really dislike using it.
I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doing local configurations, and for anything more complicated than a local configuration, I likely have access to Postgres or something. I mean, the executable for Postgres is only ten megs or twenty on Linux, so it's not even that much bigger than SQLite for modern computers.
Any good and honest tansu experience reports out there? Would be nice to understand how “bleeding edge” this actually is, in practice. The idea of a kafka compatible, but trivial to run, system like this is very intriguing!
nchmy 6 hours ago [-]
I wonder how it compares to Redpanda
anticodon 2 hours ago [-]
I've used Redpanda for local development and testing stands. It is super easy to setup in docker, starts really fast and consumes less resources than Java version. Haven't really compared it to anything, but I remember using Java version of Kafka before and it was a resource hog. It is important when you develop on laptop with constrained resources.
enether 1 hours ago [-]
to be fair, Kafka now has a GraalVM docker image[0][1] which was made for local dev/testing, and it has caught up fairly well to these alternatives re: memory and startup time
Quite cool. 7000 records per second is usable for a lot of projects.
One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:
To me it sounds like NATS Jetstream but with Rust. I wonder what the reliability looks like when it is prod ready
nchmy 2 hours ago [-]
Jetstream isn't kafka-compatible, nor does it have pluggable storage of s3, sqlite, Postgres etc...
ktzar 12 hours ago [-]
I didn't know about Tansu and probably would not use it for anything too serious (yet!). Bus as a firm believer of event sourcing and change of paradigm that Kafka brings this is certainly interesting for small projects.
tuananh 8 hours ago [-]
everything is dead. what lives on is their protocol.
same for redis, kafka, ...
brikym 10 hours ago [-]
How does it compare to Redis streams with persistent storage?
tucnak 3 hours ago [-]
This SQLite obsession is getting quite ridiculous. Now they put it in "the Cloud." What a shitshow. I wonder whether they know what SQLite is for... when Cloudflare did it, well, it made sense at least. This new generation of SQLite caro-culting is beyond anything I've ever seen.
shortishly 2 hours ago [-]
Tansu author here. Storage is a pluggable choice of: PostgreSQL, memory, SQLite or S3. There are others in the pipeline (SlateDB, ...).
youngtaff 11 minutes ago [-]
Any chance of a Parquet compatible storage choice?
Rendered at 10:14:24 GMT+0000 (Coordinated Universal Time) with Vercel.
[0] - https://www.youtube.com/watch?v=pJQ7hcsI1Dw
And one of the first patterns I wanted to use was this. Just a read-only event log that's replicated, that is very easy to understand and operate. Kafka is a beast to manage and run. We picked it at my last company -- and it was a mistake, when a simple DB would have sufficed.
https://github.com/simonw/sqlite-utils https://litestream.io/
I think part of my issue is that a lot of uses of it end up having a big global lock on the database file (see: older versions of Emby/Jellyfin) so you can't use it with multiple threads or processes, but I also haven't really ever find a case to use it over other options. I've never really felt the need to do anything like a JOIN or a UNION when doing local configurations, and for anything more complicated than a local configuration, I likely have access to Postgres or something. I mean, the executable for Postgres is only ten megs or twenty on Linux, so it's not even that much bigger than SQLite for modern computers.
https://www.sqlite.org/c3ref/busy_timeout.html
[0] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A... [1] - https://hub.docker.com/r/apache/kafka-native
One note on the backup/migrate, I think you need a shared lock on the database before you copy the database. If you dont, the database can corrupt. SQLite docs have other recommendations too:
https://sqlite.org/backup.html
same for redis, kafka, ...