Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193
I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.
The read replica FUSE layer sounds like a real nice thing to have.
> When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.
Sounds workable!
simonw 25 days ago [-]
This post is like they read my mind and implemented everything I wanted from a new Litestream. So exciting.
thewisenerd 25 days ago [-]
ben, thanks for litestream!
we're using it on production for a write-heavy interal use-case (~12GB compressed) for more than a year now; and it's costing us a couple hundred pennies per month (azure).
excited to try the new changes when they land.
tmpz22 24 days ago [-]
Mind sharing some of your operational choices for hosting/deployment? Which Azure services are you using and what configurations? What kind of throughput are you getting? Any tips regarding migrations? Are you using a dedicated server or VPS?
I'll be doing a similar deployment later this year and enjoy reading on the topic.
thewisenerd 22 days ago [-]
for this particular deployment;
we're only using the blob storage on azure. the deployments are on an on-prem kubernetes cluster with replicas=1 and strategy: recreate.
throughput: not very heavy tbf.. one webhook request every 10-ish seconds; each request leads to about 10-100+ entries added in a table.
migrations: since it's an internal console, we just took a couple hours downtime and did it.
bradgessler 25 days ago [-]
I wish Fly would polish the developer experience on top of SQLite. They're close, but it's missing:
1. A built-in UI and CLI that manages SQLite from a volume. Getting the initial database on a Fly Machine requires more work than it should.
2. `fly console` doesn't work with SQLite because it spins up a separate machine, which isn't connected to the same volume where the SQLite data resides. Instead you have to know to run `fly ssh console —pty`, which effectively SSH's into the machine with the database.
The problem in general with SQLite web apps is they tend to be small apps, so you need a lot of them to make a decent amount of money hosting them.
adenta 25 days ago [-]
Brad, what’s your take on Rails 8 w/ SQLite? Are you gravitating towards it these days over Postgres?
bradgessler 25 days ago [-]
Yep! I just migrated a Fly PG cluster database to SQLite because I over-provisioned DB resources and got tired of dealing with the occasional node crashing.
TBH I wish they had their managed PG cluster running because it would have made it easier to downsize, but I’m happy with SQLite.
I used SQLite for another project that I knew was going to max out at 100 concurrent users and it worked great. The best moment was when a user reported a production error I couldn’t recreate locally, so I downloaded the database and recreated it with the latest production data on my laptop. You couldn’t do that with a high-compliance app, but that’s not most apps.
I’m hesitant to outright say “SQLite and Rails is great”because you have to know your app will run on one node. If you know that then it’s fantastic.
25 days ago [-]
jasonthorsness 25 days ago [-]
What a coincidence, I was just researching Litestream today! I use Sqlite on my VPS and was thinking about adding this.
Am I understanding correctly that I will be able to restore a database to any point-in-time that is while the litestream process is running? Because auto-checkpointing could consume the WAL while it isn't running?
So for an extreme example if the process crashed for an hour between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the information required to restore between 2:00 and 3:00 is lost?
benbjohnson 25 days ago [-]
Litestream saves WAL segments to a given time granularity. By default, it ships off WAL changes every second so you should be able to restore to any given second in your history (within your retention period).
dolmen 24 days ago [-]
Do you have DST handling issues?
I'm asking because switching from winter time to summer time in Europe happened on March 30th with local time jumping from 2:00 to 3:00.
psanford 25 days ago [-]
This looks great! A few years ago I wrote a sqlite vfs for using dynamodb as a backing store[0] called DonutDB. With the recent addition of CAS to S3, I was thinking about making a new version of DonutDB backed by S3. I'm really glad lightstream supports this so I don't have to!
Do you have a reference for this? I assume by CAS you mean content addressable storage? I googled but can't find any AWS docs on this.
xyzzy_plugh 25 days ago [-]
Compare And Swap
gcr 25 days ago [-]
The TL;DR is that Amazon S3 now supports "conditional writes" which are guaranteed to fail if the file was written by some other writer. This is implemented by sending the ETag of an object's expected version alongside the write request.
Litestream now depends on this functionality to handle multiple writers. Think of optimistic locking.
Thanks both! In the contexts I work in CAS always means content-addressable-storage, my mistake.
ignoramous 25 days ago [-]
We have a sneaking suspicion that the robots that write LLM code are going to like SQLite too. We think what coding agents like Phoenix.new want is a way to try out code on live data, screw it up, and then rollback both the code and the state.
Prescient.
Agents would of course work well if they can go back in time to checkpoints and branch from there, exploring solutions parallely as needed.
Anyone who has experience with building workflows (Amazon SWF, Temporal, and the like) knows how difficult it is to maintain determinism in face of retries & re-drives in multi-tier setups (especially, those involving databases).
Replit recently announced their Agent's integration with Neon's time travel feature [0] for exactly the purpose outlined in TFA. Unlike Fly.io though, Replit is built on GCP and other 3p providers like Neon and it is unclear if both GCP & Databricks won't go all Oracle on them.
If you wanted to use litestream to replicate many databases (ideally, one or more per user), which is one of the use cases described here (and elsewhere), how do you tell litestream to add new databases dynamically? The configuration file is static and I haven't found an API to tell it to track a new db at runtime.
mrkurt 25 days ago [-]
I would expect this problem to get solved. It's tricky to detect new sqlites, but not impossible.
In the meantime, it's pretty straightforward to use as a library.
danielblignaut 15 days ago [-]
Have these changes landed yet? I don't see any updates in the documentation website around some of these features. Most notably, I'm interested in read replication and the ability for a read replica to take over becoming the leader (even if it requires a config update and litestream restart) in cases where we aren't using Consul (which is why LiteFS is not a workable solution in our environment).
benbjohnson 15 days ago [-]
The changes are still in progress. The blog post was just about future work that we're working on. However, we don't have plans to do failover with Litestream at the moment.
danielblignaut 15 days ago [-]
Awesome! BTW, it's really interesting work. The main reason I'm avoiding LiteFS is because of the need for Consul. Does LiteFS support Corrosion at all or a method to bypass the need for Consul? I'm digging a bit deeper into it now just in case.
srameshc 25 days ago [-]
I have been following Ben for a long time but I never knew LiteFS was based on his work. I somehow settled eventually for rqlite for self managed distributed.
I don't think they're similar at all. LiteFS uses Consul to elect a leader for a single-write-leader multiple-replica configuration, the same way you'd do with Postgres. rqlite (as I understood it last time I looked) runs Raft directly; it gets quorums for every write.
One isn't better than the other. But LiteFS isn't a "distributed SQLite" in the sense you'd think of with rqlite. It's a system for getting read-only replicas, the same way you've been able to do with log shipping on n-tier databases for decades.
apitman 25 days ago [-]
rqlite also requires you to use special client libraries, whereas litefs is transparent to the program.
rads 25 days ago [-]
What will be required from users of the existing Litestream version to upgrade to the new one? Is it a matter of bumping the version when it comes out or is there more to it?
wg0 24 days ago [-]
> Now that we’ve switched to LTX, this isn’t a problem any more. It should thus be possible to replicate /data/*.db, even if there’s hundreds or thousands of databases in that directory.
That was the show stopper. Now multi tenant with per tenant database whee (in theory) each user can roll back to certain point in time or at least completely download their database and take away for whatever they want to do with it is going to be possible.
wim 24 days ago [-]
Very cool! This is so clever and makes deploying it so simple. I just wasn't able to use it yet because we'd have (many) thousands of SQLite DBs to backup. I quickly hacked something together using fanotify + SQLite's Backup API to have some copies at least, but I'm going to try to switch to Litestream if this amount of files would be supported by the wildcard replication.
JSR_FDED 25 days ago [-]
Will the new litestream work with object stores that don’t provided conditional writes?
Skinney 24 days ago [-]
If I’m deploying a new version of my app, the typical managed solution will spawn a new server instance with that new version, and once a health check has succeeded a couple of times it will reroute trafic to this new instance and kill the old one.
Previously this would be problematic, as the new instance might miss changes made by the old server. Is this fixed by these new changes?
gwking 24 days ago [-]
It seems to me that you need to think about your server as the production database, rather than a web server instance that can be trivially spawned by a management layer.
When I deploy a new version of my python/sqlite web app, I do not replace the whole machine. I just upgrade the python package and restart the systemd service.
If I wanted to reduce downtime I could probably figure out a transition using SO_REUSEPORT. During the transition the old and new processes would be using the db concurrently, so the app would have to respect that case. If part of the upgrade requires a db schema change then I’m not sure how you could avoid some downtime. But I don’t know if it is possible with traditional dbs either.
maxmcd 24 days ago [-]
I don't think this is trivially fixed since you can still only have one writer that is holding the lease.
Your new service will come up, but it won't be able to get the write lease until the previous server shuts down. Now you have tools to detect this, stop one writer, and start the other, but the service will likely have to experience some kind of requests queueing or downtime.
malkia 25 days ago [-]
So fossil (which is built on top of sqlite) + this = SCM?
hiAndrewQuinn 25 days ago [-]
I'm still waiting for someone to make the GitHub for fossil. Bonus points if it's called Paleontology
diggan 24 days ago [-]
Is that needed to make Fossil useful? Since it's basically GitHub but all in Git, it's all works P2P without the need of a centralized service.
I guess for discovery it kind of makes sense, but wouldn't really be "GitHub for Fossil" but more like a search engine/portal.
mdaniel 23 days ago [-]
Also, as far as I know it offers zero CI integrations, which is (IMHO) table stakes for an organization-level platform
n.b. while looking up the modern state of affairs, I found this gem https://fossil-scm.org/home/doc/trunk/www/qandc.wiki#:~:text... which made me chuckle but is also, let's be real, a barrier to adoption. Yes, I am aware of the rabid army of Bugzilla fans, but I'd straight up quit before working with the garbage that is Bugzilla
hiAndrewQuinn 24 days ago [-]
Discovery and market dominance seems like basically the only thing standing in fossil's way from mass adoption, though.
mdaniel 23 days ago [-]
Depends on how one thinks of "mass," since (a) there is an unholy amount of tooling out there which only speaks the GitHub API (not even mentioning GitLab, Gitea, ...) and (b) AIUI the Fossil folks abhor history mutation which is great for them and really not great for a subset of git users. I get the lightweight impression that they took Mercurial's "please don't" and turned it up to 11
neom 25 days ago [-]
Is Litestream on a path to subsume LiteFS's capabilities? Re: PITR, would this be used to facilitate automated A/B testing of AI-generated code changes against live data subsets? I can imagine a lot of cool stuff in that direction. This is really cool Ben!
j0e1 25 days ago [-]
This is exciting! Especially glad that Litestream is still maintained. Is there a use-case for Litestream for more than backup? I am a fan of offline-first but it would be cool to have a way to synchronize on-device SQLite instances to a single central instance.
benbjohnson 25 days ago [-]
Backups & read replicas are the primary use cases. If you're interested in local-first, you can check out projects like cr-sqlite[1].
Can this be done with only Litestream, or is LiteVFS still in development? I looked into this last year but was put off by LiteFS's stated write performance penalty due to FUSE [1]; it's still marked as WIP [2] and hasn't seen updates for over a year.
Awesome stuff, this resolves my #1 feature request of being able to replicate an entire directory of SQLite *.db's from a single Litestream process - happy it's finally here.
Should make replicating Multi tenant per-user SQLite databases a lot more appealing.
oulipo 24 days ago [-]
I still don't really understand the real "advantages" of such an architecture, over, say, a centralized Postgres server, which can process just as much data no?
tptacek 24 days ago [-]
The network round trips of queries to a database server add up, so much that they influence query design (though we don't much think about that anymore, because n-tier database designs are so prevalent that everyone writes queries that way). The advantage to SQLite is that queries are incredibly fast.
oulipo 23 days ago [-]
I see! Thanks.
So it's often "presented" as "a local database which is replicated by streaming", but perhaps it would be more natural to view it as a kind of "centralized database, but with local caches that are kept near your server code, and sync'd"
I understand it's the same, but it makes the intent clearer: the intent is more to have a local cache to read/update (which is then sync'd), or at least it seems clearer to me presented that way :)
bambax 25 days ago [-]
Very cool!
There may be a typo here:
> The most straightforward way around this problem is to make sure only one instance of Litestream can replication to a given destination.
Can replicate? Or can do replications?
wiradikusuma 25 days ago [-]
For Fly.io employees here: Can I finally replace my Postgre with this a'la Cloudflare D1 (which is also Sqlite based)?
rawkode 25 days ago [-]
Amazing to see and hear about the progress. Always a pleasure when Ben works on something and shares it. Keep it up!
Nelkins 24 days ago [-]
Does anybody have a list of which S3-compatible object storage providers support Compare-And-Swap?
nodesocket 25 days ago [-]
Is there a migration guide from stable to the branch 0.5? I’m running Litestream as a Docker sidecar alongside my Python app container and it’s been great and a nice comfort knowing my SQLite db is backed up to S3.
oliwary 25 days ago [-]
Fantastic to see it's getting updated! I am a big fan of litestream, have been using it for a while together with pocketbase. It's like a cheat code for a cheap, reliable and safe backend.
fra 25 days ago [-]
Litestream has seen very little development lately and I was worried it was dead. Very glad to see Ben Johnson is continuing to push the project forward with some exciting new plans.
noroot 24 days ago [-]
I love the idea of litestream and litefs and do use it for some smaller projects, but have also been worried it was abandoned. The line is quite thin between "done" and "not maintained".
There clearly still is some untapped potential in this space, so I am glad benbjohnson is exploring and developing these solutions.
Great that the new release will offer the ability to replicate multiple database files.
> Modern object stores like S3 and Tigris solve this problem for us: they now offer conditional write support
I hope this won't be a hard requirement, since some S3 compatible storage do not have this feature (yet). I also do use the SFTP storage option currently.
avtar 25 days ago [-]
That's the conclusion I reached a couple months ago when I was evaluating similar tools. The last Litestream release was issued in 2023 and the official Docker image is over a year old. In the end it seemed like a safer bet to accept some inconvenient tradeoffs and just create backups more frequently.
tptacek 25 days ago [-]
Ben also wrote BoltDB, which was untouched (archived, even) for years despite a thriving community. Sometimes things are just done!
Zekio 24 days ago [-]
seems to have active commits from 2 weeks ago, just not on the main branch
ChocolateGod 24 days ago [-]
> It will be able to fetch and cache pages directly from S3-compatible object storage.
Does this mean your SQLite database size is no longer restricted by your local disk capacity?
bdcravens 24 days ago [-]
Looking at the LiteVFS repo, it appears so, with some limitations.
"LiteVFS is a Virtual Filesystem extension for SQLite that uses LiteFS Cloud as a backing store."
Limitations
- Databases with journal_mode=wal cannot be modified via LiteVFS (but can be read)
- Databases with auto-vacuum cannon be opened via LiteVFS at all
> Databases with journal_mode=wal cannot be modified via LiteVFS (but can be read)
Without modifying SQLite (what the Turso guys did), the WAL index is hard (but not impossible) to share across a network boundary. I'm guessing that's the why here. There's a hack that I'm pretty confident works, but I'm not sure how it behaves under latency (sure enough that I use it for Windows, and it hasn't caused issues running mptest thousands of times in CI over months).
A SQLite database that supports read-replicas and can offload cold data to object storage would be super useful.
dastbe 25 days ago [-]
asking since ben does take a look here...
will revamped litestream have a solution for ACKing only when transactions have durably committed to storage?
caleblloyd 25 days ago [-]
Is the backend pluggable? Could it be configured to write to any key value store with support for optimistic concurrency control?
benbjohnson 25 days ago [-]
We don't support plug-ins at the moment but there's several backends at the moment (S3, Azure Blob Storage, Google Cloud Storage, SFTP, etc)
yowmamasita 25 days ago [-]
tangent: in modern SQLite, are writes still serialized? That's my main concern when choosing a tech stack for an app that might have thousands of writes happening on peak periods
simonw 25 days ago [-]
Yes they are, but if you benchmark thousands of writes a second you'll likely find that SQLite does just fine.
You might start running into problems at tens or hundreds of thousands of writes a second, though even then you may be OK on the right hardware.
Is there anything like Livestream that can be just pip installed?
nico 25 days ago [-]
Very cool idea, I wonder if that works better than their Postgres instances
Recently, I deployed a little side project using a small postgres vm on fly.io
After a couple of days, and only having about 500kb of data stored in that db, the postgres vm went into an unrecoverable fail loop, saying it ran out of memory, restarting, then immediately running out of memory again, so on and so forth
It took about 3-4hrs to recover the data jumping through a lot of hoops to be able to access the data, copy it to another volume and finally download it
I would've reached for support, but it seems like the only option available is just posting on their forum. I saw a couple of related posts, all with unsatisfactory answers unfortunately
To be fair, it was incredibly easy to get up and running with them. On the other hand, almost all the time I saved by that quick start, was wasted recovering the failing db, all the while my site was down
Ironically, I originally developed the project using sqlite, but then switched to postgres to deploy
tptacek 25 days ago [-]
This post has nothing to do with Fly.io's platform offerings. Litestream is completely uncoupled from Fly.io. Ben started it before he got here.
sosodev 25 days ago [-]
Clearly it does have something to do with fly.io considering fly is and has been pushing for litefs/stream as the ideal database solution for fly users. It seems reasonable that readers would compare it to other fly offerings.
tptacek 25 days ago [-]
We have... never done that? Like ever? LiteFS is interesting for some read-heavy use cases, especially for people who are doing especially edge-deployed things, but most people who use databases here use Postgres. We actually had a managed LiteFS product --- LiteFS Cloud --- and we sunset it, like over a year ago. We have a large team working on Managed Postgres. We do not have a big SQLite team.
People sometimes have a hard time with the idea that we write about things because they are interesting to us, and for no other reason. That's also 60-70% of why Ben does what he does on Litestream.
sosodev 25 days ago [-]
I’m sorry. I think that I, and probably others, have misinterpreted it. Between Ben’s writings on the fly blog and litefs cloud it seemed like that was the case. I didn’t realize it had been discontinued.
tptacek 25 days ago [-]
Neither LiteFS nor Litestream (obviously) have been discontinued. They're both open source projects, and were both carefully designed not to depend on Fly.io to work.
mixmastamyk 24 days ago [-]
What happened to the supabase integration? Seems to have fizzled as well.
yellow_lead 25 days ago [-]
It's strange to me that they still haven't offered a managed Postgres product. Other providers like Render or even Heroku seem to have realized that this is a core part of PaaS that customers want. Instead they focused on GPUs and LiteStream. When I evaluated different PaaS for the startup I work at, I had to go with Render. I couldn't even give Fly.io a try since I knew we needed Postgres.
tptacek 25 days ago [-]
We're rolling out Managed Postgres, very slowly.
yellow_lead 25 days ago [-]
Looking forward to it!
nico 25 days ago [-]
It think they are in beta. I wished they had a managed Redis though
For Postgres I ended up going with Neon (neon.tech), very happy with them so far. Super easy to setup and get up and running, also love being able to just easily see the data from their web interface
Try Railway - nothing but good experiences with these dudes. Fairly priced and great dev UX.
internet_points 24 days ago [-]
is that an alternative to supabase?
sergiotapia 24 days ago [-]
it's an alternative between render/railway/northflank/fly. part of the new-gen paas.
apitman 25 days ago [-]
You might try paying the $29/mo. I've found the email support to be great.
norman784 24 days ago [-]
That's one of the reasons I don't use their Postgres instances and instead go with a service with a dedicated database service, but for deploying backend apps it's pretty good.
teamcampapp 24 days ago [-]
[dead]
25 days ago [-]
gavinray 25 days ago [-]
Just a heads-up, the link in the "Litestream is fully open source" callout is malformed and leads to:
For something rather new there seems to be too many choices already. Please pick a strategy under one name, good defaults, and a couple of config options.
25 days ago [-]
gitroom 24 days ago [-]
[dead]
tiffanyh 25 days ago [-]
[flagged]
vhodges 25 days ago [-]
I think that might be a structure/css issue. On Desktop it's to the left of the article (but I shrunk the window and indeed it puts it below the article).
25 days ago [-]
xmorse 24 days ago [-]
[flagged]
24 days ago [-]
internet_points 24 days ago [-]
why?
xmorse 24 days ago [-]
The landing page shows all logos of small companies, including one that is migrating away from them (Turso)
I think fly is focused on small/medium companies, bigger companies will have the manpower to use AWS/Azure directly.
mwcampbell 24 days ago [-]
Do you have a citation on Turso moving away from Fly? I'd be particularly curious about anything Turso has written on _why_ they're moving away.
tptacek 24 days ago [-]
What you're actually observing there is that we don't have a "marketing" team and we just don't care that much about the front page. As long as it's describing what we do roughly accurately, we don't think about the front page. It's not like a huge part of our funnel.
You could say this is cope, but we've been exactly the same way about this page since 2020. For a stretch of over a year it didn't even accurately describe us --- people who learned about us from the front page thought we were an edge compute company still.
xmorse 23 days ago [-]
I love fly but you really have to work harder on these things, the landing page is not something you need a marketing team for
tptacek 23 days ago [-]
No, we don't have to work harder on these things. "I love fly but I think they're going bankrupt soon because I did kremlinology on their front page" is not actually a user cohort I care much about.
I'm really tired of trying to make decisions about what we say or don't say based on these weird messaging things, so I think we're just going to stop trying, and go back to talking like we did in 2021.
fasdfdsa 25 days ago [-]
[flagged]
Rendered at 19:38:32 GMT+0000 (Coordinated Universal Time) with Vercel.
Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193
I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.
The read replica FUSE layer sounds like a real nice thing to have.
edit: Ah, it works like this: https://github.com/benbjohnson/litestream/pull/617
> When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.
Sounds workable!
we're using it on production for a write-heavy interal use-case (~12GB compressed) for more than a year now; and it's costing us a couple hundred pennies per month (azure).
excited to try the new changes when they land.
I'll be doing a similar deployment later this year and enjoy reading on the topic.
we're only using the blob storage on azure. the deployments are on an on-prem kubernetes cluster with replicas=1 and strategy: recreate.
throughput: not very heavy tbf.. one webhook request every 10-ish seconds; each request leads to about 10-100+ entries added in a table.
migrations: since it's an internal console, we just took a couple hours downtime and did it.
1. A built-in UI and CLI that manages SQLite from a volume. Getting the initial database on a Fly Machine requires more work than it should.
2. `fly console` doesn't work with SQLite because it spins up a separate machine, which isn't connected to the same volume where the SQLite data resides. Instead you have to know to run `fly ssh console —pty`, which effectively SSH's into the machine with the database.
The problem in general with SQLite web apps is they tend to be small apps, so you need a lot of them to make a decent amount of money hosting them.
TBH I wish they had their managed PG cluster running because it would have made it easier to downsize, but I’m happy with SQLite.
I used SQLite for another project that I knew was going to max out at 100 concurrent users and it worked great. The best moment was when a user reported a production error I couldn’t recreate locally, so I downloaded the database and recreated it with the latest production data on my laptop. You couldn’t do that with a high-compliance app, but that’s not most apps.
I’m hesitant to outright say “SQLite and Rails is great”because you have to know your app will run on one node. If you know that then it’s fantastic.
Am I understanding correctly that I will be able to restore a database to any point-in-time that is while the litestream process is running? Because auto-checkpointing could consume the WAL while it isn't running?
So for an extreme example if the process crashed for an hour between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the information required to restore between 2:00 and 3:00 is lost?
I'm asking because switching from winter time to summer time in Europe happened on March 30th with local time jumping from 2:00 to 3:00.
I can't wait to try this out.
[0]: https://github.com/psanford/donutdb
Do you have a reference for this? I assume by CAS you mean content addressable storage? I googled but can't find any AWS docs on this.
Litestream now depends on this functionality to handle multiple writers. Think of optimistic locking.
https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3...
Agents would of course work well if they can go back in time to checkpoints and branch from there, exploring solutions parallely as needed.
Anyone who has experience with building workflows (Amazon SWF, Temporal, and the like) knows how difficult it is to maintain determinism in face of retries & re-drives in multi-tier setups (especially, those involving databases).
Replit recently announced their Agent's integration with Neon's time travel feature [0] for exactly the purpose outlined in TFA. Unlike Fly.io though, Replit is built on GCP and other 3p providers like Neon and it is unclear if both GCP & Databricks won't go all Oracle on them.
[0] https://blog.replit.com/safe-vibe-coding
In the meantime, it's pretty straightforward to use as a library.
https://github.com/rqlite/rqlite
https://youtu.be/8XbxQ1Epi5w?si=puJFLKoVs3OeYrhS
One isn't better than the other. But LiteFS isn't a "distributed SQLite" in the sense you'd think of with rqlite. It's a system for getting read-only replicas, the same way you've been able to do with log shipping on n-tier databases for decades.
That was the show stopper. Now multi tenant with per tenant database whee (in theory) each user can roll back to certain point in time or at least completely download their database and take away for whatever they want to do with it is going to be possible.
Previously this would be problematic, as the new instance might miss changes made by the old server. Is this fixed by these new changes?
When I deploy a new version of my python/sqlite web app, I do not replace the whole machine. I just upgrade the python package and restart the systemd service.
If I wanted to reduce downtime I could probably figure out a transition using SO_REUSEPORT. During the transition the old and new processes would be using the db concurrently, so the app would have to respect that case. If part of the upgrade requires a db schema change then I’m not sure how you could avoid some downtime. But I don’t know if it is possible with traditional dbs either.
Your new service will come up, but it won't be able to get the write lease until the previous server shuts down. Now you have tools to detect this, stop one writer, and start the other, but the service will likely have to experience some kind of requests queueing or downtime.
I guess for discovery it kind of makes sense, but wouldn't really be "GitHub for Fossil" but more like a search engine/portal.
n.b. while looking up the modern state of affairs, I found this gem https://fossil-scm.org/home/doc/trunk/www/qandc.wiki#:~:text... which made me chuckle but is also, let's be real, a barrier to adoption. Yes, I am aware of the rabid army of Bugzilla fans, but I'd straight up quit before working with the garbage that is Bugzilla
[1]: https://github.com/vlcn-io/cr-sqlite
Can this be done with only Litestream, or is LiteVFS still in development? I looked into this last year but was put off by LiteFS's stated write performance penalty due to FUSE [1]; it's still marked as WIP [2] and hasn't seen updates for over a year.
[1] https://fly.io/docs/litefs/faq/#what-are-the-tradeoffs-of-us...
[2] https://github.com/superfly/litevfs
Should make replicating Multi tenant per-user SQLite databases a lot more appealing.
So it's often "presented" as "a local database which is replicated by streaming", but perhaps it would be more natural to view it as a kind of "centralized database, but with local caches that are kept near your server code, and sync'd"
I understand it's the same, but it makes the intent clearer: the intent is more to have a local cache to read/update (which is then sync'd), or at least it seems clearer to me presented that way :)
There may be a typo here:
> The most straightforward way around this problem is to make sure only one instance of Litestream can replication to a given destination.
Can replicate? Or can do replications?
There clearly still is some untapped potential in this space, so I am glad benbjohnson is exploring and developing these solutions.
Great that the new release will offer the ability to replicate multiple database files.
> Modern object stores like S3 and Tigris solve this problem for us: they now offer conditional write support
I hope this won't be a hard requirement, since some S3 compatible storage do not have this feature (yet). I also do use the SFTP storage option currently.
Does this mean your SQLite database size is no longer restricted by your local disk capacity?
"LiteVFS is a Virtual Filesystem extension for SQLite that uses LiteFS Cloud as a backing store."
Limitations
- Databases with journal_mode=wal cannot be modified via LiteVFS (but can be read)
- Databases with auto-vacuum cannon be opened via LiteVFS at all
- VACUUM is not supported
https://github.com/superfly/litevfs
Without modifying SQLite (what the Turso guys did), the WAL index is hard (but not impossible) to share across a network boundary. I'm guessing that's the why here. There's a hack that I'm pretty confident works, but I'm not sure how it behaves under latency (sure enough that I use it for Windows, and it hasn't caused issues running mptest thousands of times in CI over months).
Leaving it here, maybe Ben's interested.
https://github.com/ncruces/go-sqlite3/blob/c780ef16e277274e7...
will revamped litestream have a solution for ACKing only when transactions have durably committed to storage?
You might start running into problems at tens or hundreds of thousands of writes a second, though even then you may be OK on the right hardware.
Recently, I deployed a little side project using a small postgres vm on fly.io After a couple of days, and only having about 500kb of data stored in that db, the postgres vm went into an unrecoverable fail loop, saying it ran out of memory, restarting, then immediately running out of memory again, so on and so forth
It took about 3-4hrs to recover the data jumping through a lot of hoops to be able to access the data, copy it to another volume and finally download it
I would've reached for support, but it seems like the only option available is just posting on their forum. I saw a couple of related posts, all with unsatisfactory answers unfortunately
To be fair, it was incredibly easy to get up and running with them. On the other hand, almost all the time I saved by that quick start, was wasted recovering the failing db, all the while my site was down
Ironically, I originally developed the project using sqlite, but then switched to postgres to deploy
People sometimes have a hard time with the idea that we write about things because they are interesting to us, and for no other reason. That's also 60-70% of why Ben does what he does on Litestream.
For Postgres I ended up going with Neon (neon.tech), very happy with them so far. Super easy to setup and get up and running, also love being able to just easily see the data from their web interface
https://http//litestream.io/
https://fly.io/
You could say this is cope, but we've been exactly the same way about this page since 2020. For a stretch of over a year it didn't even accurately describe us --- people who learned about us from the front page thought we were an edge compute company still.
I'm really tired of trying to make decisions about what we say or don't say based on these weird messaging things, so I think we're just going to stop trying, and go back to talking like we did in 2021.