> The technical fix was embarrassingly simple: stop pushing to main every ten minutes.
Wait, you push straight to main?
> We added a rule — batch related changes, avoid rapid-fire pushes. It's in our CLAUDE.md (the governance file that all our AI agents follow):
> Avoid rapid-fire pushes to main — 11 pushes in 2h caused overlapping Kamal deploys with concurrent SQLite access.
Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy?
chasil 1 hours ago [-]
This is the actual problem:
"Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open."
WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers.
I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over.
Pausing requests then running two sqlites momentarily probably won’t prevent corruption. It might make it less likely and harder to catch in testing.
The easiest approach is to kill sqlite, then start the new one. I’d use a unix lockfile as a last-resort mechanism (assuming the container environment doesn’t somehow break those).
simonw 27 minutes ago [-]
I'm saying you pause requests, shut down one of the SQLite containers, start up the other one and un-pause.
Retr0id 43 minutes ago [-]
> I think you're exactly right about the WAL shared memory not crossing the container boundary.
I don't, fwiw (so long as all containers are bind mounting the same underlying fs).
The containers would need to use a path on a shared FS to setup the SHM handle, and, even then, this sounds like the sort of thing you could probably break via arcane misconfiguration.
I agree shm should work in principle though.
chasil 44 minutes ago [-]
You might consider taking the database(s) out of WAL mode during a migration.
That would eliminate the need for shared memory.
Retr0id 56 minutes ago [-]
> This is unlikely to work across containers.
Why not?
crabmusket 3 days ago [-]
Patient: doctor, my app loses data when I deploy twice during a 10 minute interval!
Doctor: simply do not do that
pavel_lishin 3 days ago [-]
Doctor: solution is simple, stop letting that stupid clown Pagliacci define how you do your work!
Patient: but doctor,
pjc50 1 hours ago [-]
pAIgliacci: as a large language model, I am unable to experience live comedy.
xnorswap 1 hours ago [-]
I'm fairly confident they let it write the blog post too.
simonw 23 minutes ago [-]
"Not as a proof of concept. Not for a side project with three users. A real store" - suggestion for human writers, don't use "not X, not Y" - it carries that LLM smell whether or not you used an LLM.
xnorswap 16 minutes ago [-]
And that's just the opening paragraph, the full text is rounded off with:
"The constraint is real: one server, and careful deploy pacing."
Another strong LLM smell, "The <X> is real", nicely bookends an obviously generated blog-post.
bombcar 1 hours ago [-]
Hey, Apple still takes their store down during product launches!
pstuart 22 minutes ago [-]
I assumed that it was to ensure that the announced products were revealed in a controlled manner rather than because they aren't able to do updates to their product listings as a regular thing.
bombcar 17 minutes ago [-]
My reading of the tea leaves is it started out as the latter and continues as the former as part of the “mystique”.
tensegrist 2 hours ago [-]
i hate to be so blunt but look around the site and then tell me you're surprised
infamia 3 days ago [-]
SQLite has a ".backup" command that you should always use to backup a SQLite DB. You're risking data loss/corruption using "cp" to backup your database as prescribed in the article.
The bottom part of the article mentions they use .backup - did they add that later or did you miss it?
BartjeD 58 minutes ago [-]
The post now says they changed it due to feedback from Hacker news. All good.
anonzzzies 2 hours ago [-]
Yeah, using cp to backup sqlite is a very bad idea. And yet, unless you know this, this is what Claude etc will implement for you. Every friggin' time.
chasil 2 hours ago [-]
It's fine if you run the equivalent of "init 1" first.
Does your OS have a single-user mode?
crazygringo 1 hours ago [-]
> Would We Choose SQLite Again? Yes. For a single-server deployment with moderate write volume, SQLite eliminates an entire category of infrastructure complexity. No connection pool tuning. No database server upgrades. No replication lag.
These are weird reasons. You can just install Postgres or MySQL locally too. Connection pool tuning certainly isn't anything you have to worry about for a moderate write volume. You don't ever need to upgrade the database if you don't want to, since you're not publicly exposing it. There's obviously no replication lag if you're not replicating, which you wouldn't be with a single server.
The reason you don't usually choose SQLite for the web is future-proofing. If you're totally sure you'll always stay single-server forever, then sure, go for it. But if there's even a tiny chance you'll ever need to expand to multiple web servers, then you'll wish you'd chosen a client-server database from the start. And again, you can run Postgres/MySQL locally, on even the tiniest cheapest VPS, basically just as easily as using SQLite.
NewEntryHN 40 minutes ago [-]
It's a spectrum. Installing Postgres locally is not 100% future-proofing since you'll still need to migrate your local Postgres to a central Postres. Using Sqlite is not 0% future-proofing since it's still using the SQL standard.
If the only argument for a piece of tech in comparison to another one is "future-proofing", that's pretty much acknowledging the other one is simpler to setup and maintain.
runako 33 minutes ago [-]
Have run PG, MySQL, and SQLite locally for production sites. Backups are much more straightforward for SQLite. They are running Kamal, which means "just install Postgres" would also likely mean running PG in a container, which has its own idiosyncrasies.
SQLite is not a terrible choice here.
crazygringo 5 minutes ago [-]
> Backups are much more straightforward for SQLite.
Not sure how? All of them can be backed up with a single command. But if you want live backups (replication) as opposed to daily or hourly, SQLite is the only one that doesn't support that.
kaibee 59 minutes ago [-]
Yeah a PG Docker container is basically magic. I too went down a rabbit-hole of trying to setup a write-heavy SQLite thing because my job is still using CentOS6 on their AWS cluster (don't ask). Once I finally got enough political capital to get my own EC2 box I could put a PG docker container on, so much nonsense I was doing just evaporated.
xnorswap 1 hours ago [-]
Yeah, it's weird "they" don't consider any middle ground between SQLite and replicated postgres cluster.
Locally running database servers are massively underrated as a working technology for smaller sites. You can even easily replicate it to another server for resiliency while keeping the local performance.
talkingtab 1 hours ago [-]
This. Spinning up Postgresql is easy once you know how. Just as SQLITE3 is easy once you know how. But I can find no benefit from not just learning postgres the first time around.
kaibee 58 minutes ago [-]
They're using AI Agents to do it in either case and using docker. There was no reason to choose SQLite.
cadamsdotcom 3 days ago [-]
The fix appears to nicely asking the forgetful unreliable agent to please (very closely pretty please!) follow the deploy instructions (and also please never hallucinate or mess up, because statistics tells us an entity with no long term memory and no incentive to get everything right will do the job right 99.99999999% of the time, which is good enough to run an eshop) not deploy too often per hour.
With one simple instruction the system (99.9999% of the time) gains the handy property that “only” two processes end up with the database files open at once.
Thanks for the vibes!
devmor 2 hours ago [-]
I have to work with agents as a part of my job and the very first thing I did when writing MCP tools for my workflow was to ensure they were read only or had a deterministic, hardcoded stopgap that evaluates the output.
I do not understand the level of carelessness and lack of thinking displayed in the OP.
mywittyname 42 minutes ago [-]
Even just having the agent write scripts to disk and run those works wonders. It keeps the agent from having to rebuild a script for the same tasks, etc.
devmor 36 minutes ago [-]
That too! Every time the agent does something I didn't intend, I end up making a tool or process guidance to prevent it from happening again. Not just add "don't do that" to the context.
jmull 3 days ago [-]
Redis, four dbs, container orchestration for a site of this modest scope… generated blog posts.
Our AI future is a lot less grand than I expected.
ramon156 2 hours ago [-]
How else will you get all those resume entries ! (/j)
add-sub-mul-div 36 minutes ago [-]
Ironically, AI de-skilling results in a robust-sounding resume.
Natfan 3 days ago [-]
llm generated article.
please consider writing it yourself. quirks in human writing is infinitely more interesting than a next-token-predicted 500 word piece
sgbeal 3 days ago [-]
> json_extract returns native types. json_extract(data, '$.id') returns an integer if the value was stored as a number. Comparing it to a string silently fails. Always CAST(json_extract(...) AS TEXT) when you need string comparison.
This is becoming the new overused LLM goto expression for describing basic concepts.
jmull 28 minutes ago [-]
I don't know if it's just me, but this whole post seems to have time traveled forward from about 3-4 days ago.
It's not just a repost. The thread includes a comment I made at the time which now from "1 hour ago".
Makes me wonder if it's an honest bug or someone has hacked the hacker news front page to sell their t-shits, mugs, and AI starter kits.
Retr0id 27 minutes ago [-]
It's an artefact of the "second chance pool" mechanism.
worksonmine 2 minutes ago [-]
Interesting choice to change the time of the comment, a deja-vu can be weird enough without staring at a comment with a recent timestamp.
nop_slide 33 minutes ago [-]
I still haven't figured out a good way to due blue/green sqlite deploys on fly.io. Is this just a limitation of using sqlite or using Fly? I've been very happy with sqlite otherwise, rather unsure how to do a cutover to a new instance.
Anyone have some docs on how to cutover gracefully with sqlite on other providers?
wolttam 20 minutes ago [-]
You accept downtime. That's the limitation of SQLite.
Or you use some distributed SQLite tool like rqlite, etc
jszymborski 3 days ago [-]
The LLM prose are grating read. I promise, you'd do a better job yourself.
faangguyindia 3 days ago [-]
I've a busy app, i just deploy to canary. And use loadbalancer to move 5% traffic to it, i observe how it reacts and then rollout the canary changes to all.
how hard and complex is it to roll out postgres?
pezh0re 3 days ago [-]
Not hard at all - geerlingguy has a great Ansible role and there are a metric crapton of guides pre-AI/2022 that cover gardening.
leosanchez 3 days ago [-]
> Backups are cp production.sqlite3 backup.sqlite3
I use gobackup[0] as another container in compose.yml file which can backup to multiple locations.
Wait, you push straight to main?
> We added a rule — batch related changes, avoid rapid-fire pushes. It's in our CLAUDE.md (the governance file that all our AI agents follow):
> Avoid rapid-fire pushes to main — 11 pushes in 2h caused overlapping Kamal deploys with concurrent SQLite access.
Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy?
"Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open."
WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers.
In case anybody needs a refresher:
https://en.wikipedia.org/wiki/Shared_memory
https://en.wikipedia.org/wiki/CB_UNIX
https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-syste...
See more: https://sqlite.org/wal.html#concurrency
I think you're exactly right about the WAL shared memory not crossing the container boundary. EDIT: It looks like WAL works fine across Docker boundaries, see https://news.ycombinator.com/item?id=47637353#47677163
I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over.
From https://kamal-deploy.org/docs/upgrading/proxy-changes/ it looks like Kamal 2's new proxy doesn't have this yet, they list "Pausing requests" as "coming soon".
The easiest approach is to kill sqlite, then start the new one. I’d use a unix lockfile as a last-resort mechanism (assuming the container environment doesn’t somehow break those).
I don't, fwiw (so long as all containers are bind mounting the same underlying fs).
Could the two containers in the OP have been running on separate filesystems, perhaps?
https://sqlite.org/wal.html
The containers would need to use a path on a shared FS to setup the SHM handle, and, even then, this sounds like the sort of thing you could probably break via arcane misconfiguration.
I agree shm should work in principle though.
That would eliminate the need for shared memory.
Why not?
Doctor: simply do not do that
Patient: but doctor,
"The constraint is real: one server, and careful deploy pacing."
Another strong LLM smell, "The <X> is real", nicely bookends an obviously generated blog-post.
https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_...
Does your OS have a single-user mode?
These are weird reasons. You can just install Postgres or MySQL locally too. Connection pool tuning certainly isn't anything you have to worry about for a moderate write volume. You don't ever need to upgrade the database if you don't want to, since you're not publicly exposing it. There's obviously no replication lag if you're not replicating, which you wouldn't be with a single server.
The reason you don't usually choose SQLite for the web is future-proofing. If you're totally sure you'll always stay single-server forever, then sure, go for it. But if there's even a tiny chance you'll ever need to expand to multiple web servers, then you'll wish you'd chosen a client-server database from the start. And again, you can run Postgres/MySQL locally, on even the tiniest cheapest VPS, basically just as easily as using SQLite.
If the only argument for a piece of tech in comparison to another one is "future-proofing", that's pretty much acknowledging the other one is simpler to setup and maintain.
SQLite is not a terrible choice here.
Not sure how? All of them can be backed up with a single command. But if you want live backups (replication) as opposed to daily or hourly, SQLite is the only one that doesn't support that.
Locally running database servers are massively underrated as a working technology for smaller sites. You can even easily replicate it to another server for resiliency while keeping the local performance.
With one simple instruction the system (99.9999% of the time) gains the handy property that “only” two processes end up with the database files open at once.
Thanks for the vibes!
I do not understand the level of carelessness and lack of thinking displayed in the OP.
Our AI future is a lot less grand than I expected.
please consider writing it yourself. quirks in human writing is infinitely more interesting than a next-token-predicted 500 word piece
More simply:
vs:This is becoming the new overused LLM goto expression for describing basic concepts.
It's not just a repost. The thread includes a comment I made at the time which now from "1 hour ago".
Makes me wonder if it's an honest bug or someone has hacked the hacker news front page to sell their t-shits, mugs, and AI starter kits.
Anyone have some docs on how to cutover gracefully with sqlite on other providers?
Or you use some distributed SQLite tool like rqlite, etc
how hard and complex is it to roll out postgres?
I use gobackup[0] as another container in compose.yml file which can backup to multiple locations.
[0]: https://gobackup.github.io/