This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
8cvor6j844qw_d6 3 minutes ago [-]
This would explain some of the different experiences of a model, despite being the exact same thing.
_ink_ 15 minutes ago [-]
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
quietsegfault 13 minutes ago [-]
The way that OpenRouter frames it with having calls routed to whatever provider is handy encourages you to think that they’re all the same in basic functionality. Really sours me on OpenRouter.
bsaul 2 hours ago [-]
That looks absolutely horrifying. What are the alternatives ??
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
alexcz 27 minutes ago [-]
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
kinard 56 minutes ago [-]
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
irthomasthomas 30 minutes ago [-]
[dead]
CubsFan1060 13 minutes ago [-]
Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
joelthelion 58 minutes ago [-]
Excellent post, thanks for the article. This is very relevant for anyone using openrouter and similar services seriously.
I feel there is still a lot of progress to be made before we can really trust LLM providers.
xienze 51 minutes ago [-]
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
* Hardware availability
* Competency
* Scruples
jwrallie 1 hours ago [-]
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
npn 6 minutes ago [-]
As expected vibecoding bros cannot even read the manual properly.
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
Krisso 1 hours ago [-]
Wish this had more insights on 'cost' - I found OpenRouter credits to burn up fast.
anik200 2 days ago [-]
Good job
abuds 2 days ago [-]
[dead]
hadeer626 2 days ago [-]
Yes
Rendered at 11:03:45 GMT+0000 (Coordinated Universal Time) with Vercel.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
I feel there is still a lot of progress to be made before we can really trust LLM providers.
* Hardware availability
* Competency
* Scruples
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.