I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
nijave 59 minutes ago [-]
I generally trust benchmarks but my personal experience in the last 6-9 months is they have little relation to real LLM performance, unfortunately.
I've had better luck just running a few of my own simple tests.
samsmallmind 2 hours ago [-]
100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)
verdverm 1 hours ago [-]
I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.
armhag 2 hours ago [-]
[dead]
pietz 3 hours ago [-]
Or, you know, it's an easy excuse because they're already behind on many of them.
girvo 3 hours ago [-]
They are? In all my evals at work, Jev seems to win, despite me wanting the open weight versions to.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
inerte 3 hours ago [-]
Same here, and I have a project 100% hosted in Cloudflare ecosystem (AI Gateway, Workers, Queues, R2, D1, DO, etc) I was so happy when I saw clef. Tested, and it's bad. Slower and wronger than Jev.
Still waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
vilos1611 2 hours ago [-]
We evaluated the Decisions API at work and it was worse in all dimensions (price, speed, and quality of the result).
jwpapi 3 hours ago [-]
Or they use it as marketing wording to get liked by devs.., which is most likely given we know that’s typesafes usp
CompleteSkeptic 2 hours ago [-]
we've been public about this before the category was even created
I used the model and it was good enough for email classification.
I was wondering if I'm supposed to be doing multiple runs for topic classification with grouping related topics i.e. if its sport,economy or tech related news and then another run for soccer,basketball or something else
or batching them like the News article topic classification example in a single run.
nico 3 hours ago [-]
[flagged]
mythz 32 minutes ago [-]
For anyone interested in a GUI around Jev I built Decision Studio for experimenting with it in our self-hosted OSS llms.py [1]
You can define tasks like sentiment analysis, support triage, or claim support, tweak the questions and criteria, and inspect the answers and probability distributions. You can also import existing recipes, save examples, and share recipes with recorded results.
Cool to see another model with support for bounding boxes on images. Really wish to see segmentation in a future release; Gemini 3.5 Flash-Lite is mandatory in my use case as it's the best model for semantic image segmentation in my experience.
moonu 4 hours ago [-]
Curious why you use Gemini 3.5 Flash Lite instead of SAM/Moondream? Any specific workflows where it outperforms?
tyushk 4 hours ago [-]
Using it as part of an agentic CAD workflow, where the model needs to be able to selectively segment from parts of a user's image as it works on a project. Using OpenSCAD so being a general purpose model is very useful to me.
bbor 8 minutes ago [-]
There's no reason that an agentic workflow has to use the primary agent model for every intuitive computing task; in fact, that would've sounded absurd before the singularity. Maybe an old flash model is the best in class for this, but A) I doubt it and B) it's still gonna be wildly more expensive.
Assuming I'm not just misunderstanding something critical (happens often!) and you haven't explored this already (very possible!), pointing your agent at this link and asking for some prototypes might be illuminating: https://huggingface.co/models?pipeline_tag=image-segmentatio...
gunalx 3 hours ago [-]
If openscad is the thing Google models has consistently outpermormed the comparable competition in my experiences. Few other models has better physical spatial reasoning skills.
shepardrtc 2 hours ago [-]
If you're looking for a decent local model, JevK5-4B is excellent. On my own benchmark for the things I work with (seeing if the answer to a question was correct given the evidence that was provided), it was within a point of jev.
- Inference for a bunch of open weights architectures
- Diffusion image generation fir Z ai turbo
- Vision input into LLM for inference
- Vulkan/fast CPU, optimized kernels
- Playground for testing
- Last but not least, system one fast inference mode with same API semantics as Jev
Decision models work by using a backbone model like qwen 3.5 9b, along with a special linear head that provides the readout for the forward pass.
aliasxneo 2 hours ago [-]
> Surprise: the Jev you all know and love (jev-1.13.0) was multimodal all along!
It really feels like they're just admitting that they had it disabled because they needed investors to subsidize turning it on.
120bits 3 hours ago [-]
Slightly unrelated, but I really like the website styling. Looks like an improved version of Linux From Scratch dos.
lukeify 3 hours ago [-]
Swiss Style/Typographic Style.
hnav 3 hours ago [-]
too bad it's just LARPing, underneath the covers it's the typical bloated Framer site.
AftHurrahWinch 3 hours ago [-]
Going from "I'll need a research team and 5 years" to Jev-based bird classifiers in just 12 years is really nice. I'm happy that some problems get easier, and I'm happy that people make things and share their work so we can all benefit.
A lot of the week 1 Jev demos involved getting image data into it somehow, via description I guess? Should be interesting to see what we get with 100ms latency decisions on visual inputs. To me the latency / response time is just a huge game changer -- one to two orders of magnitude more responsiveness opens up a large design space.
hexasquid 4 hours ago [-]
I'm curious what people are building with these models
Rudybega 4 hours ago [-]
I use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
inerte 3 hours ago [-]
For me it's using 1 Jev call to ask 5 questions, instead of what I had before.
What I had before was basically:
- 1 classifier question
- if ok, do some heavy computation
- after heavy computation, another classifier
So now what I do is:
- Jev with the 1st and 3rd classifier
- if ok, heavy computation
I can classify many things at once, and it's cheaper and faster than 1 for sure and 1 maybe classification that I had before.
SOLAR_FIELDS 13 minutes ago [-]
My favorite use case, can't take credit for it, saw it on an X post early on, is inlining it as a sql function so you can do fuzzy classifier based joins
vanuatu 4 hours ago [-]
^ Same here. so much faster and cheaper than using an LLM to return structured data
LewisOrton 4 hours ago [-]
it's actually more usefull than it looks. it is not much useful on its own but it can play a big role in a complex workflow or custom harness.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
nazgul17 4 hours ago [-]
Automatic tagging of images is what I'm doing with it. I'm using a modified Gemma model, which ingests images (and audio, for that matter).
gopalv 4 hours ago [-]
You can turn a lot of process and workflow systems into deep decision trees.
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
I found an example online, I cannot find the github again, but it was interesting: an instant dynamic generation of React component, you type what you want to build using natural language, and it directly add one to the page.
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
Sounds like an incremental change with context size change, new "box" type, and composable images.
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
therealpygon 4 hours ago [-]
Well, they do need to try to compete with SubQ’s longer context and image processing that has been coming out ahead in benchmarks, plus the thousand self-hosted options that now exist.
jofzar 4 hours ago [-]
Interesting way to release this, I'm not sure how to say this, but this has a real "enterprise"/old school taste to it verses how all the other models/AI companies are releasing things.
Very refreshing.
CompleteSkeptic 2 hours ago [-]
of all comments, "enterprise" taste is among those I'd least expect
bbor 15 minutes ago [-]
Ok. Obviously these are cool people trying to do something useful, and my real name is on this site so I definitely don't want to come off as hostile. But I have three problems with their whole... thing, here:
1. Adding bounding boxes only compounds the biggest issue with Jev, which is that it's using a massive model to do what we've been using purpose-built[1] models to do for years(/decades), only a little worse and a lot more expensive. This is fine --great, even-- if presented as a tradeoff.[2] And a product that enables "smart if statements" that are "sci-fi" does not sound like a tradeoff at all. In that light, ending the post with facetiously-labeled graphs that just show it beating a bunch of random names (including models with twenty-seven BILLION parameters) on unknown metrics feels borderline dishonest.
2. "We love the community's desire to help make models better!" feels like Manufacturing Consent but with way lower stakes. I turn on various experimental features and pick nightly builds and all that like many of us do, but this feels... well, at best it's kinda offputting rhetoric for a practice I otherwise condone. IMHO.
3. Their job postings go beyond groan-inducing to vaguely insulting. Take this "Data Demon" posting[3], which is presumably for a data scientist (data engineer...?) but is branded around mocking the Turing-award-winning scientists trying to warn the world about superintelligence. Yes, I had the same reaction: what? Why?? It's giving a corporate culture where A) executives feel entitled to shoot from the hip and inject their personal crusades on a whim, and where B) scientists/science isn't appreciated (or, worse: understood). Criticizing the concept of measurement isn't helping, nor is their choice to label output percentages as confidence scores.
I guess I hope I'm misunderstanding, cause they're gonna make millions regardless, I'm sure. I just know that I can't think of when I'd reach for Jev 1.*[4] for a prod usecase, and this is an awfully big company for a prototyping shortcut.
[4]: Which, btw, what happened to semver? Did we stop doing semver cause python did? Did I miss a memo/blog? Not a rhetorical question, I swear. Deprecating minor versions just seems unholy, much less minor version thirteen!
Havoc 3 hours ago [-]
Nice release post
MisterMunchkin 3 hours ago [-]
I have to admit, despite my initial scepticism of Jev, it is actually pretty fucking good at making decisions. I’m trying to break it with weird situations and I just can’t. And it’s so cheap!
cat-whisperer 2 hours ago [-]
the discord jam was awesome! kudos to these guys!!
matt3210 4 hours ago [-]
How does JEV prevent distillation?
himata4113 4 hours ago [-]
They don't. Generally we already have trillions tokens worth of classifier datasets so there's no reason to create synthetic data that will be worse than the data it was trained on.
nico 4 hours ago [-]
I don't think they can. But they probably monitor usage and test for patterns to then block/ban accounts
throwuxiytayq 2 hours ago [-]
[flagged]
kypro 4 hours ago [-]
I'm aware this is an unproductive comment, but why are all their releases so cringe? It's like they're written by a Reddit mod or something...
> For the builders AND dreamers
> Enter the danger zone at your own risk.
> Size does matter, apparently
> We’re trying something new out here: we’re going to treat you guys like adults
> Lots of people love to ask what kind of sci-fi we’re releasing next
That said, their docs are great and don't seem to include all the Reddit cringe.
shrinks99 4 hours ago [-]
I walked away with _mostly_ the opposite impression. They have a specific tone of voice! I think that's good.
I've yet to see any company match the copy writing skills of Valve though.
zy445566 59 minutes ago [-]
[dead]
benaduggan 4 hours ago [-]
So they held back a bunch of features because they knew they had basically no moat, and now they're trickling out a bit more functionality to try to convince people they are better than others and move fast? And using Apple advert slogans to show how cool they are?
aryabakh 4 hours ago [-]
With the speed they're adding features shows you're correct.
I guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.
woadwarrior01 3 hours ago [-]
The way that site charges a fee[1] for benchmarking models, makes any benchmarks on it very sus, IMO. TBF, the benchmarking site is probably more profitable than TypeSafe, just like the joke about how WinRAR is more profitable than OpenAI[2]. :)
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)
I've had better luck just running a few of my own simple tests.
I imagine if I fine tuned one of the open weight models for my use case it would be possible to beat it, though, but as a general drop in decision model, I’ve not really seen much that beats it especially for the cost
Still waiting on Decisions API using Luna, as this what I was using before.
Run your own evals, folks.
You can define tasks like sentiment analysis, support triage, or claim support, tweak the questions and criteria, and inspect the answers and probability distributions. You can also import existing recipes, save examples, and share recipes with recorded results.
An OpenRouter API key is required to use Jev.
[1] https://llmspy.org/docs/features/decision-studio
Videos will be next.
Assuming I'm not just misunderstanding something critical (happens often!) and you haven't explored this already (very possible!), pointing your agent at this link and asking for some prototypes might be illuminating: https://huggingface.co/models?pipeline_tag=image-segmentatio...
- Inference for a bunch of open weights architectures
- Diffusion image generation fir Z ai turbo
- Vision input into LLM for inference
- Vulkan/fast CPU, optimized kernels
- Playground for testing
- Last but not least, system one fast inference mode with same API semantics as Jev
Decision models work by using a backbone model like qwen 3.5 9b, along with a special linear head that provides the readout for the forward pass.
It really feels like they're just admitting that they had it disabled because they needed investors to subsidize turning it on.
[0]: https://xkcd.com/1425/
What I had before was basically:
- 1 classifier question
- if ok, do some heavy computation
- after heavy computation, another classifier
So now what I do is:
- Jev with the 1st and 3rd classifier
- if ok, heavy computation
I can classify many things at once, and it's cheaper and faster than 1 for sure and 1 maybe classification that I had before.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
[1] - https://github.com/prasanthj/duckdb-dual-cognition
[2] - https://demos.isotopes.ai/demo/apps/chain-command
If I remember well, in the background, it does the multi-choice question: "What is the component requested ? --> Calendar, Slider, ..."
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
Very refreshing.
1. Adding bounding boxes only compounds the biggest issue with Jev, which is that it's using a massive model to do what we've been using purpose-built[1] models to do for years(/decades), only a little worse and a lot more expensive. This is fine --great, even-- if presented as a tradeoff.[2] And a product that enables "smart if statements" that are "sci-fi" does not sound like a tradeoff at all. In that light, ending the post with facetiously-labeled graphs that just show it beating a bunch of random names (including models with twenty-seven BILLION parameters) on unknown metrics feels borderline dishonest.
2. "We love the community's desire to help make models better!" feels like Manufacturing Consent but with way lower stakes. I turn on various experimental features and pick nightly builds and all that like many of us do, but this feels... well, at best it's kinda offputting rhetoric for a practice I otherwise condone. IMHO.
3. Their job postings go beyond groan-inducing to vaguely insulting. Take this "Data Demon" posting[3], which is presumably for a data scientist (data engineer...?) but is branded around mocking the Turing-award-winning scientists trying to warn the world about superintelligence. Yes, I had the same reaction: what? Why?? It's giving a corporate culture where A) executives feel entitled to shoot from the hip and inject their personal crusades on a whim, and where B) scientists/science isn't appreciated (or, worse: understood). Criticizing the concept of measurement isn't helping, nor is their choice to label output percentages as confidence scores.
I guess I hope I'm misunderstanding, cause they're gonna make millions regardless, I'm sure. I just know that I can't think of when I'd reach for Jev 1.*[4] for a prod usecase, and this is an awfully big company for a prototyping shortcut.
[1]: https://huggingface.co/models
[2]: of dev speed for cost and accuracy, which is lowkey another way to say efficiency and efficacy. Two pretty massive things to trade!
[3]: https://jobs.ashbyhq.com/typesafe-ai/d315d436-e517-4f17-82bb...
[4]: Which, btw, what happened to semver? Did we stop doing semver cause python did? Did I miss a memo/blog? Not a rhetorical question, I swear. Deprecating minor versions just seems unholy, much less minor version thirteen!
> For the builders AND dreamers
> Enter the danger zone at your own risk.
> Size does matter, apparently
> We’re trying something new out here: we’re going to treat you guys like adults
> Lots of people love to ask what kind of sci-fi we’re releasing next
That said, their docs are great and don't seem to include all the Reddit cringe.
I've yet to see any company match the copy writing skills of Valve though.
I guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.
[1]: https://benchmarkheaven.com/submit
[2]: https://news.ycombinator.com/item?id=49797811