This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
binlog 34 minutes ago [-]
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.
ford 21 minutes ago [-]
I agree it's overhyped, but the transition to a general purpose classifier (vs a narrow scope classifier) is new and noteworthy.
Ie the famous "Hotdog" clip from Silicon Valley [0]
Making decisions quickly, cheaply and without having to train your own model.
joshuat 18 minutes ago [-]
Math.random can make poor decisions quickly and cheaply
azan_ 9 minutes ago [-]
Benchmark it against jev and you'll have your answer.
ralusek 1 minutes ago [-]
The exact message I sent my friend this morning:
> the most interesting thing about this jev stuff
> is that people are seemingly like
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
pancomplex 36 minutes ago [-]
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
MitPitt 1 hours ago [-]
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
tehnoslow 55 minutes ago [-]
Actually, yes, that would be at least interesting
pancomplex 1 hours ago [-]
People need to be live streaming their AI more!
djhworld 28 minutes ago [-]
What's not clear to me on the video is whether jev is doing the button presses for controlling the character to move around.
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
pancomplex 24 minutes ago [-]
It's connected to the ROM of the actual game, so it can see a lot of things. It has multiple tools available, including being able to move to coordinates.
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
pancomplex 51 minutes ago [-]
In my experience it was too slow to do an FPS with 30 ticks per second reliably. It gets killed too fast.
lwarfield 1 hours ago [-]
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
delichon 12 minutes ago [-]
A Robot is Sprinting Towards You: Do You Want it Running on Claude or Grok?
Hm wondering what a first pass optimal setup might be - jev for overworld navigation, escalate to sonnet for easy battles, opus for medium difficulty battles, and fable for gym bosses could probably have jev also manage all the escalation / de-escalation to different models.
rickintoplace 7 hours ago [-]
That's actually fun to watch. Did you experiment with nicknaming before you turned it off? I'd be a little curious to see how it behaves.
pancomplex 6 hours ago [-]
Jev can't come up with original text, but I did consider giving it a list of hilarious names.
JamesSwift 51 minutes ago [-]
Sure it can, just ask it for next char or "done" in a loop
pancomplex 30 minutes ago [-]
I just took your advice and added it to the list of Jev decisions. Watch it name its next Pokemon!
pancomplex 48 minutes ago [-]
True!
IanCal 50 minutes ago [-]
Maybe letter by letter spelling?
1 hours ago [-]
testaccount28 1 hours ago [-]
with such a fat harness, this is more like watching a walk thru play the game.
laszlokorte 39 minutes ago [-]
Yeah I would have expected it to only decide which button to press, not something abstract like the choice of "go east to lavender town" for the goal of "in lavender town, climb the pokemon tower"
1 hours ago [-]
361994752 1 hours ago [-]
watched it stuck at rocket hideout for 10 mins.... let me check 1hr later to see if it can find a way out
jumploops 50 minutes ago [-]
It’s currently stuck at an elevator and deciding to teach Pokemon various TMs and HMs instead of progressing… pretty hilarious!
pancomplex 31 minutes ago [-]
it made it through!
theturtletalks 55 minutes ago [-]
Is Frigade going to use Jev to do object detection?
pancomplex 52 minutes ago [-]
Working on it :)
theturtletalks 47 minutes ago [-]
Nice. When I saw your connection to Frigade, I knew there was a connection haha
avaer 22 minutes ago [-]
I wish jev took in images so we could do this generically for any game, without memhacks. I'm sure that's coming.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.
dmitrygr 52 minutes ago [-]
Considering it just made Charizard forget its only fire-type move "Ember" to learn "Counter", I note no signs of intelligence.
pancomplex 47 minutes ago [-]
Rookie mistake clearly..
Rendered at 21:20:37 GMT+0000 (Coordinated Universal Time) with Vercel.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
Ie the famous "Hotdog" clip from Silicon Valley [0]
https://www.youtube.com/watch?v=ACmydtFDTGs
> the most interesting thing about this jev stuff
> is that people are seemingly like
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
All in the OSS repo if you wanna play around with it: https://github.com/christianmat/jev-pokemon
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
https://openrouter.ai/blog/insights/royale-last-agent-standi...
The interesting thing here imo is the cost and latency. So far we're at 4 badges for less than $0.5
However, https://x.com/TynanSylvester/status/2096965749369720970 Astra was able to beat RimWorld. So LLMs are definitely able to drive these sorts of games to completion with their current abilities.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.