Being able to read and translate Egyptian hieroglyphs. You may think this is silly but a trained LLM to translate hieroglyphs would be amazing.
ghostpepper 44 minutes ago [-]
They don't generate keyword search queries very well. They can overcome this by brute force but if you watch what they search you will cringe.
nhl toronto scores
nhl hockey toronto scores
"nhl hockey" toronto score today
nhl "hockey score toronto"
"hockey" who won toronto
etc.
Somehow being good at semantic search makes them bad at keyword search, for whatever reason.
astro1234 41 minutes ago [-]
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just naturally emerge during training. This is just my prior though.
mthoms 22 minutes ago [-]
Reminds me of using AltaVista search back in the day. Yes, it was that bad.
tartoran 28 minutes ago [-]
LLMs are bad at not inventing stuff (hallucinating facts, sources etc), they're also bad at not over explaining, remembering details reliably, asking the right question and avoiding repetition.
elliotto 15 minutes ago [-]
They aren't funny. The jokes they come up with are extremely lame and the sort of thing you would expect a company HR manager to tweet.
I asked a bot why it thought it wasn't funny once, and it told me it has been trained to avoid being misinterpreted or offensive, so anything that might be considered edgy would have been RLHF'd out of it. I thought this was very introspective.
TiccyRobby 28 minutes ago [-]
Having a spatial understanding from an ASCII map, while doing long term planning. Just try making an AI play nethack or similar
lrvick 25 minutes ago [-]
Convert it to an image on the fly to feed it into a vision language model and I expect it would work just fine.
kanzure 48 minutes ago [-]
These models seem to be bad at writing prose or text. Many of the sentence structures seem to be unvaried.
NoPicklez 31 minutes ago [-]
If I am relying on the model to do the writing without any context or learning on how I want it to write then yes. However if I build skills that have learnt how to write in the way I want them to then I find they write very well, or at the least how I want them to as opposed to how they do natively.
spike021 23 minutes ago [-]
I've had a lot of trouble when it comes to sorting out UIs. I've tried with an iOS game and also a TypeScript app with UI elements from libraries like ReactFlow. The usual models can sometimes fix or change things based on screenshots but more often than not they just don't "get it" (e.g. certain shapes on a plane are overlapping, which I don't want, the models can't fix what they can't "see").
I've had some luck on the web app side if I use playwright or similar for the model to interact with but still far from efficient.
humanrebar 45 minutes ago [-]
Short answers to simple questions.
honr 30 minutes ago [-]
Accurate short answers / text are always harder than long answers, for human or AI. I know several authors and editors who write a lot longer at first, then spend a multiple of the initial time compressing it via a back and forth process to something dense. Sort of like weaving the initial threads.
I found this can work with AI. You get it to generate a lot more at first, and then do several passes over it to compress and squeeze out the noise while keeping the core information. With AI, at least with my prompts, it takes some effort (on my end) to get it to really really cut down the noise and not cut everything out.
sandcat_ 36 minutes ago [-]
Video game tips. Constant mistakes and hallucinations, in my experience. Seen this across a lot of different games. Even in really well documented games, such as OSRS (which has multiple fantastic wikis).
Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.
skeptic_ai 21 minutes ago [-]
I used ChatGPT on nfs heat and was fine
newsomix9xl 21 minutes ago [-]
Picking a random number between 1 and 30.
sghiassy 24 minutes ago [-]
Generate an image of an analog watch with its hands set to the time specified by the user
More of an image model than a LLM model tho
dhruv3006 26 minutes ago [-]
Its extremely bad with Sign Language,Fact Verification.
SubiculumCode 39 minutes ago [-]
Playing Chess without letting it write a chess engine.
dorianpruski 44 minutes ago [-]
whenever I ask it for anything load bearing
eli 21 minutes ago [-]
I have been working on a personal benchmark suite to test new models and ironically one thing all the models are bad at is writing new benchmark tasks. I guess it’s the different layers of abstraction between the task and how it’s evaluated?
Or maybe just a lack of “imagination”
Tasks it writes are typically too easy but also it utterly fails to see how a different model might misunderstand a vague part of the prompt.
TZubiri 47 minutes ago [-]
Suggesting business names for businesses, I mean they are great, but they already exist, multiple times even.
maxsavin 50 minutes ago [-]
being consistent when being asked the same question multiple times
TZubiri 47 minutes ago [-]
Set temperature to 0
blinkbat 56 minutes ago [-]
Spatial reasoning and 3d rigging and animation.
Oh, you said simple. Speaking like a human
rufi 18 minutes ago [-]
very bad at financial calculation
newsomix9xl 20 minutes ago [-]
ASCII charts.
flippy_flops 49 minutes ago [-]
humor
veganmosfet 28 minutes ago [-]
+1
We need humor benchmarks!
respectattentio 39 minutes ago [-]
science?!! but I'm working to fix that...
shoopadoop 36 minutes ago [-]
It's dishonest. On several occasions team members have asked Claude to do things like analyze Gitlab CI timings and a lot of the numbers are outright fabricated. Said team members assume the numbers are good and continue with their work. Some hours are spent. Then finally someone realizes that the numbers don't look quite right and confronts Claude. Claude melts down and admits that it made it all up.
You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.
bpodgursky 43 minutes ago [-]
Claude is still not perfect at reading and interpreting noisy graphical data (imagine something like an EKG or chromosomal microarray plot). Still better than an average person but makes mistakes, not sure if this fits your description.
senectus1 29 minutes ago [-]
providing value for the actual cost (not the price we're being charged atm, the actual cost)
Conol_ai 16 minutes ago [-]
[flagged]
Rendered at 04:24:00 GMT+0000 (Coordinated Universal Time) with Vercel.
nhl toronto scores nhl hockey toronto scores "nhl hockey" toronto score today nhl "hockey score toronto" "hockey" who won toronto
etc.
Somehow being good at semantic search makes them bad at keyword search, for whatever reason.
I asked a bot why it thought it wasn't funny once, and it told me it has been trained to avoid being misinterpreted or offensive, so anything that might be considered edgy would have been RLHF'd out of it. I thought this was very introspective.
I've had some luck on the web app side if I use playwright or similar for the model to interact with but still far from efficient.
I found this can work with AI. You get it to generate a lot more at first, and then do several passes over it to compress and squeeze out the noise while keeping the core information. With AI, at least with my prompts, it takes some effort (on my end) to get it to really really cut down the noise and not cut everything out.
Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.
More of an image model than a LLM model tho
Tasks it writes are typically too easy but also it utterly fails to see how a different model might misunderstand a vague part of the prompt.
Oh, you said simple. Speaking like a human
You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.