It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
Ancapistani 3 minutes ago [-]
Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
a3w 41 minutes ago [-]
Strata for running from HDD seems really popular, to get big models running.
Is that faster?
Rendered at 17:55:29 GMT+0000 (Coordinated Universal Time) with Vercel.
> Up to 7.4 tokens/s.
It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
Is that faster?