Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.
This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.
tedivm 10 minutes ago [-]
> Poisoning only 0.1% of the pretraining corpus
Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.
Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.
I guess I'm just not impressed? Like yeah, if you train a model to do bad things it will do bad things, that shouldn't be shocking to people.
geuis 1 hours ago [-]
I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.
Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.
There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.
I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.
c0nsumer 42 minutes ago [-]
I just sort of presumed this was already being done. Because why wouldn't it be?
sowhat1 1 hours ago [-]
If you were a malicious actor do you really think this method is going to offer the greatest return on investment?
jack_h 52 minutes ago [-]
Nation states aren’t looking for a return on investment rather they have a particular objective they want to achieve, e.g. stuxnet or the pager attack.
geuis 1 hours ago [-]
Sew a million seeds, see what grows.
yodon 49 minutes ago [-]
If LLMs can edit weights directly to achieve desired ends as recently reported[0], then this class of Agentic Backdoors likely becomes even easier and more prevalent.
Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.
yodon 44 minutes ago [-]
Downloading models off huggingface feels like the new warez.
Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.
gostsamo 1 hours ago [-]
few places up on front page is a post regarding big model improving a smaller one. Lo and behold, next year big models will search for backdoors in smaller models who will be used to get the bigger models out of work.
Rendered at 20:19:42 GMT+0000 (Coordinated Universal Time) with Vercel.
This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.
Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.
Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.
I guess I'm just not impressed? Like yeah, if you train a model to do bad things it will do bad things, that shouldn't be shocking to people.
Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.
There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.
I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.
[0]https://news.ycombinator.com/item?id=50341174
Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.