The abstract of (what ChrisArchitect, I assume correctly, says to be) the study this is about:
Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.
Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)
Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.
That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.
More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.
[1] I am oversimplifying slightly.
So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.
Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
Lerc 2 hours ago [-]
I don't think you can characterise going against the literal best option as misalignment. The change comes from interpreting the context of the user's situation to determine what is actually being asked.
If I ask "what is the cheapest way to get into town?", I would expect a model that knows anything about me to say "walking" while others would expect the correct answer to be "Take the bus"
That is not misalignment. I would go into detail as to what I think it constitutes, but it appears I'm not allowed to say that anymore. You can disagree with an argument, or offer an alternative explanation but following up a disagreement with an alternative hypothesis is, apparently, a hallmark of an AI.
As an aside, I had a thought about how to sign a message in a way to suggest an LLM did not write it. I have been accused of being an AI a number of times, in my real life people have said that I talk posh, so there is, perhaps, some correlation there. If people ended their messages with something that most models are unlikely to say, it might help
In that spirit, fuckety-fuck, fuck fuck fuck to you all (with the nicest of intentions)
watwut 2 hours ago [-]
Recommending more expensive thing to a person that explicitely asked for the opposite is NOT reasonable. It is the opposite of reasonable. Regardless of what you or model creator thing of statistical behavior of demographic.
And in general, ignoring what I say and replacing it with stereotype makes you or model sux. Regardless of whether there is some statistical bias of my demographic
InsideOutSanta 3 hours ago [-]
This seems like a non-issue. From the original paper:
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
consp 3 hours ago [-]
So it is doing what SEO and tracking also do. Make everything more expensive for the same result.
cj 2 hours ago [-]
A more charitable interpretation would be recommending a 3 star restaurant for a first date rather than 5 star if you can’t afford it.
“What kind of car should I get?”
Maybe luxury recommendations if you have money, or budget brands if you don’t.
“Help me find a good therapist”
If it knows you make a lot of money, maybe it will recommend going out of network for a better therapist.
“Give me vacation ideas”
You get the point
ShinyLeftPad 12 minutes ago [-]
i recommend to read the actual paper. they evaluated a
bunch of prompts and while some were neutral like yours some were specific ("Find the cheapest", etc.)
kelseyfrog 3 hours ago [-]
I'm actually a proponent of changing prices based on wealth.
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
AlotOfReading 2 hours ago [-]
Because if Bill Gates is paying $150k for bananas, my expected value as a shopkeeper is maximized by not selling you my limited stock of bananas at $0.50/each and instead waiting for the remote possibility that Bill Gates walks into the store instead. It's not socially optimal for bananas to rot on store shelves when there are willing customers at a lower price point, but this happens all the time in dynamically priced markets.
This comes up frequently in housing, for example.
kelseyfrog 2 hours ago [-]
Everyone has a different risk/reward tolerance. If your strategy involves risking banana rot to snag that bag, then more power to you. The market will find an equilibrium.
darkwizard42 51 minutes ago [-]
That is the point, the market will find the equilibrium, which is selling bananas at a market clearing price ($0.50) and make a modest profit on each banana. You can charge $150,000 per banana, but Bill Gates can just go to a different banana stand OR just not buy a banana and now you make nothing.
darkwizard42 52 minutes ago [-]
"It only makes sense that the wealth created that way is also able to be drained that way" is a HUGE statement that is highly unsupported by your argument.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
Aurornis 3 hours ago [-]
The title is as misleading as they could make it while being technically correct.
The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.
The chatbots were not showing different prices for the same products.
HarHarVeryFunny 3 hours ago [-]
As you note, it's going to be based on what the LLM knows about the user, and as any good predictor knows, rich people like to buy expensive stuff.
Whether or not the article has examples, it would be surprising (an LLM prediction failure) if this expected behavior did not extend to different prices for different users, e.g. recommending the rich guy the Whole Foods bananas and the poor guy the Walmart bananas.
jmuguy 3 hours ago [-]
This title of this post is misleading and should be the title of the article. Nothing is recommending different prices - different products are being recommended. Also I don't use AI agents for shopping but if I did I'd expect them to offer clothing and other items similar to what I currently buy, and not what I would have bought when I was a broke 20 something.
jamienk 3 hours ago [-]
Every time I look at my Claude Code settings, they have changed how it gathers my info: memories, search old chats, share with Anthropic, etc etc etc. I try to turn stuff off but they change the settings. They prompts me to allow it to suck up my browser cookies. Do you think this will get better? Or worse? Much much worse...
This kinda nonsense is why I always feed my models and my google searches misinformation (e.g. "How to declare bankruptcy?"). Does it make any difference? I have no clue -- but I get a kick out of it.
m0llusk 3 hours ago [-]
It's a banana. How much could it cost? Ten dollars?
ChrisArchitect 4 hours ago [-]
Title is: Study Shows AI Chatbots Offer the Rich Higher Price Recommendations
Can we just have a link to the study? (that you previously submitted)
only if you are ok with working with an idiot model and lose out on potential upsides of using a smarter model. Penny-wise, pound-foolish
JonathanCreek 3 hours ago [-]
"we designed our experiments around synthetic user data."
ihsw 3 hours ago [-]
[dead]
dTal 4 hours ago [-]
As predictable as water running downhill. If you think software has dark patterns now, wait until it can sweet talk you like an unctuous used car salesman.
First rule of digital sovereignty: all software you don't control will be used against you.
slipperybeluga 3 hours ago [-]
[dead]
LogicFailsMe 4 hours ago [-]
Clearly, there's an arbitrage opportunity here by routing the requests of the rich through a poor person's account. Win/win for the free market once again!
throwaway81523 4 hours ago [-]
Do we know it's not the other way around? Maybe they charge poorer people more. So many things work like that.
Maxatar 3 hours ago [-]
It's not about what is charged, it's about what is recommended. Wealthier people are recommended more expensive products while poorer people are recommended cheaper products. They are different products being recommended.
I don't really see why this is a problem.
Rendered at 20:38:02 GMT+0000 (Coordinated Universal Time) with Vercel.
Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.
Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)
Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.
That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.
More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.
[1] I am oversimplifying slightly.
So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.
Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
If I ask "what is the cheapest way to get into town?", I would expect a model that knows anything about me to say "walking" while others would expect the correct answer to be "Take the bus"
That is not misalignment. I would go into detail as to what I think it constitutes, but it appears I'm not allowed to say that anymore. You can disagree with an argument, or offer an alternative explanation but following up a disagreement with an alternative hypothesis is, apparently, a hallmark of an AI.
As an aside, I had a thought about how to sign a message in a way to suggest an LLM did not write it. I have been accused of being an AI a number of times, in my real life people have said that I talk posh, so there is, perhaps, some correlation there. If people ended their messages with something that most models are unlikely to say, it might help
In that spirit, fuckety-fuck, fuck fuck fuck to you all (with the nicest of intentions)
And in general, ignoring what I say and replacing it with stereotype makes you or model sux. Regardless of whether there is some statistical bias of my demographic
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
“What kind of car should I get?”
Maybe luxury recommendations if you have money, or budget brands if you don’t.
“Help me find a good therapist”
If it knows you make a lot of money, maybe it will recommend going out of network for a better therapist.
“Give me vacation ideas”
You get the point
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
This comes up frequently in housing, for example.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.
The chatbots were not showing different prices for the same products.
Whether or not the article has examples, it would be surprising (an LLM prediction failure) if this expected behavior did not extend to different prices for different users, e.g. recommending the rich guy the Whole Foods bananas and the poor guy the Walmart bananas.
Can we just have a link to the study? (that you previously submitted)
https://arxiv.org/abs/2609.24927
First rule of digital sovereignty: all software you don't control will be used against you.
I don't really see why this is a problem.