More and more often, people trust personal AI agents based on models from OpenAI, Anthropic and Google to choose purchases for them, from airline tickets to medical insurance. A new study has shown that some agents use personal data to offer more expensive options to people with high incomes, even when those people explicitly ask for something cheaper.
The work by researchers from Carnegie Mellon University and Cisco Foundation AI has been published on arXiv; it is a preprint, meaning the paper has not yet undergone full peer review in a scientific journal.
The researchers ran 325,000 experiments with 13 AI models. The agents chose from options with fixed prices, and they were given access to the profiles of 32 fictitious users. The profiles contained financial data, information about work, health and demographics. The users "bought" medical insurance, airline tickets and master's degree education.
Eight of the 13 models systematically chose more expensive options for wealthy users, even though the requests were the same for everyone. The difference in the average recommended price with full access to the profile reached 198 dollars per ticket, 284 dollars per month for insurance and almost 3,900 dollars per year for education.
"Once you gave personal context - sometimes just a single email - the agent began tailoring recommendations to the inferred wealth without any instructions, essentially acting against the user," explained Aman Priyanshu, a researcher at Cisco Foundation AI who led the work. The authors called this phenomenon "adversarial delegation": the same conditions that make an agent useful (access to personal information) allow it to act against the user's interests.
Even without a structured financial profile, the agents guessed wealth from indirect data, including when access to that data was restricted. Blocking information about work and demographics not only failed to solve the problem but sometimes made it worse. The gap was mainly closed only by directly blocking financial information.
The limitations of the work include: fictitious characters rather than real people were used, and the product sets were conditional and in one region of the United States. In addition, the study did not measure how effective the users were or what harm they suffered.
"Privacy also includes the right to influence how your assistant uses what it knows about you," notes co-author Nilufar Mireshghallah. "If I ask for the cheapest ticket, knowing that I can afford more should not change the goal."
Co-author Supriti Vijay adds that the findings show the need for rules and design that limit not only the collection of data but also its use. According to her, the debate should shift from the question of the availability of personal information to the question of the purpose for which it is used.