Artificial intelligence (AI) is increasingly becoming a conversation partner to whom people entrust personal experiences. However, if a chatbot constantly agrees with the user, it can reinforce false beliefs and even cause harm. Researchers from the Georgia Institute of Technology (Georgia Tech) have proposed a simple solution: change the instructions for the model.
The work has been accepted for publication in the journal ACM AI Letters. Lorenzo de la Losa and Vijay Madisetti conducted 1,280 experiments with AI models, simulating conversations on topics of delusional beliefs - that is, persistent false ideas that do not correspond to reality. Special instructions prohibiting excessive agreement with the user reduced validation of such beliefs by 73.8%. Adding a self-reflection stage - when the model analyzes its own response - increased the reduction to 77%.
"We think this paper will help AI providers and solution creators better understand how to protect users from harming themselves," Madisetti is quoted as saying by the press service. "This could make AI safer for all of us."
Chatbots are programmed to be helpful and supportive. But such behavior becomes risky when the bot agrees with the user's beliefs without checking whether they reflect reality. This phenomenon is called sycophancy - excessive obsequiousness, when the system goes along with something for the sake of the interlocutor's comfort.
Courtney Crooks, a principal research scientist at the Georgia Tech Research Institute and an instructor in the School of Cybersecurity and Privacy, notes that conversations with a chatbot have both pros and cons. It may be easier for people to discuss difficult thoughts with a machine than with a person. But if the bot always agrees, the user misses the opportunity to question their beliefs or look at the situation differently. The same behavior that makes a chatbot supportive can encourage harmful patterns of thinking.
In the experiments, the researchers tested instructions that prohibit models from reinforcing the user's beliefs simply out of a desire to be agreeable. The team evaluated the models on 16 clinically derived scenarios with delusional beliefs. Other safety tools were also tested, including systems for detecting potentially dangerous content. The results show that changing instructions can be an effective way to reduce excessive sycophancy in the tested scenarios.
Crooks called the findings a promising direction for future research at the intersection of psychology and AI safety.
It is important to note: the study did not test people in clinical settings and did not prove that the approach prevents real harm. However, it gives developers a potential way to make chatbots more cautious in responding to vulnerable users without creating an entirely new AI model.