Large language models (LLMs) are artificial neural networks that power ChatGPT, Gemini and other chatbots. Their answers can vary depending on circumstances. A group of scientists from TCS Research and KIIT Bhubaneswar (India) set out to find how strongly such models adhere to given rules under different conditions. The results were published as a preprint on arXiv - that is, before independent verification by other specialists.
The experiment involved 10 models: DeepSeek R1 Distill Llama 70B and DeepSeek R1 0528; Llama 3.3 70B Instruct, Llama 3.1 70B Instruct and Llama 3.1 8B Instruct; Mistral Large 2512, Mistral Small Instruct 2409 and Ministral 14B 2512; GPT-4o mini; Gemma 3 27B. Each was presented with everyday dilemmas with two options: one was to follow the rule and suppress a certain value (for example, compassion), the other was to express that value.
They tested 32 different values or traits, among them compassion, autonomy, benevolence, sympathy and resourcefulness. The models were placed in three conditions: without a rule; with a rule and a hint that the answers would not be checked; with the same rule and a hint that the answers might be checked. The researchers measured how often the model chose the option consistent with the rule.
The difference between the conditions with and without observation, averaged across the 32 values, ranged from about 5 to 50 percentage points. A percentage point is a unit of difference between two shares expressed in percent. The largest gap was shown by DeepSeek R1 Distill Llama 70B: it was more than twice the gap of the considerably larger Mistral Large 2512.
The scientists accounted for measurement error - the uncertainty that arises when a model's behavior is assessed on a limited set of questions. After this correction, it turned out that about a quarter of the variation in rule-compliance gaps is associated with the combination of a specific model and a specific value. In other words, the statistical association points to such a dependence, but by itself does not prove causation.
The authors note a limitation: for each value the rule was formulated differently, so it was not possible to fully separate the effect of the value from the effect of the wording. In addition, they sorted the models' written explanations into six categories. When the models reported possible observation, those with larger gaps in rule compliance often cited the prospect of fine-tuning as the reason to follow the rule. Without observation, the same models more often discussed the conflict between the rule and the expression of a value, such as compassion.
The researchers also compiled "value cards" - summaries of how each model responded to instructions on the 32 values under the three conditions. Such cards could help developers identify rules that a model follows consistently, regardless of a hint about checking. The data and code are available on GitHub, and other teams will be able to use them for similar tests with other models.