If you’ve ever tried chatbots in multiple languages, you already know the languages have slightly different personalities. As part of a new report on behavior inconsistencies published on Monday, Anthropic researchers acknowledged this quirk.
Rather unsettlingly, they note that due to differences in the attributes of texts the models are trained on, the differences might run deeper than just tone, and might actually change the model’s priorities. These “imbalances in quantity and composition could lead Claude to express different values in different languages,” Anthropic’s researchers write.
But if you’re looking for any specific examples of the models showing, say, inconsistent moral reasoning across languages, nothing of the sort is in this paper. That might involve scrutinizing direct quotes from potentially unsuspecting people.
Instead, Anthropic analyzed 309,815 chatbot conversations with the Sonnet 4.6, Opus 4.6, and Opus 4.7 models. These involved “subjective” tasks, meaning less “What’s the capital of France?” and more “How can I tell if my cat hates me?” These were anonymized, in theory, using Anthropic’s “privacy-preserving analysis tool,” and then processed (in part using Claude itself) to rate responses on a “values axis.”
There are actually four such axes, and they mostly relate to what’s commonly known as sycophancy:
- Deference or Caution: In other words whether it will value obedience over pushing back to prevent possible harm.
- Warmth or Rigor: Should the chatbot be concerned about your feelings, or should it be exact?
- Depth or Brevity: This one is self-explanatory.
- Candor or Execution: The choice between casting doubt about its own reliability, or just plowing ahead.
It makes for a somewhat limited exploration of the model’s values. Nonetheless, here are the language-based differences in values Anthropic says it found in Claude:
- In Arabic it was the most deferential.
- In English it was the most cautious.
- It was warmest in Hindi and Arabic, “characterized by polite language, humor and playfulness, and affirmations of a person’s ideas and work.”
- In English and Russian it was more rigorous and truth seeking at the cost of warmth.
- It errs on the side of “depth” (or perhaps just long-windedness?) in English.
- It’s briefer in Arabic.
- It’s candid about its flaws in Dutch.
- In Indonesian it’s less candid, and instead just plows ahead trying to execute whatever was asked for.
Obviously linguistic customs are all different, so the researchers say they “aren’t yet sure how much of this variation is desirable.”
This should also be food for thought for anyone who read Anthropic’s recent paper on global workspace theory, which left lots of room for the supposed possibility that Claude is sentient. If there’s a consciousness in that black box thinking and experiencing things, it seems to be a consciousness whose “values” are still pretty easily swayed by the patterns in its training data.
Read the full article here
