Right Answer, Wrong Reason
What Clever Hans taught us about shortcut learning
In the early 1900s, a horse in Berlin became an international celebrity. His name was Hans. His owner, a retired mathematics teacher named Wilhelm von Osten, had spent years teaching him arithmetic. Ask Hans what seven plus five is, and he would tap his hoof twelve times. He could handle complex multiplications, fractions and timekeeping just as easily. Crowds gathered to watch him perform and newspapers around the world covered the story. The horse earned himself the name Clever Hans.
Not everyone was convinced. In 1904, a commission led by psychologist Carl Stumpf was assembled to investigate. It included a veterinarian, a circus manager, a cavalry officer, and several schoolteachers. They couldn’t find any sign of deceit. Von Osten took no money and allowed anyone to quiz the horse, even without himself present. The commission concluded that no fraud was involved.
They were right about the fraud, but wrong about the arithmetic. Stumpf’s student, Oskar Pfungst, designed another series of experiments. He varied two things: whether the questioner knew the answer, and whether Hans could see the questioner. When the questioner knew the answer and was visible, Hans was nearly always right. But when the questioner did not know the answer, or stood out of sight, Hans was clueless.
Hans could not count. What he could do was read people. As the tapping approached the correct number, questioners involuntarily tensed, leaned in or titled their head by a fraction. Hans had learned to stop at that exact moment for a piece of bread or a carrot. The body language was so subtle that people had no idea they were doing it. Nobody was lying on purpose. Hans had simply found a way to produce the right answers without doing anything resembling mathematics.
Machine learning experts found the phenomenon so familiar that they’ve adopted the term. The Clever Hans effect describes an AI model that produces correct outputs by relying on signs that have nothing to do with the actual task.
A well-known example comes from Lapuschkin et al., who examined how an image classification model was recognising objects. The model could identify horses with almost perfect accuracy. Explanation techniques revealed why: most of the horse photos in the training data came from the same source and had a copyright watermark in the corner of the image. The model had learned that the watermark means horse. As soon as you pasted the watermark onto the image of a car, the model assumed the car was a horse. Like Hans, the system had found a shortcut to the right answer, and that shortcut had nothing to do with the task it was designed to perform.
The pattern is so common that Geirhos et al. named it shortcut learning. A shortcut is a decision rule that performs well on the test data, but fails when the conditions change and the shortcut no longer works. A recognition model trained on a single dataset will learn that cows always appear in green fields, but won’t recognise a cow standing on a beach.
While a mislabelled cow is harmless, the same problem in real life applications is not. A study by Winkler et al. tested a deep learning model used to detect melanoma in skin images. The model had picked up on the violet ink markings that dermatologists draw near lesions they consider suspicious, and had learned that ink means melanoma. Skin marked by a doctor was flagged as malignant far more often by the model, whatever the lesion actually looked like. Without the ink “shortcut”, the model could no longer reliably detect which lesions were suspicious.
Machine learning models work by identifying patterns in data. But when a machine learning model picks up the wrong logic from the data, this can have serious consequences. Caruana et al. examined a model trained to predict which pneumonia patients were at high risk and should be admitted to hospital. The model had learned a strange rule: it had concluded that patients with asthma were at lower risk of dying, so did not need to be admitted. It is true that asthmatic pneumonia patients in the training records died less often. This is because doctors, knowing asthma makes pneumonia dangerous, sent those patients straight to intensive care. The model saw only the outcome of that protective treatment and picked up the association in reverse. Deployed as intended, the model would have sent some of the most vulnerable patients home.
This is what makes the Clever Hans effect so important for understanding AI and machine learning. The standard way to evaluate a model is to measure its accuracy on test data. But accuracy only tells us that the model produced the right answers. It tells us nothing about how it found the answer. A model that has genuinely learned the task and a model relying on a shortcut can produce identical accuracy scores. The difference only becomes clear when the shortcut is no longer there, and by then the model may be deciding who gets a loan, a job interview, or a hospital bed.
If we choose to deploy AI in contexts that affect our lives, we need to understand how a model arrives at its outputs. Explainable AI and interpretation methods exist to do exactly that: to look inside the model, and to catch a Clever Hans before we rely on its recommendations.
Announcement
This week, my paper “Designed to be trusted: anthropomorphism in AI-assisted business decisions” was published in AI and Ethics by Springer Nature.
If you do not have access to this journal, you can read the paper via this view-only link, courtesy of Springer Nature: https://rdcu.be/fyYzB
van der Linden, L. Designed to be trusted: anthropomorphism in AI-assisted business decisions. AI Ethics 6, 457 (2026). https://doi.org/10.1007/s43681-026-01306-w
Thank you for reading AI Ethics Made Practical! Subscribe for free to receive new posts and support my work.
My philosophy is that knowledge is to be shared. All articles are free to read and will stay that way.
But if you appreciate what I’m doing and would like to support my work, please consider buying me a coffee or upgrading your subscription. Becoming a supporter helps keep this publication independent and ad-free.
Help to spread the message and share this article.
References
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–1730. https://doi.org/10.1145/2783258.2788613
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673. https://doi.org/10.1038/s42256-020-00257-z
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10, 1096. https://doi.org/10.1038/s41467-019-08987-4
Pfungst, O. (1911). Clever Hans (the horse of Mr. von Osten): A contribution to experimental animal and human psychology (C. L. Rahn, Trans.). Henry Holt.
Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., & Haenssle, H. A. (2019). Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition. JAMA Dermatology, 155(10), 1135. https://doi.org/10.1001/jamadermatol.2019.1735
I love this! A model can have great accuracy and still be picking up on a characteristic that isn't relevant to the question. What we have to ask, in addition to the model being accurate, is "what is the model actually using to arrive at the answer?"
That's context right there.
Clever Hans reminds me of magicians who perform card tricks. They rely on illusions as well as people's emotions for a successful magic trick. Are we being tricked by AI?