I love this! A model can have great accuracy and still be picking up on a characteristic that isn't relevant to the question. What we have to ask, in addition to the model being accurate, is "what is the model actually using to arrive at the answer?"
Clever Hans reminds me of magicians who perform card tricks. They rely on illusions as well as people's emotions for a successful magic trick. Are we being tricked by AI?
I think that, like Hans, most predictive AI models aren't designed to deliberately trick us. But once you remove all the bells and whistles that surround AI marketing, what these models actually do is perhaps not as impressive as it sounds.
So in that way, yes, it's very much like a stage magician.
I love this! A model can have great accuracy and still be picking up on a characteristic that isn't relevant to the question. What we have to ask, in addition to the model being accurate, is "what is the model actually using to arrive at the answer?"
That's context right there.
Clever Hans reminds me of magicians who perform card tricks. They rely on illusions as well as people's emotions for a successful magic trick. Are we being tricked by AI?
That's a really good comparison actually!
I think that, like Hans, most predictive AI models aren't designed to deliberately trick us. But once you remove all the bells and whistles that surround AI marketing, what these models actually do is perhaps not as impressive as it sounds.
So in that way, yes, it's very much like a stage magician.
Thank you for this and for sharing a link to your paper.