By: Richard Coad [MDB CCO]
When someone smiles, can you tell if it is revealing genuine happiness? Or is it a smile of frustration? A computer system developed at MIT can. It turns out that this same computer system does a better job of differentiating between smiles of happiness and smiles of frustration than humans do.
A study conducted in MIT's Media Lab has discovered that most people smile when they are frustrated, even though they don't think so.
According to Ehsan Hoque, a grad student at MIT's Media Lab and lead author of a paper on the subject, "The goal is to help people with face-to-face communication."
The experiments conducted at the Media Lab asked people to act out expressions of delight and frustration while webcams recorded their expressions. Then, they were either asked to fill out an online form designed to cause frustration, or they were invited to watch a video intended to elicit a response of delight and happiness.
Interestingly, Hoque realized that when asked to pretend to be frustrated, 90 percent of the subjects didn't smile. But when presented with a task that resulted in genuine frustration (filling in a tedious online form, only to watch the information get deleted when they pushed the "submit" button), 90 percent of them smiled.
Still images showed little differences between the frustrated smiles and the delighted smiles, but video analysis revealed the progression of the two smiles to be quite different.
Happy smiles built up gradually, while frustrated smiles appeared quickly but faded fast.
Hoque says most such research relies on acted out expressions of emotions, which may provide misleading results. "The acted data was much easier to classify accurately than the real responses," he says. But when trying to interpret images of real responses, people performed no better than chance, assessing these correctly only about 50% of the time.
"Understanding the subtleties that reveal underlying emotions is a major goal of this research, " Hoque says. "People with autism are taught that a smile means someone is happy, but research shows that it's not that simple."
Hoque also believes that timing has a lot to do with how people interpret expressions.
"Former British Prime Minister Gordon Brown was widely seen as having a phony smile," says Hoque. Similarly, a campaign commercial for former presidential candidate Herman Cain featured a smile that developed so slowly, taking nine seconds to appear, it was widely parodied, including a spoof by comedian Stephen Colbert. "Getting the timing right is very crucial if you want to be perceived as sincere and genuine with your smiles," says Hoque.
According to Jeffrey Cohn, a professor of psychology at University of Pittsburgh who was not involved in this research, this work "breaks new ground with its focus on frustration, a fundamental human experience. While pain researchers have identified smiling in the context of expressions of pain, the MIT group may be the first to implicate smiles in expression of negative emotion."
Cohn adds, "This is very exciting work in computational behavioral science that integrates psychology, computer vision, speech processing and machine learning to generate new knowledge...with clinical implications." Furthermore, he says this "is an important reminder that not all smiles are positive. There has been a tendency to read enjoyment whenever smiles are found. For human-computer interaction, among other fields and applications, a more nuanced view is needed."
In addition to providing training for people who have difficulty with expressions, Hoque feels the findings may be quite interesting to marketers. "Just because a customer is smiling, that doesn't necessarily mean they're satisfied." Knowing the difference could be important in gauging how best to respond to the customer, he says. "The underlying meaning behind the smile is crucial."
This work was supported by Media Lab consortium sponsors and by Proctor & Gamble Co.