Discover, December 15, 2011
I don’t usually stream Netflix onto my television to probe the inner workings of my mind, but it had that effect not long ago. While I was catching an old episode of Law & Order: Criminal Intent, the actors’ voices lagged a fraction of a second behind the movement of their mouths, making me so disoriented it completely ruined the show. Soon my irritation turned to puzzlement, and some self-observation allowed me to track my frustration to a precise source. I didn’t care that the ominous soundtrack rose half a second late when Vincent D’Onofrio and Kathryn Erbe crept into the subway tunnel where they were about to find a body. I didn’t care that the show’s trademark duh-dung! sound marking a new scene was still duh-dung-ing after the scene started.
It was only when people talked that I went batty. I would watch the characters speak, and then I’d switch to listening to them, and then I’d watch them speak again. I just couldn’t meld the two streams of information in my head.
Thanks to Netflix, I was confronted with one of the most crucial tricks that the human brain uses to make sense of the world: combining input from all five senses into a single, coherent experience, updated many times a second in virtually real time. Because the techniques our brains use to meld the senses are far from perfect, it turns out, we can fall prey to a variety of illusions—and to maddening confusion when Netflix delivers audio and video out of sync.
Neuroscientists have come a long way since the mid-1900s, when they launched their first efforts to map out the brain’s sensory pathways. They identified regions of the brain that became active when people saw things, other regions that became active when they heard sounds, and so on through the list of senses. The implications seemed straightforward enough. Separate systems of neurons handled information from different senses. Only after each system had produced a sophisticated representation of the world did the brain combine their perceptions into one experience of reality, like a film editor adding a soundtrack to a movie.
But later scientists began to discover that the brain sometimes broke these rules. And out of that rule-breaking came some intriguing illusions. One of the most famous such illusions is known as the McGurk Effect, named for its discoverer, Harry McGurk, a developmental psychologist at the University of Surrey in England. In the 1970s he filmed people repeatedly making the sound ga. Then he had a new audio track laid over the film so that ga was replaced with the sound ba. The new audio and video were perfectly in sync. Many people who watched the movie were sure that the speakers were actually saying da, a different syllable entirely. If they closed their eyes, they heard the correct ba. When they opened their eyes, it became da again. (If you don’t know about the McGurk Effect, you may want to experience it via this very impressive video.)
Today a new generation of scientists is discovering just how powerful the McGurk Effect really is. University of California, Los Angeles, psychologist Ladan Shams has been able to create a McGurk-like illusion by displaying a flash paired with varying numbers of corresponding beeps. If Shams delivered a flash with a pair of beeps, participants were more likely to say they also saw two flashes.
These illusions provide evidence of a powerful strategy our brains use to cope with the uncertain signals we get from our senses. Certain regions of the brain take input from two or more senses and then combine them in a single sensory channel to sharpen the information overall.
Michael Beauchamp and Audrey Nath, neuroscientists at the University of Texas Health Science Center at Houston, have pinpointed one of the crucial nodes where these streams of information meet. They delivered short magnetic pulses that briefly shut down different parts of the brain. When they applied the pulses to a strip of the brain near the ear, a region called the superior temporal sulcus, the McGurk Effect was diminished.
Beauchamp and Nath followed up on that study with a new one in which they scanned people’s brains with functional magnetic resonance imaging (fMRI) as they played McGurk videos of mismatched sounds and lip movements. They found that the left superior temporal sulcus became more active in people who experienced the illusion and remained less active in those who didn’t. Beauchamp and Nath’s work suggests that when the illusion occurs, it is because the superior temporal sulcus discounts some of the signals coming from one sensory region of the brain in favor of others.
We don’t mix up our senses willy-nilly, however. There is a window of less than a tenth of a second in which a stimulus from one sense can affect the others. As my misadventure with Netflix showed, my brain was accustomed to balancing sight and sound to make sense of what people were saying without my even noticing, but the sound lag during that episode of Law and Order was so wide that the two sensory streams created confusion instead.
Sight and sound are not the only senses we mingle in our brains. What we touch can affect what we see or hear. Our very understanding of the shape of our own body can be informed not just directly, through our eyes, but also by the pressure of our feet on the ground, the stretch of ligaments in our shoulders, and the wiggle of balance-sensing nerve hairs in our inner ears. Together, our senses let us control our bodies, keeping us from falling over every time we stand up.
But this much integration comes with an astonishing ability to be duped. In 1998 Matthew Botvinick and Jonathan Cohen, two psychologists then at the University of Pittsburgh, found they could make people feel as if a rubber hand were really their own. All they had to do was put a rubber hand in front of their subjects and have them put their real hand behind a screen. The scientists simultaneously began to stroke the real hand and the fake one with paintbrushes. In a matter of seconds, people reported that the rubber hand felt as if it were part of their own body and that they even felt it being stroked.
Neuroscientist Valeria Petkova of the Karolinska Institute in Sweden expanded the rubber hand illusion in 2008. Instead of making people feel a rubber hand was part of their body, she wanted to swap entire bodies. First she had volunteers put on goggles fitted with video screens for lenses. The lenses, in turn, transmitted video from cameras positioned to correspond to a mannequin’s eyes. The cameras were pointed down at the mannequin’s body, so that the mannequin seemed to occupy the precise position of the volunteer’s own body. One scientist stroked the abdomen of the mannequin, while another stroked the abdomens of the volunteers. The effect has been described as a total body swap—the synchronized acts of stroking made participants feel as if the mannequin’s body were their own.
These results were intriguing but based largely on what the volunteers told Petkova about how they felt. That left the researchers wondering what was actually going on in their brains. To find out, they redesigned the experiment so that the volunteers lay in an fMRI scanner that recorded their brain activity. While the scan was in progress, participants looked down at their bodies and a scientist stroked their hand or abdomen. Then the scientists again took participants through the whole-body swap. Once the volunteers felt that the mannequin’s body was their own, the same brain regions became active as when they actually looked down at their own bodies.
The tricks we use to integrate our senses take time to develop. As children grow up, they get better and better at combining sights and sounds. When scientists compare children of the same age, they discover a fascinating pattern: The ones who are better at combining sights and sounds tend to score higher on intelligence tests. It’s possible, some scientists suggest, that helping children combine their senses through training exercises will enable them to do better in school. Manipulating our senses could also help people who have lost a limb. Researchers at the Rehabilitation Institute of Chicago are using the rubber hand illusion to teach amputees to feel artificial limbs as their own.
Learning how the brain mingles its senses can do more than shed light on the latest glitch in streaming video. It may also be able to help some people rewire their sense of reality for their own good.
Copyright 2011 Discover Magazine. Reprinted with permission.