Incognito (9 page)

Take the McGurk effect as another example: when the sound of a syllable (
ba
) is synchronized with a video of lip movements mouthing a different syllable (
ga
), it produces the powerful illusion that you are hearing yet a third syllable (
da
). This results from the dense interconnectivity and loopiness in the brain, which allows voice and lip-movement cues to become combined at an early processing stage.
⁴⁰

Vision usually dominates over hearing, but a counter example is the illusory
flash effect: when a flashed spot is accompanied by two beeps, it appears to flash twice.
⁴¹
This is related to another phenomenon called “auditory driving,” in which the apparent rate of a flickering light is driven faster or slower by an accompanying beeping sound presented at a different rate.
⁴²
Simple illusions like these serve as powerful clues into neural circuitry, telling us that the visual and auditory systems are densely tied in with each other, trying to relate a unified story of events in the world. The assembly line model of vision in introductory textbooks isn’t just misleading, it’s dead wrong.

So what is the advantage of a loopy brain? First, it permits an organism to transcend stimulus–response behavior, and instead confers the ability to make predictions ahead of actual sensory input. Think about trying to catch a fly ball. If you were merely an assembly line device, you couldn’t do it: there’d be a delay of hundreds of milliseconds from the time light strikes your retina until you could execute a motor command. Your hand would always be reaching for a place where the ball
used
to be. We’re able to catch baseballs only because we have deeply hardwired
internal models of physics.
⁴³
These internal models generate
expectations about when and where the ball will land given the effects of gravitational acceleration.
⁴⁴
The parameters of the predictive internal models are trained by lifelong exposure in normal, Earth-bound experience. This way, our brains do not work solely from the latest sensory data, but instead construct predictions about where the ball is about to be.

This is a specific example of the broader concept of internal models of the outside world. The brain internally simulates what will happen if you were to perform some action under specific conditions. Internal models not only play a role in motor acts (such as catching or dodging) but also underlie
conscious
perception
. As early as the 1940s, thinkers began to toy with the idea that perception works not by building up bits of captured data, but instead by matching
expectations
to incoming sensory data.
⁴⁵

As strange as it sounds, this framework was inspired by the observation that our expectations influence what we see. Don’t believe it? Try to discern what’s in the figure on the following page. If your brain doesn’t have a prior expectation about what the blobs mean, you simply see blobs. There has to be a match between your expectations and the incoming data for you to “see” anything.

A demonstration of the role of expectation in perception. These blobs generally have no meaning to a viewer initially, and only after a hint does the image make sense. (Don’t worry if they still look like blobs to you; a hint comes later in the chapter.) From Ahissar and Hochstein, 2004.

One of the earliest examples of this framework came from the neuroscientist
Donald MacKay, who in 1956 proposed that the
visual cortex is fundamentally a machine whose job is to generate a model of the world.
⁴⁶
He suggested that the primary visual cortex constructs an internal model that allows it to anticipate the data
streaming up from the retina (see the appendix for an anatomical guide). The cortex sends its
predictions to the thalamus, which reports on the
difference
between what comes in through the eyes and what was already anticipated. The thalamus sends back to the cortex only that difference information—that is, the bit that wasn’t predicted away. This unpredicted information adjusts the internal model so there will be less of a mismatch in the future. In this way, the brain refines its model of the world by paying attention to its mistakes. MacKay pointed out that this model is consistent with the anatomical fact that there are ten times as many fibers projecting from the primary visual cortex back to the visual thalamus as there are going the other direction—just what you’d expect if detailed expectations were sent from the cortex to the thalamus and the forward-moving information represented only a small signal carrying the difference.

What all this tells us is that perception reflects the active comparison of sensory
inputs with internal predictions. And this gives us a way to understand a bigger concept: awareness of your
surroundings occurs only when sensory inputs
violate
expectations. When the world is successfully predicted away, awareness is not needed because the brain is doing its job well. For example, when you first learn how to ride a bicycle, a great deal of conscious concentration is required; after some time, when your sensory-motor predictions have been perfected, riding becomes unconscious. I don’t mean you’re unaware that you’re riding a bicycle, but you
are
unaware of how you’re holding the handlebars, applying pressure to the pedals, and balancing your torso. From extensive experience, your brain knows exactly what to expect as you make your movements. So you’re conscious neither of the movements nor of the sensations unless something changes—like a strong wind or a flat tire. When these new situations cause your normal expectations to be violated, consciousness comes online and your internal model adjusts.

This predictability that you develop between your own actions and the resulting sensations is the reason you cannot tickle yourself. Other people can tickle you because their tickling maneuvers are not predictable to you. And if you’d really like to, there are ways to take predictability away from your own actions so that you can tickle yourself. Imagine controlling the position of a feather with a time-delay joystick: when you move the stick, at least one second passes before the feather moves accordingly. This takes away the predictability and grants you the ability to self-tickle. Interestingly, schizophrenics can tickle themselves because of a problem with their timing that does not allow their motor actions and resulting sensations to be correctly sequenced.
⁴⁷

Recognizing the brain as a loopy system with its own internal dynamics allows us to understand otherwise bizarre disorders. Take
Anton’s syndrome, a disorder in which a stroke renders a person blind—and the patient
denies
her blindness.
⁴⁸
A group of doctors will stand around the bedside and say, “Mrs. Johnson, how many of us are around your bed?” and she’ll confidently answer, “Four,” even though in fact there are seven of them. A doctor will say, “Mrs. Johnson, how many fingers am I holding up?” She’ll say,
“Three,” while in fact he is holding up none. When he asks, “What color is my shirt?” she’ll tell him it is white when it is blue. Those with Anton’s syndrome are not
pretending
they are not blind; they truly believe they are not blind. Their verbal reports, while inaccurate, are not lies. Instead, they are experiencing what they take to be vision, but it is all internally generated. Often a patient with Anton’s syndrome will not seek medical attention for a little while after the stroke, because she has no idea she is blind. It is only after bumping into enough furniture and walls that she begins to feel that something is amiss. While the patient’s answers seem bizarre, they can be understood as her internal model: the external data is not getting to the right places because of the stroke, and so the patient’s reality is simply that which is generated by the brain, with little attachment to the real world. In this sense, what she experiences is no different from dreaming, drug trips, or hallucinations.

It is not only vision and hearing that are constructions of the brain. The
perception
of time is also a construction.

When you snap your fingers, your eyes and ears register information about the snap, which is processed by the rest of the brain. But signals move fairly slowly in the brain, millions of times more slowly than electrons carrying signals in copper wire, so neural processing of the snap takes time. At the moment you perceive it, the snap has already come and gone. Your perceptual world always lags behind the real world. In other words, your
perception of the world is like a “live” television show (think
Saturday Night Live
), which is not
actually
live. Instead, these shows are aired with a delay of a few seconds, in case someone uses inappropriate language, hurts himself, or loses a piece of clothing. And so it is with your conscious life: it collects a lot of information before it airs it live.
⁴⁹

Stranger still, auditory and visual information are processed at different speeds in the brain; yet the sight of your fingers and the
sound of the snap appear simultaneous. Further, your decision to snap
now
and the action itself seem simultaneous with the moment of the snap. Because it’s important for animals to get timing right, your brain does quite a bit of fancy editing work to put the signals together in a useful way.

The bottom line is that time is a mental construction, not an accurate barometer of what’s happening “out there.” Here’s a way to prove to yourself that something strange is going on with time: look at your own eyes in a mirror and move your point of focus back and forth so that you’re looking at your right eye, then at your left eye, and back again. Your eyes take tens of milliseconds to move from one position to the other, but—here’s the mystery—you never see them move. What happens to the gaps in time while your eyes are moving? Why doesn’t your brain care about the small absences of visual input?

And the
duration of an event—how long it lasted—can be easily distorted as well. You may have noticed this upon glancing at a clock on the wall: the second hand seems to be frozen for slightly too long before it starts ticking along at its normal pace. In the laboratory, simple manipulations reveal the malleability of duration. For example, imagine I flash a square on your computer screen for half a second. If I now flash a second square that is larger, you’ll think the second one lasted longer. Same if I flash a square that’s brighter. Or moving. These will all be perceived to have a longer duration than the original square.
⁵⁰

As another example of the strangeness of time, consider how you know when you performed an action and when you sensed the consequences. If you were an engineer, you would reasonably suppose that something you do at timepoint 1 would result in
sensory
feedback at timepoint 2. So you would be surprised to discover that in the lab we can make it seem to you as though 2 happens before 1. Imagine that you can trigger a flash of light by pressing a button. Now imagine that we inject a slight delay—say, a tenth of a second—between your press and the consequent flash. After you’ve pressed the button several times, your brain adapts
to this delay, so that the two events seem slightly closer in time. Once you are adapted to the delay, we surprise you by presenting the flash immediately after you press the button. In this condition, you will believe the flash happened before your action: you experience an illusory reversal of action and sensation. The illusion presumably reflects a recalibration of motor-sensory timing which results from a prior expectation that sensory consequences should follow motor acts without delay. The best way to calibrate timing
expectations of incoming signals is to interact with the world: each time a person kicks or knocks on something, the brain can make the assumption that the sound, sight, and touch should be simultaneous. If one of the signals arrives with a delay, the brain adjusts its expectations to make it seem as though both events happened closer in time.

Interpreting the timing of motor and sensory signals is not merely a party trick of the brain; it is critical to solving the problem of causality. At bottom, causality requires a temporal order judgment: did my motor act precede or follow the sensory
input? The only way this problem can be accurately solved in a multisensory brain is by keeping the expected time of signals well calibrated, so that “before” and “after” can be accurately determined even in the face of different sensory pathways of different speeds.

Time perception is an active area of investigation in my laboratory and others, but the overarching point I want to make here is that our sense of time—how much time passed and what happened when—is constructed by our brains. And this sense is easily manipulated, just like our vision can be.