Of Minds and Language (65 page)

Fig. 22.9. Grand averaged ERPs for the sentence-final word in the syntactic-only violation condition (left panel) and for the double violation condition (right panel) at electrodes F7 and Pz.

Source
: Hahne and Friederici 2002

(5)
verpflanzt
vs.
Verpflanzung

replanted vs. “replantment” (replanting)

Note that in these suffixed items, the semantic information (provided by the word stem) is available before the syntactic information (provided by the suffix). The prediction is that if syntactic structure-building precedes lexical-semantic processes functionally, even under this condition we should see only an ELAN and a P600 and no N400, and this is what we do find for the double violation condition (see
Fig. 22.10
). With the crucial syntactic information provided by the suffix, the early syntactic component (ELAN) is not that early when you time-lock it to the beginning of the word. But when you time-lock it to the beginning of the suffix providing the relevant word category information, it is early again. Thus we now can draw the conclusion that local structure-building processes precede lexical-semantic processes
functionally
.

Fig. 22.10. Grand averaged ERPs for the sentence-final word in the syntactic-only violation condition (left panel) and for the double violation condition (right panel) at electrodes FT7 and Pz.

Source
: Friederici et al. 2004

Fig. 22.11. MEG dipole localization results for five different subjects. Size indicates the strength of the dipole.

Source
: Friederici et al. 2000

When mapping the temporal ERP data onto the spatial networks data as revealed by the fMRI, there are still open questions. In the fMRI studies we have identified at least three areas that deal with syntax, the frontal operculum, the anterior portion of the STG, and the posterior portion of the STG, independent from the hierarchical processing domain. Because the temporal resolution of the fMRI is poor, we see all three areas active, and the question remains which areas support the early syntactic processes and which areas support the late processes.

In a next step we address this issue by using MEG, because with about 150 channels, this method gives us a good opportunity to do a valid dipole localization. Using the same sentence material, we tested five subjects (Friederici et al. 2000), who had to listen to 600 of those sentences in order to get a good signature noise ratio, which allowed us to look at the single subjects data. We observed an early syntax effect and the variation between the subjects is very small. The latency range is from 133 ms to 158 ms (see
Fig. 22.11
). For each subject we find two dipoles in each hemisphere, one dipole in the anterior portion of the STG and one in the vicinity of the frontal operculum. These two dipoles have to work together within this early time window, but since the dipole in the former region is larger, it appears that the contribution of the anterior portion of the STG is larger than the contribution of the frontal area. Now by simple logic one can make the argument that the posterior portion of the STG is somehow involved in the late integration processes. I do not have the time to go into this issue, but because late processes are very hard to capture with MEG, the only way for us now to test this hypothesis is to test patients with lesions in the posterior portion of the STG. These patients by hypothesis should show no P600, but instead an ELAN. And one can also do the reverse test. Patients with lesions in the inferior frontal gyrus should not have an ELAN but they do have a P600. Such patient studies are always an additional critical test. We conducted those studies with patients suffering from circumscribed brain lesions, and from these studies we can say that the early process of local structure-building is supported by these two areas, the anterior portion of the STG and the inferior frontal dipole.
³
With respect to patient studies we cannot say whether the frontal operculum or BA 44 is the crucial area (as lesions are never that specific), but given all the other studies, I would dare to hypothesize that it is the frontal operculum. With the studies I presented so far we have advanced a bit further in our description of temporal and spatial representation of these processes in the brain, at least with respect to syntactic and semantic processes.

Now let us turn to the prosodic processes assumed to be located in the RH. When we want to look at prosody during language processing, we somehow have to manipulate the language input such that we are able to look at the different parameters of prosodic information separately. One possibility is to delete the pitch (F0) contour, another one is to delete the segmental information so that only the F0 information remains. This is what we have done. In a first fMRI experiment (Meyer et al. 2004), as one condition, we had sentences in which all information types were present – namely semantic, syntactic, and prosodic information as in a normal sentence. For those who do not know German, the second condition probably sounds as good as the first one, but here no semantics is involved, just syntactic and prosodic information. In the third condition we have filtered out all segmental information. It sounds like somebody speaking next door. It is impossible to understand what is being said, but one can realize it is spoken language. We have called this prosodic speech, that is, we have taken out the segmental acoustic information from the signal, but a normal pitch contour is still present. What do we see when we are looking at the prosodic effect? In the fMRI data (
Fig. 22.12
) we see maximal activation in the RH, again in temporal structures and the frontal operculum – basically the homologue areas of what we had seen for syntactic processing, at least for local structure violations in the LH.

From these data we can at least tentatively draw some conclusions with regard to where prosodic information is processed in the RH. (Note, it is not only the RH which is active; there are also some LH structures involved, but the
maximal activation is in the RH.) Suprasegmental prosodic information elicits activation in and around the auditory cortex, that is, anterior and posterior to the auditory cortex in the STG and also the frontal operculum.

Fig. 22.12. Brain activation for prosodic effect. PT, planum temporale; FOP, frontal operculum; ROP, rolandic operculum; BG, basal ganglia.

Source
: adapted from Meyer et al. 2004

As the next issue we investigated the neural basis of the interaction between syntax and prosody. We did so by using sentence material of the following type.

(6)
Peter verspricht Anna zu arbeiten
#
und
…

‘Peter promises Anna to work and…'

(7)
Peter verspricht
#
Anna zu entlasten
#
und
…

‘Peter promises to support Anna and … '

Sentence (6) differs from sentence (7) only with respect to the following parameters. In a written form the two sentences are identical up to the word
Anna
, but auditorily they differ in their prosodic contour, that is with respect to their intonational phrase boundaries (#). In sentence (6),
Anna
is the object of
promise
, and in (7)
Anna
is the object of
support
. This is obvious in the English translation where the object always comes after the verb, but this is not the case in German. Interestingly, when we look at the electrophysiological response of the brain when just listening to these sentences, we find a positive wave after each of the intonational boundaries, which we called
Closure Positive Shift
(CPS) (see
Fig. 22.13
) (Steinhauer et al. 1999).

Fig. 22.13. Grand averaged ERPs for two sentence types time-locked to the sentence onset. IPh, intonational phrase boundary; CPS, Closure Positive Shift.

Source
: Steinhauer et al. 1999

This is only to demonstrate that the brain takes this information about intonational phrase boundaries into consideration. Note that intonational phrase boundaries are marked by three parameters: lengthening of the syllable before the intonational phrase boundary, change in the intonational (pitch) contour, and a pause. Interestingly enough, even when taking out the pause and leaving the other two relevant parameters (pre-final lengthening and shifting the intonational contour), we find the same results. Thus the adult system does not need the pause in order to realize the intonational phrase boundary.

With this result we had an index in the ERP for the processing of prosody, in particular the processing of intonational phrase boundaries. What we tried next, in order to see if and how and when syntactic and prosodic information interact, was to cross-splice sentences (6) and (7) in order to see whether we could garden-path the listener just by the prosodic information. The crucial third sentence consisted of the first part of sentence (7) (
Peter verspricht
#
Anna
) and the second part of sentence (6) (
zu arbeiten
…):

(8)
Peter verspricht
#
Anna zu arbeiten
#
und
…

‘Peter promises Anna to work and … '

This sentence now contains a verb that is not predicted, given the prosodic information of the sentence. The prediction is, if prosodic information influences syntactic processing, we expect an ERP effect on the critical verb
. The parser expects a transitive verb because of the prosodic break (#) after the first verb but encounters an intransitive verb. What we find is that the brain response first shows an N400, indicating “this is a lexical element I cannot integrate,”
and secondly it shows a P600 obviously trying “to integrate the different types of information” provided by the input (see
Fig. 22.14
).