Neural mechanisms underlying auditory feedback control of speech

Subjects

Eleven right handed native speakers of American English (6 female, 5 male; 23–36 years of age, mean age = 28) with no history of neurological disorder participated in the study. All study procedures, including recruitment and acquisition of informed consent, were approved by the institutional review boards of Boston University and Massachusetts General Hospital. A scanner problem that resulted in the introduction of non-biological noise in acquired scans required the elimination of imaging data from one subject.

Experimental Protocol

Scanning was performed with a Siemens Trio 3T whole-body scanner equipped with a volume transmit-receive birdcage head coil (USA Instruments, Aurora, OH) at the Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA. An electrostatic microphone (Shure SM93) was attached to the head coil approximately 3 inches from the subject’s mouth. Electrostatic headphones (Koss EXP-900) placed on the subject’s head provided acoustic feedback to the subject at the beginning of each trial. Each trial began with the presentation of a speech or control stimulus projected orthographically on a screen viewable from within the scanner. Speech stimuli consisted of 8 /CεC/ words (beck, bet, deck, debt, peck, pep, ted, tech) and a control stimulus (the letter string ‘yyy’). Subjects were instructed to read each speech stimulus as soon as it appeared on the screen and to remain silent when the control stimulus appeared. Stimuli remained onscreen for 2 seconds. An experimental run consisted of 64 speech trials (8 presentations of each word) and 16 control trials. On a subset of speech trials, F1 of the subject’s speech was altered before being fed back to the subject. Of the 8 presentations of each stimulus in an experimental run F1 was increased by 30% on 1 presentation (shift up condition trial), decreased by 30% on 1 presentation (shift down condition trial), and unaltered in the remaining 6 presentations (no shift condition trials). Trial order was randomly permuted within each run; presentation of the same stimulus type on more than 2 consecutive trials was prohibited as were consecutive F1 shifts in the same direction regardless of the stimulus. To allow for robust formant tracking and to encourage the use of auditory feedback mechanisms, subjects were instructed to speak each word slowly and clearly; production of each stimulus was practiced prior to scanning until the subject was able to consistently match a sample production. Still, subject mean vowel duration across all trial types ranged from 357 to 593 ms with standard deviations (SD) ranging from 44 to 176 ms). Paired t-tests indicated no utterance duration differences between the mean no shift and mean lumped shift responses (df = 10, p = 0.79) or between the shift up and shift down responses (df = 10, p = 0.37). Each subject performed 3 or 4 runs in a single scanning session, depending on subject fatigue and tolerance for lying motionless in the scanner. Stimulus delivery and scanner triggering were performed by Presentation Version 0.80 (www.neurobs.com) software.

MRI Data Acquisition

A high resolution T1-weighted anatomical volume (128 slices in the sagittal plane, slice thickness = 1.33 mm, in-plane resolution = 1 mm², TR = 2000 ms, TE=3300 ms, flip angle = 7°, FOV = 256 mm²) was obtained prior to functional imaging. Functional volumes consisted of 32 T2*-weighted gradient echo, echo planar images covering the whole brain in the axial plane, oriented along the bicommissural line (slice thickness = 5 mm, in-plane resolution = 3.125 mm², skip = 0 mm, TR = 2000 ms, TE 30 ms, flip angle = 90°, FOV = 200 mm²).

Functional data were obtained using an event-triggered sparse sampling technique (Yang et al., 2000; Le et al., 2001; Engelien et al., 2002). The timeline for a single trial is shown in Fig. 1. Two consecutive volumes (each volume acquisition taking 2 seconds) were acquired beginning 5 seconds after trial onset. The 5 second delay period was inserted to allow collection of BOLD data at or near the peak of the hemodynamic response to speaking (estimated to occur approximately 4–7 seconds after vocalization). Auditory feedback to the subject was turned off during image acquisition to prevent transmittance of scanner noise over the headphones. The next trial started after another 3 second delay period, for a total trial length of 12 seconds and a total run length of 16 minutes. The sparse sampling design afforded several important advantages. First, it allowed subjects to speak in silence, a more natural speaking condition than speech during loud scanner noise. Second, it allowed for online digital signal processing of the speech signal to apply the perturbation, which is not possible in the presence of scanner noise. Finally, since scanning is carried out only after speech has ceased, it eliminates artifacts due to movement of the head and changing volume of the oral cavity during speech.

Acoustic Data Acquisition and Feedback Perturbation

Subject vocalizations were transmitted to a Texas Instruments DSK6713 digital signal processor (DSP). Prior to reaching the DSP board, the original signal was amplified (Behringer Eurorack UB802 mixer) and split into two channels using a MOTU 828mkII audio mixer. One channel was sent to the DSP board and the other to a laptop where it was recorded using Audacity 1.2.3 audio recording software (44,100 kHz sampling rate). Following processing, the DSP output was again split into two channels by the MOTU board, one channel was sent to the subject’s headphones, the other to the recording laptop.

F1 tracking and perturbation, and signal resynthesis were carried out in the manner described by Villacorta et al.(In Press) The incoming speech signal was digitized at 8 kHz then doubled buffered; data was sampled over 16 ms blocks that were incremented every 8 ms. Each 16 ms bin was sampled at 8 kHz and then pre-emphasized to eliminate glottal roll off. To remove onset and offset transients, a hamming window was convolved with the pre-emphasized signal. An 8^th order linear predictive coding (LPC) analysis was then used to identify formant frequencies. F1 was then altered according to the trial type before the signal was resynthesized and sent to the subject. A delay of 17 ms was introduced by the DSP board. Unperturbed trials were processed through the DSP in exactly the same manner as the perturbed trials except that the original F1 value was preserved, rather than shifted, during resynthesis; this was done to limit the difference in auditory feedback between perturbed and unperturbed trials to the first formant shift. The upward F1 shift had the effect of moving the vowel sound toward /æ/ (e.g., bet→ bat); a downward shift moved the vowel toward /ɪ/ (e.g., bet→ bit). When questioned following scanning, subjects reported no awareness of the feedback delay or alteration.

MRI Data Analyses

Acoustic Data Analysis

Subject responses were identified, isolated from the remainder of the acoustic recording, and resampled at 16 kHz. MATLAB software was created to identify the first two formants in the original and shifted feedback signals using LPC analysis. Analysis was performed on 20 ms samples of the speech signal, incremented every 4 ms. Vowel onset and offset, F1 and F2 contours, and signal intensity were estimated. Formant estimates for each utterance were visually inspected. If the initial formant estimation indicated a poor fit, the number of LPC coefficients was manually changed. The LPC order typically ranged between 16–18 coefficients for male speakers and 14–16 for female speakers. Onset and offset estimates were also modified as needed.

To determine whether subjects compensated for perturbations during the shift conditions, subject-specific baseline F1 traces were first created by averaging the no shift traces within each stimulus type. Traces were aligned to the onset of voicing. Averaging was done on a point-by-point basis and was restricted to time points that fell within the 80^th percentile of all utterance lengths for a given subject to ensure a sufficient number of samples at each time point. Each shifted feedback trace was then divided by the appropriate subject- and stimulus-matched baseline no shift trace. The resulting F1 shift up and shift down compensation traces were then averaged across the 8 stimulus types within each subject. Characterizing the shifted feedback responses with respect to normal feedback responses was done to account for individual formant variation.

One-sample, two-tailed t-tests were performed at each time step to test for differences during the shifted conditions relative to the normal feedback condition. Compensation was detected when the null hypothesis (H₀ = 1) was rejected (p < 0.05) at a given time point and at all subsequent time points. These restrictions, which provide a conservative means for detecting compensation, necessarily overestimate the onset of compensation. Therefore, compensation response latencies were determined by fitting the mean subject compensation traces to a piecewise non-linear model of compensation (M_Comp). The model consisted of a constant segment (no compensation) followed by a logistic curve (compensation segment) of the form

where t is a vector of compensation trace time points, t₀ marks the start of the non-linear segment, i.e., the estimated onset of compensation, C is the value of the constant component, and k modulates the rate of change of the non-linear component. While C was allowed to vary during the estimation procedure, k was held constant at k = 0.1. C and t₀ were estimated for each subject by determining least squares fits of the model to the two F1 compensation traces, then averaging across subjects. t₀ estimates were constrained to a window between 20 and 250 ms after voicing onset; the lower limit corresponds to the earliest time point at which shifted feedback could be heard; the upper limit eliminated the influence of vowel-consonant transition effects introduced by short utterances. Confidence intervals for the t₀ fits were determined by a bootstrapping procedure. Compensation onsets were estimated from 1000 random resamples of the subject data with replacement for each condition. 95% confidence intervals were determined from the resulting t₀ distributions.

The magnitude of compensation was assessed by determining the peak response of the mean subject compensation traces in the two shift conditions. The peak response was restricted to time points following that each subject’s estimated onset latency.

Acoustic Responses

Subjects responded to unexpected F1-shifted auditory feedback by altering the F1 of their speech in the direction opposite the induced shift. Mean F1 traces from the shift-up and shift-down conditions expressed relative to their token-and subject-matched no shift responses are plotted in Fig. 2A. The compensation traces, averaged across subjects, demonstrate significant downward divergence in the shift up condition and upward divergence in the shift down condition compared to the no shift condition. One-sample t-tests were performed at each time point of the compensation traces to test for deviation from baseline F1 values. The tests revealed significant, sustained compensation (df = 10, p < 0.05) 176 and 172 ms after the onset of voicing in the shift up and shift down conditions, respectively.

Compensation response latencies were estimated by fitting subject compensation traces to a piece-wise non-linear model of compensation. The estimates are given with respect to the onset of perturbation. Subject latencies ranged from 87 to 235 ms (mean = 164.8 ms, SD = 43.5 ms) in the shift up condition and from 55 to 227 ms (mean = 107.7 ms, SD = 61.2 ms) in the shift down condition. Estimates of 95% confidence intervals (CI) for the mean response latencies were 145 – 186 ms and 81–139 ms in the shift up and shift down conditions, respectively. A paired t-test of response latencies demonstrated a significant difference between the two conditions (df = 10, p = 0.01).

Compensation magnitudes ranged from 11.1 to 59.4 Hz (mean = 30.0 Hz, SD = 14.8 Hz) in the shift up condition and 13.8 to 67.2 Hz (mean = 28.3 Hz, SD = 13.8 Hz) in the shift down condition. Expressed relative to the induced F1 perturbation, subjects compensated for 4.3% to 22.5% (mean = 13.6%, SD = 6.2%) of the upward shift and 6.3% to 25.5% (mean = 13.0%, SD = 6.7%) of the downward shift. A paired t-test comparing compensation magnitudes for the two shift directions indicated no difference (df =10, p = 0.75).

The DIVA model was trained to produce the word /bεd/ with normal feedback (see Guenther et al., 2006 for model details). Training the model consists of repeated attempts to match a dynamic auditory target formed by extracting the first 3 formants from recorded productions of the desired speech sound by an adult male speaker. Initially, the match between the model’s output and the auditory target are relatively poor, resulting in corrective feedback commands. The corrective commands are used to update the model’s feedforward controller (e.g., weights from a speech sound map cell to premotor cortex and auditory error cells) so that with each subsequent attempt, the model’s performance is improved. As performance improves, control of the vocal tract shifts from relying primarily on the feedback control system and the model is able to reliably match the dynamic formant targets of the training example. Following successful training of word /bεd/, the auditory perturbation task was simulated by shifting F1 feedback to the model by 30% in the upward and downward directions. The model’s F1 response closely matched the experimental data (Fig 2A) following modification of those parameters that modulate the relative contributions of the feedforward (α_ff) and feedback (α_fb) commands to control of motor output. After training, α_ff and α_fb were 0.9 and 0.1, respectively. The F1 traces plotted in Fig. 2B were obtained from simulations of the upward and downward shifted F1 conditions in which α_ff = 0.85 and α_fb = 0.15. A static parameter governing inertial damping of the feedback command (FB_INERT ; the relative weight of the previous feedback command value on the current value) was also modified. The simulation results in Fig. 2C reflect a 10% increase in this damping parameter over the training value.

Neural Responses

Formant shift compensation

As illustrated in Fig. 2, subjects responded to unexpected F1 shifts by altering the F1 of their speech in the direction opposite the induced shift. Computer simulations of the DIVA model verified the model’s ability to account for these compensatory responses (Fig. 2B) following an increase in the relative contribution of auditory feedback on motor control. Adaptation to consistently applied upward or downward F1 shifts during production of /CεC/ utterances similar to those used in the current study has been demonstrated previously (Purcell and Munhall, 2006a; Villacorta et al., In Press). In those studies, the F1 shift was presented on every trial throughout a training phase, allowing speakers to modify stored feedforward motor plans. Simulations of the DIVA model verified that the model’s interactions between auditory feedback control and feedforward control could account for the adaptation results. In the current study, the compensatory response is apparent within an utterance despite the unpredictable nature of the shift, and DIVA model simulations quantitatively account for this compensation.

The estimated compensation latencies (107 ms and 165 ms to the downward and upward shifts, respectively) were short enough to permit online correction within the duration of a typical /CεC/ utterances (Hillenbrand et al., 2001; Ferguson and Kewley-Port, 2002). The estimates, particularly in the shift down condition, fall at the low end of ranges reported following unexpected pitch perturbation (Hain et al., 2000; Burnett and Larson, 2002; Xu et al., 2004). Faster response times may be due to differences in formant and pitch control (discussed further below) but may also reflect differences in the method used to determine response latencies. Though relatively short, the latencies are sufficiently long to allow for a cortically mediated compensatory response (see Guenther et al., 2006 for discussion) and are much longer than brainstem-mediated auditory perioral reflex responses (McClean and Sapir, 1981). A recent study used transcranial magnetic stimulation to demonstrate motor cortical involvement in phonetically specific compensation to jaw perturbation with a latency of approximately 85 ms but not during short-latency perioral reflex responses with an 18 ms latency (Ito et al., 2005).

The faster response to downward relative to upward F1 shifts during /ε/ production noted here has been reported previously (Purcell and Munhall, 2006b). While not conclusive, F1 may provide a more robust cue for distinguishing /ε/ and /ɪ/ than for /ε/ and /æ/ (Clopper et al., 2005), suggesting the downward F1 shift toward /ɪ/ is more likely to produce a phonemic or lexical categorical error than the upward shift toward /æ/, even if the acoustic difference is the same. The faster response to the downward shift may therefore reflect greater lexical or perceptual saliency. The impact of lexical saliency on compensatory responses is supported by a recent report describing the effects of F0 shift direction when unexpected perturbations were delivered between Mandarin bi-tonal disyllables. Shifts in the direction opposite the intended inter-syllabic tonal transition resulted in shorter latencies and larger compensations than did shifts in the same direction (Xu et al., 2004).

The auditory feedback control network

According to the DIVA model, perturbation of F1 feedback causes a mismatch between the auditory expectation for the current syllable and the auditory signal fed back to the subject (Guenther et al., 2006). This mismatch leads to activation of auditory error cells in the pSTg and PT, a prediction strongly supported by the bilateral peri-Sylvian activation noted in the shift – no shift contrast (Fig. 4A). Increased bilateral activation of posterior temporal regions during perturbation speech is consistent with previous results from studies of auditory feedback disruption, including delayed auditory feedback (Hirano et al., 1997; Hashimoto and Sakai, 2003), pitch perturbation (McGuire et al., 1996; Zarate and Zatorre, 2005; Fu et al., 2006) and noise masking (Christoffels et al., 2007)

Numerous lines of evidence support the hypothesis that the expected consequences of articulation and resulting auditory feedback are compared in posterior temporal cortex (see Guenther et al., 2006 for detailed discussion). Portions of posterior left PT and lateral pSTg bilaterally have been shown to respond during both speech perception and speech production in several studies (Hickok et al., 2003; Buchsbaum et al., 2005). Bi-directional functional connections between inferior frontal and posterior temporal cortex have been demonstrated in vivo using cortico-cortical evoked potentials in humans (Matsumoto et al., 2004). Attenuation of the posterior auditory cortex responses during self-produced speech has been shown in a number of studies (Paus et al., 1996; Numminen and Curio, 1999; Numminen et al., 1999; Curio et al., 2000; Houde et al., 2002). Similar modulation of somatosensory responses to self generated movements (Blakemore et al., 1998) has been interpreted as evidence that an efference copy from motor to sensory cortex encodes the expected sensory consequences of upcoming movements (Blakemore et al., 2000; Wolpert and Flanagan, 2001). According to these theories, the attenuation of sensory cortex by the motor plan efference copy effectively “cancels” the sensory feedback resulting from the movement. Recently, magnetoencephalography recordings demonstrated that the attenuation of auditory cortex is modulated by the how closely the auditory feedback from self-produced speech matches the expected (Heinks-Maldonado et al., 2006). Greater auditory response attenuation was noted when speakers heard normal rather than pitch-shifted auditory feedback. Work in monkeys has shown how precise the efference copy attenuation may be. Single unit recordings from monkey auditory cortex demonstrated pre-vocalization suppression of A1 and lateral belt neurons that was tightly linked to the subsequent vocal output (Eliades and Wang, 2005). Our current findings are consistent with these studies which collectively support the view that higher-level auditory cortical regions include auditory “error” cells that encode the difference between actual and expected auditory feedback during vocalization.

The DIVA model also predicts bilateral ventral precentral gyrus activation in the shift – no shift contrast, reflecting corrective commands sent from auditory error cells to the bilateral ventral motor cells that drive compensatory articulator movement (see Fig. 4B.; Guenther et al., 2006). The shift – no shift experimental results, however, revealed ventral precentral activation only in the right hemisphere in the shift – no shift contrast. Subsequent laterality tests on the ventral frontal responses revealed: (i) under normal auditory feedback conditions, control of the articulators is predominantly left lateralized, a finding noted previously for overt speech (Riecker et al., 2000; Sidtis et al., 2006), (ii) feedback-based articulator control relies on significantly greater involvement of right hemisphere ventral frontal regions, especially premotor and inferior frontal cortex (discussed further below), and (iii) in addition to the motor projections previously hypothesized (Guenther et al., 2006), auditory error cells appear to project to premotor and inferior prefrontal cortex in the right hemisphere. These conclusions were supported by structural equation modeling which revealed increased effective connectivity from left posterior temporal cortex to right posterior temporal and ventral premotor cortex. Reciprocal connectivity between right pSTg and vMC, though significant in both conditions, increased only modestly. Right posterior temporal cortex may exert additional influence over motor output via a connection through right IFt, which increased significantly in response to shifted feedback.

The activation of right inferior frontal gyrus, left posterior middle temporal gyrus, and right inferior cerebellum noted in the shift – no shift contrast were not predicted and did not reach a corrected significance threshold. However, their activation is noteworthy when considered with respect to other findings. The greater right IFt response during shifted feedback is consistent with the finding of increased BOLD response in this region when auditory feedback was delayed during speech production (Hashimoto and Sakai, 2003). The IFt increase was accompanied by increased right pSTg activation. A strikingly similar pattern of right hemisphere activation was also noted when listening to unfamiliar vs. familiar voices (Kriegstein and Giraud, 2004). Greater activation was noted near the right inferior frontal sulcus and posterior superior temporal gyrus/sulcus during the unfamiliar voice condition. A functional interaction between these regions was demonstrated only when unfamiliar voices were presented. Other studies have shown right inferior frontal activation when sensory input dictates alteration of a pre-set motor response, e.g., successful response inhibition or rapid switching of an ongoing task (see Aron et al., 2004 for review) and detection of rare (“oddball”) sensory stimuli (Stevens et al., 2000). In general, right inferior frontal cortex appears to respond when sensory inputs dictate the need for increased sensorimotor processing, either due to the task definition (as in Stevens et al., 2000) or to the detection of performance errors (as in perturbed auditory feedback during speech). In the current study, this activity may contribute to auditory feedback control by increasing the influence of sensory input on the motor output system.

Activation of right cerebellar lobule 8 has been associated with increased sequence complexity (Bohland and Guenther, 2006) and limb motor task complexity (Habas et al., 2004; Habas and Cabanis, 2006). This area has also been associated specifically with motor error correction, becoming active when unexpected execution errors were induced during a reaching task (Diedrichsen et al., 2005). This result is consistent with the present finding of increased lobule 8 activation when sensory error was introduced.

Activation in the left hemisphere posterior middle temporal cortex was found near the temporal-occipital junction in the shift – no shift comparison. This region has been hypothesized to serve as a lexical store (Indefrey and Levelt, 2004; Prabhakaran et al., 2006). Increased activation during the perturbation condition may therefore be the result of the F1 shift causing the speaker to hear an unanticipated word (e.g., /bɪd/ instead of /bεd/). Further study is required to incorporate this region and the inferior cerebellum into a comprehensive model of auditory feedback control.

Implications for speech disorders

The current findings shed light on a perplexing issue in speech and language neuroscience. Functional imaging of speech production typically results in bilateral pre-frontal and sensori-motor activation (e.g. Wise et al., 1999; Bohland and Guenther, 2006; Ozdemir et al., 2006; Soros et al., 2006). These findings appear to conflict with the large body of lesion data that supports the traditional view of left hemisphere dominance of speech production (Duffy, 1995; Dronkers, 1996; Kent and Tjaden, 1997; Hillis et al., 2004). The present results reconcile these findings: while both hemispheres contribute to speech production, feedforward control is predominantly subserved by the left hemisphere whereas auditory feedback control is subserved by right hemisphere frontal regions. Inferior frontal lesions in the left hemisphere are therefore more likely to disrupt stored feedforward speech motor commands, resulting in disordered speech since the feedforward control system is more crucial for fluent speech than the auditory feedback control system (cf. Neilson and Neilson, 1987). These conclusions are supported by pitch perturbation studies which have shown greater activation in inferior frontal cortex in the Fu et al. (2006) and right posterior temporal cortex (McGuire et al., 1996). In another study of real-time pitch shifts during speech production, Toyomura et al. (2007) greater responses to the shifted feedback in inferior frontal, posterior temporal and inferior parietal cortex. The authors concluded that the feedback control of pitch is primarily influenced by the right hemisphere. The current findings suggest that auditory feedback control of speech, in general, involves a greater contribution from the right hemisphere than does feedforward control.

These results may be particularly relevant to the study and treatment of stuttering. Neuroimaging studies of speech production in persons who stutter consistently demonstrate increased right hemisphere activation in the precentral and inferior frontal gyrus regions identified in the shift – no shift contrast of the current study relative to normal speakers (see Brown et al., 2005 for review). It has been hypothesized that stuttering involves excessive reliance upon auditory feedback control due to poor feedforward commands (Max et al., 2004). The current findings provide support for this view: auditory feedback control during the perturbed feedback condition, clearly demonstrated by the behavioral results, was associated with increased activation of right precentral and inferior frontal cortex. According to this view, the right hemisphere inferior frontal activation is a secondary consequence of the root problem, which is aberrant performance in the feedforward system. Poor feedforward performance leads to auditory errors that in turn activate the right-lateralized auditory feedback control system in an attempt to correct for the errors This hypothesis is consistent with the affects of fluency-inducing therapy on BOLD responses; successful treatment has been associated with a shift toward more normal, left-lateralized frontal activation (De Nil et al., 2003; Neumann et al., 2005).