Audiovisual perception in adverse conditions : language, speaker and listener effects

This study investigated the relative contribution of auditory and visual information to speech perception by looking at the effect of visual and auditory degradation on the weighting given to visual cues for native and non-native speakers. Multiple iterations of /ba/, /da/ and /ga/ by five Australian English and five Mandarin Chinese speakers were presented to Australian English, British English and Mandarin Chinese participants. Tokens were presented in auditory, visual and congruent/incongruent audiovisual (AV) modes, either in clear or with visual degradation (blurring), auditory degradation (noise) or combined degradations. In the AV clear condition, English-speaking participants showed greater visual weighting for non-native speakers, but this was not found for Chinese participants. In 'single-channel degradation' conditions, the weighting of the intact channel increased significantly, with little influence of speaker language. There was no strong evidence of native-language effects on the weighting of visual cues. The degree of visual weighting varied widely across individual participants, and was also affected by individual speaker characteristics. The weighting of auditory and visual cues is therefore highly flexible and dependent on the information load of each channel; non-native speaker and language-background effects may influence visual weighting but individual perceiver and speaker strategies also have a strong impact.