The visual perception of lexical tone

Visual speech (lipreading) supports speech perception not only when the auditory signal is limited, degraded, or miss-ing, but also in mismatched auditory and visual speech component conditions when the auditory signal is clear and undegraded. While most of this research has been done with segments (consonants and vowels), research at MARCS has now provided evidence for visual speech cues for non-segmental features of language, particularly lexical tone. Early work has shown that Cantonese adults identify visual-only words differing only on tone at a rate significantly above chance; that even non- Cantonese tone (Thai) and non-tone (English) language speakers use visual information to discriminate words differing only in tone; and that one of the most likely vehicles for visual tone information is minute rigid movements of the head. In a more recent study, we investigated visual augmentation for discrimination of Mandarin tones, with F0 information degraded by using simulated cochlear implant (CI) audio. Native Mandarin and Australian English speakers were asked to discriminate between minimal pairs of Mandarin tones in five condi-tions: Auditory-Only, Auditory-Visual, CI-simulated Auditory-only, CI-simulated Auditory-Visual, and Visual-Only. Discrimination in CI-simulated audio conditions was poor compared with normal audio, but the availability of visual speech information improved discrimination in CI-simulated audio conditions, particularly on tone pairs with strong durational cues, but also for some pairs cued primarily by F0 cues. In Visual-Only, both Mandarin and Australian English speakers discriminated tones above chance, and interestingly, tone-naïve listeners outper- formed native listen-ers, suggesting firstly that visual speech information for tone is available and may be under-used by normal-hearing tone language perceivers, and secondly that the perception of such information may be language general, rather than the product of language specific learning. In a follow-up study with English-language children, it was found that point-light reductions of visual tone information did not augment tone perception, but tone perception was stronger when their pitch contours were presented as violin sounds rather than as natural speech. In future studies along this line, groups of Cantonese, Thai and Australian English children (4 to 10 years old) and adults will be tested in a dis-crimination task using auditory-only and audio-visual Cantonese and English stimuli to see how visual information (coming from both consonants and vowels or tones) is used across ages and languages.