Role of form and motion information in auditory-visual speech perception of McGurk combinations and fusions

The perception of biological motion is influenced by motion and form information. Point-light technique has been used to capture the kinematic properties of biological motion. Integration of auditory-visual information in speech perception has been shown to be influenced by such degraded forms of display. The present experiment investigates the role of global shape information and motion in multimodal speech perception. Grayscale stimuli were created from video recordings. Point-lights and point-lights joined by lines formed the stimuli that were created from motion capture data. It was hypothesized that the addition of global shape information would improve the perception of biological motion leading to a higher number of perceptual illusions and that fusion and combination McGurk effects would be identical. Twenty four Australian English subjects were asked to discriminate congruent and incongruent stimuli consisting of non-words and displayed in grayscale Video, Point-light or joined Point-light displays. Results indicate that additional global form information provided by the joint lines compared to the Point-light condition does not influence speech perception for congruent and incongruent stimuli. Nevertheless, reaction times were slower in response to this additional shape information compared with Point-light stimuli. A difference in reaction time was observed for the Video stimuli between combination and fusion responses to McGurk stimuli with subjects responding faster when the stimulus auditory /ga/ and visual /ba/ elicited a combination response /bga/ compared to the reaction time when the incongruent stimulus auditory /ba/ and visual /ga/ elicited a fusion response /da/. Fusion and Combination McGurk effects may be generated by two different perceptual processes.