Multimodal LLMs for emotion-aware human–robot interaction: design and implementation

Human–robot interaction (HRI) is constrained by an emotional bandwidth gap in which humans express emotion across multiple channels while robots typically monitor only one. In this study, we develop a tri-modal emotion pipeline that fuses facial, vocal, and linguistic cues on a Pepper robot, using classic per-modality models, namely mini-Xception for facial and Wav2Vec2-Large-XLSR for vocal, plus GPT-4o for linguistic appraisal over ASR transcripts and GPT-4o-mini to generate the response and behavior from the fused state. In a pilot within-subjects study (N=2), participants held two 5-minute conversations (health, education) with Pepper under (i) the embedded facial baseline and (ii) the proposed fusion. The baseline produced predominantly Neutral outputs (~70% of cycles), whereas the tri-modal system reduced Neutral to ~30% and revealed nuanced states aligned with context (e.g., concern during health talk, enthusiasm in education). The fused state drove the robot's adaptive responses (content complexity, pacing, and nonverbal behavior) in real time. These findings highlight the potential of MLLM-assisted fusion to enhance emotional understanding and responsiveness in HRI.