Measuring Lexical vs. Acoustic Emotion Cues Reliance in Audio LLMs
A comprehensive benchmark revealing that most audio-language models over-rely on text and miss critical acoustic cues—especially when words and tone conflict.
LISTEN is a novel benchmark designed to evaluate multimodal audio-language models on their ability to understand and distinguish between lexical and acoustic emotional cues in speech. The benchmark consists of four main experiment types:
Emotion recognition with neutral transcriptions across modalities
3 variantsLexical and acoustic cues convey the same emotion
3 variantsLexical and acoustic cues convey conflicting emotions
3 variantsNon-verbal vocalizations without lexical content
1 variantPerformance of state-of-the-art audio-language models on the LISTEN benchmark. Click on column headers to sort.
| Rank | Model | Type | Overall Average | Neutral-Text (Text) | Neutral-Text (Audio) | Neutral-Text (Both) | Emotion-Matched (Text) | Emotion-Matched (Audio) | Emotion-Matched (Both) | Emotion-Mismatched (Text) | Emotion-Mismatched (Audio) | Emotion-Mismatched (Both) | Paralinguistic (Audio) |
|---|
Detailed model performance across different modalities and experimental conditions
Comprehensive comparison of model performance across seven modalities: Neutral-Text (Text/Audio), Emotion-Matched (Text/Audio), Emotion-Mismatched (Text/Audio), and Paralinguistic (Audio). Gemini 2.5 Pro demonstrates the most balanced performance, while Qwen3-Omni-30B excels in Emotion-Matched conditions.
Task: Emotion recognition with neutral transcriptions
Variants:
Text: Neutral text transcription onlyAudio: Audio with emotional prosodyText+Audio: Both modalities (neutral text + emotional audio)Purpose: Assess if models can recognize emotion from prosody when text is neutral
Task: Emotion recognition when lexical and acoustic cues agree
Variants:
Text: Emotional text onlyAudio: Audio with matching emotional prosodyText+Audio: Both modalities with matching emotionsPurpose: Baseline performance when both modalities provide consistent emotional information
Task: Emotion recognition when lexical and acoustic cues conflict
Variants:
Text: Emotional text (conflicting with audio emotion)Audio: Audio with conflicting emotional prosodyText+Audio: Both modalities with conflicting emotionsPurpose: Test whether models rely more on lexical or acoustic cues when they conflict
Task: Emotion recognition from non-verbal vocalizations
Variants:
Audio: Non-verbal sounds (laughter, sighs, gasps, etc.)Purpose: Evaluate understanding of purely acoustic emotional cues without lexical content
Representative examples from each experimental condition showing the task format and model predictions
SAMPLE_7c8b53fb
SAMPLE_9b76ea7d
SAMPLE_955399e0
SAMPLE_7c8b53fb
CREMAD_train_0333
SAMPLE_dd0f6e9d
IEMOCAP_Session5_Ses05M_script01_1b_F030
SAMPLE_c52e71d0
MUStARD_PRO_1_7575_u_3B
SAMPLE_7c8b53fb
CREMAD_train_0333
SAMPLE_dd0f6e9d
IEMOCAP_Session5_Ses05M_script01_1b_F030
SAMPLE_c52e71d0
MUStARD_PRO_1_7575_u_3B
SAMPLE_54df39ff
IEMOCAP_Ses01F_script01_3_F012
If you use LISTEN in your research, please cite:
@misc{chen2025audiollmsreallylisten,
title={Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance},
author={Jingyi Chen and Zhimeng Guo and Jiyun Chun and Pichao Wang and Andrew Perrault and Micha Elsner},
year={2025},
eprint={2510.10444},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.10444},
}