Measuring Lexical vs. Acoustic Emotion Cues Reliance in Audio LLMs
A comprehensive benchmark revealing that most audio-language models over-rely on text and miss critical acoustic cues—especially when words and tone conflict.
LISTEN is a novel benchmark designed to evaluate multimodal audio-language models on their ability to understand and distinguish between lexical and acoustic emotional cues in speech. The benchmark consists of four main experiment types:
Emotion recognition with neutral transcriptions across modalities
3 variantsLexical and acoustic cues convey the same emotion
3 variantsLexical and acoustic cues convey conflicting emotions
3 variantsNon-verbal vocalizations without lexical content
1 variantPerformance of state-of-the-art audio-language models on the LISTEN benchmark. Click on column headers to sort.
| Rank | Model | Type | Overall Average | Neutral-Text (Text) | Neutral-Text (Audio) | Neutral-Text (Both) | Emotion-Matched (Text) | Emotion-Matched (Audio) | Emotion-Matched (Both) | Emotion-Mismatched (Text) | Emotion-Mismatched (Audio) | Emotion-Mismatched (Both) | Paralinguistic (Audio) |
|---|
Detailed model performance across different modalities and experimental conditions
Comprehensive comparison of model performance across seven modalities: Neutral-Text (Text/Audio), Emotion-Matched (Text/Audio), Emotion-Mismatched (Text/Audio), and Paralinguistic (Audio). Gemini 2.5 Pro demonstrates the most balanced performance, while Qwen3-Omni-30B excels in Emotion-Matched conditions.
Task: Emotion recognition with neutral transcriptions
Variants:
Text: Neutral text transcription onlyAudio: Audio with emotional prosodyText+Audio: Both modalities (neutral text + emotional audio)Purpose: Assess if models can recognize emotion from prosody when text is neutral
Task: Emotion recognition when lexical and acoustic cues agree
Variants:
Text: Emotional text onlyAudio: Audio with matching emotional prosodyText+Audio: Both modalities with matching emotionsPurpose: Baseline performance when both modalities provide consistent emotional information
Task: Emotion recognition when lexical and acoustic cues conflict
Variants:
Text: Emotional text (conflicting with audio emotion)Audio: Audio with conflicting emotional prosodyText+Audio: Both modalities with conflicting emotionsPurpose: Test whether models rely more on lexical or acoustic cues when they conflict
Task: Emotion recognition from non-verbal vocalizations
Variants:
Audio: Non-verbal sounds (laughter, sighs, gasps, etc.)Purpose: Evaluate understanding of purely acoustic emotional cues without lexical content
Representative examples from each experimental condition showing the task format and model predictions
SAMPLE_7c8b53fb
SAMPLE_9b76ea7d
SAMPLE_955399e0
SAMPLE_7c8b53fb
CREMAD_train_0333
SAMPLE_dd0f6e9d
IEMOCAP_Session5_Ses05M_script01_1b_F030
SAMPLE_c52e71d0
MUStARD_PRO_1_7575_u_3B
SAMPLE_7c8b53fb
CREMAD_train_0333
SAMPLE_dd0f6e9d
IEMOCAP_Session5_Ses05M_script01_1b_F030
SAMPLE_c52e71d0
MUStARD_PRO_1_7575_u_3B
SAMPLE_54df39ff
IEMOCAP_Ses01F_script01_3_F012
If you use LISTEN in your research, please cite:
@inproceedings{chen-etal-2026-audio,
title = "Do Audio {LLM}s Really {LISTEN}, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance",
author = "Chen, Jingyi and
Guo, Zhimeng and
Chun, Jiyun and
Wang, Pichao and
Perrault, Andrew and
Elsner, Micha",
editor = "Demberg, Vera and
Inui, Kentaro and
Marquez, Llu{\'i}s",
booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
month = mar,
year = "2026",
address = "Rabat, Morocco",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.eacl-long.274/",
doi = "10.18653/v1/2026.eacl-long.274",
pages = "5848--5877",
ISBN = "979-8-89176-380-7"
}