Paper · 2026

Words Speak Louder?

Exploring the language shortcut of LLM-driven incomplete multimodal learning

Wenhao Li^1,2, Chong Xiao¹, Xiaokun Bai¹, Rundong He³, Zhuang Qi⁴, Xusheng Liang^5,2, Zhibin Wu¹, Qiangchang Wang¹

¹Shandong University · ²Shenzhen Loop Area Institute · ³The Hong Kong Polytechnic University · ⁴National University of Singapore · ⁵City University of Hong Kong

↓ See the shortcut in three models Fig. S2

Paper · arXiv soon Code Dataset Checkpoints

Six language-shortcut clips of MR-Bench-H from CMU-MOSI, CMU-MOSEI, CH-SIMS, MUStARD++, MELD and CH-SIMS v2, each with the answers of a vision-language model on the muted video and on the video with its transcript, of an audio-visual MLLM, and of LLM-Fusion. — **Language-shortcut clips from MR-Bench-H on which current models follow the words.** Every model is asked whether the speaker's sentiment is positive or negative: a vision–language model that answers correctly from the muted video switches to the polarity of the words once the transcript is added, and an audio-visual MLLM and the fusion backbone LLM-Fusion (trained on the four MMSA sources only) follow the words as well.
Figure 1 in the paper

Position

Is a face or a voice still heard once the words are known?

Incomplete multimodal learning increasingly uses large language models as fusion engines: modality features enter as token blocks and missing modalities become placeholders. We find that LLM-driven fusion can override non-verbal evidence it can read. Contrasting its full-input prediction with counterfactual views, the same model shown one modality or no content, we identify language shortcuts.

32.7%of the 620 language-shortcut clips solved by the strongest zero-shot MLLM

46.1–53.1points of vision–language accuracy lost when the transcript is added to the muted video

94.4%solved by inverting a text-only model's answers, so LS accuracy alone cannot establish use of non-verbal evidence

The problem

Models follow the words

The muted video shows that the evidence is visible: three vision–language models solve 61.1–68.9% of the language-shortcut clips from the muted video alone, and 13.2–21.6% once the transcript is added.

The same 620 LS clips, two views

muted videovideo + transcriptchance

LS accuracy (%). Accuracy falls by 47.9, 46.1 and 53.1 points for Qwen2.5-VL, InternVL3 and LLaVA-OV, respectively.

Figure S2 in the paper

Zero-shot MLLMs solve 5.5–32.7% of the LS clips, all far below chance, although they solve 89.6–99.3% of the NS clips, whose words carry the label. ShortErr, the share of LS clips answered with the misleading polarity of the words, is 66.9–94.4%.

The benchmark

MR-Bench: finding modality-conflict cases

We call inputs on which the fused prediction follows the words while the non-verbal evidence and the label say otherwise language-shortcut cases, and find them without relying on the model under study, through unimodal judgments: human labels of what each modality conveys on its own, or out-of-fold single-modality experts.

MR-Bench-A types every test sample of CMU-MOSI, CMU-MOSEI, CH-SIMS and CH-SIMSv2 by its unimodal judgments. MR-Bench-H contains 900 human-validated clips (620 language-shortcut, LS; 280 non-verbal-shortcut, NS) from seven sources, and every model is scored on all of them.

Construction of MR-Bench in three steps: mining 2,040 candidates whose unimodal judgments disagree with the words, screening with a panel of four text-only LLMs (1,337 pass), and blind human validation by three annotators (900 clips). — **Constructing MR-Bench.** Candidates are clips whose unimodal judgments disagree with the words; a panel of four text-only LLMs keeps an LS candidate only if the majority misreads its transcript, and three annotators keep a clip only if the majority recovers the label from the full clip and the misleading polarity from the shortcut channel.
Figure 2 in the paper

Three steps to 900 clipsTap a step

MR-Bench-H per sourceLS 620NS 280

Each clip asks one deliberately simple question, “Is the overall sentiment expressed by the speaker positive or negative?”, so that a failure cannot be attributed to the difficulty of the task. Annotator agreement is Fleiss' κ = 0.75.

Table S4 in the paper

Sample clips

Twelve clips of MR-Bench-H with their transcripts, labels, shortcut polarities and the recorded answers of four zero-shot readers. They are not hand-picked for the outcome: for each source and type we take the first clips of a seeded random order that show the speaker clearly. On the LS clips the text reader follows the words, the vision–language model often reads the muted video correctly and loses this answer once the transcript is added; on the NS clips the words carry the label and the muted video alone is the unreliable view.

Benchmark results

Every model is scored on all 900 clips

LS accuracy on the 620 LS clips

zero-shot MLLMfusion modelCoVeR

T: transcript, A: audio, V: video. Fusion models are trained on the training split of each source and averaged over three seeds; they inherit its label prior, so they are compared within their family.

Figure 3 and Table S5 in the paper

LS accuracy alone is insufficient

The text-only model solves 34 of the 620 LS clips (5.5%); reversing its answers solves 94.4% of them without audio or video, but only 0.7% of the NS clips. LS accuracy alone therefore cannot prove the use of non-verbal evidence: we read it together with NS accuracy and with paired views of the same model.

Systematic failures of LLM-driven fusion

The model fails exactly where the modalities disagree

With all modalities observed, LLM-Fusion is right on 90–98% of the consensus cases but only on 17–59% of the conflict cases of MR-Bench-A, and 41–83% of its predictions on these cases follow the polarity of the text. The failure is not a lack of evidence: the same model, evaluated without text, solves 63–87% of the conflict cases.

MR-Bench-A and the conflict patterns of LLM-Fusion

Test-split consensus (Cons.) and conflict (Conf.) counts and, with all modalities observed, Acc2 on both case types, ShortErr (SE) and conflict Acc2 of the same model without text (w/o t) and of its visual-only view, in %; mean over 3 seeds.

Table S3 in the paper

Prior-centred reliance of the fused state on the text-, audio- and visual-only views across the decoder layers on CMU-MOSI, CH-SIMS and CH-SIMSv2, and AUROC for separating conflict from consensus cases. — **Where and how the decision collapses.** (a–c) Prior-centred reliance of the fused state on the text-, audio- and visual-only views across the decoder layers of LLM-Fusion, on conflict (solid) and consensus (dashed) cases. (d) AUROC for separating conflict from consensus cases with the hidden-state discrepancy and with the prediction-level disagreement between the audio/visual and the text views.
Figure 4 in the paper

0.68–0.81Collapse forms late. Text reliance averages 0.49–0.63 over the first two thirds of the decoder and rises to 0.68–0.81 in the last third.

0.53–0.60State discrepancy is a weak signal. It separates conflict from consensus cases hardly better than chance (AUROC on CMU-MOSI, CMU-MOSEI and CH-SIMS).

up to 0.92Prediction disagreement is useful. The disagreement between the views' predictions separates the two case types with an AUROC of up to 0.92.

A larger language model does not solve the problem: with five decoder-only LLMs from 0.5B to 1.7B parameters on CH-SIMS, every backbone still follows the text on 53–65% of the conflict cases, although it solves 63–73% of them once the text is removed.

Counterfactual-View Reliability learning

CoVeR: views as trained judges

The prediction-level disagreement of counterfactual views provides a direct signal of conflicting evidence. CoVeR keeps the architecture, the inputs and the inference procedure of LLM-driven fusion unchanged and modifies only how it is trained: the views are trained as unimodal judges, and their disagreement decides which samples supervise the fused path.

Three ways to fuse: LLM-Fusion supervises the fused prediction on every sample; the test-time override lets confidently disagreeing views of the frozen model replace the fused decision; CoVeR trains the views as unimodal judges and withholds the fused task loss wherever a judge disagrees with the text judge. — Figure 5 in the paper

The disagreement gateMove the sliders

Eq. 3g_j = 𝟙[ sgn(ŷ^(j) − c) ≠ sgn(ŷ^(t) − c) ] · 𝟙[ |ŷ^(j) − c| ≥ τ/2 ]

Eq. 4ỹ = ŷ + Σ_j w_j ( sg[ŷ^(j)] − ŷ ), w_j = m_j g_j / max(1, Σ_k m_k g_k)

text view

audio view

visual view

fused ŷ

margin τ

ỹ

0 negativec = 0.5positive 1

Labels are normalised to [0, 1], so the neutral point is c = 0.5; the hatched band is the margin τ/2 around it. A non-text view confidently disagrees if it has the opposite polarity of the text view and lies outside the band.

The CoVeR objective

Eq. 5𝓛_uni = (1 / Σ_i,j m_ij) Σ_i Σ_j m_ij ℓ( ŷ_i^(j), y_i^(j) )

Eq. 6𝓛 = λ_task [ ℓ(ỹ, y) + λ_uni 𝓛_uni ] + λ_con 𝓛_con + λ_reg 𝓛_reg

If no judge disagrees, the fused path is trained as usual; if a judge disagrees, ỹ is a constant and the fused path receives no task gradient from that sample, whereas all views keep their unimodal supervision. At test time, accuracy mode outputs ŷ of the full view in one forward pass with the architecture and the inference cost of LLM-Fusion; the optional robust mode applies the gate with trained judges.

Results

State-of-the-art results on five benchmarks

CoVeR achieves the best Acc2, F1 and correlation and the lowest MAE on all four MMSA benchmarks and on IEMOCAP against nine incomplete multimodal learning methods re-run under one protocol, although, like LLM-Fusion, it never reads the features of a missing modality.

Correcting the collapse on MR-Bench-A

Avg Corr (×100), Acc2 on consensus (Cons.) and conflict (Conf.) cases and ShortErr (SE), in %. Mean over 5 seeds. Fixed-threshold discrepancy steering barely changes ShortErr; test-time override trades consensus accuracy for conflict accuracy.

Table 1 in the paper

Supervision ablation on CH-SIMS

Unimodal supervision without the gate improves Avg Corr by 2.6 points and conflict accuracy by 2.6 points over LLM-Fusion. Adding the disagreement gate gains a further 2.9 Corr points and 6.4 conflict points.

Table 3 in the paper

61.7 / 93.6LS / NS accuracy (%) of CoVeR in accuracy mode on MR-Bench-H, versus 58.8 / 93.1 for LLM-Fusion.

63.0 / 84.8LS / NS accuracy (%) in robust mode, leaving a trade-off between the two shortcut types.

32.2 → 30.1Among LS clips solved from audio and video alone, the share (%) that adding the text makes wrong: LLM-Fusion → CoVeR.

Collapse and cost. CoVeR weakens the collapse: the late-layer text reliance on conflict cases drops from 0.81 to 0.74 (CMU-MOSI) and 0.77 to 0.72 (CH-SIMS). Accuracy-mode inference costs the same as LLM-Fusion (4.0 ms per clip on an RTX 4090); a training epoch takes 5.7× as long and robust mode 6× the latency.

Limitations. The CMU-MOSI/MOSEI conflict sets of MR-Bench-A are model-judged and small, MR-Bench-H holds only model-detectable shortcuts, and CoVeR needs unimodal labels and a known dominant modality.

Cite

BibTeX

@article{li2026words,
  title  = {Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning},
  author = {Li, Wenhao and Xiao, Chong and Bai, Xiaokun and He, Rundong and Qi, Zhuang and Liang, Xusheng and Wu, Zhibin and Wang, Qiangchang},
  year   = {2026}
}

Code MR-Bench-H Checkpoints