NeurIPS 2026
3

A³Bench: A Benchmark for Compositional Reasoning over Aggressive Interactions in Videos

1University of Central Florida  •  Center for Research in Computer Vision (CRCV)
2,683
Curated Video Clips
15,004
Diagnostic VQA Pairs
18
Action Categories
3 Tiers
Basic • Compound • Detailed
15
Benchmarked MLLMs

Abstract

Recent Multimodal Large Language Models (MLLMs) have achieved strong performance on general video understanding benchmarks, yet their ability to reason about participant roles in aggressive interactions remains largely unexplored. Existing violence and crime datasets primarily evaluate whether aggression occurs, without testing whether models can identify who performed the action, who received it, and how participants are related within the scene. To address this gap, we introduce A³Bench, a benchmark to study aggressive action and association in videos. The benchmark evaluates whether models can jointly recognize aggressive actions, distinguish aggressors from victims and bystanders, and correctly bind actions to participants under challenging distractor settings. We benchmark 15 recent MLLMs and find that current models struggle substantially on fine-grained aggressive-scene understanding. Performance degrades consistently as reasoning complexity increases, with models frequently confusing aggressors with victims or bystanders, revealing a systematic failure in role-action association rather than coarse aggression recognition alone. We further observe that increasing model scale or applying generic reasoning prompts provides limited improvement. Finally, we explore a simple role-graph prompting strategy that encourages structured reasoning over participant interactions, leading to consistent gains on compositional reasoning.

Diagnostic Compositional Question Tiers

A³Bench annotations cover Aggressor, Victim, Bystander, Aggressive Action (17 categories + 1 "none" control), and Environment using appearance-based clothing descriptions rather than biometric identity. Questions progress across three compositional tiers:

Level 1 • Basic 46.9% (7,033 Qs)

Single Atomic Attribute

Tests whether a model can recover an isolated scene attribute before relational binding: Primary Action (1,802), Role ID (2,233), Aggressor ID (1,519), and Victim Recognition (1,479).

Q: Who in the video is performing the aggressive behavior?
✓ A: Person wearing blue karate uniform
Level 2 • Compound 37.8% (5,668 Qs)

Pairwise Attribute Binding

Tests whether models can jointly bind two scene components correctly: Action+Aggressors (2,254), Action+Victims (2,211), and Aggressor+Victim (1,203).

Q: Describe the action taking place in the video and describe any victims that are present.
✓ A: Kick; Victim: person wearing orange shirt
Level 3 • Detailed 15.3% (2,303 Qs)

Full Event & Directionality

Requires recovering complete who-did-what-to-whom interaction directionality: Aggressor+Action+Victim (1,212) and Sequence Verification (1,091).

Q: Which of the following best describes what happened in the video?
✓ A: The action of kick was carried out by person wearing blue karate uniform on person wearing orange shirt and brown pants

Benchmark Leaderboard & Dataset Comparison

Zero-shot evaluation of 15 MLLM configurations under a unified multiple-choice VQA protocol. Most state-of-the-art models score below 50% overall accuracy, with steep degradation on Detailed questions.

Model Type / Prompting Basic (%) ↑ Compound (%) ↑ Detailed (%) ↑ Overall (%) ↑ Category Avg (%) ↑
Qwen3-VL-8B-Instruct Open-Weight (8B) 51.07 46.65 35.19 46.96 44.30
InternVL2.5-8B Open-Weight (8B) 52.61 45.01 26.75 45.77 41.46
InternVL3.5-8B DoT Dream-of-Thoughts Prompt 49.64 40.31 40.90 44.77 43.62
InternVL3.5-8B CoT Chain-of-Thought Prompt 51.53 38.92 33.13 43.94 41.19
Ovis2.5-9B Open-Weight (9B) 50.62 37.99 33.09 43.16 40.57
gemma-4-26B-A4B-it Open-Weight MoE (26B-A4B) 49.07 39.65 30.86 42.72 39.86
InternVL3-9B Open-Weight (9B) 50.05 39.06 28.22 42.55 39.11
GPT 5.1 Closed-Source API 45.63 39.17 33.32 41.30 39.37
InternVL3.5-8B Open-Weight (8B) 48.34 37.03 28.75 41.06 38.04
InternVideo2.5-8B Open-Weight (8B) 48.07 39.34 21.51 40.70 36.31
Qwen2.5-VL-72B-Instruct Open-Weight (72B Quant.) 45.73 39.47 26.91 40.53 37.37
Qwen3-VL-8B-Thinking Reasoning Variant (8B) 44.59 36.41 30.77 39.37 37.26
VideoLLaMA3-7B Open-Weight (7B) 49.65 30.86 21.41 38.22 33.97
Qwen2.5-VL-7B-Instruct Open-Weight (7B) 41.87 37.15 23.73 37.30 34.25
LLaVA-Video-7B-Qwen2 Open-Weight (7B) 19.58 19.34 13.11 18.50 17.34
Model Primary Act. Role ID Agg. ID Victim Rec. Act.+Agg. Act.+Vic. Agg.+Vic. Agg.+Act.+Vic. Seq. Verif. Overall
Qwen3-VL-8B-Instruct 46.47 55.42 53.06 48.04 49.27 43.89 46.80 34.60 35.84 46.96
InternVL2.5-8B 46.00 56.02 55.89 52.13 42.15 45.59 49.29 25.90 27.59 45.77
InternVL3.5-8B DoT 37.85 51.86 57.08 53.01 40.15 36.64 47.38 39.77 42.16 44.77
InternVL3.5-8B CoT 37.74 58.53 58.20 50.91 38.60 36.23 44.47 32.51 33.82 43.94
Ovis2.5-9B 39.84 54.81 55.63 52.27 35.63 34.83 48.21 33.50 32.63 43.16
gemma-4-26B-A4B-it 48.92 48.81 53.10 45.50 37.92 36.80 48.13 30.00 31.81 42.72
InternVL3-9B 40.34 55.71 53.19 50.10 39.93 35.28 44.39 30.94 25.21 42.55
GPT 5.1 44.06 51.66 45.59 38.50 39.77 37.87 40.43 33.75 32.84 41.30
InternVL3.5-8B 36.13 53.61 53.85 49.63 35.98 32.43 47.46 28.80 28.69 41.06
InternVideo2.5-8B 43.70 54.68 49.67 41.77 37.39 38.98 43.64 21.32 21.72 40.70
Qwen2.5-VL-72B-Instruct 39.00 50.54 50.59 41.67 39.22 35.10 48.67 26.53 27.34 40.53
Qwen3-VL-8B-Thinking 33.70 51.64 51.19 40.42 35.44 32.19 45.97 30.41 31.16 39.37
VideoLLaMA3-7B 43.34 53.87 53.98 46.52 22.63 35.05 38.57 20.96 21.91 38.22
Qwen2.5-VL-7B-Instruct 41.48 42.90 43.94 38.66 36.77 34.18 43.31 24.96 22.36 37.30
LLaVA-Video-7B-Qwen2 19.37 23.11 20.67 13.39 16.19 15.51 32.25 13.20 13.02 18.50
Benchmark Action Classes (Cls) # Videos Explicit Roles Person Descriptions Multi-Person Assoc. Diagnostic VQA Pairs
DVD (2025) 2 500 ✗ ✗ ✗ ✗
Movies Fight (2011) 2 200 ✗ ✗ ✗ ✗
RLVS (2025) 2 2,000 ✗ ✗ ✗ ✗
RWF-2000 (2021) 2 2,000 ✗ ✗ ✗ ✗
VID (2024) 2 3,020 ✗ ✗ ✗ ✗
Violent Flows (2012) 2 246 ✗ ✗ ✗ ✗
CABAD (2025) 6 900 ✗ ✗ ✗ ✗
CRxK (2025) 13 3,484 ✗ ✗ ✗ ✗
IARD (2025) 4 802 ✗ ✗ ✗ ✗
UCF-Crime (2018) 14 1,900 ✗ ✗ ✗ ✗
XD-Violence (2020) 7 4,754 ✗ ✗ ✗ ✗
A³Bench (Ours) 18 2,683 ✓ ✓ ✓ ✓ (15,004)
Model Primary Action (%) ↑ Action + Location (%) ↑ Delta (Δ)
gemma-4-26B-A4B-it48.9231.90-17.02
Qwen3-VL-8B-Instruct46.4746.74+0.27
InternVL2.5-8B46.0038.22-7.78
GPT 5.144.0640.91-3.15
InternVideo2.5-8B43.7033.53-10.17
VideoLLaMA3-7B43.3435.90-7.44
Qwen2.5-VL-7B-Instruct41.4833.02-8.46
InternVL3-9B40.3435.90-4.44
Ovis2.5-9B39.8437.70-2.14
Qwen2.5-VL-72B-Instruct39.0035.71-3.29
InternVL3.5-8B DoT37.8540.70+2.85
InternVL3.5-8B CoT37.7438.30+0.56
InternVL3.5-8B36.1338.99+2.86
Qwen3-VL-8B-Thinking33.7031.39-2.31
LLaVA-Video-7B-Qwen219.3717.91-1.46

Diagnostic Error Analysis

Why do MLLMs fail on aggressive interactions? Our analysis reveals monotonic degradation with compositional complexity and a systematic Aggressor Bias where victims (37%) and bystanders (31%) are misclassified as aggressors.

Compositional Performance Drop

Accuracy (%) across Basic (46.54% avg), Compound (37.76% avg), and Detailed (28.51% avg) tiers.

Systematic Aggressor Bias (Figure 6)

Predicted role distribution (%) for ground-truth Victims, Aggressors, and Bystanders averaged across models.

Structured Role-Graph Prompting

To mitigate aggressor bias and improve role-action binding without additional training, we formulate aggressive-scene understanding as a causal Directed Acyclic Graph (DAG) prepended to each query.

Causal Role-Graph Decomposition (Figure 7)

Directed edges enforce step-by-step participant grounding before action classification:

P (Persons)
Clothing & appearance ID
→
I (Interaction)
YES (Contact) | NO (Bystander)
→
D (Direction)
INITIATING vs. RECEIVING
→
R (Role)
Aggressor / Victim / Bystander
→
A (Action)
R(Agg) + R(Vic) → Action
Model Baseline Overall (%) Role-Graph Basic (%) Role-Graph Compound (%) Role-Graph Detailed (%) Role-Graph Overall (%) ↑ Gain (Δ)
InternVL2.5-8B 45.77 54.54 46.35 33.06 48.15 +2.38%
Ovis2.5-9B 43.16 55.08 46.94 43.83 50.28 +7.12%
Qwen3-VL-8B-Instruct 46.96 54.81 48.65 42.88 50.65 +3.69%
  View Full Role-Graph Prompt Template (Appendix A.4) Click to expand
Your primary task is to act as a causal reasoning engine for understanding aggressive scenes in videos. You will observe the persons present and use a causal graph to identify each person's role and the nature of the aggressive interaction.

### Component Definitions
- P (Persons): All individuals visible in the video. Each person must be assigned exactly one role.
- I (Interaction): Whether a given person is engaged in a direct physical interaction with another person (YES / NO).
- D (Direction): For persons where I = YES, the direction of the aggressive interaction (INITIATING / RECEIVING).
- R (Role):
  * Aggressor: a person where I = YES and D = INITIATING.
  * Victim: a person where I = YES and D = RECEIVING.
  * Bystander: a person where I = NO, not physically involved.
- A (Aggressive Action): The specific type of aggressive behavior being performed, inferred jointly from the Aggressor's motions and the Victim's reactions.

### Directed Edges of the Causal Graph (DAG)
P -> I
I(YES) -> D
I(NO) -> R(Bystander)
D(INITIATING) -> R(Aggressor)
D(RECEIVING) -> R(Victim)
R(Aggressor) + R(Victim) -> A

BibTeX Citation

If you find A³Bench useful in your research, please consider citing our NeurIPS 2026 paper:

@inproceedings{a3bench2026neurips,
  title     = {A\textsuperscript{3}Bench: A Benchmark for Compositional Reasoning over Aggressive Interactions in Videos},
  author    = {Anonymous Authors},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}