When Sinks Help or Hurt — ECCV 2026 Oral, Spotlight

Background

Two kinds of sinks

Large vision–language models concentrate attention on a few sink tokens. We separate them by origin: V-sinks emerge already inside the vision encoder and enter the LLM through the projected image tokens, while L-sinks emerge inside the LLM itself as generation proceeds through its layers. The two behave differently, and telling them apart is what makes the rest of the analysis possible.

Four 3D panels of per-dimension activation magnitudes for visual tokens at different depths. Orange ViT-emerged sink tokens and green LLM-emerged sink tokens show massive activations at specific hidden dimensions, gray ordinary tokens do not. — **Paper Fig. 3.** Activation patterns of visual tokens across the LVLM stack: orange = ViT-emerged sinks (V-sinks), green = LLM-emerged sinks (L-sinks), gray = ordinary tokens. Massive activations at specific hidden dimensions (650 for CLIP-ViT-L; 1415/2533 for the LLM) are what identify sink tokens.

The phenomenon

The same intervention helps some tasks and hurts others

We rescale the keys of V-sink tokens by a coefficient s_vit at a single LLM layer (pre-softmax, so attention stays a valid distribution) and measure the accuracy change on ten sub-tasks. Drag the sliders: some bars go up ▲, others go down ▼ — at the same layer, with the same coefficient.

LLM layer 6

V-sink key coefficient s_vit 0.5

Sub-task	Baseline (%)	Accuracy (%)	Δ (%p)

Precomputed sweep results; not live inference. LLaVA-1.5-7B, MMStar (6 sub-tasks) + CVBench (4 sub-tasks); the full grid is 32 layers × 11 coefficients, shipped as CSV in the code repo. The optimum is broad: in 73% of the 320 layer×task settings the best and second-best coefficient differ by less than 0.5%p — what matters is the direction (strengthen vs. suppress), not the exact value.

Heatmap of the best-case accuracy delta per sub-task and layer. Science and technology gains up to 12 percentage points in early-middle layers while coarse perception loses accuracy at almost every layer. — **Static counterpart (paper Fig. S4, top).** Best-case Δ per (sub-task, layer) over the 11 coefficients. Early–middle layers show the largest amplitudes in both directions.

Why

Sink tokens carry global information

Layer-wise linear probes on CLEVR scene attributes show that both V-sink and L-sink tokens encode substantially more global scene information (object counts, sizes, colors, shapes) than ordinary visual tokens at every depth — the gray band is a 5-seed ±1σ interval over random ordinary-token baselines. Sinks are not noise to be deleted; they are a channel the model actually uses, which is why blunt removal can hurt.

Four panels of layer-wise probing curves for Count, Size, Color, Shape. V-sink and L-sink curves sit well above the ordinary-token band in every panel. — **Paper Fig. 4.** Layer-wise linear probing on CLEVR scene attributes.

Method

Layer-wise Sink Gating (LSG)

If the right amount of sink influence depends on the layer, learn it per layer: LSG attaches a small MLP to each selected layer that predicts the key-scaling balance between V-sink tokens and the rest. It is trained with next-token prediction only — no task labels, and the backbone stays frozen (each gate is a few hundred KB). Learned gates land in the same regions the oracle sweep identifies, giving consistent gains without task supervision.

LSG architecture diagram: a frozen vision encoder, projector, and LLM; a small trainable two-layer MLP gate between layer l and layer l plus one predicts a softmax balance between ViT-sink and remaining tokens, which scales the keys at the next layer. — **Paper Fig. 6.** A small trainable 2-layer MLP between frozen LLM layers predicts the V-sink vs. rest balance and rescales the next layer's keys.

Heatmap of learned gate accuracy deltas across 32 layers and 10 sub-tasks with layers 2 and 10 highlighted. — **Paper Fig. 7.** Learned single-layer gate Δ (%p) across all 32 layers and 10 sub-tasks. Dashed boxes mark L2 (red) and L10 (green).

Generalization

The structure replicates beyond LLaVA-1.5

Repeating the taxonomy and the oracle sweep on LLaVA-OneVision-7B (SigLIP + Qwen2, a different vision encoder and a different LLM) reproduces the same layer- and task-dependent structure. Further cross-architecture results are in the paper's appendix.

Oracle sweep heatmap on LLaVA-OneVision showing the same layer and task dependent pattern of gains and losses. — **Paper Fig. S12 (Appendix E).** Oracle key-gating sweep on LLaVA-OneVision-7B.

Resources

Explore it yourself

Code

4-stage pipeline: sink_id → probing → sweep → lsg

Notebook (no GPU needed)

Rendered on GitHub — the full analysis from shipped CSVs

Checkpoints

4 gate-only checkpoints (<3 MB each)

Training subset

Cambrian 10k (NTP-only gate training)

Citation

BibTeX

@article{choi2026sinks,
  title={When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models},
  author={Choi, Jiho and Kim, Jaemin and Kim, Sanghwan and Hong, Seunghoon and Park, Jin-Hwi},
  journal={arXiv preprint arXiv:2604.03316},
  year={2026}
}