Have we scaled vision like language yet?

A few years ago at our CVPR 2023 Transformers for Vision workshop, Lucas Beyer said something that struck me by surprise. I’ve been trying to piece it together ever since.

The interaction went something like this:

Audience: “Why aren’t we scaling vision models as large as we do LLMs?”

Lucas: “You know, actually, the largest vision models are on par with the largest language models if you look at [X].”

I can never quite remember what X was — FLOPs, parameters, or token budget. Obviously now it’s not parameters. The largest recorded ViTs still tap out in the 22B regime, with the most consistent scaling amounts being 1B–7B as in DINOv3.DINOv3 (Simeoni et al, 2025)

This mystery still somewhat holds true. I think about it often enough that I’ve decided to try and trace down the original point, with Gemini as my somewhat-reliable research assistant.

X marks the treasure, and potentially AGI

Among the possibilities, parameters is clearly not it. If anything, vision models are surprisingly parameter efficient for their capabilities.

I would argue that vision models haven’t found enough signal to benefit from scaling. Despite the best efforts shown in ViT-22B and 4DS, works like DINOv3 have found large vision models to be terribly unstable to train. Anecdotally, I found these larger models to have worse performance than their smaller 1B variants — but YMMV.Scaling Vision Transformers to 22 Billion Parameters (Dehghani et al, 2023) ,Scaling 4D Representations (Carreira et al, 2024) ,[2] Scaling Vision Transformers to 22 Billion Parameters (Dehghani et al, 2023)[3] Scaling 4D Representations (Carreira et al, 2024),

A better candidate is FLOPs, i.e. compute invested during training. To compare, we need to compute tokens, since:

$$\text{FLOPs} = \text{tokens} \times \text{FLOPs per token} \approx \text{tokens} \times \text{params}$$

For this comparison, I’ll sample models released before the interaction (June 2023):

• Llama-2 as the LLM.Llama 2: Open Foundation and Fine-Tuned Chat Models (Touvron et al, 2023)
• DINOv2-g, SigLIP-1, and ViT-22B as the vision encoders.DINOv2: Learning Robust Visual Features without Supervision (Oquab et al, 2024) ,Sigmoid Loss for Language Image Pre-Training (Zhai et al, 2023) ,[5] DINOv2: Learning Robust Visual Features without Supervision (Oquab et al, 2024)[6] Sigmoid Loss for Language Image Pre-Training (Zhai et al, 2023),

Large Language Model FLOPs

Llama-2 comes in three sizes identical to its predecessor: 7B, 13B, and 70B. Each is trained on 2.0T tokens (de-duplicated) with a context length of 4k.

We can estimate FLOPs as a function of the sequence length, layers, hidden dimension, and heads ($N, L, D, H$).

Per attention block:

• QKV projections: Using $Q$ as a protypical projection, we have $$ X\; [N \times D] \cdot W_Q\; [D\times D] \to Q\; [N\times D]$$ which is $2ND^2$ FLOPs. Three of them gives $6ND^2$.
• Attention scores: $Q\; [N \times D] \cdot K^T\; [D \times N] \to \text{attn}\;[N\times N]$ is $2N^2D$ FLOPs (heads don’t matter).
• Value multiplication: $[N \times N] \cdot [N \times D] \to 2N^2D$ FLOPs.
• Output projection: $[N \times D] \cdot [D \times D] \to 2ND^2$ FLOPs.

Total per attention block: $8ND^2 + 4N^2D$.

Per feed-forward block: A $[D \to 4D \to D]$ connection gives $16ND^2$.

Per transformer block: $24ND^2 + 4N^2D$. Multiply by $3BL$ for the full model (forward + backward), where $BN$ equals total tokens.

Our parameter count is $12D^2$ per block ($4D^2$ for Q,K,V,O and $8D^2$ for FFN). When $D \gg N$, we get a common approximation for total FLOPs:

$$ \text{FLOPs} \approx 72ND^2 = 6 \times \text{params} \times \text{tokens} $$

To stay consistent with how vision researchers calculate FLOPs, we’ll track just forward FLOPs (a third of the total):

$$ \text{fwd FLOPs} \approx L(24ND^2 + 4N^2D) \approx 2 \times \text{params} \times \text{tokens} $$

Below are the model stats, largely from the Llama-1 paper.LLaMA: Open and Efficient Foundation Language Models (Touvron et al, 2023) Note that $D \approx N$ here, so the approximation doesn’t hold perfectly.

One detail: Llama-2 70B uses Grouped-Query Attention, where multiple queries share the same KV vectors. With 64 query heads but only 8 KV heads, the QKV FLOPs reduce to $4.5ND^2 + 4N^2D$.

Model	$N$	$L$	$D$	$H$	tokens	FLOPs/tkn	total fwd FLOPs
Llama-2 7B	4096	32	4096	32	2.0 T	14 G	28T GFLOPs
Llama-2 13B	4096	40	5120	40	2.0 T	26.56 G	53T GFLOPs
Llama-2 70B	4096	80	8192	64	2.0 T	130 G	260T GFLOPs

Table 1. Forward compute estimates for Llama-2 models using the heuristic we derived above.

Vision Transformer FLOPs

Now the Vision Transformers. I couldn’t find enough details about SigLIP’s exact compute budget, but the “Getting your ViT in shape” paper is a great reference.Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design (Alabdulmohsin et al, 2023) I’ll substitute in the SoViT-400m/14 model.

That paper has extensive discussion of compute budgets, and shows that FLOPs correlate well with TPU core-hours — so it’s a decent proxy.

Here are the specs for the largest size of each architecture, on a constant 14×14 patch size. These are actual traced numbers, so the comparison with LLMs won’t be entirely fair — but close enough.

Model	params	input	patches	FLOPs/tkn	pretrain	tokens	total fwd FLOPs
SoViT-400m	428M	224	256	0.86 G	40B (~13e)	10.2 T	9T GFLOPs
SoViT-400m	428M	518	1369	1.00 G	6.5B (~2e)	8.9 T	9T GFLOPs
DINOv2-g	1.01B	224	256	2.08 G	7.1B (500e)	1.8 T	3.8T GFLOPs
ViT-22B	22B	224	256	40.78 G	11.5B (~3e)	2.9 T	120T GFLOPs

Table 2. Forward compute estimates for large vision encoders, based on available open-source information.

DINOv2-g uses a true ViT-g shape (slightly modified), not counting the teacher + student. ViT-22B total estimated using the formula above. DINOv2-g pretrain tokens estimated from the GitHub training recipe of 500 epochs of LVD-142M.Estimated from the GitHub training recipe: vitg14.yaml.

Interestingly, our LLM estimate isn’t far off for ViTs. The MLP expansion factor for larger ViTs is usually ~3.7 rather than 4.0. If we account for the actual expansion factor $\alpha$:

$$(8 + 4\alpha) / (4 + 2\alpha) = 2$$

So it turns out not to matter much. The remaining error resolves with the full form above.

Musings

The first thing that surprised me was how total FLOPs is the right baseline for cross-modal comparison. It’s somewhat obvious in hindsight, but coming from epoch-land in vision made it less clear at first.

Unsurprisingly, vision researchers were far from hitting the same total FLOP count as language researchers until we scaled up both model and dataset size. ViT-22B is the closest, at 120T GFLOPs — squarely in Llama territory. This makes me believe our X is indeed total FLOP count.

However, the gains weren’t as groundbreaking as what we’d expect from an LLM of similar scale. A few observations:

Not all FLOPs are created equal. ViTs will find it hard to go beyond short sequence lengths to great success. Larger images help, but with diminishing returns (see Figure 7). The signals from higher resolutions largely overlap with those present at smaller sizes — the same goes for multi-epoch training, which vision is stuck in.

We need more sources of signal, and more tokens of it. This likely means video, or joint text-image data as already done in multimodal models.

Vision tokens are less information-dense. I first heard this from Kaiming in a talk: BERT’s optimal masking ratio is 15%, compared to MAE’s 75% and ST-MAE’s 90%.BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al, 2019) ,Masked Autoencoders Are Scalable Vision Learners (He et al, 2022) ,Masked Autoencoders As Spatiotemporal Learners (Feichtenhofer et al, 2022) [9] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al, 2019)[10] Masked Autoencoders Are Scalable Vision Learners (He et al, 2022)[11] Masked Autoencoders As Spatiotemporal Learners (Feichtenhofer et al, 2022) That’s a huge gap, suggesting vision tokens are largely redundant.

This idea resurfaced in Transfusion and Chameleon, where training mixed-modality transformers was difficult due to loss imbalance across modalities. It became a focal point of their follow-up work on MoE-Sparsity.Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (Zhou et al, 2025) ,Chameleon: Mixed-Modal Early-Fusion Foundation Models (Chameleon Team, 2024) ,Improving MoE Compute Efficiency by Composing Weight and Data Sparsity (Kilian et al, 2026) [12] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (Zhou et al, 2025)[13] Chameleon: Mixed-Modal Early-Fusion Foundation Models (Chameleon Team, 2024)[14] Improving MoE Compute Efficiency by Composing Weight and Data Sparsity (Kilian et al, 2026)

Training objectives aren’t comparable. LLMs use next-token prediction (conditional soft classification), whereas ViTs use self-supervised learning and one-hot classification. It’s unclear how these objectives affect scaling, or if there’s a unifying factor.

We need smarter compute allocation. Given that model performance doesn’t differ heavily with FLOPs/token, either our tokens need to be more informative, or we need to invest compute more wisely. Adaptive compute is one direction — but I think we need to first figure out how to use more compute effectively before optimizing how we use it.

Addendum (July 2026)

Recently I had the chance to ask Lucas about this, and he confirmed that his original point was likely training tokens. My current viewpoint is similar. FLOPs and tokens are closely related either way, but the true meat of intelligence comes from the data.

He was also able to correct some of my training FLOP estimates based on the numbers found in the “Scaling ViTs” paper.Scaling vision transformers (Zhai et al, 2022) The accurate numbers for the ViT-G and ViT-g scaling is found in Table 4 of the Appendix. The former has 5M steps at a batch size of 4096, while the latter takes 6.3M steps at the same batch size. That works out to a 5.2T or 6.6T token corpus, which is on par with SoViT-400m and still larger than LLMs at the time, even accounting for extra data scaling as prescribed by the Chinchilla laws. In hindsight, my math was a bit messy in places, but we were somewhat close and it was a fun exercise to mull over the numbers.

Lucas said he’ll come out with his own post revisiting the napkin math of this era’s training FLOPs; look out for that!