Tyler Zhu

I am a 4th year PhD student in CS at Princeton advised by Olga Russakovsky. I work on video understanding, multimodal representation learning and reasoning, and systems & inference efficiency. I received my B.S. in EECS from Berkeley in 2022 and my M.S. in 2023, advised by Jitendra Malik. I am a recipient of the Princeton President's Fellowship.

I am interested in industry roles around pretraining, especially for multimodal understanding and post-training capabilities, post-training, and in inference and systems efficiency, as well as agent orchestration. If you are interested in collaborating, or want to reach out and chat about research or advice, feel free to reach out to me at [first][last][at]princeton[dot]edu.

CV / Google Scholar / Twitter / Github / Blog

News

[05/2026] At Meta Superintelligence Labs, NYC as a Research Scientist Intern working with Nikhila Ravi and the SAM team.
[03/2026] Video maze reasoning preprint on arXiv!
[01/2026] Video text alignment accepted to ICLR 2026! Code is also released now.
[06/2025] At Google DeepMind, London as a Student Researcher working with Viorica Pătrăucean.

[05/2025] MERV (multi-encoder VideoLLM) accepted to ICML 2025!
[05/2024] xT (large image modeling) accepted to ICML 2024!
[05/2024] Co-organizing the Transformers for Vision Workshop @ CVPR 2024.
[05/2023] PaReprop (fast reversible transformers) accepted as a CVPR 2023 workshop spotlight!
[04/2023] I am starting my PhD at Princeton in Fall 2023, advised by Professor Olga Russakovsky!

Research

I am broadly interested in creating computer vision systems which can learn from, interpret, and reason about visual data as humans do.

I believe that videos should be the fundamental unit of vision. Identifying our conceptual ``tokens'' for videos will unlock our ability to reason about vision as we have for language. From there, we can build a true interplay between the modalities. My work has explored along the entire stack, from understanding how and what models learn during training, to building models on top of higher level visual primitives, to finally understanding how these models may visually reason.

While at Berkeley, I had the great fortune of working with many wonderful people, including Karttikeya Mangalam, Alvin Wan, and Dan Hendrycks. I was also heavily involved in teaching and outreach, serving on CS 70 course staff multiple times and previously leading Machine Learning @ Berkeley. You can find out more from my old website here.

Selected | All

	Video Models Reason Early: Exploiting Plan Commitment for Maze Solving Kaleb Newman, Tyler Zhu, Olga Russakovsky Workshop on Video World Models @ CVPR 2026 arXiv / tweet Video diffusion models can solve reasoning tasks like mazes, but we know little about how they arrive at solutions. We show that these models commit to a plan within the first few denoising steps, after which generation mainly refines visual details rather than reasoning. Leveraging this insight, we introduce inference-time methods that reallocate compute toward early plan search, improving long-maze accuracy from 7% to 67% and enabling long-horizon reasoning.
	Kilometer-Vision: A New Frontier for Large-Scale Spatial Awareness in VLMs Aravindh Mahendran, Michael King, Matthew Koichi Grimes, Antoine Yang, Tyler Zhu, Joseph Heyward, Tengda Han, Shiry Ginosar, Chen Sun, Dima Damen, Simon Osindero, Noah Snavely, Simon Lynen, João Carreira, Viorica Pătrăucean ECCV* 2026 pdf / challenge We introduce the first benchmark for evaluating geographic and spatial comprehension from real-world walking-tour videos spanning kilometer-scale distances, structured around landmark, route, and map-level spatial awareness. Evaluating current VLMs shows they lean on visual pattern matching and text cues rather than genuine geometric reasoning: models recognize landmarks well but struggle to infer distances, rely on semantic labels over spatial integration, and only appear to understand maps through frame matching rather than an underlying geometric model.
	Dynamic Reflections: Probing Video Representations with Text Alignment Tyler Zhu, Tengda Han, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov ICLR 2026 code / arXiv / tweet Extending the Platonic Representation Hypothesis to dynamic inputs like videos and multi-caption text dramatically improves their alignment, up to 2x. This suggests that models are more aligned than previously thought, just that their inputs were impoverished. However, representations are not temporally-sensitive yet. We also show alignment correlates well with performance on downstream video tasks as a zero-shot probe.
	Learning Dynamics of Multitask Training Data in Vision Language Models Tyler Zhu, Koome Murungi, Polina Kirichenko, Olga Russakovsky Workshop on Space in Vision, Language, and Embodied AI (SpaVLE) @ NeurIPS 2025 poster / openreview VLMs achieve great performance on a wide array of benchmarks. However, the source of this generalization is poorly understood. How do we disentangle memorization from generalization? We evaluate train & val accuracy in 1-epoch setting (seen and unseen) and break examples into visual task-specific categories for analysis. This reveals inefficiencies in question format (Bounding) and misleading task accuracies (OCR).
	Attention IoU: Examining Biases in CelebA using Attention Maps Aaron Serianni, Tyler Zhu, Olga Russakovsky, Vikram V. Ramaswamy CVPR 2025 code / arXiv We introduce Attention-IoU, a metric that uses attention maps to reveal biases within a model's internal representations, along with related scores that identify problematic image features. Validating on the synthetic Waterbirds dataset, we apply the approach to CelebA and uncover bias correlations missed by standard accuracy metrics, and show that manipulating attribute correlations in training data can expose confounding variables absent from dataset labels.
	Unifying Specialized Visual Encoders for Video Language Models Jihoon Chung, Tyler Zhu*, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, Olga Russakovsky ICML* 2025 code / arXiv / tweet We propose a framework for using many visual encoders covering broad visual categories like action recognition and spatial understanding as a unified visual encoder for video LLMs. This trend is exciting as it could allow our model to scale visual processing with the number of GPUs and run them all in parallel (sharding one expert per device) while still retaining similar runtimes to just a single expert.
	xT: Nested Tokenization for Larger Context in Large Images Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, Karttikeya Mangalam ICML* 2024 code / arXiv / tweet A simple yet effective framework for adapting vision models trained on small, 224x224 images to larger images with larger context by using linear attention encoders to integrate context over larger regions than otherwise possible. We also proposed a set of effective benchmarks for reflecting such improvements on larger images, and improve the pareto frontier of accuracy against parameters.
	PaReprop: Fast Parallelized Reversible Backpropagation Tyler Zhu, Karttikeya Mangalam Transformers for Vision Workshop @ CVPR 2023 (Spotlight Paper) code / arXiv / tweet We overcome the extra overhead of reversible transformers by parallelizing the backward pass using CUDA streams. This speeds up training for models in both vision and language, making them nearly as fast as the base models with incredible memory savings to boot.
	The many faces of robustness; A critical analysis of out-of-distribution generalization. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, Justin Gilmer ICCV 2021 code / arXiv Four new datasets measuring real-world distribution shifts, most well-known of which is ImageNet-R(enditions), as well as a new state-of-the-art data augmentation method that outperforms models pretrained with 1000x more labeled data.

Selected Writing

Have we scaled vision like language yet? — a fun investigation into how scaling in vision vs. language was like in 2022, and what it still implies for vision.
LeThoughts on JEPA: The Return of SSL — a first review of JEPA-style self-supervised learning (SSL) and my thoughts on SSL as a whole.

Misc

Preference Learning for Text-to-Image Prompt Tuning with RL
Arnav Gudibande*, Tyler Zhu*
Fall 2022 CS 285 Deep RL Final Project

We propose a framework towards automating prompt tuning for learning preferences in text-to-image synthesis using reinforcement learning with human feedback.

Service and Outreach

Guided Resource and Education Program: High School Workshop Initiative
Advisor (behind the scenes)
Machine Learning at Berkeley, Fall 2023

We piloted a free two-day workshop for local Bay Area high school students with little access to coding resources to teach them the basics of machine learning. Our goal was to be inclusive and representative of all backgrounds and experiences, and we were able to reach over 40 students evenly split between male and female participants.

Broadening Research Collaborations Workshop
Co-organizer
NeurIPS 2022

We organized a workshop at NeurIPS 2022 to bring together researchers from different backgrounds and experiences to discuss the challenges and opportunities in non-traditional collaborations beyond the standard academic and industry models.

Source code taken from Jon Barron's lovely website.