I started from unified multimodal models that represent and generate across vision, language, audio, and video. Now I study how such models become multi-agents that ground language in shared environments to communicate, plan, and act.
Agents that communicate, cooperate, and act — grounding language in shared environments and social play.
COLM 2026
A VLM planner reasons and issues language subgoals; a learned low-level controller decides how, and for how long, to act on each one — decoupling instruction understanding from motor control.
EMNLP 2024
A task and dataset for referential communication between embodied agents whose perspectives on a shared 3D scene differ.
Models that connect vision, language, audio, and video — from unified representations to any-to-any generation.
arXiv preprint 2025
A dynamic visual tokenizer that encodes each image into a variable number of tokens matched to its information content.
ICCV Workshops 2025
An open-source drop-in replacement for CLIP-style encoders that learns fine-grained visual features while preserving global semantic alignment.
CVPR 2025
Do vision-language models see color illusions the way people do? A benchmark of photorealistic illusion scenes says: partly, and revealingly.
CVPR 2024
A multimodal LLM that follows interleaved instructions across modalities for in-context, conversational any-to-any generation and editing.
NeurIPS 2023
Composable diffusion generates any combination of output modalities from any combination of inputs, including temporally aligned video and audio.
CVPR 2023Highlight
One foundation model for document AI, unifying vision, text, and layout across understanding and generation tasks.
NeurIPS 2023
Patching action knowledge into video-language models without degrading their general vision-language ability.
WACV 2023
Iterative latent attention scales vision-and-language modeling to long videos with linear complexity.
NeurIPS 2022Oral
Vision-language learning from raw pixels and audio alone, with no text anywhere in the pipeline.
NeurIPS 2021
Distilling knowledge from a video-language teacher improves purely textual language understanding.
NAACL 2021
Dense captions and entropy minimization make noisy instructional video a usable pre-training signal.
ACL 2020
Matching dense captions and gating frame selection to localize the moments that answer a video question.
Excursions beyond the two threads above.
ACL 2021
Normalizing flows as a continuous latent generative model for language generation and semi-supervised learning.
Outside of research, I spend my time on music production and video games.