Decoupling Planning and Control for Instructable Agents

Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr

COLM 2026

A VLM planner reasons and issues language subgoals; a learned low-level controller decides how, and for how long, to act on each one — decoupling instruction understanding from motor control.

Grounding Language in Multi-Perspective Referential Communication

Zineng Tang, Lingjun Mao, Alane Suhr

EMNLP 2024

A task and dataset for referential communication between embodied agents whose perspectives on a shared 3D scene differ.

arXiv paper code

Images are Worth Variable Length of Representations

Lingjun Mao, Rodolfo Corona, Xin Liang, Wenhao Yan, Zineng Tang

arXiv preprint 2025

A dynamic visual tokenizer that encodes each image into a variable number of tokens matched to its information content.

arXiv code

TULIP: Towards Unified Language-Image Pretraining

Zineng Tang, Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, David M. Chan

ICCV Workshops 2025

An open-source drop-in replacement for CLIP-style encoders that learns fine-grained visual features while preserving global semantic alignment.

arXiv code project

Evaluating Model Perception of Color Illusions in Photorealistic Scenes

Lingjun Mao, Zineng Tang, Alane Suhr

CVPR 2025

Do vision-language models see color illusions the way people do? A benchmark of photorealistic illusion scenes says: partly, and revealingly.

arXiv paper code

CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation

Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, Mohit Bansal

CVPR 2024

A multimodal LLM that follows interleaved instructions across modalities for in-context, conversational any-to-any generation and editing.

arXiv code

CoDi: Any-to-Any Generation via Composable Diffusion

Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, Mohit Bansal

NeurIPS 2023

Composable diffusion generates any combination of output modalities from any combination of inputs, including temporally aligned video and audio.

arXiv project code

Unifying Vision, Text, and Layout for Universal Document Processing

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal

CVPR 2023Highlight

One foundation model for document AI, unifying vision, text, and layout across understanding and generation tasks.

arXiv code

Paxion: Patching Action Knowledge in Video-Language Foundation Models

Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji

NeurIPS 2023

Patching action knowledge into video-language models without degrading their general vision-language ability.

arXiv code

Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention

Zineng Tang, Jaemin Cho, Jie Lei, Mohit Bansal

WACV 2023

Iterative latent attention scales vision-and-language modeling to long videos with linear complexity.

arXiv code

TVLT: Textless Vision-Language Transformer

Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal

NeurIPS 2022Oral

Vision-language learning from raw pixels and audio alone, with no text anywhere in the pipeline.

arXiv code

VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer

Zineng Tang, Jaemin Cho, Hao Tan, Mohit Bansal

NeurIPS 2021

Distilling knowledge from a video-language teacher improves purely textual language understanding.

arXiv code

DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization

Zineng Tang, Jie Lei, Mohit Bansal

NAACL 2021

Dense captions and entropy minimization make noisy instructional video a usable pre-training signal.

paper code

Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

Hyounghun Kim, Zineng Tang, Mohit Bansal

ACL 2020

Matching dense captions and gating frame selection to localize the moments that answer a video question.

paper code

Continuous Language Generative Flow

Zineng Tang, Shiyue Zhang, Hyounghun Kim, Mohit Bansal

ACL 2021

Normalizing flows as a continuous latent generative model for language generation and semi-supervised learning.

paper code