Survey · 2026 · Living reading list

A Survey on Post-training of Multimodal Large Language Models

A unified behavior-shaping view of how pretrained MLLMs become reliable, grounded, and task-oriented multimodal systems.

Haonan Zhang^1,2Pengpeng Zeng^1*Libin Cao¹Wenrui Lai¹Jinlong Li^4,5Duo Peng¹Yi Bin¹Xuanhan Wang¹Ji Zhang³Jingkuan Song¹Nicu Sebe⁴Yuchuan Wu²Yongbin Li²Heng Tao Shen¹Jieping Ye²

* Corresponding author · is.pengpengzeng@gmail.com

¹Tongji University²Qwen-Character Team, Alibaba Group³Southwest Jiaotong University⁴University of Trento⁵ETH Zürich

Explore the framework

01 / OVERVIEW

Multimodal Behavior-Shaping Loop

Multimodal pretraining builds broad perception and alignment. Post-training determines how those capabilities behave when the evidence is ambiguous, the task is complex, or the interaction has real consequences.

Overview of multimodal behavior shaping for MLLM post-training — FIG. 01Post-training algorithms steer pretrained MLLMs toward desired behaviors, while multimodal data and benchmarks close the refinement loop.

PROGRESSIVE BEHAVIOR SHAPING

MLLMs Post-training has rapidly become a central mechanism for endowing pre-trained MLLMs with the ability to exhibit more aligned and reliable behaviors, marking significant progress in multimodal intelligence.

MLLMs post-training connects multimodal learning with digital AI and physical AI, representing a core step in the progression towards Artificial General Intelligence (AGI).

Timeline of key milestones in MLLM post-training from instruction following to online-policy distillation — FIG. 02Key milestones of MLLMs post-training.

Venn diagram connecting MLLM post-training with multimodal learning, digital AI, physical AI, and AGI — FIG. 03A Venn diagram showing the interrelationships among key AI fields.

Timeline landscape of MLLM post-training research from 2023 to 2026 across five research families, datasets, and benchmarks — FIG. 04An overall landscape of MLLMs post-training research: Instruction Following, Preference Calibration, Reason Enhancement, Domain Adaptation, and Scalable Learning, with datasets and benchmarks.

02 / TAXONOMY

ACTIVATE / BEHAVIOR

Instruction Following

Visual instruction tuning aligns user intent, multimodal context, and model responses in one generative interface—from image dialogue to video, documents, and interleaved inputs.

Visual SFTData mixturesFine-grained grounding

Visual instruction-tuning pipeline for multimodal large language models — A basic view of behavior shaping for instruction following. Left: A prompt template for organizing multimodal instruction data. We use the built-in Jinja template from Hugging Face's llava-hf/llava-1.5-7b-hf as an example; it supports both training and inference. Right: The instruction-tuning paradigm used to train MLLMs. VL Inst. T: Vision-Language Instruction Tuning.

03 / EVALUATION

Desired behavior needs visible evidence.

Benchmarks define what is learned and what counts as progress—from instruction compliance and faithfulness to multimodal reasoning and domain reliability.

Evaluation system for MLLM post-training datasets and benchmarks — FIG. 05Overview of the evaluation system of MMPoT.

Instruction Following

MME · MMBench · MM-Vet · SEED-Bench · MM-IFEval · MIA-Bench

Preference Calibration

POPE · MMHal-Bench · HallusionBench · SafeBench · Multimodal RewardBench · VL-RewardBench

Reason Enhancement

MMMU · MMMU-Pro · MathVista · MathVerse · We-Math · LogicVista

Domain Adaptation

ScreenSpot · OSWorld · DocVQA · OCRBench · SLAKE · DriveLM

Reference-basedAccuracy · Exact match · F1 · mAP · IoU

Judge-basedHuman or model scoring · Pairwise preference · Win rate

04 / FUTURE DIRECTIONS

Toward dependable multimodal intelligence

Key directions for advancing MLLM post-training beyond current paradigms fall into three complementary themes: grounded behavior shaping, reliability-aware evaluation, and scaling for generalization.

Grounded behavior shaping

Post-training should learn behavior directly from multimodal evidence instead of organizing every modality around language-style supervision.

Native Multimodal Post-Training

Most pipelines eventually align visual, video, or audio inputs to text responses. Native signals should preserve spatial layout, temporal continuity, acoustic cues, and cross-modal correspondence, allowing models to learn from modality-specific structure rather than treating non-text inputs as auxiliary context.

From Digital Understanding to Physical Interaction

Future agents must move beyond understanding digital images, videos, documents, and screens. Post-training should connect perception with action-oriented reasoning in continuous environments where actions change the world and feedback can be delayed or uncertain.

Native multimodalityPhysical interaction

Reliability-aware evaluation

Evaluation must reveal whether improved scores correspond to grounded, calibrated, safe, and stable behavior.

Trustworthy Evaluation

High benchmark accuracy does not guarantee reliability. New metrics should diagnose visual grounding, calibration, consistency, safety, and robustness under distribution shift—exposing hallucination, overconfidence, shortcut reasoning, and unstable responses rather than rewarding only final-answer correctness.

Complex Real-world Scenarios

Real applications contain ambiguous evidence, long-horizon context, changing environments, and interaction constraints. Benchmarks should move beyond static images and closed-form QA so models must maintain state, handle uncertainty, and adapt across continuous multimodal inputs.

TrustworthinessRobustness

Scaling for generalization

Scaling should produce reusable behaviors that transfer across tasks, domains, modalities, and continuously changing worlds.

Post-training Scaling toward Generalist MLLMs

Progress requires more than additional data or compute. We need to understand how diverse supervision signals, model capacity, and optimization jointly drive generalization, enabling learned behaviors to be composed, reused, and adapted beyond the datasets, task formats, and reward functions seen during training.

Streaming Understanding of a Continuously Unfolding World

MLLMs need persistent memory, temporal event abstraction, selective retrieval, online feedback integration, and continual adaptation. They should absorb new experiences while preserving previous capabilities, maintaining behavioral consistency, and avoiding catastrophic forgetting or uncontrolled model drift.

Generalist MLLMsStreaming worlds

05 / CITATION

Build on the survey.

The companion repository continuously curates papers, code, datasets, and project pages across the MMPoT landscape.

Browse the reading list

BIBTEX / 2026

@article{zhang2026survey,
  title={A Survey on Post-Training of Multimodal Large Language Models},
  author={Haonan Zhang and Pengpeng Zeng and Libin Cao and Wenrui Lai and
          Jinlong Li and Duo Peng and Yi Bin and Xuanhan Wang and Ji Zhang and
          Jingkuan Song and Nicu Sebe and Yuchuan Wu and Yongbin Li and
          Heng Tao Shen and Jieping Ye},
  year={2026},
  publisher={Preprints}
}