AIarXiv
Ranked by formulaVerdict: NotableMDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions.
VerdictWorth a reader's time today.
Abstract
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.
The editor's rubric
- Leverage4 / 5
- A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
- Magnitude2 / 5
- Solid incremental gain on a meaningful problem.
- Evidence4 / 5
- Strong: large scale, preregistered, independently replicated, or a well-powered randomized trial.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory2 / 5
- Some room to improve with obvious engineering.
- Stakes3 / 5
- Meaningful benefit to many people within a few years.
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose, we report); breadth (wide range); novelty (alternative to status quo); verification (error bars, multiple benchmarks, code released); stakes (global scale, general AI).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 6.0 / 10
- Adjusted merit
- 4.9 / 10
- Attention
- 0%
- Freshness
- 47%