Skip to main navigation Skip to search Skip to main content

MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models

  • Zixin Chen (Co-first author)
  • , Hongzhan Lin* (Co-first author)
  • , Kaixin Li
  • , Ziyang Luo
  • , Yayue Deng
  • , Jing Ma*
  • *Corresponding author for this work

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

Abstract

The proliferation of memes on social media necessitates the capabilities of multimodal Large Language Models (mLLMs) to effectively understand multimodal harmfulness. Existing evaluation approaches predominantly focus on mLLMs’ detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. In this paper, we propose MemeArena, an agent-based arena-style evaluation framework that provides a context-aware and unbiased assessment for mLLMs’ understanding of multimodal harmfulness. Specifically, MemeArena simulates diverse interpretive contexts to formulate evaluation tasks that elicit perspective-specific analyses from mLLMs. By integrating varied viewpoints and reaching consensus among evaluators, it enables fair and unbiased comparisons of mLLMs’ abilities to interpret multimodal harmfulness. Extensive experiments demonstrate that our framework effectively reduces the evaluation biases of judge agents, with judgment results closely aligning with human preferences, offering valuable insights into reliable and comprehensive mLLM evaluations in multimodal harmfulness understanding. Our code and data are publicly available at https://github.com/Lbotirx/MemeArena.
Original languageEnglish
Title of host publicationProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
EditorsChristos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
PublisherAssociation for Computational Linguistics (ACL)
Pages17637–17659
Number of pages23
ISBN (Electronic)9798891763326
DOIs
Publication statusPublished - 4 Nov 2025
Event30th Conference on Empirical Methods in Natural Language Processing - Suzhou, China
Duration: 4 Nov 2025 → 9 Nov 2025
https://aclanthology.org/volumes/2025.findings-emnlp/ (Conference Proceedings)
https://aclanthology.org/volumes/2025.emnlp-main/ (Conference proceedings)
https://underline.io/events/502/reception (Conference website)

Conference

Conference30th Conference on Empirical Methods in Natural Language Processing
Abbreviated titleEMNLP 2025
Country/TerritoryChina
CitySuzhou
Period4/11/25 → 9/11/25
Internet address

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

  1. SDG 16 - Peace, Justice and Strong Institutions
    SDG 16 Peace, Justice and Strong Institutions

Fingerprint

Dive into the research topics of 'MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models'. Together they form a unique fingerprint.

Cite this