Is this chart lying to me? Automating the detection of misleading visualizations

Jonathan Tonglet1,2,3, Jan Zimny1, Tinne Tuytelaars2, Iryna Gurevych1
1UKP Lab, TU Darmstadt   2Electrical Engineering, KU Leuven  3Computer Science, KU Leuven 
Paper Code arXiv

How to detect a misleading visualization?

😯 Misleading visualizations are a dangerous form of misinformation. They misrepresent data tables by breaking various chart design conventions. We need to develop methods to detect design issues in charts that make them misleading.
In this work, we introduce two datasets for the task of misleading visualization detection: Misviz and Misviz-synth.
🤖 We evaluate three type of models on both datasets: zero-shot MLLMs, rule-based linters, and fine-tuned classifiers.

The 12 categories of misleaders included in the Misviz dataset.

Abstract

Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccurate conclusions. Prior work has shown that both humans and multimodal large language models (MLLMs) are frequently deceived by such visualizations. Automatically detecting misleading visualizations and identifying the specific design rules they violate could help protect readers and reduce the spread of misinformation. However, the training and evaluation of AI models has been limited by the absence of large, diverse, and openly available datasets. In this work, we introduce Misviz, a benchmark of 2,604 real-world visualizations annotated with 12 types of misleaders. To support model training, we also release Misviz-synth, a synthetic dataset of 81,814 visualizations generated using Matplotlib and based on real-world data tables. We perform a comprehensive evaluation on both datasets using state-of-the-art MLLMs, rule-based systems, and fine-tuned classifiers. Our results reveal that the task remains highly challenging.

Misviz and Misviz-synth

We introduce two datasets. (1) Misviz-synth contains 81,814 synthetic visualizations generated with Matplotlib. It can be used to train models. (2) Misviz contains 2,604 real-world visualizations collected from various websites. It is a challenging evaluation benchmark. The image above shows the following instance of Misviz: 
{
      "image_path": "img/68718369730_misrepresentation.png",
      "image_url": "https://64.media.tumblr.com/88844d8c3be687e0549e7b7c0a403293/tumblr_mx1as48rLr1sgh0voo1_1280.jpg",
      "chart_type": [
          "bar chart",
          "pie chart"
      ],
      "misleader": [
          "misrepresentation"
      ],
      "wayback_image_url": "https://web.archive.org/web/20250619095605/https://64.media.tumblr.com/88844d8c3be687e0549e7b7c0a403293/tumblr_mx1as48rLr1sgh0voo1_1280.jpg",
      "split": "test"
  }

Models

We consider three types of baselines: zero-shot MLLMs, rule-based linters, and fine-tuned classifiers. The latter two require to extract the axis metadata from the chart as an intermediate step.

Results

💡Our experiment results reveal several key insights. (1) The task is challenging both on synthetic and real-world visualizations even for SOTA MLLMs. (2) Fine-tuned classifiers can generalize to some extent to real-world visualizations. (3) The axis extraction phase is very error prone on real-world visualizations.
Learn more by reading our paper and stay tuned for future work!

BibTeX

@article{tonglet2025misviz,
        title={Is this chart lying to me? Automating the detection of misleading visualizations},
        author={Tonglet, Jonathan and Zimny, Jan and Tuytelaars, Tinne and Gurevych, Iryna},
        journal={arXiv preprint arXiv:2508.21675},
        year={2025},
        url={https://arxiv.org/abs/2508.21675},
        doi={10.48550/arXiv.2508.21675}
}