Enhancing harmful content detection in memes using multimodal machine learning models
Loading...
Date
Authors
Advisor(s)
Co-advisor(s)
Committee members
Graduate program
Journal Title
Journal ISSN
Volume Title
Publisher
Universidade Federal do Amazonas
Type
Access rights
DOI
Resumo (Portuguese)
This thesis focuses on detecting harmful content in memes using advanced machine learning methods. It begins with a literature review, identifying the strengths, weaknesses, and challenges of current approaches while introducing a new taxonomy to facilitate method comparison. The research presents an improvement to canonical multimodal transformer models by integrating Compact Parameter Blocks into the encoder segments, achieving superior performance compared to more complex techniques. Additionally, it explores the use of generative models, such as Multimodal Large Language Models (MLLMs), to detect aggressive memes through specific prompts. The results indicate that while these models can identify harmful content, their performance declines when high-level multimodal reasoning is required. This research contributes to the field by enhancing detection methods and exploring new generative model-based approaches, aiming to create safer online environments while preserving freedom of expression.
Abstract
Memes are a popular way of sharing messages with large audiences. They are
often funny or informative, but memes can also spread hate speech and harm-
ful content. This represents a challenge for social networks, which need to detect
and moderate this type of content to provide a safer online environment. This
thesis focuses on improving the detection of harmful content in memes using ad-
vanced machine learning methods, including canonical transformer-based models
and recent multimodal large language models (MLLMs). First, the work presents
a literature review that highlights the strengths, weaknesses, and challenges of
current methods. It also introduces a new taxonomy in order to categorize current
methods, making it easier for researchers to compare and improve detection tech-
niques. Then, this work introduces a novel approach to enhance the performance
of canonical multimodal transformer models. This is done by adding a specific
module called Compact Parameter Blocks into the encoder segments of these mod-
els. The experimental results demonstrate that the proposed method outperforms
several more complex approaches. The thesis also explores the use of generative
models, such as MLLMs, to detect aggressive memes by using specific prompts
to guide the models. The results show that these models can identify aggressive
memes, despite not being explicitly designed for this task. However, the perfor-
mance of MLLMs decreases significantly when the level of multimodality reasoning
required is very high. This research contributes to the field by providing practi-
cal improvements to current detection methods and by exploring new approaches
using generative models. These advances are important in creating safer online
environments while respecting freedom of expression.
Description
Keywords
Controlled term
Sustainable Development Goals (SDG)
Citation
QUEIROZ HERMIDA, Paulo Cezar de. Enhancing harmful content detection in memes using multimodal machine learning models. 2025. 128 f. Thesis (Doctorate in Informatics) - Federal University of Amazonas, Manaus (AM), 2025.
Collections
Endorsement
Review
Supplemented By
Referenced By
Creative Commons license
Except where otherwise noted, this item's license is described as Acesso Aberto

