Critical Digital Literacy
The promotion of digital literacy combined with critical thinking (SubProject#2) is arguably the most efficient way to tackle “fake news” and disinformation online. It has been proven to work in Finland, where five years after introducing such programs, the country has declared victory in the fight against “fake news”. Inspired by Finland’s example, this sub-project aims at promoting critical digital literacy in Qatar. We plan to achieve this through a general media literacy platform that would teach citizens and residents of Qatar how to recognize “fake news” and propaganda techniques. The platform will have lessons and exploration capabilities. It will feature tools to analyze news, social media posts, or any custom text in Arabic and English, and it would make explicit the propaganda/persuasion techniques of the discussed issues.
The tool will look for persuasion techniques such as appeal to emotions (e.g., fear, prejudices, smears, etc.) as well as logical fallacies (e.g., black & white fallacies, bandwagon, etc.). By interacting with the platform, users will become aware of the ways they can be manipulated by “fake news”, and thus they would be less likely to act based on it and also less likely to share it further, which is critical for limiting the potential impact of organized disinformation campaigns online. We will further study the role of critical digital literacy on people’s resilience to online manipulation and influence, whether legitimate, e.g., in e-commerce, or malicious, e.g., in social engineering and phishing. This literacy will also cover understanding the influence of the algorithms and the designs used in digital media, i.e., we will go beyond the literacy of how to recognize threats and how to respond to them to understand the underlying mechanics of influence and deception online.
It can be a valid argument, typically made by social media companies, that people shall be primarily responsible for managing their traits, weaknesses, worries, stress, and jealousy, whether in physical or online worlds. Generally, self-regulation is expected from users of social media. However, we argue that social media design can become too immersive and, at times, addictive. Hence, we argue that social media shall reduce triggers leading to a loss of control over their usage. Digital addiction is associated with reduced productivity and distracting sleep. Fear of missing out (FoMO) is one manifestation of how users become overly preoccupied with online spaces. We have argued that a thoughtful design process shall equip users with tools to manage it, e.g. creative versions of auto-reply, coloring schemes, and filters. Such a design can benefit those who are highly susceptible to peer pressure and possess low impulse control.
Objectives
To build a high-quality corpus annotated with propaganda and its techniques.
To develop a system for detecting the use of propaganda and its techniques in text in Arabic and English with a focus on Qatar and social media.
To develop an online platform for teaching critical digital literacy and then use the platform to study the role of critical digital literacy on people’s resilience to online manipulation and influence.
Meet Critical Digital Literacy Team members...
FIROJ ALAM
Critical Digital Literacy
WAJDI ZAGHOUANI
GEORGE MIKROS
GIOVANNI DA SAN MARTINO
MARAM HASANAIN
FATEMA AHMAD
ELISA SARTORI
University of Padova
MUAADH NOMAN
Publications
Overview of the CLEF-2023 CheckThat! Lab Task 3 on Political Bias of News Articles and News Media Proceedings Article
In: Aliannejadi, Mohammad; Faggioli, Guglielmo; Ferro, Nicola; Vlachos,; Michalis, (Ed.): Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, 2023.
Overview of the CLEF-2023 CheckThat! Lab Task 4 on Factuality of Reporting of News Media Proceedings Article
In: Aliannejadi, Mohammad; Faggioli, Guglielmo; Ferro, Nicola; Vlachos,; Michalis, (Ed.): Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, 2023.
Benchmarking arabic ai with large language models Journal Article
In: arXiv preprint arXiv:2305.14982, 2023.
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking Journal Article
In: arXiv preprint arXiv:2308.04945, 2023.
In: Behaviour & Information Technology, pp. 1–21, 2023.
Educational Material
Introduction to Critical Digital Literacy
Download the booklet: Introduction to Critical Digital literacy
Critical digital literacy is essential in today’s world, where the internet and social media are the primary sources of information and communication. As mentioned before, there are many harmful online content that can sway opinions and actions. Hence, fostering critical digital literacy skills is vital in combating the spread of fake news, harmful stereotypes, and divisive narratives. Learn more about it in the attached booklet.
Propaganda
Download the booklet: Propaganda
It is important to learn what propaganda is as a part of Critical Digital Literacy to create a safer online space, where we engage with the digital world critically.
Conferences
Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content
Read the full paper Here
Find the presentation slides: Here
Find the poster Here
Abstract: We present an overview of the second edition of the ArAIEval shared task, organized as part of the ArabicNLP 2024 conference co-located with ACL 2024. In this edition, ArAIEval offers two tasks: (i) detection of propagandistic textual spans with persuasion techniques identification in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 14 teams participated in the final evaluation phase, with 6 and 9 teams participating in Tasks 1 and 2, respectively. Finally, 11 teams submitted system description papers. Across both tasks, we observed that fine-tuning transformer models such as AraBERT was at the core of the majority of the participating systems. We provide a description of the task setup, including a description of the dataset construction and the evaluation setup. We further provide a brief overview of the participating systems. All datasets and evaluation scripts are released to the research community. We hope this will enable further research on these important tasks in Arabic.
Check-Worthiness, Subjectivity, Persuasion, Roles, Authorities, and Adversarial Robustness
Read the full paper Here
Find the presentation slides: Here
Abstract: We present an overview of the CheckThat! Lab 2024 Task 1, part of CLEF 2024. Task 1 involves determining whether a text item is check-worthy, with a special emphasis on COVID-19, political news, and political debates
and speeches. It is conducted in three languages: Arabic, Dutch, and English. Additionally, Spanish was offered for extra training data during the development phase. A total of 75 teams registered, with 37 teams submitting 236 runs and 17 teams submitting system description papers. Out of these, 13, 15 and 26 teams participated for Arabic, Dutch and English, respectively. Among these teams, the use of transformer pre-trained language models (PLMs) was the most frequent. A few teams also employed Large Language Models (LLMs). We provide a description of the dataset, the task setup, including evaluation settings, and a brief overview of the participating systems. As is customary in the CheckThat! Lab, we release all the datasets as well as the evaluation scripts to the research community. This will enable further research on identifying relevant check-worthy content that can assist various stakeholders, such as fact-checkers, journalists, and policymakers.
ArMeme: Propagandistic Content in Arabic Memes
Read the full paper here
Find the presentation here
Download the dataset from here
Abstract: With the rise of digital communication memes have become a significant medium for cultural and political expression that is often used to mislead audience. Identification of such misleading and persuasive multimodal content become more important among various stakeholders, including social media platforms, policymakers, and the broader society as they often cause harm to the individuals, organizations and/or society. While there has been effort to develop AI based automatic system for resource rich languages (e.g., English), it is relatively little to none for medium to low resource languages. In this study, we focused on developing an Arabic memes dataset with manual annotations of propagandistic content. We annotated ∼6K Arabic memes collected from various social media platforms, which is a first resource for Arabic multimodal research. We provide a comprehensive analysis aiming to develop computational tools for their detection. We made the dataset publicly available for the community.
Large Language Models for Propaganda Span Annotation
Read the full paper here
Find the poster here
Download the dataset from here
Abstract: The use of propagandistic techniques in online content has increased in recent years aiming to manipulate online audiences. Fine-grained propaganda detection and extraction of textual spans where propaganda techniques are used, are essential for more informed content consumption. Automatic systems targeting the task over lower resourced languages are limited, usually obstructed by lack of large scale training datasets. Our study investigates whether Large Language Models (LLMs), such as GPT-4, can effectively extract propagandistic spans. We further study the potential of employing the model to collect more cost-effective annotations. Finally, we examine the effectiveness of labels provided by GPT-4 in training smaller language models for the task. The experiments are performed over a large-scale in-house manually annotated dataset. The results suggest that providing more annotation context to GPT-4 within prompts improves its performance compared to human annotators. Moreover, when serving as an expert annotator (consolidator), the model provides labels that have higher agreement with expert annotators, and lead to specialized models that achieve state-of-the-art over an unseen Arabic testing set. Finally, our work is the first to show the potential of utilizing LLMs to develop annotated datasets for propagandistic spans detection task prompting it with annotations from human annotators with limited expertise. All scripts and annotations will be shared with the community.
Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles
Read the full paper here
Find the poster here
Download the dataset from here
Abstract: The use of propaganda has spiked on mainstream and social media, aiming to manipulate or mislead users. While efforts to automatically detect propaganda techniques in textual, visual, or multimodal content have increased, most of them primarily focus on English content. The majority of the recent initiatives targeting medium to low-resource languages produced relatively small annotated datasets, with a skewed distribution, posing challenges for the development of sophisticated propaganda detection models. To address this challenge, we carefully develop the largest propaganda dataset to date, ArPro, comprised of 8K paragraphs from newspaper articles, labeled at the text span level following a taxonomy of 23 propagandistic techniques. Furthermore, our work offers the first attempt to understand the performance of large language models (LLMs), using GPT-4, for fine-grained propaganda detection from text. Results showed that GPT-4’s performance degrades as the task moves from simply classifying a paragraph as propagandistic or not, to the fine-grained task of detecting propaganda techniques and their manifestation in text. Compared to models fine-tuned on the dataset for propaganda detection at different classification granularities, GPT-4 is still far behind. Finally, we evaluate GPT-4 on a dataset consisting of six other languages for span detection, and results suggest that the model struggles with the task across languages. We made the dataset publicly available for the community.
Persuasion Techniques and Disinformation Detection in Arabic Text
Find the poster Here
Find the presentation Here
Abstract: We present an overview of CheckThat! Lab’s 2024 Task 3, which focuses on detecting 23 persuasion techniques at the text-span level in online media. The task covers five languages, namely, Arabic, Bulgarian, English, Portuguese, and Slovene, and highly-debated topics in the media, e.g., the Isreali–Palestian conflict, the Russia– Ukraine war, climate change, COVID-19, abortion, etc. A total of 23 teams registered for the task, and two of them submitted system responses which were compared against a baseline and a task organizers’ system, which used a state-of-the-art transformer-based architecture. We provide a description of the dataset and the overall task setup, including the evaluation methodology, and an overview of the participating systems. The datasets accompanied with the evaluation scripts are released to the research community, which we believe will foster research on persuasion technique detection and analysis of online media content in various fields and contexts.
Workshops
Critique What You Read!
Find the presentation here
On the 8th of Sep, 2024, Critical Digital Literacy team and team MARSAD (sp#1), and in collaboration with QNL, held a public workshop to empower people to critique what they read. The workshop focused on the ways we can improve our consumption of news and online content, and empowered them with ways and tools to verify news and identify possible use of propagandistic techniques.
Prop2Hate-Meme
Download the dataset here
We adopted the ArMeme dataset for both fine- and coarse-grained hatefulness categorization. We preserved the original train, development, and test splits, with the test set released as dev_test. While ArMeme was initially annotated with four labels, for this study we retained only the memes labeled as propaganda and not_propaganda. These were subsequently re-annotated with hatefulness categories. The data distribution is provided below.
ArMeme
Download the dataset here
Read the paper here
ArMeme is the first multimodal Arabic memes dataset that includes both text and images, collected from various social media platforms. It serves as the first resource dedicated to Arabic multimodal research. While the dataset has been annotated to identify propaganda in memes, it is versatile and can be utilized for a wide range of other research purposes, including sentiment analysis, hate speech detection, cultural studies, meme generation, and cross-lingual transfer learning. The dataset opens new avenues for exploring the intersection of language, culture, and visual communication.
LLM_Propaganda Annotation
Download the dataset here
Read the paper here
Our study investigates whether large language models (LLMs), such as GPT-4, can effectively extract propagandistic spans. We further study the potential of employing the model to collect more cost-effective annotations. Finally, we examine the effectiveness of labels provided by GPT-4 in training smaller language models for the task. In this repo we release full human annotations, consolidated gold labels, and annotations provided by GPT-4 in different annotator roles.