Truth on Air: Combating Misinformation in Streaming Videos with Multimodal Fact-Checking

Streaming videos, featured in platforms such as Douyin, Kuaishou, and YouTube, have emerged as a mainstream information source for the public. However, evidences have shown that viewers on these platforms are being haunted by the misinformation presented in the videos. The catastrophic effect and prevalence of these misinformation, particularly in natural disasters, health, politics, and finance, have caused severe consequences in real world. In this regard, it is imperative to address the misinformation in streaming videos.

Existing efforts against misinformation can be divided two categories: (1) Human factchecking, which remains the primary countermeasure adopted by existing video-sharing platforms. Its process, which takes up to days to complete, lacks efficiency and scalability in handling the large volumes of contents shared on the platforms; (2) Automatic factchecking. Existing methods, despite being powered by advanced machine learning techniques such as large language models, have fallen short of being able to tackle the complex, multimodal (visual, aural, and textual) contents of streaming videos. In short,
misinformation in streaming videos has yet to be satisfactorily addressed by existing techniques.

With an aim to combat misinformation for a trustworthy digital future, this project initiates to address the novel problem of multimodal fact-checking for streaming videos, which consists of the following four milestones: (1) Motivated by the lack of relevant data, to train and to evaluate solutions for this novel problem, we plan to develop the first large-scale and comprehensive dataset of fact-checking of streaming videos, featured by its multi-platform, multi-domain, and multilingual coverage of data; (2) We will design a solution for extracting check-worthy multimodal claims from the multimodal content of streaming videos, serving as the initial step towards factchecking; (3) Next, we will develop a mechanism for verifying those extracted checkworthy multimodal claims with justification; (4) At last, we aim to make available our solutions as a service (Fact-checking as a Service) which is accessible to video-sharing platforms and individuals in the form of tailored APIs and mobile application.

This project pioneers to address treating misinformation in streaming videos. Working upon the PI’s recent relevant research outcomes on misinformation and threat-resilient ML, e.g., [25-32], and extensive resources gained from being the Associate Director (with a focus on AI) of HKBU Fact Check [33], this project will strive to achieve the milestones, protecting people in Hong Kong, the Greater Bay Area, and other regions, from the severe impact caused by misinformation.