Lip event detection using oriented histograms of regional optical flow and low rank affinity pursuit

Xin Liu, Yiu Ming CHEUNG*, Yuan Yan Tang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)

Abstract

Lip event detection is of crucial importance to the better understanding of visual speech perceptually between humans and computers. In this paper, we address an efficient lip event detection approach using oriented histograms of regional optical flow (OH-ROF) and low rank affinity pursuit. First, we align the extracted lip region sequences to reduce the impact of irrelevant motion caused by the moving cameras. Then, an optical flow field is calculated from these sequentially stabilized images and an efficient descriptor, namely OH-ROF, is presented to discriminatively code the visual appearance of each motion frame, whereby each lip motion clip can be represented by a sequence of OH-ROF vectors as its signature. Subsequently, we detect the visual silence event based on the small flow magnitude, and further propose a low rank affinity pursuit method to determine the visual speech event that incorporates the lip-dynamic states of mouth opening and closing. As a result, various kind of lip motion events can be appropriately estimated. The proposed approach neither requires any training set on the labeled videos nor learns the lip motion priors of each visual event in an unconstrained video. Experiments show a promising result in comparison with the state-of-the-art counterparts.

Original languageEnglish
Pages (from-to)153-163
Number of pages11
JournalComputer Vision and Image Understanding
Volume148
DOIs
Publication statusPublished - 1 Jul 2016

Scopus Subject Areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition

User-Defined Keywords

  • Lip event detection
  • Low rank affinity
  • Oriented histograms
  • Regional optical flow

Fingerprint

Dive into the research topics of 'Lip event detection using oriented histograms of regional optical flow and low rank affinity pursuit'. Together they form a unique fingerprint.

Cite this