A Framework for Mathematical Model-Guided Deep Neural Networks in Image Processing

Image processing is a critical area of research with extensive applications across various
fields. Over the past decades, numerous mathematical models and algorithms have been
developed, which have demonstrated significant performance and have been widely
adopted across different domains. Recently, deep neural networks have shown superior
performance in a variety of image processing tasks. However, the design of most neural
networks is empirical, and the mathematical rationale behind their success remains
largely unclear. The connections between these networks and established mathematical
models and algorithms require further investigation. Although there have been attempts
to integrate neural networks with mathematical algorithms and prior information, most
existing works merely replace components of mathematical algorithms with networks or
append network by operations induced from prior information. A systematic approach
that seamlessly incorporates neural networks with mathematical models, providing clear
mathematical explanations, is still lacking.
This proposal aims to develop a framework for designing mathematical model-guided
neural networks tailored for image processing applications. The project contains two
parts. In the first part, we develop our framework for general image processing models,
which comprise a regularization term and a fidelity term. Utilizing the hybrid splitting
method, we propose an operator-splitting algorithm to solve these models. This
algorithm is a model-guided neural network as it is essentially a neural network with
activation functions derived from the underlying mathematical model. As a result, each
component of the network has a precise algorithmic interpretation. Our framework is
general enough to be applicable to a wide range of models and tasks. In the second part,
we explore the application of the proposed framework to various types of models and
tasks, using the ROF model and Euler's elastica model as examples for image denoising,
inpainting, and deblurring.
Our framework offers a systematic approach to integrating neural networks with
mathematical models and priors. This integration leverages the strengths of classical
models within neural networks, thereby advancing the state-of-the-art in image
processing. Our framework elucidates a deep connection between deep neural networks
and mathematical models, providing an innovative perspective on the design and
explainability of neural network architectures. This approach holds the potential to
substantially advance the development of future deep learning methodologies.