TY - JOUR
T1 - Learning Hierarchical Variational Autoencoders with Mutual Information Maximization for Autoregressive Sequence Modeling
AU - Qian, Dong
AU - Cheung, William Kwok Wai
N1 - Funding information:
This research is partially supported by General Research Fund RGC/HKBU12202621.
Publisher copyright:
© 2021 IEEE.
PY - 2023/2/1
Y1 - 2023/2/1
N2 - Variational autoencoders (VAEs) are a class of effective deep generative models, with the objective to approximate the true, but unknown data distribution. VAEs make use of latent variables to capture high-level semantics so as to reconstruct the data well with the help of informative latent variables. Yet, training VAEs tends to suffer from posterior collapse, when the decoder is parameterized by an autoregressive model for sequence generation. VAEs can be further enhanced by introducing multiple layers of latent variables, but the posterior collapse issue hinders the adoption of such hierarchical VAEs in real-world applications. In this paper, we introduce InfoMaxHVAE, which integrates mutual information estimated via neural networks into hierarchical VAEs to alleviate posterior collapse, when powerful autoregressive models are used for modeling sequences. Experimental results on a number of text and image datasets show that InfoMaxHVAE can outperform the state-of-the-art baselines and exhibits less posterior collapse. We further show that InfoMaxHVAE can shape a coarse-to-fine hierarchical organization of the latent space.
AB - Variational autoencoders (VAEs) are a class of effective deep generative models, with the objective to approximate the true, but unknown data distribution. VAEs make use of latent variables to capture high-level semantics so as to reconstruct the data well with the help of informative latent variables. Yet, training VAEs tends to suffer from posterior collapse, when the decoder is parameterized by an autoregressive model for sequence generation. VAEs can be further enhanced by introducing multiple layers of latent variables, but the posterior collapse issue hinders the adoption of such hierarchical VAEs in real-world applications. In this paper, we introduce InfoMaxHVAE, which integrates mutual information estimated via neural networks into hierarchical VAEs to alleviate posterior collapse, when powerful autoregressive models are used for modeling sequences. Experimental results on a number of text and image datasets show that InfoMaxHVAE can outperform the state-of-the-art baselines and exhibits less posterior collapse. We further show that InfoMaxHVAE can shape a coarse-to-fine hierarchical organization of the latent space.
KW - Variational autoencoders (VAEs)
KW - hierarchical variational autoencoders (HVAEs)
KW - mutual information neural estimation
KW - neural autoregressive sequence modeling
UR - http://www.scopus.com/inward/record.url?scp=85127018397&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2022.3160509
DO - 10.1109/TPAMI.2022.3160509
M3 - Journal article
SN - 0162-8828
VL - 45
SP - 1949
EP - 1962
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
IS - 2
ER -