DAMix: Exploiting Deep Autoregressive Model Zoo for Improving Lossless Compression Generalization

Qishi Dong, Fengwei Zhou, Ning Kang, Chuanlong Xie, Shifeng Zhang, Jiawei Li, Heng Peng*, Zhenguo Li*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

Abstract

Deep generative models have demonstrated superior performance in lossless compression on identically distributed data. However, in real-world scenarios, data to be compressed are of various distributions and usually cannot be known in advance. Thus, commercially expected neural compression must have strong Out-of-Distribution (OoD) generalization capabilities. Compared with traditional compression methods, deep learning methods have intrinsic flaws for OoD generalization. In this work, we make the attempt to tackle this challenge by exploiting a zoo of Deep Autoregressive models (DAMix). We build a model zoo consisting of autoregressive models trained on data from diverse distributions. In the test phase, we select useful expert models by a simple model evaluation score and adaptively aggregate the predictions of selected models. By assuming the outputs from each expert models are biased in favor of their training distributions, a von Mises-Fisher based filter is proposed to recover the value of unbiased predictions that provides more accurate density estimations than a single model. We derive the posterior of unbiased predictions as well as concentration parameters in the filter, and a novel temporal Stein variational gradient descent for sequential data is proposed to adaptively update the posterior distributions. We evaluate DAMix on 22 image datasets, including in-distribution and OoD data, and demonstrate that making use of unbiased predictions has up to 45.6% improvement over the single model trained on ImageNet.

Original languageEnglish
Title of host publicationProceedings of the 37th AAAI Conference on Artificial Intelligence
EditorsBrian Williams, Yiling Chen, Jennifer Neville
Place of PublicationWashington, DC
PublisherAAAI press
Pages4250-4258
Number of pages9
Edition1st
ISBN (Electronic)9781577358800
DOIs
Publication statusPublished - 27 Jun 2023
Event37th AAAI Conference on Artificial Intelligence 2023, AAAI 2023 and the 35th Conference on Innovative Applications of Artificial Intelligence, IAAI 2023 - Washington, United States
Duration: 7 Feb 202314 Feb 2023
https://ojs.aaai.org/index.php/AAAI/issue/view/553
https://aaai-23.aaai.org/

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence, AAAI 2023
PublisherAAAI Press
Number4
Volume37
ISSN (Print)2159-5399
ISSN (Electronic)2374-3468

Conference

Conference37th AAAI Conference on Artificial Intelligence 2023, AAAI 2023 and the 35th Conference on Innovative Applications of Artificial Intelligence, IAAI 2023
Country/TerritoryUnited States
CityWashington
Period7/02/2314/02/23
Internet address

Scopus Subject Areas

  • Artificial Intelligence

User-Defined Keywords

  • DMKM: Data Compression
  • ML: Bayesian Learning
  • ML: Ensemble Methods
  • ML: Probabilistic Methods

Fingerprint

Dive into the research topics of 'DAMix: Exploiting Deep Autoregressive Model Zoo for Improving Lossless Compression Generalization'. Together they form a unique fingerprint.

Cite this