An Uplink Communication-Efficient Approach to Featurewise Distributed Sparse Optimization With Differential Privacy

Jian Lou, Yiu Ming Cheung*

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

4 Citations (Scopus)

Abstract

In sparse empirical risk minimization (ERM) models, when sensitive personal data are used, e.g., genetic, healthcare, and financial data, it is crucial to preserve the differential privacy (DP) in training. In many applications, the information (i.e., features) of an individual is held by different organizations, which give rise to the prevalent yet challenging setting of the featurewise distributed multiparty model training. Such a setting is also beneficial to the scalability when the number of features exceeds the computation and storage capacity of a single node. However, existing private sparse optimizations are limited to centralized and samplewise distributed datasets only. In this article, we develop a differentially private algorithm for the sparse ERM model training under the featurewise distributed datasets setting. Our algorithm comes with guaranteed DP, nearly optimal utility, and reduced uplink communication complexity. Accordingly, we present a more generalized convergence analysis for block-coordinate Frank–Wolfe (BCFW) under arbitrary sampling (denoted as BCFW-AS in short), which significantly extends the known convergence results that apply to two specific sampling distributions only. To further reduce the uplink communication cost, we design an active private feature sharing scheme, which is new in both design and analysis of BCFW, to guarantee the convergence of communicating Johnson–Lindenstrauss transformed features. Empirical studies justify the new convergence as well as the nearly optimal utility theoretical results.
Original languageEnglish
Pages (from-to)4529-4543
Number of pages15
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume32
Issue number10
DOIs
Publication statusPublished - Oct 2021

User-Defined Keywords

  • Differential privacy (DP)
  • distributed optimization
  • empirical risk minimization (ERM)
  • sparse optimization

Fingerprint

Dive into the research topics of 'An Uplink Communication-Efficient Approach to Featurewise Distributed Sparse Optimization With Differential Privacy'. Together they form a unique fingerprint.

Cite this