ESetStore: An erasure-coded storage system with fast data recovery

Chengjian Liu, Qiang WANG, Xiaowen CHU*, Yiu Wing LEUNG, Hai Liu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Erasure codes have been used extensively in large-scale storage systems to reduce the storage overhead of triplication-based storage systems. One key performance issue introduced by erasure codes is the long time needed to recover from a single failure, which occurs constantly in large-scale storage systems. We present ESetStore, a prototype erasure-coded storage system that aims to achieve fast recovery from failures. ESetStore is novel in the following aspects. We proposed a data placement algorithm named ESet for our ESetStore that can aggregate adequate I/O resources from available storage servers to recover from each single failure. We designed and implemented efficient read and write operations on our erasure-coded storage system via effective use of available I/O and computation resources. We evaluated the performance of ESetStore with extensive experiments on a cluster with 50 storage servers. The evaluation results demonstrate that our recovery performance can obtain linear performance growth by harvesting available I/O resources. With our defined parameter recovery I/O parallelism under some mild conditions, we can achieve optimal recovery performance, in which ESet enables minimal recovery time. Rather than being an alternative to improve recovery performance, our work can be an enhancement for existing solutions, such as Partial-parallel-repair (PPR), to further improve recovery performance.

Original languageEnglish
Article number9051846
Pages (from-to)2001-2016
Number of pages16
JournalIEEE Transactions on Parallel and Distributed Systems
Volume31
Issue number9
DOIs
Publication statusPublished - 1 Sep 2020

Scopus Subject Areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

User-Defined Keywords

  • Erasure coded storage systems
  • ESet
  • ESetStore
  • Fast data recovery

Fingerprint

Dive into the research topics of 'ESetStore: An erasure-coded storage system with fast data recovery'. Together they form a unique fingerprint.

Cite this