PAW: Data Partitioning Meets Workload Variance

Zhe Li; Man Lung Yiu; Tsz Nam Chan

doi:10.1109/ICDE53745.2022.00014

PAW: Data Partitioning Meets Workload Variance

Zhe Li, Man Lung Yiu, Tsz Nam Chan

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

3 Citations (Scopus)

Abstract

In distributed storage systems (e.g., HDFS, Amazon S3, Databricks), partitioning is applied on a dataset in order to enhance performance and availability. Recently, partitioning methods have been designed to optimize the query performance of partitions with respect to the historical query workload. Nevertheless, in practice, future query workloads may deviate from the historical query workload, thus deteriorating the performance of existing partitioning methods.
To fill this research gap, we model the variance of future query workloads from the historical query workload, then exploit this characteristic to produce partitions that perform well for future query workloads. In addition, we explore the space of irregular shaped partition regions to further optimize the query performance. Experimental results on TPC-H and real datasets show that our proposal is up to 70× more efficient than the state-of-the-art method.

Original language	English
Title of host publication	Proceedings - 2022 IEEE 38th International Conference on Data Engineering, ICDE 2022
Publisher	IEEE
Pages	123-135
Number of pages	13
ISBN (Electronic)	9781665408837
ISBN (Print)	9781665408844
DOIs	https://doi.org/10.1109/ICDE53745.2022.00014
Publication status	Published - May 2022
Event	38th IEEE International Conference on Data Engineering, ICDE 2022 - Virtual, Kuala Lumpur, Malaysia Duration: 9 May 2022 → 12 May 2022 https://icde2022.ieeecomputer.my/ https://ieeexplore.ieee.org/xpl/conhome/9835153/proceeding

Publication series

Name	Proceedings of IEEE International Conference on Data Engineering (ICDE)
ISSN (Print)	1063-6382
ISSN (Electronic)	2375-026X

Conference

Conference	38th IEEE International Conference on Data Engineering, ICDE 2022
Country/Territory	Malaysia
City	Kuala Lumpur
Period	9/05/22 → 12/05/22
Internet address	https://icde2022.ieeecomputer.my/ https://ieeexplore.ieee.org/xpl/conhome/9835153/proceeding

Scopus Subject Areas

Software
Information Systems
Signal Processing

Access to Document

10.1109/ICDE53745.2022.00014

Cite this

@inproceedings{57476eba3008461fb00d66ff0c8f6e85,

title = "PAW: Data Partitioning Meets Workload Variance",

abstract = "In distributed storage systems (e.g., HDFS, Amazon S3, Databricks), partitioning is applied on a dataset in order to enhance performance and availability. Recently, partitioning methods have been designed to optimize the query performance of partitions with respect to the historical query workload. Nevertheless, in practice, future query workloads may deviate from the historical query workload, thus deteriorating the performance of existing partitioning methods.To fill this research gap, we model the variance of future query workloads from the historical query workload, then exploit this characteristic to produce partitions that perform well for future query workloads. In addition, we explore the space of irregular shaped partition regions to further optimize the query performance. Experimental results on TPC-H and real datasets show that our proposal is up to 70× more efficient than the state-of-the-art method.",

author = "Zhe Li and Yiu, {Man Lung} and Chan, {Tsz Nam}",

note = "Funding Information: This work was supported by grant GRF 152050/19E from the Hong Kong RGC. 1For example, the typical size of a data block in HDFS is 128MB. Publisher Copyright: {\textcopyright} 2022 IEEE.; 38th IEEE International Conference on Data Engineering, ICDE 2022 ; Conference date: 09-05-2022 Through 12-05-2022",

year = "2022",

month = may,

doi = "10.1109/ICDE53745.2022.00014",

language = "English",

isbn = "9781665408844",

series = "Proceedings of IEEE International Conference on Data Engineering (ICDE)",

publisher = "IEEE",

pages = "123--135",

booktitle = "Proceedings - 2022 IEEE 38th International Conference on Data Engineering, ICDE 2022",

address = "United States",

url = "https://icde2022.ieeecomputer.my/, https://ieeexplore.ieee.org/xpl/conhome/9835153/proceeding",

}

Li, Z, Yiu, ML & Chan, TN 2022, PAW: Data Partitioning Meets Workload Variance. in Proceedings - 2022 IEEE 38th International Conference on Data Engineering, ICDE 2022. Proceedings of IEEE International Conference on Data Engineering (ICDE), IEEE, pp. 123-135, 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, 9/05/22. https://doi.org/10.1109/ICDE53745.2022.00014

PAW: Data Partitioning Meets Workload Variance. / Li, Zhe; Yiu, Man Lung; Chan, Tsz Nam.
Proceedings - 2022 IEEE 38th International Conference on Data Engineering, ICDE 2022. IEEE, 2022. p. 123-135 (Proceedings of IEEE International Conference on Data Engineering (ICDE)).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - PAW: Data Partitioning Meets Workload Variance

AU - Li, Zhe

AU - Yiu, Man Lung

AU - Chan, Tsz Nam

PY - 2022/5

Y1 - 2022/5

N2 - In distributed storage systems (e.g., HDFS, Amazon S3, Databricks), partitioning is applied on a dataset in order to enhance performance and availability. Recently, partitioning methods have been designed to optimize the query performance of partitions with respect to the historical query workload. Nevertheless, in practice, future query workloads may deviate from the historical query workload, thus deteriorating the performance of existing partitioning methods.To fill this research gap, we model the variance of future query workloads from the historical query workload, then exploit this characteristic to produce partitions that perform well for future query workloads. In addition, we explore the space of irregular shaped partition regions to further optimize the query performance. Experimental results on TPC-H and real datasets show that our proposal is up to 70× more efficient than the state-of-the-art method.

AB - In distributed storage systems (e.g., HDFS, Amazon S3, Databricks), partitioning is applied on a dataset in order to enhance performance and availability. Recently, partitioning methods have been designed to optimize the query performance of partitions with respect to the historical query workload. Nevertheless, in practice, future query workloads may deviate from the historical query workload, thus deteriorating the performance of existing partitioning methods.To fill this research gap, we model the variance of future query workloads from the historical query workload, then exploit this characteristic to produce partitions that perform well for future query workloads. In addition, we explore the space of irregular shaped partition regions to further optimize the query performance. Experimental results on TPC-H and real datasets show that our proposal is up to 70× more efficient than the state-of-the-art method.

UR - http://www.scopus.com/inward/record.url?scp=85136344378&partnerID=8YFLogxK

U2 - 10.1109/ICDE53745.2022.00014

DO - 10.1109/ICDE53745.2022.00014

M3 - Conference proceeding

SN - 9781665408844

T3 - Proceedings of IEEE International Conference on Data Engineering (ICDE)

SP - 123

EP - 135

BT - Proceedings - 2022 IEEE 38th International Conference on Data Engineering, ICDE 2022

PB - IEEE

T2 - 38th IEEE International Conference on Data Engineering, ICDE 2022

Y2 - 9 May 2022 through 12 May 2022

ER -

PAW: Data Partitioning Meets Workload Variance

Abstract

Publication series

Conference

Scopus Subject Areas

Access to Document

Other files and links

Fingerprint

Cite this