TY - GEN
T1 - DtCraft: A distributed execution engine for compute-intensive applications
AU - Huang, Tsung-Wei
AU - Lin, Chun-Xun
AU - Wong, Martin D. F.
N1 - Funding Information:
This work is partially supported by the National Science Foundation under Grant CCF-1421563 and CCF-171883.
PY - 2017/11
Y1 - 2017/11
N2 - Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 × less development efforts over hand-crafted implementation.
AB - Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 × less development efforts over hand-crafted implementation.
UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85043521519&origin=inward
U2 - 10.1109/ICCAD.2017.8203853
DO - 10.1109/ICCAD.2017.8203853
M3 - Conference proceeding
SN - 9781538630945 (Print on Demand)
T3 - Proceedings of IEEE/ACM International Conference on Computer-Aided Design
SP - 757
EP - 765
BT - 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)
PB - IEEE
T2 - 2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
Y2 - 13 November 2017 through 16 November 2017
ER -