DtCraft: A distributed execution engine for compute-intensive applications

Tsung-Wei Huang, Chun-Xun Lin, Martin D. F. Wong

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

15 Citations (Scopus)

Abstract

Recent years have seen rapid growth in data-driven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C+,+,17-based distributed execution engine that efficiently supports a new powerful programming model for building high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown promising performance on computer clusters. In a particular semicondictor design problem, we achieved 30 x speedup with 40 nodes and 15 × less development efforts over hand-crafted implementation.
Original languageEnglish
Title of host publication2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)
PublisherIEEE
Pages757-765
Number of pages9
ISBN (Electronic)9781538630938
ISBN (Print)9781538630945 (Print on Demand)
DOIs
Publication statusPublished - Nov 2017
Event2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017 - Irvine, Irvine, United States
Duration: 13 Nov 201716 Nov 2017

Publication series

NameProceedings of IEEE/ACM International Conference on Computer-Aided Design

Conference

Conference2017 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2017
Country/TerritoryUnited States
CityIrvine
Period13/11/1716/11/17

Fingerprint

Dive into the research topics of 'DtCraft: A distributed execution engine for compute-intensive applications'. Together they form a unique fingerprint.

Cite this