A General-purpose Distributed Programming System using Data-parallel Streams

Tsung-Wei Huang, Chun-Xun Lin, Guannan Guo, Martin D. F. Wong

Research output: Chapter in book/report/conference proceedingChapterpeer-review

5 Citations (Scopus)


In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.
Original languageEnglish
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery (ACM)
Number of pages4
ISBN (Print)9781450356657
Publication statusPublished - Oct 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: 22 Oct 201826 Oct 2018
https://dl.acm.org/doi/proceedings/10.1145/3240508 (Link to conference proceedings)

Publication series

NameProceedings of ACM international conference on Multimedia


Conference26th ACM Multimedia conference, MM 2018
Country/TerritoryKorea, Republic of
Internet address

User-Defined Keywords

  • Distributed System
  • Stream Processing
  • Machine Learning


Dive into the research topics of 'A General-purpose Distributed Programming System using Data-parallel Streams'. Together they form a unique fingerprint.

Cite this