A General-purpose Distributed Programming System using Data-parallel Streams

Tsung-Wei Huang, Chun-Xun Lin, Guannan Guo, Martin D. F. Wong

Research output: Chapter in book/report/conference proceedingChapterpeer-review

2 Citations (Scopus)

Abstract

In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.
Original languageEnglish
Title of host publicationMM 2018 - Proceedings of the 2018 ACM Multimedia Conference
PublisherAssociation for Computing Machinery (ACM)
Pages1360–1363
Number of pages4
ISBN (Print)9781450356657
DOIs
Publication statusPublished - Oct 2018
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: 22 Oct 201826 Oct 2018
https://dl.acm.org/doi/proceedings/10.1145/3240508 (Link to conference proceedings)

Publication series

NameProceedings of ACM international conference on Multimedia

Conference

Conference26th ACM Multimedia conference, MM 2018
Country/TerritoryKorea, Republic of
CitySeoul
Period22/10/1826/10/18
Internet address

User-Defined Keywords

  • Distributed System
  • Stream Processing
  • Machine Learning

Fingerprint

Dive into the research topics of 'A General-purpose Distributed Programming System using Data-parallel Streams'. Together they form a unique fingerprint.

Cite this