A General-purpose Distributed Programming System using Data-parallel Streams

Tsung-Wei Huang; Chun-Xun Lin; Guannan Guo; Martin D. F. Wong

doi:10.1145/3240508.3243654

A General-purpose Distributed Programming System using Data-parallel Streams

Tsung-Wei Huang, Chun-Xun Lin, Guannan Guo, Martin D. F. Wong

Office of the Provost

Research output: Chapter in book/report/conference proceeding › Chapter › peer-review

8 Citations (Scopus)

Abstract

In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

Original language	English
Title of host publication	MM 2018 - Proceedings of the 2018 ACM Multimedia Conference
Publisher	Association for Computing Machinery (ACM)
Pages	1360–1363
Number of pages	4
ISBN (Print)	9781450356657
DOIs	https://doi.org/10.1145/3240508.3243654
Publication status	Published - Oct 2018
Event	26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of Duration: 22 Oct 2018 → 26 Oct 2018 https://dl.acm.org/doi/proceedings/10.1145/3240508 (Link to conference proceedings)

Publication series

Name	Proceedings of ACM international conference on Multimedia

Conference

Conference	26th ACM Multimedia conference, MM 2018
Country/Territory	Korea, Republic of
City	Seoul
Period	22/10/18 → 26/10/18
Internet address	https://dl.acm.org/doi/proceedings/10.1145/3240508 (Link to conference proceedings)

User-Defined Keywords

Distributed System
Stream Processing
Machine Learning

Access to Document

10.1145/3240508.3243654

Cite this

@inbook{8eb75b0d2e9640659c25e05113a3a258,

title = "A General-purpose Distributed Programming System using Data-parallel Streams",

abstract = "In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.",

keywords = "Distributed System, Stream Processing, Machine Learning",

author = "Tsung-Wei Huang and Chun-Xun Lin and Guannan Guo and Wong, {Martin D. F.}",

note = "Funding Information: The works is supported by DARPA under award FA8650-18-2-7843. Publisher Copyright: {\textcopyright} 2018 Association for Computing Machinery ; 26th ACM Multimedia conference, MM 2018 ; Conference date: 22-10-2018 Through 26-10-2018",

year = "2018",

month = oct,

doi = "10.1145/3240508.3243654",

language = "English",

isbn = "9781450356657",

series = "Proceedings of ACM international conference on Multimedia",

publisher = "Association for Computing Machinery (ACM)",

pages = "1360–1363",

booktitle = "MM 2018 - Proceedings of the 2018 ACM Multimedia Conference",

address = "United States",

url = "https://dl.acm.org/doi/proceedings/10.1145/3240508",

}

Huang, T-W, Lin, C-X, Guo, G & Wong, MDF 2018, A General-purpose Distributed Programming System using Data-parallel Streams. in MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Proceedings of ACM international conference on Multimedia, Association for Computing Machinery (ACM), pp. 1360–1363, 26th ACM Multimedia conference, MM 2018, Seoul, Korea, Republic of, 22/10/18. https://doi.org/10.1145/3240508.3243654

A General-purpose Distributed Programming System using Data-parallel Streams. / Huang, Tsung-Wei; Lin, Chun-Xun; Guo, Guannan et al.
MM 2018 - Proceedings of the 2018 ACM Multimedia Conference. Association for Computing Machinery (ACM), 2018. p. 1360–1363 (Proceedings of ACM international conference on Multimedia).

Research output: Chapter in book/report/conference proceeding › Chapter › peer-review

TY - CHAP

T1 - A General-purpose Distributed Programming System using Data-parallel Streams

AU - Huang, Tsung-Wei

AU - Lin, Chun-Xun

AU - Guo, Guannan

AU - Wong, Martin D. F.

PY - 2018/10

Y1 - 2018/10

N2 - In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

AB - In this paper we present DtCraft, a distributed execution engine that enables a new powerful programming model to streamline cluster computing. Applications are described in a set of data-parallel streams, leaving difficult execution details and concurrency controls handled by our system kernel transparently. Compared with existing systems, DtCraft is unique in (1) an efficient stream-oriented programming paradigm using modern C++17, (2) an in-context resource controller and task executor based on Linux container technology, and (3) ease of development from prototyping machines to production cloud environments. These capabilities power industry applications and create new research directions in machine learning, stream processing, and distributed multimedia systems.

KW - Distributed System

KW - Stream Processing

KW - Machine Learning

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85058211774&origin=inward

U2 - 10.1145/3240508.3243654

DO - 10.1145/3240508.3243654

M3 - Chapter

SN - 9781450356657

T3 - Proceedings of ACM international conference on Multimedia

SP - 1360

EP - 1363

BT - MM 2018 - Proceedings of the 2018 ACM Multimedia Conference

PB - Association for Computing Machinery (ACM)

T2 - 26th ACM Multimedia conference, MM 2018

Y2 - 22 October 2018 through 26 October 2018

ER -

A General-purpose Distributed Programming System using Data-parallel Streams

Abstract

Publication series

Conference

User-Defined Keywords

Access to Document

Other files and links

Fingerprint

Cite this