JITA4DS: Disaggregated Execution of Data Science Pipelines Between the Edge and the Data Centre

Genoveva Vargas-Solar, Md Sahil Hassan, Ali Akoglu

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

This paper targets the execution of data science (DS) pipelines supported by data processing, transmission and sharing across several resources executing greedy processes. Current data science pipelines environments provide various infrastructure services with computing resources such as general-purpose processors (GPP), Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs) and Tensor Processing Unit (TPU) coupled with platform and software services to design, run and maintain DS pipelines. These one-fits-all solutions impose the complete externalization of data pipeline tasks. However, some tasks can be executed in the edge, and the backend can provide just in time resources to ensure ad-hoc and elastic execution environments. This paper introduces an innovative composable “Just in Time Architecture” for configuring DCs for Data Science Pipelines (JITA-4DS) and associated resource management techniques. JITA-4DS is a cross-layer management system that is aware of both the application characteristics and the underlying infrastructures to break the barriers between applications, middleware/operating system, and hardware layers. Vertical integration of these layers is needed for building a customizable Virtual Data Center (VDC) to meet the dynamically changing data science pipelines’ requirements such as performance, availability, and energy consumption. Accordingly, the paper shows an experimental simulation devoted to run data science workloads and determine the best strategies for scheduling the allocation of resources implemented by JITA-4DS.

Original languageEnglish (US)
Pages (from-to)1-26
Number of pages26
JournalJournal of Web Engineering
Volume21
Issue number1
DOIs
StatePublished - Nov 28 2021

Keywords

  • Data science pipelines
  • Disaggregated data centers
  • Edge computing

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'JITA4DS: Disaggregated Execution of Data Science Pipelines Between the Edge and the Data Centre'. Together they form a unique fingerprint.

Cite this