Containerizing the telemetry data pipeline for MMTO subsystems

J. Duane Gibson, Carlos Burguillo-Rodriguez, Timothy Pickering, Dallan Porter, Scott Swindell, Will Goble

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

The telemetry data pipeline for the MMT Observatory (MMTO) describes the flow of data sampled from diverse hardware devices within MMTO subsystems, through logging into various databases, to user interfaces and monitoring services. Subsystems within the pipeline include the telescope mount, primary and secondary mirrors, instruments, and environmental sensors. Data acquisition services within the pipeline post new data with a uniform data structure to a master Redis server. These incoming data are transported in real-time to replicated Redis servers where they are logged into local MariaDB relational databases. Database tables for logged data from the subsystems are highly optimized for data storage, allowing the archival of billions of data points for thousands of parameters over the past 10-15 years. Because of ever increasing difficulty in supporting legacy servers and software, a large-scale containerization effort is underway of the various components of the telemetry pipeline and underlying cyberinfrastructure. These critical servers and services are single points of failure that could result in up to weeks of operational downtime. Containerization helps to reduce the risk of potential hardware failure, operating system upgrades, and software incompatibilities. Containerizing a service defines all the software requirements for that service, including the code, runtime, system tools, system libraries, and settings. It allows rapid and reliable redeployment of new and legacy services with minimal concern for the underlying hardware. Finally, a summary of the ongoing and planned future work is presented.

Original languageEnglish (US)
Title of host publicationSoftware and Cyberinfrastructure for Astronomy VII
EditorsJorge Ibsen, Gianluca Chiozzi
PublisherSPIE
ISBN (Electronic)9781510653597
DOIs
StatePublished - 2022
Externally publishedYes
EventSoftware and Cyberinfrastructure for Astronomy VII 2022 - Montreal, Canada
Duration: Jul 17 2022Jul 21 2022

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
Volume12189
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X

Conference

ConferenceSoftware and Cyberinfrastructure for Astronomy VII 2022
Country/TerritoryCanada
CityMontreal
Period7/17/227/21/22

Keywords

  • Containerization
  • Cyberinfrastructure
  • Data Logging
  • Databases
  • Docker
  • MMT Observatory
  • Telemetry

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Containerizing the telemetry data pipeline for MMTO subsystems'. Together they form a unique fingerprint.

Cite this