Share Email Print

Proceedings Paper

Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model
Author(s): Paul Doyle; Fred Mtenzi; Niall Smith; Adrian Collins; Brendan O'Shea
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

The scientific community is in the midst of a data analysis crisis. The increasing capacity of scientific CCD instrumentation and their falling costs is contributing to an explosive generation of raw photometric data. This data must go through a process of cleaning and reduction before it can be used for high precision photometric analysis. Many existing data processing pipelines either assume a relatively small dataset or are batch processed by a High Performance Computing centre. A radical overhaul of these processing pipelines is required to allow reduction and cleaning rates to process terabyte sized datasets at near capture rates using an elastic processing architecture. The ability to access computing resources and to allow them to grow and shrink as demand fluctuates is essential, as is exploiting the parallel nature of the datasets. A distributed data processing pipeline is required. It should incorporate lossless data compression, allow for data segmentation and support processing of data segments in parallel. Academic institutes can collaborate and provide an elastic computing model without the requirement for large centralized high performance computing data centers. This paper demonstrates how a base 10 order of magnitude improvement in overall processing time has been achieved using the "ACN pipeline", a distributed pipeline spanning multiple academic institutes.

Paper Details

Date Published: 24 September 2012
PDF: 10 pages
Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 84510C (24 September 2012); doi: 10.1117/12.924863
Show Author Affiliations
Paul Doyle, Dublin Institute of Technology (Ireland)
Fred Mtenzi, Dublin Institute of Technology (Ireland)
Niall Smith, Cork Institute of Technology (Ireland)
Adrian Collins, Cork Institute of Technology (Ireland)
Brendan O'Shea, Dublin Institute of Technology (Ireland)

Published in SPIE Proceedings Vol. 8451:
Software and Cyberinfrastructure for Astronomy II
Nicole M. Radziwill; Gianluca Chiozzi, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?