Share Email Print
cover

Proceedings Paper

J-PAS data management pipeline and archiving
Author(s): D. Cristóbal-Hornillos; N. Gruel; J. Varela; A. López-Sainz; A. Ederoclite; M. Moles; A. J. Cenarro; A. Marín-Franch; J. Hernández-Fuertes; A. Yanes-Díaz; S. Chueca; S. Rueda-Teruel; F. Rueda-Teruel; R. Luis-Simoes
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

J-PAS survey is going to observe 8000 deg2 in 54 optical narrow band filters plus 3 broad band ones. The survey will produce 1.2 PB of raw data in six years of observations. The treatment of about 1.5 TB per night, coming from the 14 detectors of the JPCam camera in the JST/T250, plus one detector in the JAST/T80 camera, shall be performed during the day after the operations. This contribution presents the software and hardware architecture designed to process and validate the data. The processing of the images is divided in two main stages. The first one, which deals with instrumental correction and data validation, is run daily. The second stage is run when a tile is completed and do the combination of the individual corrected frames and weight maps. To perform the astrometric calibration, image coadding and source extraction the data management pipeline uses software from the community which is integrated through Python. The software uses a database to control the process by storing the operations performed, parameters used and quality checks. This allows fast reprocessing to retrieve intermediate stages of the treatment from the raw data for any data release. This approach saves disk space by avoiding the storage of the processed individual frames. The data archiving and processing will be done in a data center 30 km away from the observatory. It will be equipped with ~2.5 PB of storage capacity to store the raw data and the final mosaics of the 57 filters, and processing power to deal with the incoming data.

Paper Details

Date Published: 24 September 2012
PDF: 9 pages
Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 845116 (24 September 2012); doi: 10.1117/12.925431
Show Author Affiliations
D. Cristóbal-Hornillos, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
N. Gruel, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
J. Varela, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
A. López-Sainz, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
A. Ederoclite, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
M. Moles, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
Instituto de Astrofísica de Andalucía (Spain)
A. J. Cenarro, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
A. Marín-Franch, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
J. Hernández-Fuertes, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
A. Yanes-Díaz, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
S. Chueca, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
S. Rueda-Teruel, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
F. Rueda-Teruel, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)
R. Luis-Simoes, Ctr. de Estudios de Física del Cosmos de Aragón (Spain)


Published in SPIE Proceedings Vol. 8451:
Software and Cyberinfrastructure for Astronomy II
Nicole M. Radziwill; Gianluca Chiozzi, Editor(s)

© SPIE. Terms of Use
Back to Top