Share Email Print
cover

Proceedings Paper

Efficient data storage of astronomical data using HDF5 and PEC compression
Author(s): Jordi Portell de Mora; Enrique García-Berro; Carlos Estepa; Javier Castañeda; Marcial Clotet
Format Member Price Non-Member Price
PDF $14.40 $18.00

Paper Abstract

Future space missions are based on a new generation of instruments that often generate vast amounts of data. Transferring this data to ground, and once there, between different computing facilities is not an easy task whatsoever. A clear example of these missions is Gaia, a space astrometry mission of ESA. To carry out the data reduction tasks on ground, an international consortium has been set up. Among its tasks perhaps the most demanding one is the Intermediate Data Updating, which will have to repeatedly re-process nearly 100 TB of raw data received from the satellite using the latest instrument calibrations available. On the other hand, one of the best data compression solutions is the Prediction Error Coder, a highly optimized entropy coder that performs very well with data following realitic statistics. Regarding file formats, HDF5 provides a completely indexed, easily customizable file with a quick and parallel access. Moreover, HDF5 has a friendly presentation format and multi-platform compatibility. Thus, it is a powerful environment to store data compressed using the above mentioned coder. Here we show the integration of both systems for the storage of Gaia raw data. However, this integration can be applied to the efficient storage of any kind of data. Moreover, we show that the file sizes obtained using this solution are similar to those obtained using other compression algorithms that require more computing power.

Paper Details

Date Published: 12 October 2011
PDF: 10 pages
Proc. SPIE 8183, High-Performance Computing in Remote Sensing, 818305 (12 October 2011); doi: 10.1117/12.898203
Show Author Affiliations
Jordi Portell de Mora, Institute for Space Studies of Catalonia (Spain)
Univ. de Barcelona (Spain)
Institut de Ciencies del Cosmos (Spain)
Enrique García-Berro, Institute for Space Studies of Catalonia (Spain)
Univ. Politècnica de Catalunya (Spain)
Carlos Estepa, Institute for Space Studies of Catalonia (Spain)
Univ. Politècnica de Catalunya (Spain)
Javier Castañeda, Institute for Space Studies of Catalonia (Spain)
Univ. de Barcelona (Spain)
Institut de Ciencies del Cosmos (Spain)
Marcial Clotet, Institute for Space Studies of Catalonia (Spain)
Univ. de Barcelona (Spain)
Institut de Ciencies del Cosmos (Spain)


Published in SPIE Proceedings Vol. 8183:
High-Performance Computing in Remote Sensing
Bormin Huang; Antonio J. Plaza, Editor(s)

© SPIE. Terms of Use
Back to Top