Share Email Print
cover

Proceedings Paper

The Dark Energy Survey data management system
Author(s): Joseph J. Mohr; Darren Adams; Wayne Barkhouse; Cristina Beldica; Emmanuel Bertin; Y. Dora Cai; Luiz A. Nicolaci da Costa; J. Anthony Darnell; Gregory E. Daues; Michael Jarvis; Michelle Gower; Huan Lin; Leandro Martelli; Eric Neilsen; Chow-Choong Ngeow; Ricardo L. C. Ogando; Alex Parga; Erin Sheldon; Douglas Tucker; Nikolay Kuropatkin; Chris Stoughton
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The Dark Energy Survey (DES) collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at the National Center for Supercomputing Applications (NCSA) and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.

Paper Details

Date Published: 12 July 2008
PDF: 16 pages
Proc. SPIE 7016, Observatory Operations: Strategies, Processes, and Systems II, 70160L (12 July 2008); doi: 10.1117/12.789550
Show Author Affiliations
Joseph J. Mohr, Univ. of Illinois (United States)
National Ctr. for Supercomputing Applications (United States)
Darren Adams, National Ctr. for Supercomputing Applications (United States)
Wayne Barkhouse, Univ. of North Dakota (United States)
Cristina Beldica, National Ctr. for Supercomputing Applications (United States)
Emmanuel Bertin, Institut d'Astrophysique de Paris (France)
Y. Dora Cai, National Ctr. for Supercomputing Applications (United States)
Luiz A. Nicolaci da Costa, Observatorio Nacional (Brazil)
J. Anthony Darnell, Univ. of Illinois (United States)
Gregory E. Daues, National Ctr. for Supercomputing Applications (United States)
Michael Jarvis, Univ. of Pennsylvania (United States)
Michelle Gower, National Ctr. for Supercomputing Applications (United States)
Huan Lin, Fermi National Accelerator Lab. (United States)
Leandro Martelli, Observatorio Nacional (Brazil)
Eric Neilsen, Fermi National Accelerator Lab. (United States)
Chow-Choong Ngeow, Univ. of Illinois (United States)
Ricardo L. C. Ogando, Observatorio Nacional (Brazil)
Alex Parga, National Ctr. for Supercomputing Applications (United States)
Erin Sheldon, New York Univ. (United States)
Douglas Tucker, Fermi National Accelerator Lab. (United States)
Nikolay Kuropatkin, Fermi National Accelerator Lab. (United States)
Chris Stoughton, Fermi National Accelerator Lab. (United States)


Published in SPIE Proceedings Vol. 7016:
Observatory Operations: Strategies, Processes, and Systems II
Roger J. Brissenden; David R. Silva, Editor(s)

© SPIE. Terms of Use
Back to Top