Share Email Print
cover

Proceedings Paper

Theoretical and empirical comparison of big data image processing with Apache Hadoop and Sun Grid Engine
Author(s): Shunxing Bao; Frederick D. Weitendorf; Andrew J. Plassard; Yuankai Huo; Aniruddha Gokhale; Bennett A. Landman
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

The field of big data is generally concerned with the scale of processing at which traditional computational paradigms break down. In medical imaging, traditional large scale processing uses a cluster computer that combines a group of workstation nodes into a functional unit that is controlled by a job scheduler. Typically, a shared-storage network file system (NFS) is used to host imaging data. However, data transfer from storage to processing nodes can saturate network bandwidth when data is frequently uploaded/retrieved from the NFS, e.g., “short” processing times and/or “large” datasets. Recently, an alternative approach using Hadoop and HBase was presented for medical imaging to enable co-location of data storage and computation while minimizing data transfer. The benefits of using such a framework must be formally evaluated against a traditional approach to characterize the point at which simply “large scale” processing transitions into “big data” and necessitates alternative computational frameworks. The proposed Hadoop system was implemented on a production lab-cluster alongside a standard Sun Grid Engine (SGE). Theoretical models for wall-clock time and resource time for both approaches are introduced and validated. To provide real example data, three T1 image archives were retrieved from a university secure, shared web database and used to empirically assess computational performance under three configurations of cluster hardware (using 72, 109, or 209 CPU cores) with differing job lengths. Empirical results match the theoretical models. Based on these data, a comparative analysis is presented for when the Hadoop framework will be relevant and nonrelevant for medical imaging.

Paper Details

Date Published: 13 March 2017
PDF: 8 pages
Proc. SPIE 10138, Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications, 101380B (13 March 2017); doi: 10.1117/12.2254712
Show Author Affiliations
Shunxing Bao, Vanderbilt Univ. (United States)
Frederick D. Weitendorf, Vanderbilt Univ. (United States)
Andrew J. Plassard, Vanderbilt Univ. (United States)
Yuankai Huo, Vanderbilt Univ. (United States)
Aniruddha Gokhale, Vanderbilt Univ. (United States)
Bennett A. Landman, Vanderbilt Univ. (United States)


Published in SPIE Proceedings Vol. 10138:
Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications
Tessa S. Cook; Jianguo Zhang, Editor(s)

© SPIE. Terms of Use
Back to Top