SPIE Digital Library Get updates from SPIE Newsroom
  • Newsroom Home
  • Astronomy
  • Biomedical Optics & Medical Imaging
  • Defense & Security
  • Electronic Imaging & Signal Processing
  • Illumination & Displays
  • Lasers & Sources
  • Micro/Nano Lithography
  • Nanotechnology
  • Optical Design & Engineering
  • Optoelectronics & Communications
  • Remote Sensing
  • Sensing & Measurement
  • Solar & Alternative Energy
  • Sign up for Newsroom E-Alerts
  • Information for:
SPIE Defense + Commercial Sensing 2017 | Call for Papers

Journal of Medical Imaging | Learn more


Print PageEmail PageView PDF

Electronic Imaging & Signal Processing

Supporting experimental research in computer vision

A collaborative community paradigm promotes new levels of reproducibility, reusability, and comparison of results.
1 April 2011, SPIE Newsroom. DOI: 10.1117/2.1201103.003558

The document-analysis and exploitation (DARE) paradigm1 was sparked at Lehigh University based on observations of how current research is being conducted in computer-perception domains. While very significant advances have been made in these fields over the last decades, publish-or-perish reflexes have impacted reporting of results and the methods underlying research-problem definition.

One can identify a recurring pattern in how problems are identified, addressed, and published. A problem is chosen, sometimes by the author. A solution is then formulated. Data is found to support the problem and solution. Testing and validation is conducted. Finally, experimental results are published and sometimes the final system is fielded. Given the tendency of defining one's own problem, and finding data to support and prove one's own findings, concerned research communities have adapted to preserve a research evaluation environment that is as objective as possible. This is effected by including stringent peer-review processes, creating and sharing common data sets, and organizing competitions.

These practices unnecessarily constrain research domains. DARE provides a collaborative community paradigm for machine-perception research that promotes new levels of reproducibility, reusability, and comparison of results. The main goals of machine perception are to develop algorithms that approach human levels of performance for specific tasks or invent new methods of improving known techniques. Unfortunately, they are often tested on small, overused data sets removed from the real world. This is partially related to the fact that data-set construction is costly and burdensome. Over time, developers get to know their data sets so intimately that, unconsciously and unwillingly, their implementation and experimental choices are unduly influenced and, therefore, fail to provide evidence of generality. Moreover, human levels of performance lack a rigorous definition, and even experts can disagree on all but the most trivial cases.

Consequently, how can we know when we have succeeded or improved? The need to compare against previously published results creates reliance on standard data sets, which increases the general trend to overuse them. However, re-implementation of published algorithms may be difficult because of incomplete descriptions, inherent conflicts of interest, and other reasons. Competitions and contests try to answer these shortcomings, but are often very domain-specific and sometimes infrequent. It is, therefore, extremely difficult to keep track of actual state-of-the-art published-algorithm performance with respect to their effective impact toward evaluating human performance, apart from those operating in very specific and context-restricted situations.

DARE offers a new approach to considering experimental data and reporting experimental results. It builds upon five tenets.2 First, there is no absolute truth associated with data. It is open to multiple and possibly contradictory interpretations, and can vary with context and user. Second, data, users, interpretations, and algorithms have reputations, depending on how they are accepted, rated, and used by the community. Users and algorithms, when interacting with data, modify their reputations and that of the data they handle. Third, all data and interpretations are persistent, fully queryable, and possess full provenance, i.e., their origin, uses, and interdependencies are known and stored. This implies that a user can request, for example, a 300 dots-per-inch image set containing printed text and handwritten annotations never before seen by the user and provide a perfectly reproducible and certified reference to this data set for others to verify. Fourth, automated algorithm-evaluation results are third-party certified, and evaluation conditions are reproducible. Finally, DARE is an open, decentralized, and community-driven model, allowing for storage, execution, annotation, and extension of data and algorithms without constraints.

This paradigm does not change in any way how actual core research is conducted. It simply extends it to a new level of reproducibility and reusability, significantly changes the manner in which researchers can report their results, and alters how a research community can assess its progress and identify advances and remaining challenges. The DARE paradigm, as well as the supporting technical platform (see Figure 1), are freely available and open to extensions.1We highly encourage community appropriation by discussion and collaboration.

Figure 1. Example of the document-analysis and exploitation paradigm.

Bart Lamiroy
Nancy University
National Polytechnic Institute of Lorraine (INPL)
Nancy, France

Bart Lamiroy has been a permanent faculty member at Nancy University/INPL since 2000, and was most recently a visiting scientist at Lehigh University (2010–2011). His experience in machine perception ranges from content-based image retrieval through visual servoing to document analysis. He serves on several International Association for Pattern Recognition committees.

Daniel Lopresti
Lehigh University
Bethlehem, PA 

Daniel Lopresti received his AB from Dartmouth College in 1982 and his PhD in computer science from Princeton University in 1987. At the department of computer science at Brown University, he taught courses ranging from very-large-scale integration (VLSI) design to computational aspects of molecular biology, and conducted research in parallel computing and VLSI computer-aided design. He helped found the Matsushita Information Technology Laboratory in Princeton and later served on the research staff at Bell Labs, where his work focused on document analysis, handwriting recognition, and biometric security. In 2003, he joined the department of computer science and engineering, where he leads a research group examining fundamental algorithmic and systems-related questions in pattern recognition, bioinformatics, and security. He is co-director of the Lehigh pattern-recognition research laboratory. He became chair of the department on 1 July 2009. He serves as an associate editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence and the International Journal of Document Analysis and Recognition. He also serves on the International Association for Pattern Recognition conferences and meetings committee.

1. http://dae.cse.lehigh.edu/WIKI The DARE Wiki. Accessed 4 March 2011.
2. B. Lamiroy, D. Lopresti, H. Korth, J. Heflin, How carefully designed open resource sharing can help and expand document analysis research, Proc. SPIE 7874, pp. 78740O, 2011. doi:10.1117/12.876483