Augmented reality (AR) has the potential to allow physicians to "see" inside a patient in real-time. For this to be useful, a patient’s medical imaging must be quickly and accurately registered back to the patient. By using a radio-opaque optical marker and an iterative closest point algorithm, a method for aligning CT scans projected from a head-mounted display to within 5 mm accuracy in 0.9s is developed. These findings show that it is feasible to use a combined radio-opaque optical marker to align a CT scan with a patient.

This study looks at the effects of pressure, flow, and flow resistance with 3D-printed idealized and patient specific intracranial aneurysms (IA) using flow diverters (FD) such as the pipeline flex embolization device. 3D-printing allows development of precise benchtop experiments using embedded pressure sensors in vascular phantoms to study hemodynamic changes. In addition, micro-CT can be used to evaluate complex deployment geometries and study their correlation with flow. This study can help in the decision process for the course of treatment of FD in an IAs. This can possibly lead to higher occlusion rates post operation.

The left atrial appendage occlusion (LAAO) is detected that 80% to 90% of the thrombus is generated in the left atrium, and closes the entrance of the left atrium using occlusion devices to block the cause of thrombosis. The LAAO procedure is a breakthrough compared to conventional medication. However, LAAO depends on medical images such as computed tomography angiography (CTA) and transesophageal echocardiography (TEE) to determine the size of occlusion devices. In this study, segmentation and modeling were performed using CTA before the procedure, and patient-specific LAAO phantom was fabricated by 3D printing technology. And then, simulations were performed to predict the size of the device before the procedure prospectively. In addition, for evaluating the accuracy of 3D printed phantom, corresponding landmarks of 3D models and the phantoms were evaluated.

3D-printing of patient-specific phantoms such as the mitral valve (MV) is challenging due to inability of current imaging systems to reconstruct fine moving features and 3D printing constraints. We investigated methods to 3D-print MV structures using ex-vivo micro-CT. This approach leverages the spatial orientation of the MV annulus and papillary muscles of a patient’s cardiac CT scan to build a micro-CT stage that fixates a dissected porcine MV. With this volumetric imaging data, the chordae were segmented, and 3D printed to accurately model MV chordae diameter and length from leaflet to papillary muscle attachment sites.

3D printed Patient-Specific Neurovascular Phantoms (3DP-PSNP) containing significant portions of the neurovasculature can be used to develop and test new diagnostic tools. The purpose of this research is to assess the use of 3DP-PSNPs to study the correlation between Angiographic Parametric Imaging (API) parameters and the severity of carotid stenosis. We developed 3DP-PSNPs and successfully implemented each phantom in a physiological pulsatile flow loop and simulated angiography. Overall, API was assessed in 3DP-PSNPs and shown to have increasing differences between diseased and contralateral carotid arteries with increasing severity.

Medical imaging data extracted from a single subject's MRI can contain 100,000+ images and millions of key-value metadata pairs which must be verified to ensure instrument and research protocol compliance. Here we describe the Scalable Quality Assurance for Neuroimaging (SQAN), an open-source suite for protocol quality control (QC) and instrumental validation on medical imaging data (e.g. DICOM). This paper is the second in a series, and provides a technical description of the systems, methods, and infrastructure of the data management and quality control components of the SQAN application stack.

Advances in computer hard- and software have enabled the automated extraction of biomarkers from large scale imaging studies by means of image processing pipelines. For large cohort studies, pipelines are typically executed in parallel on one or more High Performance Computing Clusters. As processing is distributed, it is challenging and time-consuming for seasoned pipeline developers to debug large-scale experiments and monitor their progress, and nearly impossible for clinical researchers. To mitigate these challenges, we developed the Pipeline Inspection and Monitoring web application (PIM), an interactive, visualization-based web application to monitor and debug running image processing pipelines.

In this work, we extend our existing Multimodal Glioma Analysis (MGA) pipeline to an automated preprocessing framework, to address the problem of standardizing large heterogeneous MRI datasets for advanced analysis. The pipeline is equipped with a Natural Language Processing (NLP) based scan-type classifier. Additionally, it can be run in a semi-automatic mode using a Jupyter notebook based GUI for progress tracking and quality control. MGA has been implemented as a Docker container to ensure easy portability and deployment. The application can run as a stand-alone on raw DICOM data directly or using data hosted in XNAT.

Annotated clinical data concerning rare cancers is scarce, often creating a void in the understanding of these diseases. Nowadays, large biomedical data repositories, such as PubMed Central Open Access (PMC-OA), have been made freely available to the scientific community. We propose a novel framework for the curation of full-text journals in PMC-OA to generate large multimodal (images and text) datasets of rare cancer studies. We develop and evaluate an approach using visual deep learning and natural language processing models to mine out 32,486 light microscopy, human, rare cancer images. This automatically annotated multimodal dataset could be further used for advancing research.

The Stroke Neuroimaging Phenotype Repository (SNIPR) is a multi-center imaging repository of clinical scans from stroke patients, based on the XNAT imaging informatics platform. Now, SNIPR hosts and provides secure access to 4172 imaging sessions of 1587 anonymized acute stroke patients pulled from PACS worldwide. Moreover, XNAT’s standard XML Schema extension mechanism enables SNIPR to support data type extensions to facilitate integration of independent stroke phenotypic studies. Furthermore, we have developed a containerized convolutional neural network approach to automatically categorize uploaded CT sequences into defined classes, deployed it within a broader automated image processing pipeline, and obtained 96% test accuracy.

A web-based crowdsourcing platform that allows researchers to obtain close to ground truth reference segmentations is presented. The system includes a point-based segmentation interface that is used to obtain segmentations from the crowd, after a guided tutorial and performance test was passed. The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm generates estimated ground truth segmentation masks and evaluates the users performance continuously in the backend. As proof of concept, a test-study with 75 photographs of human eyes was performed by 44 users. In a few days, 2060 segmentation masks with a total of 52826 vertices along the mask contour were collected.

In this paper, we implemented a 3D network using deep learning algorithms to detect breast cancer malignancy using DBT craniocaudal (CC) view images. We created a multi-sub-volume approach, in which the most representative slice (MRS) for malignancy scans is manually selected/defined by expert radiologists. We specifically compared the effects on different selections of the MRS by two radiologists and the resulting model performance variations. The results indicate that our scheme is relatively robust for all three experiments.

Computer and imaging methods provide medical doctors assistance at any time and relieve their work load, especially for iterative processes like identifying objects of interest such as lesions and anatomical structures from the image. Microaneurysms (MAs) detection is considered to be a crucial step in some retinal image analysis algorithms for identification of diabetic retinopathy (DR) the second largest eye diseases in developed countries. The objective of this study is to compare effect of two preprocessing methods, Illumination Equalization and Contrast Enhancement, and Top-hat transformation, on retinal images to detect MAs using combination of Matching based approach and deep learning methods either in the normal fundus images or in the presence of DR. The steps for the detection are as following: 1) preprocessing, 2) Vessel segmentation and masking, 3) MAs detection using combination of Matching based approach and deep learning.

We propose a structure-enforced collaborative regression (SCoRe) model to extract co-expressed discriminative features related to schizophrenia based on imaging and epigenomics. The model links imaging and epigenomics with phenotypical information while enforcing an agreement between them. SCoRe takes advantage of group structure, which is commonly present in imaging and epigenetic data. Thus, SCoRe is more advantageous than previous methods in analyzing fMRI and DNA methylation data. Through this method, we identified 8 significant brain regions and 3 genes relevant to schizophrenia. These findings have potential in furthering the study of schizophrenia by incorporating environmental factors into neuroimaging studies.

Convolutional Neural Networks (CNNs) have proven to be effective for medical computer aided detection and diagnosis. In this research, we present and propose a two-stage architecture using CNNs for detecting pneumonia and classifying its type (bacteria/virus) in chest radiographs. In addition, we study the performance of these algorithms with the incorporation of novel lung segmentation using a U-Net architecture. We achieve an area under the receiver operating characteristic curve value of 0.996 for pneumonia detection and an accuracy of 97.8% for classification of pneumonia types on a publicly available database, thereby setting a new benchmark.

In many medical image applications, high-resolution images are needed to facilitate early diagnosis. However, due to technical limitations, it may not be easy to obtain an image with ideal resolution especially for the diffusion weighted imaging (DWI). Super-resolution (SR) technology is developed to solve this problem by generating high-resolution (HR) images from low-resolution (LR) images. The purpose of this study is to obtain the SR-DWI from the original LR image through deep super-resolution network. The effectiveness of the SR image is assessed by radiomic analysis in predicting the histological grade of breast cancer.

The cybersecurity landscape continues to rapidly evolve across all industries including healthcare. Unique to healthcare, however, is the fact that patients lives can be impacted by simply altering or withholding information. This is session we will look at the evolution of attacks on personal information, to personal health information to personal heath. In addition potential methods to balance sharing of data with protecting patients' identities will be explored to better understand concepts around federated databases and other anonymization techniques.

We provide an overview of software and hardware implementation of the prototype based on a LEAP motion sensor input device and Oculus virtual reality head-set for hand rehabilitation.

In this paper, we investigate the number of inexperienced users required to establish a reliable segmentation ground-truth (GT) and the number of vertices the user’s shall place. We employ “WeLineation”, a novel web-based system for crowdsourcing segmentations using Simultaneous Truth and Performance Level Estimation (STAPLE). 2060 sclera segmentation masks from 44 users on 75 images of human eyes were evaluated. Requiring an error below 2%, same segmentation performance is obtained with 13 experienced and 22 inexperienced users. The results indicate that knowledge about the users segmentation performance can drastically reduce the number of segmentation masks per image needed for reliable GT estimation.

Automatic image preinterpretation can help reducing dentists’ workload and improving diagnostic efficiency. We have been investigating a tooth labeling method on dental cone-beam CT images for the purpose of automatic filing of dental charts. In our previous method, two separate networks were employed for detection and classification of teeth. Although detection accuracy was promising, classification performance had a room of improvement. The purpose of this study was to investigate the use of the relation network to utilize information of positional relationship between teeth. Using the proposed method, both detection and classification performance improved.

For deep learning (DL) based medical imaging tasks, model performance improves as a function of training data set size. Using data augmentation techniques can improve the model performance when the available dataset size is limited. Different data augmentation methods for a DL based anatomy region classification task from computed tomography (CT) images was compared. The impact of data augmentation with different DL architectural layers was explored. The test accuracy improved from 88% to 97% with optimal selection of data augmentation and architecture with the same training dataset.

This paper aimed to investigate if deep image features extracted via sparse autoencoder (SAE) could be used to preoperatively predict histologic grade in pancreatic neuroendocrine tumors (pNETs). In this study, a total of 114 patients from two institutions were involved. The deep image features were extracted based on the sparse autoencoder network via a 2000-time iteration. Considering the possible prediction error due to the small patient data size, we performed 10-fold cross-validation. To find the optimal hidden size, we set the size as a range of 6-16. The maximum relevance minimum redundancy (mRMR) features selection algorithm was used to select the most histologic grade-related image features. Then the radiomics signature was generated by using the selected features with Support Vector Machine (SVM), Multivariable Logistic regression (MLR) and Artificial Neural Networks (ANN) methods. The prediction performance was evaluated using AUC value.

The rapid advancement of artificial intelligence (AI) and machine learning (ML) techniques has yielded models whose sensitivity and specificity rival those of trained human experts. However, as these models transition from proofs-of-concept to decision support tools that are used clinically, model developers and the end-users who interact with them should have a clear appreciation of how and what these models are “learning”. Future users of these models should not see them as a “black box” but demand greater transparency from model developers in conveying the rationale behind the chosen representation, how the model was trained, and the explanation associated with a model’s prediction. In this talk, I will review current and late-breaking research on the development of explainable machine and deep learning algorithms, particularly in the areas of computer-aided detection and diagnosis. I will present a taxonomy of different types of explanations and highlight techniques that interrogate the model based on internal structure or the model’s response to perturbations in the input. I will discuss experiences in applying and interpreting the results of these techniques drawn from my work in lung and breast cancer screening and other published research. Finally, I will assess the limitations and opportunities of current work in interpretable AI/ML, emphasizing the need for visualizations that aid clinical end-users with understanding model outputs and tools for ensuring the validity of model predictions over time.

General imaging has been widely performed from children to elderly people for the head, chest, abdomen, spine, limbs, and joints. Radiation technologists refer to target ranges or positioning of imaging from the past general radiography images of the same patient because patient skeleton and body type are different. Quickly and accurately positioning and setting imaging conditions reduce burden of patients and technologists. We create a patient-specific past X-ray image reference support system. Two radiation technologist assess an average time of work processing and usability of the system. Its effectiveness is clarified in the facilities for severely disabled children and persons.

This paper proposed a new generation PACS (Picture Archiving and Communication System) based on artificial intelligent visualization. It is developed from our GRIDPACS (patent number: US8805890), which combined with IHE XDS-I profile, to implement images communication, storage and display. It also uses 3D anatomical visualization model to extract multi-source data from PACS/RIS/HIS/EMR, to express patient disease location, size and severity, which was introduced as Visual Patent (VP) at previous SPIE Medical Imaging (SPIE MI 2018). It can integrate the training model of AI Imaging Diagnosis, to mark the focus and display the disease trends. The system not only has the original PACS functions, but also realizes the man-machine interaction (images and electronic medical record information between radiologist and patient) in a personalized, fast, comprehensive, quantitative and easy-to-understand way.

Reliable Deep Learning-based medical image diagnosis requires sufficient training data in terms of quality/quantity. However, in Japan, we should overcome each hospital’s different ethical code to obtain such multi-institutional data. Therefore, we built a cloud platform for (i) collecting multi-modal large-scale medical images from hospitals and (ii) conducting various Deep Learning-based Computer Aided Diagnosis research via collaboration between 6 Japanese medical societies and 6 institutes of informatics. Using over 60 million collected images, over 50 informatics researchers have conducted research on our platform. We presented the achieved results at major conferences/in medical journals and applied them to a clinical environment.

With this paper we address the challenges of providing large amounts of clinical data to external partner institutions for the purpose of knowledge extraction with a two-stage learning approach. The clinical raw data remains within the firewall of the supplying site and is analyzed on-site in a so-called QuantMed node. Thereby, we focus on the context of translational research where a modular, component-based framework supports the translation process: on the one hand it allows for a fast prototyping of DL experiments including the annotation process and the data preparation. On the other hand, prototypes can efficiently be deployed within clinical scenarios and product development is supported by a bunch of tools, e.g., for automatic distribution of processes within a GPU cluster or for the administration of the DL experiments in a comprehensible way which might support the process of facilitating translational research.

A software to delineation region of interest (ROI) of CEUS vedio is developed based on Matlab platform. Sketcher can draw bounding boxes in any frame and the software can computer the ROI of Surrounding frames through tracking algorithm. The software allows sketcher to watch and adjust the tracking results in time. The software is easy to operate and it will show the mirror ROI in double display mode. A summary of the vedio based on optical flow can tell the sketcher where is stable recording, pause recording or switch facets.

In this paper, we investigated automated localization and segmentation method for breast tumor in breast Dynamic Contrast-Enhanced MRI (DCE-MRI) scans. The proposed method takes advantage of kinetic prior and deep learning for automatic tumor localization and segmentation. We implemented our method and evaluated its performance on a dataset consisting of 74 breast MR images. We quantitatively evaluated the proposed method by comparing the segmentation with the manual annotation from an expert radiologist. Experimental results showed that the automated breast tumor segmentation method exhibits promising performance with an average Dice Coefficient of 0.89±0.06.

The external os of the uterine cervix is an important anatomical landmark. However, there has been limited work reported on segmentation of the os region in digitized cervix images. Here, we present a pipeline which consists of deep learning os region segmentation over multiple datasets, followed by comprehensive evaluation of the performance. We firstly conduct tests over datasets utilizing state-of-the-art algorithms (Mask R-CNN and MaskX R-CNN). Secondly, we use performance measurements beyond the commonly used DICE/IoU scores. The network models we obtained in this study outperform the previous methods and show potential use of quality control for data acquisition.

Most Liquid Based Cytology screening is done by cytologists, which is time consuming and expensive. Reliable automated methods are needed to assist cytologists to locate abnormal cells. State of the art cell classification assumes cells have already been segmented. However, clustered cells are very challenging to segment. We propose a two-step algorithm, avoiding cell segmentation. We train convolutional neural network models to segment nuclei and classify cell patches centered at the segmented nuclei. Evaluation with a dataset of 25 NDPI slides shows that nuclei segmentation followed by cell patch classification is a promising approach to build automated Pap test applications.

Cardiovascular diseases are the leading cause of disability and death worldwide. Coronary artery disease (CAD) is primarily caused by coronary artery narrowing due to atherosclerosis. Nowadays, coronary computed tomographic angiography (CCTA) plays a crucial role in detecting coronary artery disease (CAD) and is regarded as an effective means of ruling out CAD. Moreover, tradition visual interpretation of coronary artery stenosis exist inter-observer variability and time-consuming. Therefore, automated segment the coronary artery is important for the clinicians in evaluating patients with CAD. In this work, we propose to use a 3D attention fully convolution network (FCN) method to automatically segment the coronary artery for CCTA.

Pancreatic cancer continues showing poor prognosis with a 5-year overall survival (OS) of 9%. Stereotactic body radiotherapy (SBRT) has been increasingly adopted as the treatment option for locally advanced pancreatic cancer (LAPC). Accurate and robust segmentation of the abdominal organs on CT is essential to minimize excessive doses to organs-at-risk (OARs) such as stomach and duodenum. However, this task is tedious and time-consuming. In this work, we aimed to develop a 3D deep attention U-Net based network to automatically segment the pancreatic SBRT OARs that can significantly expedite the treatment planning process, while maintaining high segmentation accuracy comparable to the ones manually contoured by the experienced physicians.

In the last few years, Deep Learning (DL) has been showing superior performance in different modalities of bio-medical image analysis. In this paper, we propose a new DL architecture, the NABLA-N network (∇N-Net), with better feature fusion techniques in decoding units for dermoscopic image segmentation tasks. The ∇N-Net has several advances for segmentation tasks. First, this model ensures better feature representation with a combination of low to high-level feature maps. Second, this network shows better quantitative and qualitative results. In addition, the Inception Recurrent Residual Convolutional Neural Network (IRRCNN) model is used for skin cancer classification. The experimental results show superior performance on segmentation tasks compared to the Recurrent Residual U-Net (R2U-Net). The classification model shows around 87% testing accuracy for dermoscopic skin cancer classification on ISIC2018 dataset.

Mixed Reality (MR) headsets have the potential to revolutionize the way we work, learn, communicate, and get entertained. The main pillars for MR development are wearable, visual and social comfort, as well as immersion experience. Do these pillars intersect the specific needs of medical imaging and surgical guidance applications? We will review the various challenges to implement MR hardware specifically adapted to such tasks with today’s start of the art MR headset technology.

Phantoms are anatomy surrogates used to estimate the 3D distribution of radiation received by cancer patients during radiotherapy. For patients treated historically when 2D radiographs were used, phantoms are needed to simulate the radiation exposure. Currently, phantoms are built using reasonable but simple human-designed criteria, and can therefore lack precision. We propose an automatic pipeline for patient-specific phantom construction. The pipeline employs machine learning models trained to associate 2D features to 3D anatomical metrics, and a database of 3D patient imaging, to assemble patient-specific phantoms. Experiments with data of 60 pediatric patients show our approach to outperform current human-designed criteria.

This study assesses a new case-based computer-aided diagnosis (CAD) scheme of mammograms to predict likelihood of cases being malignant. To this aim, a new CAD scheme was applied to preprocess mammograms and compute a group of 12 features related to tree-based analysis on SSIM characteristics of the matched bilateral local areas from CC and MLO views separately. CAD scheme was applied to process 1000 cases in which 500 were positive, while others were benign. The area under ROC curve (AUC=0.84±0.016) and odds ratio of 19.0 demonstrated that the new CAD scheme yield higher performance to distinguish benign and malignant cases.

Schizophrenia (SZ) is a chronic and severe mental disorder that affects how a person thinks, feels, and behaves. It has been widely acknowledged that SZ is related to disrupted brain connectivity; however, the underlying neuro-mechanism has not been fully understood. In this study, we use the fMRI image from the Mind Clinical Imaging Consortium to estimate a causal brain network in SZ patients. We design a two-step Bayesian network analysis for this case-control study, in which we assume group brain networks are distinct but related. We reveal that compared to healthy people, SZ patients have a diminished ability to combine specialized information from distributed brain regions. Particularly, we identify 6 hub brain regions in the aberrant connectivity network, which are at the frontal-parietal lobe (supplementary motor area, middle frontal gyrus, Inferior parietal gyrus), insula and putamen of the left hemisphere.

We have developed a magnetic resonance (MR) image-based radiomic biopsy approach for estimation of malignancy grade in parotid gland cancer (PGC). High- versus intermediate- and low-malignancy grades were estimated using MR-radiomic biopsy approaches, i.e. 972 hand-crafted feature and transfer learning of five pre-trained deep learning (DL) architectures (AlexNet, GoogLeNet, VGG-16, ResNet-101, DenseNet-201) The DL architecture of VGG-16 demonstrated higher performance (accuracy=85.4%, AUC=0.906) in estimation of the malignancy grade for PGC than the other approaches. The VGG-16-based MR-radiomic biopsy could be feasible for the malignancy grade estimation of PGC using preoperative MR images.

Early detection of glaucoma is important to prevent total vision loss. It is difficult, however, to detect subtle nerve defects on retinal fundus photographs. Although an optical coherence tomography (OCT) is generally more sensitive, it is performed as a diagnostic exam rather than screening exam. The purpose of this study is to estimate the regions of defected nerves on retinal photographs using the deep learning model trained by OCT data. The fully convolutional network with inception modules was trained with fundus photographs and registrated OCT images. The proposed method achieved 79% accuracy in assessing the defected and non-defected regions.

To prepare for data standardisation in radiotherapy, we used deep learning methods 3D convolutional neural networks and recurrent neural networks, to build and algorithm that can predict a patient structure based on a 3D generated scan of the patient’s organs. We designed and tested this algorithm on 7698 structures from 286 radiotherapy plans from a cancer centre in Sydney. We demonstrate that a model which utilises both 3D spatial data and text data allow for great veracity in data standardisation out-performing traditional document classification methods. This demonstrates the impact deep learning can have in standardising radiology data and the onset of big data readiness in health data.

Women who are diagnosed with breast cancer are sometimes referred to Neoadjuvant Chemotherapy Treatment (NACT) before surgery. Today, the decision to select the NACT option is based on clinical variables alone, and with these variables less than half of the treated patients achieve pathological complete response. In this paper, we add Radiomics metrics namely quantitative features from multi modal medical images, to improve the prediction of NACT response. We describe a pilot system which predicts NACT response by analyzing clinical data combined with MRI and MG imaging information. It employs image processing, machine learning and deep learning algorithms to increase the set of discriminating features. Our system architecture includes the Biomedical Framework that is a framework to create configurable reusable pipelines and expose them as micro-services that can run on SPARK cluster on-prem or in-the-cloud.. We show the potential of each modality and the improvement in combining them.

In this work, we assess the one-class support vector machine (OCSVM) to solve the classification tasks on two highly imbalanced datasets, namely, space-occupying kidney lesions (including renal cell carcinoma and benign) data and breast cancer distant metastasis/non-metastasis imaging data. Experimental results show that the OCSVM exhibits promising performance compared to binary-class and other one-class classification methods.

We generalized a 3D generative adversarial network (3D-GAN) model into a self-supervised electronic cleansing (EC) scheme for dual-energy CT colonography (DE-CTC). The 3D-GAN is used to transform the acquired DE-CTC volumes into a representative cleansed CTC volume by use of a self-training scheme that adapts the method to the unique demands of each case. Our preliminary evaluation with an anthropomorphic phantom indicated that the use of the 3D-GAN EC scheme with DE-CTC features and the self-training scheme generates images of higher quality than those obtained by use of single-energy CTC or conventional training samples only.

We developed a novel survival prediction model, called pix2surv, based on a conditional generative adversarial network (cGAN), which is capable of directly estimating the survival time from chest CT images of a patient. We evaluated its effect on the prediction of the overall survival of patients with IPF in comparison with an established clinical prognostic biomarker known as the gender, age, and physiology (GAP) index as well as the composite physiologic index (CPI). The pix2surv model yielded significantly higher performance than those of the CPI and GAP index in the prediction of the overall survival of the IPF patients, indicating the high effectiveness of the pix2surv model as a prognostic imaging biomarker for patients with IPF.