This PDF file contains the front matter associated with SPIE Proceedings Volume 11597, including the Title Page, Copyright information, and Table of Contents.

Welcome and Introduction to SPIE Medical Imaging conference 11597: Computer-Aided Diagnosis

Radiology has evolved from an information sparse to an information rich domain, with increased structure and reduced subjectivity. I will discuss the changes in radiology and implications for deep networks.

In this study, we aim to investigate the role of standard and soft tissue chest radiography (CXR) images in the task of COVID-19 diagnosis at patient presentation using deep learning. The dataset consisted of the initial CXR exams of 6687 patients after their reverse transcription polymerase chain reaction tests for the SARS-CoV-2 virus, 1040 (15.6%) of whom tested positive and 5647 (84.4%) of whom tested negative for COVID-19. Each CXR exam contained a standard image and a soft tissue image obtained either from dual-energy acquisition or postprocessing technology. A curriculum learning technique was employed to train the model on a sequence of gradually more specific and complex tasks by first fine-tuning a model optimized for natural images on a previously established CXR dataset to diagnose a broad spectrum of pathologies, then refining the model on another established dataset to detect pneumonia, and finally fine-tuning the model again on the COVID-19 dataset collected for this study. In the last phase of training, the COVID-19 positive/negative classification was performed on 1) the standard images, 2) the soft tissue images, and 3) the two combined via feature fusion. The classification performances were evaluated on a held-out test set of 1338 cases with the same disease prevalence as training and validation sets using the area under the receiver operating characteristic curve (AUC). The three classification schemes with different inputs overall yielded AUC values of 0.76 [0.72, 0.80], 0.76 [0.73, 0.80], and 0.76 [0.72, 0.79]. When compared using the DeLong test, the three schemes yielded equivalent performances with an equivalence margin of ∆AUC = 0.05 was chosen prima facie. The value of the inclusion of soft tissue images will continue to be investigated in the segmentation and feature extraction of COVID-19 involvement, which may contribute to improving the performance of COVID-19 early diagnosis.

Lung cancer has high mortality and occurrence worldwide. Radiomics is a method for extracting quantitative features from medical images that can be used for predictive analysis. Radiomics has been applied quite successfully for lung nodule malignancy prediction. Along with traditional radiomics, Convolutional Neural Networks (CNN) are now used quite effectively for lung cancer analysis. Texture provides information about variation in pixel intensity in regions. Lung nodules/tumors possess a noticeable texture pattern. That’s why texture radiomics features can be used to construct predictive models to analyze malignant and benign lung nodules. As textures show visible patterns, training the CNNs using texture images is a novel idea that enables the creation of an ensemble of classifiers. In this study, 192 texture images (wavelet, Laws, gray level zone matrix, neighborhood grey tone difference, and run-length) were generated, and the same CNN architecture was trained separately on all texture images. We termed this approach, “Deep Radiomics.” The maximum classification accuracy of 73% and 0.82 AUC was achieved from both the P2L2C5 wavelet and L5E5L5 laws texture images. When multiple CNN model’s predictions were merged to generate an ensemble model, results of 81.43% (0.91 AUC) were achieved from our study, which was an improvement.

In the effort to contain the COVID-19 pandemic, quick and effective diagnosis is paramount in preventing the spread of the disease. While the reverse transcriptase polymerase chain reaction (RT-PCR) test is the gold standard method to identify COVID-19, the use of x-ray radiography (CXR) has been widely used in the clinical workup for patients suspected of infection as an additional means of diagnosis and treatment response monitoring. CXR is available in almost every medical center across the world, allowing a quick and protected means of identifying potential COVID-19 cases to subject to quarantine procedures. However, the major challenge with the use of CXR in COVID-19 diagnosis is its low sensitivity and specificity in current radiological practice due to the similarities in clinical presentation to other diseases. Machine learning methods, particularly deep learning, have been shown to perform extremely well in a variety of classification tasks, often exceeding human performance. To utilize these techniques, a large data set of over 12,000 CXR images, including over 6,000 confirmed COVID-19 positive cases, was collected to train and validate a deep learning model to differentiate COVID-19 pneumonia from other causes of CXR abnormalities. In this work we show that this deep learning method can differentiate between COVID-19 related pneumonia and non-COVID-19 pneumonia, with high sensitivity and specificity.

Lung opacities on CT scans, such as ground-glass opacities (GGOs) and consolidation, can manifest with various conditions, including lung cancers, pulmonary edema and COVID-19. Presentation of these non-specific findings can vary from isolated focal to diffuse opacities in all lobes. Moreover, disease distributions and progressions vary across disease types and patients. This unpredictability can challenge one’s ability to accurately quantify and compare the percentage of infected lung within and across patients. Despite the promise of AI models for image segmentation, the inconsistency of lung opacities, and limited access to large annotated datasets affect generalization performance of models to the cohort of lung diseases. In this paper, we developed a single-stage system to jointly localize the lung and opacifications in CT scans using a diverse real-world dataset with sparse annotations. A multi-class Dense U-Net model was designed to segment the lungs and two classes of opacity regions (GGOs and consolidations) in CT images. The model was trained on 4075 slices from 495 sparsely annotated CT studies and evaluated on 18625 slices from 103 densely annotated studies (37 positive). A comparative analysis of different training data subsets and loss functions was performed to determine optimal model design. Performance was evaluated by comparing manual and automated lung opacity percentages via Pearson Correlation Coefficient. The optimal model achieved a Pearson Correlation Coefficient of 0.99. These findings suggest the potential of developing an accurate method to localize lung opacification, unspecific to a particular disease

Fibrosing idiopathic interstitial pneumonia (fIIP) is a subclass of interstitial lung diseases, which leads to fibrosis in a continuous and irreversible process of lung function decay. Patients with fIIP require regular quantitative follow-up with CT and several image biomarkers have already been proposed to grade the pathology severity and try to predict the evolution. Among them, we cite the spatial extent of the diseased lung parenchyma and airway and vascular remodeling markers. COVID-19 (Cov-19) presents several similarities with fIIP and this condition is moreover suspected to evolve to fIIP in 10-30% of severe cases. Note also that the main difference between Cov-19 and fIIP is the presence of peripheral ground glass opacities and less or no amount of fibrosis in the lung, as well as the absence of airway remodeling. This paper proposes a preliminary study to investigate how existing image markers for fIIP may apply to Cov-19 phenotyping, namely texture classification and vascular remodeling. In addition, since for some patients, the fIIP/Cov-19 follow-up protocol imposes CT acquisitions at both full inspiration and full expiration, this information could also be exploited to extract additional knowledge for each individual case. We hypothesize that taking into account the two respiratory phases to analyze breathing parameters through interpolation and registration might contribute to a better phenotyping of the pathology. This preliminary study, conducted on a reduced number of patients (eight Cov-19 of different severity degrees, two fIIP patients and one control), shows a great potential of the selected CT image markers.

The coronavirus disease 2019 (COVID-19) pandemic continues to spread internationally as deaths climb worldwide. Patients with severe disease may progress to acute respiratory distress syndrome (ARDS), often requiring mechanical ventilation. Proning may be employed for patients undergoing mechanical ventilation in order to improve oxygenation. Hospitalized COVID-19 patients are often monitored through the use of portable chest radiography (CXR). Computational modeling of CXRs has been explored in COVID-19, and radiomic features have been used to predict disease progression and severity. However, studies have not yet drawn strong connections between specific radiomic features and clinical manifestations of the disease. In this retrospective study we analyze 23 COVID-19 patients hospitalized between March and May 2020. Each of these patients underwent mechanical ventilation with proning. CXRs were taken before, during, and after proning. Clinical laboratory data including blood gas levels, severity assessment scores, serological markers for inflammation, etc. were also measured for patients before and after proning. In our analysis, 1) we identify CXR radiomic features that change significantly after proning and 2) we demonstrate correlations between radiomic features and the progression of several clinical parameters implicated in COVID-19 pathophysiology. We report statistically significant positive correlations between several radiomic features extracted from pre-proning CXRs with changes in clinical measurements of LDH, oxygen saturation, ferritin, tidal volume, arterial oxygen partial pressure (PaO2) to fraction of inspired oxygen (FIO2) ratio, and FIO2. These preliminary findings suggest that radiomic features from CXRs might reflect changes in measurable clinical variables, thus indicating the potential for portable radiography to monitor the efficacy of the proning procedure in COVID-19 patients undergoing mechanical ventilation.

As computer-aided diagnostics develop to address new challenges in medical imaging, including emerging diseases such as COVID-19, the initial development is hampered by availability of imaging data. Deep learning algorithms are particularly notorious for performance that tends to improve proportionally to the amount of available data. Simulated images, as available through advanced virtual trials, may present an alternative in data-constrained applications. We begin with our previously trained COVID-19 x-ray classification model (denoted as CVX) that leveraged additional training with existing pre-pandemic chest radiographs to improve classification performance in a set of COVID-19 chest radiographs. The CVX model achieves demonstrably better performance on clinical images compared to an equivalent model that applies standard transfer learning from ImageNet weights. The higher performing CVX model is then shown to generalize effectively to a set of simulated COVID-19 images, both quantitative comparisons of AUCs from clinical to simulated image sets, but also in a qualitative sense where saliency map patterns are consistent when compared between sets. We then stratify the classification results in simulated images to examine dependencies in imaging parameters when patient features are constant. Simulated images show promise in optimizing imaging parameters for accurate classification in data-constrained applications.

Ductal in-situ carcinoma (DCIS) is a non-invasive proliferation that lacks the ability to metastasize. Over the past four decades, DCIS diagnoses have increased ten-fold, with treatments nearly as aggressive as those for small low-grade invasive breast cancer. In this study, we evaluate the potential of identifying intrinsic imaging phenotype of DCIS using radiomic signatures from breast DCE-MRI. The rationale is that such phenotypes may capture aspects of the heterogeneity of DCIS that can aid in identifying indolent from aggressive disease to better stratify patients for improved disease management. An initial analysis was performed on eighty- two DCIS cases from the ECOG-ACRIN E4112 trial. The Cancer Phenomics Toolkit (CapTK) was used to extract a total of 95 3-D radiomic features from each primary lesion volume in pre-treatment, pre-operative breast DCE-MRI images. Features were first filtered for robustness across the heterogeneous clinical sites of DCE-MRI acquisition and features deemed non-robust (59) were discarded. Dimensionality reduction was performed with the remaining thirty-six features via principle component analysis (PCA). Unsupervised hierarchical clustering of the resulting five principal components (PCs) capturing 85% of the original feature variance was applied. Two significant intrinsic DCIS radiomic phenotypes were identified (p<0.001). Our hypothesis is that DCIS imaging biomarkers could improve prognostic ability more reliably than biopsy alone. These findings will be further explored in the expanded analysis of ECOG-ACRIN E4112 trial.

During radiologists’ visual assessment of background parenchymal enhancement (BPE) on dynamic contrast enhanced (DCE)-MR images, presence of a tumor may erroneously inflate the BPE estimation due to angiogenesis within the tumor. With a dataset of 426 MRIs, we present an automated method to segment breasts, electronically remove the influence of lesion presence, and calculate scores to estimate BPE levels. A U-Net was trained for breast segmentation from maximum intensity projection (MIP) images. Next, fuzzy c-means (FCM) clustering was used to segment the lesions from the breast DCE-MRIs, and the lesion volume was removed to create MIP images without the influence of the lesion. U-Net outputs were applied to create MIP images of both breasts, affected breasts, and unaffected breasts before and after lesion removal. On an independent test set, a statistically significant trend was found between the radiologist BPE ratings and the calculated BPE scores for all breast regions (Kendall correlation, p < 0.001). Receiver operating characteristic (ROC) analysis was performed to determine the predictive value of the computed scores from each breast region in the binary tasks of classifying Minimal vs. Marked and Low vs. High BPE relative to a radiologist rating. Scores from all breast regions performed significantly better than guessing (p < 0.025 from the z-test) with BPE scores of the affected breast after lesion removal performing the best (AUC = 0.87). Results demonstrate the potential for an automatic BPE prediction from breast DCE-MR without the influence of lesion enhancement.

Computer-aided diagnosis based on features extracted from medical images relies heavily on accurate lesion segmentation before feature extraction. Using 994 unique breast lesions imaged with dynamic contrast-enhanced (DCE) MRI, several segmentation algorithms were investigated. The first method is fuzzy c-means (FCM), a well-established unsupervised clustering algorithm used on breast MRIs. The second and third methods are based on the convolutional neural network U-Net, a widely-used deep learning method for image segmentation—for two- or three-dimensional MRI data, respectively. The purpose of this study was twofold—1) to assess the performances of 2D (slice-by-slice) and 3D U-Nets in breast lesion segmentation on DCE-MRI trained with FCM segmentations, and 2) compare their performance to that of FCM. Center slice segmentations produced by FCM, 2D U-Net, and 3D U-Net were evaluated using radiologist segmentations as truth, and volumetric segmentations produced by 2D U-Net (slice-by-slice) and 3D U-Net were compared using FCM as a surrogate truth. Five-fold cross-validation was conducted on the U-Nets and Dice similarity coefficient (DSC) and Hausdorff distance (HD) were used as performance metrics. Although 3D U-Net performed well, 2D U-Net outperformed 3D U-Net, both for center slice (DSC p=4.13×10^-9, HD p=1.40×10^-2) and volume segmentations (DSC p=2.72×10^-83, HD p=2.28×10^-10). Additionally, 2D U-Net outperformed FCM in center slice segmentation in terms of DSC (p=1.09×10^-7). The results suggest that 2D U-Net is promising in segmenting breast lesions and could be an effective alternative to FCM.

Breast cancer is the second most common type of cancer of women in the U.S. behind skin cancer. Early detection and characterization of breast masses is critical for effective diagnosis and treatment of breast cancer. Computer-aided breast mass characterization methods would help to improve the accuracy of diagnoses, their reproducibility, and the throughput of breast cancer screening workflows. In this work, we introduce sparse representations of deep learning features for separation of malignant from benign breast masses in mammograms. We expect that the use of deep feature-based dictionaries will produce better benign/malignant class separation than straightforward sparse representation techniques, and fine-tuned convolutional neural networks (CNNs). We performed 10- and 30-fold cross-validation experiments for classification of benign and malignant breast masses on the MIAS and DDSM mammographic datasets. The results show that the proposed deep feature sparse analysis produces better classification rates than conventional sparse representations and fine-tuned CNNs. The top areas under the curve (AUC) for the receiver operating curve are 80.64% for 10-fold and 97.44% for 30-fold cross-validation in MIAS, and 77.29% for 10-fold and 76.02% for 30-fold cross-validation in DDSM. The main advantages of this approach are that it employs dictionaries of deep network features that are sparse in nature and that it alleviates the need for large volumes of training data and lengthy training procedures. The interesting results from this work prompt further exploration of the relationship between sparse optimization problems and deep learning.

Deep learning models have the potential to improve prediction of the presence of invasive breast cancer on MR images. Here we present a transfer learning framework for classifying dynamic contrast-enhanced MR images in two classes: those that have invasive breast carcinoma and those that are noninvasive (including benign findings and indolent cancers). We build and train several models based on a pre-trained VGG16 network and found that fine-tuning the last convolutional block is the best strategy for our small data scenario. Our model was trained and evaluated using 81 female patients who had a pre-operative MRI followed by surgery. All lesions have ground truth labels from the surgical pathology reports. We used a bounding box to generate cropped images centered on the lesion and extract multiple slices per lesion. Our network achieved an AUC of 0.83±0.05, sensitivity 0.83±0.16 and specificity 0.71±0.11 in predicting the presence of invasive cancer in the breast. We compared our results with state-of-the-art methods and found that our model is more accurate in distinguishing invasive from noninvasive lesions. Finally, the visual inspection of the class activation maps allowed us to better understand the decision process of our deep learning classifiers.

In radiotherapy, liver CT images are routinely acquired for treatment planning since CT images can provide important electron density information for dose calculation in treatment planning system. Besides the CT images, MRI is often acquired for diagnosis since it has superior soft tissue contrast. Therefore, for accurate contouring of the target, liver CT and MRI image registration plays an important role. It is very challenging to register the CT and MRI liver images due to the distinct image intensities and liver motion. In this study, a deep learning-based network was proposed to directly predict the DVF for MRI-CT liver image registration. To overcome the challenge of multimodal registration, we propose to incorporate a modality independent descriptor into the deep learning network to explore the correlations between the MRI and CT images. Our results show that the proposed network could accurately align MRI to CT liver images. The average Dice increased from 0.89±0.04 before registration to 0.93±0.02 after registration.

In this study, we propose a novel unsupervised deep learning-based method to register pelvic MRI and CBCT images. No ground truth deformation vector field (DVF) is needed during training. To perform registration between CBCT and MRI, a self-similarity image similarity loss, called as self-correlation descriptor, is used as loss function to learn the trainable parameters in the unsupervised deep neural networks. After training, for a new patient’s CBCT and MRI, the deformed MRI is obtained via first feeding the MRI and CBCT into the unsupervised deep neural networks to derive the DVF, then deformed via spatial transformation on MRI and DVF. Our results show that the proposed method has outperformed manual rigid registration. Target registration error calculated between CBCT and deformed MRI is used for evaluation. The average TRE is 2.95±0.94 mm among the 20 prostate patients we retrospectively investigated. The proposed method has great potential in providing accurate image registration and potentially facilitating adaptive radiation therapy by multi-imaging modality.

Serrated polyps were historically believed to be benign lesions that have no cancer potential. However, recent studies have revealed a molecular pathway where serrated polyps can develop into colorectal cancers. Because serrated polyps tend to be flat and pale lesions, they are challenging to detect in colonoscopy, whereas CT colonography can detect serrated polyps based on a phenomenon called contrast coating. However, the differentiation of contrast coating from tagged feces requires great skill from the reader. The purpose of this pilot study was to explore the performance of 3D deep learning in the detection of serrated polyps. The materials included 94 CT colonography cases with biopsy-confirmed serrated polyps. We explored how to adapt the architecture of our baseline 3D DenseNet into the limited dataset by modification of the architectural parameters. The detection performance of the different 3D DenseNets and a reference 3D ResNet and a 3D AlexNet were compared by use of 10-fold cross-validation in terms of their sensitivity and false-positive rate within a clinically meaningful performance range by use of the free-response operating characteristic analysis. Our preliminary results indicate that the optimized 3D DenseNet can yield a high detection performance for serrated polyps that is comparable to those of state-of-the-art conventional CADe systems for traditional polyps in CT colonography.

Magnetic Resonance Imaging (MRI) is increasingly used to localize prostate cancer, but the subtle features of cancer vs. normal tissue renders the interpretation of MRI challenging. Computational approaches have been proposed to detect prostate cancer, yet variation in intensity distribution across different scanners, and even on the same scanner, poses significant challenges to image analysis via computational tools, such as deep learning. In this study, we developed a conditional generative adversarial network (GAN) to normalize intensity distributions on prostate MRI. We used three methods to evaluate our GAN-normalization. First, we qualitatively compared the intensity of GAN-normalized images to the intensity distributions of statistically normalized images. Second, we visually examined the GAN-normalized images to ensure the appearance of the prostate and other structures were preserved. Finally, we quantitatively evaluated the performance of deep learning holistically nested edge detection (HED) networks to identify prostate cancer on MRI when using raw, statistically normalized, and GAN-normalized images. We found the detection network trained on GAN-normalized images achieved similar accuracy and area under the curve (AUC) scores when compared to the detection networks trained on raw and statistically normalized images. Conditional GANs may hence be an effective tool for normalizing intensity distribution on MRI and can be utilized to train downstream deep learning tasks.

Ovarian volume assessment is the measurement of the size of ovaries during an Ultrasound (US) in order to estimate the ovarian reserve. Since the ovarian reserve is used in calculating a woman’s reproductive age and is also a diagnostic criterion for polycystic ovary syndrome (PCOS), it is imperative that it is measured accurately. Furthermore, ovarian rendering has clinical significance in terms of assessing ovarian anomalies (ovarian surface epithelial cells). Thus if the spacing in the US volume is high along one direction, reducing the spacing would greatly help in both the accurate measurement of the ovarian volume as well as surface assessment. In this paper, we aim to address this problem by developing a deep learning method for super-resolving 3D US data along the axial direction. On the collected dataset, our method has achieved high PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) values, and has also resulted in a 54% improvement in ovarian volume computation accuracy. Furthermore, our solution has improved the quality of the 3D rendering of the ovary, and has also reduced the problem of fused follicles in segmentation. This proves the viability of our approach for clinical diagnostic assessment.

Kidney segmentation is fundamental for accurate diagnosis and treatment of kidney diseases. Computed tomography urography imaging is commonly used for radiologic diagnosis of patients with urologic disease. Recently, 2D and 3D fully convolutional networks are widely employed for medical image segmentation. However, most 2D fully convolutional networks do not take inter-slice spatial information into consideration, resulting in incomplete and inaccurate segmentation of targets in 3D volumes. While the spatial information is truly important for 3D volumes segmentation. To tackle these problems, we propose a computed tomography urography kidney segmentation method on the basis of spatiotemporal fully convolutional networks that employ the convolutional long short-term memory network to model inter-slice features of computed tomography urography images. We trained and tested our proposed method on kidney computed tomography urography data. The experimental results demonstrate our proposed method can effectively leverage the inter-slice spatial information to achieve better (or comparable) results than current 2D and 3D fully convolutional networks.

Accurate segmentation of the aorta in computed tomography angiography (CTA) images is the first step for analysis of diseases such as aortic aneurysm, but manual segmentation can be prohibitively time-consuming and error prone. Convolutional neural network (CNN) based models have been utilized for automated segmentation of anatomy in CTA scans, with the ubiquitous U-Net being one of the most popular architectures. For many downstream image analysis tasks (e.g., registration, diameter measurement) very accurate segmentation accuracy may be required. In this work, we developed and tested a U-Net model with attention gating for segmentation of the thoracic aorta in clinical CTA data of patients with thoracic aortic aneurysm. Attention gating helps the model focus on difficult to segment target structures automatically and has been previously shown to increase segmentation accuracy in other applications. We trained U-Nets both with and without attention gating on 145 CTAs. Performance of the models were evaluated by calculating the DCS and Average Hausdorff Distance (AHD) on a test set of 20 CTAs. We found that the U-Net with attention gating yields more accurate segmentation than the U-Net without attention gating (DCS 0.966±0.028 vs. 0.944±0.022, AHD 0.189±0.134mm vs. 0.247±0.155mm). Furthermore, we explored the segmentation accuracy of this U-Net for multi-class labeling of various anatomic segments of the thoracic aorta, and found an average DCS of 0.86 for across 7 different labels. We conclude that the U-Net with attention gating improves segmentation performance and may aid segmentation tasks that require high levels of accuracy.

Myocardial perfusion scintigraphy, which is a non-invasive imaging technique, is one of the most common cardiological examinations performed today, and is used for diagnosis of coronary artery disease. Currently the analysis is performed visually by physicians, but this is both a very time consuming and a subjective approach. These are two of the motivations for why an automatic tool to support the decisions would be useful. We have developed a deep neural network which predicts the occurrence of obstructive coronary artery disease in each of the three major arteries as well as left bundle branch block. Since multiple, or none, of these could have a defect, this is treated as a multi-label classification problem. Due to the highly imbalanced labels, the training loss is weighted accordingly. The prediction is based on two polar maps, captured during stress in upright and supine position, together with additional information such as BMI and angina symptoms. The polar maps are constructed from myocardial perfusion scintigraphy examinations conducted in a dedicated Cadmium-Zinc-Telluride cardio camera (D-SPECT Spectrum Dynamics). The study includes data from 759 patients. Using 5-fold cross-validation we achieve an area under the receiver operating characteristics curve of 0.89 as average on per-vessel level for the three major arteries, 0.94 on per-patient level and 0.82 for left bundle branch block.

Intracranial aneurysms (IAs) are mostly asymptomatic and thus often discovered incidentally on angiographic scans like 3D DSA, CTA and MRA. Skilled radiologists achieved a sensitivity of 88% by means of visual detection, which seems inadequate considering that prevalence of IAs in general population is 3-5%. Deep learning models trained and executed on angiographic scans seem best-suited for IA detection, however, reported performances across different modalities is currently insufficient for clinical application. This paper presents a novel modality agnostic method for detection of IAs. First the triangulated surfaces of vascular structures were roughly extracted from the angiograms. For IA detection purpose, the extracted surfaces were randomly parcellated into local patches and then a translation, rotation and scale invariant classifier based on deep neural network (DNN) was trained. Test stage proceeded by mimicking the surface extraction and parcellation at several random locations, then the trained DNN model was applied for classification, and the results aggregated into IA detection heatmaps across entire vascular surface. For training and validation the extracted contours were presented to skilled neurosurgeon, who marked the locations of IAs. The DNN was trained and tested using three-fold cross-validation based on 57 DSAs, 5 CTAs and 5 MRAs and showed a 98.6% sensitivity at 0.2 false positive detections per image. Experimental results show that proposed approach not only significantly improved detection sensitivity and specificity compared to state-of-the-art intensity based methods, but is also modality agnostic and thus better suited for clinical application.

Four-dimensional (4D) left ventricular myocardial velocity mapping (MVM) is a cardiac magnetic resonance (CMR) technique that allows assessment of cardiac motion in three orthogonal directions. Accurate and reproducible delineation of the myocardium is crucial for accurate analysis of peak systolic and diastolic myocardial velocities. In addition to the conventionally available magnitude CMR data, 4D MVM also acquires three velocity-encoded phase datasets which are used to generate velocity maps. These can be used to facilitate and improve myocardial delineation. Based on the success of deep learning in medical image processing, we propose a novel automated framework that improves the standard U-Net based methods on these CMR multi-channel data (magnitude and phase) by cross-channel fusion with attention module and shape information based post-processing to achieve accurate delineation of both epicardium and endocardium contours. To evaluate the results, we employ the widely used Dice scores and the quantification of myocardial longitudinal peak velocities. Our proposed network trained with multi-channel data shows enhanced performance compared to standard UNet based networks trained with single-channel data. Based on the results, our method provides compelling evidence for the design and application for the multi-channel image analysis of the 4D MVM CMR data.

Retinal toxicity among long-term users of Hydroxychloroquine manifests with loss in the Ellipsoid zone (EZ) detectable on SD-OCT imaging. This work reports an automatic deep-learning algorithm to detect and segment EZ loss in SD-OCT. The proposed model predicts EZ loss map, in a dual network architecture that operates in parallel combining scan-by-scan detections in horizontal and vertical directions. The combined model demonstrated the best overall performance with F1 score = 0.91 ± 0.07, improving the performance compared to individual models. Automatic methods for EZ loss detection could provide a useful tool to facilitate screening of patients for evidence of toxicity.

Convolutional neural networks (CNNs) have been increasingly applied to computer-aided diagnosis (CADx) for lung nodule malignancy prediction, which usually is a binary classification task. However, CNNs were often difficult to capture optimal features, thereby affect the classification performance. This study developed a CADx system based on a CNN model with auxiliary task learning to predict lung nodule malignancy in chest computed tomography (CT) scans. Our CADx system took raw CT image cubes centering at nodules as input and generated one main output and eight auxiliary outputs. The main output predicted lung nodule malignancy; the auxiliary outputs predicted lesion size and characteristics. The auxiliary tasks offered assistance for predicting the final nodule malignancy. The performance of the developed lung nodule CADx system was verified by use of the LIDC dataset. Results showed that our CADx system achieved improved performance for lung nodule malignancy prediction.

As a subclass of interstitial lung diseases, nonspecific interstitial pneumonia (NSIP), defined by an association of inflammation and fibrotic lung lesions, leads to progressive loss of pulmonary function and premature mortality. Among fibrosis manifestations, traction bronchiectasis (TB) is one of the most critical, its presence being associated to poor prognosis. Accurate follow up is crucial to treatment decision and adaptation. In this paper we propose quantitative biomarkers for TB, associating spatial localization and severity measures. Bronchial tree segmentation and TB identification first exploited a semi-automatic approach for the definition of the ground truth, then used an automatic method involving a convolutional neural network. The CNN training/testing database included 73/18 patients respectively, with both baseline and follow up exams at one year; the detection performance was assessed in terms of precision, recall and F1-score. For both ground truth and CNN-segmented data, the following TB biomarkers were derived: TB ratio over total airway, airway volume, normalized tree length, and airway caliber (absolute and relative) deviation from the normal 2-power decrease law. A correlation study between the TB biomarkers and the initial and follow up pulmonary function tests was conducted. Among ground truth-based TB biomarkers, airway volume, normalized tree length, absolute caliber deviation and TB ratio over total airway showed significant negative correlation with both initial and follow-up pulmonary function parameters, manifesting as potential prognosis biomarkers. Among CNN-based TB biomarkers, airway volume, normalized tree length, relative and absolute caliber deviation showed significant negative correlation with follow-up pulmonary function parameters confirming their predictive potential.

The coronavirus pandemic, also known as COVID-19 pandemic, has led to tens of millions of cases and over half of a million deaths as of August 2020. Chest CT is an important imaging tool to evaluate the severity of the lung involvement which often correlates with the severity of the disease. Quantitative analysis of CT lung images requires the localization of the infection area on the image or the identification of the region of interest (ROI). In this study, we propose an automatic ROI identification based on the recent feature selection method, called concrete autoencoder, that learns the parameters of concrete distributions from the given data to choose pixels from the images. To improve the discrimination of these features, we proposed a discriminative concrete autoencoder (DCA) by adding a classification head to network. This classification head is used to perform the image classification. We conducted a study with 30 CT image sets from 15 Covid-19 positive and 15 COVID19 negative cases. When we used the DCA to select the pixels of the suspected area, the classification accuracy was 76.27% for the image sets. Without DCA feature selection, the traditional neural network achieved an accuracy of 69.41% for the same image sets. Hence, the proposed DCA could detect significant features to identify the COVID-19 infected area of lung. Future work will focus on surveying more data, designing area selection layer towards group selection.

As the rapid spread of coronavirus disease (COVID-19) worldwide, X-ray chest radiography has also been used to detect COVID-19 infected pneumonia and assess its severity or monitor its prognosis in the hospitals due to its low cost, low radiation dose, and broad accessibility. However, how to more accurately and efficiently detect COVID-19 infected pneumonia and distinguish it from other community-acquired pneumonia remains a challenge. In order to address this challenge, we in this study develop and test a new computer-aided detection and diagnosis (CAD) scheme. It includes pre-processing algorithms to remove diaphragms, normalize X-ray image contrast-to-noise ratio, and generate three input images, which are then linked to a transfer learning based convolutional neural network (VGG16 model) to classify chest X-ray images into three classes of COVID-19 infected pneumonia, other community-acquired pneumonia and normal (non-pneumonia) cases. To this purpose, a publicly available dataset of 8,474 chest X-ray images is used, which includes 415 confirmed COVID-19 infected pneumonia, 5,179 community-acquired pneumonia, and 2,880 non-pneumonia cases. The dataset is divided into two subsets with 90% and 10% of images to train and test the CNN-based CAD scheme. The testing results achieve 93.9% of overall accuracy in classifying three classes and 98.6% accuracy in detecting COVID-19 infected pneumonia cases. The study demonstrates the feasibility of developing a new deep transfer leaning based CAD scheme of chest X-ray images and providing radiologists a potentially useful decision-making supporting tool in detecting and diagnosis of COVID-19 infected pneumonia.

As deep learning greatly accelerates the field of computer vision, there has been growing interest in applying deep learning models for the purpose of predicting the presence of cancer in mammography images. However, unlike in conventional object recognition where one can leverage very large diverse datasets such as ImageNet, datasets for identifying cancer with mammography images are typically small and potentially non-representative due to the high cost of acquiring medical data and labels. This makes the training and assessment of such models challenging and raises reliability as well as generalizability concerns. In this work, we propose using the jigsaw task1 as a self-supervised method to pre-train models in the case where unlabeled data is available. We show that models that are pre-trained with this task outperform randomly initialized models even when they are only trained on a half or a quarter of the train set for the malignancy prediction task. In particular, we find that when using only a quarter of the labeled data, model trained using randomly initialized weights has an area under the receiver operating characteristic curve (AUC) of 0.944. On the other hand, the model that was pre-trained with the jigsaw task achieved an AUC of 0.958 when fine-tuned on the same quarter of the training set for the malignancy prediction task, outperforming even the model that was trained on all of the labeled data starting from randomized weights (0.954 AUC). Furthermore, we propose using performance on the jigsaw task as a way to measure confidence in our model’s predictions to enable the option to abstain from making a prediction when the model is not confident. We tested multiple strategies to filter out samples on which the jigsaw model perform poorly and measured the AUC in the remaining pool of samples. We show that the best filtering strategy improves malignancy prediction performance from an AUC of 0.890 on a completely unfiltered, off-site test set from a different country to an AUC of 0.913 on the filtered set.

Characterizing case-based repeatability of lesion classification in computer-aided diagnosis (CADx) can support clinical application of CADx in breast cancer imaging. A database of 516 full-field digital mammographic (FFDM) images, 268 unique breast lesions/cases (138 malignant, 130 benign) was retrospectively collected under HIPAA/IRB approval. Lesions were automatically segmented through a region-growing method. Twenty-nine radiomic features were extracted, characterizing lesion size, shape, margin, texture, and contrast. A 0.632 bootstrap/1000 iterations was used to divide cases into training and test folds for support vector machine classification as malignant or benign. Classifier output was averaged by case. The 0.632+ bootstrap-corrected area under the receiver operating characteristic (ROC) curve (AUC) in the task of classification of lesions as malignant or benign was determined. Classifier repeatability was determined as a function of median classifier output across all cases obtained in bootstraps sorted into 10 equal bins. 95% confidence intervals (CI) of sensitivity and specificity were examined as a function of classifier output, i.e., the decision variable in ROC analysis. Actual sensitivity and specificity in the test folds were determined for a target sensitivity (0.95), a target specificity (0.95), and an ‘optimal’ combination that minimized (1-sensitivity)² + (1-specificity)². The AUC [95% CI] was 0.82 [0.77, 0.86]. Classifier output was more repeatable at lower and upper ranges. Sensitivity and specificity at the targets were (median, [95% CI]) at 0.92 [0.83, 0.98] and 0.92 [0.82, 0.98] respectively. At an ‘optimal’ decision threshold, sensitivity was 0.73 [0.59, 0.85] and specificity 0.76 [0.62, 0.88]. These results characterize SVM classification repeatability for radiomic features extracted from FFDM images.

Our ultimate goal is to develop a personalized breastscreening model. This model will provide information to an individual woman for planning her own screening options based on her imaging and other characteristics. To do so, we are developing a computerized method to estimate a near term risk of screen detectable breast cancer (sBCa) by analyzing lateral and temporal differences in screening mammograms using the Radon Cumulative Distribution Transform (RCDT), a deep convolutional neural network (dCNN). The RCDT can highlight subtle signals associated with future sBCa by exploiting lateral and temporal differences in screening mammograms of women. We used a dCNN to build a robust classification marker of having sBCa within a year from a negative screening. We explored the potential for incremental improvement over known risk factors by combining the dCNN with age and showed consistent gains in identifying asymptomatic women with a low risk of having sBCa (who could use the personalized screening option to safely skip their next screening). We used 4296 screening mammogram images of 537 women with 151 sBCa cases and fine-tuned the VGG16 network using a data split ratio of 0.64:0.16:0.2 for training, validation, and testing. The final model on the independent testing dataset achieved an AUC of 0.82, and a specificity at a sensitivity level of 98% (Sp@Se98) of 0.51 with a 95% confidence interval of 0.40 to 0.65. This indicates that the model could reliably identify at least 40% of low-risk women with a 2% error for a false negative rate.

Histologically normal areas of the breast parenchyma have been shown to share molecular similarity with breast tumors, suggesting the presence of a field effect in breast cancer. To further understand a potential cancer field effect, we compared mammographic parenchymal texture features across four regions of the breast. The study included 103 FFDMs with at least one identified malignant tumor. All FFDM images (12-bit quantization and 70 micron pixels) were acquired with a Hologic Lorad Selenia system and retrospectively collected under an IRB-approved protocol. Regions of interest (ROI) of 128x128 and 256x256 pixels were selected from four regions across the craniocaudal projection: within the identified tumor, adjacent to the tumor, distant from the tumor, and behind the nipple in the contralateral breast. Radiographic texture analysis was used to extract 45 features in each region. Kolmogorov-Smirnov (KS) and Pearson correlation tests assessed similarity between features in each region. KS test results, with a 95% confidence interval on the KS test statistic bootstrapped with 2000 iterations indicated that 81.8% (128x128) and 88.4% (256x256) of feature distributions across all ROI regions showed equivalence with a threshold equal to the critical value at the p = 0.05 level. Pearson correlation results demonstrated a majority of structure-based feature comparisons which reached statistical significance, and less intensity-based feature comparisons which reached statistical significance. These results support our hypothesis of a potential cancer field effect across tumor and non-tumor regions and support the development of computerized analysis of mammographic parenchymal patterns to assess breast cancer risk.

Deep learning models are traditionally trained purely in a data-driven approach; the information for the model training usually only comes from a single source of the training data. In this work, we investigate how to supply additional clinical knowledge that is associated with the training data. Our goal is to train deep learning models for breast cancer diagnosis using mammogram images. Along with the main classification task between clinically proven cancer vs negative/benign cases, we design two auxiliary tasks each capturing a form of additional knowledge to facilitate the main task. Specifically, one auxiliary task is to classify images according to the radiologist-made BI-RADS diagnosis scores and the other auxiliary task is to classify images in terms of the BI-RADS breast density categories. We customize a Multi-Task Learning model to jointly perform the three tasks (main task and two auxiliary tasks). We test four deep learning architectures: CBR–Tiny, ResNet18, GoogleNet, and DenseNet and we investigate the benefit of incorporating such knowledge over ImageNet pre-trained models and in the case of randomly initialized models. We run experiments on an internal dataset consisting of screening full field digital mammography images for a total of 1,380 images (341 cancer and 1,039 negative or benign). Our results show that, by adding clinical knowledge conveyed through the two auxiliary tasks to the training process, we can improve the performance of the target task of breast cancer diagnosis, thus highlighting the benefit of incorporating clinical knowledge into data-driven learning to enhance deep learning model training.

Developing a computer-aided diagnosis (CAD) scheme to classify between malignant and benign breast lesions can play an important role in improving MRI screening efficacy. This study demonstrates that extracting features from both spatial and frequency domains, and applying an efficient combination of data reduction and classifier methods, had the potential to significantly improve accuracy in classifying between malignant and benign breast masses. By applying our CAD scheme to the testing dataset, we obtained an accuracy of 83.1% for the best combination of data reduction and classification (DNE-SVM).

Elbow fractures are one of the most common fracture types. Diagnoses on elbow fractures often need the help of radiographic imaging to be read and analyzed by a specialized radiologist with years of training. Thanks to the recent advances of deep learning, a model that can classify and detect different types of bone fractures needs only hours of training and has shown promising results. However, most existing deep learning models are purely data-driven, lacking incorporation of known domain knowledge from human experts. In this work, we propose a novel deep learning method to diagnose elbow fracture from elbow X-ray images by integrating domain-specific medical knowledge into a curriculum learning framework. In our method, the training data are permutated by sampling without replacement at the beginning of each training epoch. The sampling probability of each training sample is guided by a scoring criterion constructed based on clinically known knowledge from human experts, where the scoring indicates the diagnosis difficultness of different elbow fracture subtypes. We also propose an algorithm that updates the sampling probabilities at each epoch, which is applicable to other sampling-based curriculum learning frameworks. We design an experiment with 1865 elbow X-ray images for a fracture/normal binary classification task and compare our proposed method to a baseline method and a previous method using multiple metrics. Our results show that the proposed method achieves the highest classification performance. Also, our proposed probability update algorithm boosts the performance of the previous method.

A radiomics-based model was developed to predict bone fragility based on x-ray projection images. The model was trained using ROIs of MicroCT images, in which mean von Mises stress was computed using finite element modeling. The average normalized RMSE of the estimated von Mises stress of all folds was 14.7%. The model identifies GLSZM and GLDM features as important texture features for estimating stress. This work demonstrates radiomic features from projection images can be combined with nonlinear regression techniques to predict bone fragility and may result in a more accurate projection-domain estimate than areal bone mineral density.

Meniscus segmentation from knee MR images is an essential step in finding the most suitable implant prototype for meniscus allograft transplantation using a 3D reconstruction model from the patient’s normal meniscus. However, the segmentation of the meniscus is challenging due to its thin shape, similar intensities with nearby structures such as cruciate and collateral ligaments in the knee MR images, large shape variations among patients, and inhomogeneous intensity within the meniscus. In addition, conventional deep convolutional neural network (DCNN)-based meniscus segmentation method mainly uses a pixel-wise objective function, and thus produces rather under-segmentation results due to the small shape of the meniscus or suffers from false positives that occur around the meniscus. To overcome these limitations, we propose a two-stage DCNN that combines a 2D U-Net-based meniscus localization network with a conditional generative adversarial network-based segmentation network. To efficiently localize the meniscus region to medial and lateral meniscus and feed the localized ROIs to the segmentation network, 2D U-Net-based DCNN segments knee MR images into six classes. To segment the medial and lateral meniscus while preventing under-segmentation due to intensity inhomogeneity within the meniscus and over-segmentation due to intensity similarity with surrounding structures, adversarial learning is performed repeatedly on the localized meniscus ROIs. The average DSC of the meniscus was 84.06% at the medial meniscus, and 83.19% at the lateral meniscus, respectively. These results showed that the proposed method prevented the meniscus from being over- and under-segmented by repeatedly judging and complementing the quality of segmentation results through adversarial learning.

The diagnosis of cervical spinal canal stenosis currently relies on radiological reports, based largely on subjective interpretation of magnetic resonance images (MRIs). This leads to a high degree of variation in diagnosis and treatment of pathology in the cervical spine. Despite the capabilities of MRI to provide high contrast and high spatial resolution 3D images, the prevalence of inconsistent diagnosis has presented a challenge for delivering necessary care. Recently, deep learning (DL) based segmentation algorithms have demonstrated great potential for automating the measurement and segmentation of cervical spine anatomy. We show the results of a Deep-U-Net ensemble based automated pipeline to detect and segment intervertebral discs at the C1-C7 level in sagittal spine MRIs. Training was done using scans from (n = 50) individuals wherein discs were segmented by a trained human rater (R1). Validation was performed on (n = 50) scans by comparing automated segmentations to those performed by two human raters (R1 and R2). We show that the average Dice similarity coefficient was 0.79 when comparing algorithm to R1, 0.76 when comparing algorithm to R2, and 0.78 when comparing R1 to R2. Further, segmentations by R1, R2 and algorithm differed (ANOVA with p ≤.0002), and the median Dice score comparing algorithm to R1 was greater than that comparing algorithm to R2, indicating bias toward R1. Thus, our preliminary demonstration highlights the potential of using deep learning techniques to create reliable, quantitative biomarkers to diagnose and measure cervical spine pathology. Our work additionally cautions us to the challenges posed by rater-specific bias induced by the training process in deep learning segmentation algorithms.

We are developing a decision support tool for treatment response monitoring of multiple myeloma (MM) disease in spinal MRI scan images. This study investigated the feasibility of using deep learning to stratify the risks for patients who underwent bone marrow transplant (BMT) and assessed its prognostic value in predicting time to progression (TTP) after BMT. We combined a convolutional neural network (CNN) with a recurrent neural network (RNN), referred to as C-RNN model, to classify the low and high risk groups of patients with pre- and post-BMT MRI scans. The CNN was used as the encoder with the pair of pre- and post-BMT MR images as input, and the RNN was used as the decoder to receive time-sequence vectors output from the CNN encoder for classification of patients with high and low risk of progression within a certain time. With IRB approval, 63 pairs of pre- and post-BMT T1W sagittal view of MRI scans and the time to progression (TTP) within 5 years of follow up for each patient were collected retrospectively from 63 MM patients at our institution. With respect to the TTP censored at 24 months, 41 and 22 patients were separated to the low risk and high risk groups as reference standard. Our C-RNN was trained and validated with 5-fold cross-validation. The results showed that the C-RNN achieved an average test AUC of 0.801±0.037. The Kaplan-Meier analysis showed that the high risk group patients identified by the C-RNN model had significantly shorter TTP than those low risk patients (P<0.05 by log-rank test).

Infants are sensitive to pain and discomfort in their daily lives and frequent discomfort and pain can even lead to abnormal brain development, resulting in long-term adverse neurodevelopmental outcomes. With the help of video-based monitoring, a contactless method is considered to be promising for detecting discomfort automatically. In this study, a method for distinguishing infant discomfort status from comfort is proposed. We first extract two-dimensional (2D) features from video frames using pretrained Convolutional Neural Networks (CNNs), which is followed by two different Long Short-term Memory (LSTM) networks (uni- and bi-directional LSTMs) for the comfort/discomfort classification task. The methods are evaluated using videos acquired from 23 infants. Experimental results show the best AUC of 0.89 is achieved when using the bi-directional LSTM model based on the 2D features extracted by the VGG16 network. The high detection score indicates that the proposed method is promising for clinical use.

Pregnancy requires constant monitoring by health care providers to avoid conditions that may threaten the lives of the fetus and the mother at birth. For labor management, the diagnosis of fetal presentation is essential to guarantee delivery viability. A direct indicator of fetal presentation is the fetal head location, which can be placed close to the canal birth (cephalic, head-first) or far from the canal birth (breech, feet first). Unlike urban areas, the population in rural zones experience difficulties in accessing healthcare monitoring. Although telemedicine has helped bring medical technology closer to these regions, the diagnosis still requires medical specialists. This study presents an automatic three-stage detection framework for assessing the fetal head position (and hence the fetal presentation). The first stage involves applying morphological filtering, intensity-based thresholding, and shapebased filtering for a preliminary head detection and segmentation. The second stage comprises segmentation enhancement using a combinatory approach. The third stage uses the detection results to depict the spatial location likelihood of the head, which indicates the head location and fetal presentation. Fifteen volunteers in the third trimester of pregnancy were evaluated, and the fetal presentation was diagnosed. These results were compared with the diagnosis of a radiologist (used as a gold standard). The proposed method presented a 100% accuracy in determining the fetal presentation, albeit a limited number of cases evaluated.

Medical image segmentation has a fundamental role in many computer-aided diagnosis (CAD) applications. Accurate segmentation of medical images is a key step in tracking changes over time, contouring during radiotherapy planning, and more. One of the state-of-the-art models for medical image segmentation is the U–Net that consists of an encoder-decoder based architecture. Many variations exist to the U–Net architecture. In this work, we present a new training procedure that combines U–Net with an adversarial training we refer to as Adversarial U–Net. We show that Adversarial U–Net outperformes the conventional U–Net in three versatile domains that differ in the acquisition method as well as the physical characteristics and yields smooth and improved segmentation maps.

Endoscopic submucosal dissection is a minimally invasive treatment for early gastric cancer. In endoscopic submucosal dissection, a physician directly removes the mucosa around the lesion under internal endoscopy by using the flush knife. However, the flush knife may accidentally pierce the colonic wall and generate a perforation on it. If physicians overlooking a small perforation, a patient may need emergency open surgery, since a perforation can easily cause peritonitis. For the prevention of overlooking of perforations, a computer-aided diagnosis system has a potential demand. We believe automatic perforation detection and localization function is very useful for the analysis of endoscopic submucosal dissection videos for the development of a computeraided diagnosis system. At current stage, the research of perforation detection and localization progress slowly, automatic image-based perforation detection is very challenge. Thus, we devote to the development of detection and localization of perforations in colonoscopic videos. In this paper, we proposed a supervised-learning method for perforations detection and localization in colonoscopic videos. This method uses dense layers in YOLO-v3 instead of residual units, and a combination of binary cross entropy and generalized intersection over union loss as the loss function in the training process. This method achieved 0.854 accuracy, 0.850 AUC score and 0.884 mean average precision for perforation detection and localization, respectively, as an initial study

The superior soft-tissue contrast obtained by MRI allows contouring target and organs-at-risk (OARs) accurately and reliably in head-and-neck cancer radiotherapy. However, the delineating of OARs is labor-intensive and subjective. We propose a mask regional convolutional neural network (R-CNN) for fully automated OAR delineation in head-and-neck (HN) MRI. A deep attention feature pyramid network (DAFPN) was used as the backbone to extract pyramid features, while the location and size of each region-of-interest (ROI) are estimated by the feeding features extracted from DAFPN into a regional proposal network (RPN). The final OAR contours were predicted by feeding cropped features within these ROIs (location and size) into a subnetwork to perform segmentation within ROIs. A retrospective study was performed on 45 HN cancer patients who underwent radiation therapy. MR images and their manual contours were collected for training and testing the proposed method. Five-fold cross-validation method was conducted to evaluate the proposed method. The mean Dice similarity coefficients of brain stem, esophagus, larynx, mandible, optic chiasm, left optic nerve, right optic nerve, oral cavity, left parotid, right parotid, pharynx, and spinal cord are 0.88, 0.52, 0.88, 0.77, 0.56, 0.59, 0.57, 0.90, 0.81, 0.81, 0.76, and 0.69, respectively. After the model training, all OAR can be segmented within 10 seconds. The accurate HN OAR delineation enables further development of MRI-only based radiotherapy workflow for HN cancer.

Congenital heart disease (CHD) is the most common congenital abnormality associated with birth defects in the United States. Despite training efforts and substantial advancement in ultrasound technology over the past years, CHD remains an abnormality that is frequently missed during prenatal ultrasonography. Therefore, computeraided detection of CHD can play a critical role in prenatal care by improving screening and diagnosis. Since many CHDs involve structural abnormalities, automatic segmentation of anatomical structures is an important step in the analysis of fetal echocardiograms. While existing methods mainly focus on the four-chamber view with a small number of structures, here we present a more comprehensive deep learning segmentation framework covering 14 anatomical structures in both three-vessel trachea and four-chamber views. Specifically, our framework enhances the V-Net with spatial dropout, group normalization, and deep supervision to train a segmentation model that can be applied on both views regardless of abnormalities. By identifying the pitfall of using the Dice loss when some labels are unavailable in some images, this framework integrates information from multiple views and is robust to missing structures due to anatomical anomalies, achieving an average Dice score of 79%.

Breast cancer is the second leading cause of cancer-related death in women. Ultrasound imaging has been widely used for the early detection of breast cancer because of its superior ability in imaging dense breast tissue and its lack of ionizing radiation. However, ultrasound imaging heavily depends on practitioners’ experience and thus becomes relatively subjective. In this work, we proposed a novel multi-scale view-based convolutional neural network (MSVCNN) to assist doctors to diagnose and improve classification accuracy. MSV-CNN takes full images, regions of interest (ROI), and the tumor regions with two times size of the ROI as input. It adopts three complementary branches to learn multi-scale view features from different views. The sub-networks in all branches have the same structure but with different parameters. The output of three branches is finally concatenated and fused by a fully connected layer for automated nodule classification. To assess the performance of our proposed network, we implemented breast ultrasound classification on the dataset containing 1560 images with benign nodules and 5367 images with malignant nodules. Furthermore, ResNet-18 models trained with different views were utilized as baselines. Experimental results showed that MSV-CNN achieved an average classification accuracy of 0.907. This preliminary study demonstrated that our proposed method is effective in the discrimination of breast nodules.

Despite the increasingly refined protocols in the diagnosis of Parkinson’s disease (PD), detection of PD at an early stage remains challenging. As one of the most common non-motor symptoms of PD, olfactory dysfunction yields a high prevalence of < 90% and is observed years prior to the onset of motor symptoms. Thus, olfaction-based assessment may help premotor diagnosis of PD. Nevertheless, diagnosis of PD based on olfactory dysfunction is not yet realized in clinical settings due to low specificity, therefore, it would be of high clinical relevance to differentiate PD-induced olfactory dysfunction from other forms of non-Parkinsonian olfactory dysfunction (NPOD). In the present study, we assess the feasibility of machine learning approaches in the classification of PD and NPOD patients and show that PD patients could be differentiated from NPOD patients using (a) morphometric measurements including cortical thickness, cortical surface area and volumetric measures of parcellated cortical and subcortical brain regions, or (b) T1-weighted axial magnetic resonance imaging (MRI) scans. The proposed methods give rise to efficient classification of PD and NPOD, and may thus improve the specificity of olfaction-based diagnosis of PD.

Digital subtraction angiography (DSA) is the main imaging modality used to assess reperfusion during mechanical thrombectomy (MT) when treating large vessel occlusion (LVO) ischemic strokes. To improve this visual and subjective assessment, hybrid models combining angiographic parametric imaging (API) with deep learning tools have been proposed. These models use convolutional neural networks (CNN) with single view individual API maps, thus restricting use of complementary information from multiple views and maps resulting in loss of relevant clinical information. This study investigates use of ensemble networks to combine hemodynamic information from multiple bi-plane API maps to assess level of reperfusion. Three-hundred-eighty-three anteroposterior (AP) and lateral view DSAs were retrospectively collected from patients who underwent MTs of anterior circulation LVOs. API peak height (PH) and area under time density curve (AUC) maps were generated. CNNs were developed to classify maps as adequate/inadequate reperfusion as labeled by two neuro-interventionalists. Outputs from individual networks were combined by weighting each output, using a grid search algorithm. Ensembled, AP-AUC, AP-PH, lateral-AUC, and lateral-PH networks achieved accuracies of 83.0% (95% confidence-interval: 81.2%-84.8%), 74.4% (72.0%-76.7%), 74.2% (72.8%-75.7%), 74.9% (72.2%-77.7%), and 76.9% (74.4%-79.5%); area under receiver operating characteristic curves of 0.86 (0.84-0.88), 0.81 (0.79-0.83), 0.83 (0.81-0.84), 0.82 (0.8-0.84), and 0.84 (0.82-0.87); and Matthews correlation coefficients of 0.66 (0.63-0.70), 0.48 (0.43- 0.53), 0.49 (0.46-0.52), 0.51 (0.45-0.56), and 0.54 (0.49-0.59) respectively. Ensembled network performance was significantly better than individual networks (McNemar’s p-value<0.05). This study proved feasibility of using ensemble networks to combine hemodynamic information from multiple bi-plane API maps to assess level of reperfusion during MTs.

The literature manifests that schizophrenia is associated with alterations in brain network connectivity. We investigate whether large-scale Extended Granger Causality (lsXGC) can capture such alterations using restingstate fMRI data. Our method utilizes dimension reduction combined with the augmentation of source time-series in a predictive time-series model for estimating directed causal relationships among fMRI time-series. The lsXGC is a multivariate approach since it identifies the relationship of the underlying dynamic system in the presence of all other time-series. Here lsXGC serves as a biomarker for classifying schizophrenia patients from typical controls using a subset of 62 subjects from the Centers of Biomedical Research Excellence (COBRE) data repository. We use brain connections estimated by lsXGC as features for classification. After feature extraction, we perform feature selection by Kendall’s tau rank correlation coefficient followed by classification using a support vector machine. As a reference method, we compare our results with cross-correlation, typically used in the literature as a standard measure of functional connectivity. We cross-validate 100 different training/test (90%/10%) data split to obtain mean accuracy and a mean Area Under the receiver operating characteristic Curve (AUC) across all tested numbers of features for lsXGC. Our results demonstrate a mean accuracy range of [0.767, 0.940] and a mean AUC range of [0.861, 0.983] for lsXGC. The result of lsXGC is significantly higher than the results obtained with the cross-correlation, namely mean accuracy of [0.721, 0.751] and mean AUC of [0.744, 0.860]. Our results suggest the applicability of lsXGC as a potential biomarker for schizophrenia.

Ear infections are exceedingly common, yet challenging to diagnose correctly. The diagnosis requires a clinician (such as a physician, nurse practitioner, or physician assistant) to use an otoscope and inspect the eardrum (i.e. tympanic membrane). Once visualized the clinician must rely on clinical judgment to determine the presence of changes typically associated with an ear infection such as eardrum color and/or position. Research has however consistently demonstrated systemic failure among clinicians to correctly diagnose and manage ear infections. With recent advancements of pattern recognition techniques, including deep learning, there has been increasing interest in the opportunity to automate the diagnosis of ear infections. While there are some previous studies that successfully apply machine learning to classify ear drum photos, these methods were developed and evaluated in non-real world settings and used single, crisp, still-shot photos of the eardrum that would be labor-intensive to acquire in uncooperative pediatric patients. Contrary to previous works, we present a deep anomaly detection based method that flags otoscopy video sequences as normal or abnormal, achieving a promising first step towards automated analysis of otoscopy video for in-clinic or at-home screening.

Gliomas are very heterogenous set of tumors that grow within the substance of brain and often mix with normal brain tissues. Due to its histologic complexity and irregular shapes, multiparametric magnetic resonance imaging is used to accurately diagnose brain tumor and their subregions. Current practice requires physicians to manually segment these regions on a large image dataset, which can be a very time consuming and complicated task especially with large variations among different tumor regions. Automatic segmentation of brain tumors in multimodal MRI holds a great potential in developing an effective treatment plan and improving brain tumor radiotherapy workflow. Despite continuous investigations on DL-based brain tumor segmentation, irregular shapes and histologic complexities of brain tumors introduces a major challenge in developing an effective automatic segmentation method. In this study, we develop a novel context U-net with deep supervision to segment both the whole brain tumor and their subregions of tumors. The context module was formed by an inception-like structure to extract more information regarding the brain tumors. The deep supervision in the encoder path was achieved by adding up the segmentation outputs at different levels of the network. We evaluated our method on Brain Tumor Segmentation Challenge (BraTS) 2019 training dataset in which 80% was used for training while the remaining 20% was used for performance testing. Our method achieved Dice similarity coefficients (DSC) were 0.8693, 0.8013 and 0.7782 for the whole tumor (WT), tumor core (TC) and enhancing tumor (ET), respectively. The results attained by our proposed network suggested that this technique could be used for segmentation of brain tumors and their subregions to facilitate the brain tumor radiotherapy workflow.

Purpose: to develop and validate algorithms that enable a novice user to quantitatively measure the head shape parameters associated with deformational plagiocephaly, and brachycephaly (DPB) using a smartphone or tablet. Method: We have developed a technology (called SoftSpot^TM) based on advanced imaging algorithms to detect different types and severity of DPB from top-view photos of the head acquired by a novice user at the point-of-care, i.e. in pediatric offices, and at home. Currently, the head shape parameters are measured using either a 3D scanner or a mechanical caliper by a specialist such a pediatric neurosurgeon or an orthotist. In our approach, the head contour is extracted semiautomatically using the intelligent scissors method. We then automatically compute two indices used in the clinical determination of the DPB: the cranial index (CI), and the cranial vault asymmetry index (CVAI). In this paper, we also present methods to quantify, and compensate for the user variability in the acquisition of photos, including camera angle, and distance from the head, by combining the results from different camera positions. We compared the results of our technology with ground truth measurements from 53 infants with DPB, and normal cranial parameters. Accuracy analysis was performed by Bland Altman (BA) method, and the Spearman correlation test. Results: The Spearman correlation coefficients between the new 2D method, and the 3D ground truth were 0.94 (p<0.001), and 0.96 (p<0.001) for CI and CVAI, respectively. Different camera angles, and distances from the head resulted in variation in CI and CVAI in the range of [-2.0, 6.0], and [-4.0, 4.0] units, respectively. The limit of agreement was reduced from [-3.6, 5.3], and [-3.6, 4.2] to [-0.5, 3.0], and [-1.3, 1.6] for CI and CVAI, respectively, by combining results from different camera angles, and positions in our method. The overall accuracy of the proposed technology for DPB detection was 100%. Conclusions: Photographic 2D images can be accurately analyzed to assess DPB at the point-of-care. By compensating for the error from variable camera angles, and distance from the head, our technology eliminates user variability. The algorithms will be packaged in a mobile application to enable the use of the technology at the point-of-care.

We evaluated whether a computerized decision support system for bladder cancer treatment response assessment (CDSS-T) can assist physicians from different institutions in identifying patients who have complete response after neoadjuvant chemotherapy. Pre- and post-chemotherapy CTU scans of 96 patients (114 pre- and post-treatment lesion pairs) were collected retrospectively. The pathological cancer stage after treatment was collected as the reference standard of response to treatment. 24% of the lesion pairs had T0 cancer stage (complete response) after chemotherapy. Our CDSST that combined DL-CNN and radiomics features was trained to distinguish between T0 and <T0 cases. Five abdominal radiologists and 3 oncologists participated in the observer study. One radiologist and one oncologist were from external institutions. All physicians estimated the likelihood of stage T0 disease after treatment by viewing each pre-post-treatment CTU pair displayed side by side on a specialized graphical user interface. The observer provided an estimate without CDSS-T first and then might revise the estimate, if preferred, after the CDSS-T score was displayed. The observers’ estimates with and without CDSS-T were analyzed with multi-reader, multi-case (MRMC) methodology. The AUC for prediction of T0 disease after treatment was 0.85±0.04 for the CDSS-T alone. The performance of all but one observers increased with the aid of CDSS-T. The average AUC for the observers was 0.77 (range: 0.69-0.83) without CDSS-T, and increased to 0.80 (range: 0.72-0.86), (p = 0.006) with CDSS-T. The CDSS-T could improve the performance of radiologists and oncologists from different institutions in identifying patients who fully responded to treatment. There was no apparent difference in the performance of the physicians from different institutions.

In this study, a learning-based method using mask R-CNN is proposed to automatically segment prostate and its dominant intraprostatic lesions (DILs) from magnetic resonance (MR) images. The mask RCNN is able to perform end-to-end segmentation by locating the target region-of-interest (ROI) and then segmenting target within that ROI. The ROI locating step can improve the efficiency of the segmentation step by decreasing the image size. Dual attention networks are used as backbone in mask R-CNN to extract comprehensive features from MR images. The binary mask of targets of an arrival patient’s MR image is generated by the well-trained network. To evaluate the proposed method, we retrospectively investigate 25 MRI datasets. On each dataset, prostate and DILs were delineated by physicians and was served as ground truth and training target. The proposed method was trained and evaluated by a five-fold cross validation strategy. The average centroid distance, volume difference and DSC value for prostate/DIL among all 25 patients are 0.85±2.62mm/2.77±2.13, 0.58±0.52cc/1.72±1.74cc and 0.95±0.09/0.69±0.12, respectively. The proposed method has shown accurate segmentation performance, which is promising in improving the efficiency and mitigating the observer-dependence in prostate and DIL contouring for DIL focal boost radiation therapy.

Prostate MRI is increasingly used to help localize and target prostate cancer. Yet, the subtle differences in MRI appearance of cancer compared to normal tissue renders MRI interpretation challenging. Deep learning methods hold promise in automating the detection of prostate cancer on MRI, however such approaches require large, well-curated datasets. Although existing methods that employed fully convolutional neural networks have shown promising results, the lack of labeled data can reduce the generalization of these models. Self-supervised learning provides a promising avenue to learn semantic features from unlabeled data. In this study, we apply the self-supervised strategy of image context restoration to detect prostate cancer on MRI and show this improves model performance for two different architectures (U-Net and Holistically Nested Edge Detector) compared to their purely supervised counterparts. We train our models on MRI exams from 381 men with biopsy confirmed cancer. Our study showed self-supervised models outperform randomly initialized models on an independent test set in a variety of training settings. We performed 3 experiments, where we trained with 5%, 25% and 100% of our labeled data, and observed that the U-Net based pre-training and downstream task outperformed other models. We observed the best improvements when training with 5% of the labeled training data, our selfsupervised U-Nets improve per-pixel Area Under the Curve (AUC, 0.71 vs 0.83) and Dice Similarity coefficient (0.19 vs 0.53). When training with 100% of the data, our U-Net-based pretraining and detection achieved an AUC of 0.85 and Dice similarity coefficient of 0.57.

This paper proposes an intestinal region reconstruction method from CT volumes of ileus cases. Binarized intestine segmentation results often contain incorrect contacts or loops. We utilize the 3D U-Net to estimate the distance map, which is high only at the centerlines of the intestines, to obtain regions around the centerlines. Watershed algorithm is utilized with local maximums of the distance maps as seeds for obtaining “intestine segments”. Those intestine segments are connected as graphs, for removing incorrect contacts and loops and to extract “intestine paths”, which represent how intestines are running. Experimental results using 19 CT volumes showed that our proposed method properly estimated intestine paths. These results were intuitively visualized for understanding the shape of the intestines and finding obstructions.

Automation in the field of medical image segmentation is critical in helping the oncologists and surgeons for the accurate analysis of several pathological conditions by saving time. The ability to automatically segment the liver fast and accurately enables clinicians to understand the anatomical structure of the organ, and helps in the decision making process of diagnosis, surgery planning and as an anatomical map during surgical navigation especially important when using intraoperative image modalities . This work aims to develop an automatic liver parenchyma segmentation network which is based on U-Net architecture, a widely used architecture for medical image segmentation. This modified U-Net architecture includes reduced convolutional layers and using dropout layers as well as pre-processing the dataset to overcome the constraints of a small sample set. Reduced architecture complexity and introducing dropout regularization, addresses the problem of overfitting. We experimented with a callback for observing the training failure where it follows the early stopping policy and selecting the best model. Adding Gaussian noise to data can help the model to generalise well. For choosing the appropriate loss function we tested four different loss functions; Dice, binary cross entropy, Tversky and focal Tversky and concluded that Dice performs better. The network has been trained and validated using publicly available 3D-IRCADb dataset with images from 20 patients and achieved an overall Dice score of 94.5%. The overall objective of this work is to construct a network from a small sample set without the problem of overfitting or under-fitting, but delivering an acceptable Dice score.

Pulmonary embolus (PE) refers to obstruction of pulmonary arteries by blood clots. PE accounts for approximately 100,000 deaths per year in the United States alone. The clinical presentation of PE is often nonspecific, making the diagnosis challenging. Thus, rapid and accurate risk stratification is of paramount importance. High-risk PE is caused by right ventricular (RV) dysfunction from acute pressure overload, which in return can help identify which patients require more aggressive therapy. Reconstructed four-chamber views of the heart on chest CT can detect right ventricular enlargement. CT pulmonary angiography (CTPA) is the golden standard in the diagnostic workup of suspected PE. Therefore, it can link between diagnosis and risk stratification strategies. We developed a weakly supervised deep learning algorithm, with an emphasis on novel a attention mechanism, to automatically classify RV strain on CTPA. Our method is a 3D residual block-based model with integrated attention blocks. We evaluated our model on a dataset of CTPAs of emergency department (ED) PE patients. Our results show that attention consistently improves prediction. We infer that unmarked CTPAs can be used for effective RV strain classification. This could be used as a second reader, alerting for high-risk PE patients. To the best of our knowledge, there are no previous deep learning-based studies that attempted to solve this problem.

We developed a radiomic-based reinforcement learning (R-RL) model for the early diagnosis of lung cancer. We formulated the classification of malignant and benign lung nodules with multiple years of screening as a Markov decision process. The reinforcement learning method learned a policy mapping from the set of states (patients’ clinical conditions) of the environment (patients) to the set of possible actions (decisions). The customary mapping between the two sets was based on a value function with the expected reward designed to be associated with lung cancer risk which was increased when the patient was diagnosed with lung cancer and vice versa in the Markov chains. The trained model can be deployed to a single baseline CT scan for early diagnosis of malignant nodules. 215 NLST cases including 108 positive and 107 negative cases with 431 LDCT scans collected from 3 years of screening were used as the training set and another 70 cases with 35 positive and 35 negative cases were used as the independent test set. For each screen-detected nodule in a CT exam, forty-three texture features were extracted and used as the state in reinforcement learning. An offline model-free value iteration method was used to build the R-RL model. Our R-RL model trained with 3 years of serial CT exams achieved an AUC of 0.824 ± 0.003 when deployed to the first year CT exams of the test set. In comparison, the R-RL model trained with only the first year CT scans achieved a significantly (P<0.05) lower test AUC of 0.736 ± 0.004. Our study demonstrated that the R-RL model built with serial CT scans has the potential to improve early diagnosis of indeterminate lung nodules in screening programs, thus reducing follow-up exams or unnecessary biopsy and the associated costs.

In this work, we studied the catastrophic forgetting effect in transfer learning for the development of a deep learning model to perform COVID-19 pneumonia classification using chest x-ray radiographs. We demonstrated vendor differences using a large multi-vendor chest x-ray dataset for COVID-19 classification and investigated its impact on the generalizability of the model. We showed that the model trained from one particular vendor may not generalize well to other vendors. While both vendor-based fine-tuning and joint training can improve the performance on the new vendor test set, the first approach may cause performance degradation on the previous vendor test set.

Automated diagnostic assistants in healthcare necessitate accurate AI models that can be trained with limited labeled data, can cope with severe class imbalances and can support simultaneous prediction of multiple disease conditions. To this end, we present a deep learning framework that utilizes a number of key components to enable robust modeling in such challenging scenarios. Using an important use-case in chest X-ray classification, we provide several key insights on the effective use of data augmentation, self-training via distillation and confidence tempering for small data learning in medical imaging. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.

Chest x-ray (CXR) provides valuable diagnostic information during treatment monitoring of COVID-19 pneumonia. In this preliminary study, we show that deep learning-based imaging descriptors have the potential to quantitatively assess the severity of the disease. In the first stage, a deep convolutional neural network (DCNN), GoogLeNet, was trained to perform patch-level classification of non-COVID-19 pneumonia and normal image patches from the ChestX-ray14 data set. A total of 246,753 patches were used to train the DCNN in a four-fold cross-validation. The trained DCNN generates a pixel-wise pneumonia severity map when deployed to a CXR image. Global descriptors based on the intensity of the severity map were extracted and the classification accuracy was evaluated using a random forest classifier. In the second stage, the DCNN was deployed to 202 COVID-19 positive CXRs. Global descriptors were extracted and fine-tuned to generate severity measures for COVID-19 pneumonia. These image-level global descriptors were mapped to radiologist’s severity rating using logistic regression by 2-fold cross-validation. Classification accuracy was measured using the area under the receiver operating characteristic (ROC) curve (AUC). For classification of non-COVID-19 pneumonia from normal CXR, the patch-level AUC of the DCNN was 0.91±0.03 and the AUC of the image-level global descriptors was 0.93±0.04. The COVID-19 pneumonia regression model showed that the global descriptors had a correlation of 0.68 with the severity of the pneumonia in the CXR. Using radiologist’s rating of 0 as negative and higher ratings as positive for COVID-19 pneumonia, the scores from the regression model achieved an average AUC of 0.76 for classification in the validation sets.

The outbreak of the coronavirus and its rapid spread was recently acknowledged as a worldwide pandemic. Chest CT scans show high potential for detecting pathological manifestations. Hence, the demand for computer-aided tools to support radiologists has grown exponentially. In this work, we developed a deep learning based algorithm, with an emphasis on novel transfer learning methods, to segment COVID-19 opacity in chest CT scans. Our method focuses on creating a deep encoder for feature extraction by using a Fully Convolutional Network (FCN) architecture with one shared encoder and N task-related decoders, named HydraNet. The HydraNet architecture allowed the leverage of a large variety of medical datasets from different domains, in order to achieve better performances on a limited dataset. We achieved a dice score, sensitivity, and precision of 0.724, 0.75, and 0.807 respectively, on the test set, which is competitive with known state-of-the-art results.

We propose an approach for the automatic segmentation of mammary gland regions on 3D CT images, aiming to accomplish breast cancer risk assessment through CT scans acquired in clinical medicine for various diagnostic purposes. The proposed approach uses a hybrid method that embeds a deep-learning-based attention mechanism as a module into a conventional framework, which originally uses a probabilistic atlas to accomplish Bayesian inference to estimate the pixelwise probability of mammary gland regions on CT images. In this work, we replace both the construction and application of a probabilistic atlas, which is time-consuming and complicated to realize, by a visual explanation from the attention mechanism of a classifier learned through weak supervision. In the experiments, we applied the proposed approach to the segmentation of mammary gland regions based on 174 torso CT scans and evaluated its performance by comparing the segmentation results to human sketches on 14 CT cases. The experimental results showed that the attention maps of the classifier successfully focused on the mammary gland regions on the CT images and could replace the atlas for supporting mammary gland segmentation. The preliminary results on 14 test CT scans showed that the mammary gland regions were segmented successfully with a mean value of 50.6% on the Dice similarity coefficient against the human sketches. We confirmed that the proposed approach, combining deep learning and conventional methods, shows a higher computing efficiency, much better robustness, and easier implementation than our previous approach based on a probabilistic atlas.

One major challenge in the development of new wound healing technologies is the poor translation into the clinic. This primarily stems from differences in methodologies to assess wound healing leading to discrepancies in preclinical and clinical evaluations. The current standard of planimetric software has resulted in improved evaluation accuracy and precision in the clinical setting but has not yet been standardized for pre-clinical studies, resulting low replicability, high variability, and unknown relevance to clinical conditions. To meet these shortcomings, we seek develop a set of tools to automatically characterize wound in a variety of experimental conditions. Here, we present an application of UPSNet for the segmentation of wound images in an experimental group of pigs as a first step towards our overall goal. Our results show high overall accuracy (99.54 ± 0.28), intersection over union (93.32 ± 4.57), and boundary F1 scores (91.19 ± 8.59). Additionally, predicted wound areas differ little compared to ground truth areas (0.5308 ± 0.5501 cm²). An external dataset (n = 45) of wound images yielded high accuracy (98.64 ± 0.71), intersection over union (0.8544 ± 0.0893), and boundary F1 scores (0.6514 ± 0.1928). Predicted wound areas are comparable to ground truth areas (2.6935 ± 1.8838 cm²). With integration of additional experimental subjects, replicates, and experimental conditions, we will further develop and validate this and additional tools to automate the analysis of wound images to ultimately assess wound healing technologies.

The purpose of this study is to analyze dental panoramic radiographs for completing dental files to contribute to the diagnosis by dentists. As the initial stage, we detected each tooth and classified its tooth type. Since the final goal of this study includes multiple tasks, such as determination of dental conditions and recognition of lesions, we proposed a multitask training based on a Single Shot Multibox Detector (SSD) with a branch to predict the presence or absence of a tooth. The results showed that the proposed model improved the detection rate by 1.0%, the number of false positives per image by 0.03, and the detection rate by tooth type (total number of successfully detected and classified teeth/total number of teeth) by 1.6% compared with the original SSD, suggesting the effectiveness of the multi-task learning in dental panoramic radiographs. In addition, we integrated results of single-class detection without distinguishing the tooth type and 16-class (central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, third molar, distinguished by upper and lower jaws) detection for improving the detection rate and included post-processing for classification of teeth into 32 types and correction of tooth numbering. As a result, the detection rate of 98.8%, 0.33 false positives per image, and classification rate of 92.4% for 32 tooth types were archived.

Two major challenges hinder the deployment of deep learning-based systems into clinical practice: the need for numerous high-quality well-labeled data and the lack of explainability. Attention models, originated from natural language processing, have been popular to address the label scarcity problem and encourage model explainability. In this work, we developed a domain knowledge-guided attention model for disease diagnosis with only coarse scan-level labels and the population-level domain knowledge. The use of guided attention models encourages the deep learning-based diagnosis model to focus on the area of interests in an end-to-end manner. The research interest is to diagnose subjects with idiopathic pulmonary fibrosis (IPF) among subjects with interstitial lung disease (ILD) using an axial chest high resolution computed tomography (HRCT) scan. Our dataset contains 279 IPF patients and 423 non-IPF ILD patients. The network’s performance was evaluated by the area under the receiver operating characteristic curve (AUC). We observe that without attention-based loss function, the IPF diagnosis model reaches satisfactory performance (AUC=0.972), but lack explainability; when increasing the relative importance of attention-based loss, the IPF diagnosis model increases performance (AUC=0.988), along with the model explainability. Our contributions are (1) developing an IPF diagnosis model that only uses scan-level weak supervision; (2) incorporating population-level domain knowledge into the training of IPF diagnosis model in an end-to-end manner; (3) enhancing the explainability of deep learning systems by introducing attention mechanisms.

Subarachnoid Hemorrhage (SAH) detection is a critical, severe problem that confused clinical residents for a long time. With the rise of deep learning technologies, SAH detection made a significant breakthrough in recent ten years. Whereas, the performances are significantly degraded on imbalanced data, makes deep learning models have always suffered criticism. In this study, we present a DenseNet-LSTM network with Class-Balanced Loss and the transfer learning strategy to solve the SAH detection problem on an extremely imbalanced dataset. Compared to the previous works, the proposed framework not merely effectively integrate greyscale features the and spatial information from the consecutive CT scans, but also employ Class-Balanced loss and transfer learning to alleviate the adverse effects and broaden feature diversity respectively on an extreme SAH cases scarcity dataset, mimicking the actual situation of emergency departments. Comprehensive experiments are conducted on a dataset, consisted of 2,519 cases without hemorrhage cases and only 33 cases with SAH. Experimental results demonstrate the F-measure score of SAH detection achieved a remarkable improvement, the backbone DenseNet121 gained around 33% promotion after transfer learning, and on this basis, importing the Class-Balanced Loss and the LSTM structure, the F-measure score further increased 6.1% and 2.7% sequentially.

Prognosis plays a crucial role in the customization of lung cancer care. The effective prediction of treatment response is essential to tailor treatment decisions to lung cancer patients. Molecular characterization of tumors using genomics-based approaches is important for personalized treatment planning, however, repeated tumor biopsies should be performed to capture their molecular heterogeneity, putting patients at risk of procedural complications such as a pneumothorax. Furthermore, the recent addition of immunotherapy after chemoradiotherapy for patients with unresectable stage III NSCLC can improve survival outcomes. The survival benefit achieved by stage III NSCLC patients undergoing chemoradiation is of interest since currently available biomarkers are inadequate to predict which patients are most likely to benefit from immunotherapy for first-line treatment along with chemoradiation. In this study, we investigate the association between local failure-free survival and radiomic features extracted from CT scans of stage III NSCLC adenocarcinoma patients. We retrospectively analyzed a well-curated cohort of 89 non-contrast enhanced CT scans from patients receiving homogeneous chemoradiation treatment. A set of 107 radiomic features was extracted using the pyradiomics package. In univariate analysis we performed log-rank tests per feature to predict risk of local failure. In multivariate analysis we applied principal component analysis to fit a Cox model to predict local failure-free survival. Univariate analysis showed that no individual radiomic feature can predict local failure-free survival, while multivariate analysis gave a C-index = 0.70, 95% CI = [0.56,0.85]. We conclude that radiomic features from CT scans, can predict local failure-free survival in stage III NSCLC.

Acute ischemic stroke (AIS) is not only a common cause of disability but also a leading cause of mortality worldwide. Recent studies have shown that the collateral status could play a vital role in assessing AIS and determining the treatment options for the patients. Herein, we propose a joint regression and ordinal learning approach for AIS, built upon 3-D deep convolutional neural networks, that facilitates an automated and objective collateral imaging from dynamic susceptibility contrast-enhanced magnetic resonance perfusion (DSC-MRP). DSC-MRP images of 159 AIS subjects and 186 healthy subjects are employed to evaluate the proposed approach. The collateral status is manually assessed in arterial, capillary, early and late venous, and delay phases and served as the ground truth. The proposed method, on average, obtained 0.901 squared correlation coefficient, 0.063 mean absolute error, 0.945 Tanimoto measure, and 0.933 structural similarity index. The quantitative results between AIS and healthy subjects are comparable. Overall, the experimental results suggest that the proposed network could aid in automating the evaluation of collateral status and enhancing the quality and yield of diagnosis of AIS.

In this paper, we consider the problem of Coronavirus disease (COVID-19) diagnosis from chest X-ray images in a multi-label classification scenario, where the ultimate goal is to distinguish among Healthy, non-COVID Pneumonia, and COVID-19 infection cases from the chest X-ray manifestations. Particularly, we establish the use of a rapid, non-invasive and cost-effective X-ray-based method as a key diagnosis and screening tool for COVID-19 at early and intermediate stages of the disease. To this end, we propose CoroNet, a deep learning framework that is built upon a two-stage learning methodology: 1) an AutoEncoder to extract the infected regions in the chest X-ray manifestation of COVID-19 and other Pneumonia-like diseases and 2) a deep convolutional neural network for the multi-label classification. We utilize this tailored deep architecture to extract the relevant features specific to each class to perform the task of automatic diagnosis and classification. The unsupervised part of the proposed framework helps with proper identification of the disease given the scarcity of quality datasets on COVID-19, and at the same time, facilitates exploiting the large X-ray datasets that are readily available for Healthy and non-COVID Pneumonia cases. Our numerical investigations demonstrate that the proposed framework outperforms the state-of-the-art methods for COVID-19 identification while employing approximately ten times fewer training parameters as compared to other existing methodologies. Furthermore, we make use of attribution maps, an explainable artificial intelligence tool, to interpret the diagnosis offered by the network. We have made the codes of our proposed CoroNet framework publicly available to the research community.

With the advances in deep learning analysis a question arises – if for a robust network learning cycle, the shift between raw data and image-based data, is of importance. In the current work, we start exploring this topic by focusing on the following case study: categorizing cancer cells from holographic images. The problem in the categorization process is the time consumption of the transformation from off-axis image holograms to OPD maps. While there have been attempts for fast transformation, they still take non-negligible time. We propose a novel approach for fast classification that skips the pre-processing of creating OPD maps and directly uses the raw data of holographic images. Our dataset contains two separate image acquisitions of primary cancer cells (SW480) and metastatic cancer cells (SW620) of colorectal adenocarcinoma imaged during flow. We extracted the OPD maps of those cells and used them to train and evaluate a ResNet model to create baseline results. Our convolutional neural network (CNN) model is based on Y-Net approach: during training synthetic OPD images are created from input holograms, using the real OPD maps while simultaneously classifying the cells. During inference time only the classification branch operates to further reduce running time. This approach saves computational time by over 90%.

Purpose: To determine the feasibility of using deep learning algorithms that can identify and classify types of cranial malformations, i.e., craniosynostosis and deformational plagiocephaly and brachycephaly (DPB), using top view photographs of the infant head. Method: We used 72 3D head volumes of infants with normal (13), DPB (34), and craniosynostosis (25). These volumes contain only information about the head shape and lack texture. From these 3D volumes, top-view 2D renderings were generated from different viewing angles. We generated 37 2D files were generated from each 3D volume, and we applied additional data augmentation to obtain a total of 5,254 2D images. We then used this dataset to investigate the performance of a well-known deep learning architectures for image classification, i.e., LeNet. The data were divided into training and test sets (85% and 15%, respectively with minimum one data of each class in the test set). We performed model evaluation by cross-validation. Results: The overall accuracy of the cranial shape analysis model was 87.5 ± 5.59%. Cases with craniosynostosis were identified with 0.99±0.01 accuracy, while subjects with DPB were identified with 0.78 ± 0.1. The accuracy of the model to identify normal cases without cranial deformation was 0.96 ± 0.04. Conclusions: Deep learning-based methods can be used for the accurate detection, and classification of different types of conditions with head malformation using 2D photographic data. These algorithms will be packaged as a mobile health solution to make a decision support tool available at the point-of-care.

Developing radiomic based machine learning models has drawn considerable attention in recent years. However, identifying a small and optimal feature vector to build a robust machine learning models has always been a controversial issue. In this study, we investigated the feasibility of applying a random projection algorithm to create an optimal feature vector from the CAD-generated large feature pool and improve the performance of the machine learning model. We assemble a retrospective dataset involving abdominal computed tomography (CT) images acquired from 188 patients diagnosed with gastric cancer. Among them, 141 cases have peritoneal metastasis (PM), while 47 cases do not have PM. A computer-aided detection (CAD) scheme is applied to segment the gastric tumor area and computes 325 image features. Then, two Logistic Regression models embedded with two different feature dimensionality reduction methods, namely, the principal component analysis (PCA) and a random projection algorithm (RPA). Afterward, a synthetic minority oversampling technique (SMOTE) is used to balance the dataset. The proposed ML model is built to predict the risk of the patients having advanced gastric cancer (AGC). All Logistic Regression models are trained and tested using a leave-one-case-out cross-validation method. Results show that the logistic regression embedded with RPA yielded a significantly higher AUC value (0.69±0.025) than using PCA (0.62±0.014) (p<0.05). The study demonstrated that CT images of the gastric tumors contain discriminatory information to predict the risk of PM in AGC patients, and RPA is a promising method to generate optimal feature vector, improving the performance of ML models of medical images.

Accurate characterization of suspicious microcalcifications is critical to determine whether these calcifications are associated with invasive disease. Our overarching objective is to enable the joint characterization of microcalcifications and surrounding breast tissue using mammography images and digital histopathology images. Towards this goal, we investigate a template matching-based approach that utilizes microcalcifications as landmarks to match radiographs taken of biopsy core specimens to groups of calcifications that are visible on mammography. Our approach achieved a high negative predictive value (0.98) but modest precision (0.66) and recall (0.58) in identifying the mammographic region where microcalcifications were taken during a core needle biopsy.

Papillary thyroid carcinoma (PTC) is primarily treated by surgical resection. During surgery, surgeons often need intraoperative frozen analysis and pathologic consultation in order to detect PTC. In some cases pathologists cannot determine if the tumor is aggressive until the operation has been completed. In this work, we have taken tumor classification a step further by determining the tumor aggressiveness of fresh surgical specimens. We employed hyperspectral imaging (HSI) in combination with multiparametric radiomic features to complete this task. The study cohort includes 72 ex-vivo tissue specimens from 44 patients with pathology-confirmed PTC. A total of 67 features were extracted from this data. Using machine learning classification methods, we were able to achieve an AUC of 0.85. Our study shows that hyperspectral imaging and multiparametric radiomic features could aid in the pathological detection of tumor aggressiveness using fresh surgical spemens obtained during surgery.

Medical images of a patient may have a significantly different appearance depending on imaging modality (e.g. MRI vs. CT), sequence type (e.g., T1-weighted MRI vs. T2-weighted MRI), and even manufacturer/model of equipment used for the same modality and sequence type (e.g. SIEMENS vs GE). Since in the context of deep learning training and test data often come from different institutions, it is important to determine how well neural networks generalize when image appearance varies. There is currently no systematic answer to this question. In this study, we investigate how deep neural networks trained for segmentation generalize. Our analysis is based on synthesizing a series of datasets of images with the target object of the same shape but with varying pixel intensity of the foreground object and the background. This simulates basic effects of changing equipment models and sequence types. We also consider scenarios when datasets with different image properties are combined to determine whether generalizability of the network to other scenarios is improved. We found that the generalizability of segmentation networks to changing intensities is poor. We also found that the generalizability is somewhat improved when different datasets are combined but that generalizability is typically limited to data similar to the two types of datasets included in training and not to datasets with different image intensities.

This paper presents a method for extracting the lung and lesion regions from COVID-19 CT volumes using 3D fully convolutional networks. Due to the pandemic of coronavirus disease 2019 (COVID-19), computer aided diagnosis (CAD) system for COVID-19 using CT volume is required. In the development of CAD system, it is important to extract patient anatomical structures in CT volume. Therefore, we develop a method for extracting the lung and lesion regions from COVID-19 CT volumes for the CAD system of COVID-19. We use 3D U-Net type fully convolutional network (FCN) for extraction of the lung and lesion regions. We also use transfer learning to train the 3D U-Net type FCN using the limited data of COVID-19 CT volume. As pre-training, the proposed method trains the 3D U-Net model using abdominal multi-organ regions segmentation dataset which contains a large number of annotated CT volumes. After pre-training, we train the 3D U-Net model from the pre-trained model using a small number of annotated COVID-19 CT volumes. The experimental results showed that the proposed method could extract the lung and lesion regions from COVID-19 CT volumes.

Cardiomegaly is a medical condition that leads to an increase in cardiac size. It can be manually assessed using the cardiothoracic ratio from chest x-rays (CXRs). However, as that task can be challenging in such limited examinations, we propose the fully automated cardiomegaly detection in CXR. For this, we first trained convolutional networks (ConvNets) to classify the CXR as positive or negative to cardiomegaly and then evaluated the generalization potential of the trained ConvNets on independent cohorts. This work used frontal CXR images from a public dataset for training/testing and another public and one private dataset to test the models’ generalization externally. Training and testing were performed using images cropped with a previously developed U-Net model. Experiments were performed with five topologically different ConvNets, data augmentation techniques, and a 50-50 class-weighing strategy to improve performance and reduce the possibility of bias to the majority class. The receiver operating characteristic curve assessed the performance of the models. DenseNet yielded the highest area under the curve (AUC) on testing (0.818) and external validation (0.809) datasets. Moreover, DenseNet obtained the highest sensitivity overall, yielding up to 0.971 on the private dataset with patients from our hospital. Therefore, DenseNet had a statistically higher potential to identify cardiomegaly. The proposed models, especially those trained with DenseNet convolutional core, automatically detected cardiomegaly with high sensitivity. To the best of our knowledge, this was the first work to design a novel general model for classifying specific deep-learning patterns of cardiomegaly in CXRs.

The status of the donor tissue post-keratoplasty (post-transplant), whether full or partial thickness, is currently assessed for health, function, and complications via clinical evaluations. This includes detection of visible signs of graft rejection on slit lamp biomicroscopy such as keratic precipitates or edema. Corneal endothelial cell (EC) images are utilized to indirectly assess the health of the cornea post keratoplasty with evidence that morphometric changes may occur prior to clinical signs of rejection. We extracted over 190 novel quantitative features from EC images acquired 1-12 months prior to patients´ rejection diagnosis date, and used random forest (RF) classifiers to predict future rejection. We automatically segmented the cell borders of 171 EC images using a semi-automated segmentation approach: deep learning U-Net segmentation followed by guided manual correction. Following segmentation, we extracted novel quantitative features that robustly represented the cellular morphology from the EC images. We trained and tested a RF classifier using 5-fold cross validation and minimal Redundancy Maximal Relevance (mRMR) feature selection. From the 5-fold cross validation, we report an area under the receiver operating characteristic curve (AUC) of 0.87 ± 0.03, a sensitivity of 0.86 ± 0.12, and a specificity of 0.86 ± 0.10. The results suggest we can accurately predict a patient’s future graft rejection 1- 12 months prior to diagnosis, enabling clinicians to intervene modifying and/or instituting topical corticosteroid therapy earlier with the possibility of lowering graft rejection failures. Success of this classifier could reduce health care costs, patient discomfort, vision loss and the need for repeat keratoplasty.

Inflammatory white matter brain lesions are a key pathological finding in patients suffering from multiple sclerosis (MS). Image based quantification of different characteristics of these lesions has become an elemental bio-marker in both diagnosis as well as therapy monitoring during treatment of these patients. Whilst it has been shown that the lesion load at a single point in time is only of limited value with respect to explaining clinical symptoms of the patients, a more robust estimate of disease activity can be observed by analyzing the evolution of lesions over time. Here, we propose a system for automated monitoring of temporal lesion evolution in MS. We describe an approach for analysis of lesion correspondence, along with a pipeline for fully automated computation of this model. The pipeline consists of a U-Net based lesion segmentation, a non-linear image registration between multiple studies, computation of temporal lesion correspondences, and finally an analysis module for extracting and visualizing quantitative parameters from the model.

In recent years many computer-aided diagnosis systems (CAD) using deep learning (DL) were developed for thyroid classification. However, most DL approaches have flawed clinical interpretation and often need a large amount of supervised data to ensure performance. For medical images, the costs of obtaining labeled data are relatively high, making the problem of few-shot learning (FSL) more common. We proposed a multi-task learning network for thyroid malignant probability prediction using perceptive interpretable features to overcome these limitations. With IRB approval, 1588 cases were collected with perceptive features diagnosed by experienced radiologists. The hard parameter sharing network was trained using perceptive features and pathological results as ground truth. Prior knowledge was embedded into the network by multi-task learning of perceptive features, which is how radiologists diagnose from ultrasound images. We trained the models using the 1345 cases and tested them with 243 cases. It was found that the improvement of the classifier with the proposed network (AUC of 0.879) to the baseline CNN (AUC of 0.779) was statistically significant (p <0.001).

Retinal vessels segmentation algorithms have a great significance for diagnosis of various blood-related diseases such as diabetes, high blood pressure, etc. In addition, every one of us has a different retinal vascular tree so it can be also used as a bio-metric identification. This paper describes our work to take up a challenge∗ to build the best segmentation model of blood vessels out of retinal images. One of the issues in deep learning in the medical domain is the lack of sufficient labeled data, and the DRIVE dataset given in the challenge is no exception. Therefore, in this work we propose a method to improve the performance of the state-of-the-art retinal images segmentation model by synthesizing new retinal images using StyleGAN and their corresponding segmentation maps created by a segmentation network. We show that training with additional generated images improves the segmentation performance.

In analysis and diagnosis of torso diseased organs, bone elimination from contrast enhanced CT datasets is of high importance. Yet, bone elimination from contrast enhanced CT datasets is obtained using two phases; native and desired contrast enhanced phases. This elimination is performed by segmenting bones from native phase, registering the contrast enhanced phase of the dataset with its native phase, then subtracting overlapped voxels of segmented bones on the contrast enhanced dataset. This technique for subtraction is time consuming and error prone depending on registration results. In this paper, we propose a fully automatic method to segment and classify bone though any single contrast enhanced torso CT images as well as native phase. This bone segmentation is achieved in four steps: coarse bone candidates are segmented by automatically adapted threshold, cortical bone is classified using shape and intensity, spongy bone is classified by solid angle, abdominal bone and pelvis are separated based on smoothing histogram, and. The method was examined using 64 datasets in different phases of contrast enhanced CT dataset. Bone is segmented and classified in a total 45360 slices. For evaluation, results of segmenting bones through arterial, portal and equilibrium phases were compared to results of native phase 94%, 98%, 97% respectively. These results demonstrate that the method is promising in segmenting torso bone from any single phase of CT and may have utility in clinical use. The method is capable of segmenting bone through thoracic and abdominal images separately, and to classify torso bone into ribs, spine and pelvis.

Computer-aided detection (CAD) approaches have shown promising results for early esophageal cancer detection using Volumetric Laser Endoscopy (VLE) imagery. However, the relatively slow and computationally costly tissue segmentation employed in these approaches hamper their clinical applicability. In this paper, we propose to reframe the 2D tissue segmentation problem into a 1D tissue boundary detection problem. Instead of using an encoder-decoder architecture, we propose to follow the tissue boundary using a Recurrent Neural Network (RNN), exploiting the spatio-temporal relations within VLE frames. We demonstrate a near state-of-the-art performance using 18 times less floating point operations, enabling real-time execution in clinical practice.

Nasal septal deviations are a well-known and widespread problem. According to the American Academy of Otolaryngology 80% of the population have a nasal septal deviation. Its level of severity can range from the person not being aware of it to respiratory obstruction and choking. It is therefore necessary to distinguish those patients at risk. For a proper diagnosis, the amount and location of the deviation have to be considered, but also the shape and changes in the surrounding turbinates. The segmentation of the structures of interest is an important step to reduce subjectivity in the diagnosis. Unfortunately, due to their variable and tortuous shape manual segmentation is time consuming. In this paper, the first method for the automatic segmentation of the structures in the nasal cavity and ethmoidal sinus is presented. A coupled shape model of the nasal cavity and paranasal sinus regions is trained and used to detect the corresponding regions in new CT images. The nasal septum is then segmented using a novel slice-based propagation technique. This segmentation allows the additional separation and segmentation of the left and right nasal cavities and ethmoidal sinuses and their structures by means of an adaptive thresholding with varying boundary sizes. The method has been evaluated in 10 CT images obtaining promising results for the nasal septum (DICE: 87.71%) and for the remaining structures (DICE: 72.01% - 73.01%). Based on the resulting segmentations, a web-based diagnosis tool has been designed to quantify the septal deviation using three metrics proposed by clinical experts.

Architectural distortion (AD) is one of the breast abnormal signs in medical imaging and it is hard to be detected in clinic because of its subtle appearance and similar intensity with surrounding tissues. We previously developed a deep-learning-based model for AD detection in digital breast tomosynthesis (DBT). However, for atypical ADs, the model’s detection performance was not good enough because atypical ADs do not have a radial pattern, which is the main characteristic of AD. Considering that radiologists always take surrounding tissues’ information as reference to locate atypical ADs, an ideal model should not only adapt to the different shape of atypical ADs, but also have a large receptive field. In this study, deformable convolution kernel was employed to establish a novel deep-learning-based AD detection model. A dataset of 265 DBT volumes including 64 typical ADs, 74 atypical ADs and 127 normal volumes were collected for model evaluation. Mean true positive fraction (MTPF) was used as figure-of-merit. The results of six-fold cross-validation showed that after involving deformable convolution, the MTPF improved from 0.53±0.04 to 0.56±0.04 (p=0.028) and the number of false positives (FPs) at 80% sensitivity reduced from 1.95 to 1.09. Especially for atypical AD, the MTPF improved from 0.45±0.05 to 0.51±0.04 (p=0.01) and the number of FPs at 80% sensitivity reduced from 4.79 to 1.51. These results showed that this model has potential to assist radiologists locate more suspicious ADs and improve their diagnosis efficiency.

Segmenting small brain tumors (diameter ≤ 0.5 cm) on contrast enhanced MRI images presents a particular problem, as enhancing blood vessels of similar size can be detected as false positives. The capabilities of Liquid State Machines (LSM) ensembles to separate high dimensional data are used in this project to overcome this problem. Contrast enhanced MRI images were first transformed into time series before being fed into the LSM, which consists of a 3 dimensional array of spiking neurons, the resulting activation patterns of both the excitatory and inhibitory neurons differed from each other to a high enough degree that enhancing tumors and blood vessel of similar size could be distinguished from one another. An ensemble of two LSM’s, which differed in the way the time series information was input was used to enhance data separation. The combined output of the LSM ensemble was then used as input into a random forest to classify the final result as tumor vs. non-tumor. In comparison with deep learning CNN our results show excellent small tumor recognition and generate probability maps that cover the tumors but ignore blood vessels and other contrast-enhancing objects.

Segmentation of renal parenchyma responsible for renal function is necessary to evaluate contralateral renal hypertrophy and to predict renal function after renal partial nephrectomy (RPN). Although most studies have segmented the kidney on CT images to analyze renal function, renal function analysis is required without radiation exposure by segmenting the renal parenchyma on MR images. However, renal parenchyma segmentation is difficult due to small area in the abdomen, blurred boundary, large variations in the shape of kidney among patients, similar intensities with nearby organs such as the liver, spleen and vessels. Furthermore, signal intensity is different for each data due to a lot of movement when taking abdominal MR even when photographed with the same device. Therefore, we propose cascaded deep convolutional neural network for renal parenchyma segmentation with signal intensity correction in abdominal MR images. First, intensity normalization is performed in the whole MR image. Second, kidney is localized using 2D segmentation networks based on attention UNet on the axial, coronal, sagittal planes and combining through a majority voting. Third, signal intensity correction between each data is performed in the localized kidney area. Finally, renal parenchyma is segmented using 3D segmentation network based on UNet++. The average DSC of renal parenchyma was 91.57%. Our method can be used to assess contralateral renal hypertrophy and to predict renal function by measuring volume change of the renal parenchyma on MR images without radiation exposure instead of CT images, and can establish basis for treatment after RPN.

Deep learning technologies have been successfully applied to automatic diagnostics of ex-vivo lung cancer with fluorescence lifetime imaging endomicroscopy (FLIM). Recent advance in convolutional neural networks (CNNs) by splitting input features for multi-scale feature extraction as a feature-level aggregation, has achieved further improvement in visual recognition. However, due to the splitting, correlations among input features are no longer retained. To exploit the advantages of hierarchical multi-scale architectures, while maintaining the correlations as global information, we propose a novel architecture, namely multi-scale concatenated-dilation (MSCD) at a layer level. The MSCD performs multi-scale feature extraction on input features without the splitting. In addition, we substitute the Addition aggregation in the original hierarchical architecture with the Concatenation to retrieve more features. At the same time, we also introduce dilated convolutions to replace the linear convolutions to further enlarge the receptive field. We evaluate the performance of MSCD by integrating it into ResNet, on over 60,000 FLIM images collected from 14 patients, using a custom fiber-based FLIM system, with various user-specified configurations. Accuracy, precision, recall, and the area under the receiver operating characteristic curve are used as the metrics. We first demonstrate the superiority of our MSCD model over the backbone ResNet and other state-of-the-art CNNs in terms of higher scores with lower complexity over the metrics. Moreover, we empirically demonstrate the superiority of the Concatenation aggregation over the Addition on convolution and scale efficiency. Furthermore, we compare the MSCD with Res2Net to illustrate the advantages and disadvantages of feature-/layer-level multi-scale aggregation.

A Deep-Learning (DL) based segmentation tool was applied to a new magnetic resonance imaging dataset of pregnant women with suspected Placenta Accreta Spectrum (PAS). Radiomic features from DL segmentation were compared to those from expert manual segmentation via intraclass correlation coefficients (ICC) to assess reproducibility. An additional imaging marker quantifying the placental location within the uterus (PLU) was included. Features with an ICC < 0.7 were used to build logistic regression models to predict hysterectomy. Of 2059 features, 781 (37.9%) had ICC <0.7. AUC was 0.69 (95% CI 0.63-0.74) for manually segmented data and 0.78 (95% CI 0.73-0.83) for DL segmented data.

In this research, our goal is to implement a Convolutional Neural Network (CNN) to segment Focal Cortical Dysplasia (FCD). FCD is a common lesion responsible for paediatric medically intractable focal epilepsy. MRI features of FCD can be subtle and may be missed by a radiologist. Recent advances in deep learning techniques in different fields have motivated us to develop a deep learning-based model to detect and segment the lesion responsible for FCD. We proposed a Fully Convolutional Network (FCN) for the task of FCD detection and localization. Our proposed model has four blocks of two convolutional layers followed by a pooling layer, as feature extraction part. Then, we have added three up-sampling blocks which include one convolutional layer and one up-sampling layer. The convolutional layers’ kernels are 3×3 and we are utilizing 4x and 2x upsampling layers in the decoder part. We are using skip layers as well, to get a more refined up-sampled output. We are adding the respective down-sampled feature map from the encoder part. To train and evaluate the model leaveone-out technique has been utilized where one test subject is left out of training in each experiment. We have identified 13 out 13 healthy subjects as healthy. The model has identified the lesion in 15 out of 17 MR-positive FCD subjects with 73 percent lesion coverage. For MR-negative cases, 11 out of 13 subjects were identified with lesion coverage of 64 percent. Based on our experiments, FCN holds the potential to assist specialists in detecting and localizing the lesion.

The use of low-dose Computed Tomography (CT) has been effective in reducing the mortality rate due to lung cancer. With the rapid increase in the number of studies, computer aided diagnosis (CAD) systems need to be developed to further assist radiologists in detecting lung nodules and determining their malignancy in CT scans. An important factor for determining malignancy in a sequential of scans is the rate of change and growth. In the past, deformable registration techniques have been widely applied to assess longitudinal change in data. In this paper we propose a new deep learning system based on Convolutional Neural Networks and U-Net architecture to assess longitudinal change in CT scans for a nodule over time. The VoxelMorph network [1] has been effective in deformable registration of brain MRIs. This paper extends the application of VoxelMorph to registration of lung nodules in multiple time point scans based on a subset of data from the National Lung Screening Trial (NLST), referred to as NLSTx. VoxelMorph has been modified to include both a sum-of-squared-differences (SSD) image similarity loss and spatial domain regularization to reduce the number of fold-overs and non-invertible transformations, indicated by the presence of negative Jacobian determinants. The network effectively performs 3D Deformable Registration of longitudinal nodule CT scans, providing the means to quantify nodule growth over time.

Coronary computed tomography angiography (coronary CTA) is a robust and well-established non-invasive diagnostic tool to detect and assess coronary artery disease (CAD). The accurate detection, quantification and characterization of the coronary plaque burden has become an important part of this imaging modality. The quality and performance of modern machine-learning-based data-driven learning approaches is often impacted by either insufficient or inconsistently-labeled training data and is further subject to additional bias from human annotators. To address these shortcomings for coronary plaque characterization, we have developed a synthetic lesion generating framework for CTA applications, which can produce accurate and high-quality labeled training data for data-driven learning approaches. This approach can help to ease the manual annotation burden, which is often the limiting factor in data-driven learning algorithms and instead provides reliable ground truth data for modern deep learning approaches. Furthermore, this framework can easily be used to create custom tailored training data that can be used for pre- or post-training steps of already existing machine learning approaches for CTA applications. We tested this data generation framework by inserting synthetic lesions in 11 clinical CTA scans of healthy patients resulting in a data set of ~7000 annotated 2D slices. With this data we performed several plaque detection experiments using a data-driven machine learning approach with a neural encoder architecture. In this plaque classification task we first demonstrate that the synthetic lesion generation module can consistently perform well in recognizing unseen synthetic test data with an overall classification accuracy of 93%. Next we apply the synthetic lesion framework in a transfer learning experiment, where we demonstrate the feasibility to learn to classify real clinical plaque lesions with a purely synthetic model (overall classification accuracy 84%) that never saw real clinical lesions during model training. Second, we show that using synthetically data for pre-training with a subsequent training on clinical data can enhance the overall classification accuracy (from 91% to 92%) while strongly increasing the true positive count. We conclude that the synthetic plaque lesions model faithfully covers many important image characteristics of real plaque lesions in coronary CTA imaging and can thus help reduce the annotation burden for data-driven predictive vascular systems in this domain. This allows the creation of exhaustively annotated and site-specific customizable training data with a computationally fast forward model.

Objectives: Bone segmentation can help bone disease diagnosis or post treatment assessment but manual segmentation is a time consuming and tedious task in clinical practice. In this work, three automatic methods to segment bone structures on whole body CT images were compared. Methods: A threshold-based approach with morphological operations and two deep learning methods using a 3D U-Net with different losses, one with a cross entropy/Dice loss and the second with a Hausdorff Distance/Dice loss, were developed. Ground truth bone segmentations were generated by manually correcting the results obtained with the threshold based method. The automatic bone segmentations were evaluated using a Dice score and Hausdorff distance. Visual evaluation was also performed by a medical expert. Results: Dice scores of 0.953, 0.986 and 0.978 were achieved for the Threshold-based method and the two deep learning methods, respectively. Visual evaluation showed that the deep learning method with a Hausdorff Distance/Dice loss performed the best.

In individualized screening mammography, a breast density is important to predict potential risks of breast cancer incidence and missing lesions in mammographic diagnosis. Segmentation of the mammary gland region is required when focusing on missing lesions. A deep-learning method was recently developed to segment the mammary gland region. A large amount of ground truth (prepared by mammary experts) is required for highly accurate deep-learning practice; however, this work is time- and labor-intensive. To streamline the ground truth in deep learning, we investigated a difference in acquired mammary gland regions among multiple radiological technologists having various experience and reading levels, who shared the criteria on segmentation. If we can ignore a skill level for image reading, we can increase a number of training images. Three certified radiological technologists segmented the mammary gland region in 195 mammograms. The degree of coincidence among them was assessed with respect to seven factors which indicated the feature of segmented regions including the breast density and mean glandular dose, using Student’s t-test and Bland-Altman analysis. The assessments made by the three radiological technologists were consistent considering all factors, except the mean pixel value. Thus, we concluded that the ground truths prepared by multiple practitioners with different experiences can be accepted for the segmentation of the mammary gland region and they are applicable for training images if they stringently share the criteria on the segmentation.

This study aims at developing a radiogenomic model to identify high-risk non-small cell lung cancer (NSCLC) patients and predict overall survival. Baseline CT images of 85 NSCLC patients (male/female: 58/27, event: death, adenocarcinoma/squamous cell carcinoma/unspecified: 41/32/12, in stages I/II/III/unspecified: 39/25/12/9) with gene expression profile (microarray data) of 33 genes were used from the NSCLC-Radiomics Genomics dataset, publicly available from the National Cancer Institute’s Cancer Imaging Archive (TCIA). The 33 genes were selected on the basis that they represent three major co-expression patterns in the dataset- histology, neuroendocrine (NE), and pulmonary surfactant systems (PSS) signature genes. Radiomic features (429) characterizing the primary tumor were extracted from the 3D tumor volume using PyRadiomics. As the first step to our analysis, we used the Mann-Whitney U test to identify a sub-set of 224 features that were robust (p-value < 0.05) to scan differences in slice thickness, reconstruction kernel, and contrast enhancement. Principal Component Analysis was used to extract ten principal components, capturing 85% of the variance, from the radiomic and genomic features, respectively. The following three models were created, to assess the prognostic performance of the radiomic and genomic PCs- Model 1: consists of the first five radiomic PCs; Model 2: consists of the first five genomic PCs; and Model 3: consists of the first three radiomic PCs and the first two genomic PCs. For these models, a five-fold crossvalidated multivariate Cox proportional hazards model (200 iterations) was used to compute the concordance index which measures the ability of the models to predict overall survival. The concordance index values for the Cox model are (mean (min, max)): Model 1: 0.59(0.57,0.62), Model 2: 0.57(0.54,0.61), Model 3: 0.62(0.61,0.64). In addition to the cross-validated c statistics, we also built a model on the complete dataset, for each of the three sets of predictors, to evaluate Kaplan Meier's performance in separating participants above versus below the median prognostic score. We constructed separate radiogenomic signatures for the NSCLC patients based on their histology- LUSC and LUAD and conducted similar survival analysis on them, showing that the combined radiogenomic model was the only one with statistically significant curve separation (p<0.05). This preliminary study suggests that a combination of radiomics and genomics information could give a more comprehensive assessment of the tumor’s characteristics at baseline and generate a better prognostic model in comparison to models formed using radiomics or genomics information alone.

This paper proposes an automated segmentation method of infection and normal regions in the lung from CT volumes of COVID-19 patients. From December 2019, novel coronavirus disease 2019 (COVID-19) spreads over the world and giving significant impacts to our economic activities and daily lives. To diagnose the large number of infected patients, diagnosis assistance by computers is needed. Chest CT is effective for diagnosis of viral pneumonia including COVID-19. A quantitative analysis method of condition of the lung from CT volumes by computers is required for diagnosis assistance of COVID-19. This paper proposes an automated segmentation method of infection and normal regions in the lung from CT volumes using a COVID-19 segmentation fully convolutional network (FCN). In diagnosis of lung diseases including COVID-19, analysis of conditions of normal and infection regions in the lung is important. Our method recognizes and segments lung normal and infection regions in CT volumes. To segment infection regions that have various shapes and sizes, we introduced dense pooling connections and dilated convolutions in our FCN. We applied the proposed method to CT volumes of COVID-19 cases. From mild to severe cases of COVID-19, the proposed method correctly segmented normal and infection regions in the lung. Dice scores of normal and infection regions were 0.911 and 0.753, respectively.

Repeat liver resection or transarterial chemoembolization (TACE) can be used for disease progression (PD) of hepatocellular carcinoma (HCC), but when patients developed extrahepatic metastasis or macrovascular invasion which was aggressive disease progression (aggressive-PD), the treatments became a challenge. Therefore, it was meaningful to predict aggressive-PD as early as possible considering the current prediction method in clinical was unreliable. In this study, a deep learning model was conducted to predict aggressive-PD. 333 patients receiving hepatectomy or TACE were enrolled from five hospitals. For each patient, deep learning score was calculated from a convolutional neural network model constructed based on resnet block. The model showed excellent performance for individualized, non-invasive prediction of the progression of Hepatocellular carcinoma (training set: ACC=75.61%, AUC=0.81, validation set: ACC=87.36%, AUC=0.82). Pearson correlation analysis showed albumin concentration were significantly correlated with deep learning score.

Brain computed tomography (CT) images have been routinely used by neuroradiologists in diagnosis of aneurysmal subarachnoid hemorrhage (aSAH). The purpose of this study is to develop a computer-aided detection (CAD) scheme to generate quantitative image markers computed from CT images to predict various clinical measures after aSAH. A CT image dataset involving 59 aSAH patients was retrospectively collected and used. For each patient, non-contrast CT acquired during admission into hospital is used for this study. From each CT image set, CAD scheme segments intracranial brain region, and labels each CT voxel into one of four regions namely, cerebrospinal fluid, white matter, grey matter, and leaked blood. For image slices above the level of the lateral ventricles, cerebrospinal fluid regions are also defined as sulci regions. Nest, CAD scheme computes 9 image features related to the volumes of the segmented sulci, blood, white and gray matter, as well as their ratios. We then built machine learning (ML) models by fusion of these features to predict 5 clinical measures including Delayed Cerebral Ischemia, Clinical Vasospasm, Ventriculoperitoneal Shunting, Modified Rankin Scale and Montreal Cognitive Assessment to assess prognosis of aSAH patients. Based on a leave-one-case-out cross-validation method, ML models yield performance of predicting the 5 selected clinical measures with the areas under ROC curves (AUC) ranging from 0.658 to 0.825. Study results demonstrate the promising feasibility of applying CAD-based image processing and machine learning method to generate valuably quantitative image markers and potential to assist clinicians optimally diagnosing and treating aSAH patients.

In recent years, the assessment of non-cancerous diseases on low-dose CT scans for lung cancer screening has gained significant attention. Osteoporosis shares many risk factors with lung cancer and the thoracic and upper lumbar vertebrae can be visualized within the screening scan range, making diagnosis of osteoporosis viable. However, manual assessments can be time-consuming and inconsistent. This study investigates the application of radiomic texture analysis (RTA) for the automatic detection of osteoporosis. In this retrospective analysis of 613 CT screening scans acquired from the I-ELCAP database, quantitative features, including those based on intensity, texture, and frequency, were extracted from ROIs manually placed within the central body of the T6 and L1 vertebrae on axial images. The top 4 individually performing features were selected to train an SVM classifier for the classification between osteoporotic, abnormal, and normal vertebrae. Performance was evaluated through ROC analysis, with areas under the ROC curve of 0.925 +/- 0.054 for the T6 vertebra and 0.847 +/- 0.092 for L1. Further, RTA was compared to a radiologist’s visual diagnosis and a previously published automatic bone mineral density calculation approach. The RTA technique correlated well with the automatic BMD calculation, with Pearson linear correlation coefficients of - 0.752 and -0.653 for the T6 and L1 vertebrae, respectively, and qualitative comparison to the visual assessment was favorable. Based on the ROC results and the correlation with previously established methods, RTA demonstrated significant potential in quantifying vertebral bodies in axial CT screening scans and characterizing the vertebral disease state.

Colorectal polyps are critical indicators of colorectal cancer (CRC). Classification of polyps during colonoscopy is still a challenge for which many medical experts have come up with visual models, albeit with limited success. An early detection of CRC prevents further complications in the colon, which makes identification of abnormal tissue a crucial step during routinary colonoscopy. In this paper, a classification approach is proposed to differentiate between benign and pre-malignant polyps using features learned from a Triplet Network architecture. The study includes a total of 154 patients, with 203 different polyps. For each polyp an image is acquired with White Light (WL), and additionally with two recent endoscopic modalities:Blue Laser Imaging (BLI) and Linked Color Imaging (LCI). The network is trained with the associated triplet loss, allowing the learning of non-linear features, which prove to be a highly discriminative embedding, leading to excellent results with simple linear classifiers. Additionally, the acquisition of multiple polyps with WL, BLI and LCI, enables the combination of the posterior probabilities, yielding a more robust classification result. Threefold cross-validation is employed as validation method and accuracy, sensitivity, specificity and area under the curve (AUC) are computed as evaluation metrics. While our approach achieves a similar classification performance compared to state-of-the-art methods, it has a much lower inference time (from hours to seconds, on a single GPU). The increased robustness and much faster execution facilitates future advances towards patient safety and may avoid time-consuming and costly histhological assessment.

We developed an image-based unsupervised survival prediction model, called pix2surv, based on a conditional generative adversarial network (cGAN), and evaluated its performance based on chest CT images of patients with the coronavirus disease 2019 (COVID-19). The architecture of the pix2surv model includes a time generator that consists of an encoding convolutional network and a fully connected prediction network, and a discriminator network. The time generator is trained to generate survival-time images from chest CT images of each patient. The discriminator is a patch-based convolutional network that is trained to differentiate between “fake pairs” of a chest CT image and a generated survival-time image from “true pairs” of the chest CT image and the corresponding observed survival-time image of the patient. For evaluation, we retrospectively collected high-resolution chest CT images of COVID-19 patients. The survival predictions of the pix2surv model on these patients were compared with those of existing clinical prognostic biomarkers by use of a two-sided t-test with bootstrapping. Concordance index (C-index) and relative absolute error (RAE) were used as measures of the prediction performance. The bootstrap evaluation yielded C-index and RAE values of 80.4% and 15.6% for the pix2surv model, whereas those for the extent of the well-aerated lung parenchyma were 49.8% and 33.6%, and for a combination of blood tests of lactic dehydrogenase, lymphocyte, and C-reactive protein were 69.8% and 25.5%, respectively. The increase in survival prediction by the pix2surv model was statistically significant (p < 0.0001), indicating high effectiveness of the pix2surv model as a prognostic biomarker for the survival of patients with COVID-19.