The Segment Anything Model (SAM) is a recently developed all-range foundation model for image segmentation. It can use sparse manual prompts such as bounding boxes to generate pixel-level segmentation in natural images but struggles in medical images such as low-contrast, noisy ultrasound images. We propose a refined test-phase prompt augmentation technique designed to improve SAM’s performance in medical image segmentation. The method couples multi-box prompt augmentation and an aleatoric uncertainty-based false-negative (FN) and false-positive (FP) correction (FNPC) strategy. We evaluate the method on two ultrasound datasets and show improvement in SAM’s performance and robustness to inaccurate prompts, without the necessity for further training or tuning. Moreover, we present the Single-Slice-to-Volume (SS2V) method, enabling 3D pixel-level segmentation using only the bounding box annotation from a single 2D slice. Our results allow efficient use of SAM in even noisy, low-contrast medical images. The source code has been released at: https://github.com/MedICL-VU/FNPC-SAM

Purpose: Accurate tool segmentation is essential in computer-aided procedures. However, this task conveys challenges due to artifacts’ presence and the limited training data in medical scenarios. Methods that generalize to unseen data represent an interesting venue, where zero-shot segmentation presents an option to account for data limitation. Initial exploratory works with the Segment Anything Model (SAM) show that bounding-box-based prompting presents notable zero-short generalization. However, point-based prompting leads to a degraded performance that further deteriorates under image corruption. We argue that SAM drastically over-segment images with high corruption levels, resulting in degraded performance when only a single segmentation mask is considered, while the combination of the masks overlapping the object of interest generates an accurate prediction. Method: We use SAM to generate the over-segmented prediction of endoscopic frames. Then, we employ the ground-truth tool mask to analyze the results of SAM when the best single mask is selected as prediction and when all the individual masks overlapping the object of interest are combined to obtain the final predicted mask. We analyze the Endovis18 and Endovis17 instrument segmentation datasets using synthetic corruptions of various strengths and an In-House dataset featuring counterfactually created real-world corruptions. Results: Combining the over-segmented masks contributes to improvements in the IoU. Furthermore, selecting the best single segmentation presents a competitive IoU score for clean images. Conclusions: Combined SAM predictions present improved results and robustness up to a certain corruption level. However, appropriate prompting strategies are fundamental for implementing these models in the medical domain.

Previous work on methods for cross domain generalization in medical imaging found a simple but very effective method called ”global intensity non-linear” (GIN) augmentation. Our goal in this study is to use the GIN approach to train a model as powerful as TotalSegmentator for MRI data, despite having neither sufficient amounts of MRI data nor ground truth organ contours. Instead, we employ the GIN augmentation approach to show qualitatively and quantitatively that this is indeed feasible for a diverse set of anatomical structures including abdominal and thoracic organs as well as bones. The models are trained on the TotalSegmentator and AMOS22 datasets. For evaluation we apply them to whole body MRI scans from the German National Cohort (NAKO) study with a set of in-house reference masks. With GIN augmentation the mean Dice score of the model increases from 0.18 to 0.52 on Dixon water images, when using TotalSegmentator data for training. The improvements can be further split into 0.47 to 0.66 for abdominal organs, 0.55 to 0.79 for thoracic organs and 0.00 to 0.40 for bones.

Three-dimensional lesion segmentation is required for analysis of radiomic features and lesion growth kinetics. In clinical trials, radiologists apply the Response Evaluation Criteria in Solid Tumors (RECIST), by manually annotating the long and short diameters of a lesion on a single 2D axial slice (RECIST slice), where the lesion looks largest. We developed a novel approach that leverages the RECIST annotations to segment lesions in 3D on CT scans. We start with bounding box and center point prompts derived from RECIST long and short diameters on RECIST slice. Iteratively, we perform prompted segmentation using Segment Anything Model (SAM) on off-RECIST slices towards the superior and inferior direction until all slices are segmented. To optimize the performance of SAM, we fine-tuned the mask decoder. In addition, it is crucial to detect where the lesions disappear at the superior and inferior direction to prevent over segmentation. We developed a multi-task framework for lesion existence classification and segmentation and further compared the parallel framework and cascaded framework. We used an internal dataset consisting of 2053 and 200 3D lesions for fine-tuning of SAM decoder and testing, respectively. Baseline SAM, SAM with fine tuning, SAM with parallel multi-task fine tuning, and SAM with cascaded multitask fine tuning have Dice scores of 0.4745±0.2138, 0.7136±0.1277, 0.6985±0.1312, and 0.7239±0.1321, respectively. Our experiments proves that multi-task learning is an effective way for 3D segmentation with SAM, and cascaded framework performs better than parallel framework.

Magnetic resonance images are often acquired as several 2D slices and stacked into a 3D volume, yielding a lower through-plane resolution than in-plane resolution. Many super-resolution (SR) methods have been proposed to address this, including those that use the inherent high-resolution (HR) in-plane signal as HR data to train deep neural networks. Techniques with this approach are generally both self-supervised and internally trained, so no external training data is required. However, in such a training paradigm limited data are present for training machine learning models and the frequency content of the in-plane data may be insufficient to capture the true HR image. In particular, the recovery of high frequency information is usually lacking. In this work, we show this shortcoming with Fourier analysis; we subsequently propose and compare several approaches to address the recovery of high frequency information. We test a particular internally trained self-supervised method named SMORE on ten subjects at three common clinical resolutions with three types of modification: frequency-type losses (Fourier and wavelet), feature-type losses, and low-resolution re-gridding strategies for estimating the residual. We find a particular combination to balance between signal recovery in both spatial and frequency domains qualitatively and quantitatively, yet none of the modifications alone or in tandem yield a vastly superior result. We postulate that there may either be limits on internally trained techniques that such modifications cannot address, or limits on modeling SR as finding a map from low-resolution to HR, or both.

Diffusion magnetic resonance imaging (dMRI) offers the ability to assess subvoxel brain microstructure through the extraction of biomarkers like fractional anisotropy, as well as to unveil brain connectivity by reconstructing white matter fiber trajectories. However, accurate analysis becomes challenging at the interface between cerebrospinal fluid and white matter, where the MRI signal originates from both the cerebrospinal fluid and the white matter partial volume. The presence of free water partial volume effects introduces a substantial bias in estimating diffusion properties, thereby limiting the clinical utility of DWI. Moreover, current mathematical models often lack applicability to single-shell acquisitions commonly encountered in clinical settings. Without appropriate regularization, direct model fitting becomes impractical. We propose a novel voxel-based deep learning method for mapping and correcting free-water partial volume contamination in DWI to address these limitations. This approach leverages data-driven techniques to reliably infer plausible free-water volumes across different diffusion MRI acquisition schemes, including single-shell acquisitions. Our evaluation demonstrates that the introduced methodology consistently produces more consistent and plausible results than previous approaches. By effectively mitigating the impact of free water partial volume effects, our approach enhances the accuracy and reliability of DWI analysis for single-shell dMRI, thereby expanding its applications in assessing brain microstructure and connectivity.

Generative adversarial networks (GANs) have been used to successfully translate images between multiple imaging modalities. While there is a significant amount of literature on the use cases for these approaches, there has been limited investigation into the optimal model design and evaluation criteria. In this paper, we demonstrated the performance of different approaches on the task of cone-beam computer tomography (CBCT) to fan-beam computer tomography (CT) translation. We examined the implications of choosing between 2D and 3D models, the size of 3D patches, and the integration of the Structural Similarity Index Measure (SSIM) into the cycle-consistency loss. Additionally, we introduced a partially-invertible VNet architecture into the RevGAN framework, enabling the use of 3D UNet-like architectures with minimal memory footprint. We compared image similarity metrics to visual inspection as an evaluation method for these models using held-out patient data and phantom scans to demonstrate their generalizability. Our findings suggest that 3D models, despite requiring a longer training time to converge due to the number of parameters, produce fewer image perturbations compared to 2D models. Training with larger patches also improved stability and significantly reduced artifacts, but increased the training time, while the SSIM-L1 cycle-consistency loss function enhanced performance. Interestingly, our study revealed a discrepancy between standard image similarity metrics and visual evaluation, with the former failing to adequately penalize visually evident artifacts in synthetic CT scans. This underscores the need for tailored and standardized evaluation metrics for medical image translation, which would facilitate more accurate comparisons across studies. To further the clinical applicability of image-to-image translation, we have open-sourced our methods and experiments, available at github.com/ganslate-team.

Kernel Synthesis (KS) in CT is an advanced image processing technique that involves the conversion of an image acquired with one specific convolution kernel into an image that appears as if it were acquired with a different kernel. This process doesn't require raw sinogram or reconstruction algorithm to generate images. While kernel synthesis methods exhibit promise, they can introduce noise and artifacts during transformation, highlighting the importance of effective noise and artifact management in the input data before synthesis. In this work, we represent kernel synthesis as a deep neural network regularized inverse problem. We optimize the convolutional neural network (CNN) weights in a data-consistent manner where CNN acts as an implicit prior to regularize the solution. Since, the learned CNN priors are more generic than hand crafted priors (like sparsity and total-variation), they helps control artifacts while preserving the details in the image. Experimental results for a low-resolution kernel (STND) data to a high resolution kernel (LUNG) data conversion clearly indicates that the output images from the proposed method resemble closely with images reconstructed using high-resolution kernel (LUNG) while simultaneously reducing the noise and clutter.

Magnetic Resonance Imaging with tagging (tMRI) has long been utilized for quantifying tissue motion and strain during deformation. However, a phenomenon known as tag fading, a gradual decrease in tag visibility over time, often complicates post-processing. The first contribution of this study is to model tag fading by considering the interplay between T₁ relaxation and the repeated application of radio frequency (RF) pulses during serial imaging sequences. This is a factor that has been overlooked in prior research on tMRI post-processing. Further, we have observed an emerging trend of utilizing raw tagged MRI within a deep learning-based (DL) registration framework for motion estimation. In this work, we evaluate and analyze the impact of commonly used image similarity objectives in training DL registrations on raw tMRI. This is then compared with the Harmonic Phase-based approach, a traditional approach which is claimed to be robust to tag fading. Our findings, derived from both simulated images and an actual phantom scan, reveal the limitations of various similarity losses in raw tMRI and emphasize caution in registration tasks where image intensity changes over time.

Deep neural networks have emerged as the preferred method for semantic segmentation of CT images in recent years. However, understanding their limitations and generalization properties remains an active area of research and a relevant topic for clinical applications. One crucial factor among many is the X-ray radiation dose, which is always kept as low as reasonably possible during CT acquisition. Therefore, potential dose reductions may pose a challenge for existing segmentation models. In this paper, we investigate robustness of the recently proposed TotalSegmentator model for anatomical segmentation with respect to dose reduction. TotalSegmentator combines a large CT dataset and the well-established nnU-Net framework to train deep learning models, resulting in state-of-the-art performance for anatomical segmentation. Our method relies on accurate low-dose simulations derived from acquired full-dose projections. For a set of registered low- and full-dose CT images, we measure the Dice score between the corresponding segmentations. Our results reveal a high level of robustness in the segmentation outcomes. Comprehensive quantitative comparisons demonstrate that at a 20% dose level, the Dice score declines by at most 3%. Visual comparisons reveal only minor differences at the boundaries of the segmented organs. These findings may have a large potential for dose reduction when CT data are acquired predominantly for segmentation purposes, such as for the planning of interventional or surgical procedures.

Intraprocedural 3D real-time magnetic resonance imaging (MRI) provides a way for accurate and precise radiofrequency catheter targeting during ventricular tachycardia ablation. However, the limited data acquisition time needed to freeze cardiac motion results in highly undersampled k-space data that are challenging to reconstruct. In this work, we evaluated several deep learning (DL) based methods for real-time reconstruction of highly undersampled 3D real-time cardiac MRI. Algorithm reconstruction performance and speed were compared between classical algorithms and DL-based methods. Generative adversarial networks with attention layers in the generator were used to perform reconstructions in the image domain, which strived to balance reconstruction speed and image quality. In addition, variational networks were implemented by iterating data consistency in k-space and enforcing image smoothness via neural network-based regularization. In a preliminary study of heartbeat-resolved highly undersampled 3D cardiac MRI for 11 healthy volunteers, we observed that DL reconstruction methods provided good image quality with a significant increase in computational speed.

Invasive treatment of coronary artery disease (CAD) is costly and burdensome for the patient. Therefore, prediction of treatment success would be of clinical value. This study presents a deep learning method for the prediction of the post-treatment fractional flow reserve (FFR) in patients with coronary artery disease (CAD) from pre-treatment coronary CT angiography (CCTA). To simulate post-treatment FFR, pre-treatment coronary artery characteristics are modified to mimic invasive coronary treatment. Artery characterization and subsequent prediction of the FFR values along the artery are performed using deep learning. The method was tested on CCTA scans of 29 patients with invasive pre- and post-treatment FFR measurements along the artery. Achieved accuracy for the prediction of the presence of a functionally significant stenosis was 0.77 before and 0.63 after simulated treatment. The analysis took 0.8 s per artery. The results indicate that real-time treatment planning might be feasible.

The heterogeneity in myofiber helicity across the cardiac wall causes twisting (torsion) in the left ventricle (LV) during contraction, which is a significant contributor to its pumping function. Although important progress has been made in identifying and studying a quantitative “global” metric for torsion, four-dimensional (4D) torsion characteristics in the LV remain underexplored. We propose an imaging-based framework that uses myocardial motion obtained from cine cardiac magnetic resonance (CMR) scans of the LV to calculate torsion. We performed our LV torsion analysis in a mitral valve prolapse (MVP) human patient (n=1) pre- and post-MV repair. Establishing a high-fidelity torsion measure in the LV offers a rigorous regional marker to investigate the potential reversal of cardiac remodeling post-surgical interventions, such as MV repair. After image acquisition, a non-rigid image registration algorithm was used to calculate 4D LV displacements. Twisting was evaluated through (i) an in-plane rotation-based approximation (referred to as T_2D) and (ii) a three-dimensional formulation involving in-plane and through-plane strains (referred to as T_3D). Comparing T_2D to T_3D, broad regions of positive (counter-clockwise) rotation, captured through T_3D, were unrepresented by T_2D despite a qualitative agreement between the two metrics in capturing the average regional torsion. Also, the presence of comprehensive transmural positive torsion at the apical section was opposed to negative epicardial torsion at the midsection. Such variations in torsional behavior, captured by T_3D, are expected due to transmural helicity in fiber orientation. Overall, the underlying effects of through-plane shear in characterizing LV torsion were evidenced, and the image registration framework offered a comprehensive tool to capture 4D myocardial torsion that can complement conventional LV global markers.

Aortic valve morphometry plays a crucial role in understanding and diagnosing cardiovascular diseases. Precise localization of landmarks in three-dimensional (3D) computed tomography (CT) images of the heart, particularly landmarks on the aortic valve, enables accurate assessment of valve structure and dimensions. Such information is vital for planning surgical interventions, evaluating valve function, and monitoring disease progression. Reliable landmark localization methods aid clinicians in making informed decisions, leading to improved patient outcomes and enhanced overall cardiovascular healthcare. In this study, we present a comprehensive comparison of two landmark localization methods, i.e. the spatial configuration network (SCN) and communicative multi-agent reinforcement learning (C-MARL), for detecting six distinctive landmarks on the aortic cusps from 160 3D CT images of healthy and pathological subjects. Both methods were individually trained on images from 80 healthy subjects, and their robustness to new, unseen pathological images was assessed by evaluating the trained models on 40 images from healthy and 40 images from pathological subjects. SCN exhibited superior performance in accurately localizing landmarks in healthy subjects (mean distance ± standard deviation against reference landmarks of 1.14±0.78 mm), showcasing its proficiency in normal anatomy scenarios. On the other hand, C-MARL demonstrated remarkable adaptability to the complexity of pathology, yielding better results for pathological subjects (2.66±3.99 mm). Both methods offer valuable insights for biomedical imaging applications.

Accurate classification of medical images is crucial for disease diagnosis and treatment planning. Deep learning (DL) methods have gained increasing attention in this domain. However, DL-based classification methods encounter challenges due to the unique characteristics of medical image datasets, including limited amounts of labeled images and large image variations. Self-supervised learning (SSL) has emerged as a solution that learns informative representations from unlabeled data to alleviate the scarcity of labeled images and improve model performance. A recently proposed generative SSL method, masked autoencoder (MAE), has shown excellent capability in feature representation learning. The MAE model trained on unlabeled data can be easily tuned to improve the performance of various downstream classification models. In this paper, we performed a preliminary study to integrate MAE with the self-attention mechanism for tumor classification on breast ultrasound (BUS) data. Considering the speckle noise, image quality variations of BUS images, and varying tumor shapes and sizes, two revisions were adopted in using MAE for tumor classification. First, MAE’s patch size and masking ratio were adjusted to avoid missing information embedded in small lesions on BUS images. Second, attention maps were extracted to improve the interpretability of the model’s decision-making process. Experiments demonstrated the effectiveness and potential of the MAE-based classification model on small labeled datasets.

Convolutional Neural Networks (CNNs) exhibit strong performance in medical image segmentation tasks by capturing high-level (local) information, such as edges and textures. However, due to the limited field of view of convolution kernels, it is hard for CNNs to fully represent global information. Recently, transformers have shown good performance for medical image segmentation due to their ability to better model long-range dependencies. Nevertheless, transformers struggle to capture high-level spatial features as effectively as CNNs. A good segmentation model should learn a better representation from local and global features to be both precise and semantically accurate. In our previous work, we proposed CATS, which is a U-shaped segmentation network augmented with transformer encoder. In this work, we further extend this model and propose CATS v2 with hybrid encoders. Specifically, hybrid encoders consist of a CNN-based encoder path paralleled to a transformer path with a shifted window, which better leverage both local and global information to produce robust 3D medical image segmentation. We fuse the information from the convolutional encoder and the transformer at the skip connections of different resolutions to form the final segmentation. The proposed method is evaluated on three public challenge datasets: Beyond the Cranial Vault (BTCV), Cross-Modality Domain Adaptation (CrossMoDA) and task 5 of Medical Segmentation Decathlon (MSD-5), to segment abdominal organs, vestibular schwannoma (VS) and prostate, respectively. Compared with the state-of-the-art methods, our approach demonstrates superior performance in terms of higher Dice scores. Our code is publicly available at https://github.com/MedICL-VU/CATS.

Ultrasound imaging is a cost-effective and radiation-free modality for visualizing anatomical structures in real-time, making it ideal for guiding surgical interventions. However, its limited field-of-view, speckle noise, and imaging artifacts make it difficult to interpret the images for inexperienced users. In this paper, we propose a new 2D ultrasound to 3D CT registration method to improve surgical guidance during ultrasound-guided interventions. Our approach adopts a dense feature matching method called LoFTR to our multimodal registration problem. We learn to predict dense coarse-to-fine correspondences using a Transformer-based architecture to estimate a robust rigid transformation between a 2D ultrasound frame and a CT scan. Additionally, a fully differentiable pose estimation method is introduced, optimizing LoFTR on pose estimation error during training. Experiments conducted on a multimodal dataset of ex vivo porcine kidneys demonstrate the method’s promising results for intraoperative, trackerless ultrasound pose estimation. By mapping 2D ultrasound frames into the 3D CT volume space, the method provides intraoperative guidance, potentially improving surgical workflows and image interpretation.

Pulmonary sarcoidosis (PS) is an inflammatory interstitial lung disease, causing clusters of inflamed tissue called ‘granulomas’ within the lung. However, PS may mimic other conditions such as malignancy or infection, which often leads to a delayed diagnosis, leading to worsening respiratory function. Chest computed tomography (CT) is used to diagnose PS but with varying specificities, as the appearance of PS and other diffuse lung diseases on chest CT are myriad and considered difficult to diagnose by many radiologists. In this work we used a radiomics-guided ensemble of 3D CNN and Vision Transformers (ViT) features (Rad-CNNViT) to classify between PS and other interstitial lung diseases (o-ILDs). The input to the network was the 3D radiomics map that received the top feature score in a Random Forest (RF)-based feature selection approach. The input map was then fed into an ensemble network of CNN and ViT to capture local and global features respectively for diffuse lung disease classification. Training datasets of PS (n=61) and o-ILD chest CTs (n=154) were used for feature discovery using RF and train the Rad-CNNViT and a radiomics-based machine learning (Rad-ML) framework. On a separate test cohort of PS (n=65) and o-ILD (n=96), Rad-CNNViT ensemble network differentiated PS from o-ILDs with higher AUC=0.89 compared to a Rad-ML approach with AUC=0.77.

Self-supervised pretraining has shown great performance in improving the accuracy of downstream tasks. Although pretraining on a large dataset improves performances, it becomes challenging to further optimize the model by solely enlarging the dataset. In contrast, additional adaptation of pretrained models to the target domain has shown promise in NLP. Inspired by the success of continual pretraining, we investigated the efficacy of adapting the target domain dataset to a pretrained model in medical imaging, particularly in the context of segmentation. We present a study based on a self-supervised pretraining framework using the SwinUNETR backbone. In this study, we improved the generalizability of the self-supervised pretraining by adapting a foundational model pretrained on 5k CT volumes to data of the downstream segmentation task. In detail, we employed 385 abdominal CT volumes for the continual task-adaptive pretraining and 24 abdominal CT volumes for the downstream segmentation task, all sourced from the same dataset. Additionally, we conducted comparative experiments to demonstrate the benefits of this task-adapting pretraining approach. Our method has shown that continual pretraining helps to improve the performances, achieving an average Dice score for 10-class organ segmentation of 87.8%.

In this presentation, I will present Metrics Reloaded - a comprehensive framework guiding researchers in the problem-aware selection of metrics in biomedical image analysis. I will first demonstrate how the choice of performance metrics often fails to align with the clinicians’ interests, thereby impeding scientific progress and the practical application of machine learning (ML) algorithms. Next, I will present the Metrics Reloaded framework, which has been developed by an international expert consortium to overcome the aforementioned issues.

Reproducibility is a key component of science, as the replication of findings is the process by which they become knowledge. It is widely considered that many fields of science, including medical imaging, are undergoing a reproducibility crisis. This has led to the publications of various guidelines to improve research reproducibility, such as the checklists proposed by NeurIPS or MICCAI. In this presentation, after introducing various concepts related to reproducibility, we will discuss the fact that using such checklists is not straightforward, both for the authors that have to fill them and for the reviewers that have to comment them, and that their use remains to be clarified for reproducible research to grow in the medical image processing field.

Despite the expert-level performance of artificial intelligence (AI) models for various medical imaging tasks, real-world performance failures with disparate outputs for various minority subgroups limit the usefulness of AI in improving patients’ lives. AI has been shown to have a remarkable ability to detect protected attributes of age, sex, and race, while the same models demonstrate bias against historically underserved subgroups of age, sex, and race in disease diagnosis. Therefore, an AI model may take shortcut predictions from these correlations and subsequently generate an outcome that is biased toward certain subgroups even when protected attributes are not explicitly used as inputs into the model. This talk will discuss various types of bias from shortcut learning that may occur at different phases of AI model development. I will also summarize current techniques for mitigating bias from preprocessing (data-centric solutions) and during model development (computational solutions) and postprocessing (recalibration of learning).

The human cerebral cortex has many bumps and grooves called gyri and sulci. Even though there is a high interindividual consistency for the main cortical folds, this is not the case when we examine the exact shapes and details of the folding patterns. Because of this complexity, characterizing the cortical folding variability and relating them to subjects’ behavioral characteristics or pathologies is still an open scientific problem. Classical approaches include labeling a few specific patterns, either manually or semi-automatically, based on geometric distances, but the recent availability of MRI image datasets of tens of thousands of subjects makes modern deep-learning techniques particularly attractive. Here, we build a self-supervised deep-learning model to detect folding patterns in the cingulate region. We train a contrastive self-supervised model (SimCLR) on both Human Connectome Project (1101 subjects) and UKBioBank (21070 subjects) datasets with topological-based augmentations on the cortical skeletons, which are topological objects that capture the shape of the folds. We explore several backbone architectures (convolutional network, DenseNet, and PointNet) for the SimCLR. For evaluation and testing, we perform a linear classification task on a database manually labeled for the presence of the ”double-parallel” folding pattern in the cingulate region, which is related to schizophrenia characteristics. The best model, giving a test AUC of 0.76, is a convolutional network with 6 layers, a 10-dimensional latent space, a linear projection head, and using the branch-clipping augmentation. This is the first time that a self-supervised deep learning model has been applied to cortical skeletons on such a large dataset and quantitatively evaluated. We can now envisage the next step: applying it to other brain regions to detect other biomarkers. The GitHub repository is publicly available on https://github.com/neurospin-projects/2022 jchavas cingulate inhibitory control.

Magnetic resonance spectroscopy (MRS) is one of the few non-invasive imaging modalities capable of making neurochemical and metabolic measurements in vivo. Traditionally, the clinical utility of MRS has been narrow. The most common use has been the "single-voxel spectroscopy" variant to discern the presence of a lactate peak in the spectra in one location in the brain, typically to evaluate for ischemia in neonates. Thus, the reduction of rich spectral data to a binary variable has not classically necessitated much signal processing. However, scanners have become more powerful and MRS sequences more advanced, increasing data complexity and adding 2 to 3 spatial dimensions in addition to the spectral one. The result is a spatially- and spectrally-variant MRS image ripe for image processing innovation. Despite this potential, the logistics for robustly accessing and manipulating MRS data across different scanners, data formats, and software standards remain unclear. Thus, as research into MRS advances, there is a clear need to better characterize its image processing considerations to facilitate innovation from scientists and engineers. Building on established neuroimaging standards, we describe a framework for manipulating these images that generalizes to the voxel, spectral, and metabolite level across space and multiple imaging sites while integrating with LCModel, a widely used quantitative MRS peak-fitting platform. In doing so, we provide examples to demonstrate the advantages of such a workflow in relation to recent publications and with new data. Overall, we hope our characterizations will lower the barrier of entry to MRS processing for neuroimaging researchers.

Multiple sclerosis (MS) is a demyenalinating inflammatory neurological disease. In vivo biomarkers of myelin content are of major importance for patient care and clinical trials. Positron Emission Tomography (PET) with Pittsburgh Compound B (PiB) provides a specific myelin marker. However, it is not available in clinical routine. In this paper, we propose a method to generate myelin maps by synthesizing PiB PET from clinical routine MRI sequences (T1-weighted and FLAIR). To that purpose, we introduce a new curriculum learning strategy for training generative adversarial networks (GAN). Specifically, we design a curricular approach for training the discriminator: training starts with only lesion patches and random patches (from anywhere in the white matter) are progressively introduced. We relied on two distinct cohorts of MS patients acquired each on a different scanner and in a different country. One cohort was used for training/validation and the other one for testing. We found that the synthetic PiB PET was strongly correlated to the ground-truth both at the lesion level (r = 0.70, p < 10⁻⁵) and the patient level (r = 0.74, p < 10⁻⁵). Moreover, the correlations were stronger when using the curricular learning strategy compared to starting the discriminator training from random patches. Our results demonstrate the interest of this new curriculum learning strategy for PET image synthesis. Even though further evaluations are needed, our approach has the potential to provide a useful biomarker for clinical routine follow-up of patients with MS.

White matter signals in resting state blood oxygen level dependent functional magnetic resonance (BOLD-fMRI) have been largely discounted, yet there is growing evidence that these signals are indicative of brain activity. Understanding how these white matter signals capture function can provide insight into brain physiology. Moreover, functional signals could potentially be used as early markers for neurological changes, such as in Alzheimer’s Disease. To investigate white matter brain networks, we leveraged the OASIS-3 dataset to extract white matter signals from resting state BOLDFMRI data on 711 subjects. The imaging was longitudinal with a total of 2,026 images. Hierarchical clustering was performed to investigate clusters of voxel-level correlations on the timeseries data. The stability of clusters was measured with the average Dice coefficients on two different cross fold validations. The first validated the stability between scans, and the second validated the stability between subject populations. Functional clusters at hierarchical levels 4, 9, 13, 18, and 24 had local maximum stability, suggesting better clustered white matter. In comparison with JHU-DTI-SS Type-I Atlas defined regions, clusters at lower hierarchical levels identified well defined anatomical lobes. At higher hierarchical levels, functional clusters mapped motor and memory functional regions, identifying 50.00%, 20.00%, 27.27%, and 35.14% of the frontal, occipital, parietal, and temporal lobe regions respectively.

Analysis of microcirculation is crucial for several clinical outcomes. Heterogeneity of blood flow and low perfusion in microcirculation is linked to outcomes in sepsis, cardiogenic shock and hemodynamically compromised patients. Although imaging advancements have progressed substantially in the last couple of decades, manual analysis of microcirculation remains the gold standard. Capillary detection in microcirculation videos is difficult because of sublingual or hand motion in the handheld videos, and intermittent visibility of capillaries due to plasma gaps. Most automated methods perform capillary segmentation on a mean-image over time, which fail to detect intermittently visible capillaries due to their weaker contrast as a result of averaging. To normalize the spatial contrast in capillaries, we propose to use a relative perfusion feature map. Our feature map can be used in both unsupervised and supervised settings. Experiments show that our method leads to consistent improvements in total vessel density and centerline recall over baselines. Statistically, our method has lesser bias, achieves better agreement with the ground truth, and correlates better with manual annotations, making it a clinically viable benchmark.

Cell segmentation is a crucial task in brightfield microscopy for cellular analysis. Conventional segmentation methods, including thresholding and supervised deep learning approaches, have limitations due to weak cell boundaries and the requirement for expert annotations of cells. To address these challenges, this work proposes an unsupervised cell-segmentation model. Specifically, the model maximizes mutual information between an artificially created data pair to train a neural network for cell segmentation. The proposed method achieves the fastest inference time and requires only a small fraction of network parameters compared to reported state-of-the-art alternatives from literature, making the proposal more efficient for live-cell experiments where large amounts of data are collected. A quantitative comparison with other state-of-the-art models shows that the unsupervised model outperforms relevant supervised models in terms of the Jaccard index. In detail, the unsupervised model achieves a remarkable Jaccard score of 0.71 on a out-of-sample proprietary dataset which is fully annotated. This represents a notable improvement of 4.0% compared to the supervised Cellpose model and an impressive 11.4% increase compared to the unsupervised Leopart model. In conclusion, the proposed model effectively segments cells of six types without the need for expert annotations, thereby making it a valuable tool for cellular analysis. The source code is available at https://github.com/wesselvannierop/wescell.

Deep learning has revolutionized medical image analysis, promising to significantly improve the precision of diagnoses and therapies through advanced segmentation methods. However, the efficacy of deep neural networks is often compromised by the prevalence of imperfect medical labels, while acquiring large-scale, accurately labeled data remains a prohibitive challenge. To address the imperfect label issue, we introduce a novel learning framework that iteratively optimizes both a neural network and its label set to enhance segmentation accuracy. This framework operates through two steps: initially, it robustly trains on a dataset with label noise, distinguishing between clean and noisy labels, and subsequently, it refines noisy labels based on high-confidence predictions from the robust network. By applying this method, not only is the network trained more effectively on imperfect data, but the dataset is progressively cleaned and expanded. Our evaluations are conducted on retina Optical Coherence Tomography datasets using U-Net and SegNet architectures, and demonstrate substantial improvements in segmentation accuracy and data quality, advancing the capabilities of weakly supervised segmentation in medical imaging.

Purpose: The complexity of three-dimensional (3D) image segmentation exceeds that of its 2D counterpart, primarily due to increased memory demands resulting from the larger size of input images. To mitigate these challenges, a common strategy is to extract patches from the original preprocessed images. The use of sophisticated models can sometimes offset the trade-off associated with input image size, but increased image sizes tend to yield superior performance. In this study, we aim to alleviate memory constraints while improving the accuracy of segmenting 9 distinct substructures within the knee anatomy as compared to state-of-the-art models.
Approach: We utilized a cohort of 183 knee MRI scans, with nine anatomical structures annotated by radiologists. This dataset was divided into training, validation, and independent testing sets at a ratio of 8:1:1, respectively. The second standard convolutional layer of the nnU-Net is substituted with our MS-SCConv module at each down-sampling stage. The MS-SCConv module employs soft attention mechanisms to weight both the original and refined feature maps. The refined feature maps are produced using the SCConv process, which integrates two units specifically designed to minimize feature redundancy from both spatial and channel perspectives. Comprehensive ablation studies were performed to evaluate the specific contributions of the MS-SCConv module within the informative nnU-Net framework. The performance of our proposed model was compared with several other models, including U-Net, UNETR, DynU-Net and nnU-Net.
Results: Informative nnU-Net demonstrated superior segmentation performance, achieving the high Dice coefficient (0.83) compared to other models, namely U-Net (0.71), UNETR (0.71), DynU-Net (0.74), and nnU-Net (0.81) (p<0.05). Notably, Informative nnU-Net achieves improved accuracy while reducing the number of parameters from 27.2M to 26.9M.
Conclusions: Our experiments revealed that with MS-SCConv and informative nnU-Net, the number of parameters can be decreased while improving segmentation performance.

Despite its benefits for tumour detection and treatment, the administration of contrast agents in dynamic contrast-enhanced MRI (DCE-MRI) is associated with a range of issues, including their invasiveness, bioaccumulation, and a risk of nephrogenic systemic fibrosis. This study explores the feasibility of producing synthetic contrast enhancements by translating pre-contrast T1-weighted fat-saturated breast MRI to their corresponding first DCE-MRI sequence leveraging the capabilities of a generative adversarial network (GAN). Additionally, we introduce a Scaled Aggregate Measure (SAMe) designed for quantitatively evaluating the quality of synthetic data in a principled manner and serving as a basis for selecting the optimal generative model. We assess the generated DCE-MRI data using quantitative image quality metrics and apply them to the downstream task of 3D breast tumour segmentation. Our results highlight the potential of post-contrast DCE-MRI synthesis in enhancing the robustness of breast tumour segmentation models via data augmentation. Our code is available at https://github.com/RichardObi/pre_post_synthesis.

This paper proposes an intestine segmentation method to segment intestines from CT volumes for helping clinicians diagnose intestine obstruction. For large-scale labeled datasets, fully-supervised methods have shown superior results. However, medical image segmentation is usually difficult to achieve accurate prediction due to the limited number of labeled data available for training. To address this challenge, we introduce a novel multi-view symmetrical network (MVS-Net) for intestine segmentation and incorporate bidirectional teaching to utilize unlabeled datasets. Specifically, we design the MVS-Net, which can use different sizes of convolution kernels instead of a fixed kernel size, enabling the network to capture multi-scale features from images’ different perceptual fields and ensure segmentation accuracy. Additionally, the pseudo-labels are generated by bidirectional teaching, which can make the network captures semantic information from large-scale unlabeled data for increasing the training data. We repeated the experiment five times, and used the averaged result on the intestines dataset to represent the segmentation accuracy of the proposed method. The experimental results showed the average Dice was 78.86%, the average recall 84.50%, and the average precision 75.94%, respectively.

Image quality assessment of CT scans is of utmost importance in balancing radiation dose and image quality. Nonetheless, estimating the image quality of CT scans is a highly subjective task that cannot be adequately captured by a single quantitative metric. In this work, we present a novel vision Transformer network for no-reference CT image quality assessment. Our network combines convolutional operations and multi-head self-attention mechanisms by adding a powerful convolutional stem in the beginning of the traditional ViT network. To enhance the performance and efficiency of the network, we introduce a distillation methodology, comprised of two sequential steps. In Step I, we construct a “teacher ensemble network” by training five Vision Transformer networks using a five-fold division schema. In Step II, we train a single vision Transformer, referred to as the “student network”, by using the teacher’s predictions as new labels. The student network is also optimized using the original labeled dataset. The effectiveness of the proposed model is evaluated on the task of predicting image quality scores from low-dose abdominal CT images from the LDCTIQAC2023 Grand Challenge. Our model demonstrates remarkable performance, ranking 6th during the testing phase of the challenge. Additionally, our experiments highlight the effectiveness of incorporating a convolutional stem in the ViT architecture and the distillation methodology.

Recent development in deep learning based image processing has spurred research in development of large and complex imaging models for detection of disease or segmentation of regions of interest like masses/nodules or organs in high-dimensional medical imaging data like CPT and MRI volumes. Given the complexity of the model and dimensionality of the input, large amounts of annotated training data is required to build such models. However, the annotation process for medical imaging data is challenging because of tedious processes and limited availability of clinical experts such as radiologists. Hence, only small amounts of annotated imaging data is generally available for any novel task – resulting in overfitted models with poor generalization capacity. A common solution to this challenge has been to pre-train the models on large imaging dataset of generic images like ImageNet in a supervised manner using class/object labels available with the data and then fine tuning them on small amounts of annotated medical imaging data [1]. This strategy often fails when pre-training and downstream tasks do not share the domain and thus the low-level visual features (e.g. classification of tumor molecular subtype using model pretrained on ImageNet dataset). In contrast to fine-tuning, foundation models [2] are deep models with multiple layers trained on large datasets in self-supervised fashion at a scale which attempt to learn generic low-level features, with the potential to transfer their knowledge of a variety of downstream target tasks - even when the self-supervised and target tasks are not similar. Attempts have been made to train such models in various domains including natural language processing (NLP), image processing, as well as combined vision-language research tasks [3] [4] [5] [6] [7] [8] [9] [10]. Recently, the NLP domain has arguably undergone revolutionary advances with foundation models allowing release of large language models (LLMs), such as GPT [3]. Similar ideas of self-supervision were translated in the domain of image processing. Such models are used to learn image representations, encode fundamental visual characteristics, and are tested for downstream prediction tasks like object detection on datasets like ImageNet. Simultaneously, complex vision-language models like CLIP [6], DALL-E [8], FLAVA [9], and Socratic models [12] were trained in a self-supervised way for understanding correlation in multi-modal data, and enabling reasoning across multiple modalities, and even applied in generation tasks. Inspired by vast research performed in the field of image and natural language processing for self-supervised training of foundation models [13] [4] [5], we proposed radiology-specific task agnostic self-supervision techniques. We experimented with a systematic solution for handling limited annotated data problems for a commonly performed imaging exam (CT of the chest) that can be independent of the downstream clinical tasks. Potential downstream applications for chest CT are vast including segmentation of multiple organs like lungs and heart and/or segmentation of lung nodules, and detection of various abnormalities, such as pulmonary embolism, COVID-19, pneumonia, and lung cancer, coronary calcium scoring. Many of these tasks have been released with datasets as multiple public challenges. To counteract scarcity of annotated data at individual institutions, data in challenges are often pooled from multiple healthcare organizations and can make large, annotated sets available for relevant tasks like PE detection and lung nodule segmentation. However, this approach is not scalable for producing annotated sets for all relevant applications outside of the intended target (e.g., using publicly available PE dataset for cardiac segmentation). Thus, a foundation model could fill this gap by extending its ability for various downstream tasks outside of its core training task. The foundation model can reduce the need for large volume annotated data for any chest CT axial view based downstream task providing a task-independent solution to data annotation challenge. We evaluated the performance of the foundation model on two separate clinical tasks (pulmonary embolism (PE) detection, lung nodule segmentation) and compared them to state-of-the-art models (SOTA).

Deep learning (DL) has led to significant improvements in medical image synthesis, enabling advanced image-toimage translation to generate synthetic images. However, DL methods face challenges such as domain shift and high demands for training data, limiting their generalizability and applicability. Historically, image synthesis was also carried out using deformable image registration (DIR), a method that warps moving images of a desired modality to match the anatomy of a fixed image. However, concerns about its speed and accuracy led to its decline in popularity. With the recent advances of DL-based DIR, we now revisit and reinvigorate this line of research. In this paper, we propose a fast and accurate synthesis method based on DIR. We use the task of synthesizing a rare magnetic resonance (MR) sequence, white matter nulled (WMn) T1-weighted (T1-w) images, to demonstrate the potential of our approach. During training, our method learns a DIR model based on the widely available MPRAGE sequence, which is a cerebrospinal fluid nulled (CSFn) T1-w inversion recovery gradient echo pulse sequence. During testing, the trained DIR model is first applied to estimate the deformation between moving and fixed CSFn images. Subsequently, this estimated deformation is applied to align the paired WMn counterpart of the moving CSFn image, yielding a synthetic WMn image for the fixed CSFn image. Our experiments demonstrate promising results for unsupervised image synthesis using DIR. These findings highlight the potential of our technique in contexts where supervised synthesis methods are constrained by limited training data.

It is known that deep neural networks (DNNs) are vulnerable to adversarial noises. Improving adversarial robustness of DNNs is essential. This is not only because unperceivable adversarial noise is a threat to the performance of DNNs models, but also adversarially robust DNNs have a strong resistance to the white noises that may present everywhere in the actual world. To improve adversarial robustness of DNNs, a variety of adversarial training methods have been proposed. Most of the previous methods are designed under one single application scenario: image classification. However, image segmentation, landmark detection, and object detection are more commonly observed than classifying the entire images in the medical imaging field. Although classification tasks and other tasks (e.g., regression) share some similarities, they also differ in certain ways, e.g., some adversarial training methods use misclassification criteria, which is well-defined in classification but not in regression. These restrictions/limitations hinder application of adversarial training for many medical imaging analysis tasks. In our work, the contributions are as follows: (1) We investigated the existing adversarial training methods and discovered the challenges that make those methods unsuitable for adaptation in segmentation and detection tasks. (2) We modified and adapted some existing adversarial training methods for medical image segmentation and detection tasks. (3) We proposed a general adversarial training method for medical image segmentation and detection. (4) We implemented our method in diverse medical imaging tasks using publicly available datasets, including MRI segmentation, Cephalometric landmark detection, and blood cell detection. The experiments substantiated the effectiveness of our method.

Biased outcomes in machine learning models can arise due to various factors, including limited training data, imbalanced class distribution, suboptimal training methodologies, and overfitting. In training neural networks with Stochastic Gradient Descent (SGD) and backpropagation, the choice of hyperparameters like learning rate and momentum is crucial to influencing the model’s performance. A comprehensive grid search study was conducted using static hyperparameters with standard SGD and dynamic hyperparameters with the Adam optimizer. The investigation focused on a multifaceted analysis across different tasks —- classification, segmentation, and detection — and was applied to four image-based applications: digit classification using the MNIST dataset, tuberculosis detection from chest X-ray images, lung segmentation in chest X-ray images, and the detection of malaria parasites in blood smear images. In the first comparative study on the MNIST dataset, the SGD algorithm consistently outperformed the Adam algorithm as noise levels increased. SGD held a slight advantage in accuracy over Adam in a noise-free environment. This advantage became more apparent as noise was introduced and increased to moderate levels. At high noise levels, both algorithms experienced a significant decline in performance, yet SGD maintained a relatively better accuracy compared to Adam. This trend underscored SGD’s superior ability to generalize across varying noise conditions. For TB detection in the second application, the DenseNet121 architecture was used, and it was found that Adam showed better performance on the larger TBX11K dataset. However, SGD outperformed Adam on a smaller subset of TBX11K. In the third comparative study for lung segmentation using COVID-19 and NLM datasets, SGD slightly outperformed Adam based on the mean and Hausdorff distances. In the case of malaria parasite detection using the YOLOv8 architecture, SGD and Adam optimizers showed varying performances across different conditions. Initially, both achieved high accuracies on the full dataset, with Adam slightly outperforming SGD. However, with decreasing dataset size, SGD maintained more consistent performance, while Adam’s accuracy fluctuated significantly. In noise tests, both showed equal accuracy on clean data, but under noise conditions, SGD maintained higher accuracies, suggesting better generalization capabilities. Further experiments assessed how well SGD and Adam could cope with domain shifts, mainly when using data from different countries. SGD’s generalization performance was superior to Adam’s performance in these experiments. In summary, although Adam is arguably the more popular of the two optimization techniques, SGD held its own in the experiments and showed superior performance when it came to generalization or classifying in noisy conditions.

Caliper placement is an integral part of ultrasound clinical workflow, e.g., kidney volume measurement. Automated approaches utilize anatomical segmentation followed by application-specific caliper placement. Robust clinical outcomes require confidence/uncertainty associated with such predictions be indicated. Conventional methods estimating uncertainty (MC Dropout, Deep Ensembles) with high computational load are impractical for deployment. We exploit the existence of uncertainty only on boundary pixels for any predicted segmentation. We utilize disagreement between independent predictions – region segmentation edge and direct boundary prediction, to identify uncertainty on anatomical boundary. We demonstrate our Boundary-Aware Segmentation Uncertainty (BASU) on cross-sections of kidney, correlating with ground-truth and clinician’s intuitions.

Medical image segmentation has improved with deep-learning methods, especially for tumor segmentation. However, variability in tumor shapes, sizes, and enhancement remains a challenge. Breast MRI adds further uncertainty due to anatomical differences. Informing clinicians about result reliability and using model uncertainty to improve predictions are essential. We study Monte-Carlo Dropout for generating multiple predictions and finding consensus segmentation. Our approach reduces false positives using per-pixel uncertainty and improves segmentation metrics. In addition, we study the correlation of model performance to the perceived ease of manual segmentation. Finally, we compare the per-pixel uncertainty with the inter-rater variability as segmented by six different radiologists. Our code is available at https://github.com/smriti-joshi/uncertainty-segmentation-mcdropout.git.

Automated lesion segmentation is essential to provide fast, reproducible tumor load estimates. Though deep learning methods have achieved unprecedented results in this field, they are often difficult to interpret, hampering their potential integration in the clinic. An interpretable deep learning approach is proposed for segmenting melanoma lesions on whole-body fluorine-18 fluorodeoxyglucose ([¹⁸F]FDG) positron emission tomography (PET) / computed tomography (CT). This consists of an automated PET thresholding step to identify FDGavid regions, followed by a three-channel nnU-Net considering the binary mask in addition to the PET and CT images. This segmentation step differentiates healthy from malignant tissue and removes the restriction on lesion boundaries imposed by the thresholding. The proposed method, trained on 267 images and evaluated on two sets acquired at the same institute, achieved mean Dice similarity coefficients (DSC) of 0.779 and 0.638 with mean absolute volume differences of 15.2mL and 22.0 mL. The DSC proved significantly higher compared to a direct, two-channel nnU-Net considering only the PET and CT. The same was observed when retraining and testing on subsets of the public data of the autoPET challenge, containing melanoma, lung cancer and lymphoma patients. In addition, overall results proved superior to a previously proposed two-step approach, where a classification network categorized each component of increased tracer uptake as healthy or malignant. The proposed lesion segmentation method for whole-body [¹⁸F]FDG PET/CT incorporates prior thresholding information while allowing more flexibility in the lesion delineation than a pure thresholding approach and increased interpretability over a direct segmentation network.

Foundation models rapidly emerge as a transformative force in medical imaging, leveraging extensive datasets and sophisticated pre-trained models to decode and interpret complex medical images. Our presentation will begin with an in-depth exploration of the essential concepts that underpin these models, with a special emphasis on the synergy between vision-language models and medical imaging. We aim to elucidate how the integration of prompts, language, and vision catalyzes a groundbreaking shift in the foundation of artificial intelligence. We will then analyze how these four critical elements - prompts, language, vision, and foundation models - will collaboratively shape state-of-the-art AI solutions in medical imaging. Our objective is to ignite a vibrant dialogue about leveraging the collective strength of these components at the SPIE Medical Imaging Conference.

Radiology workflow automation requires knowledge of exam contents in an image series such as anatomy region, injected contrast phase, presence of metals, so that appropriate post-processing steps and analysis can be invoked automatically. This paper investigates the applicability of DL to the task of classifying an entire image series into one of fourteen common exam types. A total of 2300 independent computed tomography (CT) image series, each manually labeled for its exam category by clinical experts, was used to train DL models. An additional 593 series were labeled and used as an independent test set. Each CT image series containing a 3D volume acquisition is converted to a special 2D multiplanar-reconstruction (MPR) image. DL based classifier was trained to classify the image series based on this 2D representation, which could be an AP view, a Lateral view or both. Different convolutional neural network architectures with varying block depths were compared. Global average pooling (GAP) layer was used in the final classification block and the impact of input view was studied. The impact of depth of feature extraction layer, input image type, data augmentation techniques and learning rates were studied. The best single class prediction accuracy achieved was 97%. The top-two classes classification accuracy reached > 99%. This method avoids the cost of inferencing each image in a 3D series but still provides very high classification accuracy.

The recent advent of clinical data warehouses (CDWs) has facilitated the sharing of very large volumes of medical data for research purposes. MRIs can be affected by various artefacts such as motion, noise or poor contrast that can severely degrade the overall quality of an image. In CDWs, a large amount of MRIs are unusable because corrupted by these diverse artefacts. Given the huge number of MRIs present in CDWs, manually detecting these artefacts becomes an impractical task. Therefore, it is necessary to develop an automated tool that can efficiently identify and exclude corrupted images. We previously proposed an approach for the detection of motion artefacts in 3D T1-weighted brain MRIs. In this paper, we propose to extend our work to two other types of artefacts: poor contrast and noise. We rely on a transfer learning approach, which leverages synthetic artefact generation, and comprises two steps: model pre-training on research data using synthetic artefacts, followed by a fine-tuning step, where we generalise the pre-trained models to clinical routine data relying on the manual labelling of 5000 images. The main objectives of our study were two-fold: to be able to exclude images with severe artefacts and to detect moderate artefacts. Our approach excelled in meeting the first objective, achieving a balanced accuracy of over 84% for the detection of severe noise and very poor contrast, which closely matched the performance of human annotators. Nevertheless, performance in the pursuit of the second objective was less satisfactory and inferior to that of the human annotators. Overall, our framework will be useful for taking full advantage of MRIs present in CDWs.

Subject head motion during the acquisition of diffusion-weighted imaging (DWI) of the brain induces artifacts and affects image quality. Information about the frequency and extent of motion could reveal which aspects of motion correction are most necessary. Therefore, we investigate the extent of translation and rotation among participants, and how the motion changes during the scan acquisition. We analyze 5,380 DWI scans from 1,034 participants. We measure the rotations and translations in the sagittal, coronal and transverse planes needed to align the volumes to the first and previous volumes, as well as the displacement. The different types of motion are compared with each other and compared over time. The largest rotation (per minute) is around the right – left axis (median 0.378 °/min, range 0.000 - 11.466°) and the largest translation (per minute) is along the anterior – posterior axis (median 1.867 mm/min, range 0.000 - 10.944 mm). We additionally observe that spikes in movement occur at the beginning of the scan, particularly in anterior – posterior translation. The results show that all scans are affected by subtle head motion, which may impact subsequent image analysis.

The establishment of automated image segmentation methods in medical imaging allows the analysis of very large datasets. However, visual quality control (QC) of segmentation results is impractical in large datasets, hence the need for automatic QC. In this paper, we introduce a novel automatic approach for QC of segmentation results. We developed a QC deep learning model (referred to as QC model) that, for a given patient, predicts the accuracy of the corresponding automatic segmentation (in our work the Dice score) provided by a deep learning segmentation model (referred to as segmentation model) in the absence of a ground truth annotation. To train the QC model, we introduce data augmentation by using the early epochs of the segmentation model. These early epochs allow us to feed the training of the QC model with examples of poor segmentation. We applied our approach to the QC of automatic segmentation of the choroid plexuses of the brain from MRI in controls and patients with multiple sclerosis. However, the method is generic and could be used with any segmentation model. The experiments showed that the proposed approach is very effective for predicting the segmentation accuracy with a correlation coefficient of 0.92, an R² of 0.763, a mean absolute error (MAE) of 0.078, and a mean squared error (MSE) of 0.009. Overall, this work shall provide a valuable tool for the automatic QC of segmentation results.

The reconstruction kernel in computed tomography (CT) generation determines the texture of the image. Consistency in reconstruction kernels is important as the underlying CT texture can impact measurements during quantitative image analysis. Harmonization (i.e., kernel conversion) minimizes differences in measurements due to inconsistent reconstruction kernels. Existing methods investigate harmonization of CT scans in single or multiple manufacturers. However, these methods require paired scans of hard and soft reconstruction kernels that are spatially and anatomically aligned. Additionally, a large number of models need to be trained across different kernel pairs within manufacturers. In this study, we adopt an unpaired image translation approach to investigate harmonization between and across reconstruction kernels from different manufacturers by constructing a multipath cycle generative adversarial network (GAN). We use hard and soft reconstruction kernels from the Siemens and GE vendors from the National Lung Screening Trial dataset. We use 50 scans from each reconstruction kernel and train a multipath cycle GAN. To evaluate the effect of harmonization on the reconstruction kernels, we harmonize 50 scans each from Siemens hard kernel, GE soft kernel and GE hard kernel to a reference Siemens soft kernel (B30f) and evaluate percent emphysema. We fit a linear model by considering the age, smoking status, sex and vendor and perform an analysis of variance (ANOVA) on the emphysema scores. Our approach minimizes differences in emphysema measurement and highlights the impact of age, sex, smoking status and vendor on emphysema quantification.

Reconstructing a tomographic image with sparse-view sampling is a major challenge in low-dose computed tomography. Recently, several studies have reported that deep-learning-based methods can reconstruct images of 512 × 512 pixels from 60-view X-ray projections without large artifacts. In this study, a U-Net variant with residual connections and attention gates is proposed for sparse-view computed tomography. A pair of the proposed U-Nets with a loss function based on the structural similarity index measure can be applied to synthesize sparse-view sampling sinograms and denoise reconstructed images. The experimental results indicate the performance of the proposed method is superior to those of other U-Net-based methods for fewer than 60 projection views. Experiments on a public data set of chest tomographic images validated that the proposed method can be used for COVID-19 identification.

Unsupervised anomaly detection is a popular approach for the analysis of neuroimaging data as it allows identifying a wide variety of anomalies from unlabelled data. It relies on reconstructing a subject-specific model of healthy appearance to which a subject’s image can be compared to detect anomalies. In the literature, it is common for anomaly detection to rely on analysing the residual image between the subject’s real image and its pseudo-healthy reconstruction. This approach however has limitations partly due to the pseudo-healthy reconstructions being imperfect and to the lack of natural thresholding mechanism. Our proposed method, inspired by Z-scores, leverages the healthy population variability to overcome these limitations. Our experiments conducted on 3D FDG PET scans from the ADNI database demonstrate the effectiveness of our approach in accurately identifying simulated Alzheimer’s disease related anomalies.

Deep models have been shown to tend to fit the target function from low to high frequencies (a phenomenon called the frequency principle of deep learning). One may hypothesize that such property can be leveraged for better training of deep learning models, in particular for segmentation tasks where annotated datasets are often small. In this paper, we exploit this property to propose a new training method based on frequency-domain disentanglement. It consists of three main stages. First, it disentangles the image into high- and low-frequency components. Then, the segmentation network model learns them separately (the approach is general and can use any segmentation network as backbone). Finally, feature fusion is performed to complete the downstream task. The method was applied to the segmentation of the red and dentate nuclei in Quantitative Susceptibility Mapping (QSM) data and to three tasks of the Medical Segmentation Decathlon (MSD) challenge under different training sample sizes. For segmenting the red and dentate nuclei and the heart, the proposed approach resulted in considerable improvements over the baseline (respectively between 8 and 16 points of Dice and between 5 and 8 points). On the other hand, there was no improvement for the spleen and the hippocampus. We believe that these intriguing results, which echo theoretical work on the frequency principle of deep learning, are of interest for discussion at the conference. The source code is publicly available at: https://github.com/GuanghuiFU/frequency_disentangled_learning.

The transformation model is an essential component of any deformable image registration approach. It provides a representation of physical deformations between images, thereby defining the range and realism of registrations that can be found. Two types of transformation models have emerged as popular choices: B-spline models and mesh models. Although both models have been investigated in detail, a direct comparison has not yet been made, since the models are optimized using very different optimization methods in practice. B-spline models are predominantly optimized using gradient-descent methods, while mesh models are typically optimized using finite-element method solvers or evolutionary algorithms. Multi-objective optimization methods, which aim to find a diverse set of high-quality trade-off registrations, are increasingly acknowledged to be important in deformable image registration. Since these methods search for a diverse set of registrations, they can provide a more complete picture of the capabilities of different transformation models, making them suitable for a comparison of models. In this work, we conduct the first direct comparison between B-spline and mesh transformation models, by optimizing both models with the same state-of-the-art multi-objective optimization method, the Multi-Objective Real-Valued Gene-pool Optimal Mixing Evolutionary Algorithm (MO-RV-GOMEA). The combination with B-spline transformation models, moreover, is novel. We experimentally compare both models on two different registration problems that are both based on pelvic CT scans of cervical cancer patients, featuring large deformations. Our results, on three cervical cancer patients, indicate that the choice of transformation model can have a profound impact on the diversity and quality of achieved registration outcomes.

The correct pose of the patient during radiography is of critical importance to ensure an adequate diagnostic quality of radiographs, which are the basis for diagnosis and treatment planning. However, correct patient positioning is not a standardized process, often resulting in inadequate radiographs and repeated radiation exposure. We propose a novel approach using Time-of-Flight cameras to assess the patient’s pose and therefore predict the expected diagnostic quality of the radiograph, before it is even captured. As a first step towards this goal, we acquired a new dataset, consisting of depth images and corresponding radiographs of the ankle using two anatomical preparations in multiple poses. The radiographs were labeled by radiologists for their diagnostic quality related to the patient’s pose. These labels serve as quality label for the corresponding pose. Using this dataset we trained deep neural networks and were able to correctly assess the diagnostic quality of a pose with a mean accuracy of up to 90.2%, demonstrating that shared features for pose assessment across patients exist and can be learned.

Cone-beam computed tomography (CBCT) is a widely established technique for dental imaging that provides detailed three-dimensional representations of the maxillofacial region. However, its application in clinical practice is often impaired by patient movement during image acquisition, leading to severe image artifacts and compromised diagnostic accuracy. Cone beam CT system manufacturers have incorporated mechanical fixations to prevent patient movement, however, these fixtures lack complete rigidity and still permit some degree of movement. In clinical situations, short scan acquisitions are a common scenario where only a limited portion of the full 360° rotation is covered during the scan. A shorter scan has the advantage of a shorter exposure time and requires a device of smaller size which is very convenient due to space constraints in many dentist’s clinics. This paper proposes a multi-stage approach to compensate for patient motion in short-scan CBCT reconstructions. In contrast to conventional iterative techniques that require multiple reconstructions during compensation, this approach requires only a few reconstructions computed with the Feldkamp-Davis-Kress (FDK) algorithm as the reference for compensation. Before the motion compensation stage, the reference volume undergoes regularization. The motion compensation is then accomplished by optimizing motion parameters through a regularized 3D-2D image registration process. The results show that the proposed motion compensation approach effectively reduces motion-induced artifacts in the final reconstruction.

Self-supervised learning (SSL) has become a crucial approach for pre-training deep learning models in natural and medical image analysis. However, applying transformations designed for natural images to three-dimensional (3D) medical data poses challenges. This study explores the efficacy of specific augmentations in the context of self-supervised pre-training for volumetric medical images. A 3D non-contrastive framework is proposed for in-domain self-supervised pre-training on 3D gray-scale thorax CT data, incorporating four spatial and two intensity augmentations commonly used in 3D medical image analysis. The pre-trained models, adapted versions of ResNet-50 and Vision Transformer (ViT)-S, are evaluated on lung nodule classification and lung tumor segmentation tasks. The results indicate a significant impact of SSL, with a remarkable increase in AUC and DSC as compared to training from scratch. For classification, random scalings and random rotations play a fundamental role in achieving higher downstream performance, while intensity augmentations show limited contribution and may even degrade performance. For segmentation, random intensity histogram shifting enhances robustness, while other augmentations have marginal or negative impacts. These findings underscore the necessity of tailored data augmentations within SSL for medical imaging, emphasizing the importance of task-specific transformations for optimal model performance in complex 3D medical datasets.

The Segment Anything Model (SAM) has drawn significant attention from researchers who work on medical image segmentation because of its generalizability. However, researchers have found that SAM may have limited performance on medical images compared to state-of-the-art non-foundation models. Regardless, the community sees potential in extending, fine-tuning, modifying, and evaluating SAM for analysis of medical imaging. An increasing number of works have been published focusing on the mentioned four directions, where variants of SAM are proposed. To this end, a unified platform helps push the boundary of the foundation model for medical images, facilitating the use, modification, and validation of SAM and its variants in medical image segmentation. In this work, we introduce SAMM Extended (SAMME), a platform that integrates new SAM variant models, adopts faster communication protocols, accommodates new interactive modes, and allows for fine-tuning of subcomponents of the models. These features can expand the potential of foundation models like SAM, and the results can be translated to applications such as image-guided therapy, mixed reality interaction, robotic navigation, and data augmentation.

Glioblastoma (GBM) is most aggressive and common adult brain tumor. The standard treatments typically include maximal surgical resection, followed adjuvant radiotherapy and chemotherapy. However, the efficacy of these treatment is often limited, as tumor often infiltrate into the surrounding brain tissue, often extending beyond the radiologically defined margins. This infiltration contributes to the high recurrence rate and poor prognosis associated with GBM patients, necessitating advanced methods for early and accurate detection of tumor infiltration. Despite the great promise traditional supervised machine learning shows in predicting tumor infiltration beyond resectable margins, these methods are heavily reliant on expert-drawn Regions of Interest (ROIs), which are used to construct multi-variate models of different Magnetic Resonance (MR) signal characteristics associated with tumor infiltration. This process is both time consuming and resource intensive. Addressing this limitation, our study proposes a novel integration of fully automatic methods for generating ROIs with deep learning algorithms to create predictive maps of tumor infiltration. This approach uses pre-operative multi-parametric MRI (mpMRI) scans, encompassing T1, T1Gd, T2, T2-FLAIR, and ADC sequences, to fully leverage the knowledge from previously drawn ROIs. Subsequently, a patch based Convolutional Neural Network (CNN) model is trained on these automatically generated ROIs to predict areas of potential tumor infiltration. The performance of this model was evaluated using a leave-one-out cross-validation approach. Generated predictive maps binarized for comparison against post-recurrence mpMRI scans. The model demonstrates robust predictive capability, evidenced by the average cross-validated accuracy of 0.87, specificity of 0.88, and sensitivity of 0.90. Notably, the odds ratio of 8.62 indicates that regions identified as high-risk on the predictive map were significantly more likely to exhibit tumor recurrence than low-risk regions. The proposed method demonstrates that a fully automatic mpMRI analysis using deep learning can successfully predict tumor infiltration in peritumoral region for GBM patients while bypassing the intensive requirement for expert-drawn ROIs.

Unlike for other brain tumors, there has been little work on the automatic segmentation of primary central nervous system (CNS) lymphomas. This is a challenging task due the highly variable pattern of the tumor and its boundaries. In this work, we propose a new loss function that controls border irregularity for deep learning-based automatic segmentation of primary CNS lymphomas. We introduce a border irregularity loss which is based on the comparison of the segmentation and it smoothed version. The border irregularity loss is combined with a previously proposed topological loss to better control the different connected components. The approach is general and can be used with any segmentation network. We studied a population of 99 patients with primary CNS lymphoma. 40 patients were isolated from the very beginning and formed the independent test set. The segmentations were performed on post-contrast T1-weighted MRI. The MRI were acquired in clinical routine and were highly heterogeneous. The proposed approach substantially outperformed the baseline across the various evaluation metrics (by 6 percent points of Dice, 40mm of Hausdorff distance and 6mm of mean average surface distance). However, the overall performance was moderate, highlighting that automatic segmentation of primary CNS lymphomas is a difficult task, especially when dealing with clinical routine MRI. The code is publicly available here: https://github.com/rosanajurdi/LymphSeg.

Selective internal radiation therapy (SIRT) is a targeted treatment for liver tumors, particularly hepatocellular carcinoma (HCC), that involves the precise delivery of radioactive microspheres to the tumor’s blood supply. Image registration is critical to SIRT, ensuring precise alignment of diagnostic images with the treatment plan to optimize microsphere delivery and minimize side effects. In the context of SIRT, our goal was to develop a fully-automatic hybrid registration pipeline, using liver segmentation masks from a 3D UNet model to achieve performance comparable to expert registrations. The pipeline combines conventional and deep learning methods for automatic alignment of pre-treatment magnetic resonance images (MRI) on single-photon emission computed tomography (SPECT)/computed tomography (CT) images. This hybrid pipeline, which uses conventional global rigid registration and a deep learning-based approach for local deformation, outperformed conventional and manual expert registration. Quantitative assessment on a dataset of 69 HCC patients showed an improved Dice similarity coefficient (DSC) of 0.928 compared to 0.917 with the conventional methods. A subset analysis of 61 patients with expert registrations showed a mean DSC of 0.922, while our proposed method remained at a mean DSC of 0.928. These results demonstrate the effectiveness of our hybrid approach in achieving accurate liver registration, which is critical for precise microsphere delivery during SIRT. The improvement over conventional methods highlights the potential of incorporating deep learning techniques into multimodal liver registration, thereby improving the overall quality and effectiveness of SIRT in the management of HCC. Our method provides clinicians with a reliable and automated registration pipeline that can positively optimize treatment planning and reduce the burden of manual registration. As a result, our hybrid approach holds promise for more accurate and precise registration results.

Useful first order delta radiomics can be calculated after each lung SBRT radiotherapy treatment using CBCTs routinely acquired for image guidance only under certain conditions whereas retrospective use of unsorted data may be inconsistent and inconclusive. Patients can be classified as having either minimal, moderate, or large density changes after each treatment fractions and with further investigation these categories may then be associated with treatment outcome and complication probability. Sensitivity and consistency checks were performed to validate results for various examples of interesting findings within our sample data set. Detailed first order radiomic features were computed for demonstration purposes. From our patient data base we selected 10 cases for analysis. A majority of cases were disqualified due to the presence of fiducials which were required for IGRT and gated treatments, proximity to other irradiated targets, and PTV included non-lung tissue. Intrafraction and Interfraction reproducibility was excellent. The most interesting finding appears a case with significant density changes after each treatment fraction. The unique target classification such as spiculation, concurrent chemotherapy, stage, or lung cancer type is the subject of continued investigation. Successful first order delta radiomics indicate that clinically relevant higher order features can also be calculated in selected cases.

The lack of general screening for thoracic aortic aneurysm (TAA) compared to abdominal aortic aneurysm is in part due to the higher costs of CT scan or MRI compared to abdominal ultrasound. Here, we introduce a deep learning method to easily detect TAA from a less expensive imaging modality: a single chest x-ray. We use a two-step approach. First, we use image-to-image translation using conditional generative adversarial network to separate the tissue layers (e.g. bone, lung, ascending/descending aorta). Next, we apply a skeletonization algorithm to compute the diameters of the ascending and descending aorta. We validate this method on a paired CXR and CT dataset and find high agreement between the computed diameter from the CXR and the true diameter from the CT scan. The mean absolute error for all cross section diameters and maximum diameters is 3.58mm and 1.97mm, respectively. In detecting dilation (>35mm), our method achieves an F1 score of 0.87 and Cohen's kappa of 0.75, showing high agreement.

Arterial spin labeling (ASL) perfusion MRI is the only non-invasive imaging technique for quantifying regional cerebral blood flow (CBF), which is a fundamental physiological variable. ASL MRI has a relatively low signal-to-noise-ratio (SNR). In this study, we proposed a novel ASL denoising method by simultaneously exploiting the inter- and intra-receive channel data correlations. MRI including ASL MRI data have been routinely acquired with multichannel coils but current denoising methods are designed for denoising the coil-combined data. Indeed, the concurrently acquired multi-channel images differ only by coil sensitivity weighting and random noise, resulting in a strong low-rank structure of the stacked multi-channel data matrix. In our method, this matrix was formed by stacking the vectorized slices from different channels. Matrix rank was then approximately measured through the logarithm-determinant of the covariance matrix. Notably, our filtering technique is applied directly to complex data, avoiding the need to separate magnitude and phase or divide real and imaginary data, thereby ensuring minimal information loss. The degree of low-rank regularization is controlled based on the estimated noise level, striking a balance between noise removal and texture preservation. A noteworthy advantage of our framework is its freedom from parameter tuning, distinguishing it from most existing methods. Experimental results on real-world imaging data demonstrate the effectiveness of our proposed approach in significantly improving ASL perfusion quality. By effectively mitigating noise while preserving important textural information, our method showcases its potential for enhancing the utility and accuracy of ASL perfusion MRI, paving the way for improved neuroimaging studies and clinical diagnoses.

Tumor vasculature varies widely among patients and is an important factor in disease progression and treatment response. Characterizing tumor fluid flows, often using dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI), is therefore a key component of disease assessment. In our work, we applied a computational fluid dynamical model we have developed, called the unbalanced regularized optimal mass transport (urOMT) method, to quantify and visualize fluid flow behaviors in breast tumors. Unlike the popular Tofts model, the urOMT model includes cross-voxel transport by solving an advection-diffusion equation. The urOMT outputs can reveal time-varying changes of physical transport properties at a local voxel level and can also visualize directional trend of the cross-voxel flows. Results for DCE-MRI studies of ten breast cancer patients each at four time points during neoadjuvant chemotherapy indicate that urOMT-produced metrics, flux, influx and efflux are potentially valuable biomarkers for evaluating therapeutic responses.

Automated cellular nuclei segmentation is often an important step for digital pathology and other analyses such as computer aided diagnosis. Most existing machine learning methods for microscopy image analysis require postprocessing such as watershed transform or connected component analysis to obtain instance segmentation from semantic segmentation results. This becomes prohibitively expensive computationally especially when used with 3D microscopy volumes. UNet Transformers for Instance Segmentation (UNETRIS) is proposed to eliminate the postprocessing steps necessary for nuclei instance segmentation in 3D microscopy images. UNETRIS, an extension of UNETR which utilizes a transformer as the encoder in the successful “U-shaped” network design for the encoder-decoder structure of U-Net, uses additional transformers to separate individual instances of cell nuclei directly during the inference step without the need for expensive postprocessing steps. UNETRIS does not require but can use manual ground truth annotations for training. UNETRIS was tested on a variety of microscopy volumes collected from multiple regions of organ tissues.

Human capacity for "lifelong learning" encompasses a continuous process of acquiring knowledge, adapting to new environments, and developing new skills throughout one's life. In order to bridge the gap between human intelligence and artificial intelligence, an increasing number of researchers have begun to explore the concept of lifelong learning within the field of machine learning, also referred as continuous learning or incremental learning. Incremental learning enables machine to learn from a continuous stream of data, thereby achieving lifelong learning capability. Incremental learning can be categorized into three types based on the different data batch and task settings: Task Incremental Learning (TIL), Domain Incremental Learning (DIL), and Class Incremental Learning (CIL). These scenarios describe the incremental learning challenges within supervised learning and encompass the majority of incremental learning settings. However, there is currently no unified and specific definition for incremental learning situations with the same task and data distribution while in a continuous information flow. Therefore, in this paper, we concentrated on the situation defined as data incremental learning, where all training samples belong to the same task and share the same data distribution. Currently, there is limited in-depth research on data incremental learning. Hence, this study conducts a preliminary experimental exploration of data incremental learning characteristics within the context of image classification task. Moreover, it reviews existing strategies utilizing deep neural networks for incremental learning and compares these methods in the context of data increment learning, offering insights for future research and exploration in the field.

Connectivity matrices derived from diffusion MRI (dMRI) provide an interpretable and generalizable way of understanding the human brain connectome. However, dMRI suffers from inter-site and between-scanner variation, which impedes analysis across datasets to improve robustness and reproducibility of results. To evaluate different harmonization approaches on connectivity matrices, we compared graph measures derived from these matrices before and after applying three harmonization techniques: mean shift, ComBat, and CycleGAN. The sample comprises 168 agematched, sex-matched normal subjects from two studies: the Vanderbilt Memory and Aging Project (VMAP) and the Biomarkers of Cognitive Decline Among Normal Individuals (BIOCARD). First, we plotted the graph measures and used coefficient of variation (CoV) and the Mann-Whitney U test to evaluate different methods’ effectiveness in removing site effects on the matrices and the derived graph measures. ComBat effectively eliminated site effects for global efficiency and modularity and outperformed the other two methods. However, all methods exhibited poor performance when harmonizing average betweenness centrality. Second, we tested whether our harmonization methods preserved correlations between age and graph measures. All methods except for CycleGAN in one direction improved correlations between age and global efficiency and between age and modularity from insignificant to significant with p-values less than 0.05.

Morphological abnormalities in biological cell nuclei are used as essential features for diagnosing diseases, determining cell cycle stages, and conducting other fundamental cell biological research. While many deep learning approaches have been proposed for segmenting normal elliptical nuclei, less work has been done on segmenting abnormally shaped nuclei. One issue is that acquiring a significant number of annotated nuclei data poses a challenge for many deep learning segmentation methods, particularly due to the generally high cost associated with obtaining annotated data. The lack of the use of shape analysis is another problem that causes segmentation to not perform well for abnormally shaped nuclei. To address these problems, we propose a system to segment abnormally shaped nuclei with limited training data. We generate synthetic ground truth images of abnormally shaped nuclei to supplement the limited amount of training images available. Six Mask R-CNNs are trained to segment abnormally shaped nuclei. We then introduce an ensemble strategy, known as Weighted Mask Fusion, to combine the segmentation results from the six Mask R-CNNs. We describe a shape analysis step, based on a convexity measure, that is used on the segmented nuclei in the ensemble result to further improve the segmentation performance. Our proposed system is compared with other segmentation methods for abnormally shaped nuclei. The evaluation demonstrates the effectiveness of the ensemble processing, fusion, and convexity measures for segmenting abnormally shaped nuclei.

Diffusion Tensor Cardiac Magnetic Resonance (DT-CMR) is the only in vivo method to non-invasively examine the microstructure of the human heart. Current research in DT-CMR aims to improve the understanding of how the cardiac microstructure relates to the macroscopic function of the healthy heart as well as how microstructural dysfunction contributes to disease. To get the final DT-CMR metrics, we need to acquire diffusion weighted images of at least 6 directions. However, due to DWI’s low signal-to-noise ratio, the standard voxel size is quite big on the scale for microstructures. In this study, we explored the potential of deep-learning-based methods in improving the image quality volumetrically (×4 in all dimensions). This study proposed a novel framework to enable volumetric super-resolution, with an additional model input of high-resolution b0 DWI. We demonstrated that the additional input could offer higher super-resolved image quality. Going beyond, the model is also able to super-resolve DWIs of unseen b-values, proving the model framework’s generalizability for cardiac DWI super-resolution. In conclusion, we would then recommend giving the model a high-resolution reference image as an additional input to the low-resolution image for training and inference to guide all super-resolution frameworks for parametric imaging where a reference image is available.

In this paper, we present ClinicaDL, an open-source software platform that aims at enhancing the reproducibility and rigor of research for deep learning in neuroimaging. We first provide an overview of the software platform and then focus on recent advances. Features of the software aim at addressing three key issues in the field: the lack of reproducibility, the methodological flaws that plague many published studies and the difficulties using neuroimaging datasets for people with little expertise in this application area. Key existing functionalities include automatic data splitting, checking for data leakage, standards for data organization and results storing, continuous integration and integration with Clinica for preprocessing, amongst others. The most prominent recent features are as follows. We now provide various data augmentation and synthetic data generation functions (both standard and advanced ones including motion and hypometabolism simulation). Continuous integration test data are now versioned using DVC (data version control). Tools for generating validation splits have been made more generic. We made major improvements regarding usability and performance. We now support multi-GPU training and automatic mixed precision (to exploit tensor cores). We created a graphical interface to easily generate training specifications. We allow tracking of experiments through standard tools (MLflow, Weights&Biases). We believe that ClinicaDL can contribute to enhance the trustworthiness of research in deep learning for neuroimaging. Moreover, its functionalities and coding practices may serve as inspiration for the whole medical imaging community, beyond neuroimaging.

The field of systems neuroscience strives to comprehend the intricate links between brain activity and behavior, with the goal of understanding neurological disorders and targeted interventions, notably in feedback-based therapies like Deep Brain Stimulation (DBS). However, challenges persist in correlating neuroimaging data with behavior, especially in freely behaving animals and across different imaging modalities. In the last decade, deep learning has emerged as a crucial tool for analyzing image data and enabling the correlation or prediction from different imaging modalities. This study investigates the application of a sophisticated deep learning architecture, specifically a pre-trained ResNet50-BiLSTM model, to predict behaviors from neuroimaging data. This study acquired multicontrast neuroimaging data (fluorescence or FL, intrinsic optical signal or IOS, and laser speckle contrast or LSC) synchronized with behavioral data from five awake, healthy mice. Behavioral "syllables" (i.e. running, nest building/eating, and minimally mobile) or classes were annotated using Behavior Cloud software. Neural activity (i.e. FL channel) image sequences were associated with behavioral syllables, consisting of 12 image frames spanning one minute. A pre-trained ResNet50-BiLSTM architecture was used for image sequence classification. We added a fully connected layer after BiLSTM and fine-tuned it to compute class scores. Since we had only five mice, we evaluated the model’s performance and generalizability using a 5-fold cross-testing strategy. Test accuracy was utilized to assess the reliability and robustness of predictions, and performance evaluation. Generalizability results showcase the promising performance of the ResNet50-BiLSTM model for predicting behavior from neural activity images with test accuracy reaching 93.75%. We intend to conduct additional research on this pipeline’s generalizability by analyzing confusion matrices, ROC curves, and AUCs, within a One-vs-All (OvA) framework. This initial study lays the groundwork for the exploration of multi-modal approaches (e.g. vision transformers and autoencoders) that encompass different neuroimaging modalities for accurate behavior prediction.

Among patients with early-stage non-small cell lung cancer (NSCLC) undergoing surgical resection, identifying who is at high risk of recurrence can inform clinical guidelines with respect to more aggressive follow-up and/or adjuvant therapy. While predicting recurrence based on pre-surgical resection data is ideal, clinically important pathological features are only evaluated postoperatively. Therefore, we developed two supervised classification models to assess the importance of pre- and post-surgical features for predicting 5-year recurrence. An integrated dataset was generated by combining clinical covariates and radiomic features calculated from pre-surgical computed tomography images. After removing correlated radiomic features, the SHapley Additive exPlanations (SHAP) method and recursive feature elimination (RFE) were used for feature selection. A support vector machine (SVM) classification model was then trained to predict recurrence. We demonstrate that the post-surgical model outperforms the pre-surgical model in predicting lung cancer recurrence, with tumor pathological features and peritumoral radiomic features contributing significantly to the model’s performance.

In this study, we investigated the feasibility of using masked auto-encoder (MAE) for sinogram inpainting to reduce a streak artifact in sparse view CT, taking their successful application in image inpainting tasks. To handle the sparse view sinogram, the flattener operator, which flattened the image patch into a token, was modified to handle the 1D image patches. We compared the artifact reduction performance for trained MAE using random sampling (i.e., MAE (Random)) and periodic sparse view sampling (i.e., MAE (Sparse)). To generate the training and validation samples, a software phantom consisting of multiple simple figures was used for the CT simulation with Siddon algorithm and filtered back-projection algorithm. To evaluate the performance of MAE for streak artifact reduction in sparse view CT, we implemented RED-CNN and compared the streak artifact reduction performance of RED-CNN and MAE. SSIM and PSNR were used to quantitatively measure the performance of MAE and RED-CNN. MAE provided better performance for streak artifact reduction in sparse view CT than conventional deep-learning-based artifact reduction technique (i.e., RED-CNN). MAE with random view sampling showed better performance than with sparse view CT images. We expect that MAE for medical imaging could be applied in other fields of medical imaging study.

Metal artifact reduction (MAR) is crucial for improving the quality of dental cone-beam computed tomography (CBCT) images. However, inaccurate extraction of metal in MAR can lead to incomplete suppression of metal artifacts, obscuring adjacent bone structures in resultant images. The pre-existing metal artifacts such as streaks, shadowing, and cupping complicate the extraction of metallic segments. In particular, dental CBCT applications are vulnerable to metal artifacts due to the lack of anti-scatterers and the frequent appearance of metallic restorations. Conventional image processing methods for metal extraction have some limitations, for example, relying on predefined parameters and requiring manual interventions. To overcome these challenges, we introduce a deep learning-based metal extraction method in an unsupervised manner, eliminating the labor-intensive label annotation process. This method combines several methods, including unpaired image-to-image translation, weakly supervised learning, and geometric transformations, followed by supervised learning for real-time metal extraction. As a result, the proposed method achieves high-quality metal extraction with an accuracy of 90%, as evaluated with manual annotations, and shows a significant improvement in MAR performance for clinical CBCT images compared to our conventional method.

This study assessed automated bone density measurement technologies in pediatric groups, focusing on lumbar spine localization and spine segmentation models initially trained on adult data. The research involved three phases: training models using YOLOv5 and U-Net on adult images, adapting these models with pediatric data via transfer learning, and external validation categorized by age to account for anatomical variances. The adult-trained model showed decreased sensitivity in younger ages, with the lowest performance in the youngest group. Conversely, the pediatric-trained model achieved high sensitivity, over 90% in children under 10, and perfect scores in the 10-12 group, demonstrating improved accuracy. Qualitative analysis for segmentation indicated better performance in the pediatric model across all age groups, particularly in those under 13. The study concludes that transfer learning enhances the performance and generalizability of models for pediatric spine analysis, suggesting a potential for more accurate diagnostics.

Lung cancer has both high incidence and mortality rates compared to other cancer types. One important factor for improved patient survival is early detection. Deep learning for lung nodule detection has been extensively studied, as a tool to facilitate clinicians with early nodule detection and classification. Many publications are reporting high detection accuracy and several models have been introduced to clinical practice. However, certain models may have reduced performance in real-world clinical practice. In this study, we introduce a method to assess the robustness of lung nodule detection models. Medically relevant image perturbations are used to assess the robustness of these models. The perturbations include noise and motion perturbations, which have been created in consultation with an expert radiologist to ensure the clinical relevance of the artifacts for thoracic computed tomography (CT) scans. The evaluated models demonstrate robustness to clinically relevant noise simulations, but it shows less resilience to motion artifacts in perturbed CT scans. This robustness evaluation method, incorporating simulated relevant artifacts, can be extended for use in other applications involving the analysis of CT scans.

Prostate cancer is a significant contributor to cancer-related deaths in men. Detecting prostate cancer early can greatly increase the likelihood of successful treatment. However, detecting and assessing prostate lesions from multiparametric magnetic resonance images (MRI) is time-consuming and variable across radiologists with different levels of experience. We present an integrated framework for segmenting and classifying prostate lesions from MRI. The proposed approach is in contrast with most existing automated prostate analysis approaches, which treat segmentation and classification of prostate lesions as two separate tasks with no interactions between them. In the proposed framework, preliminary lesion boundaries were first segmented from T2-weighted (T2W) and diffusion-weighted images (DWI) by a three-stream network. The region of interest (ROI) enclosing the segmented lesion was fed to a weakly supervised classification network, which predicted the Gleason grade of the lesion and provided the class activation maps (CAMs) corresponding to multiple MRI modalities. Finally, MR images of different modalities with the corresponding CAMs were fed to a six-stream network to generate an enhanced lesion mask. Our experiments showed that CAMs generated by the proposed weakly supervised classifier improved segmentation performance. Our proposed method has a great potential to improve the accuracy and efficiency of prostate MRI interpretation workflow.

Purpose: Neuronavigation has become integral to neurosurgery. By helping neurosurgeons localize their surgical instruments according to the patient’s anatomy in real time, neuronavigation facilitates improved patient outcomes and lower procedure costs. This technology has been out of reach of surgeons in lower resource settings due to cost and infrastructure limitations. Our group has developed NousNav, an open-source low-cost navigation system accessible in such environments to address this need. Neuronavigation requires skin surface segmentation from preoperative images, which is necessary for image-to-patient registration. Thus, accurate skin segmentation is essential for effective neuronavigation. However, manual skin surface segmentation is time-consuming, reliant on technical expertise, and susceptible to inter-rater variability. We propose to implement deep learning into the skin surface segmentation step of NousNav, mitigating the limitations of manual segmentation.

Approach: We built a nnU-Net model to automate skin segmentation, training it on 36 healthy brain T1-FLASH MRIs and testing it on another 5 healthy brain MRIs. We evaluated the model’s performance versus that of three human raters, using Hausdorff distance (in mm) and Dice metrics.

Results: On average, comparing the human raters to each other and the automated segmentation model to the human raters yielded Hausdorff distances of 1.3534mm and 1.0198mm and Dice scores of 0.9750 and 0.9820, respectively, demonstrating that automated skin surface segmentation performs comparably to manual segmentation.

Conclusions: Here we uniquely implemented automated segmentation for meeting the unmet need of accessible, open-source neuronavigation systems. Considering the promising performance of our deep learning model, this step may be feasibly incorporated into the NousNav workflow.

Diffusion MRI (dMRI) streamline tractography, the gold standard for in vivo estimation of brain white matter (WM) pathways, has long been considered indicative of macroscopic relationships with WM microstructure. However, recent advances in tractography demonstrated that convolutional recurrent neural networks (CoRNN) trained with a teacherstudent framework have the ability to learn and propagate streamlines directly from T1 and anatomical contexts. Training for this network has previously relied on high-resolution dMRI. In this paper, we generalize the training mechanism to traditional clinical resolution data, which allows generalizability across sensitive and susceptible study populations. We train CoRNN on a small subset of the Baltimore Longitudinal Study of Aging (BLSA), which better resembles clinical protocols. Then, we define a metric, termed the epsilon ball seeding method, to compare T1 tractography and traditional diffusion tractography at the streamline level. Under this metric, T1 tractography generated by CoRNN reproduces diffusion tractography with approximately two millimeters of error.

Optical coherence tomography (OCT) established as an essential part of the diagnosis, monitoring and treatment programs of patients suffering from wet age-related macular degeneration (AMD). To further improve disease progression monitoring and just-in-time therapy, home OCTs such as the innovative self-examination low-cost full-field OCT (SELFF-OCT) are developed, enabling self-examination by patients due to its technical simplicity and cost efficiency, but coming at the cost of reduced image quality indicated by a low signal-to-noise ratio (SNR). Although deep learning denoising methods based on convolutional neural networks (CNN) or generative adversarial networks (GAN) achieve state-of-the-art denoising performance in improving the SNR for better image interpretability, they usually require noise-free images for training, which are not available for OCT imaging or can only be approximated by repeated scanning followed by complex and error-prone registration and multi-frame averaging processes. To circumvent this drawback, the denoising approach proposed in this work is based on utilizing paired SELFF-OCT images acquired from the retina of an artificial eye to train a Noise2Noise (N2N) network by repeatedly mapping one noisy image to another noisy realization of the same image. The performance of the proposed approach is evaluated by denoising unseen SELFF-OCT images from the artificial eye as well as real human eyes, utilizing standard image quality assessment (IQA) metrics as well as non-reference quality metrics. Qualitative and quantitative results of the evaluation verify the effectiveness of the proposed N2N approach by an improved SNR, while important structure information is preserved. Furthermore, the results reveal a superior denoising performance of the proposed approach compared to the application of conventional denoising methods.

Supine and prone colon registration is challenging due to the change in the patient’s position and the consequent severe distortion of the colon shape. We propose a novel method for colon registration in the cylinder domain. Our method uses Teichm¨uller Map (T-Map) to match the prescribed feature points in supine and prone positions. In addition, we propose a scheme of virtual boundaries for categorizing feature points as constraints for T-Map. We validate our proposed method on a dataset of 40 colon data and demonstrate the superiority of our method over the planar domain-based method.

While chronological age is based on time elapsed from the time of the birth of a person, cellular or biological age (BA) is a measure of gradual decline in cell division and proper functioning. BA may be affected by intrinsic factors like genetic changes and cellular metabolism as well as extrinsic factors like exposure to environmental toxins, lifestyle, and diet [1][2][3]. While chronological age is often used as a crude estimate of BA to assess risk of mortality and morbidity, BA can provide a more accurate risk estimate. Even with the lack of a universally agreed upon definition and quantification of BA, laboratory test based BA estimation measures have been well-established [4][5][6]. Biological age related markers may be incidentally collected in medical data such as physical activity from wearable devices[7][8] and medical imaging data [9][10][11][12][13][14][15][16]. Age-related changes in bone density[17][18], body composition[19][20][21], cardiovascular structures[22][23], lungs[24][25], and ocular structures[26][27] have been demonstrated on imaging. Deep learning based image processing models have recently been designed to quantify such biomarkers for biological age estimation using magnetic resonance imaging of the brain[28][29], retinal photograph[30][31], and chest X-rays[32]. “Brain age” has been established as indicative of cognitive decline and mortality[28][29]. Degree of atrophy in brain tissue captured in radiological images such as brain MR and head CT, can indicate biological age of a person which may be different from chronological age. Biological age estimation studies have often connected risk of mortality and morbidity with larger gaps between chronological and biological age[30][31]. While this gap may be caused to some extent by intrinsic factors, social determinants of health (SDoH) can influence biological age, accelerating cellular aging and overall health outcomes[33]. Electronic health records (EHR) collected for patients in hospitals may provide clues to social determinants of health but cannot guarantee quantification of its influence. Objective of the current study is to understand the relation between biological age calculated using imaging features derived from head CT studies and social determinants of health. Our hypothesis is that higher biological age than chronological age primarily represents poor quality of life including unhealthy diet, stress, exposure to environmental toxins, sedentary lifestyle, etc. Social deprivation index (SDI) is a composite measure of seven demographic characteristics collected in the American Community Survey (ACS) to quantify social determinants based on geolocation: percent living in poverty, percent with less than 12 years of education, percent single-parent households, the percentage living in rented housing units, the percentage living in the overcrowded housing unit, percent of households without a car, and percentage unemployed adults under 65 years of age. SDI can address challenges in quantifying social determinants of health to best guide clinical and community health interventions. The SDI measure was calculated at the four geographic areas: county, census tract, aggregated Zip Code Tabulation Area (ZCTA), and Primary Care Service Area (PCSA, v 3.1). While raw values are directly computed by the formula using the seven measures described above, SDI score is a value normalized between 0- 100 with higher value indicating higher extent of social disadvantage. We use SDI score as a surrogate for social determinant of health and study the relation between gaps in biological age estimated by our deep learning-based imaging model and chronological age of the patient at the time of the imaging exam. In order to prove our hypothesis, we adopted two parallel experiments - (i) (post-processing) establish the correlation between model computed biological age and SDI; (ii) (in-processing) include SDI during the learning of the model which may boost the model performance for biological age by providing surrogate for social determinant of the patient.

Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease diagnosis due to its ability to provide complementary information by analyzing a relationship between multimodal images on the same subject. Acquiring all MR modalities, however, can be expensive, and, during a scanning session, certain MR images may be missed depending on the study protocol. The typical solution would be to synthesize the missing modalities from the acquired images such as using generative adversarial networks (GANs). Yet, GANs constructed with convolutional neural networks (CNNs) are likely to suffer from a lack of global relationships and mechanisms to condition the desired modality. To address this, in this work, we propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features using the modality infuser. Furthermore, the modality infuser captures long-range relationships among all brain structures, leading to the generation of more realistic images. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities, and our experimental results demonstrate the superiority of our proposed method in terms of synthesis quality. In addition, we conducted experiments on a brain tumor segmentation task and different conditioning methods.

Imaging findings inconsistent with those expected at specific chronological age ranges may serve as early indicators of neurological disorders and increased mortality risk. Estimation of chronological age, and deviations from expected results, from structural magnetic resonance imaging (MRI) data has become an important proxy task for developing biomarkers that are sensitive to such deviations. Complementary to structural analysis, diffusion tensor imaging (DTI) has proven effective in identifying age-related microstructural changes within the brain white matter, thereby presenting itself as a promising additional modality for brain age prediction. Although early studies have sought to harness DTI’s advantages for age estimation, there is no evidence that the success of this prediction is owed to the unique microstructural and diffusivity features that DTI provides, rather than the macrostructural features that are also available in DTI data. Therefore, we seek to develop white-matter-specific age estimation to capture deviations from normal white matter aging. Specifically, we deliberately disregard the macrostructural information when predicting age from DTI scalar images, using two distinct methods. The first method relies on extracting only microstructural features from regions of interest (ROIs). The second applies 3D residual neural networks (ResNets) to learn features directly from the images, which are nonlinearly registered and warped to a template to minimize macrostructural variations. When tested on unseen data, the first method yields mean absolute error (MAE) of 6.11 ± 0.19 years for cognitively normal participants and MAE of 6.62 ± 0.30 years for cognitively impaired participants, while the second method achieves MAE of 4.69 ± 0.23 years for cognitively normal participants and MAE of 4.96 ± 0.28 years for cognitively impaired participants. We find that the ResNet model captures subtler, non-macrostructural features for brain age prediction.

Vestibular Schwannoma is a benign brain tumour that grows from one of the balance nerves. Patients may be treated by surgery, radiosurgery or with a conservative ”wait-and-scan” strategy. Clinicians typically use manually extracted linear measurements to aid clinical decision making. This work aims to automate and improve this process by using deep learning based segmentation to extract relevant clinical features through computational algorithms. To the best of our knowledge, our study is the first to propose an automated approach to replicate local clinical guidelines. Our deep learning based segmentation provided Dice-scores of 0.8124 ± 0.2343 and 0.8969 ± 0.0521 for extrameatal and whole tumour regions respectively for T2 weighted MRI, whereas 0.8222 ± 0.2108 and 0.9049 ± 0.0646 were obtained for T1 weighted MRI. We propose a novel algorithm to choose and extract the most appropriate maximum linear measurement from the segmented regions based on the size of the extrameatal portion of the tumour. Using this tool, clinicians will be provided with a visual guide and related metrics relating to tumour progression that will function as a clinical decision aid. In this study, we utilize 187 scans obtained from 50 patients referred to a tertiary specialist neurosurgical service in the United Kingdom. The measurements extracted manually by an expert neuroradiologist indicated a significant correlation with the automated measurements (p < 0.0001). Our code is publicly available at https: //github.com/navodini/AutomatedReportGenerationVS.

Medical image registration aims to identify the spatial deformation between images of the same anatomical region and is fundamental to image-based diagnostics and therapy. To date, the majority of the deep learning-based registration methods employ regularizers that enforce global spatial smoothness, e.g., the diffusion regularizer. However, such regularizers are not tailored to the data and might not be capable of reflecting the complex underlying deformation. In contrast, physics-inspired regularizers promote physically plausible deformations. One such regularizer is the linear elastic regularizer, which models the deformation of elastic material. These regularizers are driven by parameters that define the material’s physical properties. For biological tissue, a wide range of estimations of such parameters can be found in the literature, and it remains an open challenge to identify suitable parameter values for successful registration. To overcome this problem and to incorporate physical properties into learning-based registration, we propose to use a hypernetwork that learns the effect of the physical parameters of a physics-inspired regularizer on the resulting spatial deformation field. In particular, we adapt the HyperMorph framework to learn the effect of the two elasticity parameters of the linear elastic regularizer. Our approach enables the efficient discovery of suitable, data-specific physical parameters at test time. To the best of our knowledge, we are the first to use a hypernetwork to learn physics-inspired regularization for medical image registration. We evaluate our approach on 3D intrapatient lung CT images. The results show that the linear elastic regularizer can yield comparable results to the diffusion regularizer in unsupervised learning-based registration while predicting deformations with fewer foldings. With our method, the adaptation of the physical parameters to the data can successfully be performed at test time. Our code is available at https://github.com/annareithmeir/elastic-regularization-hypermorph.

This study aims to simplify radiation therapy treatment planning by proposing an MRI-to-CT transformer-based denoising diffusion probabilistic model (CT-DDPM) to generate high-quality synthetic computed tomography (sCT) from magnetic resonance imaging (MRI). The goal is to reduce patient radiation dose and setup uncertainty by eliminating the need for CT simulation and image registration during treatment planning. The CT-DDPM utilizes a diffusion process with a shifted-window transformer network to transform MRI into sCT. The model comprises two processes: a forward process, adding Gaussian noise to real CT scans to create noisy images, and a reverse process, denoising the noisy CT scans using a Vshaped network (Vnet) conditioned on the corresponding MRI. With an optimally trained Swin-Vnet, the reverse process generates sCT scans matching the MRI anatomy. The method is evaluated using mean absolute error (MAE) of Hounsfield unit (HU), peak signal-to-noise ratio (PSNR), multi-scale Structural Similarity index (MS-SSIM) and normalized cross-correlation (NCC) between ground truth CTs and sCTs. For the brain dataset, CT-DDPM demonstrated state-of-the-art quantitative results, exhibiting an MAE of 45.210±3.807 HU, a PSNR of 26.753±0.861 dB, an SSIM of 0.964±0.005, and an NCC of 0.981±0.004. In the context of the prostate dataset, the model also showed impressive performance with an MAE of 55.492±8.281 HU, a PSNR of 28.912±2.591 dB, an SSIM of 0.894±0.092, and an NCC of 0.945±0.054. Across both datasets, CT-DDPM significantly outperformed competing networks in most metrics, a finding corroborated by the student’s paired t-test. The source code is available: https://github.com/shaoyanpan/Synthetic-CT-generation-from- MRI-using-3D-transformer-based-denoising-diffusion-model

One of the major neuropathological consequences of traumatic brain injury (TBI) is intracranial hemorrhage (ICH), which requires swift diagnosis to avert perilous outcomes. We present a new automatic hemorrhage segmentation technique via curriculum-based semi-supervised learning. It employs a pre-trained lightweight encoder-decoder framework (MobileNetV2) on labeled and unlabeled data. The model integrates consistency regularization for improved generalization, offering steady predictions from original and augmented versions of unlabeled data. The training procedure employs curriculum learning to progressively train the model at diverse complexity levels. We utilize the PhysioNet dataset to train and evaluate the proposed approach. The performance results surpass those of supervised model with an average Dice coefficient and Jaccard index of 0.573 and 0.428, respectively. Additionally, the method achieves 87.86% accuracy in hemorrhage classification and Cohen's Kappa value of 0.81, indicating substantial agreement with ground truth.

We present a robust vertebral segmentation framework that can bootstrap the segmentation task with little to no dataset. Our deformable model-based framework jointly optimizes the appearance and shape of the spine model using the novel differentiable appearance modeling method. Because our framework learns the appearance of the spine only from the given image, it does not rely on the dataset or handcrafted image features and adapts robustly to the appearance of the image. With our proposed differentiable signed distance operator and spectral mesh optimization, the shape of the spine model can be refined via a gradient-based optimizer. Our framework was tested on the VerSe’20 training dataset, and it achieved an average Dice score of up to 90% for selected vertebral labels. Our results suggest that utilizing the explicit knowledge from the template model can significantly reduce the need for a large training dataset.

Accurate and reliable registration of longitudinal spine images is essential for assessment of disease progression and surgical outcome. Implementing a fully automatic and robust registration is crucial for clinical use, however, it is challenging due to substantial change in shape and appearance due to lesions. In this paper we present a novel method to automatically align longitudinal spine CTs and accurately assess lesion progression. Our method follows a two-step pipeline where vertebrae are first automatically localized, labeled and 3D surfaces are generated using a deep learning model, then longitudinally aligned using a Gaussian mixture model surface registration. We tested our approach on 37 vertebrae, from 5 patients, with baseline CTs and 3, 6, and 12 months follow-ups leading to 111 registrations. Our experiment showed accurate registration with an average Hausdorff distance of 0.65 mm and average Dice score of 0.92.

Breast masses are the most critical clinical symptom of breast cancer, underscoring the importance of early detection for accurate diagnosis and the need for improved accuracy based on high detection rates and low false-positive rates. In a standard routine mammography screening, cranial-caudal (CC) and mediolateral-oblique (MLO) views are acquired per breast. Employing the two standard views aids radiologists in making more dependable decisions compared to relying on a single view, as it offers information on correspondence, thus enhancing reliability. As a result, this research introduces a deep learning model, the Paired-mammogram view Network, based on a convolutional neural network (CNN) that simultaneously utilizes both CC view and MLO view in mammography to improve the performance of breast mass detection. To assess the efficacy of the suggested approach, we conducted a performance comparison between the proposed method and both single and paired view models using the identical dataset. The proposed network based on Resnet50 reached a sensitivity of 0.922, precision of 0.884, and false positives per image of 0.156; The contrast single view model reached a sensitivity of 0.888, precision of 0.853, and false positives per image of 0.188. This work demonstrates the proposed algorithm based on a clinical approach can be utilized for early diagnosis of breast cancer.

Recent studies indicate that malignant breast lesions can be predicted from structural changes in prior exams of preventive breast MRI examinations. Due to non-rigid deformation between studies, spatial correspondences between structures in two consecutive studies are lost. Thus, deformable image registration can contribute to predicting individual cancer risks. This study evaluates a registration approach based on a novel breast mask segmentation and non-linear image registration based on data from 5 different sites. The landmark error (mean ± standard deviation [1^st quartile, 3^rd quartile]), annotated by three radiologists, is 2.9 ± 2.8 [1.3, 3.2] mm when leaving out two outlier cases from the evaluation for which the registration failed completely. We assess the inter-observer variabilities of keypoint errors and find an error of 3.6 ± 4.7 [1.6, 4.0] mm, 4.4 ± 4.9 [1.8, 4.8] mm, and 3.8 ± 4.0 [1.7, 4.1] mm when comparing each radiologist to the mean keypoints of the other two radiologists. Our study shows that the current state of the art in registration is well suited to recover spatial correspondences of structures in cancerous and non-cancerous cases, despite the high level of difficulty of this task.

Medical image segmentation benefits from machine learning advancements, offering potential automation. Yet, accuracy depends on substantial annotated data and significant computing resources. Transfer learning addresses these challenges by leveraging a model's knowledge from one task for another with minor adjustments. The idea is to adapt learned features to new tasks, even with differing datasets but shared characteristics. Studies explore the impact of using large source datasets for limited target datasets. This investigation focuses on transferring knowledge from a limited source to enhance model versatility across various tasks. Our goal involved transferring knowledge from an advanced model trained on T2 weighted MR images related to Autosomal Dominant Polycystic Kidney Disease (ADPKD) for kidney and cyst segmentation (referred to as "Lsource"). This transfer was directed towards five distinct target datasets: CT liver, CT kidneys, CT spleen, MRI kidneys, and CT multimodal data (target datasets 1 through 5). The primary objective was to achieve accurate segmentation on these target datasets while saving time and computational resources. This approach is especially valuable when obtaining a substantial, labeled mouse PKD MRI target dataset is challenging, and the source dataset itself is resource-intensive. Using transfer learning from source 1 onto target sets 1 to 5 resulted in mean Dice Similarity Coefficients (DSCs) of 0.94±0.04, 0.97±0.02, 0.95±0.03, 0.96±0.01, 0.93±0.02, respectively. Similarly, employing source 2 yielded mean DSCs of 0.95±0.04, 0.96±0.02, 0.95±0.02, 0.96±0.02, and 0.93±0.02 for the same target sets. Despite variations in pathological conditions, image characteristics, and imaging modalities, the transfer learning approach produced DSC values comparable to the initial published outcomes. This accomplishment was achieved with reduced training requirements, faster convergence times, and decreased computational complexities.

Unpaired image synthesis is a particularly active area of research, especially in medical imaging, where paired datasets are rare. Disentangled representations are an important part of the techniques used, following those based on GAN. However, by relying on the factorization of an image into independent variation latent codes, these methods can offer greater control over the synthesis result than GANs. This work investigates the use of disentangled representation learning for high-resolution dynamic MRI synthesis.

Recent image registration methods built upon unsupervised learning have achieved promising diffeomorphic image registration performance. However, the bijective consistency of spatial transformations is not sufficiently investigated in existing image registration studies. In this study, we develop a multi-level image registration framework to achieve diffeomorphic image registration in a coarse-to-fine manner. A novel stationary velocity field computation method is proposed to integrate forward and inverse stationary velocity fields so that the image registration result is invariant to the order of input images to be registered. Moreover, a new bijective consistency regularization is adopted to enforce the bijective consistency of forward and inverse transformations at different time points along the stationary velocity integration paths. Validation experiments have been conducted on two T1-weighted magnetic resonance imaging (MRI) brain datasets with manually annotated anatomical structures. Compared with four state-of-the-art representative diffeomorphic registration methods, including two traditional diffeomorphic registration algorithms and two unsupervised learning-based diffeomorphic registration approaches, our method has achieved better image registration accuracy with superior topology preserving performance.

Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes—i.e., articulatory-acoustic relation—is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals. A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.

Segmentation is an essential tool for quantification and characterization of tissue properties, with applications ranging from assessment of body composition, disease diagnosis, to development of imaging biomarkers. In this work, we propose a multi-method and multi-atlas methodology for automated segmentation of functional muscle groups in 3D Dixon MR images of the mid-thigh. The functional muscle groups addressed in this paper lie anatomically close to each other, that makes segmentation an arduous task for accuracy. We propose an approach that uses anatomical mappings enabling delineation of adjacent muscle groups that are difficult to separate using conventional intensity-based patterns only. We segment the four functional muscle groups of the thigh in both legs by multi-atlas anatomical mappings and fuse the labels to improve delineation accuracy. We investigate the fusion of segmentation from multiple atlases and multiple deformable registration methods. For performance evaluation we applied cross-validation by excluding the scans that served as templates in our framework and report DSC values on the remaining test scans. We evaluated four individual deformable models, free-form deformation (FFD), symmetric normalization (SYN), symmetric diffeomorphic demons (SDD), and Voxelmorph (VXM), and the joint multi-method fusion. Multi-atlas and multi-method fusion produced the top average DSC of 0.795 over all muscles on the test scans.

The purpose of this research is to address the critical challenge of improving the detectability of small perfusion defects in deep learning (DL) denoising for low dose Myocardial Perfusion Imaging (MPI) with Single-Photon Emission Computed Tomography (SPECT). By developing a 3D convolutional auto-encoder (CAE) incorporated with an edge-preservation mechanism, the study aims to mitigate potential blurring effects associated with DL-based denoising methods. The CAE is optimized to enhance noise reduction on low-dose SPECT-MPI scans while seeking to maintain the integrity of image-edge features which are vital for preserving subtle myocardial perfusion defects after denoising.

Emphysema progression in COPD is highly variable, and there is a lack of prognostic markers that describe the evolution of the disease. We hypothesize that local CT features can provide suitable information for describing the evolution of emphysema. In this work, we propose a deep learning approach to identify “deep local phenotypes” by learning the evolution of lung density and emphysema progression at a timed event using longitudinal autoencoders. Results show that there are local traits, or deep local phenotypes, that can predict the 5-year evolution of the local density patterns. Local features learned from the latent space representation of the emphysema progression seem to provide prognostic markers of progression due to the good performance in identifying regions (32 by 32 pixels) with and without emphysema progression. The model’s performance on our test dataset (n=364 millon of regions from 295 subjects) was AUC_5%=0.87, PPV_5%=0.57, Sens._5%= 0.55, ACC_5%=0.84, Balanced ACC_5% = 0.73, F1_5%=0.56 and K_5%=0.46. Prognostication tools to detect local patterns of emphysema progression could be essential to manage COPD patients better, as there are no therapeutic options to reverse (or slow down) emphysema damage. Our approach could also improve the selection of patients in much-needed emphysema treatment trials.

Medical image classification plays a vital role in disease diagnosis, tumor staging, and various clinical applications. Deep learning (DL) methods have become increasingly popular for medical image classification. However, medical images have unique characteristics that pose challenges for training DL-based models, including limited annotated data, imbalanced distribution of classes, and large variations in lesion structures. Self-supervised learning (SSL) methods have emerged as a promising solution to alleviate these issues through directly learning useful representations from large-scale unlabeled data. In this study, a new generative self-supervised learning method based on the StyleGAN generator is proposed for medical image classification. The style generator, pretrained on large-scale unlabeled data, is integrated into the classification framework to effectively extract style features that encapsulate essential semantic information from input images through image reconstruction. The extracted style feature serves as an auxiliary regularization term to leverage knowledge learned from unlabeled data to support the training of the classification network and enhance model performance. To enable efficient feature fusion, a self-attention module is designed for this integration of the style generator and classification framework, dynamically focusing on important feature elements related to classification performance. Additionally, a sequential training strategy is designed to train the classification model on a limited number of labeled images while leveraging large-scale unlabeled data to improve classification performance. The experimental results on a chest X-ray image dataset demonstrate superior classification performance and robustness compared to traditional DL-based methods. The effectiveness and potential of the model were discussed as well.

Single-image super resolution (SR) task has evolved mainly in natural imaging and achieved impressive performance by employing vision transformer-based methods. We can consider using these deep-learning-based SR methods to obtain high-resolution medical images to confirm the possibility of improving the resolution of radiography images. In this paper, we apply a vision transformer as a single-image SR method in dynamic X-ray radiography (DXR). Because utilizing high-resolution images in DXR is difficult due to the limitation of resources, applying a binning technique to downsize images is inevitable even though the technique degrades the image quality. However, the SR simulations show that the network model can restore the aliased signals and significantly improve the modulation transfer function from the downsized images. Therefore, using the SR method based on the vision transformer, we can effectively restore high-resolution images from downsized images and thus develop high-performance DXR systems.

Reticular pseudodrusen (RPD) are subretinal drusenoid deposits that represent an important disease feature in age-related macular degeneration (AMD). RPD are of particular interest because their presence is a strong predictor of progression to advanced AMD. RPD features can be characterized using volumetric spectral-domain optical coherence tomography (SD-OCT). In this work, we curated a dataset from the Age-Related Eye Diseases Study 2 (AREDS2) ancillary OCT study. The dataset included 826 SD-OCT scans, with RPD present in 222 SD-OCT scans. Binary RPD labels were transferred from fundus autofluorescence (FAF) images taken at the same visits as the SD-OCT scans. The dataset was split at the participant level into training (70%), validation (10%), and test sets (20%). We proposed a 3D classification network to detect RPD from SD-OCT scans. We compared it to a baseline 2D network with average bagging and a 3D network with multi-tasking. The proposed network achieved the highest accuracy of 0.7784, area under receiver characteristic operating curve of 0.8689, and mean average precision of 0.7706 for detecting RPD from SD-OCT scans.

Histopathology whole-slide image (WSI) captures detailed structural and morphological features of tumor tissue, offering rich histological and molecular information to support clinical practice. With the development of artificial intelligence, deep learning (DL) methods have emerged to assist in automatically analyzing histopathology WSIs. It alleviates the need for tedious, time-consuming, and error-prone inspections by clinicians. Up to now, employing DL models for histopathology WSI analysis is still challenging due to the intrinsic complexity of histology characteristics of tumor tissue, high image resolution, and large image size. In this study, we proposed a transformer-based classifier with feature aggregation for cancer subtype classification using histopathology WSIs while addressing these challenges. Our method shows three advantages to improve classification performance. First, an aggregate transformer decoder is employed to learn both global and local features from WSIs. Second, the transformer architecture facilitates the decoder to learn spatial correlations among different regions in a WSI. Third, the self-attention mechanism of the transformer facilitates the generation of saliency maps to highlight regions of interest in WSIs. We evaluated our model on three cancer subtype classification tasks and demonstrated its effectiveness and performance.

It is known that lumbar disc degeneration, a progressive structural wear and tear of lumbar intervertebral disc, is regarded as an essential factor on low back pain, a global health concern. Magnetic Resonance Imaging (MRI) plays a crucial role in detecting the morphologic changes and internal information of tissues. Thus, it is essential to develop accurate semantic segmentation of lumbar spine MRI images, which can assist doctors in recognizing the state of spinal degeneration in order to determine a suitable treatment. Recently, deep learning approaches have shown remarkable performance in medical image segmentation. However, the existing segmentation models often generate erroneously fragmented regions due to reasons related to texture similarity, image quality, or noises. In this work, we propose a novel neural network for the geometry reconstruction of lumbar spine from 2D mid-sagittal MRI images. We incorporated a UNet-style backbone with sequential geometry deformation modules to predict shape deformation on varying scales. We developed image feature extraction in a shape-specific manner to reject irrelevant information. We utilized self-attention mechanism to further process the extracted shape representation fused with image features and with a template as position embedding. We compared our model with some well-known models for image segmentation, including UNet++, Attention UNet, TransUNet, Swin-Unet, and UTNet. The results demonstrate that our model has the best performance, and its segmentation results are highly accurate and free of erroneous fragments. The source code is available at https://github.com/linchenq/SPIE2024-Geometry-Deformation-Lumbar-Spine-Segmentation.

Following the acquisition of images in CT, a crucial post-processing step involves orienting the volumetric image to align with standard viewing planes, facilitating the assessment of disease extent and other pathologies. However, manual alignment is not only time-consuming but can also pose challenges in achieving consistent standard plane views, particularly for lesser skilled technologists. Existing automated solutions, primarily based on registration techniques, encounter reduced accuracy in cases involving significant rotations, pediatric patients, and instances with pronounced pathological effects. This limitation arises due to their reliance on symmetry. In severe scenarios, registration-based methods can exacerbate image misalignment compared to the original input. To address these concerns, this study introduces a landmark-based automated image alignment method. This method presents three key advantages: robust alignment across diverse data variations, the capability to identify algorithm failures and gracefully terminate, and the ability to align images with different standard planes. The effectiveness of our method is showcased through a comparative evaluation with registration-based approaches. The evaluation employs a test dataset comprising various head cases across different age groups, reaffirming the effectiveness of our proposed method.

Positron emission tomography (PET) provides valuable functional information that is widely used in clinical domains such as oncology and neurology. However, the structural quality of PET images may not be sufficient to effectively evaluate small regions of interest. Image super-resolution techniques aim to recover a high-resolution image from an input low-resolution version. We study adaptations of deep convolutional neural network architectures for improving the spatial resolution of PET images. The proposed super-resolution model involves a deep architecture that uses convolutional blocks together with various residual connections for more effective and efficient training. We use the supervised setting where the downscaled versions of the original PET images are given as the low-resolution input to the deep networks and the original images are used as the high-resolution target data to be recovered. Experiments show that the proposed model performs better than a multi-scale convolutional architecture according to both quantitative performance metrics and visual qualitative evaluation.

Geometrical parameters of the spine have been identified as valuable for describing spinal deformities, and their measurement is commonly based on landmarks, identified in radiographs but often also in three-dimensional (3D) magnetic resonance (MR) images of the spine. However, existing studies make use of manually pre-selected midsagittal cross-sections of 3D MR images that best align with the middle of the spines. In this work, we propose an algorithm for lumbar vertebra landmarking in 3D MR images by using a combination of neural networks for the identification of the correct mid-sagittal cross-section and landmark area segmentation. In a database of 3D MR images from 70 patients, four landmark groups were defined on lumbar vertebrae and sacral endplates: antero-superior, postero-superior, anterior-inferior and postero-interior. A ResNet was first applied to estimate the displacements of sagittal cross-sections and obtain the location of the mid-sagittal cross-section within the 3D image. A U-Net was then used to segment the landmark areas in several cross-sections that were predicted as candidates for the mid-sagittal cross-section. Both networks were finally combined into a landmark detection framework. The overall landmark detection error, measured as the mean absolute distance (± standard deviation) between manually placed and automatically detected landmarks, was 2.2±1.3 mm, with 92.7% of all landmarks detected correctly. The resulting error can be decomposed into 1.6±1.4mm along the sagittal direction (ResNet) and 1.3±0.9mm within the sagittal plane (U-Net), which is close to human performance.

Spinal imaging serves as an invaluable tool in the non-invasive visualization and evaluation of spinal pathologies. A key basis for quantitative medical image analysis pertinent to clinical diagnosis and spinal surgery planning is the segmentation of vertebrae in computed tomography (CT) images. While fully convolutional networks in general dominate over medical image segmentation, with the U-Net being the architecture of choice, alternative methodologies may offer potential advancements. One promising approach is the deep implicit statistical shape model (DISSM), known for generating high-quality surfaces without discretization and for its robustness, underpinned by the use of rich and explicit anatomical priors, particularly for challenging cross-dataset clinical samples. This paper explores the utilization of DISSM for vertebra segmentation on two image datasets: a collection of 1005 CT spine images known as CTSpine1K for the shape decoder, and a set of 319 CT images known as VerSe2020 for the pose estimation encoders (translation, rotation, scaling and principal component analysis). These images and their corresponding vertebra segmentations are used for the preparation, preprocessing, and training and testing of DISSM. The preprocessing and learning techniques are based on a DISSM software package (AshStuff/dissm) with our custom modifications. The obtained segmentation results yielded an overall mean Dice coefficient of 0.767, average symmetric surface distance of 1.93 mm, and 95^th percentile Hausdorff distance of 5.71 mm. We can therefore conclude that DISSM has the potential to further advance the field of vertebra segmentation.

Managing patients with hydrocephalus and cerebrospinal fluid disorders requires repeated head imaging. In adults, this is typically done with computed tomography (CT) or less commonly magnetic resonance imaging (MRI). However, CT poses cumulative radiation risks and MRI is costly. Transcranial ultrasound is a radiation-free, relatively inexpensive, and optionally point-of-care alternative. The initial use of this modality has involved measuring gross brain ventricle size by manual annotation. In this work, we explore the use of deep learning to automate the segmentation of brain right ventricle from transcranial ultrasound images. We found that the vanilla U-Net architecture encountered difficulties in accurately identifying the right ventricle, which can be attributed to challenges such as limited resolution, artifacts, and noise inherent in ultrasound images. We further explore the use of coordinate convolution to augment the U-Net model, which allows us to take advantage of the established acquisition protocol. This enhancement yielded a statistically significant improvement in performance, as measured by the Dice similarity coefficient. This study presents, for the first time, the potential capabilities of deep learning in automating hydrocephalus assessment from ultrasound imaging.

In modern neuroscience, functional magnetic resonance imaging (fMRI) has been a crucial and irreplaceable tool that provides a non-invasive window into the dynamics of whole-brain activity. Nevertheless, fMRI is limited by hemodynamic blurring as well as high cost, immobility, and incompatibility with metal implants. Electroencephalography (EEG) is complementary to fMRI and can directly record the cortical electrical activity at high temporal resolution, but has more limited spatial resolution and is unable to recover information about deep subcortical brain structures. The ability to obtain fMRI information from EEG would enable cost-effective, naturalistic imaging across a wider set of brain regions. Further, beyond augmenting the capabilities of EEG, cross-modality models would facilitate the interpretation of fMRI signals. However, as both EEG and fMRI are high-dimensional and prone to noise and artifacts, it is currently challenging to model fMRI from EEG. Indeed, although correlations between these two modalities have been widely investigated, few studies have successfully used EEG to directly reconstruct fMRI time series. To address this challenge, we propose a novel architecture that can predict fMRI signals directly from multi-channel EEG without explicit feature engineering. Our model achieves this by implementing a Sinusoidal Representation Network (SIREN) to learn frequency information in brain dynamics from EEG, which serves as the input to a subsequent encoder-decoder to effectively reconstruct the fMRI signal in a specific brain region. We evaluate our model using a simultaneous EEG-fMRI dataset with 8 subjects and investigate its potential for predicting subcortical fMRI signals. The present results reveal that our model outperforms a recent state-of-the-art model and indicate the potential of leveraging periodic activation functions in deep neural networks to model functional neuroimaging data.

Although Digital Subtraction Angiography (DSA) is the most important imaging for visualizing cerebrovascular anatomy, its interpretation by clinicians remains difficult. This is particularly true when treating arteriovenous malformations (AVMs), where entangled vasculature connecting arteries and veins needs to be carefully identified. The presented method aims to enhance DSA image series by highlighting critical information via automatic classification of vessels using a combination of two learning models: An unsupervised machine learning method based on Independent Component Analysis that decomposes the phases of flow and a convolutional neural network that automatically delineates the vessels in image space. The proposed method was tested on clinical DSA images series and demonstrated efficient differentiation between arteries and veins that provides a viable solution to enhance visualizations for clinical use.

Image registration serves as a central pre-processing step in medical image processing and plays a crucial role in various applications such as medical image analysis, diagnostics and surgical navigation. Existing deformable registration methods have mainly focused on the efficient extraction of features from two images, coupled with the automatic derivation of deformation fields. However, these methods tend to overlook the inherent low-level nature of deformable registration, which is considered unnecessary for feature extraction operations. Instead, deformable registration tasks should place more emphasis on detecting dissimilarities between the two images, thereby obtaining a more accurate deformation field. In this work, we present an unsupervised deformable image registration network called Cross-Matching-based Deformable Image Registration Network (CMBMorph). We introduce a cross-matching-based mechanism designed to facilitate the learning of feature disparities between the two images. By focusing attention on the non-aligned regions of the two images, our model achieves high-quality medical image registration results. We then evaluate our proposed method on publicly available datasets. Compared to baseline models and state-of-the-art techniques, our method achieves a Dice accuracy of 0.812, establishing a new state-of-the-art performance level.

Accurately segmenting complete teeth from CBCT images is not only crucial in the field of orthodontics but also holds significant importance in forensic science. However, fully automated teeth segmentation is challenging due to the tight interconnection of teeth and the complexity of their arrangement, as well as the difficulty in distinguishing them from the surrounding alveolar bone due to similar densities. Currently, U-Net-based approaches have demonstrated remarkable success across a spectrum of medical image processing tasks, particularly in the task of segmentation. This work compares some U-Net-based segmentation methods (U-Net, U-Net++, U²-Net, nnU-Net, and TransUNet) on clinical teeth segmentation. We assess the enhancements introduced by these networks over the original U-Net and validate their performance on the identical dataset. Experimental results, both qualitative and quantitative, reveal that all methods perform well, with TransUNet demonstrating the best performance, achieving a Dice coefficient of 0.9364. Notably, U-Net, serving as the foundational model, outperforms U-Net++ and U²-Net, highlighting its robust generalization capability.