Image-Guided Procedures, Robotic Interventions, and Modeling

Artificial intelligence and machine learning (AI/ML) are powerful tools for building computer vision systems that support the work of clinicians, leading to high interest and explosive growth in the use of these methods to analyze clinical images. These promising AI techniques create computer vision systems that perform some image interpretation tasks at the level of expert radiologists. In radiology, deep learning methods have been developed for image reconstruction, imaging quality assurance, imaging triage, computer-aided detection, computer-aided classification, and radiology documentation. The resulting computer vision systems are being implemented now and have the potential to provide real-time assistance, thereby reducing diagnostic errors, improving patient outcomes, and reducing costs. We will show examples of real-world AI applications that indicate how AI will change the practice of medicine and illustrate the breakthroughs, setbacks, and lessons learned that are relevant to medical imaging.

Intelligent medical systems adept at acquiring and analyzing sensor data to offer context-sensitive support are at the forefront of modern healthcare. However, various factors, often not immediately apparent, significantly hinder the effective integration of contemporary machine learning research into clinical practice. Using insights from my own research team and extensive international collaborations, I will delve into prevalent issues in current medical imaging practices and offer potential remedies. My talk will highlight the vital importance of challenging every aspect of the medical imaging pipeline from the image modalities applied to the validation methodology, ensuring that intelligent imaging systems are primed for genuine clinical implementation.

Ultrasound tomography (UST) is an emerging medical imaging modality that has found its way into clinical practice after its recent approval by the Food and Drug Administration (FDA) for breast cancer screening and diagnostics. As an active area of research, UST also shows promise for applications in brain, prostate, limb and even whole-body imaging. The historical development of ultrasound tomography is rooted in the idea of “seeing with sound” and the concept borrows heavily from diverse disciplines, including oceanography, geophysics and astrophysics. A brief history of the field is provided, followed by a review of current reconstruction methods and imaging examples. Unlike other imaging modalities, ultrasound tomography in medicine is computationally bounded. Its future advancement is discussed from the perspective of ever-increasing computational power and Moore's Law.

Finding desired scan planes in ultrasound imaging is a critical first task that can be time-consuming, influenced by operator experience, and subject to inter-operator variability. This work presents a new approach that leverages deep reinforcement learning to automate probe positioning during intraoperative ultrasound imaging. A dueling deep Q-network is applied and evaluated for kidney imaging. The agent was trained on images resliced from CT images, with a novel reward function that used image features. Evaluations on an independent test dataset demonstrated the agent’s ability to reach target views with an accuracy of 76% ± 8% within an average of 18 ± 11 steps.

This study describes the development of an image-guided TORS (igTORS) framework utilizing intraoperative imaging, instrument tracking, and real-time synchronized visualization with a heads-up display integrated to the da Vinci Surgical System. The navigation efficacy was evaluated in a phantom study where a TORS-experienced surgeon performed target localization tasks following the standard-of-care TORS and igTORS protocols, respectively. The target localization error of the igTORS trial was significantly reduced when compared to the standard-of-care TORS trial (2.10mm±1.42mm vs. 4.60mm±3.29mm; p=0.05). The authors successfully demonstrated the efficacy of the igTORS framework in improving surgical accuracy. Future work includes system evaluation in cadaveric studies and enabling image overlay.

Minimally invasive approaches for intracerebral hemorrhage evacuation have shown promising results in improving patient outcomes. However, these approaches are still disruptive to surrounding normal brain tissue and do not allow for the near-total evacuation of a hemorrhage. In this MRI-guided study, we assessed an MRI-Compatible robotic aspiration device in a sheep brain phantom. The robot was advanced into the clot and aspiration was performed with real-time intraoperative MR imaging. The volume of the clot was reduced by 83% in 21 seconds and the phantom did not have any unexpected damage from the procedure.

The accurate reconstruction of surgical scenes from surgical videos is critical for various applications, including intraoperative navigation and image-guided robotic surgery automation. However, previous approaches, mainly relying on depth estimation, have limited effectiveness in reconstructing surgical scenes with moving surgical tools. To address this limitation and provide accurate 3D position prediction for surgical tools in all frames, we propose a novel approach called SAMSNeRF that combines Segment Anything Model (SAM) and Neural Radiance Field (NeRF) techniques. Our approach generates accurate segmentation masks of surgical tools using SAM, which guides the refinement of the dynamic surgical scene reconstruction by NeRF. Our experimental results on public endoscopy surgical videos demonstrate that our approach successfully reconstructs high-fidelity dynamic surgical scenes and accurately reflects the spatial information of surgical tools. Our proposed approach can significantly enhance surgical navigation and automation by providing surgeons with accurate 3D position information of surgical tools during surgery.

Accurate models of the mitral valve are highly valuable for studying the physiology of the heart and its various pathologies, as well as creating replicas for cardiac surgery training. Currently, heart simulator technologies are used which rely on patient-specific data to create valve replicas. Alternatively, mathematical models of the mitral valve have been developed for computational applications. However, currently mathematically models do not include both the mitral valve’s leaflets and its saddle-shaped annulus in one design together. Mathematical models to date have not been replicated as dynamic, physical valve models and validated in a heart simulator system. We propose a new parametric representation of the mitral valve based on a combination of valve models from prior literature, combining both accurate leaflet shape, and annular geometry. A physical silicone replica of the model is created and validated in a pulse duplicator. Using a transesophageal echocardiography probe with color Doppler imaging, we demonstrate that our combined model replicates healthy valve behaviour, showing no regurgitation at realistic pressure gradients across the valve.

Kawasaki disease, predominantly affecting children, can lead to potential complications in the coronary arteries, potentially causing inflammation of blood vessel walls if left untreated. Intravascular Optical Coherence Tomography (IV-OCT) offers vital coronary artery imaging guidance to cardiologists, but its operation demands skilled expertise and adherence to intricate protocols. Our study introduces a novel approach utilizing polyvinyl alcohol cryogel (PVA-c) to fabricate patient-specific coronary OCT phantoms. These phantoms closely mimic human tissue, serving as valuable tools for training cardiologists and deepening understanding of the OCT image formation process. By designing 3D molds based on real OCT arterial images, we create PVA-c phantoms that capture the morphological characteristics and visual features of diseased coronary arteries. Our findings indicate that these phantoms effectively emulate the structures and appearances observed in OCT, closely resembling human tissue.

The increasing incidence of laryngeal carcinomas requires approaches for early diagnosis and treatment. In clinical practice, white light endoscopy of the laryngeal region is typically followed by biopsy under general anesthesia. Optical coherence tomography (OCT) has been proposed to study sub-surface tissue layers at high resolution. However, accessing the region of interest requires miniature OCT probes that can be inserted in the working channel of a laryngoscope. Typically, such probes generate single column depth images which are difficult to interpret. We propose a novel approach using endoscopic images to spatially align these images. Given the natural tissue motion and movements of the laryngoscope, resulting OCT images show a three-dimensional representation of sub-surface structures, which is simpler to interpret. We present a motion tracking method and assess the precision of spatial alignment. Furthermore, we demonstrate the in-vivo application, illustrating the benefit of spatially meaningful alignment of OCT images to study laryngeal tissue.

Localizing the electrode array (EA) in cochlear implant (CI) postoperative computed tomography (CT) images is needed in image-guided CI programming, which has been shown to improve hearing outcomes. Postoperative images with adequate image quality are required to allow the EA to be reliably and precisely localized. However, these images sometimes are affected by motion artifacts which can make the localization task unreliable or even fail. Thus, flagging these low-quality images prior to the subsequent clinical use is important. In this work, we propose to assess the image quality by using a 3D convolutional neural network to classify the level (no/mild/moderate/severe) of the motion artifacts that affect the image. To address the challenges of subjective annotations and class imbalance, several techniques (a new loss term, an oversampling strategy, and motion artifact simulation) are used during training. Results demonstrate the proposed method has the potential to reduce time and efforts on image quality assessment that is traditionally by visual inspection.

The management of lung cancer requires efficient tools like three-dimensional (3D) computed tomography (CT) imaging and bronchoscopy, but fusing their data remains challenging due to the difficulty in acquiring depth and camera pose information directly from bronchoscopic video frames. We propose a self-supervised training strategy to train a network to predict them among bronchoscopic sequences. Our strategy uses the so-called monodepth2 training procedure on bronchoscopic video sequences and cycleGAN-synthesized bronchoscopic view and depth pairs to enhance the depth prediction. Results highlight precise depth prediction and pose accuracy, showcasing the strategy's potential for CT-video fusion.

Breast conserving surgery is a common treatment option for women with early-stage breast cancer, but these procedures have high and variable reoperation rates due to positive resection margins. This work proposes an image guidance system for breast conserving surgery that combines stereo camera soft tissue monitoring with nonrigid registration for deformation correction. A series of breast phantom deformation experiments were performed to demonstrate system capabilities, and validation studies with human volunteers are ongoing. Overall, this system may allow for better navigation and tumor localization during breast conserving surgeries.

This work presents a new system for electromagnetic catheter navigation during endovascular interventions. A custom catheter instrument was designed and constructed to integrate a 5 single EM coil sensor at its tip. The tracked sensor was used in (1) dynamically reconstructing the instrument shape as it is advanced or retracted within the vessels; (2) visualizing the tip direction to guide it through vessel bifurcations; and (3) registering its path to vessel centerlines to provide image overlay. Experimental studies demonstrate sufficient accuracy (4.1 mm and 3.4°) for guiding the catheter through the main arteries.

Minimally invasive surgical techniques have improved patient outcomes and postoperative recovery, but it's limited by its field of view and difficulty in locating subsurface targets. Our proposed solution applies an augmented reality (AR) based system to overlay pre-operative images acquired from magnetic resonance imaging (MRI) onto the target organ providing the location of subsurface lesions and a proposed surgical guidance path in real-time. An infrared motion tracking camera system was employed to obtain real-time position data of the phantom model and surgical instruments. To perform hologram registration, fiducial markers were used to track and map virtual coordinates to the real -world. Phantom models of each organ were constructed to test the reliability of the AR system. Our results show a registration root-mean-square error of 2.42 ± 0.79 mm and a procedural targeting error of 4.17 ± 1.63 mm using our AR-guided laparoscopic system.

Locating non-palpable lesions and lymph nodes during cancer surgery is crucial for management of the disease. Unfortunately, currently available surgical guidance techniques, such as radioguided surgery, cannot always provide precise localization of the cancerous lesions. This research investigates the use of deep learning algorithms to improve the resolution of lesion detection in a hand-held gamma probe. Preliminary results demonstrate that a neural network achieves up to a 10-fold improvement in resolution compared to existing clinically available gamma probes for detection of high-energy radionuclides. These results show promise for efficiently guiding a surgeon towards the lesion of interest and thus improving the surgical accuracy.

Surgical Navigation Systems (SNS) employing optical tracking systems (OTS) have become the industry standard for computer-aided position tracking of medical instruments and patients. However, OTS face challenges due to line-of-sight issues caused by occluded or contaminated markers. To overcome these limitations, this paper proposes a novel approach using real surgery data to simulate occlusion and evaluate instrument visibility based on the idea to develop a markerless system with multiple RGBD-cameras, AI-based techniques, and optical-geometrical postprocessing for precise instrument tracking. The simulation introduces the "task occlusion score" (TOS) to measure average instrument occlusion. Results indicate that optimal camera placement for visibility is above the situs, contrary to traditional setups. This simulation enhances the usability of navigated surgery, offering potential for marker-based systems with different marker geometries, and further possibilities for optimizing tracking accuracy using multiple cameras.

Machine learning models that detect surgical activities in endoscopic videos are instrumental in scaling post-surgical video review tools that help surgeons improve their practice. However, it is unknown how well these models generalize across various surgical techniques practiced at different institutions. In this paper, we examined the possibility of using surgical site information for a more tailored, better-performing model on surgical procedure segmentation. Specifically, we developed an ensemble model consisting of site-specific models, meaning each individual model was trained on videos from a specific surgical site. We showed that the site-specific ensemble model consistently outperforms the state-of-the-art site-agnostic model. Furthermore, by examining the representation of video-frames in the latent space, we corroborated our findings with similarity metrics comparing videos within and across sites. Lastly, we proposed model deployment strategies to manage the introduction of videos from a new site or sites with insufficient data.

This research proposes a novel, deep-learning-based method for catheter path reconstruction in high-dose-rate prostate brachytherapy. The proposed method incorporates a lightweight spatial attention-based convolutional neural network to accurately segment volumetric ultrasound images in near real-time and a 3D catheter path reconstruction algorithm. Using automated data augmentation, structured dropout, and batch normalization techniques, the model training pipeline was designed to be robust to various issues, including overfitting and limited annotated data. The model detected 98% of the tested catheter paths and achieved faster inference times than existing methods. This 3D path-tracking pipeline has the potential to significantly improve the accuracy and efficiency of high-dose-rate prostate brachytherapy.

The rise of minimally invasive surgery (MIS) can mainly be attributed to the exponential growth in technology and the evolution of laparoscopic instrumentation over the past two decades. Deep Learning has had a major impact on a range of surgical procedures, such as optimizing workflow, surgical training, intraoperative assistance, patient safety, and efficiency. However, it also requires high computational and memory resources. There has been a lot of research into optimizing deep learning models to balance performance and accuracy under limited resources. Techniques like post-training quantization can significantly reduce model size and latency. In this paper, we explore TensorRT-based techniques with Yolo-based instrument detection technique on edge devices to achieve real-time inference without compromising accuracy under limited compute. This paper gives a review looking at how deep learning and edge computing intersect and how to optimize deep learning for edge devices with limited resources.

The examination of suspicious peripheral pulmonary lesions (PPLs) is an important part of lung cancer diagnosis. The physician performs bronchoscopy and then employs radial-probe endobronchial ultrasound (RP-EBUS) to examine and biopsy suspect lesions. Physician skill, however, plays a significant part in the success of these procedures. This has driven the introduction of image-guided bronchoscopy systems. Unfortunately, such systems do not provide guidance on how to use RP-EBUS. Our recently proposed image-guided bronchoscopy system does offer guidance for both the bronchoscope and RP-EBUS. Unfortunately, the system relies on a time-consuming, error-prone, interactive approach to generate device maneuvers. We propose a methodology that performs automatic planning of all device maneuvers and that is fully integrated into our system. Results show that the planning the device maneuvers now occurs in under one second.

Accurate and reliable medical image analysis, particularly in lung nodule segmentation, plays a crucial role in data-driven healthcare assistance technologies. Current evaluation metrics for segmentation algorithm performance lack specificity to individual use cases and may not adequately assess the accuracy of 2D segmentation in context. In this preliminary work, we propose a novel evaluation approach that incorporates use case-specific evaluation metrics, focusing particularly on the spatial congruence and mass center accuracy of the nodule segmentation in the context of robot-assisted image-guided interventions. By promoting the adoption of use case-specific metrics, we aim to improve the performance of segmentation algorithms, and ultimately, the outcome of critical healthcare procedures.

In this work, we propose a system for taking pre-operative and post-operative photos for patients in a facial reconstructive surgery. The system allows the pair of images to be taken from very similar perspectives, making further analysis easier. A mobile device is used to track the pose of the patient and post-operative photo is taken automatically when the head pose matches the pre-operative pose. We evaluated the system by comparing to using only the front-facing camera of a phone and we found that we were able to more accurately register the pre- and post-operative images using the proposed pipeline.

Cone Beam CT (CBCT) has become a routine clinical imaging modality in interventional radiology. Extended Field of View (FOV) CBCT is of great clinical importance for many medical applications, especially for cases where the Volume of Interest (VOI) is outside the standard FOV. In this study, we investigate FOV extension by optimizing customized source-detector CBCT trajectories using Simulated Annealing (SA) algorithm, a heuristic search optimization algorithm. The SA algorithm explores different elliptical trajectories within a given parameter space, attempting to optimize image quality in a given VOI. Kinematic constraints (e.g., due to collisions of the imager with the patient or other medical devices) are taken into account when designing the trajectories. Our experimental results have shown that our proposed customized trajectories can lead to an extended FOV and enable improved visualization of anatomical structures in extreme positions while taking into account the available kinematic constraints.

Modern spinal procedures are moving to smaller exposures for patient welfare. The surgical scene in these procedures is constantly changing during surgery due to intervertebral motion. Hand-held stereovision systems can be used to drive a deformation model to generate an updated CT using intraoperative data, however they require a large spine exposure for robust data collection. This study uses simulated narrow exposures to test the robustness of the deformation model. The 3 HHS datasets were manually segmented in the following ways: out to the transverse process, out to the facet joints, and out to the lamina. The mean values for L2 norms for the transverse process segmentation data, facet segmentation data, and lamina segmentation are 2.04±1.10mm, 3.18±2.18mm, and 4.59±2.28mm respectively.

This study introduces an AR system for nail implantation in complex tibial fractures, using a CNN to accurately segment bone and metal objects from pre- and post-operative CT data. Successful segmentation of bone and metal, even in cases with artifacts, is demonstrated. Integration into clinical workflows could enhance surgical outcomes and safety by reducing radiation exposure and intervention time.

Monocular depth estimation is a popular task in natural image datasets. Due to the difficulty of obtaining true depth labels for the bronchus and the characteristics of the bronchial image such as scarcity of texture, smoother surfaces and more holes, there are many challenges in bronchial depth estimation. Hence, we propose to use a ray tracing algorithm to generate virtual images along with their corresponding depth maps to train an asymmetric encoder-decoder transformer network for bronchial depth estimation. We propose the edge-aware unit to enhance the awareness of the bronchial internal structure considering that the bronchus has few texture features and many edges and holes. And asymmetric encoder-decoder is proposed by us for multi-layer features fusion. The experimental results of the virtual bronchial demonstrate that our method achieves the best results in several metrics, including MAE of 0.915 +- 0.596 and RMSE of 1.471 +- 1.097.

In this study, we evaluate some of the key training aspects of unsupervised optical flow in soft tissue keypoint tracking, we use the unsupervised ARFlow method with the SurgT challenge dataset and metrics to evaluate accuracy and robustness. Our results showed comparable trends for the loss functions to "What Matters in Unsupervised Optical Flow", but different behavior in other components. These results point to potential bottlenecks that impact overall performance and need to be addressed for further improvements in tracking keypoints in laparoscopic videos.

The generation of 3D printed, patient specific, biomechanically accurate physical models for planning treatment of Adult Spinal Deformities (ASD) and other ailments has the potential to be automated for the ease and benefit of medical professionals. This study by researchers at the University at Buffalo Departments of Biomedical Engineering and Neurosurgery in cooperation with the Canon Stroke and Vascular Research Center and University at Buffalo's Jacobs Medical School proposes automation of segmentation of spinal CTs via 3D dense u-net and the Keras Deep Learning API for Python.

Laparoscopic surgery is a minimally invasive way of cancer resection, which is expected to increase in number. However, because a typical laparoscope can only receive visible light, there is a risk of accidentally damaging nerves that are similar in color to other tissues. To solve this problem, near-infrared (NIR) light (approximately 700-2,500 nm) is considered to be effective because of its feature; component analysis based on the molecular vibrations specific to biomolecules. Previously, we have developed NIR multispectral imaging (MSI) laparoscopy, which acquires NIR spectrum at 14 wavelengths with a band-pass filter. However, since the wavelength is limited, the optimal wavelength for identification cannot be studied. In this study, we developed the world's first laparoscopic device capable of NIR hyperspectral imaging (HSI) with an increased number of wavelengths. Furthermore, NIR-HSI was conducted in a living pig, and the machine-learning was demonstrated to identify nerves and other tissues; accuracy was 0.907.

This work explores the potential of different head-worn eye-tracking solutions for tele-robotic surgery, as derived metrics from gaze-tracking and pupillometry show promise for cognitive load assessment. Current eye-tracking solutions face challenges in tele-robotic surgery due to close-range interactions, leading to extreme angles of the pupil and occlusion. A matched-user study was performed to compare the effectiveness of the Tobii Pro 3 Glasses and the Pupil Labs Core with regards to the stability of the estimated gaze and pupil diameter. Results show, that both systems perform similarly in both regards without an outdated calibration.

The robotic component artifact influence over the region of surgical interest (ROSI) is to be mitigated for complications as well as to provide accurate guidance for the surgeon. This study defines a large MRI phantom design for specimen submersion to verify and quantify artifact generation from robotic system components as well as provide a better visualization platform for robotic performance during preliminary testing and evaluation. The main topics of focus for the phantom design are fluid selection, phantom shape, phantom containment material, and 3D printed artifact measurement evaluation grids. After image equalization from the acquired MRI images, the image uniformity was determined through the ACR method while the SNR and CNR values were calculated in Fiji. The results illustrated the preferred environmental constraints according to the main topics: food grade mineral oil, cylindrical, motion artifact interference, and PETG 3D printed grid.

The article introduces a new non-invasive quantitative method for evaluating regional diaphragmatic structure and function in pediatric patients with thoracic insufficiency syndrome (TIS), before and after VEPTR surgery. Despite minimal changes in diaphragm shape, we observed significant improvement in diaphragm motion after surgery, indicating a positive impact on diaphragmatic function. This promising approach offers comprehensive insights into TIS patient management, potentially leading to improved treatment planning and patient outcomes.

Providing manual formative feedback in minimally invasive surgery training requires an expert observer whose availability is limited. Using a simple deep learning method and descriptive motion features, we developed an automatic method to assess technical surgical skills. Our method outperforms the state-of-the-art technique for robotic minimally invasive surgery skills assessment and is also suitable for non-robotic laparoscopic training. As opposed to most methods that classify students in broad skill level categories, we focused on predicting the ratings of specific surgical technical skills. Therefore students can know where to direct their training efforts.

The advent of computed tomography significantly improves patients’ health regarding diagnosis, prognosis, and treatment planning and verification. However, tomographic imaging escalates concomitant radiation doses to patients, inducing potential secondary cancer by 4%. We demonstrate the feasibility of a data-driven approach to synthesize volumetric images using patients’ surface images, which can be obtained from a zero-dose surface imaging system. This study includes 500 computed tomography (CT) image sets from 50 patients. Compared to the ground truth CT, the synthetic images result in the evaluation metric values of 26.9 ± 4.1 Hounsfield units, 39.1 ± 1.0 dB, and 0.965 ± 0.011 regarding the mean absolute error, peak signal-to-noise ratio, and structural similarity index measure. This approach provides a data integration solution that can potentially enable real-time imaging, which is free of radiation-induced risk and could be applied to image-guided medical procedures.

Structural heart disease (SHD) is a recently recognized subset of heart disease, and minimally invasive, transcatheter treatments for SHD rely heavily on guidance from multiple imaging modalities. Mentally integrating the information from these images can be challenging during procedures and can take up time and increase radiation exposure. This study used the free Unity graphics engine and tailored LabVIEW and Python algorithms, along with deep learning, to merge echocardiography, CT-derived 3D heart models, and fiber optic shape sensing data with fluoroscopic imaging. Tests were performed on a patient specific ballistic gel heart model. This is the first attempt at fusing the above four imaging modalities together and can pave the way for more advanced guidance techniques in the future.

It has been shown that individuals may develop vertebral compression fracture (VCF) after stereotactic body radiotherapy (SBRT), and it is necessary to identify possible risk groups prior to performing SBRT. In this study, we propose a multi-modal deep network for risk prediction of VCF after SBRT that uses clinical records, CT images, and radiotherapy factors altogether without explicit feature extraction. The retrospective study was conducted on a cohort of 131 patients who received SBRT for spinal bone metastasis. A 1-D feature vector was generated from clinical factors. We augmented and cropped a 3-D patch of the lesion area from pretreatment CT images and planning dose images. We designed a two-branch multi-modal deep learning network. From the k-fold validation and ablation study, our proposed multi-modal network showed the highest performance with an area under the curve (AUC) of 0.7468 and average precision(AP) of 0.7134. The prediction model would play a valuable role not only in the treated patients’ welfare but also in the treatment planning for those patients.

Tracked intraoperative ultrasound (iUS) is growing in surgical use. Accurate spatial calibration is essential to enable iUS navigation. Utilizing sterilizable ultrasound probes introduces new calibration challenges that can be solved by a robust, efficient and user independent calibration technique to be performed sterilely at the time of surgery in the sterile field. This study investigates the effect of pose variation on the accuracy of a plane-based ultrasound calibration technique through analysis of a comprehensive dataset. The location of the tracked tool attached to the probe is decoupled into 6 degrees of freedom and monitored during data acquisition. An intelligent line detection algorithm is deployed to US video feed during acquisition. The range of the degrees of freedom of the data set are iteratively reduced to study its effect on the spatial calibration accuracy. Analysis of reducing the translation and rotation range increased both TRE and standard deviation emphasizing the need for increasing pose variation during calibration data acquisition to ensure high calibration accuracy This work facilitates a larger development toward sterile time-of-surgery calibration.

This paper describes an adaptive octree cube refinement method for deformable organ models. Surgical simulation is one of the most promising ways for surgical training. Laparoscopic surgery simulators are already in practical use and have been evaluated for their effectiveness. To realize a high-quality simulator, it is important to efficiently process organ deformation models. In this study, we extend adaptive mesh refinement and apply it to an octree cube structure. Refinement of the structure is performed based on the grasping position. This approach improves the resolution of the octree around the grasping position. In addition, it makes it easier to detect interference between the grasp model and the high-resolution grid of the octree. Simulation results showed there were 199 cubes before and 339 cubes after refinement, and the FPS decreased from 44.1 FPS to 32.4 FPS, which is still within real-time processing.

Surgical instrument segmentation in laparoscopy is vital for computer-assisted surgery. While Deep Learning has made strides, laparoscopic dynamics pose challenges. nnU-Net, excelled in 33 of 53 global challenges. Easy auto-configuration and low expertise needs made it a popular base. This study explores Optical Flow's (OF) use to enhance nnU-Net. OF estimates motion and depth in videos, beneficial for estimating surgical tools' movement. Incorporating OF into nnU-Net, indirectly adds temporal information without altering architecture. OF improves tool detection but blurs differentiation. Its easy integration and low computational demands are key. However, using OF limits augmentations, conflicting with low expertise aim. OF-based nnU-Net improves detection, not differentiation. More research needed for optimal nnU-Net+OF use.

In image-guided neurosurgery, preoperative magnetic resonance (pMR) images are rigidly registered with the patient’s head in the operating room. Image-guided systems incorporate this spatial information to provide real-time information on where surgical instruments are located with respect to preoperative imaging. The accuracy of these systems become degraded due to intraoperative brain shift. To account for brain shift, we previously developed an image-guidance updating framework that incorporates brain shift information acquired from registering intraoperative stereovision (iSV) surface with the pMR surface to create an updated magnetic resonance image (uMR). To register the iSV surface and the pMR surface, the two surfaces must have some matching features that can be used for registration. To capture features falling outside of the brain volume, we have developed a method to improve feature extraction, which involves performing a selective dilation in the region of the stereovision surface. The goal of this method is to capture useful features that can be use to improve image registration.

There has been growing interest in regional biomarkers to quantify the health of lung parenchyma. Image registration through the use of dynamic imaging has emerged as a powerful tool to assess the kinematic and deformation behavior of lung parenchyma during respiration. However, the difficulty in validating the results provided by image registration has limited its use in clinical settings. To overcome this barrier, we propose to generate a phantom computed tomography (CT) image based on a finite-element (FE) mesh. We believe that this will serve as an essential step towards the implementation of dynamic imaging and image registration in clinical settings to assess regional deformation in patients as a diagnostic and risk-stratification tool.

Percutaneous Microwave ablation (MWA) is a minimally invasive technique to treat liver tumors. The Penne bioheat equation explains heat distribution in tissues, including factors like blood perfusion rate (BPR) and metabolic heat (MH). We employed 3D patient-specific models and sensitivity analysis to examine how BPR and MH affect MWA results. Numerical simulations using a triaxial antenna and 65 Watts power on tumors demonstrated that lower BPR led to less damage and complete tumor destruction. Models without MH had less liver damage. The study highlights the importance of tailored ablation parameters for personalized treatments, revealing the impact of BPR and MH on MWA outcomes.

Nonrigid surface-based soft tissue registration is crucial in surgical navigation systems yet faces challenges due to complex surface structures of intra-operative data. By employing nonrigid registration, surgeons can achieve a real-time visualization of the patient’s complex pre- and intra-operative anatomy in a common coordinate system to improve navigation accuracy. To address limited access to liver registration methods, we compare the robustness of the three open-source optimization-based nonrigid registration methods to a reduced visibility ratio (reduced partial views of the surface) and an increasing deformation level (mean displacement), reported as the root mean square error (RMSE) between the pre-and intra-operative liver surface meshed following registration. The Gaussian Mixture Model - Finite Element Model (GMM-FEM) method consistently yields a lower post-registration error than the other two tested methods in the presence of both reduced visibility ratio and increased intra-operative surface displacement, therefore offering a potentially promising solution for pre- to intra-operative nonrigid liver surface registration.

Cochlear implants (CIs) are neural prosthetics for patients with severe-to-profound hearing loss. CIs induce hearing sensation by stimulating auditory nerve fibers (ANFs) using an electrode array that is surgically implanted into the cochlea. After the device is implanted, an audiologist programs the CI processor to optimize hearing performance. Without knowing which ANFs are being stimulated by each electrode, audiologists must rely solely on patient performance to inform programming adjustments. Patient-specific neural stimulation modeling has been proposed to assist audiologists, but requires accurate localization of ANFs. In this paper, we propose an automatic neural-network-based method for atlas-based localization of the ANFs. Our results show that our method is able to produce smooth ANF predictions that are more realistic than those produced by a previously proposed semi-manual localization method. Accurate and realistic ANF localizations are critical for constructing patient-specific ANF stimulation models for model guided CI programming.

Title: Enhancing Patient Education on Neurosurgery through Mixed Reality Technology Author List: Yanwen Wang, Yushen Chen, Jonathan Collier, David Hocking, Terry M. Peters and Elvis C.S.Chen Aims: This study examines the potential of Mixed Reality (MR) technology to enhance patient understanding and autonomy in the informed consent process in neurosurgery.

As surgical robotics are made progressively smaller, and their actuation systems simplified, the opportunity arises to re-evaluate how we integrate them into operating room workflows. Several research groups have shown that robots can be made so small and light that they can become hand-held tools. This hand-held paradigm enables robots to fit much more seamlessly into existing clinical workflows. In this paper, we compare an onboard user interface approach against the traditional offboard approach. In the latter, the surgeon positions the robot, and a support arm holds it in place while the surgeon operates the manipulators using the offboard surgeon console. The surgeon can move back and forth between the robot and the console as often as desired. Three experiments were conducted, and results show that the onboard interface enables statistically significantly faster performance in a point-touching task performed in a virtual reality environment.

CT-guided renal tumor ablations have been considered an alternative to treat small renal tumors, typically 4 cm in size or smaller, especially for patients who are ineligible to receive nephron-sparing surgery. For this procedure, the radiologist must compare the pre-operative with the post-operative CT to determine the presence of residual tumors. Distinguishing between malignant and benign kidney tumors poses a significant challenge. To automate this tumor coverage evaluation step and assist the radiologist in identifying kidney tumors, we proposed a coarse-to-fine U-Net-based model to segment kidneys and masses. We used the TotalSegmentator tool to obtain an approximate segmentation and region of interest of the kidneys, which was inputted into our 3D segmentation network trained using the nnUNet library to fully segment the kidneys and masses within them. Our model achieved an aggregated DICE score of 0.777 on testing data, and on local CT kidney data with tumors collected from the London Health Sciences University Hospital, our model achieved a DICE score of 0.7 for tumour segmentation. Our results indicate the model will be useful for tumour identification and evaluation.

With a growing role for virtual simulation skills training, we propose a low-cost solution to automate creation of high-fidelity 3D holographic hand animations using motion capture data from the Oculus Quest 2 mixed reality headset for surgical skills training. Using this methodology, we successfully developed a 3D holographic animation of one-handed knot ties used in surgery. With regards to the quality of the produced animation, our qualitative pilot study demonstrated comparable successful learning of knot-ties from the holographic animation to in-person demonstration. Furthermore, participants found learning knot-ties from the holographic animation to be easier, more effective, were more confident in mastery of the skill in comparison to in-person demonstration, and also found the animation comparable to real hands showing promise for application in surgical skills training applications.

Reconstruction of stereoendoscopic video has been explored for guiding minimally-invasive procedures across many surgical subspecialties, and may play an increasingly important role in navigation as stereo-equipped robotic systems become more widely available. Capturing stereo video for the purpose of offline reconstruction requires dedicated hardware, a mechanism for temporal synchronization, and video processing tools that perform accurate clip extraction, frame extraction, and lossless compression for archival. This work describes a minimal hardware setup comprising entirely off-the-shelf components for capturing video from the da Vinci and similar 3D-enabled surgical systems. Software utilities are also provided for synchronizing data collection and accurately handling captured video files. End-to-end testing demonstrates that all processing functions (clipping, frame cropping, compression, un-compression, and frame extraction) operate losslessly, and can be combined to generate reconstruction-ready stereo pairs from raw surgical video.

CT image synthesis from MR images is necessary for MR-only treatment planning, MRI-based quality assurance, and treatment assessment in radiation therapy (RT). For pediatric cancer patients, reducing ionizing radiation from CT scans is preferred for which MRI-based RT planning and assessment is truly beneficial. For accurate pediatric CT image synthesis, we investigated a 3D conditional generative adversarial network (cGAN)-based transfer learning approach due to the lack of sufficient pediatric data compared to adult data. Our model was first trained using adult data with downscaling to simulate pediatric data, followed by fine-tuning on a smaller number of pediatric data. The proposed 3D cGAN-based transfer learning was able to accurately synthesize pediatric CT images from MRI, allowing us to realize pediatric MR-only RT planning, QA, and treatment assessment.

For those experiencing severe-to-profound sensorineural hearing loss, the cochlear implant (CI) is the preferred treatment. Augmented reality (AR) surgery may improve CI procedures and hearing outcomes. Typically, AR solutions for image-guided surgery rely on optical tracking systems to register pre-op planning information to the display so that hidden anatomy or other information can be overlayed co-registered with the view of the surgical scene. In this work, our goal is to develop a method that permits direct 2D-to-3D registration of the microscope video to the pre-operative CT scan without the need for external tracking equipment. Our proposed solution involves surface-mapping a portion of the incus in the video and determining the pose of this structure relative to the surgical microscope by solving the perspective-n-point pose computation to achieve 2D-to-3D registration. This registration can then be applied to pre-operative segmentation of other hidden anatomy as well as the planned electrode insertion trajectory to co-register this information for AR display.

The transoral approach to resecting oral and oropharyngeal tumors is associated with lower morbidity than open surgery, but is associated with a high positive margin rate. When margins are positive, it is critical that resection specimens be accurately oriented in anatomical context for gross and microscopic evaluation, and also that surgeons, pathologists, and other care team members share an accurate spatial awareness of margin locations. With clinical interest in digital pathology on the rise, this work outlines a proposed framework for generating 3D specimen models intraoperatively via robot-integrated stereovision, and using these models to visualize involved margins in both ex vivo (flattened) and in situ (conformed) configurations. Preliminary pilot study results suggest that stereo specimen imaging can be easily integrated into the transoral robotic surgery workflow, and that the expected accuracy of raw reconstructions is around 1.60mm. Ongoing data collection and technical development will support a full system evaluation.

Point of Care ultrasound (POCUS) is a term to describe a field of ultrasound imaging that is portable, fast, and accessible. Such a device can now perform an echocardiogram while connected to a smartphone. While the accessibility of performing a test has been greatly improved, expertise is still required to provide usable results and diagnoses. The goal of this study is to improve the clinical utility of mobile ultrasound echocardiograms with AI machine learning. By integrating artificial intelligence into this workflow, feedback can be given to the provider during its operation to maximize the usability of the ultrasound data and allow more tests to be performed properly. The Intel GETi framework was used to create computer vision models that could quantify the readability of frames taken from an echocardiogram. These models determine the quality and the orientation of each frame. Feedback from these models can alert the user to proper positioning and technique to gather good ultrasound data. The accuracy of the models ranges from 77% - 99%, depending on factors such as how the model was trained and the ratio of training to testing data. Testing accuracy can also be improved with

Medical imaging data is often used inefficiently, and this happens most often for patients with abnormal imaging who require a complex procedure. This talk describes those patients, how their medical images undergo Computer Aided Design (CAD), and how that data reaches a Final Anatomic Realization, one of which is 3D printing. This talk highlights “keys” to “unlock” value when this clinical service line is performed in a hospital, and the critical role for medical engineers who work in that infrastructure. The talk includes medical oversight, data generation, and a specific, durable definition of value for medical devices that are 3D printed in hospitals. The talk also includes clinical appropriateness, and how it folds into accreditation for 3D printing in hospitals and universities. Up to the minute information on reimbursement for medical devices that are 3D printed in hospitals and universities will be presented.

Increasingly powerful technological developments in surgery such as modern operating rooms (OR), featuring digital and interconnected as well as robotic devices provide a huge amount of valuable data which can be used to improve patient therapy. Although a lot of data is available, the human ability to use these possibilities especially in a complex and time-critical situation such as surgery is limited and is extremely dependent on the experience of the surgical staff. This talks focuses on AI-assisted surgery with a specific focus on analysis of intraoperative video data. The goal is to democratize surgical skills and enhance the collaboration between surgeons and cyber-physical systems by quantifying surgical experience and make it accessible to machines. Several examples to optimize the therapy of the individual patient along the surgical treatment path are given. Finally, remaining challenges and strategies to overcome them are discussed.

Laparoscopic and robotic surgery, as a type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that comprises of a high-resolution color camera and a snapshot hyperspectral camera, and we employ super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experiment results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.

Augmented reality (AR) has seen increased interest and attention for its application in surgical procedures. While previous works have utilized pre-operative imaging such as computed tomography or magnetic resonance images, registration methods still lack the ability to accurately register deformable anatomical structures across modalities and dimensionalities. This is especially true of minimally invasive abdominal surgeries due to limitations of the monocular laparoscope. Surgical scene reconstruction is a critical component towards AR-guided surgical interventions and other AR applications such as remote assistance or surgical simulation. In this work, we show how to generate a dense 3D reconstruction with camera pose estimations and depth maps from video obtained with a monocular laparoscope utilizing a state-of-the-art deep-learning-based visual simultaneous localization and mapping (vSLAM) model. The proposed method can robustly reconstruct surgical scenes using real-time data and provide camera pose estimations without stereo or other sensors, which increases its usability and is less intrusive.

Ureteroscopic intrarenal surgery is a surgical operation in the upper urinary tract, commonly for kidney stones or upper tract urothelial carcinoma (UTUC), which comprises the passage of a flexible ureteroscope through the ureter into the kidney. Flexible ureteroscopes (fURS) are limited by their visualization ability and fragility, which can cause missed regions during the procedure in hard-to-visualize locations and/or due to scope breakage. This contributes to a high recurrence rate for both kidney stone and UTUC patients. We introduce an automated patient-specific analysis for determining viewability in the renal collecting system using pre-operative CT scans.

Depth estimation in surgical video plays a crucial role in many image-guided surgery procedures. However, it is difficult and time consuming to create depth map ground truth datasets in surgical videos due in part to inconsistent brightness and noise in the surgical scene. Therefore, building an accurate and robust self-supervised depth and camera ego-motion estimation system is gaining more attention from the computer vision community. Although several self-supervision methods alleviate the need for ground truth depth maps and poses, they still need known camera intrinsic parameters, which are often missing or not recorded. Moreover, the camera intrinsic prediction methods in existing works depend heavily on the quality of datasets. In this work, we aim to build a self-supervised depth and ego-motion estimation system which can predict not only accurate depth maps and camera pose, but also camera intrinsic parameters. We propose a cost-volume-based supervision approach to give the system auxiliary supervision for camera parameters prediction.

In image-guided open cranial surgeries, brain deformation after dural opening degrades image guidance accuracy. A biomechanical model can update pre-operative MR images to match intraoperative conditions using sparse data and boundary conditions from intraoperative stereovision (iSV). Current methods use manual region of interest (roi) for cortical surface segmentation, which requires user expertise and additional time during surgery for intraoperative image updating, showing the need for an automatic method. We adopted the Fast Segment Anything Model (FastSAM) for such task. We evaluated segmentation results against ground truth, and assessed image updating performance for one clinical case. Additionally, we compared the performance of FastSAM segmentation against a manual segmentation, an automatic segmentation using a UNET model in terms of segmentation accuracy and image updating accuracy. Preliminary results show that the performance of FastSAM is similar to the current manual segmentation method and shows promise to improve efficiency in the operating room by eliminating user input and expertise.

Radiation therapy (RT) planning for pediatric brain cancer is a challenging task. RT plans are typically optimized using CT, thus exposing patients to ionizing radiation. Manual contouring of organs-at-risk (OARs) is time-consuming, particularly difficult due to the small size of brain structures, and suffers from inter-observer variability. While numerous methods have been proposed to solve MR to CT image synthesis or OAR segmentation separately, there exist only a handful of methods tackling both problems jointly, even less specifically developed for pediatric brain cancer RT. We propose a multi-task convolutional neural network to jointly synthesize CT from MRI and segment OARs (eyes, optic nerves, optic chiasm, brainstem, temporal lobes, and hippocampi) for pediatric brain RT planning.

Measuring errors in neuro-interventional pointing tasks is critical to better evaluating human experts as well as machine learning algorithms. If the target may be highly ambiguous, different experts may fundamentally select different targets, believing them to refer to the same region, a phenomenon called an error of type. This paper investigates the effects of changing the prior distribution on a Bayesian model for errors of type specific to transcranial magnetic stimulation (TMS) planning. Our results show that a particular prior can be chosen which is analytically solvable, removes spurious modes, and returns estimates that are coherent with the TMS literature. This is a step towards a fully rigorous model that can be used in system evaluation and machine learning.

In an effort to improve hearing outcomes for cochlear implant recipients, many computational models of electrical stimulation in the inner ear have been developed to provide clinicians with objective information that can assist their decision-making. These models exist on a range of complexity, including highly-detailed, patient-specific models designed to more accurately simulate an individual’s experience. One limitation of these models is the large amount of data required to create them, with the resulting model being highly optimized to these single sets of measurements. Thus, it is desirable to create a new model of equal or better quality that does not require this data to create the model and that is adaptable to new sets of clinical data. In this work, we present a methodology for one component of such a model, which uses simulations of voltage spread in the cochlea to estimate patient-specific electrical tissue resistivity values.

This research addresses the challenges of liver resection planning and execution, leveraging intraoperative ultrasound (IOUS) for guidance. We propose an AI-driven solution to enhance real-time vessel identification (inferior vena cava (IVC), right hepatic vein (RHV), left hepatic vein (LHV), and middle hepatic vein (MHV)) using a visual saliency approach that integrates attention blocks into a novel U-Net model with integrated attention blocks. The study encompasses a dataset of IOUS video recordings from 12 patients, acquired during liver surgeries. Employing leave-one-out cross-validation, the model achieves mean dice scores of 0.88 (IVC), 0.72 (RHV), 0.53 (MHV), and 0.78 (LHV). This innovative approach holds the potential to revolutionize liver surgery by enabling precise vessel segmentation, with future prospects including broader vasculature segmentation and real-time application in the operating room.

Brachytherapy is a common treatment technique for cervical cancer diagnoses. Radiation is delivered using specialized applicators or needles that are inserted within the patient using medical imaging guidance. However, advanced imaging modalities may be unavailable in underfunded healthcare centers, suggesting a need for accessible imaging techniques during brachytherapy procedures. This work focuses on the development and validation of a spatially tracked mechatronic arm for 3D trans-abdominal and trans-rectal ultrasound imaging. The arm will allow automated acquisition and inherent registration of two 3D ultrasound images, resulting in a fused image volume of the whole female pelvic region. The results of our preliminary testing demonstrate this technique as a suitable alternative to advanced imaging for providing visual information to clinicians during brachytherapy applicator insertions, potentially aiding in improved patient outcomes.

Percutaneous nephrostomy is a commonly performed procedure to drain urine to provide relief in patients with hydronephrosis. Current percutaneous nephrostomy needle guidance methods can be difficult, expensive, or not portable. We propose an open-source based real-time 3D anatomical visualization aid for needle guidance with live ultrasound segmentation and 3D volume reconstruction using deep learning and free, open-source software . Participants performed needle insertions with visualization aid and conventional ultrasound needle guidance. Visualization aid guidance showed a significantly higher accuracy, while needle insertion time and success rate were not statistically significant at our sample size. Participants mostly responded positively to visualization aid needle guidance and 80% found it easier to use than ultrasound needle guidance. We found that real-time 3D anatomical visualization aid for needle guidance produced increased accuracy and an overall mostly positive experience.

This research introduces an innovative mirror-based ultrasound system for hand gesture classification and explores data analysis using Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures. Hand gesture recognition with ultrasound has gained interest in prosthetic control and human-computer interaction. Traditional methods used for hand gesture estimation involve placing an ultrasound probe perpendicular to the forearm causing discomfort and interference with arm movement. To address this, a novel approach utilizing acoustic reflection is proposed wherein a convex ultrasound probe is strategically positioned in parallel alignment with the forearm, and a mirror is placed at a 45-degree angle for transmission and reception of ultrasound waves. This positioning enhances stability and reduces arm strain. CNNs and ViT are employed for feature extraction and classification. The system's performance is compared to the traditional perpendicular method, demonstrating comparable results. The experimental outcomes showcase the potential of the system for efficient hand gesture recognition.

This paper presents the development and evaluation of an educational system for training and assessing student skills in ultrasound-guided interventional procedures. The system consists of an ultrasound needle guidance system which overlays virtual needle trajectories on the ultrasound screen, and custom anatomical phantoms tailored to specific anesthesiology procedures. The system utilizes artificial intelligence-based optical needle tracking. It serves two main functions: skill evaluation, providing feedback to students and instructors, and as a learning tool, guiding students in achieving correct needle trajectories. The system was evaluated in a study with nursing students, showing significant improvements in guided procedures compared to non-guided ones.

Image perception, observer performance, and technology assessment have driven many of the advances in breast imaging. Technology assessment metrics were used to develop mammography systems, first with screen-film mammography and then to digital mammography and digital breast tomosynthesis. To optimize these systems clinically, it became necessary to determine what type of information a radiologist needed to make a correct diagnosis. Image perception studies helped define what spatial frequencies were necessary to detect breast cancers and how different sources of noise affected detectability. Finally, observer performance studies were used to show that advances in the imaging system led to better detection and diagnoses by radiologists. In parallel to these developments, these three concepts were used to develop computer-aided diagnosis systems. In this talk, I will highlight how image perception, observer performance, and technology assessment were leveraged to produce technologies that allow radiologists to be highly effective in detecting breast cancer.

While modern medical imaging coupled to contemporary methods in machine learning has allowed for dramatic expansions of diagnostic discrimination, similar advances in procedural medicine have lagged due to systematic barriers associated with the intrinsic data limitations of the procedural environment. This reality motivates many questions, both exhilarating and provocative. The assertion in this talk is that treatment platform technologies of the future will need to be intentionally designed for the dual purpose of treatment and discovery. While it is difficult to be prescient on the forms that these forward-thinking systems will take, it is clear that new requirements associated with data integration/acquisition, automation, real-time computation, and cost will likely be critical factors. Exemplar surgical and interventional technologies will be discussed that involve complex biophysical models, methods of automation and procedural field surveillance, efforts toward data-driven procedures and therapy forecasting, and approaches integrating disease phenotypic biomarkers. The common thread to the work is the use of computational models driven by sparse procedural data as a constraining environment to enable guidance and therapy delivery.

Image registration often requires retrospective tuning of model parameters to optimize registration accuracy. However, these procedures may not produce results that optimally generalize to inter- and intra-dataset variabilities. We present a parameter estimation framework based on the Akaike Information Criterion (AIC) that permits dynamic runtime adaptation of model parameters by maximizing the informativeness of the registration model against the specific data constraints available to the registration. This parameter adaptation framework is implemented in a frequency band-limited reconstruction approach to efficiently resolve modal harmonics of soft tissue deformation in image registration. Our approach automatically selects optimal model complexity via AIC to match informational constraints via a parallel-computed ensemble model that achieves excellent TRE without the need for any hyperparameter tuning.

Image registration is a cornerstone of medical imaging, ensuring spatial alignment across different coordinate systems. While deformable registration offers the ability to accommodate complex anatomical changes, it introduces significant computational demands when using traditional iterative optimization methods. Learning-based solutions like VoxelMorph have made progress, but they still consume considerable resources. In response, we introduce GhostMorph, a novel framework inspired by the GhostNet architecture. Applied to the Liver Tumor Segmentation Benchmark (LiTS) dataset, GhostMorph demonstrates both competitive performance and improved computational efficiency compared to leading methods. This balance of speed and accuracy positions GhostMorph as an invaluable tool, especially in resource-limited clinical settings.

We propose an uncertainty-aware model for accurate deformation fields generation and risk estimation via a joint synthesis and registration network. By framing warping field prediction as a pixel-wise regression problem, we employ pixel-wise evidential deep learning to predict uncertainties. Visualized uncertainty maps revealed a strong correlation between high warping magnitude and uncertainty. Numeric outcomes on segmentation maps substantiated the benefit of uncertainty integration, yielding improvements significantly better than the training without uncertainty, which shows that introducing uncertainty to the registration network holds great promise.

Although Cochlear Implants (CIs) have been remarkably successful at restoring sound sensation, the electroneural interface is typically unknown to audiologists who tune the CI programming. Thus, many programming sessions are needed and usually lead to suboptimal results. Previously, our group has developed an ANF localization approach in order to simulate the neural response triggered by CIs. That method relies heavily on manual adjustment and is error prone. In this work, we introduce a fully automatic and accurate ANF localization method, where the peripheral and central axon of an ANF can be estimated individually based on five sets of automatically generated landmarks; the fast marching method can be used to find geodesic paths between landmarks; and cylindrical coordinate systems can be constructed based on the landmarks in order to smoothly interpolate trajectories between landmarks. Experiments show that our proposed method outperforms the original method and achieves impressive performance qualitatively and quantitatively.

Rib fractures occur in 10% of all trauma patients. Rib fractures can be observed in X-ray and CT scans allowing for better surgical planning. However, translating the surgical plan to the operating table through mental mapping remains a challenging task. Using augmented reality (AR), a preoperative plan can be intraoperatively visualized in the field of view of the surgeon, allowing for a more accurate determination of the size and location of the incision for optimum access to the fractured ribs. This study aims to evaluate the use of AR for guidance in rib fracture procedures. To that end, an AR system using the HoloLens 2 was developed to visualize surgical incisions directly overlayed on the patient. To evaluate the feasibility of the system, a torso phantom was scanned for preoperative planning of the incision lines. A user study with 13 participants was conducted to align the preoperative model and delineate the visualized incisions. For a total of 39 delineated incisions, a mean distance error of 3.6 mm was achieved. The study shows the potential of using AR as an alternative to the traditional palpation approach for locating rib fractures, which has an error of up to 5 cm.

Image-guided spine surgery relies on surgical trackers for real-time localization of surgical instruments, which are susceptible to local changes in anatomy due to patient repositioning or changes imparted during the procedure. This study presents an ultrasound-guided approach and an integrated real-time system for verifying and recovering tracking accuracy following spinal deformations. The approach combines deep-learning segmentation of the posterior vertebral cortices with a multi-step point-to-surface registration to map reconstructed US features to the 3D CT image. The solution was evaluated in cadaver specimens with induced deformation and demonstrated 1.7 ± 0.5 mm of registration error in localizing vertebrae.

We have evaluated the feasibility of using AI-segmented 3D spine ultrasound for the evaluation of scoliosis in pediatric patients. Our system uses motion tracking cameras to track a wireless ultrasound probe and waist belt in conjunction with proprietary SpineUs™ software using neural networks to build a volumetric reconstruction of the spine in real-time on a laptop computer. Transverse process angles from both the ultrasound reconstructions and the patient’s radiographic imaging were compared for five pediatric patients; the results demonstrate a strong linear correlation between the angles obtained from the two imaging methods with minimal variance. The SpineUs™ system shows promise as a potential alternative to x-ray imaging to reduce radiation dose in children, and integrates well into a busy clinic workflow with minimal disruption and additional staff training.

Estimating the severity of scoliosis is time consuming and imprecise. This paper aims to contribute to developing a fully automated method of estimating Cobb angles—a measurement commonly used for scoliosis—through the use of a specialized image segmentation model trained specifically on x-rays to automatically identify vertebrae within x-ray images. This model is named the Adaptive Loss Engine for X-ray Segmentation (ALEXS). Besides training, another method of improving the performance of the ALEXS is altering the original x-ray image without changing the locations of the vertebrae. Sharpening the image and increasing its contrast allowed the ALEXS to identify many more vertebrae than before. Based on the results that were obtained, using the ALEXS combined with altered images produces superior results compared to some previous attempts. These improved methods allow for a more accurate end-to-end process for automatically diagnosing scoliosis.

Statistical surgical process modeling (SPM) presents a powerful framework for computational modeling of workflow, emerging technologies, and analysis of key outcome variables. This work developed statistical SPMs for image-guided spine surgery based on fluoroscopy or CT + navigation, quantifying the benefits of advanced imaging, registration, and planning methods in terms of cycle time, radiation dose, and geometric accuracy.

To bring natural intelligence (NI) in the form of anatomical information into AI methods effectively, we have recently introduced the hybrid intelligence (HI) concept (NI+AI) for image segmentation [20,21]. This HI system has shown remarkable performance/ robustness to image deficiencies. In this paper, we introduce several advances related to modeling of the NI component, so the HI-system becomes substantially more efficient. We demonstrate a 9-40 fold computational improvement in the auto-segmentation task for RT planning via clinical studies obtained from 4 different RT centers, while retaining state-of-the-art accuracy of the previous system in segmenting 28 objects in Thorax and Neck.

This work presents a detailed study of several Image Transformer architectures for the classification of prostate cancer in ultrasound images. It seeks to establish a baseline for the performance of these architectures on cancer detection, both in specific regions of interest (ROI-scale methods) and across the entire biopsy core using multiple ROIs (multi-scale methods). This work also introduces a novel framework for multi-objective learning with transformers by combining the loss for individual ROI predictions with the loss for the core prediction, thereby improving performance over baseline methods.

Wireless Capsule Endoscopy (WCE) offers a promising approach to painless endoscopic imaging of the whole of the small bowel as well as other parts of the Gastro-Intestine through wireless image capturing. This method detects various diseases and pathologies. However, existing capsules are often passive, necessitating active control to enhance localization and identification accuracy. We propose a deep learning-based system to estimate camera position for active endoscopic capsule robots. Unsupervised methods rely on synthetic data, where pose network outputs guide depth predictions. Our network incorporates supervision through image warping based on predicted depth and ego-motion, with a comprehensive loss covering image synthesis, depth, and pose. We introduce a visual transformer into the Visual Odometry pipeline for improved accuracy, building based on the Pyramid Vision Transformer (PVT) structure to address limitations. Our framework incorporates PVTv2, enabling joint training of depth and pose networks for single-image depth regression. Consecutive frames predict depth maps and relative poses, supervised via photometric loss.

We developed an MRI-guidance system for cervical cancer brachytherapy, providing automatic segmentation of organs-at-risk and the high-risk clinical target volume (HR-CTV), and real-time active needle tracking. The segmentation module comprises coarse segmentation step for organ localization, followed by fine segmentation models separately trained for each organs. Size-dependent segmentation is performed for HR-CTV. The needle-tracking module communicates with active stylets, and displays the stylet-tip location/orientation on the MRI in real-time. These modules were incorporated into a brachytherapy treatment planning system, and validated in five cervical cancer cases, demonstrating its clinical utility in increasing procedure efficiency.

Cochlear implant insertion using percutaneous cochlear access involves drilling a single hole through the skull surface and traversing the facial recess, a region approximately 1.0–3.5 mm in width bounded posteriorly by the facial nerve and anteriorly by the chorda tympani. Thus, it is very important that these clinical structures are segmented accurately for trajectory planning. In this work, we propose the use of a conditional generative adversarial network (cGAN) to automatically segment the facial nerve. Our network utilized weakly supervised approach, being trained on a small sample of 12 manually segmented images and an additional 120 images automatically segmented using atlas-based methods. We also leverage endpoint predictions generated by the network to fix noisy or disconnected segmentations by postprocessing the facial nerve skeleton with a minimum cost path search function. Our method generated segmentations with an average mean surface error of only 0.22mm, improving upon the original method by ~50%.

Additive manufacturing (i.e. 3D printing) offers transformative potential in the development of biomedical devices and medical imaging systems, but at the same time presents challenges that continue to limit widespread adoption. Within medical imaging, 3D printing has numerous applications including device design, radiographic collimation, anthropomorphic phantoms, and surgical visualization. Continuous technological development has resulted in improved plastic materials as well as high-throughput fabrication in medical-grade metal alloys. Nonetheless, additive manufacturing has not realized its full potential, due to a number of factors. The regulatory environment for medical devices is geared towards conventional manufacturing techniques, making it challenging to certify 3D-printed devices. Additive manufacturing may still not be competitive when scaled up for industrial production, and the need for post-processing may negate some of the benefits. In this talk, we will describe the current state of 3D printing in medical imaging and explore future potential, including links to 3D design and finite-element modeling.

Forthcoming

Advanced imaging and enhanced visualization techniques are critical for precision surgical interventions that can improve outcomes and save lives. Various imaging modalities and visualization approaches are being developed to aid surgeons to complete procedures with high accuracy, reducing inadvertent errors and reoperation rates. For example, one emerging imaging modality is called hyperspectral imaging that has been increasingly explored for image-guided surgery, including laparoscopic procedures. Augmented reality systems have also been developed to enhance the visualization of human organs and lesions for potential applications in biopsy and surgery. Advanced imaging, enhanced visualization, AI tools, and surgical robotics will revolutionize the operating room of the future.

Tonsillectomy, one of the most common surgical procedures worldwide, is often associated with postoperative complications, particularly bleeding. Tonsil laser ablation has been proposed as a safer alternative; however, its adoption has been limited because it can be difficult for a surgeon to visually control the thermal interactions that occur between the laser and the tissue. In this study, we propose to monitor the ablation caused by a CO2 laser on ex vivo tonsil tissue using photoacoustic imaging. Soft tissue’s unique photoacoustic spectra were used to distinguish between ablated and non-ablated tissue. Our results demonstrate that photoacoustic imaging was able to visualize necrosis formation and calculate the necrotic extent, offering the potential for improved tonsil laser ablation outcomes.

Lung cancer, a leading cause of global cancer-related deaths, can be detected early with a combination of bronchoscopy imaging techniques: white-light (WLB), autofluorescence (AFB), and narrow-band imaging (NBI). However, each modality requires separate tedious manual examinations, leading to no direct link of the three sources. To address this, we propose a framework for multimodal video synchronization and fusion, built into an interactive graphical system. Key airway video-frame landmarks and lesion frames are noted, registered, and fused to a patient's CT-based 3D airway tree model. Our method eases user interaction, is skill-independent, and facilitates true multimodal analysis of a bronchoscopic airway exam.

Free-breathing based quantitative dynamic MRI (QdMRI) provides a practical solution to evaluate the regional dynamics and architecture of thorax for TIS patients. Our current aim is to investigate if QdMRI can also measure thoracic architecture for TIS patients before and after surgery as well as in healthy children. Architectural parameters (3D distances and angles from multiple object centroids) were computed and compared via T-testing. The distance between the right lung and right hemi-diaphragm is larger at end-inspiration than that at end-expiration for TIS patients and healthy children, and after surgery becomes closer to that of healthy children.

Neuroblastoma is the most common type of extracranial solid tumor in children. It is often present in the adrenal and kidney glands. Focused ultrasound is the ideal way of treating tissue that is deep within the body, without using ionizing radiation. The goal of this project was to develop an augmented reality (AR) system for the guided treatment of neuroblastoma with focused ultrasound. Our project’s initial step focused on generating 3D models of neuroblastoma lesions obtained from PET/CT and then displaying them in our AR system. Displaying images in the AR headset involved registering and rendering them in real time. Other unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process the PET/CT images for optimal visualization.

This study aimed to develop a versatile vascular model of a liver with a tumour with applications in training interventional radiologists as well as in research programs to improve embolization therapy. The phantom uses exchangeable, single-use tumour models fabricated using 3D-printing, while mimicking the anatomical, hemodynamic, and radiographic properties of the liver. The modular phantom was used to mimic fluoroscopically guided embolization procedures, demonstrating the visual characteristics of the procedures, including reflux that would lead to non-target embolization. The 3D printed modular phantom design represents an adaptable and versatile model for training and research applications.

This study introduces a simulation approach combining eXtended Finite Element Method (XFEM) for retraction modeling with medical image updates to enhance target visualization and localization accuracy. XFEM simulates tissue retraction, representing complex mechanical behavior during surgery. Utilizing XFEM-derived displacement fields, preoperative images are updated, aiding in visualizing tissue deformation. Experimental validation shows an average displacement error of 2.5 to 3.8 mm, showcasing significant improvement in target accuracy compared to traditional methods.

A limitation of image-guided biopsies is the lack of optimal visualization of the organ and its surrounding structures, leading to missed target lesions. In this study, we propose an augmented reality (AR) system to increase the accuracy of biopsies. Our AR-guided biopsy system uses high-speed motion tracking technology and an AR headset to display a holographic representation of the organ, lesions, and other structures of interest superimposed on real objects. We apply this system to prostate biopsy by incorporating preoperative computed tomography (CT) scans and real-time ultrasound images. This AR system enables clinicians to gain a better understanding of the lesion’s real-time location. With the enhanced visualization of the prostate, lesion, and surrounding organs, surgeons can perform prostate biopsies with increased accuracy. Our AR-guided biopsy system yielded an average targeting accuracy of 2.94 ± 1.04 mm and can be applied for real-time guidance of prostate biopsy as well as other biopsy procedures.

Localized microwave ablation (MWA) of lung tumors is a safe clinically established treatment for non-surgical candidates. This study aims to aid physicians in optimizing patient selection by developing a model to predict local tumor progression (LTP) after MWA treatment. Our model utilizes specialized 3D three-channels data: pre-ablation CT (channel 1), post-ablation CT depicting the resulting ablation zone (channel 2), and overlapping data of the tumor and ablation zone (channel 3). By spatially registering pre- and post-ablation CTs, we establish a clear spatial relationship between the tumor and ablation zone. The model achieved a C-statistic (AUC) of 0.849, outperforming prior work.

Metal artifacts from cryoablation probes interfere with probe placement and ablation monitoring for CT-guided interventional oncology procedures. We developed an approach to training deep learning-based metal artifact reduction (MAR) models that uses phantom-based methods for simulating metal artifacts as well as novel loss functions and data augmentation steps to achieve optimal results. Qualitative comparisons demonstrate that the proposed method can reduce probe-induced metal artifacts while maintaining a high level of anatomic detail. The proposed method does not require access to raw projection data and therefore can be applied to any combination of probes and CT scanners.

We aimed to demonstrate the ability of spectral CT to provide temperature mapping within the treatment volume during CT-guided hypo- and hyper-thermal tumor ablations. We collected high dose spectral CT data spanning a wide range of temperatures and generated look up tables that map CT signal to temperature for ranges of clinical interest in cryoablation and hyperthermal ablations. Using electron density images generated from spectral CT data, we demonstrated a sensitivity to temperature changes of 1.2 and 4.1HU-equivalent per 10C in the freezing and heating temperature ranges, respectively. At the clinical radiation dose level for our Interventional Oncology practice, we obtained a maximum precision of 7C and 2C within a 33 mm3 ROI of electron density images for freezing and heating temperatures, respectively. This information was used to develop a clinical-ready CT thermometry protocol that was independently validated and demonstrated a median absolute error of 12.2C and 3.4C for freezing and heating temperature data, respectively.

Magnetic Resonance-guided Laser Interstitial Thermal Therapy (MRgLITT) is a minimally invasive brain tumor treatment involving the insertion of a laser fiber guided by real-time MR thermometry images. However, repositioning the laser is invasive, and accurately predicting thermal spread close to heat sinks poses challenges. To address this issue, we propose the development of MR thermometry prediction using artificial intelligence (AI) modeling. U-Net was trained to model the nonlinear mapping from anatomical magnetic resonance imaging (MRI) planning images to MR thermometry, enabling neurosurgeons to predict heat propagation and choose the best laser trajectory before treatment.

This study investigates the robustness of device segmentation and tracking in continuous-sweep limited angle fluoroscopy. This technique was developed to provide real-time 3D device navigation during catheter-based procedures. A porcine study is presented were image sequences at different noise levels were acquired and the device automatically tracked using a deep learning-based segmentation approach.

Lung cancer, the second most common cancer in the United States, is diagnosed and staged through the analysis of biopsy specimens, often obtained through transbronchial biopsy (TBB). However, accurate TBB for small nodules is hindered by CT body divergence – misalignment between pre-operative CT and intra-operative coordinate frames. We propose a comprehensive image guidance system, leveraging a stationary multi-source fluoroscopy imager together with deformable 3D/2D registration to solve for a motion field parameterized by implicit neural representations(INR) to jointly track pulmonary and bronchoscopic motion. We evaluate our algorithm using a simulated imaging chain and a 4D-CT dataset, as well as on simulated TBB. Using 5 views, we demonstrate a median landmark TRE of 1.42 mm and a bronchoscope tip error of 2.8 mm. We demonstrate a promising 3D image guidance approach to improving the accuracy of trans-bronchial biopsy using a multi-view stationary imager and estimation of patient motion through deformable 3D/2D registration, which can be extended to track respiratory and bronchoscope motion over time for real-time navigation.

A deformable liver motion model was developed to advance treatment planning for CBCT-guided histotripsy. The model is FEM-based, informed by displacements at not only external boundaries (liver and gallbladder surfaces), but internal ones as well (vessel surfaces). This method can accurately predict how the target volume has deformed between a high-quality diagnostic scan for sophisticated treatment planning and the day-of CBCT to account for changes in patient positioning.