This PDF file contains the front matter associated with SPIE Proceedings Volume 10951, including the Title Page, Copyright information, Table of Contents, Author and Conference Committee lists.

Purpose. We report the initial implementation of an algorithm that automatically plans screw trajectories for spinal pedicle screw placement procedures to improve the workflow, accuracy, and reproducibility of screw placement in freehand navigated and robot-assisted spinal pedicle screw surgery. In this work, we evaluate the sensitivity of the algorithm to the settings of key parameters in simulation studies. Methods. Statistical shape models (SSMs) of the lumbar spine were constructed with segmentations of L1-L5 and bilateral screw trajectories of N=40 patients. Active-shape model (ASM) registration was devised to map the SSMs to the patient CT, initialized simply by alignment of (automatically annotated) single-point vertebral centroids. The atlas was augmented by definition of “ideal / reference” trajectories for each spinal pedicle, and the trajectories are deformably mapped to the patient CT. A parameter sensitivity analysis for the ASM method was performed on 3 parameters to determine robust operating points for ASM registration. The ASM method was evaluated by calculating the root-mean-square-error between the registered SSM and the ground-truth segmentation for the L1 vertebra, and the trajectory planning method was evaluated by performing a leave-one-out analysis and determining the entry point, end point, and angular differences between the automatically planned trajectories and the neurosurgeon-defined reference trajectories. Results. The parameter sensitivity analysis showed that the ASM registration algorithm was relatively insensitive to initial profile length (P_Linitial) less than ~4 mm, above which runtime and registration error increased. Similarly stable performance was observed for a maximum number of principal components (PC_max) of at least 8. Registration error ~2 mm was evident with diminishing return beyond a number of iterations, N_iter, ~2000. With these parameter settings, ASM registration of L1 achieved (2.0 ± 0.5) mm RMSE. Transpedicle trajectories for L1 agreed with reference definition by (2.6 ± 1.3) mm at the entry point, by (3.4 ± 1.8) mm at the end point, and within (4.9° ±2.8°) in angle. Conclusions. Initial results suggest that the algorithm yields accurate definition of pedicle trajectories in unsegmented CT images of the spine. The studies identified stable operating points for key algorithm parameters and support ongoing development and translation to clinical studies in free-hand navigated and robot-assisted spine surgery, where fast, accurate trajectory definition is essential to workflow.

Intraoperative imaging systems are seeing an increased role in support of surgical guidance and quality assurance in the operating room for interventional approaches. However, image quality sufficient to detect complications and provide quantitative assessment of the surgical product are often confounded by image noise and artifacts. In this work, we translated a 3D image reconstruction method (referred to as “Known-Component Reconstruction,” KC-Recon) for the first time to clinical studies with the aim of resolving both limitations. KC-Recon builds upon an optimization-based reconstruction method to reduce noise and incorporates a model of surgical instruments in the image to reduce artifacts. The first clinical pilot study involved 17 spine surgery patients imaged using the O-arm before and after spinal instrumentation. Imaging performance was evaluated in terms of low-contrast soft-tissue visibility, the ability to assess screw placement within bone margins, and the potential to image at lower radiation doses. Depending on the imaging task, dose reduction up to an order of magnitude appeared feasible while maintaining soft-tissue visibility. KC-Recon also yielded ~30% reduction in blooming artifact about the screw shafts and ~60% higher tissue homogeneity at the screw tips, providing clearer depiction of pedicle and vertebral body for assessment of potential breaches. Overall, the method offers a promising means to reduce patient dose in image-guided procedures, extend the use of cone-beam CT to soft-tissue surgeries, provide a valuable check against complications in the operating room (cf., post-operative CT), and serve as a basis for quantitative evaluation of quality of the surgical construct.

Purpose: A method for automatic computation of global spinal alignment (GSA) metrics is presented to mitigate the high variability of manual definitions in radiographic images. The proposed algorithm segments vertebral endplates in CT as a basis for automatic computation of metrics of global spinal morphology. The method is developed as a potential tool for intraoperative guidance in deformity correction surgery, and/or automatic definition of GSA in large datasets for analysis of surgical outcome. Methods: The proposed approach segments vertebral endplates in spine CT images using vertebral labels as input. The segmentation algorithm extracts vertebral boundaries using a continuous max-flow algorithm and segments the vertebral endplate surface by region-growing. The point cloud of the segmented endplate is forward-projected as a digitally reconstructed radiograph (DRR), and a linear fit is computed to extract the endplate angle in the radiographic plane. Two GSA metrics (lumbar lordosis and thoracic kyphosis) were calculated using these automatically measured endplate angles. Experiments were performed in seven patient CT images acquired from Spineweb and accuracy was quantified by comparing automatically-computed endplate angles and GSA metrics to manual definitions. Results: Endplate angles were automatically computed with median accuracy = 2.7°, upper quartile (UQ) = 4.8°, and lower quartile (LQ) = 1.0° with respect to manual ground-truth definitions. This was within the measured intra- observer variability = 3.1° (RMS) of manual definitions. GSA metrics had median accuracy = 1.1° (UQ = 3.1°) for lumbar lordosis and median accuracy = 0.4° (UQ = 3.0°) for thoracic kyphosis. The performance of GSA measurements was also within the variability of the manual approach. Conclusions: The method offers a potential alternative to time-consuming, manual definition of endplate angles for GSA computation. Such automatic methods could provide a means of intraoperative decision support in correction of spinal deformity and facilitate data-intensive analysis in identifying metrics correlating with surgical outcomes.

Brain shift during neurosurgery can compromise the fidelity of image guidance and potentially lead to surgical error. We have developed a finite element model-based brain shift compensation strategy to correct preoperative images for improved intraoperative navigation. This workflow-friendly approach precomputes potential intraoperative deformations (a ‘deformation atlas’) via a biphasic-biomechanical-model accounting for brain deformation associated with cerebrospinal fluid drainage, osmotic agents, resection, and swelling. Intraoperatively, an inverse problem approach is employed to provide a combinatory fit from the atlas that best matches sparse intraoperative measurements. Subsequently, preoperative image is deformed accordingly to better reflect patient’s intraoperative anatomy. While we have performed several retrospective studies examining model’s accuracy using post- or intra-operative magnetic resonance imaging, one challenging task is to examine model’s ability to recapture shift due to the aforementioned effects independently with clinical data and in a longitudinal manner under varying conditions. The work here is a case study where swelling was observed at the initial stage of surgery (after craniotomy and dura opening), subsequently sag was observed in a later stage of resection. Intraoperative tissue swelling and sag were captured via an optically tracked stylus by identifying cortical surface vessel features (n = 9), and model-based correction was performed for these two distinct types of brain shift at different stages of the procedure. Within the course of the entire surgery, we estimate the cortical surface experienced a deformation trajectory absolute path length of approximately 19.4 ± 2.1 mm reflecting swelling followed by sag. Overall, model reduced swelling-induced shift from 7.3 ± 1.1 to 1.8 ± 0.5 mm (~74.6% correction); for subsequent sag movement, model reduced shift from 6.4 ± 1.5 to 1.4 ± 0.5 mm (~76.6% correction).

Deep brain stimulation (DBS) electrode placement is a burr-hole procedure for the treatment of patients with neuro- degenerative disease such as Parkinson’s disease, essential tremor and dystonia. Accurate placement of electrodes is the key to optimal surgical outcome. However, the accuracy of pre-operative images used for surgical planning are often degraded by intraoperative brain shift. To compensate for intraoperative target deviation, we have developed a biomechanical model, driven by partially sampled displacements between pre- and postCT, to estimate a whole brain displacement field based on which updated CT (uCT) can be generated. The results of the finite element model depend on sparse data, as the model minimizes the difference between model estimates and sparse data. Existing approaches to extract sparse data from brain surface are typically geometry or feature-based. In this paper, we explore a geometry- based iterative closest point (ICP) algorithm and a feature-based image registration algorithm, and drive the model with 1) geometry-based sparse data only, 2) feature-based sparse data only, and 3) combined data from 1) and 2). We assess the model performance in terms of model-data misfit, as well as target registration errors (TREs) at the anterior commissure (AC) and posterior commissure (PC). Results show that the model driven by the geometry-based sparse data reduced the TREs of preCT from 1.65mm to 1.26 mm and 1.88 mm to 1.58 mm at AC and PC, respectively by compensating majorly along the direction of gravity and the longitudinal axis, whereas feature-based sparse data contributed to the compensation along the lateral direction at PC.

Clinical tracking systems are popular but typically require specific tracking markers. During the last years, scanning speed of optical coherence tomography (OCT) has increased to A-scan rates above 1MHz allowing to acquire volume scans of moving objects. Therefore, we propose a markerless tracking system based on OCT to obtain small volumetric images including information of sub-surface structures at high spatio-temporal resolution. In contrast to conventional vision based approaches, this allows identifying natural landmarks even for smooth and homogeneous surfaces. We describe the optomechanical setup and process ow to evaluate OCT volumes for translations and accordingly adjust the position of the field-of-view to follow moving samples. While our current setup is still preliminary, we demonstrate tracking of motion transversal to the OCT beam of up to 20mms􀀀1 with errors around 0:2mm and even better for some scenarios. Tracking is evaluated on a clearly structured and on a homogeneous phantom as well as on actual tissue samples. The results show that OCT is promising for fast and precise tracking of smooth, monochromatic objects in medical scenarios.

Automatic motion compensation and adjustment of an intraoperative imaging modality's field of view is a common problem during interventions. Optical coherence tomography (OCT) is an imaging modality which is used in interventions due to its high spatial resolution of few micrometers and its temporal resolution of potentially several hundred volumes per second. However, performing motion compensation with OCT is problematic due to its small field of view which might lead to tracked objects being lost quickly. We propose a novel deep learning-based approach that directly learns input parameters of motors that move the scan area for motion compensation from optical coherence tomography volumes. We design a two-path 3D convolutional neural network (CNN) architecture that takes two volumes with an object to be tracked as its input and predicts the necessary motor input parameters to compensate the object's movement. In this way, we learn the calibration between object movement and system parameters for motion compensation with arbitrary objects. Thus, we avoid error-prone hand-eye calibration and handcrafted feature tracking from classical approaches. We achieve an average correlation coefficient of 0:998 between predicted and ground-truth motor parameters which leads to sub-voxel accuracy. Furthermore, we show that our deep learning model is real-time capable for use with the system's high volume acquisition frequency.

Quality assurance in current 4D radiotherapy workflows is of great importance to assure a positive treatment outcome, i.e. total tumor eradication. Especially for the treatment of lung and liver tumors, which are subject to high motion magnitudes due to patient breathing, it is crucial to verify the applied dose to the target volume. In this study, we present a new 4D Monte Carlo dose accumulation approach that accounts for internal patient motion during treatment and is therefore able to predict the actual 3D dose distribution delivered to the patient for quality assurance purposes. Monte Carlo simulations are conducted using the EGSnrc software toolkit, which models the propagation of photons, electrons and positrons. However, to consider dynamic beam parameters and the movement of internal patient geometry, we developed a method to compute the dose for each control point of the actual VMAT patient treatment plan to account for breathing induced internal patient motion. The internal motion during treatment is predicted using correspondence modeling, which correlates patient-specific DIR-based internal motion information and external breathing signals and is trained on 4D CT data of the patient. For each VMAT control point, a corresponding motion vector field is predicted and applied to the original patient CT to allow for dose computation on the patient geometry as it was irradiated during treatment. Thus, density changes while treatment due to patient breathing motion are taken into account during computation of the resulting dose distribution.

We present a new scheme for bronchoscopic navigation by exploiting visual SLAM for bronchoscope tracking. Bronchoscopic navigation system is used to guide physicians by providing 3D space information about the bronchoscope during bronchoscopic examination. Existing bronchoscopic navigation systems mainly used CT-video or sensor for bronchoscope tracking. CT-video based tracking estimates the bronchoscope pose by registration of real bronchoscope images and virtual images generated from computed tomography (CT) images, which requires lots of time. Sensor based tracking calculates the bronchoscope pose based on information from sensor, which is easily in uenced by examination tools. We improve the bronchoscope tracking by using visual simultaneous localization and mapping (VSLAM), which can overcome the aforementioned shortcomings. VSLAM is an approach to estimate the camera pose and reconstruct surrounding structure around a camera (called map). We use the adjacent frames to increase the points used for tracking, and use VSLAM for bronchoscope tracking. Tracking performance of VSLAM were evaluated with phantom and in-vivo videos. Reconstruction performance of VSLAM was evaluated by root mean square (RMS) value, which is calculated using aligned reconstructed points and segmented bronchus from pre-operative CT volumes. Experimental results showed that the successfully tracked frames in the proposed method increased more than 700 frames compared with the original ORB-SLAM for six cases. The average RMS in phantom case between estimated bronchus from SLAM and bronchus shape from segmented bronchus was 2.55 mm.

Current image-guided prostate radiotherapy often relies on the use of implanted fiducial markers (FMs) or transducers for target localization. Fiducial or transducer insertion requires an invasive procedure that adds cost and risks for bleeding, infection and discomfort to some patients. We are developing a novel markerless prostate localization strategy using a pre-trained deep learning model to interpret routine projection kV X-ray images without the need for daily cone-beam computed tomography (CBCT). A deep learning model was first trained by using several thousand annotated projection X-ray images. The trained model is capable of identifying the location of the prostate target for a given input X-ray projection image. To assess the accuracy of the approach, three patients with prostate cancer received volumetric modulated arc therapy (VMAT) were retrospectively studied. The results obtained by using the deep learning model and the actual position of the prostate were compared quantitatively. The deviations between the target positions obtained by the deep learning model and the corresponding annotations ranged from 1.66 mm to 2.77 mm for anterior-posterior (AP) direction, and from 1.15 mm to 2.88 mm for lateral direction. Target position provided by deep learning model for the kV images acquired using OBI is found to be consistent that derived from the implanted FMs. This study demonstrates, for the first time, that highly accurate markerless prostate localization based on deep learning is achievable. The strategy provides a clinically valuable solution to daily patient positioning and real-time target tracking for image-guided radiotherapy (IGRT) and interventions.

In radiation therapy, tumor tracking is a challenging task that allows a better dose delivery. One practice is to acquire X-ray images in real-time during treatment, that are used to estimate the tumor location. These informations are used to predict the close future tumor trajectory. Kalman prediction is a classical approach for this task. The main drawback of X-ray acquisition is that it irradiates the patient, including its healthy tissues. In the classical Kalman framework, X-ray measurements are taken regularly, i.e. at a constant rate. In this paper, we propose a new approach which relaxes this constraint in order to take measurements when they are the most useful. Our aim is for a given budget of measurements to optimize the tracking process. This idea naturally brings to an optimal intermittent Kalman predictor for which measurement times are selected to minimize the mean squared prediction error over the complete fraction. This optimization problem can be solved directly when the respiratory model has been identified and the optimal sampling times can be computed at once. These optimal measurement times are obtained by solving a combinatorial optimization problem using a genetic algorithm. We created a test benchmark on trajectories validated on one patient. This new prediction method is compared to the regular Kalman predictor and a relative improvement of 9:8% is observed on the root mean square position estimation error.

Proton beam therapy has recently been proposed as a noninvasive approach for treating ventricular tachycardia (VT), where target regions are identified in the myocardium and treated using external beam therapy. Effective treatment requires that lesions develop at target sites of myocardial tissue in order to stop arrhythmic pathways. Precise characterization of the dose required for lesion creation is required for determining appropriate dose levels in future clinical treatment of VT patients. In this work, we use a deformable registration algorithm to align proton beam delivery isodose lines planned from baseline computed-tomography scans to follow-up delayed contrast-enhanced magnetic resonance imaging scans in three swine studies. The relationship between myocardial lesion formation and delivered dose from external proton beam ablation therapy is then quantitatively assessed. The current study demonstrates that myocardial tissue receiving a dose of 20Gy or higher tends to develop into lesion, while tissue exposed to less than 10Gy of dose tends to remain healthy. Overall, this study quantifies the relationship between external proton beam therapy dose and myocardial lesion formation which is important for determining dose levels in future clinical treatment of VT patients.

We present an analytical LV systolic model derived from human CT data to serve as the ground truth for optimization and validation of a previously published CT-derived regional strain metric called SQUEEZ. Physiologically-accurate strains were applied to each vertex of a clinically derived end-diastolic LV mesh to create analytical end-systolic poses exhibiting normal function as well as regional hypokinesia of four sizes (17.5mm, 14mm, 10.5mm, and 7mm in diameter), each with a programmed severe, medium, and subtle dysfunction. Regional strain estimates were obtained by registering the end-diastolic mesh to each end-systolic mesh condition using a non-rigid registration algorithm. Ground-truth models of normal function and of severe hypokinesia were used to identify the optimal parameters in the registration algorithm, and to measure the accuracy of detecting regional dysfunction of varying sizes and severities. We found that for normal LV systolic contraction, SQUEEZ values in all 16 AHA segments of the LV were accurately measured (within ±0.05). For cases with regional dysfunction, the errors in SQUEEZ in the region around the dysfunction increased with decreasing size of regional dysfunction. The mean SQUEEZ values of the 17.5mm and 14mm diameter dysfunctional regions, which we hypothesize are the most clinically relevant, were accurate to within 0.05.

Transthoracic echocardiography (echo) is the most common imaging modality for diagnosis of cardiac conditions. Echo is acquired from a multitude of views, each of which distinctly highlights specific regions of the heart anatomy. In this paper, we present an approach based on knowledge distillation to obtain a highly accurate lightweight deep learning model for classification of 12 standard echocardiography views. The knowledge of several deep learning architectures based on the three common state-of-the-art architectures, VGG-16, DenseNet, and Resnet, are distilled to train a set of lightweight models. Networks were developed and evaluated using a dataset of 16,612 echo cines obtained from 3,151 unique patients across several ultrasound imaging machines. The best accuracy of 89.0% is achieved by an ensemble of the three very deep models while we show an ensemble of lightweight models has a comparable accuracy of 88.1%. The lightweight models have approximately 1% of the very deep model parameters and are six times faster in run-time. Such lightweight view classification models could be used to build fast mobile applications for real-time point-of-care ultrasound diagnosis.

New high-frame-rate ultrasound imaging techniques are being developed to image tissue motion and blood flow with high sensitivity and at high temporal resolution. An emerging application for these new techniques is diagnosing inutero and neonatal cardiac disease. We have developed a morphologically and hemodynamically accurate neonatal heart phantom to provide a high-fidelity physical model for laboratory testing of ultrafast color Doppler echocardiography methods. This paper summarizes the design and functionality of the simulator by measuring pressure gradients across the mitral valve at a physiologic heart-rate range and stroke volume and by evaluating valve function using 2D transesophageal echocardiography (TEE) and Doppler images. The phantom achieved normal physiological pressures across the mitral valve ranging from 42 to 87 mmHg in systole and 2.4 to 4.2 mmHg in diastole at heartrates of 100, 125 and 150 beats per minute (bpm), with a realistic neonatal stroke volume of 7 ml. 2D ultrasound images were obtained at 60 bpm.

Mitral valve disease affects 2% of the Canadian population and 10% of those over the age of 75. Mitral valve regurgitation is a common valve disease often requiring surgical intervention for repair or replacement. Repair is often preferred over replacement, as it is associated with improved outcomes. Current mitral valve repair training is typically limited primarily to intraoperative experience. Additionally, the outcome of complex repair procedures is often unknown preoperatively, and is particularly true of new, off-pump repair techniques. Further challenges include identifying the most effective repair technique based on patient pathology, as multiple approaches exist. We present a hemodynamically accurate mitral valve phantom for testing previously validated patient specific pathological mitral valves. The device can be used for surgical resident training as well as complex procedure planning. The simulator is validated using pressure measurements across the mitral valve demonstrating realistic hemodynamics across a range of heart rates , and by evaluating valve function using 2D and 3D transesophageal echocardiography (TEE).

Concentric tube and steerable needle robots provide minimally invasive access to confined or remote spaces in the human body. While the modeling and control of these devices has received a great deal of attention in the robotics literature, comparatively less attention has been paid to date to the design of the mechatronic system that grasps the tubes/needles at their bases and applies axial twists and telescopic motions to component tubes, which we refer to as an actuation unit. Toward moving these systems to clinical use, this paper explores the design of a new, compact modular robotic actuation unit that incorporates new approaches to homing and tool changes. In particular, we accomplish homing using sensors that require no moving wires, eliminating potential failure points on the robot. We also present a new quick-connect mechanism that enables a collection of tubes to be rapidly coupled to or decoupled from the robot. This paper describes our new actuation unit design, illustrating our new tube coupling and homing concepts.

Cochlear implant surgery typically requires a wide-field mastoidectomy to access the cochlea. This portion of the surgery can leave a visible and palpable depression behind the patient's ear, which can be cosmetically displeasing to the patient. For the surgeon, a wide-field mastoidectomy is challenging to perform because bone must be gradually removed by freehand drilling guided primarily by visual feedback in an effort to detect, yet avoid, vital anatomy including the facial nerve which controls motion of the face. Toward overcoming these issues and standardizing surgery, imaged-guided, minimally invasive approaches have been developed in which the cochlea is accessed using a single pre-planned drill trajectory. This approach promises decreased invasiveness, but the limited surgical view and long narrow opening to the cochlea present significant challenges for inserting electrode arrays. This paper describes the first cadaver experiments using a new manual insertion tool which provides a roller mechanism to enable the physician to deploy a cochlear implant electrode array through the narrow drilled hole created by this minimally invasive, image-guided access technique. Results demonstrate that the new tool enables consistent and successful insertions similar to insertions with the traditional tool while increasing the ease of the insertion and freeing the surgeon to monitor progress and make fine adjustments as needed.

The desire to improve patient safety and clinical precision has prompted research in the development of a real-time, single operator, image-guided solution for neuraxial anesthesia. Ultrasound is ideal for this application given that it is real-time, non-ionizing, and with recent advances, ultra-portable. Previous work has investigated the use of 3D ultrasound and 2D in-plane imaging to track needle insertions but faced barriers to successful clinical translation. The EpiGuide 2D is a novel multi-channel out-of-plane needle guide that addresses deficiencies observed in prior designs. Specifically, it leverages beam thickness, an inherent imaging artefact, to provide needle visibility over a range of depths. The current work investigates the ability of the EpiGuide 2D to visualize out-of-plane needle insertions. Two different needle types are explored with 9 needle angles over 5 distinct imaging depths. Benchtop testing is performed to assess stability of the guide’s open channels. Subsequent water bath testing is used to establish baseline visibility metrics across all angles. Finally, testing on an ex vivo porcine model is performed. A total of n=424 needle insertions are performed. Visible range and contrast-to-noise ratios are measured for each insertion. As needle angle approached parallel to the imaging plane, visible range increased. Needle echogenicity also increased the visible range of the needle in the water bath setting but was not found to have a statistically significant effect on visible range in the porcine model. The EpiGuide 2D accommodates needle visualization in tissue for depths of 21 mm to 53 mm. Further in vivo studies are warranted.

PURPOSE: MR-guided injections are safer for the patient and the physician than CT-guided interventions but require a significant amount of hand-eye coordination and mental registration by the physician. We propose a low-cost, adjustable, handheld guide to assist the operator in aligning the needle in the correct orientation for the injection. METHODS: The operator adjusts the guide to the desired insertion angle as determined by an MRI image. Next, the operator aligns the guide in the image plane using the horizontal laser and level gradient. The needle is inserted into the sleeve of the guide and inserted into the patient. To evaluate the method, two operators inserted 5 needles in two facet joints of a lumbar spine phantom. Insertion points, final points and trajectory angles were compared to the projected needle trajectory using an electromagnetic tracking system. RESULTS: On their first attempt, operators were able to insert the needle into the facet joint 85% of the time. On average, operators had an insertion point error of 2.92 ± 1.57 mm, a target point error of 3.39 ± 2.28 mm, and a trajectory error of 3.98 ± 2.09 degrees. CONCLUSION: A low-cost, adjustable, handheld guide was developed to assist in correctly positioning a needle in MR guided needle interventions. The guide is as accurate as other needle placement assistance mechanisms, including the biplane laser guides and image overlay devices when used in lumbar facet joint injections in phantoms.

Pathologies of protective laryngeal reflexes such as the laryngeal adductor reflex (LAR) can increase the risk of aspiration pneumonia, a potentially life-threatening inflammation of the lungs caused by the intrusion of foreign particles into the lower airways. To estimate this risk, a standardized and non-invasive LAR screening method is highly desirable. In previous work, a commercially available high-speed laryngoscope has been equipped with a pressurized fluid system to shoot droplets onto the laryngeal structures for LAR stimulation and subsequent reflex latency evaluation. This Micro-droplet Impulse Testing of the LAR (MIT-LAR) lacks droplet impact site prediction for an unbiased and stimulation site-dependent reflex latency assessment. In this work, a two- phase algorithm leveraging stereoscopic image data for droplet impact site prediction and visualization of this prediction in the laryngoscopic image is proposed. A high-speed stereolaryngoscope requiring only a single camera was designed from scratch by combining two rod lens optics in one common shaft. This setup was used for stereoscopic high-speed image data acquisition of droplets shot at different muzzle energies and endoscope roll angles. Surface reconstruction of the target region is performed using a phantom of the human larynx. The point of intersection between the reconstructed surface and the droplet trajectory approximation recorded previously at known droplet formation parameters is calculated. The proposed approach allows stimulation site prediction for an enhanced MIT-LAR procedure. Quantitative evaluation of impact site prediction accuracy for different droplet muzzle energies and nozzle inclinations yields an average prediction error of 2.9 mm (n = 20).

Purpose: A new version of a grid-template-mimicking MR-compatible robot is developed to assist during in-gantry MRI-guided focal laser ablation of prostate cancer. This robot replaces the grid template and provides higher positioning resolution, and allows autonomous needle alignment directly from the targeting and navigation software, with the needle insertion manually performed for safety. Method: A substantially more compact solution is designed and prototyped to allow comfortable accommodation between the patient’s legs while in MRI bore. The controller software was reconfigured and embedded into the custom navigation and multi-focal ablation software, OncoNav (NIH). OncoNav performs robot-to-image registration, target planning, controlling the robot, ablation planning, and 3D temperature analysis for monitoring. For free space accuracy study, 5 targets were selected across the workspace and the robot was commanded 5 times to each target. Then, a thermochromic phantom study was designed consisting of acrylamide gel and color changing ink for testing the overall workflow. 4 spherical metal fiducials were embedded in the phantom at different locations. After each targeting, laser ablation was applied in two of the targets. Finally, the phantom was sliced for gross observation of guidance and treatment accuracy. Results: in-the-air accuracy was 0.38±0.27 mm. The overall targeting accuracy including robot, registration, and insertion error was 2.17±0.47 mm in phantom. Ablation successfully covered ellipsoids around the targets. The workflow was acceptably smooth. Conclusions: The new robot can accurately assist in targeting small targets followed by focal laser ablation. Sterility and regulatory hurdles will be addressed with specific design approaches as the next step.

Accurate segmentation of organs at risk (OARs) is a key step in image guided radiation therapy. In recent years, deep learning based methods have been widely used in medical image segmentation. Among them, U-Net and V-Net are the most popular ones. In this paper, we evaluate a customized V-Net on 16 OARs throughout the body using a large CT dataset. Specifically, two customizations are used to reduce the GPU memory cost of V-Net: 1) multi-resolution V-Nets, where the coarse-resolution V-Net aims to localize the OAR in the entire image space, while the fine-resolution V-Net focuses on refining detailed boundaries of OAR; 2) a modified V-Net architecture, which is specifically designed for segmenting large organs, e.g., liver. Validated on 3483 CT scans of various imaging and disease conditions, we show that, compared with traditional methods, the customized V-Net wins in speed (0.7 second vs 20 seconds per organ), accuracy (average Dice score 96.6% vs 84.3%), and robustness (98.6% successful rate vs 83.3% successful rate). Moreover, the customized V-Net is very robust against various image artifacts, diseases and slice thicknesses, and has much better performance even on the organs with large shape variations (e.g., the bladder) than traditional methods.

Surgical robot technology has revolutionized surgery toward a safer laparoscopic surgery and ideally been suited for surgeries requiring minimal invasiveness. Sematic segmentation from robot-assisted surgery videos is an essential task in many computer-assisted robotic surgical systems. Some of the applications include instrument detection, tracking and pose estimation. Usually, the left and right frames from the stereoscopic surgical instrument are used for semantic segmentation independently from each other. However, this approach is prone to poor segmentation since the stereo frames are not integrated for accurate estimation of the surgical scene. To cope with this problem, we proposed a multi encoder and single decoder convolutional neural network named StreoScenNet which exploits the left and right frames of the stereoscopic surgical system. The proposed architecture consists of multiple ResNet encoder blocks and a stacked convolutional decoder network connected with a novel sum-skip connection. The input to the network is a set of left and right frames and the output is a mask of the segmented regions for the left frame. It is trained end-to-end and the segmentation is achieved without the need of any pre- or post-processing. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets that includes robotic instruments as well as anatomical objects and non-robotic surgical instruments. Compared with the previous instrument segmentation methods, our approach achieves a significant improved Dice similarity coefficient.

This paper presents a colonoscope tracking method utilizing a colon shape estimation method. CT colonography is used as a less-invasive colon diagnosis method. If colonic polyps or early-stage cancers are found, they are removed in a colonoscopic examination. In the colonoscopic examination, understanding where the colonoscope running in the colon is difficult. A colonoscope navigation system is necessary to reduce overlooking of polyps. We propose a colonoscope tracking method for navigation systems. Previous colonoscope tracking methods caused large tracking errors because they do not consider deformations of the colon during colonoscope insertions. We utilize the shape estimation network (SEN), which estimates deformed colon shape during colonoscope insertions. The SEN is a neural network containing long short-term memory (LSTM) layer. To perform colon shape estimation suitable to the real clinical situation, we trained the SEN using data obtained during colonoscope operations of physicians. The proposed tracking method performs mapping of the colonoscope tip position to a position in the colon using estimation results of the SEN. We evaluated the proposed method in a phantom study. We confirmed that tracking errors of the proposed method was enough small to perform navigation in the ascending, transverse, and descending colons.

In microsurgery, lasers have emerged as precise tools for bone ablation. A challenge is automatic control of laser bone ablation with 4D optical coherence tomography (OCT). OCT as high resolution imaging modality provides volumetric images of tissue and foresees information of bone position and orientation (pose) as well as thickness. However, existing approaches for OCT based laser ablation control rely on external tracking systems or invasively ablated artificial landmarks for tracking the pose of the OCT probe relative to the tissue. This can be superseded by estimating the scene flow caused by relative movement between OCT-based laser ablation system and patient. Therefore, this paper deals with 2.5D scene flow estimation of volumetric OCT images for application in laser ablation. We present a semi-supervised convolutional neural network based tracking scheme for subsequent 3D OCT volumes and apply it to a realistic semi-synthetic data set of ex vivo human temporal bone specimen. The scene flow is estimated in a two-stage approach. In the first stage, 2D lateral scene flow is computed on census-transformed en-face arguments-of-maximum intensity projections. Subsequent to this, the projections are warped by predicted lateral flow and 1D depth flow is estimated. The neural network is trained semi-supervised by combining error to ground truth and the reconstruction error of warped images with assumptions of spatial flow smoothness. Quantitative evaluation reveals a mean endpoint error of (4.7 ± 3.5) voxel or (27.5 ± 20.5) μm for scene flow estimation caused by simulated relative movement between the OCT probe and bone. The scene flow estimation for 4D OCT enables its use for markerless tracking of mastoid bone structures for image guidance in general, and automated laser ablation control.

Motivation/Purpose: This work reports the development and validation of an algorithm to automatically detect and localize vertebrae in CT images of patients undergoing spine surgery. Slice-by-slice detections using the state-of-the art 2D convolutional neural network (CNN) architectures were combined to estimate vertebra centroid location in 3D including a method that combined detections in sagittal and coronal slices. The solution facilitates applications in image guided surgery and automatic computation of image analytics for surgical data science. Methods: CNN-based object detection models in 3D (volume) and 2D (slice) images were implemented and evaluated for the task of vertebrae detection. Slice-by-slice detections in 2D architectures were combined to estimate the 3D centroid location including a model that simultaneously evaluated 2D detections in orthogonal directions (i.e., sagittal and coronal slices) to improve the robustness against spurious false detections – called Ortho-2D. Performance was evaluated in a data set consisting of 85 patients undergoing spine surgery at our institution, including images presenting spinal instrumentation/implants, spinal deformity, and anatomical abnormalities that are realistic exemplars of pathology in the patient population. Accuracy was quantified in terms of precision, recall, F1 score, and the 3D geometric error in vertebral centroid annotation compared to ground truth (expert manual) annotation. Results: Three CNN object detection models were able to successfully localize vertebrae, with Ortho-2D model that combined 2D detections in orthogonal directions achieving best performance: precision = 0.95, recall = 0.99, and F1 score = 0.97. Overall centroid localization accuracy was 3.4 mm (median) [interquartile range (IQR) = 2.7 mm], and ~97% of detections (154/159 lumbar cases) yielded acceptable centroid localization error <15 mm (considering average vertebrae size ~25 mm). Conclusions: State-of-the-art CNN architectures were adapted for vertebral centroid annotation, yielding accurate and robust localization even in the presence of anatomical abnormalities, image artifacts, and dense instrumentation. The methods are employed as a basis for streamlined image guidance (automatic initialization of 3D-2D and 3D-3D registration methods in image-guided surgery) and as an automatic spine labeling tool to generate image analytics.

During high-dose-rate (HDR) interstitial brachytherapy of gynecologic malignancies, precise placement of multiple needles is necessary to provide optimal dose to the tumor while avoiding overexposing nearby healthy organs, such as the bladder and rectum. Needles are currently placed based on preoperative imaging and clinical examination but there is currently no standard for intraoperative image guidance. We propose the use of a three-dimensional (3D) ultrasound (US) system incorporating three scanning geometries: 3D transrectal US (TRUS), 360° 3D sidefire transvaginal US (TVUS), and 3D endfire TVUS, to provide an accessible and versatile tool for intraoperative image guidance during interstitial gynecologic brachytherapy. Images are generated in 12 - 20 s by rotating a conventional two-dimensional US probe, providing a reconstructed 3D image immediately following acquisition. Studies of needles in patient images show mean differences in needle positions of 3.82 ± 1.86 mm and 2.36 ± 0.97 mm in TRUS and sidefire TVUS, respectively, when compared to the clinical x-ray computed tomography (CT) images. A proof-of-concept phantom study of the endfire TVUS mode demonstrated a mean positional difference of 1.91 ± 0.24 mm. Additionally, an automatic needle segmentation tool was tested on a 360° 3D TVUS patient image resulting in a mean angular difference of 0.44 ± 0.19 ° and mean positional difference of 0.78 ± 0.17 mm when compared to manually segmented needles. The implementation of 3D US image guidance during HDR interstitial gynecologic brachytherapy provides a versatile intraoperative system with the potential for improved implant quality and reduced risk to nearby organs.

The brain is significantly deformed during neurosurgery, in particular because of the removal of tumor tissue. Because of this deformation, intraoperative data is needed for accurate navigation in image-guided surgery. During the surgery, it is easier to acquire ultrasound images than Magnetic Resonance (MR) images. However, ultrasound images are difficult to interpret. Several methods have been developed to register preoperative MR and intraoperative ultrasound images, to allow accurate navigation during neurosurgery. Model-based methods need the location of the resection cavity to take into account the tissue removal in the model. Manually segmenting this cavity is extremely time consuming and cannot be performed in the operating room. It is also difficult and error-prone because of the noise and reconstruction artifacts in the ultrasound images. In this work, we present a method to perform the segmentation of the resection cavity automatically. We manually labelled the resection cavity on the ultrasound volumes from a database of 23 patients. We trained a Unet-based artificial neural network with our manual segmentation and evaluated several variations of the method. Our best method results in 0.82 mean Dice score over the 10 testing cases. The Dice scores range from 0.67 to 0.96, and eight out of ten are higher than 0.75. For the most difficult test cases, lacking clear contour, the manual segmentation is also difficult but our method still yields acceptable results. Overall the segmentations obtained with the automatic methods are qualitatively similar to the manual ones.

Conventional B-mode ultrasound imaging lacks soft tissue contrast to differentiate various tissue types. Emerging ultrasound imaging technologies have therefore focused on extracting tissue parameters important for tissue differentiation such as scatterer size, tissue elasticity, and micro-vasculatures. Among these technologies, shear wave elastography (SWE) is an approach that measures tissue viscoelastic parameters. Our group has proposed Temporal Enhanced Ultrasound (TeUS) that differentiates tissue types without requiring any external stimuli. Through analytical derivations and simulations, we previously showed that the source of tissue typing information in TeUS is physiological micro-vibrations resulting mainly from perfusion. We further demonstrated that TeUS is sensitive to the size and distributions of scatterers in the tissue, as well as its visco-elasticity. In this paper, we designed ultrasound phantoms to mimic tissue with two different elasticities and two scatterer sizes. A exible microtube was embedded in the phantoms to generate local micro-vibrations. We experimentally demonstrate the relationship between TeUS and SWE and their sensitivity to tissue elasticity and scatterer size. This work indicated that while shear wave measurements are sensitive to the phantoms viscoelasticity, they are not sensitive to ultrasound scatterer size. On the contrary, the TeUS amplitude depends on both scatterer size and tissue viscoelasticity. This work could potentially inform clinicians of choosing imaging modalities and interventions based on each cancer's unique traits and properties.

Ultrasound is a cost-effective and real-time modality for image-guided intervention for challenging and complication- prone procedure. Conventional ultrasound machines usually require the use of expensive probes and bulky electronics because of the need to acquire and process hundreds of channel data simultaneously. In contrast, a single-element ultrasound system creates a virtual array by scanning an ultrasound element with robotic actuation and tracking instead of using a physical array of ultrasound elements as in conventional ultrasound machine. It not only enables visualization of procedures that otherwise require a custom probe but also dramatically reduces cost and improves the accessibility of image-guided intervention in point-of-care applications. In this work, we present a single-element ultrasound imaging system with a delta configuration actuator modified from a low-cost commercial off-the-shelf 3-D printer, which can serve as both a prototype for clinical application and a research platform. We demonstrated the capability of the system with experiments of spine visualization. We scanned a spine phantom with a needle-based ultrasound. The results indicate the feasibility of compact and economically friendly single-element ultrasound imaging solution for spinal intervention applicable to guiding lumbar puncture and epidural needle insertion.

Ultrasound-guided interventions are progressively incorporating additional augmented reality (AR) components to improve navigation. A fundamental requirement to integrate ultrasound (US) images into AR environments is US probe calibration, which places images from a tracked US probe in the context of the tracker. To improve the incompatibility of common US probe calibration methods with clinical environments, Chen et al. developed an US calibration method (GUScal) with a focus for intraoperative application. To understand the effect of image quality and tracking accuracy on this calibration method, novice users were recruited to perform the calibration under three different conditions: (1) free-hand with one focal depth, (2) free-hand with three focal depths, and (3) using a mechanical arm to fix the needle with one focal depth. An expert user repeated this process 15 times per condition. The resultant transformation matrices and associated times were recorded for all calibrations. Numerical and visual analysis was conducted to compare the users calibrations results to each other and to the expert results. Based on our results we concluded including multiple focal depths produced the most precise calibrations for novice users. The expert user results showed that stabilizing the needle through a mechanical arm improved the calibration with practice. We recommend the inclusion of additional focal depths and a method to stabilize the needle to produce an accurate calibration in approximately 5 minutes.

Liver cancer is the second and sixth most frequent cause of cancer mortality worldwide in men and women, respectively, with high prevalence in under developed and developing countries. Minimally invasive focal ablation of liver tumors is an alternative technique to resection and transplantation for early-stage cancer and is focused on reducing patient complications and recovery times. Although promising, the therapeutic benefits are currently present with high local cancer recurrence. One potential source of error arises when performing therapy applicator guidance with 2D ultrasound (US) since the field-of-view is limited and requires the physician to build a mental image of the anatomy. Our solution to this limitation has been the development of a novel mechanically assisted 3D US imaging and guidance system capable of providing geometrically variable images. A three-motor mechanical mover was designed to provide linear, tilt, and hybrid geometries with adjustable ranges of motion for variable 3D US fields-of-view. This mover can manipulate any clinically available 2D US transducer via transducer-specific 3D-printed holders to guide applicator insertions intraoperatively. This mover is held by a counterbalanced mechanical “arm and wrist”, which contain electromagnetic brakes and five encoders to track the position of the transducer. This mechanical support is mounted on a portable cart with coarse adjustable height to accommodate gross differences in patient sizes. This work represents the design, construction, software implementation, preliminary 3D volume reconstruction evaluation, and the first qualitative human volunteer scans. Geometric errors performed on a grid phantom were <3% and human volunteer images were clinically applicable.

In transperineal prostate biopsy or ablation, a grid-template is typically used to guide the needle. The guidance method has limited positioning resolution and lack of needle angulation selections that are referenced to ultrasound imaging or TRUS-MRI fusion targets. To overcome the limitation, a novel augmented reality (AR) system that use smart see-through glasses and smartphone as a needle guidance device for transperineal prostate procedure was developed. The AR system is comprised of a MRI/CT scanner, a pre-procedural image analysis and visualization software, AR devices (smart-glasses, smartphone), a newly-developed AR app, as well as a local network. The AR app displays the lesion and planned needle trajectory, which are derived from the pre-procedural images, on the AR devices. A special designed image marker frame that affixed to the patient’s perineum was used to track the pre-procedural image with the AR devices. The displayed needle plan was always referenced to the patient and remains independent from the position and orientation of the devices. Multiple devices can be used simultaneously and communicate via a local network. We evaluated the AR system accuracy with iPhone and R-7 glasses in a phantom study. The image overlay accuracy was 0.58±0.43^o and 1.62±1.52^o in iPhone and R-7 glasses respectively. The accuracy of iPhone guidance was 1.9±0.97 mm (lateral) and 1.0±0.5 mm (in-direction), the accuracy of R-7 guidance was 2.8±1.4mm (lateral) and 2.3±1.5mm (indirection). AR system using smart-glasses and smartphone can provide accurate needle guidance and see-through-the-skin display for needle based transperineal prostate interventions like biopsy and ablation.

Brain cancer surgery has the goal of performing an accurate resection of the tumor and preserving as much as possible the quality of life of the patient. There is a clinical need to develop non-invasive techniques that can provide reliable assistance for tumor resection in real-time during surgical procedures. Hyperspectral imaging (HSI) arises as a new, noninvasive and non-ionizing technique that can assist neurosurgeons during this difficult task. In this paper, we explore the use of deep learning (DL) techniques for processing hyperspectral (HS) images of in-vivo human brain tissue. We developed a surgical aid visualization system capable of offering guidance to the operating surgeon to achieve a successful and accurate tumor resection. The employed HS database is composed of 26 in-vivo hypercubes from 16 different human patients, among which 258,810 labelled pixels were used for evaluation. The proposed DL methods achieve an overall accuracy of 95% and 85% for binary and multiclass classifications, respectively. The proposed visualization system is able to generate a classification map that is formed by the combination of the DL map and an unsupervised clustering via a majority voting algorithm. This map can be adjusted by the operating surgeon to find the suitable configuration for the current situation during the surgical procedure.

Immersive, stereoscopic displays may be instrumental to better interpreting 3-dimensional (3D) data. Further- more, the advent of commodity-level virtual reality (VR) hardware has made this technology accessible for meaningful applications, such as medical education. Accordingly, in the current work we present a commodity- level, immersive simulation for interacting with human ear anatomy. In the simulation, users may interact simultaneously with high resolution computed tomography (CT) scans and their corresponding, 3D anatomical structures. The simulation includes: (1) a commodity level, immersive virtual environment presented by the Oculus CV1, (2) segmented 3D models of head and ear structures generated from a CT dataset, (3) the ability to freely manipulate 2D and 3D data synchronously, and (4) a user-interface which allows for free exploration and manipulation of data using the Oculus touch controllers. The system was demonstrated to 10 otolaryngolo- gists for evaluation. Physicians were asked to supply feedback via both questionnaire and discussion in order to determine the efficacy of the current system as well as the most pertinent applications for future research.

Background: Current display technologies can be suboptimal, even inadequate, when conveying spatially-complex healthcare concepts to patients and providers. This can lead to difficulties in sharing medical information, with potentially deleterious effects on patient outcomes, research, and trainee education. Methods: Investigators used off-the-shelf augmented reality (AR) technologies to facilitate visual communication for healthcare. Using untethered headset devices (Microsoft HoloLens), proof-of-concept work was completed for two use-cases: 1.) multi-user shared AR visualizations and 2.) AR-guided invasive procedural performance. The research team collaborated to create: 1.) a shared AR environment that enabled multiple users to independently visualize and manipulate 3D patient anatomic models with position, rotation, and scale synchronized across users; and 2.) a hybrid [AR-physical] covered box configuration containing CT-scanned targets and custom trajectory guidance system for simulated needle aspiration. As a pilot study exploring technical feasibility and experimental viability of the selected approach, measurements of 1.) size, displacement, angular rotation, and 2.) expert aspiration success were used as primary metrics. Results: The mean difference between AR models and physical objects was 2.0±0.4% and 1.7±0.4% of all dimensions on two HoloLens devices. One shared model configuration exhibited deviations of 7.8±2.8mm in location and 0.5±0.9° in angular orientation, and another showed differences of 6.5±2.1mm and 0.1±0.7° . For AR-guided procedure simulations, two expert surgeons required 3 attempts in 10 minutes and 1 attempt in 3 minutes to successfully aspirate the hybrid targets. Conclusion: AR technologies were used to enable core elements of an interactive shared medical visualization environment and a guided procedure simulation.

PURPOSE: There is a lack of open-source or free virtual reality (VR) software that can be utilized for research by medical professionals and researchers. We propose the design and implementation of such software. We also aim to assess the feasibility of using VR as a modality for navigating 3D visualizations of medical scenes. METHODS: To achieve our goal, we added VR capabilities to the open-source medical image analysis and visualization platform, 3D Slicer. We designed the VR extension by basing the software architecture on VTK’s vtkRenderingOpenVR software module. We extended this module by adding features such as full interactivity between 3D Slicer and the VR extension during VR use, variable volume rendering quality based on user headset motion etc. Furthermore, the VR extension was tested in a feasibility study in which participants were asked to complete specific tasks using bot the conventional mouse-monitor and VR method. For this experiment, we used 3D Slicer to create two virtual settings, each having an associated task. Participants were asked to maneuver the virtual settings using two approaches, the conventional method, using mouse and monitor, and VR using the head-mounted-display and controllers. The main outcome measure was total time to complete the task. RESULTS: We developed a VR extension to 3D Slicer—SlicerVirtualReality (SlicerVR). Additionally, from the experiment we conducted we found that when comparing mean completion times, participants, when using VR, were able to complete the first task 3 minutes and 28 seconds quicker than the mouse and monitor method (4 minutes and 24 seconds vs. 7 minutes and 52 seconds, respectively); and the second task 1 minute and 20 seconds quicker (2 minutes and 37 seconds, vs. 3 minutes and 57 seconds, respectively). CONCLUSION: We augmented the 3D Slicer platform with virtual reality capabilities. Experiments results show a considerable improvement in time required to navigate and complete tasks within complex virtual scenes compared to the traditional mouse and monitor method.

PURPOSE: Lumpectomy is the resection of a tumor in the breast while retaining as much healthy tissue as possible. Navigated lumpectomy seeks to improve on the traditional technique by employing computer guidance to achieve the complete excision of the cancer with optimal retention of healthy tissue. Setting up navigation in the OR relies on the manual interactions of a trained technician to align three-dimensional virtual views to the patient’s physical position and maintain their alignment throughout surgery. This work develops automatic alignment tools to improve the operability of navigation software for lumpectomies. METHODS: Preset view buttons were developed to refine view setup to a single interaction. These buttons were tested by measuring the reduction in setup time and the number of manual interactions avoided through their use. An auto-center feature was created to ensure that three-dimensional models of anatomy and instruments were in the center of view throughout surgery. Recorded data from 32 lumpectomy cases were replayed and the number of auto-center view shifts was counted from the first cautery incision until the completion of the excision of cancerous tissue. RESULTS: View setup can now be performed in a single interaction compared to an average of 13 interactions (taking 83 seconds) when performed manually. The auto-center feature was activated an average of 33 times in the cases studied (n=32). CONCLUSION: The auto-center feature enhances the operability of the surgical navigation system, reducing the number of manual interactions required by a technician during the surgery. This feature along with preset camera view options are instrumental in the shift towards a completely surgeon-operable navigated lumpectomy system.

Introduction. Treatment choice for extracranial carotid artery widening, also called aneurysm, is difficult. Blood flow simulation and experimental visualization can be supportive in clinical decision making and patient-specific treatment prediction. This study aims to simulate and validate the effect of flow-diverting stent placement on blood flow characteristics using numerical and in vitro simulation techniques in simplified carotid artery and aneurysm models. Methods. We have developed a workflow from geometry design to flow simulations and in vitro measurements in a carotid aneurysm model. To show feasibility of the numerical simulation part of the workflow that uses an immersed boundary method, we study a model geometry of an extracranial carotid artery aneurysm and put a flow-diverting stent in the aneurysm. We use ultrasound particle image velocimetry (PIV) to visualize experimentally the flow inside the aneurysm model. Results. Feasibility of ultrasound visualization of the flow, virtual flow-diverting stent placement and numerical flow simulation are presented. Flow is resolved to scales much smaller than the cross section of individual wires of the flow-diverting stent. Numerical analysis in stented model introduced 25% reduction of the blood flow inside the aneurysm sac. Quantitative comparison of experimental and numerical results showed agreement in 1D velocity profiles. Discussion/conclusion. We find good numerical convergence of the simulations at appropriate spatial resolutions using the immersed boundary method. This allows us to quantify the changes in the flow in model geometries after deploying a flow-diverting stent. We visualized the physiological blood flow in a 1-to-1 aneurysm model, using PIV, showing a good correspondence to the numerical simulations. The novel workflow enables numerical as well as experimental flow simulations in patient-specific cases before and after flow-diverting stent placement. This may contribute to endovascular treatment prediction.

Modern image-guided cervical cancer brachytherapy involves the insertion of hollow applicators in the uterus and surrounding the cervix to deliver a radioactive source. These applicators are imaged and manually digitized following insertion for treatment planning. We present an algorithm to automatically digitize these applicators using MRI for cervical cancer brachytherapy planning. Applicators were digitized in vivo using T2-weighted MR images (1.5 T) from 21 brachytherapy fractions including 9 patients. The model-to-image registration algorithm was implemented in C++ involving a 2D matched filter to identify the applicator center, and a 3D surface model to identify local position by maximizing the intensity gradient normal to the surface. Surface models were produced using training MR images. Errors in the algorithm results were calculated as the 3D distances of the applicator tip and center from those identified manually. A model based on manufacturer data was also used for applicator digitization to assess algorithm sensitivity to surface model variation. The algorithm correctly identified the applicator in 20 out of 21 images with mean execution time of 2.5 s. Mean±SD error following digitization using the MRI and manufacturer-based surface models was 1.2±0.6 mm and 1.3±0.7 mm for the tandem tip (p = 0.52), and 1.4±0.9 mm and 1.3±0.7 mm for the ring center (p = 0.61). The algorithm requires no manual initialization with consistent results across surface models, showing promise for clinical implementation.

In this paper we report development of an integrated RF coil for our body-mounted arthrography robot called Arthrobot. Arthrography is the evaluation of joint conditions using imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). Current arthrography practice requires two separate stages; an intra-articular contrast injection guided by fluoroscopy or ultrasound followed by MR imaging. Our body-mounted robot is intended to enable needle placement in the MRI environment, streamlining the procedure. To improve imaging with our robot, a single loop coil was created and embedded into the mounting adaptor of the robot. This coil provides enough spatial coverage and sensitivity to localize anatomical points of interest and registration fiducials on the robot frame. In this paper we report the results of a SNR and heating study using our custom-made RF coil in four different scenarios using T1 and T2 weighted MR images: 1) no robot present, 2) robot off, 3) robot powered on, and 4) robot running. We also report an end-to-end robotic-assisted targeting study in a Philips MRI scanner suite using Arthrobot and our custommade RF coil for image acquisition. The SNR results and targeting results were promising. SNR dropped 32% for T1 weighted images compared to baseline (no robot present) images. For T2 weighted images the SNR drop was 42%. The average targeting error was 2.91 mm with a standard deviation (SD) of 1.82 mm. In future work we plan to replace the passive fiducials embedded in the base of Arthrobot with active fiducials that are tracked by the MRI system. These active fiducials will enable real-time tracking of the robot base and could allow breathing motion compensation during robotic procedures.

We propose a new method to support tympanoplasty operations for ear, nose and throat (ENT) surgery, namely the reconstruction of the eardrum. This intervention needs precise distance and contour measurements to allow a successful operation. Currently, surgeons only have limited tools to measure patient specific anatomy and rely on their experience and visual judgement to estimate anatomical dimensions. Therefore, we provide an image-based augmented reality (AR) measuring tool using a complete digital processing chain, giving instant feedback about anatomical structures with high metric accuracy. Our method detects the center of gravity of a marked spherical instrument tip in the stereoscopic image pair of a digital surgical microscope and triangulates points in 3D space of the calibrated stereo system. We track the tip using a self-updating template-matching algorithm. An accurate subpixel refinement of the tip center prevents drift and guarantees highly accurate stereo correspondences. GPU implementation and a color-based pre-detection allows real-time tracking in high-resolution images. Reconstructed trajectories form a true-scale virtual stencil, which is projected directly into the surgeon’s field of view as a precise registered AR overlay. This overlay supports the surgeon while cutting a patient specific shape from autologous tissue. Measurement accuracy and real-time tracking performance are evaluated using a depth-of-field test body and a temporal bone model, where the obtained 3D path-reconstruction is compared to a CT scan. Our approach provides great potential to improve state-of-the-art surgical workflows by reducing operating times and facilitating intraoperative decisions.

Head and neck squamous cell carcinoma (SCCa) is primarily managed by surgical resection. Recurrence rates after surgery can be as high as 55% if residual cancer is present. In this study, hyperspectral imaging (HSI) is evaluated for detection of SCCa in ex-vivo surgical specimens. Several methods are investigated, including convolutional neural networks (CNNs) and a spectral-spatial variant of support vector machines. Quantitative results demonstrate that additional processing and unsupervised filtering can improve CNN results to achieve optimal performance. Classifying regions that include specular glare, the average AUC is increased from 0.73 [0.71, 0.75 (95% confidence interval)] to 0.81 [0.80, 0.83] through an unsupervised filtering and majority voting method described. The wavelengths of light used in HSI can penetrate different depths into biological tissue, while the cancer margin may change with depth and create uncertainty in the ground-truth. Through serial histological sectioning, the variance in cancer-margin with depth is also investigated and paired with qualitative classification heat maps using the methods proposed for the testing group SCC patients.

The primary liver cancer including intrahepatic bile duct cancer pose significant global burden of illness with increasing incidence and mortality in US and around the world. Surgery remains the most effective form of treatment. However, surgical complication rates for medium to high complexity hepatectomies persist in 30-40% range even in highly skilled hands and at high volume centers. The critical challenges appear to be attributable to navigating liver parenchymal dissection, where size of resection surface, associated with blood loss and missed bile leaks from the liver parenchyma, and prolonged operative time during dissection pose significant obstacle. In this work, we present a new laparoscopic real-time liver flow display of subsurface liver structures (e.g., intrahepatic artery, portal vein, and bile duct) by creating a ‘Surgical Map’ to guide liver parenchymal dissection in hepatobiliary surgery. The intelligent display of intrahepatic critical structures and functional physiology in real-time can make the hepatic dissection safer and more efficient for any liver surgery. We integrated multimodal optical imaging technologies into a single laparoscopic vision tool, created a continuously evolving quantitative surgical map based on Bayesian framework, and finally validated the usefulness of Surgical Map through preclinical porcine studies.

Because coronary artery calcified plaques can hinder or eliminate stent deployment, interventional cardiologists need a better way to plan interventions, which might include one of the many methods for calcification modification (e.g., atherectomy). We are imaging calcifications with intravascular optical coherence tomography (IVOCT), which is the lone intravascular imaging technique with the ability to image the extent of a calcification, and using results to build vesselspecific finite element models for stent deployment. We applied methods to a large set of image data (<45 lesions and < 2,600 image frames) of calcified plaques, manually segmented by experts into calcified, lumen and “other” tissue classes. In optimization experiments, we evaluated anatomical (x, y) versus acquisition (r,θ) views, augmentation methods, and classification noise cleaning. Noisy semantic segmentations are cleaned by applying a conditional random field (CRF). We achieve an accuracy of 0.85 ± 0.04, 0.99 ± 0.01, and 0.97 ± 0.01, and F-score of 0.88 ± 0.07, 0.97 ± 0.01, and 0.91 ± 0.04 for calcified, lumen, and other tissues classes respectively across all folds following CRF noise cleaning. As a proof of concept, we applied our methods to cadaver heart experiments on highly calcified plaques. Following limited manual correction, we used our calcification segmentations to create a lesion-specific finite element model (FEM) and used it to predict direct stenting deployment at multiple pressure steps. FEM modeling of stent deployment captured many features found in the actual stent deployment (e.g., lumen shape, lumen area, and location and number of apposed stent struts).

The use of pre-operative CT and MR images for navigation during endo-nasal skull-base endoscopic surgery is a well-established procedure in clinical practice. Fusion of CT and MR images on the endoscopic view can offer an additional advantage by directly overlaying surgical-planning information in the surgical view. Fusion of intraoperative images, such as cone beam computed tomography (CBCT), represents a step forward since these images can also account for intra-operative anatomical changes. In this work, we present a method for intra-operative CBCT image fusion on the endoscopic view for endo-nasal skull-base surgery, implemented on the Philips surgical navigation system. This is the first study which utilizes an optical tracking system (OTS) embedded in the flat-panel detector of the C-arm for endoscopic-image augmentation. In our method the OTS, co-registered in the same CBCT coordinate system, is used for tracking the endoscope. Accuracy in CBCT image registration in the endoscopic view is studied using a calibration board. Image fusion is tested in a realistic surgical scenario by using a skull phantom and inserts that mimic critical structures at the skull base. Overall performances tested on the skull phantom show a high accuracy in tracking the endoscope and registration of CBCT on endoscopic view. It can be concluded that the implemented system show potential for usage in endo-nasal skull-base surgery.

We present our latest work on designing a magnetically anchored wireless stereoscopic robot with 2 degrees of freedom (DOF) Pan-Tilt unit for single-port minimally invasive surgery (MIS). This camera could reduce the tool clashing issue in MIS and could provide better angulation and visualization of surgical field. After introduction of the robot through umbilicus (belly button), it is anchored to internal abdominal wall using a magnet from outside. Surgeon can change view angle of the camera remotely via a wireless joystick and a real-time stereo view will be displayed on a user interface screen. Since the robot is anchored using an external magnet on the abdominal wall during the surgical operation, surplus shocks and slight tremble of the robot will result in poor visualization. Therefore, we developed a real-time video stabilization scheme to eliminate these affects. Our proposed method uses a high frequency inertial measurement sensory data fused with visual optical flow vectors, extracted from the stereo camera, to estimate the unwanted shocks during the video streaming. This method compensates and stabilizes video streams in real-time by shifting the video images in the opposite direction of the estimated motion vector. We conducted several experiments including robot control, video streaming performance, and real-time video stabilization to investigate the system function. The results of these experiments are reported in this paper.

Over the last 25 years, the number of papers written that involve image guidance, liver, and registration has, on average, doubled every 6-8 years. While image guidance has a long history within the neurosurgical theatre, it’s translation to other soft-tissue organs such as the liver has been slower given the inherent difficulty in image-to-physical registration. More specifically, deformations have been recognized to compromise image guidance fidelity in virtually all soft-tissue image guidance applications. As a result, an active area of investigation is the development of sparse-data-driven nonrigid image-to-physical liver registration techniques to compensate for deformation and provide accurate localization for image guided liver surgery. In this work we have leveraged our extensive human-to-phantom registration testing framework based on the work in [1] and Amazon Web Services to create a sparse data challenge for the image guided liver surgery community (https://sparsedatachallenge.org/). Our sparse data challenge will allow research groups from across the world to extensively test their approaches on common data and have quantitative accuracy measurements provided for assessment of fidelity. Welcome to the Sparse Data Challenge for image-to-physical liver registration assessment.

Deep metrics have been shown effective as similarity measures in multi-modal image registration; however, the metrics are currently constructed from aligned image pairs in the training data. In this paper, we propose a strategy for learning such metrics from roughly aligned training data. Symmetrizing the data corrects bias in the metric that results from misalignment in the data (at the expense of increased variance), while random perturbations to the data, i.e. dithering, ensures that the metric has a single mode, and is amenable to registration by optimization. Evaluation is performed on the task of registration on separate unseen test image pairs. The results demonstrate the feasibility of learning a useful deep metric from substantially misaligned training data, in some cases, the results are significantly better than from Mutual Information. Data augmentation via dithering is, therefore, an effective strategy for discharging the need for well-aligned training data; this brings deep learning based registration from the realm of supervised to semi-supervised machine learning.

Purpose. Intraoperative 2D virtual long-film (VLF) imaging is investigated for 3D guidance and confirmation of the surgical product in spinal deformity correction. Multi-slot-scan geometry (rather than a single-slot “topogram”) is exploited to produce parallax views of the scene for accurate 3D colocalization from a single radiograph. Methods. The multi-slot approach uses additional angled collimator apertures to form fan-beams with disparate views (parallax) of anatomy and instrumentation and to extend field-of-view beyond the linear motion limits. Combined with a knowledge of surgical implants (pedicle screws and/or spinal rods modeled as “known components”), 3D-2D image registration is used to solve for pose estimates via optimization of image gradient correlation. Experiments were conducted in cadaver studies emulating the system geometry of the O-arm (Medtronic, Minneapolis MN). Results. Experiments demonstrated feasibility of multi-slot VLF and quantified the geometric accuracy of 3D-2D registration using VLF acquisitions. Registration of pedicle screws from a single VLF yielded mean target registration error of (2.0±0.7) mm, comparable to the accuracy of surgical trackers and registration using multiple radiographs (e.g., AP and LAT). Conclusions. 3D-2D registration in a single VLF image offers a promising new solution for image guidance in spinal deformity correction. The ability to accurately resolve pose from a single view absolves workflow challenges of multiple-view registration and suggests application beyond spine surgery, such as reduction of long-bone fractures.

In this paper, a deformable registration method is proposed that enables automatic alignment of preoperative PET/CT to intraoperative ultrasound in order to achieve PET-determined focal prostate brachytherapy. Novel PET imaging agents such as prostate specific membrane antigen (PSMA) enables highly accurate identification of intra/extra-prostatic tumors. Incorporation of PSMA PET into the standard transrectal ultrasound (TRUS)-guided prostate brachytherapy will enable focal therapy, thus minimizing radiation toxicities. Our registration method requires PET/CT and TRUS volume as well as prostate segmentations. These input volumes are first rigidly registered by maximizing spatial overlap between the segmented prostate volumes, followed by the deformable registration. To achieve anatomically accurate deformable registration, we extract anatomical landmarks from both prostate boundary and inside the gland. Landmarks are extracted along the base-apex axes using two approaches: equiangular and equidistance. Three-dimensional thin-plate spline (TPS)-based deformable registration is then performed using the extracted landmarks as control points. Finally, the PET/CT images are deformed to the TRUS space by using the computed TPS transformation. The proposed method was validated on 10 prostate cancer patient datasets in which we registered post-implant CT to end-of-implantation TRUS. We computed target registration errors (TREs) by comparing the implanted seed positions (transformed CT seeds vs. intraoperatively identified TRUS seeds). The average TREs of the proposed method are 1.98±1.22 mm (mean±standard deviation) and 1.97±1.24 mm for equiangular and equidistance landmark extraction methods, respectively, which is better than or comparable to existing state-of-the-art methods while being computationally more efficient with an average computation time less than 40 seconds.

Laser interstitial thermal therapy (LITT) is a novel minimally-invasive neurosurgical ablative tool that is par ticularly well-suited for treating patients suffering from drug-resistant mesial temporal lobe epilepsy (mTLE). Although morbidity to patients is lower with LITT compared to the open surgical gold standard, seizure freedom rates appear inferior, likely a result of our lack of knowledge of which mesial temporal subregions are most critical for treating seizures. The wealth of post-LITT imaging and outcomes data provides a means for elucidating these critical zones, but such analyses are hindered by variations in patient anatomy and the distribution of these novel data among multiple academic institutions, each employing different imaging and surgical protocols. Adequate population analyses of LITT outcomes require normalization of imaging and clinical data to a common reference atlas. This paper discusses a method to nonrigidly register preoperative images to an atlas and quantitatively evaluate its performance in our region of interest, the hippocampus. Knowledge of this registration error would allow us to both select an appropriate registration method and define our level of confidence in the correspondence of the postoperative images to the atlas. Once the registration process is validated, we aim to create a statistical map from all the normalized LITT ablation images to analyze and identify factors that correlate with good outcomes.

Cochlear implants (CIs) are considered the standard-of-care treatment for severe-to-profound, sensorineural hearing loss. The positioning of the array within the cochlea affects which auditory nerve fibers are stimulated by which electrode and is known to affect hearing outcomes. Image-Guided CI Programming (IGCIP) techniques, where estimates of the position of the electrodes relative to the nerve fibers are provided to the programming audiologist, have been shown to lead to significantly improved hearing outcomes. With the current IGCIP approach, assumptions are made about electrical current spread to estimate which fiber groups are activated based on their distance to the electrode. To improve our estimates, we are developing an approach for creating patient-customized, high-resolution, electro-anatomical models of the electrically stimulated cochlea coupled with computational auditory nerve fiber models (ANFMs) to permit physics-based estimation of neural stimulation patterns. In this paper, our goal is to evaluate semi- and fully-automatic techniques for segmenting auditory nerve fibers that will be used in creating ANFMs, as well as to quantify the effect of change in fiber location on the neural activation patterns. Our semi-automatic approach uses path finding algorithms to connect automatically estimated landmarks, and our automatic approach is atlas-based. We found that repeatability in fiber localization with semi-automatic segmentation is 0.1 mm on average and results in modeled activation patterns that have 83% overlap. The difference between the semi-automatic and automatic segmentations led to higher average differences of 0.19 mm and lower activation pattern overlap of 74%.

Cochlear implants (CIs) use a surgically implanted electrode array to treat severe-to-profound sensorineural hearing loss. Audiologists program CIs by selecting a number of stimulation parameters for the CI processor to optimize hearing performance. It has been shown in previous research that audiologists arrive at CI settings that lead to a better hearing outcome when they are provided an estimate of which regions of the auditory nerve are being activated by each electrode for a patient. If the neural fibers could be localized, neural fiber models could be used to estimate activa tion in response to electrode activation for individual patients. However, the neural fibers are so small they are not visible in clinical images. In this project, our aim is to develop an active-shape model based solution to automatically localize the Internal Auditory Canal (IAC), which houses the auditory nerves and has borders that are visible in CT scans, to serve as a landmark for localizing the nerve fibers . Seven manually segmented IAC volumes were used to create and validate our method using a leave-one-out approach. We found that the mean surface errors of the dataset ranged from ~0.4 to ~1.2 CT voxels (0.13 mm to 0.37 mm). These results suggest that our IAC segmentation is highly accurate and could provide an excellent landmark for estimating fiber position.

For prostate cancer patients, large organ deformations occurring between the sessions of a fractionated radiotherapy treatment lead to uncertainties in the doses delivered to the tumour and the surrounding organs at risk. The segmentation of those structures in cone beam CT (CBCT) volumes acquired before every treatment session is desired to reduce those uncertainties. In this work, we perform a fully automatic bladder segmentation of CBCT volumes with u-net, a 3D fully convolutional neural network (FCN). Since annotations are hard to collect for CBCT volumes, we consider augmenting the training dataset with annotated CT volumes and show that it improves the segmentation performance. Our network is trained and tested on 48 annotated CBCT volumes using a 6-fold cross-validation scheme. The network reaches a mean Dice similarity coefficient (DSC) of 0:801 ± 0:137 with 32 training CBCT volumes. This result improves to 0:848 ± 0:085 when the training set is augmented with 64 CT volumes. The segmentation accuracy increases both with the number of CBCT and CT volumes in the training set. As a comparison, the state-of-the-art deformable image registration (DIR) contour propagation between planning CT and daily CBCT available in RayStation reaches a DSC of 0:744 ± 0:144 on the same dataset, which is below our FCN result.

Segmentation of anatomical structures in intraoperative ultrasound (iUS) images during image-guided interventions is challenging. Anatomical variances and the uniqueness of each procedure impede robust automatic image analysis. In addition, ultrasound image acquisition itself, especially acquired freehand by multiple physicians, is subject to major variability. In this paper we present a robust and fully automatic neural-network-based segmentation of central structures of the brain on B-mode ultrasound images. For our study we used iUS data sets from 18 patients, containing sweeps before, during, and after tumor resection, acquired at the University Hospital Essen, Germany. Different machine learning approaches are compared and discussed in order to achieve results of highest quality without overfitting. We evaluate our results on the same data sets as in a previous publication in which the segmentations were used to improve iUS and preoperative MRI registration. Despite the smaller amount of data compared to other studies, we could efficiently train a U-net model for our purpose. Segmentations for this demanding task were performed with an average Dice coefficient of 0.88 and an average Hausdorff distance of 5.21 mm. Compared with a prior method for which a Random Forest classifier was trained with handcrafted features, the Dice coefficient could be increased by 0.14 and the Hausdorff distance is reduced by 7 mm.

Lung cancer is the number one cause of cancer in the United States. With the rollout of new lung cancer screening programs, the number of early-stage lung cancer patients is expected to increase. This greatly increases the premium on proper central-chest lymph node staging. Physicians typically use minimally invasive bronchoscopy for these staging procedures. On another front, it is now accepted practice that a bronchoscopic procedure is greatly assisted using image-guided virtual-bronchoscopy navigation (VBN) systems, whereby the procedure plan and follow-on live guidance draw upon information derived from a patient's three-dimensional (3D) computed- tomography (CT) chest scan. To date, VBN systems only provide a means for managing individual diagnostic sites. Truly effective lymph node staging, however, demands procedure plans that take into account multiple nodes. To bridge this gap, we propose an approach for planning a more comprehensive lymph node staging plan, suitable for VBN-based guidance systems. In addition, during the live guided procedure, existing VBN systems do not provide visualization aids pertinent to the demands of the lymph node staging problem. We discuss our progress toward developing a computer-based planning and guidance system tailored to more comprehensive lymph node staging. In particular, our development features three interconnected contributions. First, a system modernized to the newest eighth edition of the IASLC nodal-station map is presented. Second, a method for locating station-specific 3D soft-tissue regions, where candidate lymph nodes could reside, is discussed. Finally, an approach for computing airway routes leading to each nodal station, along with associated visualization views is given. We demonstrate our methodology using data from human lung cancer patients.

For peripheral pulmonary lesion diagnosis, surgical thoracoscopy and percutaneous needle biopsy are common invasive options, but entail significant risks; e.g., percutaneous biopsy carries a 15% pneumothorax rate and risk of other complications. The development of new bronchoscopic devices, such as radial-probe endobronchial ultrasound (RP-EBUS), however, enables far less risky lesion diagnosis. Based on recent research, an image- guided bronchoscopy system can be used to navigate the bronchoscope close to the lesion, while RP-EBUS, which provides real-time extraluminal information on local tissue and lesions, can then be used for lesion localization and biopsy site selection. Unfortunately, physician skill in using RP-EBUS varies greatly, especially for physicians not at expert centers. This results in poor biopsy yields. Also, current state-of-the-art image-guided bronchoscopy systems provide no means for guiding the use of the RP-EBUS. We describe progress toward devising a methodology that facilitates synchronization of the known chest CT-based guidance information to possible locations for invoking RP-EBUS. In particular, we describe a top-level CT-based mechanism that mimics the possible positions of the RP-EBUS probe, supplemented with an approach that simulates possible RP-EBUS views. Results with human patient data demonstrate the potential of the methodology.

During bronchoscopy, a physician uses the endobronchial video to help navigate and observe the inner airways of a patient's lungs for lung cancer assessment. After the procedure is completed, the video typically contains a significant number of uninformative frames. A video frame is uninformative when it is too dark, too blurry, or indistinguishable due to a build-up of mucus, blood, or water within the airways. We develop a robust and automatic system, consisting of two distinct approaches, to classify each frame in an endobronchial video sequence as informative or uninformative. Our first approach, referred as the Classifier Approach, focuses on using image-processing techniques and a support vector machine, while our second approach, the Deep-Learning Approach, draws upon a convolutional neural network for video frame classification. Using the Classifier Approach, we achieved an accuracy of 78.8%, a sensitivity of 93.9%, and a specificity of 62.8%. The Deep-Learning Approach, gave slightly improved performance, with an accuracy of 87.3%, a sensitivity of 87.1%, and a specificity of 87.6%.

Artificial Intelligence (AI) is increasingly becoming a tool to enhance various medical image analysis tasks with accuracies comparable to expert clinicians. Computer assisted detection and diagnosis, and image segmentation and registration have significantly benefited from AI. However, integration of AI into the clinical workflow has been slow due to requirements for libraries that are specific to each model, and also environments that are specific to clinical centers. These challenges demonstrate the need for an AI-based solution that can be integrated into any environment with minimum hardware and software overhead. Tesseract-Medical Imaging (Tesseract-MI) is an open-source, web-based platform which enables deployment of AI models while simultaneously providing standard image viewing and reporting schemes. The goal of Tesseract-MI is to augment 3D medical imaging and provide a 4^th dimension (AI) when requested by a user. As a case study, we demonstrate the utility of our platform and present ProstateCancer.ai, a web application for identification of clinically significant prostate cancer in MRI.

Standard treatment of prostate cancer has traditionally been invasive and radical, with the whole gland being treated, even in those instances of localized prostate cancer. The ongoing development of computational tools and improvements to MRI techniques has led to an increase in research exploring alternative treatment solutions. The goal of this study was to investigate the application of open-source computational tools to MR guided Prostate Cancer Focal Laser Ablation (PCaFLA). To this end, we explore the use of image registration tools within 3D Slicer to assess target coverage with the treated ablation zone. Three different registration tools were investigated for registration between a planning MRI (day of procedure MRI examination prior to ablation) and a post-treatment MRI (day of procedure MRI examination immediately after ablation). Prostate whole gland, focal tumor volume ROI, and ablation zone were segmented manually for each case. Overlap between the ablation zone and tumor ROI was quantified using volume overlap measures and directed Hausdorff distance (HD) metrics and experimental results were reported. Experimental results demonstrate the ability to apply open-source computational tools to investigate outcomes of focal laser ablation of the prostate from imaging. This study provides a preliminary evaluation of the utility of open-source computational tools in intra-procedural feedback and post-treatment assessment for focal therapy.

Surgical instrument segmentation in laparoscopic image sequences can be utilized for a variety of applications during surgical procedures. Recent studies have shown that deep learning-based methods produce competitive results in surgical instrument segmentation. Difficulties, however, lie in the limited number of training datasets involving surgical instruments in laparoscopic image frames. Even though there are publicly available pixelwise training datasets along with trained models from the Robotic Instrument Segmentation challenge, we are not able to relate them to laparoscopic image frames from different surgical scenarios without any pre- or postprocessing. This is because they contain different instrument shapes, image backgrounds, and specular reflections, which implies laborious manual segmentation for training dataset generation. In this work, we propose a novel framework for semi-automated training dataset generation for the purpose of robust segmentation using deep learning. To generate training datasets in various surgical scenarios faster and more accurately, we utilize the publicly available trained model from the Robotic Instrument Segmentation challenge and then use the Watershed Segmentation-based method. For robust segmentation, we use a two-step approach: first, we obtain a coarse segmentation obtained from a deep convolutional neural network architecture, and then we refine the segmentation result via the GrabCut algorithm. Through experiments using four different laparoscopic image sequences, we demonstrate the ability of our proposed framework to provide robust segmentation quality.

Dual-energy subtraction angiography (DESA) using fast kV switching has received attention for its potential to reduce misregistration artifacts in thoracic and abdominal imaging where patient motion is difficult to control; however, commercial interventional solutions are not currently available. The purpose of this work was to adapt an x-ray angiography system for 2D and 3D DESA. The platform for the dual-energy prototype was a commercially available xray angiography system with a flat panel detector and an 80 kW x-ray tube. Fast kV switching was implemented using custom x-ray tube control software that follows a user-defined switching program during a rotational acquisition. Measurements made with a high temporal resolution kV meter were used to calibrate the relationship between the requested and achieved kV and pulse width. To enable practical 2D and 3D imaging experiments, an automatic exposure control algorithm was developed to estimate patient thickness and select a dual-energy switching technique (kV and ms switching) that delivers a user-specified task CNR at the minimum air kerma to the interventional reference point. An XCAT-based simulation study conducted to evaluate low and high energy image registration for the scenario of 30-60 frame/s pulmonary angiography with respiratory motion found normalized RMSE values ranging from 0.16% to 1.06% in tissue-subtracted DESA images, depending on respiratory phase and frame rate. Initial imaging in a porcine model with a 60 kV, 10 ms, 325 mA / 120 kV, 3.2 ms, 325 mA switching technique demonstrated an ability to form tissuesubtracted images from a single contrast-enhanced acquisition.

Transcatheter interventions for structural heart disease demand real-time visualization of catheter devices and their relationship to cardiac anatomy. Co-registration of x-ray fluoroscopy with echocardiography has been proposed to provide the necessary device and soft tissue visualization for these procedures. Development of real-time 3D x-ray/echo registration systems with device tracking has been hampered by the lack of a suitable test model. This study presents a phantom that is compatible with x-ray, CT, transthoracic (TTE), and transesophageal echo (TEE) for testing the feasibility and accuracy of new registration solutions. The phantom consists of a 20.3-cm diameter, 15-cm tall cylindrical shell with acoustic windows for TTE and an access port for a TEE probe. The interior contains 24 dual-modality targets, 5-mm in diameter, suspended in a three-turn helix occupying a volume that is similar to an adult heart. An ultrasound-compatible, tissue-mimicking slurry medium fills the remainder of the phantom. The dual-modality targets are agar based with barium sulfate (BaSO₄) powder and glass beads added to generate contrast in both x-ray and ultrasound. Appropriate concentrations of these additives were determined experimentally with contrast measurements in x-ray, CT, and ultrasound. Selected concentrations were 150 mg/mL BaSO₄ and 100 mg/mL of 53-63 μm diameter glass beads. Average target contrast (± SD) was 16% ± 2% in x-ray fluoroscopy (90 kV) and 1805 ± 99 HU in CT (100 kV). In ultrasound, target CNR was 4.30 ± 0.62 in 2D B-mode and 4.03 ±1.06 in 4D-mode images acquired at a center frequency of 2.8 MHz.

Ultrasound (US)-guided renal biopsy is a critically important tool in the evaluation and management of non-malignant renal pathologies with diagnostic and prognostic significance. It requires a good biopsy technique and skill to safely and consistently obtain high yield biopsy samples for tissue analysis. This project aims to develop a virtual trainer to help clinicians to improve procedural skill competence in real-time ultrasound-guided renal biopsy. This paper presents a cost-effective, high-fidelity trainer built using low-cost hardware components and open source visualization and interactive simulation libraries: interactive medical simulation toolkit (iMSTK) and 3D Slicer. We used a physical mannequin to simulate the tactile feedback that trainees experience while scanning a real patient and to provide trainees with spatial awareness of the US scanning plane with respect to the patient’s anatomy. The ultrasound probe and biopsy needle were modeled using commonly used clinical tools and were instrumented to communicate with the simulator. 3D Slicer was used to visualize an image sliced from a pre-acquired 3-D ultrasound volume based on the location of the probe, with a realistic needle rendering. The simulation engine in iMSTK modeled the interaction between the needle and the virtual tissue to generate visual deformations on the tissue and tactile forces on the needle which are transmitted to the needle that the user holds. Initial testing has shown promising results with respect to quality of simulated images and system responsiveness. Further evaluation by clinicians is planned for the next stage.

The accuracy of image guidance in spinal surgery can be compromised by intervertebral motion between preoperative supine CT images and intraoperative prone positioning. Patient registration and image updating approaches have been developed to register CT images with intraoperative spine and compensate for posture and alignment changes. We have developed a hand-held stereovision (HHS) system to acquire intraoperative profiles of the exposed spine and facilitate image registration and surgical navigation during open spinal surgery. First, we calibrated the stereo parameters using a checkerboard pattern, and the mean reprojection error was 0.33 pixel using 42 image pairs. Second, we attached an active tracker to the HHS device to track its location using a commercial navigation system. We performed spatial calibration to find the transformation between camera space and tracker space, and the error was 0.73 ± 0.39 mm. Finally, we evaluated the accuracy of the HHS using an ex-vivo porcine specimen. We used a tracked stylus to acquire locations of landmarks such as spinous and transverse processes, and calculated the distances between these points and the reconstructed stereovision surface. The resulting accuracy was 0.91 ± 0.58 mm, with an overall computational efficiency of ~ 5s for each image pair. Compared to our previous microscope-based stereovision system, the accuracy and efficiency of HHS are similar while HHS is more practical and functional, and would be more broadly applicable in spine procedures.

Transrectal ultrasound (TRUS) is considered the standard of care for imaging the prostate during biopsy and brachytherapy procedures. However, interpretation of TRUS images is challenging due to high specularity, making it difficult to recognize prostate boundaries. Image-guided brachytherapy and fusion-guided prostate biopsies require accurate non-rigid registration of magnetic resonance pre-operative image to intra-operative TRUS. State of the art techniques suggest semi-automated segmentation of the prostate on the TRUS images. However, due to the high variability, segmentation of the prostate is challenging. Segmentation errors could lead into poor localization of the biopsy target and can impact the registration of pre-operative images. In general, this kind of registration is challenging since the prostate anatomy undergoes motion due to TRUS probe pressure. In this paper, we propose a non-rigid surface registration approach for MR-TRUS fusion based on a statistical deformation model. Our method builds a statistical deformation model (SDM) of pre-operative to intra-operative deformations on a prostate dataset. In order to compute the fusion for an unseen MR-TRUS pair, the trained SDM is incorporated into the registration process to increase the fusion accuracy. The proposed approach is evaluated on a dataset of 23 patients with prostate cancer, for which the MRI-TRUS scans were available. We compared the proposed non-rigid SDM registration to non-rigid Iterative closest point (NICP) and rigid ICP approaches. Experiments demonstrate that the proposed SDM based method outperforms both NICP and ICP approaches, yielding a mean squared distance of 0.52 ± 0.26mm at the base, 0.45 ± 0.17mm mid-gland and 0.59 ± 0.13mm at the apex. These results show the advantage of integrating prior knowledge of deformation fields due to probe pressure for MR-TRUS fusion prostate interventions.

Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. This work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. In this paper, we introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filter-wheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 nm up to 975 nm. They are calibrated and validated using the same database and calibration set. For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus, the results acquired with both settings become transferable and comparable to each other as well as between different interventions. Clinical in-vivo data of two different oral and maxillofacial surgical procedures underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both setups. Thereby, we demonstrate their feasibility for the in-vivo analysis and differentiation of different human soft tissues.

Brain shift compensation attempts to model the deformation of the brain which occurs during the surgical removal of brain tumors to enable mapping of presurgical image data into patient coordinates during surgery and thus improve the accuracy and utility of neuro-navigation. We present preliminary results from clinical tumor resections that compare two methods for modeling brain deformation, a simple thin plate spline method that interpolates displacements and a more complex finite element method (FEM) that models physical and geometric constraints of the brain and its material properties. Both methods are driven by the same set of displacements at locations surrounding the tumor. These displacements were derived from sets of corresponding matched features that were automatically detected using the SIFT-Rank algorithm. The deformation accuracy was tested using a set of manually identified landmarks. The FEM method requires significantly more preprocessing than the spline method but both methods can be used to model deformations in the operating room in reasonable time frames. Our preliminary results indicate that the FEM deformation model significantly out-performs the spline-based approach for predicting the deformation of manual landmarks. While both methods compensate for brain shift, this work suggests that models that incorporate biophysics and geometric constraints may be more accurate.

Fluoroscopic imaging provides good visibility of prosthetic valve stent frames during transcatheter aortic valve replacement (TAVR) procedures. Recently, efforts have been made to perform 3D tracking of valve frames based on biplane fluoroscopic imaging; however, x-ray-based imaging lacks contrast in the soft tissue structures of the heart. The purpose of this work was to develop a system for real-time 3D visualization of transcatheter prosthetic valve frames and cardiac soft tissue anatomy using the fusion of biplane fluoroscopy and echocardiography. The system is a workstation that accepts real-time image streams from commercial biplane x-ray and echocardiography systems. The image datasets can be registered based on an x-ray visible fiducial apparatus attached to the echocardiography probe, or alternatively, manually registered. During the procedure, a real-time display shows the 3D valve representation together with adjustable cut-planes through the real time 3D ultrasound data. To validate the system, a phantom with cylindrical cavities was created using an agar-graphite mixture. Dual-modality fiducials were attached to the phantom for manual registration and evaluation of registration error. At maximum speed, the frame rate of the fusion system was 24.6 fps. The fiducial registration error between the ultrasound and x-ray imaging systems was 1.3 mm. The target registration error between the 3D prosthetic valve model reconstructed from biplane x-ray and a reference CT acquisition was 1.2 mm. The proposed system works with commercially available angiography and ultrasound systems and provides a novel 3D visualization of the prosthetic valve within the ultrasound image space in real-time.

Thermal ablation is a clinical procedure that aims at destroying pathological tissue minimally invasively through temperature changes. Temperature monitoring during the treatment is instrumental to achieve a precise and successful ablation procedure: ensuring a complete target ablation while preserving as much healthy tissue as possible. Ultrasound (US) is a promising low cost and portable modality, that could provide real-time temperature monitoring. However, the validation of such a technique is challenging. It is usually done with thermometers. They provide temperature measurements with good temporal resolution but only at a few local points. Magnetic Resonance Imaging (MRI) is the gold standard in term of temperature monitoring nowadays. It could also be used for validation of other thermometry techniques with a more accurate spatial resolution, but it requires MR-compatible devices. In this paper, we propose to leverage the use of a novel bipolar radiofrequency (RF) ablation device that provides 10 different ablation shapes to validate an ultrasound-based temperature monitoring method. The monitoring method relies on an external ultrasound element integrated with the bipolar RF ablation probe. This element send through the ablated tissues ultrasound waves that carry time-of-flight information. The ultrasound waves are collected by a clinical diagnostic ultrasound probe and can be related to the changes in temperature due to the ablation since ultrasound propagation velocity in biological tissue changes as temperature increases. We use this ultrasound-based method to monitor temperature during RF ablation. First on simulation data and then on two ex-vivo porcine liver experiments. Those dataset are used to show that we can validate the proposed temperature reconstruction method using the novel conformal radiofrequency ablation device by generating different ablation shapes.

Multi-modality image fusion of 3D coronary venous anatomy from fluoroscopic venograms with left ventricular (LV) epicardial surface from single-photon emission computed tomography (SPECT) myocardial perfusion image (MPI) can provide both LV physiological information and venous anatomy for guiding CRT LV lead placement. However, it is difficult to match the time points between MPI and venograms because of heart beating and thus image acquisition of the different cardiac frames, which affects the accuracy of 3D fusion. To address this issue, this study introduces a scale ratio iterative closest point (S-ICP) algorithm to non-rigidly fuse images from two different modalities. Three steps, including the image reconstruction, image registration, and image overlay were implemented to complete the images fusion. First, the 3D fluoroscopic venous anatomy and SPECT LV epicardial surface were reconstructed. Second, a landmark-based registration method was performed as an initial registration of S-ICP. With the initialization, the S-ICP algorithm with a preset scale range completed a fine registration for SPECT-vein fusion. Third, the registered venous anatomy was overlaid onto the SPECT LV epicardial surface. Moreover, in order to validate the accuracy of the fusion, 3D CT venous anatomy was manually fused with the same SPECT LV epicardial surface, and then the distance-based mismatch errors between fluoroscopic veins and CT veins were evaluated. Five patients were enrolled. As a result, the overall mismatch error was 5.6±4.1mm, which is smaller than the pixel size of SPECT images (6.4mm).

Image-guided surgery near anatomical or functional risk structures poses a challenging task for surgeons. To this end, surgical navigation systems that visualize the spatial relation between patient anatomy (represented by 3D images) and surgical instruments have been described. The provided 3D visualizations of these navigation systems are often complex and thus might increase the mental effort for surgeons. Therefore, an appropriate intraoperative visualization of spatial relations between surgical instruments and risk structures poses a pressing need. We propose three visualization methods to improve spatial perception in navigated surgery. A pointer ray encodes the distance between a tracked instrument tip and risk structures along the tool’s main axis. A side-looking radar visualizes the distance between the instrument tip and nearby structures by a ray rotating around the tool. Virtual lighthouses visualize the distances between the instrument tip and predefined anatomical landmarks as color-coded lights flashing between the instrument tip and the landmarks. Our methods aim to encode distance information with low visual complexity. To evaluate our concepts’ usefulness, we conducted a user study with 16 participants. During the study, the participants were asked to insert a pointer tool into a virtual target inside a phantom without touching nearby risk structures or boundaries. Results showed that our concepts were perceived as useful and suitable to improve distance assessment and spatial awareness of risk structures and surgical instruments. Participants were able to safely maneuver the instrument while our navigation cues increased participant confidence of successful avoidance of risk structures.

Intracardiac echocardiography (ICE) systems are routinely used in percutaneous cardiac interventions for interventional and surgical navigation. Conavi's Foresight ICE is a new ICE system that uses a mechanically rotating transducer to generate a 2D conical surface image in 3D space, in contrast to the more typical side-firing phased array. When combined with magnetic tracking technology, this unique imaging geometry poses new calibration challenges and opportunities. Existing ultrasound calibration methods are designed for 2D planar images and cannot be trivially applied to unique 2:5D conical surface images provided by the Foresight ICE system. In this work a spatial and temporal calibration technique applied to the unique case of conical ultrasound image data is described and validated. Precision of calibration parameters is used to quantify the validation of our calibration method and the overall system accuracy is validated using point source and sphere centroid localization. We re- port a maximum error of 5:07mm for point reconstruction accuracy and 1:94mm for sphere centroid localization accuracy.

Workflow driven support systems in the peri-operative area have the potential to optimize clinical processes and to allow new situation-adaptive support systems. We started to develop a workflow management system supporting all involved actors in the operating theatre with the goal to synchronize the tasks of the different stakeholders by giving relevant information to the right team members. Using the OMG standards BPMN, CMMN and DMN gives us the opportunity to bring established methods from other industries into the medical field. The system shows each addressed actor their information in the right place at the right time to make sure every member can execute their task in time to ensure a smooth workflow. The system has the overall view of all tasks. Accordingly, a workflow management system including the Camunda BPM workflow engine to run the models, and a middleware to connect different systems to the workflow engine and some graphical user interfaces to show necessary information or to interact with the system are used. The complete pipeline is implemented with a RESTful web service. The system is designed to include different systems like hospital information system (HIS) via the RESTful web service very easily and without loss of data. The first prototype is implemented and will be expanded.

The existing hybrid cardiac imaging approaches focus on predicting the adverse cardiac events or disease diagnosis, yet do not offer any insight into the pathological advance in the repair process. Angiogenesis is one of the most important mechanism in the repair process after ischemic injury and has shown benefit to the prognosis of occlusive cardiovascular disorders, thus becomes a target of molecular therapies. In vivo monitoring of angiogenesis and comprehensive evaluation of cardiac function associated with angiogenesis are urgently needed in both research and clinical practice. In this paper, a multimodality image fusion strategy was proposed for angiogenesis and viable myocardium identification. Imaging approaches including coronary computed tomography angiography(CCTA), 2-deoxy-2-[¹⁸F]fluoro-D-glucose ([¹⁸F]DG) PET/CT, [⁶⁸Ga]-1,4,7-triazacyclononane-1,4,7-triacetic acid-(Arg-Gly-Asp)2 ([⁶⁸Ga]-NOTA-PRGD2) PET/CT and ^99mTc-sestamibi (^99mTc-MIBI) myocardial perfusion SPECT/CT scanning were performed to acquire both anatomy and three kinds of function information. All of these modality images were then fused by an automatic strategy consisting of ROI segmentation and cross modality registration. The left ventricle myocardium was categorized into 4 groups based on fusion result according to the respective relative tracer uptake. The final results intuitively reflected the extent of the [¹⁸F]DG and ^99mTc-MIBI uptake defect, the perfusion-metabolism mismatch area, as well as the location of the [⁶⁸Ga]-NOTA-PRGD2 signal. The hybrid CCTA-PET-SPECT image verified the occurrence of angiogenesis based on the in vivo noninvasive molecular imaging approaches and visualized the hibernating myocardium. The presented fusion strategy is helpful in facilitating the study of the relationship between viability, perfusion and blocked coronary arteries, as well as angiogenesis.

PURPOSE: Identification of vertebral landmarks with ultrasound is a challenging task. We propose a step-wise computer-guided landmark identification method for developing 3D spine visualizations from tracked ultrasound images. METHODS: Transverse process bone patches were identified to generate an initial spine segmentation in real - time from live ultrasound images. A modified k-means algorithm was adapted to provide an initial estimate of landmark locations from the ultrasound image segmentation. The initial estimations using the modified k-means algorithm do not always provide a landmark on every segmented image patch. As such, further processing may improve the result captured from the sequences, owing to the spine’s symmetries. Five healthy subjects received thoracolumbar US scans. Their real- time ultrasound image segmentations were used to create 3D visualizations for initial validation of the method. RESULTS: The resulting visualizations conform to the parasagittal curvature of the ultrasound images. Our processing can correct the initial estimation to reveal the underlying structure and curvature of the spine from each subject. However, the visualizations are typically truncated and suffer from dilation or expansion near their superior and inferior-most points. CONCLUSION: Our methods encompass a step-wise approach to bridge the gap between ultrasound scans, and 3D visualization of the scoliotic spine, generated using vertebral landmarks. Though a lack of ground-truth imaging prevented complete validation of the workflow, patient-specific deformation is clearly captured in the anterior-posterior curvatures. The frequency of user-interaction required for completing the correction methods presents a challenge in moving towards full automation and requires further attention.

Evaluating tumor regression of rectal cancers via MRI after standard-of-care chemoradiation therapy (CRT) remains highly challenging for radiologists. While the tumor region-of-interest (ROI) on post-CRT rectal MRI is difficult to localize, an underexplored region is the perirectal fat (surrounding tumor and rectum) where residual cancer cells and positive lymph nodes are known to be present. Recent studies have shown that physiologic environments surrounding tumor regions may provide complementary information that is predictive of response to CRT and patient survival. We present initial results of characterizing perirectal fat regions on MRI via radiomics, towards capturing sub-visual details related to rectal tumor or nodal response to CRT. A total of 37 rectal cancer patients for whom MRIs as well as pathologic tumor staging were available post-CRT were included in this study. Region-wise radiomic features were extracted from expert annotated perirectal fat regions and a 2-stage feature selection was employed to identify the most relevant features. Radiomic entropy of perirectal fat was found to be over-expressed in patients with poor tumor or nodal response post-CRT, albeit with different spatial distributions. In a leave-one-patient-out cross validation setting, a quadratic discriminant analysis (QDA) classifier trained on top radiomic features from the perirectal fat achieved AUCs of 0.77 (for differentiating incomplete vs marked tumor regression) and 0.75 (for differentiating lymph node positive from negative patients). By comparison, perirectal fat intensities achieved significantly poorer AUCs in both tasks. Our results indicate perirectal fat on post-CRT MRI may be highly relevant for evaluating CRT response and informing follow-on interventions in rectal cancers.

PURPOSE: Spatially tracked ultrasound-guided needle insertions may require electromagnetic sensors to be clipped on the needle and ultrasound probe if not already embedded in the tools. It is assumed that switching the electromagnetic sensor clip does not impact the accuracy of the computed calibration. We propose an experimental process to determine whether or not devices should be calibrated on a more frequent basis. METHODS: We performed 250 calibrations. Of these, 125 were performed on the needle and 125 on the ultrasound. Every five calibrations, the tracking clip was removed and reattached. Every 25 calibrations, the tracking clip was exchanged for an identical 3D-printed model. From the resulting transform matrices, coordinate transformations were computed. Data reproducibility was analyzed through looking at the difference between mean and grand mean, standard deviation and the Shapiro-Wilks normality constant. Data was graphically displayed to visualize differences in calibrations in different directions. RESULTS: For the needle calibrations, transformations parallel to the tracking clip and perpendicular to the needle demonstrated the greatest deviation. For the ultrasound calibrations, transformations perpendicular to the sound propagation demonstrated the greatest deviation. CONCLUSION: Needle and ultrasound calibrations are reproducible when changing the tracking clip. These devices do not need to be calibrated on a more frequent basis. Caution should be taken to minimize confounding variables such as bending the needle or ultrasound beam width at the time of calibration. KEY WORDS: Calibration, tracking, reproducibility, electromagnetic, spatial, ultrasound-guided needle navigation, transformation, standard deviation.

PURPOSE: In the operating theatre surgeons are accustomed to using spatially navigated tools in conjunction with standard clinical imaging during a procedure. This gives them a good idea where they are in the patients’ anatomy but doesn’t provide information about the type of tissue they are dissecting. In this paper we demonstrate an integrated system consisting of a spatially navigated surgical electrocautery combined with real-time molecular analysis of the dissected tissue using mass spectrometry. METHODS: Using the 3D Slicer software package, we have integrated a commercially available neurosurgical navigation system with an intra-operative mass spectrometer (colloquially referred to as the intelligent knife, or iKnife) that analyzes the charged ions in the smoke created during cauterization. We demonstrate this system using a simulated patient comprised of an MRI scan from a brain cancer patient deformably registered to a plastic skull model. On the skull model we placed porcine and bovine tissues to simulate cancerous and healthy tissue, respectively. We built a PCA/LDA model to distinguish between these tissue types. The tissue classifications were displayed in a spatially localized manner in the pre-operative imaging, in both 2D and 3D views. RESULTS: We have demonstrated the feasibility of performing spatially navigated intra-operative analysis of tissues by mass spectrometry. We show that machine learning can classify our sample tissues, with an average computed confidence of 99.37 % for porcine tissue and 99.36% for bovine tissue. CONCLUSION: In this paper we demonstrate a proof of concept system for navigated intra-operative molecular analysis. This system may enable intra-operative awareness of spatially localized tissue classification during dissection, information that is especially useful in tumor surgeries where margins may not be visible to the unassisted eye.

Responsive neurostimulation (RNS) is a novel surgical intervention for treating medically refractory epilepsy. A neurostimulator implanted under the skull monitors brain activity in one or two seizure foci and provides direct electrical stimulation using implanted electrodes to prevent partial onset seizures. Despite significant successes in reducing seizure frequency over time, outcomes are less than optimal in a number of patients. To maximize treatment efficacy, it is critical to identify the factors that contribute to the variance in outcomes, including accurate knowledge of the final electrode location. However, there is as yet no automated algorithm to localize the RNS electrodes in the brain. Currently, physicians manually demarcate the positions of the leads in postoperative images, a method that is affected by rater bias and is impractical for largescale studies. In this paper, we propose an intensity feature based algorithm that can automatically identify the electrode positions in postoperative CT images. We also validate the performance of our algorithm on a multicenter dataset of 13 implanted patients and test how it compares with expert raters.

PURPOSE: Raman Spectroscopy is amongst several optical imaging techniques that have the ability to characterize tissue non-invasively. To use these technologies for intraoperative tissue classification, fast and efficient analysis of optical data is required with minimal operator intervention. Additionally, there is a need for a reliable database of optical signatures to account for variable conditions. We developed a software system with an inexpensive, flexible mechanical framework to facilitate automated scanning of tissue and validate spectroscopic scans with histologic ground truths. This system will be used, in the future, to train a machine learning algorithm to distinguish between different tissue types using Raman Spectroscopy. METHODS: A sample of chicken breast tissue is mounted to a microscope slide following a biopsy of fresh frozen tissue. Landmarks for registration and evaluation are marked on the specimen using a material that is recognizable in both spectroscopic and histologic analysis. The slides are optically analyzed using our software. The landmark locations are extraction from the spectroscopic scan of the specimen using our software. This information is then compared to the landmark locations extracted from images of the slide using the software, ImageJ. RESULTS: Target registration error of our system in comparison to ImageJ was found to be within 1.1 mm in both x and y directions. CONCLUSION: We demonstrated a system that can employ accurate spectroscopic scans of fixed tissue samples. This system can be used to spectroscopically scan tissue and validate the results with histology images in the future.

Purpose: Colonoscopy is a complex procedure with considerable variation among patients, requiring years of experience to become proficient. Understanding the curvature of colons could enable practitioners to be more effective. The purpose of this research is to develop methods to analyze the curvature of patients’ colons, and compare key segments of colons between supine and prone positions. Methods: The colon lumen in CT scans of ten patients are segmented. The following steps are automated by Python scripts in the 3D Slicer application: a set of center points along the colon are generated, and a curve is fit to these points. By identifying local maximums and local minimums in curvature, curves can be defined between two local curvature minimums. The angle of each curve is calculated over the distance of curves. Results: This automated process was used to identify and quantitatively analyze curves on the colon centerline in different patient positions. On average, there are 4.6 ± 3.8 more curves in supine position than prone. In the descending colon, there are more curves in the supine position, but curves in the prone position are larger. Conclusion: This process can quantify the curvature of colons, and can be adapted to consider other patient groups. Descriptive statistics indicate supine position has more curves in the descending colon, and prone has sharper curves in the descending colon. These preliminary results motivate further work with a larger sample size, which may reveal additional significant differences.

Three-dimensional reconstruction of nerve fascicle is important in the analysis of biological characteristics in the arm. The topology of fascicle has been used by doctors to investigate the nerve direction and the relationship between the individual nerve fascicle. However, there still does not exist an ideal internal fascicle and 3D model in the human peripheral nerve. Accurate segmentation of fascicle from CT images is a crucial step to obtain reliable 3D nerve fascicle model. Traditional method in the fascicle segmentation is not efficient due to time consuming, manual work and poor generalization capacity. In this study, we proposed an efficient deep segmentation network and then reconstruct 3D nerve fascicle model. The proposed network explores the intra-slice contextual features with convolutional long short-term memory for accurate fascicle segmentation, and model long-range semantic information among image slices. Transfer learning technique is integrated with ResNet34, and the discriminative capability of intermediate features are further improved. The proposed network architecture is efficient, flexible and suitable for separating the adhesive fascicle. Our approach is the first deep learning method for nerves segmentation. The proposed approach achieves state-of-the-art performance on our dataset, where the mean Dice of our method is 95.4% and at least 5% more than other methods.

In brain tumor ablation procedures, imaging for path planning and tumor ablation are performed in two different sessions. Using pre-operative MR images, the neurosurgeon determines an optimal ablation path to maximize tumor ablation in a single path ablation while avoiding critical structures in the brain. After pre-operative path planning the patient undergoes brain surgery. Manual planning for brain tumor ablation is time-intensive. In addition, the preoperative images may not precisely match the intra-operative images due to brain shift after opening the skull. Surgeons sometimes therefore adjust the path planned during the surgery, which leads to increased anaesthesia and operation time. In this paper, a new heuristic-based search algorithm is introduced to find an optimal ablation path for brain tumors, that can be used both pre- and intra-operatively. The algorithm is intended to maximize the safe ablation region with a single path ablation. Given the tumor location, healthy tissue locations, and a random start point on the skull from medical images, our proposed algorithm computes all plausible entry points on the skull and then searches for different ablation paths that intersect with the tumor, avoids the critical structures, and finds the optimal path. We implemented Breadth First Search (BFS), Dijkstra, and our proposed heuristic based algorithms. In this paper we report the results of a comparative study for these methods in terms of the search space explored and required computation time to find an optimal ablation path.

Segmentation of the prostate in magnetic resonance (MR) images has many applications in image-guided treatment planning and procedures such as biopsy and focal therapy. However, manual delineation of the prostate boundary is a time-consuming task with high inter-observer variation. In this study, we proposed a semiautomated, three-dimensional (3D) prostate segmentation technique for T2-weighted MR images based on shape and texture analysis. The prostate gland shape is usually globular with a smoothly curved surface that could be accurately modeled and reconstructed if the locations of a limited number of well-distributed surface points are known. For a training image set, we used an inter-subject correspondence between the prostate surface points to model the prostate shape variation based on a statistical point distribution modeling. We also studied the local texture difference between prostate and non-prostate tissues close to the prostate surface. To segment a new image, we used the learned prostate shape and texture characteristics to search for the prostate border close to an initially estimated prostate surface. We used 23 MR images for training, and 14 images for testing the algorithm performance. We compared the results to two sets of experts’ manual reference segmentations. The measured mean ± standard deviation of error values for the whole gland were 1.4 ± 0.4 mm, 8.5 ± 2.0 mm, and 86 ± 3% in terms of mean absolute distance (MAD), Hausdorff distance (HDist), and Dice similarity coefficient (DSC). The average measured differences between the two experts on the same datasets were 1.5 mm (MAD), 9.0 mm (HDist), and 83% (DSC). The proposed algorithm illustrated a fast, accurate, and robust performance for 3D prostate segmentation. The accuracy of the algorithm is within the inter-expert variability observed in manual segmentation and comparable to the best performance results reported in the literature.

Head and neck cancer (HNC) includes cancers in the oral/nasal cavity, pharynx, larynx, etc., and it is the sixth most common cancer worldwide. The principal treatment is surgical removal where a complete tumor resection is crucial to reduce the recurrence and mortality rate. Intraoperative tumor imaging enables surgeons to objectively visualize the malignant lesion to maximize the tumor removal with healthy safe margins. Hyperspectral imaging (HSI) is an emerging imaging modality for cancer detection, which can augment surgical tumor inspection, currently limited to subjective visual inspection. In this paper, we aim to investigate HSI for automated cancer detection during image-guided surgery, because it can provide quantitative information about light interaction with biological tissues and exploit the potential for malignant tissue discrimination. The proposed solution forms a novel framework for automated tongue-cancer detection, explicitly exploiting HSI, which particularly uses the spectral variations in specific bands describing the cancerous tissue properties. The method follows a machine-learning based classification, employing linear support vector machine (SVM), and offers a superior sensitivity and a significant decrease in computation time. The model evaluation is on 7 ex-vivo specimens of squamous cell carcinoma of the tongue, with known histology. The HSI combined with the proposed classification reaches a sensitivity of 94%, specificity of 68% and area under the curve (AUC) of 92%. This feasibility study paves the way for introducing HSI as a non-invasive imaging aid for cancer detection and increase of the effectiveness of surgical oncology.

We have presented a variety of methods for object recognition based on the Automatic Anatomy Recognition (AAR) framework at previous SPIE conferences, including AAR recognition via optimal threshold on intensity, AAR recognition via composite information from intensity and texture, and AAR recognition with the optimal hierarchical structure design, and via neural networks to learn object relationships. The purpose of this paper is to introduce new features for the AAR-based recognition procedure and improve the performance of object localization for autocontouring in head and neck (H&N) radiation therapy planning, specifically for some of the most challenging objects. The proposed super-mask technique first registers images used for model building among themselves optimally by using a minimal spanning tree in the complete graph formed with images as nodes to determine the order of registering images. Subsequently, we build a super-mask by combining the similarly registered binary images corresponding to each object by taking (S1) union of all binary images, (S2) intersection among all binary images, or (S3) the votingbased fuzzy mask created by adding the binary images. The super-mask is then used to confine search for optimum localization of the object in the given image. A large-scale H&N computed tomography (CT) data set with 216 subjects and over 2000 3D object samples were utilized in this study. The super-mask-based object localization approach within the AAR framework improved the recognition accuracy by 25-45% compared with the previous AAR strategy, especially for the most challenging H&N objects. On low quality images, the new method achieves recognition accuracy within 2 voxels on 50-60% of the cases.

The accurate tracking of patients during a surgery is an essential requirement of computer assisted surgery. Many tracking systems are based on permanently installed infrared camera systems to detect reflective spheres. These tracking concepts need a certain amount of installation effort and are associated with high investments. An alternative are planar fiducial markers, which can be tracked only through RGB data and can therefore be used with different camera systems. The objective of this work is to introduce a set of similarity metrics to compare fiducial markers for pose estimation. We propose eight different similarity metrics to unify the process of evaluation and comparison of marker systems. These are the size and outer margin of the marker, the distance to the camera, the pose estimation accuracy, the runtime of the algorithm, the robustness against external influences, the affection to the sensor system and the number of used markers. We also describe the methodology for evaluating these metrics. We then apply these metrics to compare the ArUco and AprilTag open source marker systems. Our tests conclude that the optical tracking of open source fiducial markers is possible at submillimeter range at distances up to one meter. In addition, the tracking result can be greatly improved by using multiple markers. Accuracy is increased and fluctuations are minimized. The similarity metrics presented by us are suitable for evaluating and comparing marker systems in detail. This can serve as a basis for selecting a suitable system for a specific medical procedure.

The proton resonance frequency shift (PRFS) method is the most frequently used method to perform volumetric thermometry during MRI-guided thermal therapies. However, one of the main drawbacks of the PRFS method is its sensitivity to inter-frame motion and magnetic field drifts, which can result in incorrect estimation of temperature profiles. To address these problems, several techniques have been proposed, such as the reference less, multi-baseline, and hybrid methods. While the hybrid method has demonstrated the best performance, it assumes focal heating, which may be valid when using energy modalities such as high intensity focused ultrasound, but does not hold for heating using diffuse sources such as needle- and catheter-based microwave applicators. Here, we present an enhanced hybrid method suitable for MRI thermometry in the presence of motion during microwave thermal therapy. The presented model-based method uses the sparsity of wavelet coefficients of the phase shift based on the fact that heat-induced phase shifts exhibit a correlation structure due to smoothness. The presented enhanced hybrid method is compared to the previously presented hybrid and conventional PRFS methods for temperature estimation during microwave heating of a tissue-mimicking phantom with a 2.45 GHz directional microwave antenna integrated with 14.1 T high-field MRI. Experimental results demonstrate that the proposed method estimates microwave heating-induced temperature changes within 0.3-0.5 ^oC (mean error of 5.9 % over 5 min of heating) of fiber-optic temperature sensors, compared to 1.5 ^oC (mean error of 36.3% over 5 min of heating) with the hybrid technique.

Despite a strong evidence of the clinical and economic benefits of minimally invasive surgery (MIS) for many common surgical procedures, there is a gross underutilization of MIS in many US hospitals, potentially due to its steep learning curve. Intraoperative videos captured using a camera inserted into the body during MIS procedures are emerging as an invaluable resource for MIS education, skill assessment and quality assurance. However, these videos often have a duration of several hours and there is a pressing need for automated tools to help surgeons quickly find key semantic segments of interest within MIS videos. In this paper, we present a novel integrated approach for facilitating content-based retrieval of video segments that are semantically similar to a query video within a large collection of MIS videos. We use state-of-theart deep 3D convolutional neural network (CNN) models pre-trained on large public video classification datasets to extract spatiotemporal features from MIS video segments and employ an iterative query refinement (IQR) strategy where in a support vector machine (SVM) classifier trained online based on relevance feedback from the user is used to refine the search results iteratively. We show that our method outperforms the state-of-the-art on the SurgicalActions160 dataset containing 160 video clips of typical surgical actions in gynecologic MIS procedures.

A major clinical challenge in rectal cancer currently is non-invasive identification of tumor regression to standard- of-care neoadjuvant chemoradiation (CRT). Multi-parametric MRI is routinely acquired after CRT, but expert radiologists find it highly challenging to assess the degree of tumor regression on both T2-weighted (T2w) and Gadolinium contrast-enhanced (CE) MRI; resulting in poor agreement with gold-standard pathologic evaluation. In this study, we present initial results for integrating quantitative image appearance (radiomic) features from post-CRT T2w and CE MRI towards in vivo assessment of pathologic rectal tumor response to chemoradiation. 29 rectal cancer patients with post-CRT multi-parametric 3 T MRI (with T2w, initial and delayed CE phases) were included in this study. Through spatial co-registration, the treated region of the rectal wall was identified and annotated on T2w and all CE phases (as well as correcting for motion artifacts in CE MRI). 165 radiomic features (including Haralick, Gabor, Laws, Sobel/Kirsch) were separately extracted from each of T2w and 2 CE phases; within the entire rectal wall. The top 2 response-associated radiomic features for each of (a) T2w, (b) 2 CE phases, (c) combined T2w+CE phases were identified via feature selection and evaluated in a leave- one-patient-out cross validation setting. Integrating T2w and CE radiomic features was found to be markedly more accurate (AUC=0.93) for assessing post-CRT pathologic tumor stage, compared to T2w radiomic features (AUC=0.80) and CE radiomic features (AUC=0.63) individually. Top-ranked features captured heterogeneity of gradient responses on T2w MRI and macro-scale Gabor wavelet responses of contrast enhancement on CE MRI. Combining radiomic features from post-CRT T2w and CE MRI may hence enable more comprehensive evaluation of response to neoadjuvant therapy in rectal cancers, which can be used to better guide follow-up interventions.

Cholesteatomas are benign lesions that form in the middle ear (ME). They can cause debilitating side effects including hearing loss, recurrent ear infection and drainage, and balance disruption. The current approach for positively identifying cholesteatomas requires intraoperative visualization either by lifting the ear drum or transmitting an endoscope through the ear canal and tympanic membrane – procedures which are typically done in and operating room with the patient under general anesthesia. We are developing a novel endoscope that can be inserted trans-nasally and could potentially be used in an outpatient setting allowing clinicians to easily detect and visualize cholesteatomas and other middle ear conditions. A crucial part of designing this device is determining the degrees of freedom necessary to visualize the regions of interest in the middle ear space. To permit virtual evaluation of scope design, in this work we propose to create a library of models of the most difficult to visualize region of the middle ear, the retrotympanum (RT), which is located deep and posterior to the tympanic membrane. We have designed a semi-automated atlas-based approach for segmentation of the RT. Our approach required 2-3 minutes of manual interaction for each of 20 cases tested. Each result was verified to be accurate by an experienced otologist. These results show the method is efficient and accurate enough to be applied to a large scale dataset. We also created a statistical shape model from the resulting segmentations that can be used to synthesize new plausible RT shapes for comprehensive virtual evaluation of endoscope designs and show that it can represent new RT shapes with average errors of 0.5 mm.

Dynamic Optical Contrast Imaging (DOCi) is an imaging technique that generates image contrast through ratiometric measurements of the autouflorescence decay rates of aggregate uorophores in tissue. This method enables better tissue characterization by utilizing wide-field signal integration, eliminating constraints of uniform illumination, and reducing time-intensive computations that are bottlenecks in the clinical translation of traditional fluorescence lifetime imaging. Previous works have demonstrated remarkable tissue contrast between tissue types in clinical human pilot studies [Otolaryngology-Head and Neck Surgery 157, 480 (2017)]. However, there are still challenges in the development of several subsystems, which results in existing works to use relative models. A comprehensive mathematical framework is presented to describe the contrast mechanism of the DOCi system to allow intraoperative quantitative imaging, which merits consideration for evaluation in measuring tissue characteristics in several important clinical settings.

Multimodal registration improves surgical planning and the performance of interventional procedures such as transarterial chemoembolizations (TACE), since it allows to combine complementary information provided by pre- and intrainterventional data about tumor localization and access. However, no registration methods specifically developed for the multimodal registration of abdominal scans exist and as a result only general-purpose methods are available for this application. In this paper, we evaluate and optimize the performance of three standard registration methods which rely on different similarity metrics, namely Advanced Mattes Mutual Information (AMMI), Advanced Normalized Correlation (ANC) and Normalized Mutual Information (NMI), for the registration of preinterventional T1- and T2-weighted MRI to preinterventional CT as well as intrainterventional Cone Beam CT (CBCT) to preinterventional CT of the liver. Moreover, different variants of the registration algorithms, based on the introduction of masks and different resolution levels in multistage registrations, are investigated. To evaluate the performance of each registration method, the capture range was estimated based on the calculation of the mean target registration error.

Although resection and transplantation are primary curative methods of treatment for hepatocellular carcinoma, many patients are not candidates. In these cases, other treatment methods such as selective internal radiation therapy, chemotherapy, or external beam radiation are used. While these treatments are effective, patient-specific customization of treatment could be beneficial. Recent advances in personalized medicine are making this possible, but often there are multiple phenotypes within a proliferating tumor. While not standard, one could envision a serial longitudinal biopsy approach with more phenotypically-targeted therapeutics if one could detect responding and non-responding regions of tumor over time. This work proposes a method to determine active regions of the tumor that differentially respond to treatment to better guide biopsy for longitudinal personalization of treatment. While PET may serve this purpose, it is not easily used for real-time image guidance, is not effective for many types of tumors, and can be confounded by inflammatory responses. In this work, ten total patients with imaging sequences from before and after treatment were retrospectively obtained. Five of these were selected for analysis based on the total liver volume change. A two-phase alignment process comprised of an intensity-based rigid registration followed by a nonrigid refining process driven by bulk deformation of the organ surface was performed. To assess the accuracy of the registration, two metrics were used for preliminary results. The mean closest point surface distance was used to quantify how well the surfaces of the registered livers match and was found to be 2.65±3.54mm. Anatomical features visible in pre- and post-treatment images were also identified. After registration, the mean Euclidean distance between features was found to be 5.22±4.06mm. To assess potential areas of tumor change, the registered tumor pre- and post-treatment were overlaid.

PURPOSE: Partial nephrectomy is the preferred method for managing small renal masses. This procedure has significant advantages over radical nephrectomy. However, partial nephrectomy is under-used due to its difficulty. We propose a navigation system for laparoscopic partial nephrectomy. In this study, we evaluate the usability and accuracy of the navigation system. METHODS: An electromagnetically tracked navigation system for partial nephrectomy was developed. This system tracks the positions of the laparoscopic scissors, ultrasound probe, tumor, and calyces and vasculature. Phantom kidneys were created using mixtures of plastisol and cellulose. To test the system, the navigation display quality was measured through measurement of lag and frames per second displayed. The accuracy of the system was determined through fiducial registration. Finally, a study consisting of ten participants was conducted to assess the usability of the navigation system using the System Usability Survey. RESULTS: The mean System Usability Score of the navigation system was 82.5. The navigation display had an average lag of 243 milliseconds and showed 5 frames per second. The accuracy was measured with fiducial registration and found to have an RMS error of 2.84 mm. CONCLUSION: The results of this study suggest that the partial nephrectomy navigation system developed is both usable and accurate. Future work will include the conversion of the laparoscopic scissor tool tracking to optical. Further studies will be conducted to determine the effectiveness of this technology in tumor resection and avoidance of calyx and vasculature damage. We will additionally explore this system as a training tool.

Neurosurgery typically requires craniectomy and meticulous dissection to achieve sufficient exposure for subsequent surgical intervention. This highly invasive process requires hours of operating time, long recovery periods and leaves patients with visible surgical scars. Non-invasive high-intensity focused ultrasound (HIFU) has shown some promise yet remains challenged by the attenuation of ultrasonic waves while passing through the skull. Consequently, the clinical impact of this technology remains limited, particularly in the treatment of neuro-oncology. In order to compensate for acoustic attenuation, excessive use of power for HIFU devices has been investigated, although it is undesirable from a regulatory and patient safety standpoint. Here, we report the design and development of a novel HIFU device prototype for neurologic lesion ablation. This device concept is envisioned to access the ventricular space via a minimally invasive ventriculostomy, allowing ultrasound to reach targets deep in the brain, while eliminating the need for high power to penetrate the skull.

Ideal treatment of trauma, especially that which is sustained during military combat, requires rapid management to optimize patient outcomes. Medical transport teams `scoop-and-run' to trauma centers to deliver the patient within the `golden hour', which has been shown to reduce the likelihood of death. During transport, emergency medical technicians (EMTs) perform numerous procedures from tracheal intubation to CPR, sometimes documenting the procedure on a piece of tape on their leg, or not at all. Understandably, the EMT's focus on the patient precludes real-time documentation; however, this focus limits the completeness and accuracy of information that can be provided to waiting trauma teams. Our aim is to supplement communication that occurs en-route between point of injury and receiving facilities, by passively tracking and identifying the actions of EMTs as they care for patients during transport. The present work describes an initial effort to generate a coordinate system relative to patient's body and track an EMT's hands over the patient as procedures are performed. This `patient space' coordinate system allows the system to identify which areas of the body were the focus of treatment (e.g., time spent over the chest may indicate CPR while time spent over the face may indicate intubation). Using this patient space and hand motion over time in the space, the system can produce heatmaps depicting the parts of the patient's body that are treated most. From these heatmaps and other inputs, the system attempts to construct a sequence of clinical procedures performed over time during transport.

Prostate cancer has the second highest noncutaneous cancer incidence in men worldwide. A common treatment technique for intermediate and high-risk localized prostate cancer is ultrasound (US)-guided high-dose-rate brachytherapy. This minimally invasive procedure uses a radioactive source passed through multiple needles to deliver radiation to the prostate and relies on accurate identification of needle tips to ensure patient safety and delivery of the prescribed doses. Image artifacts from nearby needles and the surrounding tissue often limit the accuracy of needle tip identification when using standard US imaging. To overcome these limitations and improve the accuracy of intraoperative needle tip identification, we propose the use of power Doppler (pD) US imaging while a mechanical perturbation is applied to the needle of interest. A mock procedure employing the standard clinical workflow was completed in a tissue-mimicking agar phantom. Inserted needles were imaged using standard US, followed by pD imaging of the same needles while a custom-made mechanical oscillator was used to perturb the needle. Physical measurements of the needle end lengths were used to estimate insertion depth errors (IDEs). 13 unobstructed needles and 10 shadowed needles were imaged using standard and pD US, resulting in mean IDEs ± standard deviation of 2.2 ± 0.9 mm and 1.3 ± 0.9 mm, respectively, for unobstructed needles, and 2.1 ± 1.6 mm and 1.6 ± 1.2 mm for shadowed needles. Mean IDEs were reduced in all cases when pD imaging was used, suggesting our method may be useful in improving HDR-BT treatment accuracy and patient safety.

Patient-specific dose response against chemotherapeutics can be assessed through local in situ release of drugs at sub-therapeutic concentrations. Such controlled release can be performed in patient-derived xenografts (PDXs), which offer pre-clinical methods for mimicking the tumor microenvironment. However, the prolonged co-existence of intermingled human and mouse tissues poses a number of challenges for histological image analysis. Manual annotation of regions of human tissue is labor-intensive and lacks reproducibility and scalability, complicating the investigation of multiplexed local drug effects near drug-dispensing microdevices. To this end, we apply a random forest algorithm for segmenting histological images to obtain binary masks for refined image analysis. Region-of-interest masks obtained using this supervised learning approach allow for a spatially refined assessment of the dose response in heterogeneous tissue8. We achieved a Dice similarity coefficient score (DSC) of 0.56 with the random forest classifier.

Detailed localization of the rectal wall after chemoradiation on standard-of-care post-chemoradiation (CRT) MRIs could enable more targeted follow-up interventions, but it is a challenging and laborious task for radiologists. This may be because the primary tumor site (i.e. primary" wall) and the remaining adjacent" wall areas depict visually overlapping intensity characteristics as a result of chemoradiation-induced noise and treatment effects. In this study, we present initial results for developing and optimizing fully convolutional networks (FCNs) to automatically segment the rectal wall on post-CRT MRIs. Our cohort comprised 50 post-CRT, T2-weighted MRIs from rectal cancer patients with expert annotations of the entire length of the rectal wall (with separate indications for extent of primary wall as well as adjacent wall). The FCN framework was designed to provide a pixel-wise segmentation of the rectal wall while utilizing the original T2w intensity images, and was tested on 20% of the cohort that was held-out from training. Our results showed that (a) the best-performing FCN for segmenting primary wall areas utilized a training set comprising primary wall sections alone (median DSC = 0.71), while (b) optimal segmentations of adjacent wall areas were achieved by an FCN trained on both primary and adjacent wall sections (median DSC = 0.68). Notably, the primary wall FCN performed poorly when applied to adjacent wall and vice versa; perhaps indicating that fundamental physiological differences exist between these wall areas that must be accounted for within automated CN segmentation approaches. FCNs may hence have to be optimized on a region-specific basis to obtain detailed, accurate delineations of the entire rectal wall on post-CRT T2w MRI, towards more targeted excision surgery and adjuvant therapy.

Despite a number of recent advances in robot-assisted surgery, achieving minimal access still requires that surgeons operate with reduced faculties for perception and manipulation as compared to open surgery. Image guidance shows promise for enhancing perception during local navigation (e.g. near occluded endophytic tumors), and we hypothesize that these methods can be extended to address the global navigation problem of efficiently locating and exposing a target organ and its associated anatomical structures. In this work we describe the high-level architecture of an augmented reality system for guiding access to abdominal organs in laparoscopic and robot-assisted procedures, and demonstrate the applicability of an array of assimilation algorithms through proof-of-concept simulation. Under the proposed framework, a coarse model of procedure-specific internal anatomy is initialized based on segmented pre-operative imaging. The model is rigidly registered to the patient at the time of trocar placement, then non-rigidly updated in an incremental manner during the access phase of surgery based on limited views of relevant anatomical structures as they are exposed. Observations are assumed to derive primarily from reconstruction of stereoscopic imaging; however, the assimilation framework provides a means of incorporating measurements made with other sensing modalities. Simulation results show that standard state estimation algorithms are suitable for accommodating large-scale displacement and deformation of the observed feature configuration relative to the initial model. Future work will include development of a suitable 3D model of anatomical structures involved in partial nephrectomy as well as provision for leveraging intraoperative dynamics in the assimilation framework.

Modern handheld structured light scanners show potential in the field of medical imaging and image-guided surgery. For the effective use of such scanners as a rapid registration tool, anatomical regions of interest must be identified. The purpose of this study is to investigate the use of machine learning to classify various anatomical tissues using the textural information collected from structured light scanners. We performed an ex vitro study using three fresh frozen knee specimens. Each specimen underwent multiple stages of dissection to reveal different anatomical tissues. At each stage of dissection, the specimens were scanned with a structured light scanner (Artec Spider, Artec Group, Palo Alto, USA). Using the texture information of the scanned model, a domain expert manually segmented four tissues of interest: muscle, tendon, cartilage, and bone. The RGB and HSL values of the data points in the manually segmented models were extracted for use in training and evaluating a random forest classifier. Our trained random forest classifier obtained a four-class classification accuracy of 77% and a three-class classification accuracy of 82%. The results of this study demonstrate the feasibility of a random forest to aid in semi-automatic or automatic segmentation of anatomical tissues using only textural information. Further experiments with in vivo tissues will need to be done to validate the application of such classifiers in an intraoperative environment.

Respiratory motion models aim at improving the quality of free breathing image acquisition protocols and yield increased targeting accuracy during image guided interventions. Respiratory motion can deviate pre-defined targets and trajectories determined preoperatively during treatment procedures. In this context, motion models offer a mean to estimate spatio-temporal displacements of the organ and correct the target position in real time during an intervention. To construct a motion model, data of the entire organ of interest must be acquired. However, existing techniques for 3D dynamic imaging have poor spatial and temporal resolution. Therefore, to capture the organ’s temporal behavior, series of dynamic 2D slices covering the entire organ are typically acquired. Then, these slices are reordered retrospectively according to their motion phase within the respiratory cycle and stacked to form 3D dynamic volumes known as 4D images (3D + t). On the other hand, while numerous metrics were proposed to assess the spatial quality of the reordering, little attention has been paid to metrics that assess the coherent temporal behavior of the reconstructed dynamic volumes. This work proposes a method combining image-based matching approach with manifold alignment and compares it with two state of the art slice reordering methods. Methods were evaluated on a dataset of 7 volunteers using new metrics to assess the spatial quality and the temporal behavior, with the proposed method outperforming in terms of both spatial and temporal quality.

A revised version of this paper was published on 14 June 2019. Details of the revision are provided in the text that accompanies this Erratum. The original paper has been updated.