We are investigating new approaches for augmented reality in robotic surgery. Top: We envision combining preoperative and intraoperative data to register a liver tumor (shown in green) on the patient's anatomy and track it in real time. Bottom: Our AR functions assist the surgeon by overlaying 2D preoperative data and intraoperative cues (a live view of the operating room, the 3D distance between points selected with the instruments, a warning symbol for out-of-view instruments).
Following recent advances in optics, digital image acquisition, and computer vision, augmented reality (AR) applications are being vigorously researched and effectively deployed in several areas of the healthcare industry. In robot-assisted minimally invasive surgery (RMIS), AR has the potential to reduce the surgeon's cognitive load and thereby increase focus and efficiency by delivering computational, diagnostic, and visualization tools directly in the surgeon console.
In current clinical practice, AR is successfully applied to neurological surgery for navigation and guidance; after preoperative data are matched with the intraoperative scene via a registration process, the surgeon can superimpose the preoperative data onto the patient's anatomy. In RMIS, soft tissues and deformable organs in the abdomen make accurate superimposition and tracking extremely challenging. As such, we aim to achieve accurate registration and tracking by performing a robust anatomical 3D reconstruction of the intraoperative scene. We believe the resulting image-based guidance will have the potential to help surgeons during critical steps of minimally invasive procedures.
We have also explored novel uses and interaction methods for AR in RMIS [ ]. In particular, we developed four voice-controlled functions to view 2D preoperative images, view a live video of the operating room, measure 3D distances, and warn users about instruments that have moved outside the visual field. A user study with eight experienced RMIS surgeons performing dry-lab lymphadenectomy showed that the functions improved the procedure; surgeons particularly appreciated the possibility of accessing patient images on demand, measuring distances intraoperatively, and interacting with the functions using voice commands. Our low-cost platform can be easily integrated into any surgical robot equipped with a stereo camera and a stereo viewer [ ].