Publication: Deep learning for force estimation and haptic feedback in robotic surgery
Abstract
Robotik minimal invazif cerrahi, uzaktan kontrol edilen robotik cerrahi enstrümanların kullanıldığı, hastalara daha az komplikasyon ve hızlı iyileşme imkanı, cerrahlara ise daha hassas bir kontrol ve arttırılmış ergonomi sunan bir cerrahi yöntemidir. Ancak mevcut cerrahi robotlar, cerrahın ameliyatı sadece kamera görüntüleriyle yapmasına dayanır ve haptik (dokunsal) geri bildirim sağlamaz. Haptik geri bildirim cerrahın sağlıklı dokulara zarar vermemesi için minimal invazif cerrahide kritik öneme sahiptir. Literatürde haptik geri bildirim sağlamaya yönelik birçok çalışma bulunmasına rağmen, robotik cerrahi sistemlerinde kuvvet sensörü kullanmadan gerçekçi dokunma hissi sağlayabilen bir yöntem sunulmamıştır. Bu çalışmada, öncelikle bu sorunun çözümüne yönelik olarak derin öğrenme temelli bir dinamik model ve kuvvet kestirimi algoritması geliştirilmiştir. Bu algoritma, farklı robotlara ve değişen ortam koşullarına transfer öğrenmesi aracılığıyla daha hızla adapte olmakta ve operasyon sırasında robottan kaynaklanan dinamik kuvvetler ile dışarıdan etkiyen kuvvetleri ayırt ederek cerraha sadece dış kuvvetleri geri bildirebilmektedir. Bu çalışmada haptik geri bildirim, yine bir robotik cerrahi sisteminde ilk kez kullanılan yüksek şeffaflığa sahip dört kanallı bir teleoperasyon sistemi aracılığıyla sağlanmıştır. Buna ek olarak sistemin haptik geri bildirim kalitesini arttırmak amacıyla robotların bir yüzeyle temasını makine öğrenmesi aracılığıyla tespit edebilen ve buna göre geri bildirimi açıp kapayan yeni bir (x-lateral) teleoperasyon mimarisi de sunulmaktadır. Tez kapsamında geliştirilen metodlar, Johns Hopkins Üniversitesi'ndeki da Vinci Research Kit (dVRK) ve Marmara Üniversitesi'ndeki Phantom Omni robotları üzerinde test edilmiştir. dVRK üzerinde kullanıcı deneyleri de gerçekleştirilmiş ve kullanıcıların dokunarak tümör bulma performansının kaydadeğer seviyede arttığı gösterilmiştir.
Robotic minimally invasive surgery is performed by surgeons who remotely control robotic surgical instruments. This method provides all the benefits of conventional laparoscopy such as rapid recovery and fewer complications for patients, as well as increased accuracy and ergonomics for the surgeons. However, with the current technology, surgeons rely only on visual feedback from endoscopic cameras and the robots do not provide haptic feedback to the surgeon. The sense of touch (haptic feedback) is critical in minimally invasive surgery especially to prevent tissue damage by applying excessive forces.Although there are many studies in the literature aimed at providing haptic feedback, no method has been presented that can provide a realistic sense of touch in robotic surgical systems without the use of force sensors. In this study, a dynamic model identification and a force estimation algorithm based on deep learning have been developed primarily to solve this problem. This algorithm can adapt more quickly to different robots and changing environmental conditions through transfer learning and can distinguish between robot dynamic forces and external forces, and can feedback only the external force to the surgeon. In this study, haptic feedback was provided through a transparency optimized four-channel teleoperation system, used for the first time in a robotic surgical system. In addition, to enhance the quality of haptic feedback, a new (x-lateral) teleoperation architecture that can detect the robot's contact with an environment through machine learning and accordingly switch the feedback on and off is presented.The methods presented within this thesis were tested on the da Vinci Research Kit (dVRK) at Johns Hopkins University and the Phantom Omni robots at Marmara University. User studies conducted on the dVRK show that sensorless haptic feedback can significantly improve tumor detection performance through palpation.
Robotic minimally invasive surgery is performed by surgeons who remotely control robotic surgical instruments. This method provides all the benefits of conventional laparoscopy such as rapid recovery and fewer complications for patients, as well as increased accuracy and ergonomics for the surgeons. However, with the current technology, surgeons rely only on visual feedback from endoscopic cameras and the robots do not provide haptic feedback to the surgeon. The sense of touch (haptic feedback) is critical in minimally invasive surgery especially to prevent tissue damage by applying excessive forces.Although there are many studies in the literature aimed at providing haptic feedback, no method has been presented that can provide a realistic sense of touch in robotic surgical systems without the use of force sensors. In this study, a dynamic model identification and a force estimation algorithm based on deep learning have been developed primarily to solve this problem. This algorithm can adapt more quickly to different robots and changing environmental conditions through transfer learning and can distinguish between robot dynamic forces and external forces, and can feedback only the external force to the surgeon. In this study, haptic feedback was provided through a transparency optimized four-channel teleoperation system, used for the first time in a robotic surgical system. In addition, to enhance the quality of haptic feedback, a new (x-lateral) teleoperation architecture that can detect the robot's contact with an environment through machine learning and accordingly switch the feedback on and off is presented.The methods presented within this thesis were tested on the da Vinci Research Kit (dVRK) at Johns Hopkins University and the Phantom Omni robots at Marmara University. User studies conducted on the dVRK show that sensorless haptic feedback can significantly improve tumor detection performance through palpation.
Description
Keywords
Dinamik modelleme, kuvvet kestirimi, derin öğrenme, haptik, telerobotik ve teleoperasyon Dynamic identification, force estimation, deep learning, haptics, telerobotics and teleoperation