Surgical robotic tools face dynamic environments, changing kinematics, and complex tasks. Machine learning can help us overcome these challenges.
Author: Francesco Cursi
[1] Francesco Cursi, George P. Mylonas, and Petar Kormushev, “Adaptive Kinematic Modelling for Multiobjective Control of a Redundant Surgical Robotic Tool”, In Robotics, vol. 9, no. 3, pp. 1-16, 2020. | Read the full paper | Download the PDF | Watch the video
Introduction
Thanks to the advances in medical technologies, the number of surgical procedures transitioning from open to minimally invasive has been increasing in the last decades. Minimally invasive surgery (MIS) differs from open surgery in limiting the size of the incisions needed for operations: patients are no longer subjected to large incisions and organs no longer exposed to open air, limiting the risk of infection and reducing the complications deriving from open surgery. Many studies have shown that this less invasive method turned out to have many benefits for the patient thanks to reduction of blood losses, wound infection, pain, and postoperative recovery.
In the last twenty years, further progress has been made to augment surgeons’ capabilities by introducing robots in surgical operations, leading to the rise of robot-assisted surgery and minimally invasive robotic surgery (MIRS).
Surgical progression throughout history.
However, there are still some challenges that prevent a wide spread use of robots for surgery. In order to be able to perform surgeries autonomously, robots need to be safe, precise, and capable of adapting to different situations. Robots for minimally invasive surgery have usually complex structures, being very articulated or even continuum robots. Therefore, modelling their kinematics may be very challenging. Inaccurate robot models make the control more tedious and imprecise, with increasing possibility of damages to the patient’s tissues and organs.
Robot Kinematics
The kinematic model of a robot allows us to map motor values to the robot’s tip pose, generally expressed in a Cartesian space as 3D position and 3D orientation. Given a desired tip position and orientation, the kinematic model is used to compute the corresponding motor values to reach the desired goal.
Inverse kinematics driving a robot to a desired position in the 3D world.
Different methods exist to model the kinematic structure of robots, but the modelling of surgical robots is much more challenging. Most of the surgical robots are tendon-driven, meaning that motors are not directly connected to the joints. Motors are usually placed at the proximal end, which needs to be placed outside the human body, and tendons are routed around the motors. Tendons are inserted inside the robotic structure and connected to the corresponding link at the distal end.
The surgical robot used in this work (Shang et. al.)
This kind of transmission causes many nonlinear effects that make robot modelling complicated. Analytical models are hard to derive, and, consequently, data-driven models based on machine learning techniques are preferred.
Machine Learning for Robot Modelling
Different approaches have been developed and used in the field of robotics to model complex systems. These techniques belong to regression methods, which allow to find relationships between some continuous input and output data. Among the most successful and widespread are Feedforward Artificial Neural Networks (ANN). ANN are considered to be universal approximators that can model any function.
A single layer feedforward artificial neural network (innoarchitech)
ANNs have a cascade structure, generally consisting of one input layer, several hidden layers, and one output layer, each layer having many nodes. The nodes receive inputs from the previous layer, weight and combine them, and output a value which is passed on to the next layer. The final output of the network is a combination of nonlinear functions which approximates the desired mapping.
The Need for Adaptation
Even though machine learning for robot modelling is growing very fast and helps solving challenging problems, it still has some drawbacks:
- Machine learning approaches are data-driven, and, as such, heavily rely on the provided data. This means that the models obtained from these techniques are not usually generalizable, thus requiring new models to be built if there is a change to the environment or the robot.
- The environment may not be structured and it may largely differ from that on which a model has been built.
- The behaviour of robotic systems can change during the operation. Especially in surgery, fluids like blood can penetrate the robotic instrument or tendons may break, thus affecting how the robot responds to motor commands.
Model adaptation allows the robot to overcome this issues by adapting the model of the robot whenever new data is acquired. If the output provided by the original model differs too much from the new provided data, then the data points are used to update the model of the robot.
Our Approach
In our work we focused on modelling and control the Micro-IGES surgical robotic tool, developed at Hamlyn Centre. The Micro-IGES is a surgical robotic tool, composed of a rigid shaft and a flexible section. The shaft is responsible for the roll Degrees of Freedom (DOFs), whereas the articulated end consists of 2 elbows for pitch and yaw, with each elbow made of a pair of coupled joints, a 1 DOF revolute joint for the wrist pitch, and the jaws. The jaws provide two more DOFs: one for the wrist yaw and one for the gripper’s opening and closing. Each joint of the articulated part is driven by an antagonistic pair of tendons, with each pair being connected to the corresponding driving capstan at the proximal drive unit. The coupling of the two pairs of joints of the elbows occurs at the driving unit: the two capstans that drive the two serial joints for each DOF of the elbow (pitch and yaw) are coupled by a series of gears with 1:2 ratio.
An initial model of the system is first built offline, mapping the motor values to the 3D Cartesian tip position. During operation, however, an adaptive control strategy is enabled in order to improve the accuracy of the model and cope with possible unpredicted changes in the system’s behaviour. The proposed re-adaptation works as in the flow chart blow.
Flowchart showing the adaptation process.
With the current model, (1) the motor values to reach the desired position are computed. These motor values are then commanded to the robot and (2) the actual tip position Pa is measured and the expected tip position computed. If the error between the expected position and the actual robot position (for any Cartesian component) is higher than a certain threshold, (3) the network weights are updated. Once the model is updated, the control strategy is used again to compute the motor values to reach the desired position. The adaptation is repeated until convergence. Afterwards, (4) a new desired trajectory point is assigned and the process repeated until the whole task has been completed.
Results
We tested our approach on two different tasks, simulating a tumor resection:
Autonomous tracking
in this scenario the robot is required to follow a circular path autonomously. To model unpredicted changes in the model the Elbow 1 joint is considered locked at an angle of 10◦. Detecting which joint is not functioning anymore may be challenging or even impossible in a real scenario and, therefore, the system model needs to adapt accordingly. In this case, no information about the broken joint is given, and the controller only receives data about the tip position and the commanded motor values. Results show that the proposed method
is able to effectively relearn the model and ensure correct execution of the task. Apart from some initial errors, the adaptation is able to converge to an accurate model. When a simple feedback controller is used instead, the system becomes unstable and fails the tracking task, due to the unmodelled errors.
Autonomous tracking with a broken joint. Left: without adaptation. Right: with adaptation.
Teleoperation
To further prove the capabilities of the proposed method, a teleoperation test is also performed. This test serves for two purposes. On one hand, surgical robots are usually teleoperated and, therefore, the adaptation must be fast and accurate enough to allow the robot to follow the master commands. On the other hand, through teleoperation it was possible to generate paths completely different from the training one, which might happen due to tissue deformations. Also in this case the primary task is following the path in the x, y Cartesian space. The same thresholds and memory size of the autonomous tracking are used. Thanks to the fast model adaptation, there is very little lag between the commanded motion and the executed one. Moreover, the model adaptation proved effective and able to allow the system to compensate for the errors and follow the desired path with high accuracy.
Teleoperation of the surgical robot. Left: no model errors. Right: broken joint.