Sensorization of Robotic Hand Using Optical Three-Axis Tactile Sensor: Evaluation with Grasping and Twisting Motions

Problem statement: Sensitization of robot hand is still remaining as crucial issue since most of robot hand systems nowadays are only capable to grasp a predefined specific object. It is still difficult for robot hand system to realize human-like tactile sensation. Some common problems in robot hand system are low accuracy sensing device, sensors are not robust enough for long time work and heavy duties, inconsistence tactile sensing detection and difficulties in control of sensing fusion with robot trajectory. These problems are apparently drawback the progress to commercializing robot hands as real consumer products. Approach: This study presented the application of optical three-axis tactile sensor to robot hand to improve sensitization quality in robotic hand system. The proposed tactile sensor system was designed with compliance modules to communicate with robot hand control system. The sensing principle used in this tactile sensor comparatively provides better sensing accuracy to detect contact phenomena from acquisition of three axial directions of forces. Methodology of force and slippage detection in the tactile sensor system was presented. Accordingly, the optimization of robot hand control algorithm to comply with the tactile sensor system was presented and verified in experiment of grasping and twisting. Results: The tactile sensor presented in this study is capable of detecting normal and shear force simultaneously. The proposed methodology was verified in experiment with paper cup and water, in which the result shows the robot control system managed to respond to the proposed object stiffness distinction parameters and effectively respond to sudden change of object weight during grasping. An experiment of grasping and twisting motions was conducted using a bottle cap. In order to perform simultaneous grasping and twisting tasks, optimization of the control algorithm was conducted with additional parameters to satisfy the desired tasks. Conclusion: Experimental result shows that the robot hand managed to perform grasping and twisting of bottle cap smoothly. The overall results revealed good performance of the proposed optical three-axis tactile sensor system and robot hand control algorithm for future application in a real artificial robot hand. In addition, slippage sensation measured in a robot control system could contribute a better maneuvering of the robot arm-finger system.


INTRODUCTION
Grasping and twisting are two common human abilities that support human's daily life. Grasping in robotics has been studied from both biological and engineering points of view (Edin et al., 2008). Primarily, robot grasping is synthesized by the ability of robot grippers to measure and detect sensation of touch, pressure, slippage and stiffness of object (Xiong et al., 1999). Twisting motion by robot hand is an advanced ability that performed based on grasping technology. Recent development of high performance robot hand such as the Gifu Hand III (Mouri et al., 2009), the UTAH/M.I.T Hand (Jacobsen et al., 1984), Shadow Hand and the high-speed robot hand produced at the University of Tokyo, which perform impressive acts of dexterity and skillful manipulation (Senoo et al., 2009), have urged the development of reliable sensing devices and control algorithms to merge perfectly with robot controller. However, sensitization of robot hand is still remaining as crucial issue since most of robot hand systems are only capable to grasp a predefined specific object (Mouri et al., 2007).
It is still difficult for robot hand system to realize human-like tactile sensation. Some common problems in robot hand system are low accuracy sensing device, sensors are not robust enough for long time work and heavy duties, inconsistence tactile sensing detection and difficulties in control of sensing fusion with robot trajectory. These problems are apparently drawback the progress to commercializing robot hands as real consumer products.
Apparently, sensorization of robot hand is an important issue that requires intelligent solution. In order to contribute to this effort, the direction of our research has focused on development of high performance tactile sensing device for application in robotic hand. Basically, tactile sensor is a device that can measure a given property of an object or contact event through physical contact between the sensor and the object. So far several researchers have mounted tactile sensors on robotic hand to evaluate grasping abilities and manipulation performance Maekawa et al., 1992).
The proposed tactile sensor in this research known as optical three-axis tactile sensor is based on optical waveguide transduction method (Ohka et al., 2004;Nicholls, 1990). The sensing principle used in this tactile sensor comparatively provides better sensing accuracy to detect contact phenomena from acquisition of three axial directions of forces (Ohka et al., 2005). Therefore, it is capable of measuring both normal and shear force simultaneously and suitable for application in dexterous robotic hands (Yussof et al., 2007). In this research, sensorization of robot hand is realized using the optical three-axis tactile sensors that are mounted on the tips of two robot hands attached to a robot arm as shown in Fig. 1. The robot hands are functioning as fingers for the arm. The total degree-of-freedoms for this robot arm is 11-dofs: 2-dofs at the shoulder joint, 1dof at the elbow joint, 2-dofs at the wrist joint and 2 units of 3-dof robot hands.

Hardware structure and measurement principle:
The hardware structure of the optical three-axis tactile sensor, as shown in Fig. 2, consists of an acrylic hemispherical dome, an array of 41 pieces of sensing elements made from silicon rubber, a light source, an optical fiber scope and a CCD camera. The optical fiber-scope is connected to the CCD camera to acquire the images of sensing elements touching the acrylic dome inside the sensor. The silicone rubber sensing element is comprised of one columnar feeler and eight conical feelers that remain in contact with the acrylic surface. The light emitted from the light source is directed towards the edge of the hemispherical acrylic dome through optical fibers. When an object contacts the columnar feelers, resulting in contact pressure, the feelers collapse. At the points where the conical feelers collapse, light is diffusely reflected out of the reverse surface of the acrylic surface because the rubber has a higher reflective index. The contact phenomena, which consist of bright spots caused by the collapse of the feelers, are observed as image data, retrieved by the optical fiber scope connected to the CCD camera and transmitted to the computer. In the measurement process, the normal force of the F x , F y and F z values is calculated using integrated gray-scale value G, while shearing force is based on horizontal center point displacement. The displacement of gray-scale distribution u is defined in Eq. 1, where i and j are the orthogonal base vectors of the x-and yaxes of a Cartesian coordinate, respectively. This equation is based on calibration experiments and material functions are identified with piecewise approximate curves (Ohka et al., 2006). Finally, each force component is defined in Eq. 2: F x = f(u x ), F y = f(u y ), F z = g(G) (2) Control system structure: The system layout of the optical three-axis tactile sensor is shown in Fig. 3. When contact pressure is applied on the tactile sensor elements, a bright spot area are appeared inside the tactile sensor which then captured as image data by a CCD camera. The image data retrieved by the CCD camera are delivered to PC via PCI bus of image processing board Himawari PCI/S. Then the image data are saved in an internal buffer area that created inside the PC internal memory space. The control system architecture of the robot hand is shown in Fig. 4. It is comprised of three modules: A Connection Module, Thinking Routines and a Hand Control Module. The architecture is connected to the tactile sensor controller by the connection modules using TCP/IP. Thinking Routines Module function is basically to define what kind of information is to be acquired from the tactile sensor, how to translate and utilize this information and how to send commands to the robot finger to properly control the velocity of the finger motion. As shown in Fig. 4, inside the Thinking Routines Module, there is a Thinking Routine Chooser that consists of a Pin Status Analyzer and a Velocity Generator. Moreover, there is a Motion Information Structure that connects to both the Pin Status Analyzer and the Velocity Generator. The Pin Status Analyzer module receives information from the tactile sensor about the condition of the sensing elements and uses this information to determine a suitable motion mode. Then it sends a list of sensing elements to the Connection Module to acquire tactile information.
Meanwhile, the Velocity Generator Module determines finger velocity based on the Finger Information and Motion Information Structure. The Motion Information Structure consists of initial velocity and motion flag modes, which are used to control the finger movement. Meanwhile, the Finger Information Structure provides connections to all modules so that they can share finger orientation, joint angle and tactile sensing data from each sensor element. A User Interface was designed so that the operator can provide commands to the finger control system. The finger control module controls finger motions by calculating joint velocities and angles.

Force detection and analysis:
Normal force: Basically, normal force is measured based on the changes of brightness at conical feelers of sensing element that touches acryl dome surface based on image analysis is a specific sampling time. The changes of brightness are proportional with the force applied to the sensing element. According to Eq. 1 and 2, the integrated gray-scale value of contact area g(x, y) is proportional with contact force p(x, y), as shown in (3): Here: Based on this relationship, we define the normal force from the gray-scale distribution of the contact area. The measurement of contact force P is defined by the following integration, where S is size of the grayscale measurement area.
Here, when (3) is applied to (4), we can define measurement of contact force P as following Eq. 5: Shear force: Shear force is measured based on centroid displacement of sensing element, which is the center point of the bright spots area that equals to the center position of the integrated gray-scale measurement area calculates in a specific progress time. When tangential force is applied to the sensor element as shown in Fig. 5, the conical feeler`s contact area at sensing element with the acryl surface is shifted horizontally. This horizontal centroid point displacement at the x and y axes are measured to define the shear force.
The centroid positions at the xy-axes, which are described as x G and y G are defined by applying the increment of integrated gray-scale value ∆g(x, y) within the measurement area of the integrated gray-scale value, as shown in Eq. 6 and 7, respectively: Based on the above equations, the displacement of the centroid point at the xy-axes in time t is defined as follows: Slippage detection: Slippage detection is an important component in grasp synthesis analysis particularly to define optimum grasp pressure when deals with unknown object stiffness. In this study, due to the cylindrical shape of the sensing elements, slippage normally occurs when the grasped object surface alternates between sticking to the sensing element and sliding over the sensing element. In our system, this 'stick-slip' phenomenon can be measured by calculation of time derivation of centroid point displacement in shear force detection. Since the present control algorithm allows robot hands to refine their grasp pressure when slippage is detected, the robot hands will be able to handle various types of objects by adjusting the parameter value of the centroid point displacement threshold. In the control system, the amount of centroid change dr for x-directional (dx G ) and y-directional (dy G ) of the fingertip coordinate frame, by means of shear force distribution are considered. In order to correlate the stiffness distinction of hard and soft objects, we utilized the increment of normal force ∆F, which was calculated within a specified progress time, as a stiffness distinction parameters. If ∆F is over the dr value, the finger re-pushes toward the object to prevent it from slipping. If the detected ∆F was lower than a specified value, the finger system uses the dr value to control the finger's re-push velocity so that the grasping becomes gentler and finally stops when the centroid change is over a specified dr value.

Control parameters and algorithm:
Control parameters of this system are defined from primary calibration test that was conducted during development of the tactile sensor. The test was conducted using both hard and soft objects. The robot control parameters are shown in Table 1. The robot hand movements are basically controlled by thresholds of normal force F 1 and F 2 . If the normal force is over the F 1 value, both fingers will not further re-push toward the object. F 2 is used for emergency stops in case of over push towards the object occurred.
In the robot hand control algorithm, at first the control system sees the threshold of centroid change dr before performing stiffness distinction using the increment of normal force ∆F. Then the fingers reinforce the grasping pressure to re-push the object according to the velocity ratio. Regarding the threshold of normal force, if the object was detected as soft object, the finger will no longer re-push the object when the normal force detected exceeds F 1 .

Verification of force detection:
A set of experiments was conducted to verify the performance of force detection in the proposed control system by recognizing the stiffness of unknown objects and responding to the changes of object weight during object grasping tasks, as shown in Fig. 6. The experiment was conducted using robot hands and optical three-axis tactile sensors were mounted on the hand tips.
The object was an empty paper cup that weighed about 4 grams. Motion planning was designed so that both fingers could move along the x-axis direction to grip the cup, lifting it up along the y-axis direction. As shown in Fig. 6, at 40 sec we poured 60 mL of water into the cup, then at 55 sec we poured another 30 mL and finally 20 mL at about 70 sec. This is to analyze the control system performance against sudden changes of the object's weight.
At first both fingers softly touch the cup to recognize its stiffness and define the optimum grasping pressure. Based on the proposed control algorithm, the system recognized stiffness of the paper cup. When optimum gripping pressure is satisfied, the hands lift the cup without crushing it. At this moment, the parameters of F 1 and F 2 are used to control the grip force so that the fingers do not crush the cup. The fingertip movements and the detected normal force and centroid change data of both fingers are compiled in graphs as shown in Fig. 6. In this Fig. 6 the movement of the right hand at the x-axis against the detected normal and shear forces can be observed. In this experiment, when water is poured into the cup, slippage was detected by tactile sensors and resulting centroid changes at the x-directional of the sensor elements. The hand system responds by adjusting the hand position to tighten the grip so that the paper cup will not slip out. The movement of hand during adjusting grasp pressure can be observed in the above graphs.
Optimization of control algorithm: The objective of this study is to evaluate performance of the proposed system conducting dexterous grasping and twisting motions. In order to accomplish the objective of the present case study, it is necessary to optimize the control algorithm. Basically the algorithm is divided into two phases: one for grasp, move and release motions and second for grasp and twist motions during the twisting task.
In the robot hand system, the sensor control program and hand control program are executed in different computers because CPU time is efficiently consumed using the multi-task program method. These programs are synchronized as the following five flags according to the tri-axial tactile data and finger motions: SEARCH: Hands search for an object until normal force of a sensing element exceeds a threshold F 1 or a Slip flag is raised.

MOVE:
This flag is raised whenever the robotic hand manipulates an object. TOUCH: This flag is raised whenever one of the fingers touches an object. SLIP: This flag is raised whenever time derivative of shearing force exceeds a threshold dr. OVER: This flag is raised when normal force of a sensing element exceeds a threshold F 2 . Figure 7 shows the algorithm of the flag analyzer in the control system. The outputs of this algorithm are TOUCH, SLIP and OVER flags. In the flag analyzer, the robot control system realizes the status as touching an object when normal force of a sensing element exceeds F 1 or absolute time derivative of shearing force exceeds dr (SLIP flag is raised). At this moment, the TOUCH flag is raised. The OVER flag is raised when normal force of a sensing element exceeds F 2 to prevent over squeezing toward the object. Figure 8 shows the algorithm of the velocity generator inside the Thinking Routines. The velocity of the hand is determined based on the five flag values. Whenever the SLIP flag is raised, a sensing element of the largest normal force is identified and re-compressive velocity of the hand is determined as an inward normal line of the sensing element. The re-compressive velocity is added to the current velocity and the resultant velocity is applied to the control module.

RESULTS
An experiment of grasping and twisting motion was conducted to evaluate the performance of the proposed tactile sensor system and the robot hand control algorithm. The object is a mineral water bottle cap. For this experiment, direction of slippage is utilized to improve the performance of the control system. When slippage direction is at the positive side, the robot will continue the task, but if the direction is at negative side the robot hand will release the object. In this experiment, the cap and bottle are located at the place where only the geometrical positions of the xand y-axes are primarily defined. The height at the zaxis is not specified. This is to evaluate slippage direction algorithm because the robot control system will use the slippage direction to justify the timing to lift and release the object. The dr value is 0.004 (mm) for the whole tasks except during twisting task which was increased 1.3 times so that the tactile sensor can optimize slippage detection to justify the cap tighten condition. Figure 9 shows pictures of the experimental condition of grasping and twisting motions. At first, the robot hands grasp the object and adjust its grasp pressure. At this moment, since the first detected shear force direction that exceeded the dr value is at the positive side, the robot arm continues lifting the object and moves to the bottle position.
When the robot hand moves toward the bottle, the cap touched the bottle head surface causing great slippage due to rotation of the cap. Since the detected slippage direction that exceeded dr value is at the negative side, the robot hands gently release the cap. Next, the robot hands performed twisting tasks. At the third cycle of the tasks, the robot controller realized that the cap was already tightened.

DISCUSSION
This study introduced optical three-axis tactile sensor to contribute to the effort of improving sensorization quality in robot hand system. The tactile sensor presented in this study is capable of detecting normal and shear force simultaneously. Methodology of force and slippage detection was presented with definition of control parameters acquired by the tactile sensor system.
The proposed methodology was verified in experiment with paper cup and water, in which the result shows the robot control system managed to respond to the proposed object stiffness distinction parameters and effectively respond to sudden change of object weight during grasping. This experiment also verified that the proposed control algorithm is effective in responding towards slippage sensation during grasping.
Finally, an experiment of grasping and twisting motions was conducted using a bottle cap. In order to perform simultaneous grasping and twisting tasks, optimization of the control algorithm was conducted with additional parameters to satisfy the desired tasks. Experimental result shows that the robot hand managed to perform grasping and twisting of bottle cap smoothly. The overall results revealed good performance of the proposed optical three-axis tactile sensor system and robot hand control algorithm for future application in a real artificial robot hand.
In addition, slippage sensation measured in a robot control system could contribute a better maneuvering of the robot arm-finger system when handling an object, particularly when the geometrical orientation of the object is unknown.

CONCLUSION
In this study, we developed a new control algorithm in a robot arm based on tactile information acquired to enhance sensorization quality of robotic hand. The proposed optical three-axis tactile sensors are mounted on fingertips of the robot hands. The control algorithm was evaluated in an experimental case study of grasp, move and twisting motions. Prior to the experiment, we presented the methodology of force detection and slippage measurement in the optical three-axis tactile sensor and optimized these characteristics to develop suitable control algorithm. Basically, the algorithm was classified in two phases, grasp-move-release and grasp-twist.
The experimental results using a bottle cap revealed good performance of the proposed algorithm for future application in a real artificial robot hand. This result shows that integration of tactile and slippage sensation in a robot control system could contribute a better maneuvering of the robot arm-finger system when handling an object, particularly when the geometrical orientation of the object is unknown and year 2008-2010 Grant-in-Research by the Japan Society for the Promotion of Science (JSPS) no. P08062. The authors are also grateful with the support from University Technology MARA.