Manipulating Deformable Linear Objects: Characteristic Features for Vision-Based Detection of Contact State Transitions

Transcription

1 Maniplating Deformable Linear Objects: Characteristic Featres for Vision-Based Detection of Contact State Transitions Jürgen Acker Dominik Henrich Embedded Systems and Robotics Lab. (RESY) Faclty of Informatics, Bilding 48 Kaiserslatern Uniersity of Technology, D Kaiserslatern, Germany [acker, Abstract This paper deals with the handling of deformable linear objects (DLOs), sch as hoses, wires, or leaf springs. It inestigates sable featres for the isionbased detection of a changing contact sitation between a DLO and a rigid polyhedral obstacle and a classification of sch contact state transitions. The reslt is a complete classification of contact state transitions and of the most significant featres for each class. This knowledge enables reliable detection of changes in the DLO contact sitation, facilitating implementation of sensor-based maniplation skills for all possible contact changes. 1 Introdction In most cases, indstrial robot systems work only with rigid objects; howeer maniplation of deformable linear objects (DLOs) sch as hoses, wires or leaf springs is desirable, too. For example, the atomotie indstry mst handle DLOs. In particlar, the atomated installation of cables and hoses in the motor compartment reqires a concept for maniplating DLOs. Other application fields are hot wire maintenance or assembly of control cabinets. The main problem in handling DLOs is coping with ncertainties. One particlar problem is determining the exact shape of a DLO at the start of a maniplation process, since the shape depends on the preceding maniplation steps. Also, de to manfactring tolerances, the shape of each indiidal DLO may differ. Dring the maniplation, the shape of the DLO changes de to graity and contact forces. The prediction of sch ariations with sfficient precision is typically ery difficlt. The obios approach is the se of sensors to compensate for these ncertainties. Some sch possibilities are se of a force/torqe sensor [6], or implementation of a ision system and force/torqe sensor like Nakagaki et al [10]. Both try to sole a special form of the peg-in-hole task and ths inestigate the soltion of clearly specified single tasks. How those special soltions can be applied to more general cases remains nclear. Mch preios research has been performed inoling rigid work pieces. The main problem addressed was deelopment of robst and flexible rotines (skills) for typical assembly or disassembly tasks. Hasegawa et al. [3] presented the skills moe-to-toch, rotate-to-leel and rotate-to-insert for handling rigid objects. Those skills encapslate the programming-intensie sensor data processing and can be sed for soling complex assembly or disassembly problems, like the disassembly of a ale [3]. Later research addressed the problem of finding a niersal set of maniplation skills. Morrow and Khosla proposed a taxonomy to deelop maniplation task primities for composing sets of robot skills likely to coer a gien domain [8] and [9]. For deformable linear objects, a similar approach based on contact states was introdced by Henrich et al. [4]. As this was the basis for the work presented here, the next section describes this approach in more detail. Remde [11] analyzed the possible transitions between contact states and implemented a force-based recognition of those contact state transitions in [12] and [13]. This method was implemented in [14], sing assembly of a leaf spring as an example of se. The reslts showed that some transitions between contact states are hard to recognize with force sensor data, bt the additional se of other types of sensors may sole this problem. We propose ision sensors sch as a CCD camera as promising soltions. To erify this, ision-based recognition is inestigated independently. A first attempt at ision-based recognition of contact state transitions was made in [1], based on a qalitatie description of the progress of for featres, bt is not complete. Frther, the releance of the proposed featres as the basis for the recognition is not clear and the image processing depends on some strong assmptions, which shold be weakened. Here, we propose to find the featres most characteristic of transitions that are widely independent of the nderlying low-leel algorithms for image processing or object representation. Another objectie is assigning the different transitions to classes and generating sensor drien rotines for detecting sch transitions. In Section 2, we first describe preios work and or working assmptions. Section 3 describes or basic approach for isal recognition of contact state transitions. In each of the following sections (Section 4 throgh 6), we introdce one class of transitions and describe the characteristic featres for each class. In

2 Section 7, we smmarize the reslts and gie an oeriew on the ftre work. 2 Preios Work The work described here, depends heaily on the contact states for DLOs introdced in [4] and [11], ths, the following is a smmary of that research. Since a polyhedral enironment is assmed in [4], only ertices, edges, and faces exist as geometric primities. The DLO is modeled by two ertices with an edge between them. The reslting contact states are shown in Figre 1. A contact is considered to be stable if and only if a small moement in any direction does not change the contact state. (Een if the position of the contact point or contact line changes.) This reqires a contact force, since otherwise only instable contacts exist. Only stable contact states can be established and released in a reliable manner. The instable contact states behae ncontrollably as described below. since the instable ones change spontaneosly to a stable state and sch spontaneos transitions cannot be controlled. Since the spontaneos transitions lead to stable contact states, the instable ones may still occr as transient states after an initiated state transition leads to an instable state. The reslting contact states and state transitions are shown in Figre 2, with regard only to single contacts. In addition to the assmptions mentioned aboe, sch as a polyhedral enironment and contact forces, we make some assmptions for the ision system. First, we restrict orseles presently to elastic DLOs with emphasis on low elastic deformation (E+, E classes in [4]). Highly elastic objects may oscillate after acceleration, bt we assme here that objects will either not oscillate or that an actie damping [15] took place. After an initial acceleration, the acceleration of the gripper shold be zero (linear motion) ntil the contact state transition takes. Frther, the camera is placed at the optimal obseration point for each transition and the enironment is assmed to be static, so only the robot and the maniplated object moe, and the other obstacles remain nchanged dring the maniplation process. E / V N V / E E / E L V / V E / F E / E P V / F Figre 1: Enmeration of contact states 1. Before the transitions between contact states are examined, we first frther inestigate the effects of the reqired contact force. This contact force presses the DLO against the obstacle, casing a deformation. For the stable E/F Point and the instable E/E Point contact states, the force flattens the DLO at the contact point, reslting in a short line contact instead of real point contact. Therefore, both contacts behae like their corresponding line contacts, so we will only frther discss the stable E/F Line and stable E/E Point contact states. The instable E/E Line contact state is also regarded. Becase of the obstacle s geometric properties, the stable E/E Point and the E/V contact remain point contacts. After defining contact states, we mst now consider the possible transitions between them. A contact state transition is a change from one contact state to another withot passing any intermediate contact states. For the handling of DLOs, the stable states are most interesting, Figre 2: State transition graph, solid edges indicate initiated and dashed indicate spontaneos transitions 2. 3 Basic Approach One method for ision-based detection of contact state transitions is the se of an object recognition system and generation of a 3D model of the DLO and the enironment. Howeer, since compter ision is still mch inferior to hman ision, the deelopment of a system for recognizing DLOs and calclation of a 3D model is still an expensie task, especially the calibration of the camera(s). Fortnately, the introdced approach does not reqire a precise model for DLOs de to abstracing from geometric details. In [13], a force/torqe sensor was sed to detect contact state transitions instead of a ision system. Since 1 From [4,11] bt the instable E/E point contact has been added. 2 From [11] bt the instable states E/E and V/V hae been added.

3 the force/torqe sensor is nable to sense the enironment and the method described here does not se a 3D model, or ision-based system may not need sch a 3D model. The internal forces measred with the force/torqe sensor deform the work-piece, reslting in obserable deformation. Indeed, [7] describes an approach for fsing ision and force sensor data, whereby the ision sensor is sed as an additional force/torqe sensor by calclating the forces based on object deformation. This means identifying ertices, edges or faces is not necessary for detecting contact state transitions. For this reason and becase we assme a static enironment, we can se the work piece motion to segment backgrond and foregrond [5]. Or approach is basically a flow-based one, since we always regard a seqence of images, bt we do not restrict orseles to any special techniqe sch as optical flow. More concretely, we se a stationary gray-ale camera to acqire images and remoe the static enironment from the image by calclating difference images. In the next step, binary images are calclated by applying a threshold ale to the difference images. In the reslting binary images, only the work-piece and the robot remain (Figre 3). The remaining seqence of threshold-applied difference images (hereafter: difference images) shold contain sfficient information for sccessfl ision-based detection of contact state transitions. Bt now we mst answer the qestion of recognizing contact state transitions in sch a seqence of difference images. Since we do not want to restrict orseles to any specific algorithm, or objectie is the identification of general featres that are significant for detecting state transitions. The deelopment of algorithms for detecting/extracting those featres from the image seqence is part of ftre work. classes, with each class handled by one skill and its inersion. Since force-based transition detection ses one skill for eery single transition, the following classification mainly deals with ision-based methods. The next sections describe three classes of contact state transitions and the featres for detecting them. The classes are similar bt not identical to the three skills sed by Hasegawa [3] for handling rigid objects, since here we regard deformable objects based on the state transitions shown in Figre 2. While no set of transitions corresponds to his rotate-to-insert skill and the last class described here only applies to deformable objects, his moe-totoch and rotate-to-leel skills are almost identical to the first and second class described below. 4 Establishing and Releasing Contact The moe-to-toch skill is sefl for handling deformable objects and is represented in Figre 2 by all edges from N to any other state. In contrast to Hasegawa [3], we also introdce a moe-to-detach skill. This skill is the inersion of the moe-to-toch skill and is represented by any initiated transitions to N. Both skills belong to the establishing- and releasing-contact class. Since we always assme a contact force for any existing contact, only motions leading to sch a contact force are allowed. Two examples for allowed motions are gien in Figre 4b and 4d. The motions 4a and 4c are examples of establishing a contact withot deformation of the work piece. Sch motions are not allowed, since they do not lead to a contact force. The rest of the paper concentrates on translatory motions like 4b, bt the featres described work also with rotatory motions like 4d. (a) (b) (c) (d) (a) (b) Figre 3: The original image taken by a grey-ale camera (a) and the reslting binary image (b). Any algorithms deeloped mst flfill some reqirements, an important one being reliability. A reliable algorithm mst detect eery transition and the contact states after the transition mst be stable. In particlar, we need a contact force to achiee a stable contact. So, for rigid objects the transitions mst be detected as fast as possible; otherwise the object or enironment may sffer damage. In contrast, with a DLO, the transition mst not be detected so qickly or an instable contact may be the reslt. Instead of looking for featres for eery single contact state transition, similar transitions are groped together in Figre 4: Establishing contact ia translatory motions (a,b) or rotatory motions (c,d) while deforming (b,d) or not deforming (a,c) the work piece. The first featre examined for ision-based transition detection is of corse deformation. As long as the DLO is not in contact with any obstacle (bt the gripper), it may deform de to graity. Establishing a stable contact leads to a deformation cased by the contact force. This deformation may be measred by obsering the cratre of the DLO. The main characteristic of an established contact is that at least a part of the DLO stops moing in at least one direction, while other parts contine their motion. This can be seen in Figre 5, where two seqences of difference images are shown. Eery seqence consists of for binary images; the brighter the DLO, the older the corresponding difference image. In

4 both seqences, the DLO is moed in the -direction ntil it hits an obstacle (box). In the left seqence, the tip of the DLO stops moing in the -direction and starts moing in the negatie -direction. If de to friction or the DLO s shape it does not moe in the -direction, the -direction motion will stop at the contact point, since the obstacle preents frther motion. Either stoppage of motion in the -direction or additional motion in the -direction will always occr. The right-hand seqence also shows this effect, whereby the marked part stops moing in the - direction. Ths, the common featre is that part of the DLO stops moing in one direction as it hits an obstacle. This is the characteristic featre for all transitions from N to any other contact state, inclding all instable ones. This loss of motion may be detected by comparing sccessie difference images. The reslt of sch a comparison of the last two images is shown at the bottom line of Figre 5. The later an image is taken, the more the DLO has moed in the motion direction ntil the DLO hits the obstacle. We can then distingish p to three areas a -, a + and a 0. In a + the DLO contines moing bt in a 0 it remains at the same place, whereas in area a -, the DLO een moes backwards. The existence of an area a 0 is eqialent to a stop in motion; ths, it appears in both examples. The reerse transitions from any state to N can be detected in the same way, bt the characteristics are inerted. This means that we start with an a 0 area and after the contact state transition, only the a + area remains. (a) a a a (b) a a a Figre 5: Seqence of difference images for the transition N V/F (a) and for N E/E P (b); for better orientation the obstacle is also shown. More difficlt than detection of contact is the identification of the reslting contact state. Since or basic approach ses difference images, the obstacles are inisible in the binary images. Additionally, it cannot be decided whether the obstacle is an edge or a face. The seqence in Figre 5b shows an N E/E P transition. An obstacle represented by the dotted line wold prodce the same seqence of images, bt the transitions wold be N E/F. Since we cannot sense what geometric primitie the DLO is in contact with, we can only try to decide whether the edge or the ertex of the DLO is in contact. Howeer, een this task is difficlt since the deformation de to stopped motion may not appear at the tip een for a ertex contact (Figre 5a). Bt since the tip of the DLO stops moing in the motion direction for any ertex contact, this enables one to distingish DLO ertex- from edge-contacts. 5 Stable State Transitions The second class contains all immediate transitions from any stable contact state other than N to any other stable contact state other than N, like V/F E/F. This class corresponds to Hasegawa s rotate-to-leel skill for rigid objects, becase his skill and this class share the main characteristic, namely the change from a point to a line contact. Bt in contrast to the case of rigid objects, where this can only be done by a rotatory motion, in DLOs both translatory and rotatory motion can change a point to a line contact. This class also contains the reerse transitions from line to point contacts. (a) l,l 1 2 l3 l4 c c c,c (b) c 4 c 2 c 3 c 1 l 3 l 2 l 1 l 4 Figre 6: Seqence of difference images i = 1,...,4 for the transition E/E P E/F (a) and V/F E/F (b) with contact length l i and centre of contact c i. The deformation change depends on the motion. For example, the transition V/F E/F can be made withot frther (isible) deformation of the work piece by sing the contact point as rotation centre. Translatory motions mst deform the work piece, for this reason a rotate-toleel motion cannot be sbstitted with a translatory motion for rigid objects. The deformation offers a way to recognize sch contact state changes, assming a translatory motion. Bt the bending of the DLO may increase (Figre 6b) or decrease (Figre 6a), depending on the specific geometric sitation. Althogh the shape changes heaily, it is difficlt to determine the transition point, so additional featres sch as the length of the contact shold be regarded. It is obios that a point contact is smaller in length than a line contact, so this can easily be sed as a featre. The examples in Figre 6 show two transitions from a point to a line contact, whereby the contact length l i is drawn for eery image. Another featre is the motion of the center of contact c i. This centre is nchanged as long as the work-piece is in the E/E P contact state (Figre 6a) bt after the transition to E/F, it starts moing. In the second seqence (Figre 6b), the centre first moes to the left while the work piece is in V/F and after reaching E/F it moes back (c 4 ). The reerse transitions

5 E/F V/F and E/F E/E P can be detected in the same way, bt the characteristics mst be inerted. The deformation and the length of contact are qantitatie featres, since the featres do not change their sign at the contact point. In contrary, the contact point motion is more a qalitatie featre becase the centre of contact either begins moing and stops after the transition or changes the direction of its motion. Ths, this featre is expected to be more resistant to noise, making it more reliable than others. The contact state alidation here is as difficlt as for the establishing and releasing contact class bt as the transitions can be flly controlled, an additional alidation is not really necessary. 6 Spontaneos Transitions All transitions in the transition graph (Figre 2) marked spontaneos and those leading from stable to instable states still remain to be classified. (a) Figre 7: Seqences of difference images for a transition V/F V/E N (a) and V/F 1 V/E V/F 2 (b). Those transitions ending in an instable state cannot be detected with the featres discssed in the last section becase the contact wold remain a point (V/F V/E) or line contact (E/F E/E L ). Figre 7b shows sch a transition (V/F V/E) and the following spontaneos transition (V/E V/F). Hmans can recognize the V/F V/E transition in the original seqence of grayale images. Howeer, in the seqence of difference images een a hman fails to detect this transition, de to the need for edge-recognition capability. Fortnately, detection of this transition is not een necessary, since (de to instability of V/E) another recognizable transition takes place. Here, this spontaneos transition leads to the stable state V/F, a transition recognizable een in seqences of difference images. In general, seeral stable states can be reached from an instable one. Which one is actally reached depends on seeral conditions; an exact analysis of the transition conditions can be fond in [7]. Whateer stable contact state reslts, there is always a spontaneos transition after reaching an instable contact state leading to a stable one. Ths, the third class does not contain single transitions bt seqences of transitions. Sch transition seqences consist of an initiated transition to an instable state and a (b) spontaneos transition to a stable state. Since the spontaneos part is isible in the seqence of difference images, the seqence is recognizable een if the first part cannot detected by or approach. Contrary to initiated transitions, spontaneos transitions are one-way transitions, so only one skill is needed for this class. Frther, since sch spontaneos transitions lead often to state N, this class wold intersect with the establishing and releasing contact class. Ths, the restriction to "initiated transitions" for the establishing and releasing contact class is made. First, we assme the reslting state is not N. One way to detect sch a transition is obsering the contact point motion. If the DLO is dragged oer an obstacle, the contact point always follows the srface of the obstacle as long as it is in contact. Since only a polyhedral enironment is considered, the contact point always moes along a straight line (see Figre 7). A transition from any stable contact state to another contact state ia an instable one cases a discontinity. In Figre 7b, the face before and after the V/E contact state mst hae different normal ectors, otherwise both faces are identical and there is no transition. In general, the orientation of the geometric primities before and after the transition seqence mst differ. Since the contact point mst follow the srface, this change of orientation means the contact point now follows a straight line with a different direction ector than before. In the second case, (Figre 7a), the contact is released after the spontaneos transition, bt this can be sensed with the featres discssed in Section 4 or by obsering the deformation of the DLO. The characteristic of spontaneos transitions is the release of stress. This makes sch a transition ncontrollable bt also redces bending of the work piece. The motion before the spontaneos transition may deform the DLO and may een redce bending, bt the transition itself at least cases an additional redction, so there is always a discontinity while obsering the change of deformation. In addition, sally straightening occrs rather qickly and initiated transitions case continal (n)bending, so fast changes are typical for spontaneos transitions. A state alidation is again difficlt bt at least it is possible to distingish the transition leading to N from those leading to any other stable state. This is possible by examining featres sed to detect release of a contact. 7 Conclsions We identified three classes in particlar: one for establishing and releasing contact, one for changing from one stable contact state to any other state bt N and one for the spontaneos transitions. For each class, some of what we beliee to be characteristic featres for isionbased detection are gien. The listed featres allow for recognition all transitions in Figre 2 bt the transitions from any stable state (except N) to any instable state. For recognizing sch transitions, the recognition of the enironment is necessary. Howeer, the instability of

6 sch contacts reslts in a spontaneos transition leading to a stable contact state. Since the detection of sch spontaneos transitions is possible, recognition of the enironment is not needed. Bt one disadantage remains, since withot recognition of the enironment, we cannot identify the contact state after the transition. Only a weak estimation is possible withot frther actiities sch as probing or exploration motions. The adantage of or approach is the simplicity; we foresee neither the need for calibrated cameras nor for a large, time consming system for object recognition. Remaining steps inclde deelopment of some lowleel image processing algorithms for extracting the described featres from the seqence of difference respectie binary images. Or focs here is again on simple, reliable and fast algorithms. The algorithm will be based on comparison of sccessie difference images like those shown in Figres 5 throgh 7. The sccessie difference images will be merged as in these figres, bt as eery image is to be sed, the seqence becomes mch smoother than in the figres shown. The reslting image is a good basis for calclating optical flow [15], and the optical flow field is expected to be sfficient for detecting the described featres. It is obios that for the recognition of any transition, the camera mst be able to obsere it. Therefore, the placement of the camera is another topic for ftre work. The image seqences shown in Figres 5 throgh 7 are taken from a side iew. Indeed, an angle of 90 degrees between the camera axis and the plane spanned by the moement ector of the DLO and the normal ector of the obstacle s srface was sed. Bt as we expect a fair amont of anglar independence, an angle of perhaps 45 degrees shold also work. Howeer, there is a big difference between this side iew and a top iew. Since een for a hman recognizing the contact state transitions based on a top iew is ery difficlt, we do not expect any algorithm based on the featres described aboe to fnction in this case. In general, more thorogh inestigation of where we can or mst place the camera is needed. In this context, we will also inestigate the reqired lightning conditions. References [1] Abegg F., Henrich D. and Wörn H.: Maniplating deformable linear objects - Vision-based recognition of contact state transitions. In: Proc. of the 9 th Int. Conf. on Adanced Robotics, Tokyo, Japan 1999 [2] Dais J. W.: Recognizing Moement sing Motion Histograms in: MIT Technical Report #487, 1999 [3] Hasegawa T., Sehiro T., and Takase K.: A modelbased maniplation system with skill-based exection. In: IEEE Transactions on Robotics and Atomation, ol. 8, No. 5, pp , October 1992 [4] Henrich D., Ogasawara T., Wörn H.:"Maniplating deformable linear objects: Contact states and point contacts". In: Proc. IEEE International Symposim on Assembly and Task Planning, Porto, Portgal, [5] Jain R, Martin W.N. and Aggarwal J.K.: Segmentation Throgh the Detection of Change De to Motion In: Compter Graphics Image Processing No. 1 September 1979, pp [6] Kras W., McCarragher B. J.: "Case stdies in the maniplation of flexible parts sing a hybrid position/force approach". In: Proc Int. Conf. on Robotics and Atomation, ol.. 1, pp , Albqerqe, USA, April, [7] Lo Y. and Nelson B.J.: Fsing Force and Vision Feedback for Maniplating Deformable Objects, Jornal of Robotic Systems 18(3), pp [8] Morrow J. D., Khosla P. K.: "Maniplation task primities for composing robot skills". In: Proc IEEE Int. Conf. on Robotics and Atomation, pp , Albqerqe, USA, April [9] Morrow J. D., Nelson B. J., Khosla P. K.: "Vision and force drien sensorimotor primities for robotic assembly skills". In: Proc IEEE/RSJ Int. Conf. on Intelligent Robots and System, Pittsbrgh, Pennsylania, USA, Agst [10] Nakagaki H., et al.: "Stdy of deformation and insertion tasks of a flexible wire". In: Proc Int. Conf. on Robotics and Atomation, ol. 3, pp , Albqerqe, USA, April [11] Remde A., Henrich D., and Wörn H.: "Maniplating deformable linear objects: Contact state transitions and transition conditions". In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Kyongj, Korea, October [12] Remde A., Pfaffenberger E., and Wörn H.: Maniplating deformable linear objects: Force based detection of contact state transitions. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Takamats, Japan, October/Noember [13] Schlechter A. and Henrich D.: Maniplating Deformable Linear Objects: Characteristics in Force Signals for Detecting Contact State Transitions In: Proc. of 10 th Int. Conf. on Adanced Robotics Bdapest, Agst [14] Schlechter A and Henrich D. Maniplating Deformable Linear Objects: Programming sing Different Maniplation Skills In: VDI Bericht 1679 Tagngshandbch zr Robotik 2002 Jne 19-20, Ldwigsbrg Germany 2002 [15] Schlechter A. and Henrich D.: Maniplating Deformable Linear Objects: Maniplation Skill for Actie Damping of Oscillations In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Lasanne, Switzerland 2002.