Robotic Guide Dog: Leading a Human with Leash-Guided Hybrid Physical Interaction

[00:00:00]hi everyone i'm zhongyu i'm going to present our work where we develop a cochlear robot that can lead a visual impaired person via a leash to safely navigate in narrow environments we call this robot a robotic guide dog this is a teamwork with ancient winter league gene enemy and they were advised by kosher so first of all why we need a robotic guide dog let me show you some numbers training guide dogs is both time and labor intensive with the process not easily scalable it takes about 20 months in uk to train from a puppy to a guidelock in addition the skills from one dog cannot be transferred to another one and only 40 percent of the trained puppies are qualified to serve as skydogs as you may expect it changing a qualified guido is very expensive and in the cost of around 50k dollars in u.s for the guide dog school and around 1k dollars per year for daily expenses but if we use a coachbar robot to serve as a robotic guide dog the aeros developed on one robot can be easily transferred to another one therefore we can mass produce robotic guide dogs to serve more people who need a guide and having a coach coachback robot is quite affordable these days it takes about 10k dollars to buy one and just needs to be recharged every day moreover the biggest advantage of using a robot is that we can enable a robot with sort of intelligence such as safe navigation autonomy and speech communications to better serve visual impaired persons they are already previously endeavors to develop robotic guiding systems and there are two major types first one is using a mobile robot to provide a human with physical assistance and active guidance another one is to use robotic cam to help the visual impaired person to detect the surroundings and to navigate however most of the previous work use rigid robot arm to lead the human and this has a drawback that is the system can get stuck in narrow spaces as you may notice that most of the previous work only show the ability to lead a person in some open spaces therefore in this paper we want to address this problem by introducing a leash here is our system we use a coachback robot that is a mini chitter this dynamic robot is able to perform algebra maneuvers with small footprints moreover it's morphologically similar to a puppy thus and does make it more welcome in the human community we build a sensor kit on top of it it has a rgbd camera to detect a human being led a 2-degree field and gimbal to follow the human by rotation in the pitching we also have a 2d lidar to do mapping localization and to detect obstacles there is also an on-board computer to process the real-time data in this system we're first introducing a leash to lead the human and the leash can be either slack or taught and they will have a force sensor at the end of the leash to measure the interactive force this clip shows advantage of using a leash the robot is leading a blindfolded person to navigate in narrow space when the system is getting stuck the robot will let the disk go slack and since there's no tension in the leash the robot the human will stop moving at this mode the robot can reposition and reorient itself to find a better configuration to lead human to pass through this narrow space without collision and now we are going to introduce how this guide autonomy work with the leash here is the outline of the talk today we firstly developed and validate a hybrid model to describe the dynamics between the human and the robot while using a guide leash which could be either slack or torque i think this is one of the first hybrid physical human robot induction model in this scenario this hybrid model is then utilized in organization based local planner to enable the robot to lead human while avoiding obstacles and in the last part we build up one of the first end-to-end robotic guide autonomy using the coachbar robot to lead blindfolded persons to safely navigate through narrow spaces let's introduce in modeling first we describe the robot leading human system in the 2d world frame the robot has his position and the turning yaw the human being led has his or her positions the robot using a leash to guide the human so the leash has its own lens and his orientation with respect to the robot we can assume that human will always hold the leash therefore the human position can be represented by owning the system states which include the robot 2d position xy turning your theta the leash angle fee and the dish end each length scale we also define the robot frame attached to the robot body then we can define the robot velocity in the rope in the inner body frame since the leash can go either slack or torque this makes the robot leaving human system hybrid the virtual input to this system includes the robot velocity and turning your rate when the leash is stopped there is eternal force in the leash we assume that the human will move along the leash direction while the leash lens keeps being fixed when the leash goes slack the leash length is varied below the original and they will we assume that human won't move because there's no tension in the leash the mode will switch from slack to taut when the robot is moving away from the human while the entirety force is larger than the certain threshold and if the robot and him are approaching to each other or the leash force is below the threshold the leash will go slack again however both of these models are built based on some assumptions but are they true we validate these models by experiments we firstly let the human operator to control the robot to lead blindfold the person in order to always keep the leash in the total mode we use the proposed dynamics in the third mode to predict robot and human trajectories this is the robot pass the light yellow line is matched with robot position which is the brown shoes while the top yellow is the predicted robo position by the model this is the human path the light blue is the ground choose human position and the dark blue is a predicted human position and we did a lot of these similar experiments and the proposed thought mode dynamics shows reasonably good prediction accuracy for both human and the robot therefore we can feel safe to assume that the dynamics of the top move is correct what about select mode dynamics this is the measured force in the leash during another robot leading human experiments and this is to match the human speed as you can see that when the interactive force drops below the threshold which is 12 newton shown in the dashed line the human tend to stop stop moving such as from 40 seconds to 60 seconds where the speed of the humor is below 0.05 meters per second therefore the assumption that human won't move if the force is near zero can work well in the select mode dynamics during the experiments once we obtain such hybrid physical hri model we can utilize this in opposition-based local planner to enable the robot to lead the human to reach the given waypoints while avoiding obstacles this is the system at the current state the robot and the human have their own dimensions we want to move the system to the topic state therefore we formulate a collocation based mixed integer or transition problem where we want to minimize the distance between the final node to the target node while having small virtual inputs and we want to subject it to the initial state conditions and we enforce the hybrid dynamics we just developed via collocations and we also have we also enforce the states and input bonds by solving these problems we can have the trajectories for both robot and human to approach to the to the given waypoints and this local planner can be embedded to an end-to-end robotic guide dogs framework this is the framework we developed for this work firstly after being given a gold location we use a star serving as a global planner to quickly find the path of the configuration for the entire system it will output the next waypoint to the local planner to check this local planner is what we just introduced using the hybrid physical hri model and we output the virtual inputs to the system which are the desired robo-working and the turning your velocity these commands are sent to a pre-built velocity checking controller for the minion and online we use a 2d lidar to detect obstacles and to do localizations and we have a rgbd camera to detect the human being led and this camera is able to move by a 2d gimbal to always keep the person in the wheel this this human following system will update the human position in the real time now we introduce our experiment results we deployed the ergozone on the mini cheddar and led it to lead blindfolded persons via a leash after selected glow a gold location the global planner generates a path for the entire system and the planned parts for the human is shown here which is a green line the match of the human is marked as a yellow cylinder and the robot current positions is marked as a blue box when the robo-leading human system entered the narrow spaces using the local planner and the hybrid physical hi model the robot will let the leash go slack because the system cannot pass through this space while the bleach being torqued human then stop because there's no force in the leash and no information from the outside at this time the robot can re-position and reorientate itself to find a better configuration by the local planner and then the robot will let the leash go taught again to inform the human to move with it in order to travel through this space we also test the system starting from different initial pose such thought slag switching may happen many times during the navigation the tightest space in this experiment setup is about one meter why the human robot separation is about 1.6 meter when the leash is taut we also test and evaluate our system on multiple human subjects without changing or teaching the human how to use the robot in conclusion we first developed a hybrid model to capture a dynamic relationship in the system of the robo-leading human with the leash to and validate this by preliminary experiments this hybrid model is then utilized in organization-based local planner to optimize for a trajectory for the robot to lead the human to safely reach the target position while taking advantage of the top-slag switching of the leash and according to the best of our knowledge we build up one of the first end-to-end robotic guide dog autonomy wizalish we showed advantage of this system by enabling the robot to lead the blindfolded presence through the narrow spaces we also show the potential to outperform guidos by using coach variables in terms of autonomous navigation our work is also featured by lots of media because they think this work may bring benefits to the visual impaired community and to the entire society and there are some guide dog schools show great interest to collaborate with us we will continue to push this project forward in the future we want to add speech interface on the robot and to enable a robot to operate elevators for the human being net ok this is all the material that i want to share thank you