{"id":233,"date":"2017-12-08T17:20:33","date_gmt":"2017-12-08T17:20:33","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2017teamh\/?page_id=233"},"modified":"2018-05-11T20:42:31","modified_gmt":"2018-05-11T20:42:31","slug":"gaze-subsystem-implementation","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2017teamh\/gaze-subsystem-implementation\/","title":{"rendered":"Gaze Subsystem"},"content":{"rendered":"

Oct 20. The gaze subsystem set-up<\/h1>\n

Two of the cameras faces towards the pupils, and the other faces towards the environment. From the image of the rest camera, we will locate the pupils, and analyze the direction. After calibration, the location of the gaze is mapped to the world-frame images. At the same time, the gaze tracking subsystem is also listening to the information from the environment perception system, which is publishing the objects detected in the working space and their position. According to the relative distance from the gaze and the center of the object, a confidence is obtained.<\/p>\n

Communication of the gaze node and AprilTag node is finished. The environment reconstruction is finished in AprilTag Node.<\/p>\n

 <\/p>\n

\"\"<\/p>\n

 <\/p>\n

Oct 27. Gaze based possibility<\/h1>\n

The model between gaze based and confidence should be following:<\/p>\n

When the gaze is in one of the bounding box, the gaze based confidence for that<\/p>\n

object should be very high. And as long as the gaze is in the bounding box, which means no matter which part of the object you are gazing at, the confidence should remain the same. If your gaze is not in the bounding box of any of the objects, then the gaze based confidence should be proportional to one over the distance.<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Nov 10.\u00a0 Intent Prediction<\/h1>\n

When the user wants one of the objects in the work space, he would gaze at the object,<\/p>\n

and give a command containing the objects. Under such assumption, given the intent,<\/p>\n

the gaze would not add more information for speech, so they are conditional independent.<\/p>\n

The relationship is described in the following figure.<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Nov 22. Update for the gaze possibility<\/h1>\n

The gaze possibility is accumulated over the time of speech, assuming that when the user is giving the command, the gaze would locate around the target most of the time.<\/p>\n

February 22. We Go Deep<\/h1>\n

While the previous approach has been very successful and has fulfilled our requirements. The intent prediction requires the user to stare at the object constantly and causes significant strain to user’s eyes in a prolonged interaction. So we decided to use deep learning techniques to learn the user’s gaze pattern and use that pattern to predict the object that the user intends to pick up. We have decided to use a multi-modal approach where we combine the information from the environment-facing camera and the gaze location. Below are our two proposed neural networks.<\/p>\n

 <\/p>\n

\"\" \"\"<\/p>\n

 <\/p>\n

March 23: Data Collection and Initial Results<\/h1>\n

To train our model, we collected and labeled our\u00a0own data set. The data set consists of 260 interactions where the user requested for an item on the table and is handed that item by the experimenter. After collecting the data we annotated the time when the gaze is shown to be expressing the intent of the object to be picked up.<\/p>\n

 <\/p>\n

This image demonstrated the experimental setup. Here we have the user staring at the table with the operator standing across the table. The user will request for an item they want and the operator will hand it over. The red dot here represent the location that the user is gazing at.<\/p>\n

\"\"<\/p>\n

This is the example of our data annotation process. We annotate the start and end time when the user’s gaze exhibits intend<\/p>\n

\"\"<\/p>\n

This graph is the initial training result of our neural network. The red line is the training error and the blue line is the validation error. While we obtained very high training error, the model overfitted the dataset and, consequently, had a relatively low validation error.<\/p>\n

\"\"<\/p>\n

 <\/p>\n

April.11th: LSTM-based Model for Intention Prediction<\/h1>\n

 <\/p>\n