{"id":155,"date":"2020-01-12T17:35:48","date_gmt":"2020-01-12T17:35:48","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/?page_id=155"},"modified":"2020-12-17T13:49:42","modified_gmt":"2020-12-17T18:49:42","slug":"design","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/design\/","title":{"rendered":"Design"},"content":{"rendered":"
<\/div>
\t\t\t

Mandatory System Requirements<\/h2>\n

Mandatory Performance Requirements<\/h3>\n

The system will:<\/p>\n

    \n
  1. Register camera-robot coordinate frames with a 10 mm maximum root mean-square-error (RMSE)<\/li>\n
  2. Register Blaser-robot coordinate frames with a 10 pixel unit maximum reprojection error<\/li>\n
  3. 3D-scan the surface of phantom liver in under 5 minutes<\/li>\n
  4. Segment the liver, including near organ occlusions, at 95% IOU<\/li>\n
  5. Estimate the motion of the phantom liver within 0.5 Hz<\/li>\n
  6. Generate a 3D point cloud of phantom liver within 10 mm RMSE when compared with known geometry<\/li>\n
  7. Palpate the phantom liver in under 10 minutes<\/li>\n
  8. Complete the entire surgical procedure in under 20 minutes<\/li>\n
  9. Detect >90% of cancerous tissue<\/li>\n
  10. Misclassify <10% of healthy tissue as cancerous tissue<\/li>\n<\/ol>\n

    Mandatory Nonfunctional Requirements<\/h3>\n

    The system shall:<\/p>\n

    M.N.1 – Comply with Food and Drug Administration (FDA) regulations
    \nM.N.2 – Reduce the cognitive overload of surgeon
    \nM.N.3 – Not occlude the phantom liver’s visibility to the surgeon
    \nM.N.4 – Exit autonomous mode immediately upon prompt
    \nM.N.5 – Yield itself to regular safety checks
    \nM.N.6 – Cost less than $5000<\/p>\n

    <\/h3>\n<\/div>\n\t\t<\/div><\/div>
    <\/div><\/div>
    <\/div>
    \t\t\t

    Functional Architecture<\/h2>\n
    \"\"<\/figure>\n<\/div>\n\t\t<\/div>
    \t\t\t

    The updated<\/strong> functional architecture in Fig. 1\u00a0 demonstrates the function of the chopsticks system in the simulation environment. It has 3 main subsystems and is a visual representation of the flow of data throughout our system. The sensors subsystem comprises of the sensors. These sensors are used to determine the visual and physical properties of the region of interest within the phantom organ scene.
    \nMore specically, the sensors will measure the force response of and image the phantom organs. The sensed signals will be passed to the data processing & organ modeling subsystem and the searching & palpating subsystem.<\/p>\n

    The data processing & organ modeling subsystem is broken into two steps. The first step is to align the data from the multiple visual imaging sensors into a consistent frame and the second is to use the visual data to determine the shape and movement of the organ. Each block within the subsystem represents a specic algorithm. The sensing information is fed into the rst block that computes transformations between the frames of each sensor and then transformed in the next block. These two blocks represent the data processing portion of this subsystem.<\/p>\n

    Afterwards, the transformed visual data is used to segment the phantom liver from the surrounding organs. The segmented data is passed to two parallel blocks that determine the 3D point cloud and estimate the movement of the organ. The final block combines these
    \nparallel outputs to create a moving 3D model of the organ. This is the final output of the data processing & organ modeling subsystem. The output of this system gets passed on to the searching & palpating subsystem.<\/p>\n

    Using the point cloud that represents the moving organ, the first block in the searching & palpating subsystem creates a searchable surface for use in the rest of the subsystem. The search, robot kinematics, force reading transformation, and stiffness data calculation blocks form a loop. This reflects how the system transforms each force reading into stiffness data and uses this to reason about where to poke next (tumors are faked in simulation, so force feedback is not necessary). After an adequate amount of stiffness data has been accumulated, the stiffness data is overlaid onto the 3D point cloud outputted by the data processing & organ modeling subsystem. The stiffness map together with the point cloud are the final output of the entire subsystem. These are rendered on the surgeon’s camera feed.<\/p>\n

    <\/h3>\n<\/div>\n\t\t<\/div><\/div>
    <\/div><\/div>
    <\/div>
    \t\t\t

    Cyberphysical Architecture<\/h2>\n
    \"\"<\/figure>\n<\/div>\n\t\t<\/div>
    \t\t\t

    The updated<\/strong> cyberphysical architecture in Fig. 2 demonstrates the functionality of our robotic surgical system in simulation, including methodology and the flow of data. The architecture is divided into 3 main subsystems: the sensing subsystem, the data processing & organ modeling subsystem, and the searching & palpating subsystem.<\/p>\n

    The sensing subsystem is responsible for taking visual and force data of the organ scene\u00a0and the region of interest. This signal information is passed in to the data processing &\u00a0organ modeling subsystem and the searching & palpating subsystem.<\/p>\n

    The data processing & organ modeling subsystem will take in raw sensor data and transform\u00a0it all into the camera frame. The liver data will be segmented from all the visual data and\u00a0passed to the organ modeling algorithms. These will find the point cloud and movement\u00a0equation of the organ and combine these into a single model of the organ. This model is\u00a0passed on to the searching & palpating subsystem.<\/p>\n

    The searching & palpating subsystem will perform a search on this model with the aim of finding regions of high stiffness which correspond to embedded tumors. The search algorithm will calculate a location to poke and direct the force sensor to palpate there (A force sensor was not developed in simulation, hence tumors were arbitrarily assigned on the liver’s surface, and the palpation procedure was a sampling-based process). After converting these force readings into stiffness data, the search algorithm will make an informed decision about where to poke next to gain the most information. Once the poking procedure is complete, the stiffness information will be overlaid onto the 3D model, and these will be rendered onto the surgeon’s camera feed.<\/p>\n

    <\/h3>\n<\/div>\n\t\t<\/div><\/div>
    <\/div><\/div>
    <\/div>
    \t\t\t

    System Design Description<\/h2>\n
    \"\"

    Figure 3: Depiction of the Chopsticks Surgical System<\/p><\/div>\n

    The system is composed of four main subsystems: sensors, motion simulation platform, data processing & 3D modeling, and searching & palpating. The section will elucidate each subsystem and propose alternatives, where applicable.<\/div>\n<\/div>\n\t\t<\/div>
    \t\t\t

    Sensors Subsystem<\/h3>\n
    The purpose of the sensors subsystem is to provide information to obtain the 3D point cloud, the stiffness heat map, and the movement information. It comprises the Blaser scanner, a force sensor, and a stereo camera. This was determined during subsystem trade studies. The goal of this subsystem is to capture sufficient data for the data processing & 3D modelling subsystem to estimate organ motion.\u00a0 The stereo camera will image the phantom liver from two views and aid in segmenting the liver from the rest of the image. The Blaser will 3D-scan the segmented liver region to obtain its corresponding point cloud. and the force sensor will be used to palpate the liver and obtain a stiffness map.\u00a0Figure 4 depicts the primary sensors of the system.<\/div>\n
    \n
    \n
    \n\"\" \"\" \"\"
    \nFigure 4: Main sensors of the Chopsticks Surgical System\n<\/div>\n<\/div>\n
    <\/div>\n<\/div>\n<\/div>\n\t\t<\/div>
    \t\t\t

    Movement Simulation Platform<\/h3>\n

    The purpose of the Movement Simulation Platform(MSP) is to simulate the motion pattern of the liver due to the patient\u2019s respiration and heartbeat rhythm during surgery.\u00a0It was built based on Professor Ken Goldberg\u2019s <\/span>Stewart Platform Research Kit for simulating organ motion for surgical robot prototypes.\u00a0<\/span>The MSP subsystem, which is a combination of software and hardware,\u00a0has 6 degrees of freedom (DOF) to move and can rotate in 3D space. It can exhibit a translation of 1.27 cm along each of the x<\/em>, y<\/em>, and z<\/em> axes, and a rotation of 15 degrees in roll, pitch, and yaw.<\/p>\n

    Hardware:<\/strong>
    \nThe main structure of the MSP was built in acrylic and actuated by 6\u00a0XL320 servo motors connected in series.\u00a0 The work surface was designed to be wide enough to contain the silicone models of the liver and other organs in its vicinity. The geometry of the hardware structure was developed such that it can support the weight of the silicon models above, and translate the rotational motion of the step motors into desired movements at the platform work surface.<\/p>\n

    \n\"\" \"\"
    \nFigure 5: Movement Simulation Platform of the surgical system\n<\/div>\n

    Software:
    \n<\/strong>The software interface provides an interface for the user to input different motion commands, and controls and estimates the movements of the step motors.\u00a0The MSP is programmed in C via the onboard OpenCM 9.04-C microcontroller. In order to control the servo motors connected in series,\u00a0commands are given in a form of [ServoID, Command Type, Command Goal].<\/p>\n<\/div>\n\t\t<\/div>

    \t\t\t

    Data Processing & 3D Modelling Subsystem<\/h3>\n

    The data processing and 3D modeling subsystem includes all algorithms of preprocessing before searching and palpating the organ, including: frame registration algorithms, 3D point cloud reconstruction algorithms and organ motion estimation algorithms. The subsystem takes points from laser scanner, images from stereo camera and points from robot markers as input, and outputs a moving 3D model of the organ in the camera reference frame for further processing by the searching and palpating system. The algorithms within this subsystem are explained below.<\/p>\n

    Frame Registration Algorithms<\/h5>\n

    Registration algorithms were used to get full knowledge of transformations between robot and laser, camera and robot, so that inputs from different sensors can be fused in the same reference frame and all the final outputs will be visualized in camera frame, which can then be displayed to the surgeon on the visual feed. The stereo reconstruction procedure used the least-square error method to get the transformation between the stereo\u2019s left camera and the right camera.\u00a0 The force sensor used to get the stiffness map is attached to the robot arm, and so all the force readings are represented in the robot reference frame. Since the final visualization will be in the camera frame, Horn\u2019s method was used to do camera-robot registration. Laser-robot registration was performed using bundle adjustment.<\/p>\n

    3D Point Cloud Reconstruction Algorithms<\/h5>\n

    This algorithm uses the transform between the Blaser-frame and the robot frame to stitch 2D line scans into 3D point clouds. The Blaser is mounted on PSM1 of the dVRK and moved around in a raster scan-like fashion to capture the profile of the phantom liver.<\/p>\n

    Segmentation Algorithms<\/h5>\n

    The segmentation algorithm segments out phantom liver pixels from the stereo camera image of all visible abdominal organs such as the liver, kidneys, stomach, etc. This information is used to 3D-scan the segmented liver region. We use simple HSV segmentation along with foreground segmentation.<\/p>\n

    Motion Estimation Algorithms<\/h5>\n

    The motion estimation algorithms will estimate the direction along which the organ exhibits the most movement, as well as the frequency of this movement using the visual data from the stereo and laser over time. The former was estimated using Principal Component Analysis (PCA) and the latter using Fast Fourier Transform (FFT).<\/p>\n<\/div>\n\t\t<\/div>

    \t\t\t

    Searching & Palpating Subsystem<\/h3>\n

    This subsystem includes all the algorithms necessary to take a 3D model of an organ and output a stiffness map. This includes algorithms that transform the data to the correct form, search over the organ to detect tumors, control robot kinematics, and render the stiffness map.<\/p>\n

    Searchable Surface Transformation<\/h5>\n

    There will be two main transformation algorithms. The first algorithm will take the 3D point cloud returned by the processing subsystem and convert it to a 3D searchable surface. This surface should be compatible with the current 3D Gaussian Process search algorithm developed by the lab.<\/p>\n

    Alternatives: We may instead convert the 3D point cloud into a 2D projection and do a 2D search on this surface. This simplifies the problem to a 2D search, which means it would not have to be compatible with the Biorobotics Lab\u2019s original algorithm.<\/p>\n

    Force Reading Conversion<\/h5>\n

    The second transformation algorithm will convert the force readings from the Blaser frame to the stereo camera frame. Since the 3D point cloud is in the stereo camera, this transformation is important to ensure the search algorithm correctly decides where to poke next in the stereo frame based on the force readings. The transformation was previously calculated in the data processing subsystem.<\/p>\n

    Force Reading to Stiffness Data<\/h5>\n

    This algorithm will convert the readings from the force sensor into a Multivariate Gaussian Distribution\u00a0 that models the stiffness of regions of the organ based on the magnitude of the force readings. This means we will not have to poke the organ in as many locations because the readings can be generalized over the surface. This algorithm has been implemented by the lab so we do not anticipate needing an alternative. The force readings will be converted to stiffness data that indicate both the shape and location of tumors in the organ, and overlaid on the visual feed for the surgeon to see, as depicted in Fig. 3.<\/p>\n

    Search Over the Organ<\/h5>\n

    The search algorithm\u00a0 will use active LSE to determine where the robot should poke the organ for maximum information gain. It will take the searchable surface and output coordinates in the robot frame of where to poke next.<\/p>\n

    Alternatives: Instead of intelligently determining where to poke the organ, the organ can be divided into a meshgrid and we can direct the robot to poke within each box of the grid in a brute-force manner.<\/p>\n

    Inverse Robot Kinematics<\/h5>\n

    The dVRK already has PID control and robot kinematics implemented. The current robot kinematics allow the robot to move into \u201cunreachable positions\u201d from which the robot cannot recover. We will develop a helper algorithm on top of the robot kinematics to ensure that the robot does not move into these positions by checking if the desired position and orientation are within the operating range of the dVRK.<\/p>\n

    Display<\/h5>\n
    \"\"

    Figure 6: Stiffness map overlaid on phantom organ<\/p><\/div>\n

    Given a stiffness map, the Biorobotics Lab at CMU has implemented an algorithm to render it into stereo images. We will have to ensure that the stiffness map aligns with the moving organ. This will be done with rendering software with some preprocessing.<\/p>\n

    This subsystem works a little differently in simulation because we did not simulate a stereo camera. Once the robot completes the palpation procedure, a visualization of the results as shown in Figure 7 appears for the user\/surgeon by switching on two ROS topics. The first topic publishes the shape and location of the ground truth tumor, shown in green, while the second topic publishes palpated points in white. During development, we overlay both visualizations on the liver to get an understanding of what percentage of the tumor(s) was classified correctly. The minimum bounding circle shown in yellow is sketched offline to show how we calculated our accuracy percentages.
    \n

    \"\"

    Figure 7: Stiffness map in simulation<\/p><\/div><\/p>\n<\/div>\n\t\t<\/div><\/div>

    <\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"

    Mandatory System Requirements Mandatory Performance Requirements The system will: Register camera-robot coordinate frames with a 10 mm maximum root mean-square-error (RMSE) Register Blaser-robot coordinate frames with a 10 pixel unit maximum reprojection error 3D-scan the surface of phantom liver in under 5 minutes Segment the liver, including near organ occlusions, at 95% IOU Estimate the
    + Read More<\/a><\/p>\n","protected":false},"author":182,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-155","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/pages\/155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/users\/182"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/comments?post=155"}],"version-history":[{"count":148,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/pages\/155\/revisions"}],"predecessor-version":[{"id":860,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/pages\/155\/revisions\/860"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teama\/wp-json\/wp\/v2\/media?parent=155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}