Brinkmanship

When we began our MRSD project, brinkmanship was a concept that had just begun to be explored within the MoonRanger organization. Another student named Neil Khera had written a paper on the subject and had developed some code to use a RealSense depth camera to detect brinks and steep slopes.

We knew that the rover that would be sent to the moon would not have a depth sensor installed, so we needed to develop the same brinkmanship functionality using only image data. In addition, we needed to integrate brinkmanship with our navigation and locomotion code, so that our rover could adjust its behavior when it detected the pit edge.

02/14/2020 – Stereo Feature Matching and Reconstruction

We were able to use the stereo data from the realsense camera and perform feature matching. We then generated a 3D point cloud that the existing brinkmanship code could accept as an input. Thus, we were able to develop a pipeline that could detect pit edges by taking in stereo image data.

03/19/2020 – Dense Reconstruction Using a Depthmap (SGBM OpenCV)

The feature matching performed initially generated a sparse point cloud which was not of much use. Hence, we switched to generating a depthmap to obtain a dense point cloud.

03/22/2020 – Completing the Stereo Brinkanship Pipeline

Converted the point cloud to the correct format and passed it through the brinkmanship code. The results can be seen below.

04/03/2020 – Field Test for Data Acquisition

In order for this pipeline to assist the rover in navigating its surroundings, we need to identify the characteristics of safe and unsafe terrain, and tune the brinkmanship system so that it can classify the rover’s surroundings appropriately. We scouted some locations in Schenley Park which had good examples of safe and unsafe terrain, including small cliffs and slopes of varying steepness. We secured our rover with a rope, to prevent it from being damaged, and then drove the rover over these cliffs and slopes while recording odometry and camera data. From this data, we will be able to characterize the different types of terrain as perceived by the brinkmanship system, and adjust our thresholds accordingly so that the rover accurately detects dangerous slopes and brinks.

In order to gather information about the rover’s ability to detect edges, we needed to test in the field with a rover surrogate. A rover called Blue had previously been built to provide a testing platform for the MoonRanger and PitRanger platforms, but we did not have consistent access to that rover. We chose to build a copy of Blue (“Blue 2”) that we could use for our own purposes. The COVID-19 pandemic and resulting quarantine of the CMU campus slowed our progress in assembling Blue 2, but we were able to order additional parts and assemble Blue 2 in time for our field test on April 3. We were able to use the teleoperation code that had been developed for Blue to operate Blue 2 during the test.

04/08/2020 – Developed a Simple Heuristic For Brinkmanship

After acquiring a significant amount of data, we converted all of it to a usable format. We then proceeded with the development of an algorithm that would allow the rover to stop right in front of an edge. We integrated all our previously developed code in this process. The brinkmanship code developed earlier was not used in this case.

04/19/2020 – Tested Integrated Code in Second Field Test

The camera operation subsystem and the brinkmanship subsystem were successfully integrated and tested. Multiple tests were performed with different camera positions and different thresholds used in our heuristic. The rover stopped right in front of the edge almost all the time. After the robot stopped, it captured multiple images of the surrounding at different angles. These were used later to generate a panorama.

09/16/2020 – Improved Brinkmanship Heuristics

Prior to this date, the heuristic used to detect edges only counted the total number of points in the point clouds generated from the stereo camera images. The terrain on the lunar surface is varied enough that this heuristic would not be suitable for an actual mission. The team worked together to come up with a more robust heuristic that could be implemented.

The solution that the team arrived at was inspired by the way 3D environments are represented in video games and other virtual depictions. These computer-generated environments are composed of meshes of many small polygons, usually triangles, which approximate complex terrain. We reasoned that if we could convert our point clouds into triangular meshes, we could then determine the surface normals of each polygon in the mesh and use those vectors to distinguish between safe and unsafe terrain. Furthermore, gaps in the mesh could be used to indicate the presence of sheer brinks.

10/08/2020 – First Successful Point Cloud to Mesh Conversion

On this date, we were able to transform a point cloud generated from the simulated rover’s perception of its lunar environment into a triangular mesh. To accomplish this, we used the python library PyVista, which provides tools for manipulating and displaying point clouds and meshes. The point cloud was collected using the depth sensor attached to the simulated rover, and shows a swath of the lunar surface in front of the rover’s current position. This raw point cloud was very dense, which caused the conversion process to take a long time to complete. Awadhut developed a procedure for creating a more sparse point cloud by clustering the points in the original cloud together. This sparse point cloud could then be converted quickly into a triangular mesh. Visual inspection of the mesh showed that its shape matched that of the original surface. Additional tests with more varied terrain will be necessary to prove that this approach is truly suitable for our needs.

10/14/2020 – Updated Brinkmanship In Sim To Use Generated Meshes.

Once the mesh was generated, the surface normals of triangles in the mesh were used to determine if there is the presence of sloped terrain near the rover. We used the following criteria for performing brinkmanship with the mesh data.

1. If the number of triangles for which the normal is off from the vertical axis by a fixed value is above a certain threshold, we conclude that the terrain in front of the rover has an unsafe slope. Both the angle from the vertical and the number of meshes used to make this decision can be adjusted easily to suit different conditions.
2. The mesh is also used to locate brinks which may represent the edge of a pit. A pit edge is said to be detected when there is an absence of terrain near the rover which is reflected in the generated triangle mesh. The mesh is divided into a fixed number of cells and we conclude that the rover has reached near a shear brink when the cells which are close to the rover become empty. The brinkmanship subsystem will trigger an alert signal in such a case and prevent the rover from getting too close to the pit edge.

10/28/2020 – Generated Triangle Mesh From Real World Point Cloud Data

We were able to test our system developed in sim on real-world data around the time of our tenth progress review. The point cloud generated from stereo images of the real world was noisier than the one obtained in simulation. We had to process and filter the raw cloud to remove outliers before generated a triangle mesh to obtain the best results.

11/08/2020 – Tested Full System At Gascola

We tested our rover at two different sites at Gascola. We conducted around 5 straight tests at the first site where we met our success criteria related to the rover’s distance from the edge. Additionally, none of these tests had the occurrence of a mission failing incident. At the second site, we conducted at least 10 tests which again met the criteria related to the rover distance and mission failure. Images from our visit to Gascola are shown below.