System Performance

Spring Validation Experiment

Test Results

During our Spring Validation Experiment, we tested most of the key subsystems and their requirements.  Our experiments were for the most part successful, with reliability being the only metric that succeeded in all tests but was not shown in the SVE.

Requirements Expectations FVE FVE-ENCORE
MP1.1, MP1.2, MP1.3 Docking platform shall move according to the given input frequency in Z-direction Successful

Dock changed to given frequency as detected by IR

Successful
MP1.4 Docking platform shall have a locking mechanism which supports weight of 5 kg Successful

Dock held quadcopter for 30 seconds without external support

Successful
MP1.2 Sensor gathers data from the motion of the platform and outputs the frequency within an accuracy of 0.05 Hz Successful

 

Successful
MP2.1 Quadcopter shall localize w.r.t. platform within 50mm accuracy Successful

Quadcopter maintained XY centered hover with a distance of less than 50mm between center of quadcopter and center of platform

Successful
MP2.2 Quadcopter shall dock to the platform autonomously and without colliding within 10 minutes Successful

Quadcopter docked 4 times and rejected once, with an average relative velocity of 38 cm/s and an average time of TIME

 

Successful

 

MP2.2 Quadcopter shall dock to the platform autonomously and without colliding in 80% of tests Not Successful

Quadcopter docked 4 times and rejected once out of 6 attempts, a rate of 67%

However, exhaustive testing outside of demo showed a failure rate of 3 out of 30, 90%

 

Not Successful

 

DP2.1 Quadcopter shall localize w.r.t. platform within 30mm accuracy Sucessful Sucessful
DP2.2. Quadcopter shall dock to the platform within 5 minutes Successful

Docking occurred in an average of TIME

Successful

Table 1 – SVE Results with FVE goals included

Fall Validation Experiment

Test Results

During our Fall Validation Experiment, we tested most of the key subsystems and their requirements.  Our experiments were for the most part successful, with navigation being the only subsystem that needed to have further development scheduled for the Spring semester in addition to all other planned implementation.

REQUIREMENTS EXPECTATIONS FVE FVE-ENCORE
MFR1.1 Docking platform shall move according to the given input frequency Successful (0.15 to 0.3 achieved with 10% error)
Successful (added qualitative functionality, no quantitative improvement)
MFR2 Sensor gathers data from the motion of the platform and outputs the frequency Successful (output with less than 5% error)
Successful (no change)
MFR2.1 Quadcopter shall be able to autonomously hover Successful (quadcopter successfully hovered for 20 seconds)
Successful (simulated, quadcopter unavailable during encore)
MFR2.2 Quadcopter shall move autonomously from point A to point B Failure due to lack of spare parts Failure due to lack of spare parts
MFR2.1 AprilTag should detect the camera and ascertain the distance moved within 5% error Successful (AprilTag system output within 5% error)
Successful (AprilTag system output within 5% error)
MFR2.1 AprilTag should detect the camera moving and therefore make the quadcopter move accordingly in the simulation Not planned initially Successful (simulated quadcopter moved correct amount based on motion of April Tag)

Table 2 – FVE Results

Strengths and Weaknesses

Our system and subsystems are also for the most part acceptable.  Our strengths are in the robustness of our dock and vision, and most of our weaknesses are solvable with further improvement to control of the quadcopter.  Our dock’s robustness, however, comes at the cost of being large and heavy, necessitating special arrangements to find a location for it that will facilitate testing and evaluation of the system as a whole.

STRENGTHS WEAKNESSES
Docking Platform is robust Velocity control not stable in Matrice
Motor can withstand the desired load Flight controller code is not accessible
April Tag works suitably even in low lighting Cannot provide state estimation values to flight controller
IMU is giving accurate readings Platform is heavy. Not able to find place to mount it permanently
Indoor hovering is stable using guidance

Table 3 – Strengths and Weaknesses of System

The motor and platform are strong enough to withstand the weight of the quadcopter after it has docked. A 100% margin was used while selecting the motor. Currently, it can withstand weight of 5 kg at 60V and the maximum take-off weight of the quadcopter is 3.4 kg. Higher torques can be achieved at higher voltages. The accuracy of IMU readings is critical to our project because these values will be transmitted to the quadcopter and used to determine the suitable moment to initiate the docking operation. The April Tag detection is working suitably in low light conditions as well as when the resolution is reduced to one-fourth of the original resolution. Also, the tag detection rate is 20 Hz in poor lighting conditions. We are able to achieve stable hover of quadcopter indoors using the Guidance. However, when we try to give velocity commands for position control, it drifts significantly. This is because the default position control in Matrice 100 is done using GPS coordinates.

Another weakness is that the flight controller’s code is not accessible to us and we cannot provide state estimation values to it. Thus, the only possible option to do position control is to provide velocity commands. The loop is closed using information provided from the IMU sensors and the Guidance’s cameras. The IMU does not provide accurate position estimates due to high noise. The Guidance uses optical flow to calculate velocity. This is effective only if there are significant non-repeating features available for tracking.

To overcome the weaknesses of our quadcopter we first need to set safe limits to yaw, pitch and roll commands. Further, we need to confirm that the manual override is functional by testing it at least ten times.  For CV we need to increase the detection rate further. Implementing Lucas-Kanade tracking should help us in achieving this. Further, using a camera with higher frame rate should also contribute to an increased detection rate.

Unit Tests

Our Unit Tests for Fall are tracked on a separate page.

Our Unit Tests for Spring are tracked in our Test Plan.