Robotic Arm for Fruit Picking
Bio-Robotics
Dr. Sunny Jung
Individual
August 2022 - December 2022
Skills:
-
Embedded Systems
-
Inverse kinematics
-
Servo motors
-
Computer vision (TensorFlow, openCV2)
-
Robot arm manipulation
Overview
The final project for the Bio-Robotics class was to use a 4 degree-of-freedom (DoF) robotic arm and a camera to identify where a cotton ball was in front of the robot, move to it, and grab it. If our robot worked, we received an A. If it did not work, we did not receive an A.
Project
The job given to us was to have our robot arm grab a cotton ball while avoiding all of the green ping pong balls. The purpose was to have us simulate building a robot which could pick fruit while detecting ripe fruit from rotten fruit.
There were two components to the task: developing a machine learning model which could classify cotton balls, ping pong balls, and dowels, and actually coding the arm to move to the cotton balls.
Machine Learning
The machine learning model was a neural network. I didn't have to actually build the network from scratch; we used opencv2 and TensorFlow, two Python libraries, to create our model. I used the EfficientDet_Lite0 model in TensorFlow as a classifier. We then trained this model on a multitude of images. Fortunately, we did not have to be able to figure out how far the cotton balls were from our camera as we did not have RealSense cameras, so we only had to figure out the x and y of the cotton balls (in relation to the origin of our robot arm, which I defined as the base)
Robotic Arm
For the robotic arm, I had to solve the inverse kinematics of the system to know how to move the robot. Due to our robot arm only having two links, I could use a numerical solution and it would not be too complicated. I solved for the rotation matrices, then used the inverses to convert my desired end location into joint angles (of which I only had two). To avoid redundancies, I limited it so that my end effector (the robotic gripper) was always lower than the elbow joint between the two links so that the robot gripper would always point downwards.
Solution:
After I had solved the inverse kinematics of the system and developed my machine learning model, I had to combine it all. This involved a lot of debugging and troubleshooting. The first issue was calibrating our camera and figuring out its field of view and how wide/tall it could see. Another big issue I had was how I would actually move my gripper to grab the cotton ball and not knock it out. In the end, I decided that the easiest way to grab the cotton ball was to move to the side of the cotton ball and grab it from the side. I ended up completing the task, as shown in the video below, and receiving my A!