Abstract
Interest in agricultural robotics has significantly increased due to its potential to increase productivity and reduce labor in farming. However, developing robotic systems for fruit harvesting in unstructured environments presents considerable challenges, particularly regarding robot perception, planning, and manipulation.To address these challenges, we propose a dual-arm robot that integrates perception and control for fruit detection and harvesting. The robot, equipped with an RGB-D camera and with tools for cutting and collecting fruit, uses a Hierarchical Quadratic Programming (HQP) control strategy to move in such complex environments while obeying constraints such as joint limits, self-collisions, and potential collisions with both the fruit and tree. By combining deep learning with traditional image processing, our approach enables accurate detection and tracking of fruits and tree trunks in controlled environments. These methods are validated through real-world testing on RGB-D images and simulations, demonstrating the robustness and reliability of the system.Building on this, our research extends to optimize the efficiency and adaptability of dual-arm robots through global planning and scheduling algorithms. To cope with the dynamic and unpredictable nature of agricultural settings, we implement a scheduling algorithm based on the Traveling Salesperson Problem (TSP) to minimize the distance required to harvest all visible fruits. Global motion planning ensures collision-free operation, increasing both safety and efficiency. Our methodology is validated using two distinct robots: Baxter, a cost-effective and widely available robot, and BAZAR, a custom-built, high-precision dual-arm platform. Real-world experiments conducted on apples and oranges illustrate the adaptability of our methods.Additionally, our work tackles the specific challenges caused by occlusions during fruit harvesting, occlusions which can lead to inaccurate localization or impede the robot from picking the fruit. We developed a method to identify these occlusions using a combination of deep learning and traditional image processing techniques, then clear them by having the robot arm push the branch responsible for the occlusion. Our setup uses an RGB-D camera and a single robot arm to detect partially occluded fruits and estimate their full appearance with a generative neural network. A novel 3D extension of the 2D Hough transform detects tree branches in the point cloud, allowing the robot to identify which branches cause occlusions and to strategically push them aside. This technique has been tested across different fruit types and lighting conditions, proving its effectiveness in increasing visibility and clearing obstructions, thus ensuring successful fruit harvesting.By integrating these three strategies: coordinating dual-arm robots for robust harvesting, optimizing planning and scheduling for efficiency, and dynamically handling occlusions for accuracy, our research presents a comprehensive solution to the complexities of robotic fruit harvesting in unstructured environments.