Vision-Based Control for a Vine Pruning Robot

Fadi Gebrayel PhD defense

Soutenance

18.03.26 - 18.03.26

Agriculture is a crucial global sector which faces unprecedented challenges, including rising food demand from a growing population, significant labor shortages, and the urgent need for sustainable practices. The convergence of an aging agricultural workforce, youth migration to urban areas, and climate change, places agriculture at a juncture. In response, agricultural robotics is emerging as an element of a possible solution, providing a means to automate laborious tasks and address these pressing issues. This thesis aims to improve the performance of autonomous vine pruning systems. A significant limitation in current approaches is the absence of sensor based control, which frequently results in a high incidence of task failures—where the end-effector fails to reach the designated pruning location. This research addresses this critical gap by proposing a visual feedback–driven system designed to accurately reach the cutting poses, even under vine or robot movement. Given the intricate structure of vine branches and the cluttered nature of vineyard environ- ments, an investigation is first conducted on classical methods for extracting visual descriptors (SIFT, ORB, D-SIFT, etc.) and the feasibility of fitting geometrical primitives (lines, cylinders, etc.) to be subsequently used for control. The results highlight the difficulty of the task when relying on classical methods. Then, a novel and general approach to vision-based vine pruning is proposed. It combines Iterative Closest Point (ICP) point cloud alignment with position-based visual servoing (PBVS). Four relevant ICP variants are compared within PBVS in vine pruning scenarios: standard ICP, Levenberg–Marquardt ICP, Point-to-Plane ICP, and Symmetric ICP. A dedicated ICP initial guess is incorporated to improve alignment speed and accuracy, as well as a procedure for generating reference point clouds at pruning locations. Live experiments conducted on a Franka Emika manipulator equipped with a stereo camera are reported, involving three real vines under laboratory conditions. Further, this system is augmented with an online point cloud based waypoint planner. Using a Nonlinear Model Predictive Control (NMPC) scheme, a sequence of 3D poses is computed in real time. Then, it is navigated by a sliding-reference PBVS ensuring smooth traversal without stopping. An ICP based point cloud alignment featuring a specific initial guess is incorporated for pose estimation. Qualitative and quantitative evaluations are conducted on a Franka Emika manipulator fitted with an eye-in-hand stereo camera, which performs pruning sequences on three increasingly complex vine stocks. The system robustness is validated against unexpected disturbances, including vine and robot base movement. Last, recent advances in AI based visual processing are drawn upon, so as to overcome the difficulties in the extraction, matching and tracking of 2D visual features, and enable image-based visual servoing (IBVS). Deep learning (DL) based methods are used for feature extraction and matching. With this in mind, a dedicated dataset is built in order to train DL strategies to feature matching and vine branch segmentation. The IBVS algorithm is designed so as to ensure convergence despite the absence of depth information. Results of live experiments on real vines are provided, using a Franka Emika manipulator equipped with a single RGB camera. The work concludes with a discussion and directions for future research.

published on 26.02.26