Enhanced vine pruning thanks to artificial intelligence and augmented reality
Vine pruning is a complex procedure that requires years of knowledge and experience. Poor pruning can have catastrophic consequences for the health and productivity of the vine. To address the shortage of skilled and knowledgeable pruners, the startup 3D2cut SA, located in Valais, Switzerland, asked Idiap to develop a system based on artificial intelligence and integrate it into augmented reality goggles. The resulting device identifies the different structural parts of each individual vine plant and as a result of this information recommends precise cutting points. This work has been made possible thanks to the company’s founders who are world-renowned master pruners, in addition to support from the Ark, a foundation for innovation based in Valais.
“Up to 1/5th of the world's vines are not productive. The International Organization of Vine and Wine identified wood diseases of the stump as the main cause of this problem and pruning as the main means of prevention,” Massimo Guidici, co-founder of 3D2cut, explains. To ensure preventive "soft pruning”, augmented reality glasses can visually pinpoint and recommend cutting points on the vine. “To achieve this, a deep understanding of the vine and the pruning decisions for each plant is necessary,” Marco Simonit, master pruner and co-founder of 3D2cut, explains. It is therefore essential that the rules laid down by vineyard experts are followed.
Result obtained by the visual system developed by Idiap applied to an image of a vine. Circles highlight nodes and buds for each branch, and red lines represent the predicted structure of the vine. The various colors seen on the nodes or buds of each branch characterize the different parts of the vine predicted by the system (trunk, spur, cane, shoots, etc.)
How does it work?
The visual system developed at Idiap allows identification of the structure of a vine plant by identifying the nodes (buds) as well as the vine branches and their type. “To achieve this, we drew inspiration from the technologies used to detect human posture. Indeed, instead of predicting in an image both the key points of a human body, such as joints—wrist, elbow, knee—and their connections—arms, thighs, torso—, the device detects nodes on vine plants as well as the branches that connect them. It also determines the type of branches: trunk, spur, rods, shoots and in general, all the necessary information on the vine to allow an expert system to recommend where the pruning cuts should be made,” Jean-Marc Odobez, Head of the Perception & Activity Understanding group at Idiap, explains.
Node and branch detection is carried out by a neural network specially designed for this task, and trained on vine images annotated by specialists. The final structure of the plant is extracted from this detection thanks to a new algorithm based on the research of a path of least resistivity of a graph. This method allows to adapt to the number of nodes and connections, unlike what is used for humans for whom the articulation structure is fixed. This work has been published in the Computers & Electronics in Agriculture journal. The images and annotations of the vines created for this project have been made accessible in open access for non-commercial use.
A promising tool
A first version has been integrated and deployed by 3D2cut on a tablet and tested on real-life shots. For the experts, results are very encouraging. To offer reliable and efficient technology, many challenges remain. The vision system will have to work in different conditions and take into account aspects such as variations in luminosity, a wide variety of backgrounds including other vines, and different viewing angles. Above all, the system must be able to process images very quickly on on-board equipment.
“Cooperating with Idiap has been invaluable. We have the technical knowledge regarding vines and the ability to collect large quantities of data, researchers bring us the necessary expertise in AI to quickly reach a solution,” Jérôme Corre, CTO of 3D2cut, explains. This partnership with the institute could prove even more fruitful in the future. “Once we get a proven product for vines, this system could be applied to other fruit trees requiring pruning, such as apple, pear, orange, and even cocoa and coffee trees,” Henrico Dolfing, co-founder of 3D2cut, concludes.
More information
- Scientific paper
- Vines images database and annotations used to train the model
- 3D2cut SA