Machine learning algorithm for strawberry fruit recognition and classification of their maturity level

The paper presents the results of research on the development of an algorithm based on a modern deep learning Convolutional Neural Networks model, which improved the accuracy of strawberry fruit recognition and classification of the degree of removable ripeness. The research utilized transfer learning method to adapt the used YOLOv10-M (You Only Look Once version 10 medium) model, originally trained on COCO dataset, to solve the task of strawberry berry ripeness degree classification. Image annotation (markup) was performed to create a training dataset. The Supervisely service was used, which allowed assigning appropriate classes to the selected areas of interest using rectangular frames. As a result of image annotation, 3 classes of garden strawberries of different maturity levels were identified: immature berry (class "unripe_strawberry"), mature berry (class "ripe_strawberry") and unripe berry (class "half_ripe_strawberry"). The unripe berry class included fruits that had less than 60% of the total red area in the image. Augmentation of the dataset, including operations such as cropping, resizing, rotation, vertical reflection, blurring, contrast variation, noise addition, random color correction, and addition of weather effects, increased the sample size to 4500 images. The YOLOv10-M model was trained on the created data sample, 500 epochs were used, the batch size was set to 8 (batch size). Stochastic Gradient Descent (SGD) with an initial learning rate of 0.01 (learning rate) was chosen as the optimization algorithm. Analysis of the graphs and metrics of binary and multiclass classification to evaluate the quality of the model allowed us to determine the optimal settings and to select a confidence threshold (0.7) that achieves the best balance between accuracy (precision metric 0.93) and completeness (recall metric 0.89). The mean absolute percentage error (MAPE) of recognition of the test sample images for all classes was 3.4%. The greatest difficulty in recognition occurred with the unripe berry class, for which the mean absolute percentage error (MAPE) was 4.6%.

1. Utkov Yu.A., Filippov R.A., Hort D.O., Kutyrev A.I. The development of strawberry berries harvesting machines in Russia. Istoriya nauki i tekhniki = History of Science and Engineering, 2020, no. 8, pp. 58–76. (In Russian). DOI: 10.25791/intstg.08.2020.1207.

2. Woo S., Uyeh D.D., Kim J., Kim Y., Kang S., Kim K.C., Lee S.Y., Ha Y., Lee W.S. Analyses of Work Efficiency of a Strawberry-Harvesting Robot in an Automated Greenhouse. Agronomy, 2020, no. 10, p. 1751. DOI: 10.3390/agronomy10111751.

3. Kurpaska S., Bielecki A., Sobol Z., Bielecka M., Habrat M., Śmigielski P. The Concept of the Constructional Solution of the Working Section of a Robot for Harvesting Strawberries. Sensors, 2021, no. 21, p. 3933. DOI: 10.3390/s21113933.

4. Cao L., Chen Y., Jin Q. Lightweight Strawberry Instance Segmentation on Low-Power Devices for Picking Robots. Electronics, 2023, no. 12, p. 3145. DOI: 10.3390/electronics12143145.

5. Khort D.O., Maistrenko N.A., Tereshin A.N., Vershinin R.V. Studying the methods of harvesting garden strawberry with robotic machines. Sel'skokhozyajstvennye mashiny i tekhnologii = Agricultural Machinery and Technologies, 2020, no. 14 (1), pp. 27–33. (In Russian). DOI: 10.22314/2073-7599-2020-14-1-27-33.

6. Khort D.O., Kutyrev A.I., Smirnov I.G., Moiseev G.V., Solovyov V.I. Agricultural autonomous robotic platform motion control. Sel'skokhozyajstvennye mashiny i tekhnologii = Agricultural Machinery and Technologies, 2023, no. 17 (1), pp. 25–34. (In Russian). DOI: 10.22314/2073-7599-2023-17-1-25-34.

7. Xiong Y., Peng C., Grimstad L., From P.J., Isler V. Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper. Computers and electronics in agriculture, 2019, vol. 157, pp. 392–402. DOI: 10.1016/j.compag.2019.01.009.

8. Li Y., Wang W., Guo X., Wang X., Liu Y., Wang D. Recognition and Positioning of Strawberries Based on Improved YOLOv7 and RGB-D Sensing. Agriculture, 2024, no. 14, p. 624. DOI: 10.3390/agriculture14040624.

9. Tao Z., Li K., Rao Y., Li W., Zhu J. Strawberry Maturity Recognition Based on Improved YOLOv5. Agronomy, 2024, no. 14, p. 460. DOI: 10.3390/agronomy14030460.

10. Cai C., Tan J., Zhang P., Ye Y., Zhang J. Determining Strawberries’ Varying Maturity Le-vels by Utilizing Image Segmentation Methods of Improved DeepLabV3+. Agronomy, 2022, no. 12, p. 1875. DOI: 10.3390/agronomy12081875.

11. Guo Z., Hu X., Zhao B., Wang H., Ma X. StrawSnake: A Real-Time Strawberry Instance Segmentation Network Based on the Contour Learning Approach. Electronics, 2024, no. 13, p. 3103. DOI: 10.3390/electronics13163103.

12. Zhang Y., Zhang L., Yu H., Guo Z., Zhang R., Zhou X. Research on the Strawberry Recognition Algorithm Based on Deep Learning. Applied Sciences, 2023, no. 13, p. 11298. DOI: 10.3390/app132011298.

13. Yu Y., Zhang K., Yang L., Zhang D. Fruit detection for strawberry harvesting robot in non-structural environment based on MaskRCNN. Computers and Electronics in Agriculture, 2019, no. 163, p. 104846. DOI: 10.1016/j.compag.2019.06.001.

14. Ge Y., Xiong Y., Tenorio G.L. Fruit Localization and Environment Perception for Strawberry Harvesting Robots. IEEE Access, 2019, vol. 7, pp. 147642–147652. DOI: 10.1109/ACCESS.2019.2946369.

15. Wang C., Wang H., Han Q., Zhang Z., Kong D., Zou X. Strawberry Detection and Ripeness Classification Using YOLOv8+ Model and Image Processing Method. Agriculture, 2024, no. 14 (5), p. 751. DOI: 10.3390/agriculture14050751.

16. Amraoui K., Ansari M., Lghoul M., Alaoui M., Abanay A., Jabri B., Masmoudi L., Valente de Oliveira J. Embedding a Real-Time Strawberry Detection Model into a Pesticide-Spraying Mobile Robot for Greenhouse Operation. Applied Sciences, 2024, no. 14 (16), p. 7195. DOI: 10.3390/app14167195.

17. Terven J., Córdova-Esparza D.-M., RomeroGonzález J.-A. A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS. Machine Learning and Knowledge Extraction, 2023, no. 5 (4), pp. 1680–1716. DOI: 10.3390/make5040083.

18. Hussain M., Khanam R. In-Depth Review of YOLOv1 to YOLOv10 Variants for Enhanced Photovoltaic Defect Detection. Solar, 2024, no. 4, pp. 351–386. DOI: 10.3390/solar4030016.

19. Iman M., Arabnia H.R., Rasheed K. A Review of Deep Transfer Learning and Recent Advancements. Technologies, 2023, no. 11, p. 40. DOI: 10.3390/technologies11020040.

20. Maxwell A.E., Warner T.A., Guillén L.A. Accuracy Assessment in Convolutional Neural Network-Based Deep Learning Remote Sensing Studies. Part 1: Literature Review. Remote Sensing, 2021, no. 13 (13), p. 2450. DOI: 10.3390/rs13132450.