Weed Image Augmentation by ControlNet-Added Stable Diffusion

Robust weed recognition relies on curating large-scale, diverse datasets, which are, however, practically difficult to come by. Deep generative modeling has received widespread attention in synthesizing visually realistic images beneficial for wide-ranging applications. This study investigates the efficacy of state-of-the-art deep learning-based diffusion models as an image augmentation technique for synthesizing weed images towards enhanced weed detection performance. A 10-weed-class dataset was created as a testbed for image generation and weed detection tasks. A ControlNet-added Stable Diffusion model was trained to generate weed images with broad intra-class variations of targeted weed species and diverse backgrounds to adapt to changing field conditions. The quality of generated images was assessed using metrics including the Fréchet Inception Distance and Inception Score. The generated images had an average FID score of 0.98 and an IS score of 3.63. YOLOv8l was trained for weed detection. Combining the generated with real images yielded consistent improvements (1.2-1.4% mAP@50:95) in weed detection, compared to modeling using only real images. Further research is needed to exploit controllable diffusion models for generating high-fidelity, diverse weed images and enhancing multi-class weed detection.