Evaluation of methods to improve synthetic training images for semantic segmentation of the environment from UAV perspective

Tushar REWATKAR

Master's thesis

Abstract:

This thesis addresses the significant challenge of the Simulation to Real (sim-to-real) gap in Unmanned aerial vehicles (UAVs) based semantic segmentation tasks. Autonomous flight for UAVs requires robust environmental perception, often achieved through neural networks trained on diverse datasets. However, gathering and annotating real-world aviation data is challenging and resource intensive. To mitigate this, synthetic datasets generated by advanced game engines have been employed as an alternative. Despite their potential, these synthetic datasets introduce a gap in model performance when applied to real world scenarios due to differences in textures, lighting, and visual characteristics or maybe some unknown characteristics. This research explores three primary approaches to bridge this sim-to-real gap. The first approach investigates the impact of basic image augmentation techniques such as brightness, contrast, saturation, and RGB value adjustments on model performance. The second approach refines these augmentations by aligning synthetic images more closely with real images using mean-based adjustments. The third and most advanced approach employs various Image-to-Image (I2I) translation methods, to transform synthetic images into more realistic ones. The evaluation of these approaches is conducted using semantic segmentation models trained on augmented and translated datasets, with their performance assessed against real-world data. Metric such as Mean Intersection Over Union (mIoU) are used to measure effectiveness. The results demonstrate that while basic augmentations provide minimal improvements, I2I translation methods, particularly single-image techniques, significantly reduce the sim-to-real gap, enhancing model robustness and performance. The findings of this thesis suggest that advanced image translation methods are essential for improving the applicability of synthetic datasets in real-world applications, providing a foundation for future research and development in this field.

Abstract:

This thesis addresses the significant challenge of the Simulation to Real (sim-to-real) gap in Unmanned aerial vehicles (UAVs) based semantic segmentation tasks. Autonomous flight for UAVs requires robust environmental perception, often achieved through neural networks trained on diverse datasets. However, gathering and annotating real-world aviation data is challenging and resource intensive. To mitigate this, synthetic datasets generated by advanced game engines have been employed as an alternative. Despite their potential, these synthetic datasets introduce a gap in model performance when applied to real world scenarios due to differences in textures, lighting, and visual characteristics or maybe some unknown characteristics. This research explores three primary approaches to bridge this sim-to-real gap. The first approach investigates the impact of basic image augmentation techniques such as brightness, contrast, saturation, and RGB value adjustments on model performance. The second approach refines these augmentations by aligning synthetic images more closely with real images using mean-based adjustments. The third and most advanced approach employs various Image-to-Image (I2I) translation methods, to transform synthetic images into more realistic ones. The evaluation of these approaches is conducted using semantic segmentation models trained on augmented and translated datasets, with their performance assessed against real-world data. Metric such as Mean Intersection Over Union (mIoU) are used to measure effectiveness. The results demonstrate that while basic augmentations provide minimal improvements, I2I translation methods, particularly single-image techniques, significantly reduce the sim-to-real gap, enhancing model robustness and performance. The findings of this thesis suggest that advanced image translation methods are essential for improving the applicability of synthetic datasets in real-world applications, providing a foundation for future research and development in this field.

Keywords

Artificial intelligence, deep learning, image processing, object detection, pixel-level integration, Split and Merge, generative adversarial networks (GANs), CycleGAN, CUT, SinCUT, SinMCL, Pix2PixHD, real-time perception, dataset generation, feature extraction, model training, evaluation metrics, performance improvement, UAV applications, environmental monitoring, vision-based navigation, computational efficiency, model robustness, texture transformation, image generation.

Language used: English

Date on which the thesis was submitted / produced: 11. 2. 2025

Thesis defence

• Supervisor: prof. Dr. Christina Bauer

UNIVERSITY OF SOUTH BOHEMIA IN ČESKÉ BUDĚJOVICE

Faculty of Science

Master programme / field:
Artificial Intelligence and Data Science / Artificial Intelligence and Data Science