Construction sites are highly dynamic environments where misplaced tools and equipment pose risks to both autonomous robots and human workers. Reliable real-time perception is therefore essential for safe robotic operation. However, traditional object detection systems are limited to predefined categories and struggle to generalize to unseen tool variants commonly found on construction sites. This thesis proposes a two-stage framework for open-world understanding of construction tools and equipment. In the first stage, a real-time YOLOv12 detector was trained using a single-class configuration to accurately localize tools under challenging floor conditions. To support training, a synthetic dataset generation pipeline was developed that combines cropped tool images with diverse background scenes that represent a robot’s downward-facing perspective. The trained detector achieved very high performance, reaching an mAP@50 of approximately 99.5% and an mAP@50–95 of about 98.7%, with precision and recall both reaching 100% on the evaluation dataset. In the second stage, the detected regions were analyzed using a Vision-Language Model (VLM) to provide semantic interpretation and hazard reasoning. By combining fast spatial localization with high-level multimodal reasoning, the system moves beyond rigid class labels toward flexible tool identification and contextual understanding. Experimental evaluation showed that the model correctly interpreted most tools and produced meaningful safety descriptions. The proposed approach demonstrates that integrating real-time detection with vision-language reasoning provides a scalable, adaptable solution for robotic perception in dynamic construction environments, supporting safer, more reliable autonomous robot operation.     «

Construction sites are highly dynamic environments where misplaced tools and equipment pose risks to both autonomous robots and human workers. Reliable real-time perception is therefore essential for safe robotic operation. However, traditional object detection systems are limited to predefined categories and struggle to generalize to unseen tool variants commonly found on construction sites. This thesis proposes a two-stage framework for open-world understanding of construction tools and equip...     »