Decoding the city: harnessing machine learning to extract building attributes from street-view imagery

• Main Author: Chia, Zhi Yi
• Other Authors: David Lallemant
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Earth and Environmental Sciences; Machine learning; Computer vision; Object detection; Image classification; Attribute prediction; Street-view imagery; Multi-task learning; Building attributes; Faster- RCNN
• Online Access: https://hdl.handle.net/10356/174813
• Institution: Nanyang Technological University

Building attribute data can be used to improve the accuracy of hazard risk models, but detailed per-building data is generally unavailable due to the difficulty of collecting and maintaining such data. Machine learning models could be used to automate part of the collection process, by analysing widely available street-view data to locate the buildings and classify their attributes. Prior research has explored the use of object detection models to locate the buildings and image classification models to classify their attributes. However, it may be possible to instead modify an object detection model to accomplish both object detection and attribute prediction, as demonstrated in other fields. To test this possibility, a dataset of street-view images with annotated building attributes was constructed, and several modified versions of an object detection model were tested on the dataset. Another baseline model was constructed based on the approach outlined in prior research and tested on the dataset. Comparing the modified models, the modification with the greatest separation between the object detection and attribute prediction tasks performed the best, likely because of conflicts between the tasks. However, completely separating the tasks, like in the baseline model, only slightly improves object detection performance at the cost of substantially worsened attribute prediction performance on our dataset. Hence, the modified object detection model approach is superior for retrieving building attribute data, at least on this dataset. The potential of such an approach should be further explored, and more extensively verified by testing on more data.