CiviViL: multimodal interpretation of composite urban facility scenes with spatial-semantic-aware bidirectional cross-attention

Urban infrastructure and industrial facilities serve as the major carriers of urban lifeline systems, playing a pivotal role in enhancing urban resilience. These critical facilities are typically complexes composed of multiple components arranged in non-rigid layouts, characterized by form-function decoupling, limited data availability, and a lack of abundant crowd-sourced geospatial information. Existing approaches excel at backbone optimization for general scenes classification but fail to model the complex structural context and intricate relational dependencies essential for interpreting composite urban facility scenes. Though large vision-language models (VLMs) have revolutionized general scene understanding, their potential remains untapped in this domain due to the lack of specialized datasets and mechanisms for encoding domain-specific spatial semantics. To bridge this gap, we construct the first large-scale, fine-grained image-caption multimodal dataset for composite critical facility scenes, comprising 52,603 pairs, and then propose a structured representation method augmented by spatial-semantic decomposition, named CiviViL. Specifically, a vision foundation model is used to extract global image features, while a pre-trained transformer-based language model encodes multi-dimensional discrete spatial contexts associated with the configuration of functional components, including component distribution, facility-component relationships, and view-dependent spatial geometry. A bidirectional cross-attention mechanism (image-to-text and text-to-image) is designed to achieve collaborative reinforcement between visual perception and spatial-relational semantics, enabling the integration of complex associations into a unified scene representation. Extensive experimental results on 13 categories of infrastructure and industrial facilities demonstrate that CiviViL significantly outperforms 21 state-of-the-art baselines, underscoring the effectiveness and superiority of cross-modal collaborative enhancement for fine-grained interpretation of composite urban facility scenes.