Smart Traffic Fund application — AI-powered Road Digital Twin for Hong Kong

During my postdoctoral research at City University of Hong Kong (CityU), I contributed to a Smart Traffic Fund (STF) research and application proposal led by Prof. Xiaowei Luo — “LLM-powered Road Maintenance Digital Twin for Hong Kong”. The project aims to transform how Hong Kong monitors, diagnoses, and maintains its road network by integrating fleet-based sensing, multi-modal AI diagnostics, and an LLM-powered cognitive digital twin into operational road asset management workflows.

Proposed maintenance-oriented Road Digital Twin framework for Hong Kong

Proposed framework — sensing, diagnostics, digitalisation, and deployment

Motivation

Hong Kong’s road network spans 2,241 km, with annual highway maintenance spending of approximately HK$1.69 billion. In 2023 alone, more than 3,400 road damage or defect cases were reported. Conventional monitoring relies on labour-intensive manual inspections and costly professional survey vehicles, offering limited spatial and temporal coverage. Maintenance is often reactive — defects are addressed only after they worsen — which affects driving comfort, safety, and operational efficiency.

The Smart Traffic Fund — a HK$1 billion initiative established by the Transport Department — supports research and application projects that use innovation and technology to improve the convenience, efficiency, and safety of road users. Our proposal aligns with this scope by developing vehicle-related technologies for smarter, proactive, and data-driven road asset management.

Proposed solution — four integrated modules

The proposed maintenance-oriented Road Digital Twin (DT) integrates four modules across the full management cycle:

Module Purpose
Sensing Fleet-vehicle toolkit and smartphone app for real-time, city-wide road monitoring
Diagnostics Multi-modal AI models for defect classification, quantification, and roughness analysis
Digitalisation Dynamic road digital twin with road-specific ontology and interoperable data structures
Deployment Predictive maintenance strategies and LLM-generated inspection reports
1. Sensing module

Unlike conventional survey vehicles with limited coverage, this module deploys a low-cost sensing toolkit on fleet vehicle windscreens (e.g. buses, taxis, and garbage trucks), turning widely available public vehicles into a scalable data source.

The onboard hardware design includes:

  • Main controller: LubanCat RK3588
  • Depth camera: Orbbec Femto Bolt
  • IMU: HIPNUC HI229 (under evaluation)
  • GPS module: Beitian GPS (under evaluation)
  • Mini IPS display for local interaction (under evaluation)
  • Microphone for acoustic sensing (full configuration)
  • Interfaces: USB ports, 12 V DC power, TF card slot

Key innovations include VLM models optimised for mobile-grade hardware, enabling responsive on-vehicle detections rather than relying solely on conventional survey instruments. A complementary smartphone app (iOS and Android) will support public participation in crowdsourced data collection, covering both response-based comfort monitoring and visual AI-based defect detection.

Target: a functional toolkit costing under HK$1,000 per unit, enabling scalable deployment across urban routes.

2. Diagnostics module

AI models process multi-modal data — camera visuals, vehicle motion (IMU), and acoustic signals — to evaluate pavement and roadside asset conditions. Compared with existing CV-only solutions (e.g. Blackmoth, 3DAI) that detect defects with bounding boxes, this module aims to provide advanced defect quantification (width, depth, and length) for maintenance prioritisation.

The diagnostics pipeline explores:

  • RGB stream: lightweight architectures (Inception-based CNNs, MobileNet) with data augmentation
  • IMU + acoustic stream: LSTM-based models with time warping and magnitude scaling augmentation
  • Sensor fusion: feature-level and decision-level fusion across modalities
  • LLM/VLM validation: outputs validated against human-labelled data and engineering guidelines

Target: ≥ 95% accuracy in defect detection and classification, with daily high-resolution road assessments.

3. Digitalisation module — Cognitive Road Digital Twin

Current systems such as HyD’s MARCH platform store critical asset information but face fragmented data schemas and limited interoperability with BIM and other management tools. This module proposes a Cognitive Digital Twin that:

  • Develops a road-specific ontology graph and flexible data structures
  • Continuously ingests fleet-contributed sensing and diagnostic outputs
  • Integrates geospatial data, maintenance logs, and inspection records
  • Provides a unified, real-time representation of road asset conditions across Hong Kong
4. Deployment module — LLM-powered maintenance intelligence

The deployment module addresses the labour-intensive manual interpretation of multi-modal inspection data. Domain-specific LLMs will be fine-tuned on labelled images, inspection reports, and Hong Kong road asset guidelines to:

  • Generate autonomous inspection reports (Scan-to-Report)
  • Support predictive maintenance planning using historical maintenance, climate, and traffic data
  • Provide actionable maintenance recommendations benchmarked against engineering standards

Research objectives

  1. Develop a scalable, low-cost road asset sensing system — VLM-powered devices on public fleet vehicles for real-time, city-wide monitoring with open and actionable data collection.
  2. Advance AI-based road asset diagnostics — multi-modal AI models for accurate, quantifiable detection of road anomalies, integrating visual and vibration data.
  3. Establish an LLM-powered Cognitive Road Digital Twin — unify, interpret, and generate actionable insights from diverse road asset data to enhance maintenance decision-making.

R&D milestones and deliverables

The project is structured in four milestones:

Milestone Focus Key deliverable
M1 Sensing D1: Road condition data collection system (toolkit + smartphone app)
M2 Diagnostics Multi-modal pavement condition assessment framework
M3 Digitalisation D5: Digital Twin system for road asset managing authorities
M4 Deployment LLM-powered routine inspection report generation and maintenance planning

A field experiment in Hong Kong, involving local bus and taxi operators, will validate the platform’s ability to reduce manual inspection costs, achieve high-accuracy defect detection, and generate predictive maintenance plans more efficiently than conventional methods.

My contribution

As part of the CityU project team, my work on this STF application included:

  • Drafting and refining the technical proposal and project abstract

  • Designing the on-vehicle sensing hardware architecture (edge computing platform, depth camera, IMU, and GPS integration)

  • Supporting the development of the diagnostics and digital twin framework, building on my prior PhD research on smartphone-based pavement roughness estimation, crowdsourced data, and LLM-assisted inspection reporting

  • Preparing presentation materials and literature review for state-of-the-art benchmarking against existing road monitoring and digital twin solutions

  • Smart Traffic Fund: stf.hkpc.org