Enhancing Person Identification for Smart Cities: Fusion of Video Surveillance and Wearable Device Data based on Machine Learning

For smart cities, video surveillance has been widely used for security and management purposes. In video surveillance, a fundamental challenge is person identification (PID), which involves promptly tagging individuals in videos with their IDs. Using RFID and fingerprint/iris/face recognition is a possible solution. However, the identification results are highly related to environmental factors, such as line of sight, lighting conditions, and distance. Fingerprint/face recognition also has privacy concerns. In this work, we show how to achieve immediate PID through two sensor data sources: (i) human objects and their pixel locations retrieved from videos and (ii) user trajectory data retrieved from wearable devices through indoor localization. By fusing these pixel trajectories and indoor trajectories, we demonstrate an enhancing-vision capability in the sense that PID can be achieved on surveillance videos even when no clear human biological features are seen. Two types of fusion are proposed: (i) similarity-based and (ii) machine learning-based. We have developed lightweight prototyping with off-the-shelf equipment and validated our results through extensive experiments. The performance evaluation showed that our system has an accuracy of up to 92% for person identification.