Toward Lightweight End-to-End Semantic Learning of Real-Time Human Activity Recognition for Enabling Ambient Intelligence

Building accurate human behavior models is necessary for ambient intelligence. However, human activity recognition (HAR) in continuously monitored physical space faces challenges to achieve a good performance yet using only simple computing resources. In this work, we model HAR as an edge classification problem for a collaborative event graph of context entities in a sequential bipartite graph form. We design a semantic learning framework, called KGAR, to perform HAR by mining, encoding, and exploiting deep semantic knowledge of activities in an end-to-end fashion. KGAR has three components: preprocessor, KGEncoder, and predictor. The preprocessor builds offline a tiny knowledge graph of activities, to model and capture multidimensional semantic relationships between activities and core context entities. KGEncoder encodes the knowledge graph of activities using improved graph neural networks (GNNs) models, to avoid different confusing context patterns. The predictor can be deployed using lightweight deep neural networks to produce real-time labels. Experimental results show that using KGEncoder in KGAR improves the performance of original deep neural networks by 25% - 439% on five datasets. The time of labeling each sensor event during testing with event streams is less than 0.5ms. We have also conducted extensive experimental study to show that KGAR outperforms different types of models in more complex activity scenarios. We believe KGAR could be used for real-time HAR in real life with its high prediction performance and low computing requirement.