Organic Computing Techniques for Run-Time Self-Adaptation of Ubiquitous, Multi-Modal Activity Recognition Systems

• Applicants: Professor Dr. Paul Lukowicz; Professor Dr. Bernhard Sick
• Subject Area: Computer Architecture, Embedded and Massively Parallel Systems
• Term: from 2015 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 276698135

• Final Report Year: 2020

Final Report Abstract

Using sensors to recognize and monitor human activities has important applications in areas ranging from sports and wellness through maintenance and manufacturing support to health and elderly care. As more and more sensors become available, one would expect such systems to get better and better. Unfortunately, this is often not the case. This is due to the lack of labeled training data for the new sensors, which is costly and difficult to record. In the project OC-SAM, we have developed learning methods that allow new sensors to be integrated into an activity recognition system without or with only very little new labeled training data. The first is based on structural analysis of the feature space that results from adding one or multiple new sensors to an existing system and using appropriate similarity measures to create synthetic labels. The second either exploits the broad availability of easy to label video data to create synthetic labeled training data for a variety of other sensors, or it is based on active learning. An unexpected outcome of the project was the development of an approach that helps to notice the theft of a smart device based on (learned) activities of its owner.

Publications

• Towards self-improving activity recognition systems based on probabilistic, generative models. In 2016 IEEE International Conference on Autonomic Computing (ICAC), pages 285–291, July 2016
  M. Jänicke, S. Tomforde, and B. Sick
  
• Label propagation: An unsupervised similarity based method for integrating new sensors in activity recognition systems. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–24, 2017
  V. F. Rey and P. Lukowicz
  
• Self-Adaptation of Activity Recognition Systems to New Sensors
  D. Bannach, M. Jänicke, V. F. Rey, S. Tomforde, B. Sick, and P. Lukowicz
  
• Hijacked smart devices – methodical foundations for autonomous theft awareness based on activity recognition and novelty detection. In Proceedings of the 10th International Conference on Agents and Artificial Intelligence, volume 2, pages 113–142, 2018
  M. Jänicke, V. Schmidt, B. Sick, S. Tomforde, and P. Lukowicz
  
• Self-adaptive multi-sensor activity recognition systems based on gaussian mixture models. Informatics, 5(3):38, 2018
  M. Jänicke, B. Sick, and S. Tomforde
  
• Agents and Artificial Intelligence. ICAART 2018, chapter Smart Device Stealing and CANDIES, pages 247–273. Number 11352 in Lecture Notes in Computer Science. Springer, Cham, Switzerland, 2019. (revised selected papers of ICAART 2018)
  M. Jänicke, V. Schmidt, B. Sick, S. Tomforde, P. Lukowicz, and J. Schmeißing
  
• Let there be imu data: generating training data for wearable, motion sensor based activity recognition from monocular rgb videos. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, pages 699–708, 2019
  V. F. Rey, P. Hevesi, O. Kovalenko, and P. Lukowicz