Advanced Learning for Tracking and Detection in Medical Workflow Analysis

• Applicant: Professor Dr. Nassir Navab, Ph.D.
• Co-Applicants: Professor Dr. Horst Bischof; Professor Dr. Pascal Fua; Professor Dr. Vincent Lepetit
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2011 to 2015
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 179168991

• Final Report Year: 2015

Final Report Abstract

Surgical workflow models are built in order to derive and analyse statistical properties of a surgery for recovering the phase of the operation, staff training, data visualization, report generation and monitoring. Building a workflow model requires sufficient amount of data from different sources and sensors such as medical instruments and imaging devices. In this project, we propose to use a multi-view RGB-camera system that automatically estimates the body pose of the surgeons and medical staff. Our goal is to perform 3D human pose estimation of multiple individuals from multiple views inside the operating room. The body poses compose an additional input signal to the framework of the surgical workflow modelling. To that end, we introduce different human body models for pose estimation in the 2D and 3D space based on RGB image input. Moreover, we present a unique dataset for human pose estimation in the operating room that captures a simulated medical operation using a multi-view camera system. We evaluate our models in standard human pose datasets, as well as in the operating room and demonstrate state-of-the-art performance.

Publications

• “A user-centered and workflow-aware unified display for the operating room,” in MICCAI Workshop on Modeling and Monitoring of Computer Assisted Interventions (M2CAI), 2012
  R. Stauder, V. Belagiannis, L. Schwarz, A. Bigdelou, E. Soehngen, S. Ilic, and N. Navab
  
• “Segmentation based particle filtering for real-time 2d object tracking,” in Computer Vision–ECCV 2012, pp. 842–855, Springer, 2012
  V. Belagiannis, F. Schubert, N. Navab, and S. Ilic
  
• “3D pictorial structures for multiple human pose estimation,” in CVPR 2014-IEEE International Conference on Computer Vision and Pattern Recognition, 2014
  V. Belagiannis, S. Amin, M. Andriluka, B. Schiele, N. Navab, and S. Ilic
  (See online at https://doi.org/10.1109/CVPR.2014.216)
• “Holistic human pose estimation with regression forests,” in Articulated Motion and Deformable Objects, pp. 20–30, Springer, 2014
  V. Belagiannis, C. Amann, N. Navab, and S. Ilic
  
• “Multiple human pose estimation with temporally consistent 3D pictorial structures,” in Computer Vision–ECCV 2014, ChaLearn Looking at People Workshop, Springer, 2014
  V. Belagiannis, X. Wang, B. Schiele, P. Fua, S. Ilic, and N. Navab
  
• “Robust optimization for deep regression,” in Computer Vision (ICCV), 2015 IEEE International Conference on, IEEE, 2015
  V. Belagiannis, C. Rupprecht, G. Carneiro, and N. Navab
  (See online at https://doi.org/10.1109/ICCV.2015.324)
• “3D pictorial structures revisited: Multiple human pose estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 10, pp. 1929-1942, 1 Oct. 2016
  V. Belagiannis, S. Amin, M. Andriluka, B. Schiele, N. Navab, and S. Ilic
  (See online at https://doi.org/10.1109/TPAMI.2015.2509986)
• “Parsing human skeletons in an operating room,” Machine Vision and Applications, October 2016, Volume 27, Issue 7, pp 1035–1046
  V. Belagiannis, X. Wang, H. Beny Ben Shitrit, K. Hashimoto, R. Stauder, Y. Aoki, M. Kranzfelder, A. Schneider, P. Fua, S. Ilic, H. Feussner, and N. Navab
  (See online at https://doi.org/10.1007/s00138-016-0792-4)