Deep temporal architectures for activity recognition

Abstract

The amount of video content generated increases daily, three hundred hours of video content is uploaded to YouTube every 60 seconds1. There exists a need to sort, summarise, describe, categorise and retrieve video data based on the content (i.e. the activities occurring in the video). Activity recognition (i.e. automatically naming activities) is an important area for video analysis. Activity recognition has applications in robotics, video surveillance, multimedia retrieval, behaviour analysis, disaster warning systems and content-based browsing. Automatically categorising activities given a video clip poses two main challenges, namely object detection and motion learning. An activity recognition system must detect and localise the agent as well as learn to categorise the action the agent is performing. This research hypothesises that learning models incorporating spatial and temporal aspects from video data should outperform models that learn only spatial or temporal features on activity recognition learning tasks. The above hypothesis is investigated by developing two deep learning architectures for activity recognition that learn temporally independent and dependent features respectively.minima do not exist. A recurrent network (structurally constrained gated recurrent unit (SCGRU)) that adds contextual feature learning to gated recurrent units (GRUs) is proposed. Adding contextual features stabilises the hidden state of a GRU layer. The approach taken to investigate activity recognition architectures in this research involved examining the architectures on four benchmark datasets and analysing the results to 1) find the best model for activity recognition, 2) examine the model’s ability to learn salient temporal features, and 3) examine the model’s computational complexity. SCGRU based models outperform GRU based models on the majority of the investigated activity recognition models and datasets.