Egocentric Action Recognition by Automatic Relation Modeling

Egocentric action recognition aims to recognize the actions of the camera wearers from egocentric videos. In egocentric action recognition, relation modeling is important, because the interactions between the camera wearer and the recorded persons or objects form complex relations in egocentric videos. However, only a few of existing methods model the relations between the camera wearer and the interacting persons, and moreover they require prior knowledge or auxiliary data to localize the interacting persons. In this work, we consider modeling the relations in a weakly supervised manner, i.e., without using annotations or prior knowledge about the interacting persons or objects, for egocentric action recognition. We form a weakly supervised framework by unifying automatic interactor localization and explicit relation modeling for the purpose of automatic relation modeling. Firstly, we learn to automatically localize the interactors, i.e., the body parts of the camera wearer and the interacting persons or objects, by learning a series of keypoints directly from video data. Secondly, more importantly, we develop an ego-relational LSTM to search for the optimal connections for explicit relation modeling, which reduces the human efforts for structure design. Extensive experiments on egocentric video datasets illustrate the effectiveness of our method.