VideoBERT: A Joint Model for Video and Language Representation Learning

[parthp]

VideoBERT: A Joint Model for Video and Language Representation Learning - ICCV 2019 - Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, the paper proposes a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, the authors build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. VideoBERT is used n numerous tasks, including action classification and video captioning.  VideoBERT can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, it outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.