Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

Videos often capture objects, their motion, and the interactions between different objects. Although real-world objects have physical properties associated with them, many of these properties (such as mass and coefficient of friction) are not captured directly by the imaging pipeline. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce a new video question answering task for reasoning about the implicit physical properties of objects in a scene, from videos. For this task, we introduce a dataset – CRIPP-VQA, which contains videos of objects in motion, annotated with hypothetical/counterfactual questions about the effect of actions (such as removing, adding, or replacing objects), questions about planning (choosing actions to perform in order to reach a particular goal), as well as descriptive questions about the visible properties of objects. We benchmark the performance of existing video question answering models on two test settings of CRIPP-VQA: i.i.d. and an out-of-distribution setting which contains objects with values of mass, coefficient of friction, and initial velocities that are not seen in the training distribution. Our experiments reveal a surprising and significant performance gap in terms of answering questions about implicit properties (the focus of this paper) and explicit properties (the focus of prior work) of objects.

Note: Video-player works in Google Chrome but not on Safari.


Few examples of descriptive/counterfactual/planning questions

Input video


Counterfactual reference videos for readers:


Annotations are available at: Drive Link
Questions & Answers are available at: Drive Link
Mask-RCNN features are available at: Drive Link
To re-create the high-quality videos from the annotations follow instructions at: Code Link
Train/Val/Test distribution: [0-4000) / [4000, 4500) / [4500, 5000]