Situation Recognition: Visual Semantic Role Labeling for Image Understanding
Citations Over TimeTop 1% of 2016 papers
Abstract
This paper introduces situation recognition, the problem of producing a concise summary of the situation an image depicts including: (1) the main activity (e.g., clipping), (2) the participating actors, objects, substances, and locations (e.g., man, shears, sheep, wool, and field) and most importantly (3) the roles these participants play in the activity (e.g., the man is clipping, the shears are his tool, the wool is being clipped from the sheep, and the clipping is in a field). We use FrameNet, a verb and role lexicon developed by linguists, to define a large space of possible situations and collect a large-scale dataset containing over 500 activities, 1,700 roles, 11,000 objects, 125,000 images, and 200,000 unique situations. We also introduce structured prediction baselines and show that, in activity-centric images, situation-driven prediction of objects and activities outperforms independent object and activity recognition.
Related Papers
- → Shallow Semantic Parsing Based on FrameNet, VerbNet and PropBank(2006)22 cited
- → A comparative study on generalization of semantic roles in FrameNet(2009)27 cited
- → FrameNet-Based Fact-Seeking Answer Processing: A Study of Semantic Alignment Techniques and Lexical Coverage(2008)2 cited
- Using Ontologies for Semi-automatic Linking VerbaLex with FrameNet(2010)
- Czech Verbs in FrameNet Semantics(2009)