Learning the Visual Interpretation of Sentences
Citations Over TimeTop 1% of 2013 papers
Abstract
Sentences that describe visual scenes contain a wide variety of information pertaining to the presence of objects, their attributes and their spatial relations. In this paper we learn the visual features that correspond to semantic phrases derived from sentences. Specifically, we extract predicate tuples that contain two nouns and a relation. The relation may take several forms, such as a verb, preposition, adjective or their combination. We model a scene using a Conditional Random Field (CRF) formulation where each node corresponds to an object, and the edges to their relations. We determine the potentials of the CRF using the tuples extracted from the sentences. We generate novel scenes depicting the sentences' visual meaning by sampling from the CRF. The CRF is also used to score a set of scenes for a text-based image retrieval task. Our results show we can generate (retrieve) scenes that convey the desired semantic meaning, even when scenes (queries) are described by multiple sentences. Significant improvement is found over several baseline approaches.
Related Papers
- Chinese Named Entity Recognition with Conditional Random Fields(2006)
- → Gradual transition detection with conditional random fields(2007)7 cited
- → Result identification for biomedical abstracts using Conditional Random Fields(2008)6 cited
- Named entity recognition in Chinese medical records based on cascaded conditional random field(2014)
- → Biomedical Named Entity Recognition Using Second-Order Conditional Random Fields(2011)