Storytelling from an Image Stream Using Scene Graphs

Citations Over TimeTop 10% of 2020 papers

Abstract

Visual storytelling aims at generating a story from an image stream. Most existing methods tend to represent images directly with the extracted high-level features, which is not intuitive and difficult to interpret. We argue that translating each image into a graph-based semantic representation, i.e., scene graph, which explicitly encodes the objects and relationships detected within image, would benefit representing and describing images. To this end, we propose a novel graph-based architecture for visual storytelling by modeling the two-level relationships on scene graphs. In particular, on the within-image level, we employ a Graph Convolution Network (GCN) to enrich local fine-grained region representations of objects on scene graphs. To further model the interaction among images, on the cross-images level, a Temporal Convolution Network (TCN) is utilized to refine the region representations along the temporal dimension. Then the relation-aware representations are fed into the Gated Recurrent Unit (GRU) with attention mechanism for story generation. Experiments are conducted on the public visual storytelling dataset. Automatic and human evaluation results indicate that our method achieves state-of-the-art.

Related Papers

• → Storytelling is not just for marketing: Cultivating a storytelling culture throughout the organization(2023)41 cited
• → Storytellers and Storytelling in Northeast Thailand(2008)4 cited
• → Review of Storytelling in Organizations: Why Storytelling Is Transforming 21st Century Organizations and Management by John Seeley Brown(2006)2 cited
• → Review of Storytelling in Daily Life: Performing Narrative by Langellier and Peterson(2006)1 cited
• Film Story: Telling & Showing(2018)