Automatically evaluating content selection in summarization without human models
Citations Over TimeTop 10% of 2009 papers
Abstract
We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen-Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.
Related Papers
- Multilingual Summarization Evaluation without Human Models(2010)
- → Experiences with and Reflections on Text Summarization Tools(2009)9 cited
- On the Applications of the Experience Summarization in Modern Teaching and Research(2000)
- → Dynamic Summarization: Another Stride Towards Summarization(2007)
- → Prompting LLMs with content plans to enhance the summarization of scientific articles(2023)