A hybrid Markov/semi-Markov conditional random field for sequence segmentation

Citations Over TimeTop 10% of 2006 papers

Abstract

Markov order-1 conditional random fields (CRFs) and semi-Markov CRFs are two popular models for sequence segmentation and labeling. Both models have advantages in terms of the type of features they most naturally represent. We propose a hybrid model that is capable of representing both types of features, and describe efficient algorithms for its training and inference. We demonstrate that our hybrid model achieves error reductions of 18% and 25% over a standard order-1 CRF and a semi-Markov CRF (resp.) on the task of Chinese word segmentation. We also propose the use of a powerful feature for the semi-Markov CRF: the log conditional odds that a given token sequence constitutes a chunk according to a generative model, which reduces error by an additional 13%. Our best system achieves 96.8% F-measure, the highest reported score on this test set.

Related Papers

• → Recurrent conditional random field for language understanding(2014)112 cited
• → Hybrid semi-Markov CRF for Neural Sequence Labeling(2018)56 cited
• → Gradual transition detection with conditional random fields(2007)7 cited
• → Upgrading CRFS to JRFS and its Benefits to Sequence Modeling and Labeling(2020)5 cited
• → Hybrid semi-Markov CRF for Neural Sequence Labeling(2018)3 cited