Fully unsupervised word segmentation with BVE and MDL

Daniel Hewlett; Paul Cohen

Fully unsupervised word segmentation with BVE and MDL

Daniel Hewlett, Paul Cohen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Several results in the word segmentation literature suggest that description length provides a useful estimate of segmentation quality in fully unsupervised settings. However, since the space of potential segmentations grows exponentially with the length of the corpus, no tractable algorithm follows directly from the Minimum Description Length (MDL) principle. Therefore, it is necessary to generate a set of candidate segmentations and select between them according to the MDL principle. We evaluate several algorithms for generating these candidate segmentations on a range of natural language corpora, and show that the Bootstrapped Voting Experts algorithm consistently outperforms other methods when paired with MDL.

Original language	English (US)
Title of host publication	ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics
Subtitle of host publication	Human Language Technologies
Pages	540-545
Number of pages	6
State	Published - 2011
Event	49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL-HLT 2011 - Portland, OR, United States Duration: Jun 19 2011 → Jun 24 2011

Publication series

Name	ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
Volume	2

Other

Other	49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL-HLT 2011
Country/Territory	United States
City	Portland, OR
Period	6/19/11 → 6/24/11

ASJC Scopus subject areas

Language and Linguistics
Linguistics and Language

Cite this

Hewlett, D., & Cohen, P. (2011). Fully unsupervised word segmentation with BVE and MDL. In ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 540-545). (ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies; Vol. 2).

Fully unsupervised word segmentation with BVE and MDL. / Hewlett, Daniel; Cohen, Paul.

ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011. p. 540-545 (ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Hewlett, D & Cohen, P 2011, Fully unsupervised word segmentation with BVE and MDL. in ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 2, pp. 540-545, 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL-HLT 2011, Portland, OR, United States, 6/19/11.

Hewlett, Daniel ; Cohen, Paul. / Fully unsupervised word segmentation with BVE and MDL. ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011. pp. 540-545 (ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies).

@inproceedings{2c68c9f3ae344d649b6fddb8750acecc,

title = "Fully unsupervised word segmentation with BVE and MDL",

abstract = "Several results in the word segmentation literature suggest that description length provides a useful estimate of segmentation quality in fully unsupervised settings. However, since the space of potential segmentations grows exponentially with the length of the corpus, no tractable algorithm follows directly from the Minimum Description Length (MDL) principle. Therefore, it is necessary to generate a set of candidate segmentations and select between them according to the MDL principle. We evaluate several algorithms for generating these candidate segmentations on a range of natural language corpora, and show that the Bootstrapped Voting Experts algorithm consistently outperforms other methods when paired with MDL.",

author = "Daniel Hewlett and Paul Cohen",

year = "2011",

language = "English (US)",

isbn = "9781932432886",

series = "ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies",

pages = "540--545",

booktitle = "ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics",

note = "49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL-HLT 2011 ; Conference date: 19-06-2011 Through 24-06-2011",

}

TY - GEN

T1 - Fully unsupervised word segmentation with BVE and MDL

AU - Hewlett, Daniel

AU - Cohen, Paul

PY - 2011

Y1 - 2011

N2 - Several results in the word segmentation literature suggest that description length provides a useful estimate of segmentation quality in fully unsupervised settings. However, since the space of potential segmentations grows exponentially with the length of the corpus, no tractable algorithm follows directly from the Minimum Description Length (MDL) principle. Therefore, it is necessary to generate a set of candidate segmentations and select between them according to the MDL principle. We evaluate several algorithms for generating these candidate segmentations on a range of natural language corpora, and show that the Bootstrapped Voting Experts algorithm consistently outperforms other methods when paired with MDL.

AB - Several results in the word segmentation literature suggest that description length provides a useful estimate of segmentation quality in fully unsupervised settings. However, since the space of potential segmentations grows exponentially with the length of the corpus, no tractable algorithm follows directly from the Minimum Description Length (MDL) principle. Therefore, it is necessary to generate a set of candidate segmentations and select between them according to the MDL principle. We evaluate several algorithms for generating these candidate segmentations on a range of natural language corpora, and show that the Bootstrapped Voting Experts algorithm consistently outperforms other methods when paired with MDL.

UR - http://www.scopus.com/inward/record.url?scp=84859036614&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84859036614&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84859036614

SN - 9781932432886

T3 - ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

SP - 540

EP - 545

BT - ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics

T2 - 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, ACL-HLT 2011

Y2 - 19 June 2011 through 24 June 2011

ER -

Fully unsupervised word segmentation with BVE and MDL

Abstract

Publication series

Other

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this