Combining probability model and Web mining model: A framework for proper name transliteration

The rapid growth of the Internet has created a tremendous number of multilingual resources. There are Web pages in almost every popular language. However, language boundaries prevent information sharing and discovery across countries. Proper names play an important role in search queries and knowledge discovery. However, often foreign names need to be translated and this phonetic translation is referred to as transliteration. Previous transliteration models can be categorized into three approaches: A rule-based approach, a machine learning approach, and a statistical approach.. In this research we proposed a generic proper name transliteration framework which incorporates an enhanced Hidden Markov Model (HMM) and a Web mining model. We improved the traditional statistical-based transliteration in three areas: 1) incorporated a simple phonetic transliteration knowledge base; 2) incorporated a bigram and a trigram HMM; 3) incorporated a Web mining model that uses word frequency of occurrence information from the Web. We evaluated the framework on two different language pairs, English-Arabic and English-Chinese. Both experiments showed that when using HMM alone, a combination of the bigram and trigram HMM approach performed the best for English-Arabic transliteration. While the bigram model alone achieved fairly good performance, the trigram model alone did not. The Web mining approach boosted the performance by 46%. For English-Chinese transliteration, we found the trigram model out-performed the bigram. The Web mining approach again improved the performance by 12%. Overall, our framework achieved a precision of 86%-93% when the 8 best transliterations were considered. Our results are encouraging and show promise for using transliteration techniques to improve multilingual Web retrieval.