针对汉语语言模型中的数据稀疏问题,利用词语语义信息,将词语相似度同back-off平滑技术相结合,提出一种基于词语相似度的汉语语言模型平滑技术,并且设计了一种能够自动优化模型中各项参数的迭代算法,最后,将这种平滑技术由低阶语言模型推广到高阶语言模型中,将上述技术应用到音字转换领域。实验表明,这项技术使语言模型的性能获得了较大的提高,并有效地降低了音字转换系统的错误率。
By using word semantic information, this paper introduces a similarity-based smoothing algorithm for Chinese language modeling which combines word similarity calculation with back-off smoothing method, and presents an iterative method to optimize the parameters in the algorithm. Furthermore, the similarity-based smoothing algorithm is extended from low-level language model to high-level model. By applying the method to Pinyin-to-Chamcter conversion system, the experiment shows that the method improves the performance of language model significantly and reduces the error rate of pinyin-to-character conversion system effectively.