在Unicode编码方案中维、哈、柯文字符安排在阿拉伯字符区域,三种语言中共享字符比较多,跟阿拉伯字符区域混在一起,没有专用的语言ID。在信息检索和自然语言处理领域对维、哈、柯文的识别、处理带来不便。该文首先分析并总结了维、哈、柯文三种语言中的专用字符、复合字符、某些字符在某种语言中出现形势的独特性等特征,然后在此基础上设计了维、哈、柯文种识别算法。实验结果表明该文提出的文种识别算法的正确率在文本多于70词时达到96.67%以上。
In Unicode encoding consortium, Uyghur, Kazak and Kyrgyz characters are arranged in the Arabic characters area and mixed with Arabic characters. Some characters in these languages shares same code without language ID,which brings difficulty in information retrieval and natural language processing. After analyzing the unique characters, compound characters and the special features of some characters in certain language context, this paper designs a language identification algorithm of Uyghur, Kazak and Kyrgyz. The experimental results show that the accuracy achieves 96.67% for texts with 70 words or more.