汉语自动分词是中文信息处理的前提,如何提高分词效率是中文信息处理技术面临的一个主要问题。基于词典和基于统计的分词方法是现有分词技术的主要方法,但是前者无法处理歧义字段,后者需要大量的词频计算耗费时间。本文提出优化最大匹配与统计结合的分词方法,首先提出优化最大匹配算法,在此基础上提出了规则判断与信息量统计两种消歧策略。然后,给出了优化最大匹配与统计结合的分词算法,提高了分词的效率。最后,基于分词算法实现中文分词系统,并通过实验对算法进行了分析和验证。
Chinese automatic segmentation is the precondition of Chinese information processing. A primary problem of Chinese information processing is how to improve segmentation efficiency. The segmentation method based on dictionary and statistics is main method of present segmentation technology; the former can not deal with ambiguity and the latter need a large amount of time to calculate word frequency. A method based on optimization maximum matching integrated with statistics is proposed. The method uses the se...