自动分词是中文信息处理的关键步骤。由于具有结构简单、易于实现和开发周期短等优点,基于词典的分词方法被广泛应用。结合中文多字词数量少,使用频度低的特点,设计实现了一种新的词典机制,在此基础上,把分治策略引入到分词中,提出了一种新的分词算法,幷对该算法进行了理论分析和实验验证。
The primary stage in Chinese information processing is the automatic word segmentation. As the advantage of simple structure and easy realization as well as short development cycle, the method of word segmentation based on dictionary is very popular. In this paper, a new dictionary mechanism which is achieved in accordance with Chinese word characteristics is designed. Based on above, the divide-and-conquer strategy is introduced in word segmentation and a new word segmentation algorithm is presented. Furthermore, theory analyses as well as experiment verification are done.