规则与统计相结合的中文新词识别研究

针对目前分词方法无法识别网络中不断出现的普通新词,设计了一种新的基于规则与统计相结合的分词方法。针对新词不同的构词模式特点,利用语言学的知识把新词识别问题分类细化,将单字串模式的新词和后缀串模式的新词作为本文的主要识别对象。对于单字串模式的候选新词,在使用内部词概率模型的基础上,通过分析候选新词的内部紧密结合度以及上下文环境依赖程度等特征信息,采用平均互信息和左右邻接信息熵相结合的方法进行识别;对于后缀模式的候选新词,通过大规模语料训练得到的噪声尾词典进行噪声串的过滤得到新词.

Research on Chinese New Word Identification Based on Regulations and Statistics

In view of the current word segmentation methods＇ incapability in recognizing the newly emerged words on the Internet,we design a new sub-word method based on regulations and statistics.According to the different features of component models of new words,we make use of linguistic knowledge to classify the problem of recognizing new words,and take single character mode and suffix character mode as the research objects.With regard to the candidate new words of single character mode,the recognition is realized by means of inspecting internal combination and external linguistic environment of those candidates which are on the basis of inside word probability,and employing the combination of the average mutual information and the left and right entropy model.As for the candidate new words of the suffix character mode,new words are obtained by filtering noises with the noise tail dictionary based on the large-scale corpus training.