NMT语料库中语符不平衡度的测评研究OACSCDCSTPCD

Token imbalance is a common phenomenon in the corpus of neural machine translation(NMT).It is of great significance to evaluate the token imbalance degree of NMT corpus to improve the quality of corpus and translation effect.Aiming at the defects and deficiencies in the algorithm and word segmentation scope of the existing studies on the measurement of the token imbalance degree,this paper proposes the dispersion of token distribution(DTD)algorithm to cal-culate the token imbalance degree,expands the word segmentation scope,and evaluates the corpus from three granularity:character,subword and word.The experimental results show that the accuracy,validity and robustness of the proposed al-gorithm are greatly improved compared with previous studies.There are great differences in the token imbalance degree of corpora under different word segmentation granularity,in which character granularity has the highest token imbalance de-gree,followed by sub word granularity and word granularity.