编制价格指数的爬虫数据抽样方法研究OA北大核心CHSSCDCSSCICSTPCD

Aiming at the problem of high cost of compiling price index with full crawler data,this paper proposes a sampling method.This method adopts the idea of"big data-small data",and fully captures the commodity transaction data of the e-com-merce platform through web crawler technology in the base period to form a sampling frame.Sampling techniques are used in con-tinuous surveys;according to the idea of stratified sampling,clustering algorithms and silhouette coefficients are used to achieve overall data stratification;representative samples of each stratum are obtained through random sampling with unequal probability.Considering the non-response phenomenon of the selected samples in the continuous survey,the idea of formal and alternative samples is proposed.For each formal sample,the nearest neighbor matching algorithm is used to select several alternative sam-ples.When the formal sample has no answer,the alternative sample is used as a substitute to complete the price index compila-tion.Finally,the grain and oil category in Tmall mall is used as an example for experimental validation,and the results show that in the captured data,the full-amount crawler data in the base period is 18351,the average sampling ratio of the continuous survey from 2 to 8 periods is 10.18%,and the average relative error of sampling is 0.59%,which indicates that the method is feasible.