计算机技术与发展2019，Vol.29Issue(3)：72-76,5.DOI:10.3969/j.issn.1673-629X.2019.03.015
基于Spark平台的K-means算法的设计与优化
Design and Optimization of K-means Algorithm Based on Spark Platform
摘要
Abstract
The clustering center needs to be set manually is the biggest problem of K-means algorithm, and it is usually impossible to determine the classification of data in reality. In order to solve the problem, we propose a new OCC K-means algorithm. Different from the traditional algorithm, which generates the clustering center in the way of random selection, this algorithm carries out necessary preprocessing, and uses UPGMA and maximum and minimum distance algorithm to screen data points for the ones that can reflect data distribution characteristics as the initial clustering center, so as to improve the accuracy of clustering. From the two experimental results, it can be found that in different data sets, the improved algorithm is better in the measurement of clustering accuracy, recall, F-measurement than the traditional K-means algorithm. This is because the center point selected by OCC algorithm comes from different and data-intensive areas, and noise data and edge data interference to the experiment are excluded in the process of screening. At the same time, in order to conform to the trend of big data development, the parallelization implementation is carried out on Spark platform with Scala language, which improves the ability of the algorithm to deal with massive data, and the better parallelization of the algorithm is verified by experimental indexes.关键词
聚类/聚类中心/K-means/最大最小距离算法/非加权组平均法Key words
clustering/clustering center/K-means/maximum and minimum distance algorithm/unweighted pair group method with arithmetic mean分类
信息技术与安全科学引用本文复制引用
王义武,杨余旺,于天鹏,沈兴鑫,李猛坤..基于Spark平台的K-means算法的设计与优化[J].计算机技术与发展,2019,29(3):72-76,5.基金项目
国家自然科学基金(61640020) （61640020）
江苏省农业自主创新项目(CX(13)3054、CX(16)1006) （CX(13）
江苏省重点研发计划(BE2016368-1) （BE2016368-1）
江苏省科技重点及面上项目(SBE2018310371) （SBE2018310371）
弹总装线***技术研究(JCKY2017***) （JCKY2017***）
Postgraduate Research&Practice Innovation Program of Jiangsu Province(SJCX17_0107) （SJCX17_0107）
北京市教育委员会科技计划面上项目(KM201510028019) （KM201510028019）
