计算机技术与发展Issue(2)：144-149,6.DOI:10.3969/j.issn.1673-629X.2016.02.033
基于Hadoop的网络舆情监控平台设计与实现
Design and Implementation of Network Consensus Monitoring System Based on Hadoop
摘要
Abstract
A network consensus monitoring system based on Hadoop was designed and realized. The system adopts HDFS as the underly-ing storage system,and then it builds a distributed database based on HBase with it to realize unified storage and management on the net-work consensus information. Firstly,it grabs the data with the distributed web crawler based on MapReduce to solve the problems of low efficiency and poor expansibility of single crawler. Then it uses the secondary clustering algorithm with Canopy combined with K-means, which can overcome the shortages of single K-means clustering algorithm and could improve the efficiency and precision of text cluste-ring. Finally,it could realize the topics tracking strategy based on query,also could be effective track and analysis of hot topics. The simu-lation experiment results show that compared with the traditional methods,the false negative and false positive of Canopy-Kmeans cluste-ring method is lower at 1. 24% and 0. 09% respectively,the minimum standard price is lower at 1. 681%. Through providing the visual-ized analysis of network consensus,the system proposed could provide scientific and systematical technology support for enterprises and scientific institutions to learn the hot network consensus and make network consensus strategy.关键词
Hadoop/MapReduce/舆情监控/文本聚类/热点发现/话题跟踪Key words
Hadoop/MapReduce/monitoring public opinion/text clustering/hot topic founding/topic tracking分类
信息技术与安全科学引用本文复制引用
李晨,杨子江,朱世伟,于俊凤..基于Hadoop的网络舆情监控平台设计与实现[J].计算机技术与发展,2016,(2):144-149,6.基金项目
山东省科学院青年基金项目(2013QN036) （2013QN036）
山东省科技发展计划(2013GGX10127,2014GGX101013) （2013GGX10127,2014GGX101013）
