计算机与现代化Issue(12)：65-70,76,7.DOI:10.3969/j.issn.1006-2475.2017.12.013
Storm流处理平台中负载均衡机制的实现
Realization of Load Balancing Mechanism in Storm Streaming Processing Platform
摘要
Abstract
Compared with Hadoop,Storm has advantage of real-time data stream processing,which provides an efficient,fast and real-time data processing framework for multi-source heterogeneous data processing.However,the worker assignments in the Storm cluster only consider the sort of available Slot between different nodes,while ignoring the current load condition of different nodes,which may fail to meet the command of load balancing when more than one topology running in the cluster.In order to improve the efficiency and achieve load balancing of real-time stream processing,a Storm scheduling algorithm is proposed which is weighted sorting of available Slot and node load conditions and based on Storm-based distributed flow processing system to reduce load imbalance.And through designing the data structure reasonably,the paper designs the rowkey in Hbase randomly and evenly,which can ensure the load balance of the various RegionServer,improve the utilization of cluster resources and increase the speed of data writing greatly.Through the comparison experiment with the original Storm system,it is shown that the above algorithm improvement and mechanism optimization ensure the fast writing of data and improve the utilization rate of cluster resources.The improved system has obvious advantages in practicality and efficiency.关键词
Storm/流处理/分布式计算/批量处理/负载均衡Key words
Storm/streaming processing/distributed computing/batch processing/load balancing分类
信息技术与安全科学引用本文复制引用
张楠,柴小丽,谢彬,唐鹏..Storm流处理平台中负载均衡机制的实现[J].计算机与现代化,2017,(12):65-70,76,7.基金项目
中国电子科技集团公司第三十二研究所自立项目(ZQ160006 （）
ZQ160007) （）
