软件导刊2019，Vol.18Issue(1)：45-47,55,4.DOI:10.11907/rjdk.181652
Spark Streaming中参数与资源协同调整策略
Parameter and Resource Coordination Adjustment Strategy for Spark Streaming
摘要
Abstract
Spark Streaming is a typical batched streaming processing system that can be used to process continuously arriving data streams.The two most important characteristics of streaming data are its volatility and timeliness.The method of dynamical parameter configuration and dynamical resource allocation are proposed to guarantee the end to end latency with different data arrival rates.However, the method of dynamical parameter configuration has limitation on scope of application, and the method of dynamical resource allocation will bring greater cost to users.Therefore, this paper proposes a parameter and resource coordination adjustment strategy, using dynamic neighborhood particle swarm algorithm to find a solution that can achieve resource minimization on the premise of meeting the SLO goal.Experiments show that AdaStreaming reduced latency by 59％ against DyBBS, and reduced the amount of resources by 34％ against DRA.关键词
Spark Streaming/动态邻域粒子群/参数配置/资源分配Key words
Spark Streaming/DNPSO/parameter configuration/resource allocation分类
信息技术与安全科学引用本文复制引用
梁毅,刘飞,常仕禄,程石帆..Spark Streaming中参数与资源协同调整策略[J].软件导刊,2019,18(1):45-47,55,4.基金项目
国家自然科学基金项目(91546111, 91646201) （91546111, 91646201）
国家重点研发计划项目(2017YFC0803300) （2017YFC0803300）
北京市教委项目(KZ201610005009) （KZ201610005009）
