计算机应用研究2017，Vol.34Issue(11)：3219-3222,3228,5.DOI:10.3969/j.issn.1001-3695.2017.11.004
基于资源签名与遗传算法的Hadoop参数自动调优系统
Hadoop parameter automatic tuning system based on resource signature and genetic algorithm
摘要
Abstract
In different Hadoop clusters,fast and accurate configuration is difficult because of the complexity and correlation of parameters.In this case,this paper built a Hadoop cluster automatic tuning system.Firstly,in order to classify the tasks,this system obtained the job's resource consumption characteristics.Secondly,it used a genetic algorithm based optimization framework to optimize parameters by iteration.This system built a library with the parameter configuration and a resource acquisition device.The experimental results show that the cluster task's finishing time significantly reduced after tuning and cluster resource utilization has been obviously improved.关键词
Hadoop集群/资源签名/遗传算法/参数配置/自动调优Key words
Hadoop cluster/resource signature/genetic algorithm/parameter configuration/automatic tuning分类
信息技术与安全科学引用本文复制引用
马跃,余骋远,于碧辉..基于资源签名与遗传算法的Hadoop参数自动调优系统[J].计算机应用研究,2017,34(11):3219-3222,3228,5.基金项目
国家科技重大专项资助项目(2012ZX01029001-002,2014ZX04009031) （2012ZX01029001-002,2014ZX04009031）
