电子科技大学学报2016，Vol.46Issue(6)：986-991,6.DOI:10.3969/j.issn.1001-0548.2016.06.019
一种Mapreduce作业内存精确预测方法
An Innovative Memory Prediction Approach for Mapreduce Job
摘要
Abstract
It is difficult to predict the amount of memory for a mapreduce job. Based on the fact that Java virtual machine (JVM) divides the heap space managed by the JVM garbage collector into young and old generations, a generational memory prediction method is put forward. We build up a function that models the relationship between the amount of young generation and the total garbage collection time, and then we use a constrained nonlinear optimization model to find the rational footprint of young generation. The memory model for the map phase is established, the phase of a mapreduce job is reduced, then a relationship between map/reduce tasks’ performance (runtime of a task) and the amount of memory of the old generation is set up, and finally, the reasonable old generation memory size is obtained. The experimental results show that the proposed approach can accurately predict the memory size of map and reduce the tasks of a mapreduce job. In comparison with the default configuration, the proposed approach can give us 6 times performance improvement than default settings.关键词
垃圾回收/Java虚拟机/mapreduce/资源管理Key words
garbage collection/JVM/mapreduce/resource management分类
信息技术与安全科学引用本文复制引用
罗永刚,陈兴蜀,杨露..一种Mapreduce作业内存精确预测方法[J].电子科技大学学报,2016,46(6):986-991,6.基金项目
国家科技支撑计划(2012BAH18B05) （2012BAH18B05）
