通信学报2017，Vol.38Issue(9)：133-147,15.DOI:10.11959/j.issn.1000-436x.2017188
基于分配适应度的Spark渐进填充分区映射算法
Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree
摘要
Abstract
The job execution mechanism of Spark was analyzed, task efficiency model and Shuffle model were estab-lished, then allocation fitness degree (AFD) was defined and the optimization goal was put forward. On the basis of the model definition, the progressive filling partitioning and mapping algorithm (PFPM) was proposed. PFPM established the data distribution scheme adapting Reducers' computing ability to decrease synchronous latency during Shuffle process and increase cluster the computing efficiency. The experiments demonstrate that PFPM could improve the rationality of workload distribution in Shuffle and optimize the execution efficiency of Spark.关键词
并行计算/Spark/渐进填充/分区映射/分配适应度Key words
parallel computing/Spark/progressive filling/partitioning and mapping/allocation fitness degree分类
信息技术与安全科学引用本文复制引用
卞琛,于炯,修位蓉,廖彬,英昌甜,钱育蓉..基于分配适应度的Spark渐进填充分区映射算法[J].通信学报,2017,38(9):133-147,15.基金项目
国家自然科学基金资助项目(No.61262088, No.61462079, No.61562078, No.61363083, No.61562086) （No.61262088, No.61462079, No.61562078, No.61363083, No.61562086）
新疆维吾尔自治区自然科学基金资助项目(No.2017D01A20) （No.2017D01A20）
新疆维吾尔自治区高校科研计划基金资助项目(No.XJED2016S106) （No.XJED2016S106）
新疆财经大学科研博士启动基金资助项目(No.2015BS007) The National Natural Science Foundation of China (No.61262088, No.61462079, No.61562078, No.61363083, No.61562086), The Natural Science Foundation of Xinjiang Uygur Autonomous Region (No.2017D01A20), The Educational Re-search Program of Xinjiang Uygur Autonomous Region (No.XJEDU2016S106), The Doctoral Research Foundation of Xinjiang University of Finance and Economics (No.2015BS007) （No.2015BS007）
