计算机工程与应用2019，Vol.55Issue(13)：66-71,6.DOI:10.3778/j.issn.1002-8331.1803-0095
Hadoop海量数据迁移系统开发及应用
Development and Application of Hadoop Massive Data Migration System
摘要
Abstract
With more and more data generated by High Energy Physics(HEP)experiments, Hadoop has been a solution for HEP data analysis while facing with the demand of data migration. However, existing data migration tools do not sup-port data transmission between HDFS and other file systems, and have obvious performance deficiency. Based on the requirements of high-energy physical data synchronization and archiving, this paper designs and implements a universal mass data migration system, which uses MapReduce to directly move data between HDFS and other storage systems or media by extending the HDFS data access methods. In addition, dynamic priority scheduling model is proposed to do multi-tasks dynamic priority assignment and selection. The system has been applied to the data migration in LHAASO experiment, and the actual operation results indicate that the system achieves good performance and meets the data migra-tion requirements of various experiments.关键词
高能物理/数据迁移/GridFTP协议/动态优先级调度/多属性决策/Hadoop系统Key words
High Energy Physics(HEP)/ data migration/ GridFTP protocol/ dynamic priority scheduling algorithm/multiple attribute decision-making/ Hadoop system分类
信息技术与安全科学引用本文复制引用
YIN Qiao,WEI Zhanchen,HUANG Qiulan,SUN Gongxing,SHI Jingyan..Hadoop海量数据迁移系统开发及应用[J].计算机工程与应用,2019,55(13):66-71,6.基金项目
国家自然科学基金(No.11775249,No.11775250). （No.11775249,No.11775250）
