计算机工程与应用Issue(22)：43-49,7.DOI:10.3778/j.issn.1002-8331.1212-0302
Hadoop平台数据访问监控机制研究
Data access monitoring mechanism in Hadoop platform
摘要
Abstract
Aiming on the issue of task scheduler considering the data location information for locality-based data processing in Hadoop Map tasks, a novel data access behavior monitoring mechanism is proposed in this paper. It is argued that the data access monitoring mechanism of Hadoop platform should not only serve to promote the efficiency of data access, but also serve to promote the execution efficiency of parallel Map/Reduce jobs. It is necessary to monitor the balance of data access overhead in the parallel execution of multiple Map tasks. The granularity and information set of data access monitoring in Hadoop platform is defined;The master-slave-based monitoring architecture is presented, which works with the support of Hadoop existing function modules; The detail implementation of the main monitoring function modules is discussed and the experimental results is analyzed.关键词
Hadoop/Map/Reduce/监控/数据访问Key words
Hadoop/Map/Reduce/monitoring/data access分类
信息技术与安全科学引用本文复制引用
王玉凤,梁毅,金翊,李光瑞..Hadoop平台数据访问监控机制研究[J].计算机工程与应用,2014,(22):43-49,7.基金项目
北京市教委科技计划项目（No.JC007013201101）；国家自然科学基金（No.61202075）；北京市自然科学基金预探索项目（No.4133081）。 （）
