信阳师范学院学报（自然科学版）2024，Vol.37Issue(4)：433-441,9.DOI:10.3969/j.issn.1003-0972.2024.04.003
基于HDF5的多层次结构并行IO算法
Multilevel Structure Parallel IO Algorithm Based on HDF5
摘要
Abstract
A multi-level parallel IO(Input/Output)scheme based on Hierarchical Data Format(HDF5)was proposed for large-scale data input and output applications.The parallel IO scheme was divided into two layers:Inter-node IO data was taken as unit,intra-node IO data was allowed to work cooperatively or independently.According to the internal working mode of nodes,a multi-level parallel IO algorithm and a multi-level sentinel parallel IO algorithm were proposed respectively,which could effectively improve IO efficiency and avoid redundancy of output files.Considering the two typical application scenarios of heterogeneous computing and pure CPU computing,multi-group experiments with a maximum of 4096 cores and 256G data were carried out on Shuguang platform and Intel platform,respectively.The results showed that the IO efficiency of multi-level parallel IO algorithm was increased by 1.97～25.87 times.The IO efficiency of multi-level sentinel parallel IO algorithm was increased by 6.53～9.36 times,and the number of output files was reduced to 1/4 and 1/32 of the number of parallel IO algorithms.关键词
层次存储格式/大规模并行计算/并行IO/数据存储Key words
Hierarchical Data Format(HDF5)/massively parallel computing/parallel IO/data storage分类
信息技术与安全科学引用本文复制引用
马文鹏,翟环欣,李瑞莹,袁武..基于HDF5的多层次结构并行IO算法[J].信阳师范学院学报（自然科学版）,2024,37(4):433-441,9.基金项目
国家重点研发计划项目(2020YFB1709500) （2020YFB1709500）
河南省重点研发与推广专项(科技攻关)(222102210162) （科技攻关）
