电子学报2025，Vol.53Issue(2)：344-353,10.DOI:10.12263/DZXB.20240611
分布式存储系统中支持近数据处理的纠删码技术
An Erasure Coding Technology Supporting Near-Data Processing in Distributed Storage Systems
摘要
Abstract
Erasure coding and near-data processing are two cornerstones for building efficient cloud-edge-end collab-orative data management systems.The former ensures system availability by adding coding redundancy to the original data,while the latter avoids significant network transmission overhead by processing data at the storage end.Cloud-edge-end col-laborative data management systems typically adopt mature distributed storage systems as the underlying storage engine.However,the erasure coding implementation in mainstream distributed storage systems can not efficiently support near-data processing.This paper proposes an erasure coding architecture that supports near-data processing.Its basic principle is to re-organize the data to be encoded to ensure that semantically related data is stored in the same storage device,thereby avoid-ing cross-node data transmission during near-data processing.The scheme has been implemented on the distributed storage system Ceph,and the read and write performance under typical scenarios are tested.The experimental results show that the performance of reading objects in the near-data processing scenario and the conventional data reading scenario are im-proved by about 59.4%and 10%respectively,while the object writing performance remains consistent with the original ver-sion.关键词
纠删码/分布式存储/Ceph/近数据处理/云边端协同数据管理Key words
erasure coding/distributed storage/Ceph/near-data processing/cloud-edge-end collaborative data man-agement分类
计算机与自动化引用本文复制引用
李浩然,黄志杰,史宇龙,赵承佳,赵楠楠,张晓..分布式存储系统中支持近数据处理的纠删码技术[J].电子学报,2025,53(2):344-353,10.基金项目
国家自然科学基金(No.62272394,No.62202382) （No.62272394,No.62202382）
国家重点研发计划(No.2022YFB2702101) （No.2022YFB2702101）
广东省基础与应用基础研究基金(No.2021A1515110080) National Natural Science Foundation of China(No.62272394,No.62202382) （No.2021A1515110080）
National Key Research and Development Program(No.2022YFB2702101) （No.2022YFB2702101）
Guangdong Basic and Applied Basic Research Foundation(No.2021A1515110080) （No.2021A1515110080）
