计算机工程与科学2025，Vol.47Issue(5)：775-786,12.DOI:10.3969/j.issn.1007-130X.2025.05.002
面向算力网络的跨集群数据迁移系统的设计和实现
Design and implementation of a cross-cluster data migration system for computational networks
摘要
Abstract
In the construction of computational networks,how to conduct efficient and reliable data migration between clusters in different regional computing centers is a key research topic.In view of this,this paper designs and implements a high-performance transmission software based on RSYNC,namely SCOW-SYNC.The main research results are as follows:Firstly,SCOW-SYNC adopts the queue and thread pool architecture to optimize the traditional RSYNC.By parallelly establishing multi-ple TCP connections and parallel transmission,the bandwidth utilization rate is improved.In addition,SCOW-SYNC also supports functions such as automatic large file splitting,dynamic compression,back-ground operation,real-time progress query,and SSH connection pool management.Through testing,SCOW-SYNC can achieve a speedup ratio of 125％to 130％compared with RSYNC.Secondly,in order to improve the security of transmission,this paper proposes a reliable cross-cluster transmission system architecture for computing centers.Data transmission only occurs between"transmission nodes"and is encrypted by"transmission keys",which are dynamically checked,generated,and distributed by the management node".Finally,this paper integrates SCOW-SYNC into the high-performance computing portal and management platform SCOW,and implements the cross-cluster transmission module of SCOW,so that users can perform high-performance data migration between different clusters through the browser,and deploys it to the cross-cluster environment of Peking University through containeriza-tion technology,which improves the production efficiency.关键词
高性能计算系统软件/算力网络/并行传输/RSYNC/集群安全Key words
high performance computing system software/computational network/parallel transmis-sion/RSYNC/cluster security分类
计算机与自动化引用本文复制引用
李俊哲,付振新,杨宏辉,马银萍,李若淼,樊春..面向算力网络的跨集群数据迁移系统的设计和实现[J].计算机工程与科学,2025,47(5):775-786,12.基金项目
2023年湖南省十大技术攻关项目(2023GK1010) （2023GK1010）
