计算机应用与软件2016，Vol.33Issue(8)：76-80,5.DOI:10.3969/j.issn.1000-386x.2016.08.017
面向城轨线网的海量小文件存储方法
AN APPROACH TO STORING MASSIVE SMALL FILES FOR URBAN RAIL TRANSIT NETWORK
刘靖 1廖家赵 2刘琼2
作者信息
- 1. 广州市地下铁道总公司建设事业总部 广东 广州 510380
- 2. 华南理工大学计算机科学与工程学院 广东 广州 510006
- 折叠
摘要
Abstract
Because of the great amount of small files for urban rail transit network system,traditional distributed file system is difficult to provide read and write process with high throughput and low latency meeting the demand for massive small files storage.According to the data characteristics of urban rail transit network system and the data access method cycled in day,we propose an approach of storing massive small files for urban rail transit network,which is based on FastDFS distributed file system and Redis key-value database.The method merges the small files with correlation for urban rail transit into a big file to accomplish the aggregated writing operation;It forms the global index according to the big file indexes returned by FastDFS,the initial offset of small file storage and the lengths of small files,and uses Redis to store key-value pair of the filename of small file and global index;It adopts data prefetching mechanism to prefech the files with adjacent creation time.Experimental results show that compared with FastDFS distributed file system,the read throughput and write throughput of small files in FastDFS-Redis storage system increase by 9.35% and 4.45% respectively,which reaches the goal of noticeably improving the small file access efficiency for urban rail transit network.关键词
小文件存储/城轨线网/FastDFS/Redis/访问性能Key words
Small file storage/Urban rail transit network/FastDFS/Redis/Access performance分类
信息技术与安全科学引用本文复制引用
刘靖,廖家赵,刘琼..面向城轨线网的海量小文件存储方法[J].计算机应用与软件,2016,33(8):76-80,5.
