计算机应用与软件2018，Vol.35Issue(2)：134-140,7.DOI:10.3969/j.issn.1000-386x.2018.02.025
面向国内直播行业的分布式弹幕爬虫研究
RESEARCH ON DISTRIBUTED DANMAKU CRAWLING FOR CHINESE LIVE-STREAMING SERVICES
摘要
Abstract
Danmaku services have grown in popularity recently thanks to the blooming of online video and live -streaming industries.However, the environment is seriously under-regulated, with frequent rule violations from both casters and audience,resulting in negative publicity of the industry,hence a need for research on danmaku crawling and processing is realized.A distributed live-streaming danmaku crawler scheme was devised according to technical characteristics of popular Chinese danmaku services.Light-weight clients were developed for services implementing open standards.For services based on Adobe Flash and proprietary interfaces, Chromium browser-based Electron with modified PPAPI plugin interface implementation was used to simulate browsing of streaming room pages,side-channeling Flash network traffic for crawling.Experiments on a well-known danmaku service showed that the system was able to dispatch IP addresses for large-scale crawling and the performance was good with achieving an average speed of 134 per second and peak speed of more than 1 000 per second for the danmaku traffic.关键词
直播弹幕/爬虫/浏览器模拟/PPAPI旁路Key words
Live-streaming danmaku/Crawler/Browser simulation/PPAPI side-channeling分类
信息技术与安全科学引用本文复制引用
王雪瑞,刘渊..面向国内直播行业的分布式弹幕爬虫研究[J].计算机应用与软件,2018,35(2):134-140,7.基金项目
国家科技支撑计划课题(2015BAH54F00) （2015BAH54F00）
国家重点研发计划项目(2016YFB0800305). （2016YFB0800305）
