计算机与数字工程2012，Vol.40Issue(9)：63-65,3.
采用多种策略的分布式Web Spider
New Distributed Web Spider by Applying Many Optimization Strategies
陈炎龙 1段红玉1
作者信息
- 1. 郑州牧业工程高等专科学校信息工程系 郑州450011
- 折叠
摘要
Abstract
For the increasingly prominent web access problems, A New Distributed Web Spicier (NDWS) was proposed NDWS uses central control node to coordinate actions of all web spiders,employs Breadth-First search to obtain high-quality web pages, caches DNS to improve speed of access to web server, increases number of concurrent threads to increase download speed of web pages. Meanwhile, NDWS also can dynamically add web spider node and sub-cent ralcontrol-node so that NDWS has strong flexibility and expansion capability. Experi-mental results show that as a front-end of search engine, NDWS can quickly and efficiently download web pages, and has better performance.关键词
中央控制节点/宽度优先搜索/线程/搜索引擎Key words
central control node/breadth-first search/thread/search engine分类
信息技术与安全科学引用本文复制引用
陈炎龙,段红玉..采用多种策略的分布式Web Spider[J].计算机与数字工程,2012,40(9):63-65,3.
