欢迎您！东篱公司
退出
-
申报数据库
-
立项数据库
-
成果数据库
-
项目获奖数据库
中文摘要：
多线程爬行是高效率网络机器人实现的关键。分析在单个计算机节点上网络机器人多线程爬行的关键流程,并借鉴硬件指令流水线的执行过程研究不同多线程并发调度策略下系统的性能特点。最后,给出实现网络机器人多线程爬行的关键数据结构、并发调度方法及URL消重策略。实验数据表明：有效地控制并发和有选择地消除重复URL可以提高网络机器人的爬行速度。
英文摘要：
Multi-thread crawling is the key in implementing a high efficient crawler.Core working flow of crawler＇s multi-thread crawling at a single computer node is analyzed in the paper.It focuses on the system performances in different multi-thread concurrent controlling policies with the reference of the execution process of hardware instruction pipelining.Finally,a key data structure for the implementation of crawler＇s multi-thread crawling,the concurrent controlling approach and the duplicated URLs elimination policy are given.The experimental data show that to effectively control the concurrent and to selectively eliminate the repeated URLs could accelerate the crawler＇s working speed.
同期刊论文项目
同项目期刊论文
期刊信息
位置：