随着互联网爆炸式的发展,网络爬虫的重要性越来越重要.一个搜索引擎搜索结果的数量以及质量在一定程度上取决于网络爬虫爬取结果的质量,而如何能更好的组织这些爬虫也成了一件能影响爬虫效率的事情.随着在服务器上部署爬虫的增加,对一个能够有效管理爬虫监控系统的需求也就越来越紧迫.本文对爬虫监控系统的设计和实现将会给爬虫的管理带来很大的方便.
With the explosive growth of Internet, the importance of web crawler is becoming increasingly important.The quantity and quality of search results of a search engine is depends to some extent on the quality ofweb crawler's crawling results, and how to organize these crawlers better have become a things can affectthe effi ciency of crawler. With the increase of crawlers deployed on the server, the need for an effectivemanagement of monitoring system is more and more urgent. In this paper, the design and implementationof reptile monitoring system will bring great convenience to the crawler's management.