如何从海量的Web资源中获取有用的信息是Web研究领域的重要研究内容。针对特定领域信息的获取,目前主要采用聚焦爬虫策略。该策略只爬取与主题相关的页面,忽略不相关页面。但目前的聚焦爬虫技术在爬行效率和页面质量两个方面仍存在一定的不足。因此,本文主要从这两个方面进行改进,并在此基础上设计和实现了一个面向大学领域的聚焦爬虫系统。该系统采用基于改进的Context Graphs方法的搜索策略和基于支持向量机(SVM)的目标页面分类器方法获取有用的资源。实验结果表明该系统在爬虫结果的收益率和准确率上分别提高了10%和8%。
How to obtain useful information from massive Web resource is crucial in the field of Web research.The main method to obtain domain-specific resources on Web is the strategy of focused crawler and this strategy only traverses pages related to the topic while neglects those irrelevant ones.However,the present strategy of focused crawler has deficiency in crawling efficiency and page quality.This article tries to make some improvements in the strategy from these two aspects,based on which we design and implement a university-oriented focused crawler system.The system uses a search strategy based on improved Context Graphs and a target page classifier based on Support Vector Machine(SVM)to acquire useful resources.The experimental results show that the system increases the harvest and accuracy of the crawling result by 10% and 8%respectively.