cách truy cập khác "tương tự như" người sử dụng và nhìn thấy những gì các nội dung khác có sẵn. Phát triển các nguyên tắc phân loại kiểm soát, so folksonomies có thể tốn kém cả về thời gian của người sử dụng bằng cách sử dụng phân loại tư duy cứng nhắc, và về các chi phí phát triển để duy trì nó. | 146 Chapter 6 Intelligent web crawling Simpo PDF Merge and Split Unregistered Version - http from external sites which can then be used in your application. Search engines such as Google and Yahoo constantly crawl the web to gather data for their search results. HOW BIG IS In late July 2008 Google announced that they had detected more than a THE WEB trillion unique URLs on the web with the internet growing by several billion individual pages every day. Of course not all the content has been indexed by Google but a large portion has. To get a sense of the number of pages indexed by Google it is useful to look at the number of pages indexed by Google for a site type site website for example site to search for the pages indexed by Google for Facebook this number incidentally was more than 76 million pages as ofJuly 2008 . Other providers such as Alexa and Quantcast also provide useful data on the kinds of searches carried out on various sites. This chapter is organized in three sections First we look at the field of web crawling how it can be used in your application the details of the crawling process how the process can be made intelligent how to access pages that aren t retrievable using traditional methods and the available public domain crawlers that you can use. Second to understand the basics of intelligent focused crawling we implement a simple web crawler that highlights the key concepts related to web crawling. Third we use Apache Nutch an open sourceJava-based scalable crawler. We also discuss making Nutch distributed and scalable using concepts known as Hadoop and MapReduce. Introducing web crawling Web crawling is the automated process of visiting web pages with the aim of retrieving content. The content being extracted could be in many forms text images or videos. A web crawler is a program that systematically visits web pages retrieves content extracts URLs to other relevant links and then in turn visits .
