Information retrieval techniques: Lecture 33. The main topics covered in this chapter include: web search using IR; spiders (robots/bots/crawlers); what any crawler must do; what any crawler should do; search strategies; basic crawl architecture; restricting spidering; robot exclusion; . Please refer to the content of document. |