Webspider

WebSpider is a program that provides a mechanism to get contents from the Web. With the extended classes, you can grab URLs from a specified base URL, analyze the contents of a list of URLs, and get specific files from web sites.

Also it is an Internet bot that systematically browses the WWW, typically for the purpose of Web indexing.