A 2nd generation spider to crawl any article site, automatic read title and article. the speed of spider is about 700 thousand documents per day, 22 million per month, and the maximize crawling speed is 450 per minute, avg 80 per minute, the memory cost is about 200 megabytes on each spider kernel.
Comments