CSIRO Arch is an open source free extension of Apache Nutch, a popular general purpose search engine that is capable of indexing billions of web pages using clusters of computers.
Such environments typically have one or more web sites, with web content provided for external readers and internal use, and one or more "intranet" sites that provide content for internal use only.
Arch can be used to search both the external access and restricted access sites and produces extremely high quality search results.