Please do not spider search results in an automated fashion. It can generate significant system load. Instead, please use the XML Sitemap that we provide.