Alexa which is owned by Amazon.com released Alexa Web Search Platform Beta. This service could be the breakthrough if web developers can develop some applications to access and sort through the giant vault of information.
According to Alexa:
Since 1996, Alexa has been crawling and storing the Web at millions of pages per day. Alexa has also been building out the infrastructure to store and analyze the data and serve it to toolbars, browsers, and websites worldwide. Now, all of that infrastructure is yours to use via the Alexa Web Search Platform:
* Three online web snapshots of up to 100 terabytes each
* Powerful tools to sift through the content to create your own data set
* Upload, compile and run your own programs on a processing cluster across the data set
* Store your output on a storage cluster
* Integrate your data into a search index
* Access your new search via Amazon Web Services
This service is fairly affordable as well.