On Wednesday, November 18th while at the Web 2.0 Conference at the Jacob Javits Center in New York City, I found an interesting web crawling service provider, 80legs. Their product is a service platform for web crawling which provides their customers (web site owners) the ability to design their own crawls and the power to crawl up to 2 billion pages per day. What this means is that web crawler technology is now affordable for smaller companies and individuals -- clients will pay for only what they use (e.g., $2.00 / million pages, $.03 per CPU hour, etc.). Although there is much controversy in regards to the future of Web becoming more real time centric vs. indexed searches, many industry experts, including 80legs CEO Shion Deysarkar, believe indexed crawlers will be around for a long time as there is so much data compiled vs. a smaller amount of relevant real time populations.
Deysarkar actually got into web crawling because he felt it was one of the best use cases for the grid computing platform provided by his sister company, Plura Processing. "We initially thought of building specific tools that took advantage of web crawling, but we ultimately felt that providing a platform for web crawling would be more powerful and interesting," says Deysarkar.
When asked where he thinks Web 3.0 will be going versus Web 2.0, Deysarkar believes that Web 3.0 is all about turning the unstructured content (e.g., information retrieval from keyword searches via google, yahoo, etc.) on the web into structured data (e.g., databases from query html forms, flickr, etc.) . “At 80legs we can really help with web 3.0 because we can throw a whole lot of bandwidth and compute-power towards those big barriers to solve the web 3.0 problems,” says Deysarkar.Deysarkar actually got into web crawling because he felt it was one of the best use cases for the grid computing platform provided by his sister company, Plura Processing. "We initially thought of building specific tools that took advantage of web crawling, but we ultimately felt that providing a platform for web crawling would be more powerful and interesting," says Deysarkar.
To view Web 2.0 Facebook Album go to http://www.facebook.com/album.php?aid=122609&id=794796434&l=87f2b2fa43
For more information go to: http:80legs.com, http://80legs.wordpress.com/, or http://twitter.com/80legs
written by Gloria Buono Daly (c) 2009
Thanks for the good info.
ReplyDeleteI found interesting information here. thanks!
ReplyDeleterochester mn web designer
Good job - thanks so much!
ReplyDeleteBrooklyn Park Web Design