On Monday, Google announced that its new web indexing system, called Caffeine, is complete. The Internet giant promises that with the new system, results for web searches will be more comprehensive and will be updated more quickly, including news stories, blog entries, forum posts, and other website updates.
Google hopes that this update will appease Web users with a high demand for real-time content and real-time search results. “[P]eople’s expectations for search are higher than they used to be,” said Google software engineer Carrie Grimes. “Searchers want to find the latest relevant content and publishers expect to be found the instant they publish.”
Google’s new index created with Caffeine uses 100 million GB of storage in one database, and adds hundreds of thousands of gigs of new information per day.
Camille Ricketts of VentureBeat, who has worked for Google once upon a time, writes:
Google’s old indexing system — which made it famous and blew Yahoo and Ask.com out of the water when it launched a decade ago — took a layered approach. Top-tier content (determined based on linking) would be refreshed more often than lower-tier content. But each one of this refreshes required Google to review all web content, and discover and rank new pages. This obviously required a lot of time and computing power.
Ricketts explains that Caffeine breaks that task into “bite-size pieces” by looking at smaller portions of the Internet at a time, allowing for the quicker updates. This is the biggest change to Google’s search engine methodology in four years.
Google’s official blog entry on the update reminds us that search engine results aren’t the same as the actual live Web. What we’re actually searching is the index that Google has made of websites and pages across the Internet. The company made a video a while back to help explain how it works.
Source: “Google Completes Caffeine Update After Nearly a Year,” WebProNews, 06/09/10
Source: “Google launches Caffeine search indexing for a faster, more chaotic web,” VentureBeat, 06/08/10
Source: “Our new search index: Caffeine,” The Official Google Blog, 06/08/10
Image by mararie, used under its Creative Commons license.