Surprise! Google Says The Web Is Really, Really Big

25 July 2008, 1:24 pm

Google has announced that its search index, which chronicles the vast majority of the internet, has hit a major milestone: 1 trillion unique URLS. That’s 1,000,000,000,000 pages - more than twice as many as there are stars in the Milky Way. The index has grown exponentially since Google first launched. In 1998, the site had ranked approximately 26 million pages. The site reached the 1 billion page milestone in 2000, and has since grown by more than one thousand fold. Over the last decade Google has changed the way it indexes the web in order to accommodate its massive scale. In the early days it would sporadically recrawl the web to create a new index, which would last for a fixed period of time. Now, the site is continuously monitoring the web, updating the index multiple times a day. Google’s post clarifies that the size of the internet really depends on what you classify as a unique webpage. In theory, there are an infinite number of pages, as many sites can automatically generate new ones (for example, a calendar could create a new page for every day in the future). The database also ignores pages that are near-exact reproductions of other sites. Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.... read more