MapReduce: Simplifed Data Processing on Large Clusters

"MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. "
Statistics of jobs in august 2004, as usual impressive amount of data!!!
Number of jobs 29,423
Average job completion time 634 se
Machine days used 79,186 days
Input data read 3,288 TB
Intermediate data produced 758 TB
Output data written 193 TB

...

"It has been used across a wide range of domains within Google, including:&160;&160; large-scale machine learning problems,&160;&160; clustering problems for the Google News and Froogle products, MapReduce has been so successful because it makes it possible to write a simple program and run it ef ciently on a thousand machines in the course of half an hour, greatly speeding up the development and prototyping cycle. Furthermore, it allows programmers who have no experience with distributed and/or parallel systems to exploit large amounts of resources easily." more in the PDF here

You might like also

Scoop: the moon is made of cheese!
Yes the moon is made of cheese..... :-)In honor of the first manned Moon landing, which took place on July 20, 1969, Google added some NASA imagery to the Google Maps interface and present Google map moon.So Google localizing offering is now made of:Google map moon, this is where You can see with your own eyes that the moon is made of cheese (just use the zoom function to see the maximum of details). " Google Moon only has as …
5176 Days ago
Google toolbar soon for firefox!
Google HAS released its first version of the Google toolbar for Mozilla Firefox LINK HERE Since 4 years, it was only available for IE but it seems that Google Inc do not want to let the increasing number of Firefox users without their toolbar.In the meantime, You can still try the open-source Googlebar, developed and supported by Google fans:Update: the boutton "Agree and install" is not working, it is complaining about my firefox even with the latest version 1.0.4, so …
5190 Days ago
No Thumbnail was found
Google Video enables you to search a growing archive of televised content – everything from sports to dinosaur documentaries to news shows. In order to fully appreciate this new service, You will have to install a free Google viewer (Firefox and IE are supported) …
5198 Days ago
No Thumbnail was found
Google is the new Internet behemoth, snatching up small companies left and right. So, in this article, I ask: what tech gems are in the running for Google's growing subsidiary menagerie? To help predict, I will first take a look at who Google has acquired in the past and what Google has done for them, and then I'll throw out a few possibilities for Googlification and discuss where they might fit into Google's strategy. Read more at http://www.kuro5hin.org …
5212 Days ago
No Thumbnail was found
Internet search phenomenon Google has overtaken a swathe of venerable rivals to become the world's biggest media company by stock market value. ...Some analysts believe Google's shares are now overpriced, a rerun of the heady valuations seen during the internet boom of the late 1990s. ... Initially, Google got 10,000 queries per day compared with 200 million today. Read more HERE at BBC news …
5219 Days ago
No Thumbnail was found
Google Sitemaps is an easy way for you to help improve your coverage in the Google index. It's a collaborative crawling system that enables you to communicate directly with Google to keep us informed of all your web pages, and when you make changes to these pages. Read more at Google Sitemap …
5222 Days ago
No Thumbnail was found
i've came through some really good articles describing the future of satellite views search engine...first the two Giant: Google and M$Google + Keyhole = Google Earth at www.windypundit.comMSN Virtual Earth at blog.searchenginewatch.com The Nasa is also providing since August 2004, it's famous software World Wind which lets you zoom from satellite altitude into any place on Earth. Leveraging Landsat satellite imagery and Shuttle Radar Topography Mission data, World Wind lets you experience Earth terrain in visually rich 3D, just as …
5222 Days ago
Google desktop enterprise edition is out!
What are the features of Google desktop enterprise edition, and why You may prefer use it, as a simple user, to the Google desktop public editionIt is free! as all other Google search products...It support web history of the most famous alternate web browser: Firefox!It can search on all drive, and be connect to the a Google appliance server (if Your company has put one in its intranet)The data created by google desktop search on your disk are encrypted... (by …
5237 Days ago
No Thumbnail was found
Google has filled a patent that reveal some of the technologies behind their famous pageranking algorithm: from www.wwwcoder.com If you thought you cracked the Google Code and had Google all figured out … guess again.Google’s sweeping changes confirm the search giant has launched a full out assault against artificial link inflation & declared war against search engine spam in a continuing effort to provide the best search service in the world… and if you thought you cracked the Google Code …
5247 Days ago
No Thumbnail was found
The new edition of Linux Journal has a special insert: The GLAT (Google Labs Aptitude Test) is a Google recruiting quiz presented as a spoof of standardised aptitude tests. It is filled with math and Google-related trivia. On slashdot, a lot of users have tried to find the answers to the test. …
5334 Days ago