Building with big data

IN A short story called “On Exactitude in Science”, Jorge Luis Borges described an empire in which cartographers became so obsessive that they produced a map as big as the empire itself. This was so cumbersome that future generations left it to disintegrate. (“[I]n the western deserts, tattered fragments of the map are still to be found, sheltering some occasional beast or beggar.”)

As usual, the reality of the digital age is outpacing fiction. Last year people stored enough data to fill 60,000 Libraries of Congress. The world’s 4 billion mobile-phone users (12% of whom own smartphones) have turned themselves into data-streams. YouTube claims to receive 24 hours of video every minute. Manufacturers have embedded 30m sensors into their products, converting mute bits of metal into data-generating nodes in the internet of things. The number of smartphones is increasing by 20% a year and the number of sensors by 30%.

The McKinsey Global Institute (MGI) has no Borges-like qualms about the value of all these data. In a suitably fact-packed new report, “Big data: the next frontier for innovation, competition and productivity”, MGI argues that data are becoming a factor of production, like physical or human capital. Companies that can harness big data will trample data-incompetents. Data equity, to coin a phrase, will become as important as brand equity. MGI insists that this is not just idle futurology: businesses are already adapting to big data.

Companies are assembling more detailed pictures of their customers than ever before. Tesco, a British retailer, collects 1.5 billion nuggets of data every month and uses them to adjust prices and promotions. Williams-Sonoma, an American retailer, uses its knowledge of its 60m customers (which includes such details as their income and the value of their houses) to produce different iterations of its catalogue. Amazon, an online retailer, has claimed that 30% of its sales are generated by its recommendation engine (“you may also like”). The mobile revolution adds a new dimension to customer-targeting. Companies such as America’s Placecast are developing technologies that allow them to track potential consumers and send them enticing offers when they get within a few yards of a Starbucks.

The data revolution is disrupting established industries and business models. IT firms are nosing their way into the health-care market: Google Health and Microsoft HealthVault both allow consumers to track their health and record their treatments. Manufacturers are hastening their transformation into service companies: all those sensors allow them to monitor their products and see if they need repairing long before they break down. BMW uses sensor-data to tell its customers when their cars need to be serviced, for example. Insurance firms can now monitor the driving styles of their customers and offer them rates based on their competence (or recklessness) rather than their age and sex.

This revolution is changing government, too. Tax authorities are getting better at spotting spongers (for example, by flagging people who claim unemployment pay as well as occupational-injury benefits). Health services are mining clinical data to gauge the cost-effectiveness of drugs. After a detailed study of its clients, the German Federal Labour Agency managed to cut its annual spending by €10 billion ($14 billion) over three years while also reducing the length of time people spent out of work.

MGI argues that the data deluge could create a new wave of productivity growth. Properly used, big data could save the American health-care system $300 billion a year and the European public sector €250 billion. It could also enable retailers to increase their operating margins by 60%. But it is hard to read these figures without succumbing to Borges-style doubts. Will big companies and big governments use big data to trample on the little man? And is this mountain of data really as useful as MGI’s data-heads think?

Power to the little people

The McKinseyites provide good answers to the first question. The data revolution is clearly handing power to the little people as well as the big ones. You can now buy a device that will store all the world’s recorded music for just $600. Shoppers can use their mobile phones to scan bar codes to see if there is a better deal elsewhere. Citizens can use publicly available information to demand better public services. Britain’s Open Knowledge Foundation has used government databases to develop a useful site called wheredoesmymoneygo.org. Dr Foster Intelligence provides patients with information about the quality of health care.

But on the second question, they are silent. Big data has the same problems as small data, but bigger. Data-heads frequently allow the beauty of their mathematical models to obscure the unreliability of the numbers they feed into them. (Garbage in, garbage out.) They can also miss the big picture in their pursuit of ever more granular data. During the 2008 presidential campaign Mark Penn provided Hillary Clinton with reams of micro-data, thus helping her to craft micro-policies aimed at tiny slices of the electorate. But Mrs Clinton was trounced by a man who grasped that people wanted to feel part of something bigger. The winning slogans were vague and broad (“hope” and “change”).

The sheer size of today’s data banks means that companies need to be more careful than ever to treat data as a slave rather than a master. There is no substitute for sound intuition and wise judgment. But if firms can preserve a little scepticism, they can surely squeeze important insights from the ever-growing store of data. In the 1980s and 1990s retailers such as Walmart used their mastery of retailing data to launch the “big-box” revolution (huge out-of-town stores with ultra-low prices). Today’s big data will provide the raw material for further revolutions.