Big Data and the Role of Intuition-Harvard Business Review

http://blogs.hbr.org/2013/12/big-data-and-the-role-of-intuition/?utm_source=Socialflow&utm_medium=Tweet&utm_campaign=Socialflow

HBR Blog Network

Big Data and the Role of Intuition

by Tom Davenport  |   8:00 AM December 24, 2013

• Comments (18)
•        

Many people have asked me over the years about whether intuition has a role in the analytics and data-driven organization. I have always reassured them that there are plenty of places where intuition is still relevant. For example, a hypothesis is an intuition about what’s going on in the data you have about the world. The difference with analytics, of course, is that you don’t stop with the intuition — you test the hypothesis to learn whether your intuition is correct.

Another place where intuition is found in analytical companies is in the choice of the business area where analytical initiatives are undertaken. Few companies undertake a rigorous analytical study of what areas need analytics the most! The choice of a target domain is typically based on the gut feelings of executives. For example, at Caesars Entertainment — an early and continuing user of analytics in its business — the initial focus was on analytics for customer loyalty and service. CEO Gary Loveman noted that he knew that Caesars (then Harrah’s) had low levels of customer loyalty across its nationwide network of casinos. He had also done work while at Harvard Business School on the “service profit chain” — a theory that companies that improve customer service can improve financial results. While the theory had been applied and tested in several industries, it hadn’t been applied to gaming firms. But Loveman’s intuition about the value of loyalty and service was enough to propel years of analytics projects in those areas.
Of course, as with hypotheses, it’s important to confirm that your intuitions about where to apply analytics are actually valid. Loveman insists on an ROI from each analytics project at Caesars. Intuition plays an important role at the early stages of analytics strategy, however. In short, intuition’s role may be more limited in a highly analytical company, but it’s hardly extinct.
But how about with big data? Surely intuition isn’t particularly useful when there are massive amounts of data available for analysis. The companies in the online business that were early adopters of big data — Google, Facebook, LinkedIn, and so forth — had so much clickstream data available that no one needed hunches any more, correct?
Well, no, as it turns out. Major big data projects to create new products and services are often driven by intuition as well. Google’s self-driving car, for example, is described by its leaders as a big data project. Sebastian Thrun, a Google Fellow and Stanford professor, leads the project. He had an intuition that self-driving cars were possible well before all the necessary data, maps, and infrastructure were available. Motivated in part by the death of a friend in a traffic accident, he said in an interview that he formed a team to address the problem at Stanford without knowing what he was doing.
At LinkedIn, one of the company’s most successful data products, the People You May Know (PYMK) feature, was developed by Jonathan Goldman (now at Intuit) based on an intuition that people would be interested in what their former classmates and colleagues are up to. As he put it in an interview with me, he was “playing with ideas about how to help people build their networks.” That certainly sounds like an intuitive process.
Pete Skomoroch, who became Principal Data Scientist at LinkedIn a few years after PYMK was developed, believes that creativity and intuition are critical to the successful development of data products. He told me in an interview this week that companies with the courage to get behind the intuition of data scientists — without a lot of evidence yet that their ideas will be successful — are the ones that will develop successful data products. As with traditional analytics, Skomoroch notes that you have to eventually test your creativity with data and analysis. But he says that it may take several years before you know if an idea will really pay off.
So whether you’re talking about big data or conventional analytics, intuition has an important role to play. One might even say that developing the right mix of intuition and data-driven analysis is the ultimate key to success with this movement. Neither an all-intuition nor an all-analytics approach will get you to the promised land.

Independent Report: Pivotal Data Dispatch Reached Payback in 3 Months with 377% Annual ROI at NYSE Euronext

Independent Report: Pivotal Data Dispatch Reached Payback in 3 Months with 377% Annual ROI at NYSE Euronext

December 18, 2013 • Case Studies • By Stacey Schneider

This week, Nucleus Research compiled an independent report profiling how NYSE Euronext solved the challenge of big data in their organization with big returns using Pivotal technologies that we have now released to the market called Pivotal Data Dispatch.  By all accounts, NYSE Euronext was caught between the proverbial rock and a hard place with their data requirements. With their stock trades representing one third of the world’s equities volume, and federal regulations requiring them to keep 7 years history, by 2006 their data volume was staggering. The cost to maintain this data and the penalties the business was paying with the latency of data results was unacceptable.
However, instead of following the norm in the industry and continuing to invest in monolithic data stores that had fixed data ceilings, NYSE Euronext read the future and decided to embrace modern big data strategies early, resulting in a scalable and cost affordable data solution. As a result, they are now recognized as a early leader in the big data market, achieving payback for their efforts in just 3 months and enjoying a staggering 377% annual ROI.

Background

To give an idea of the scope of the problem, by 2007, when NYSE Euronext started to look at viable solutions to their growing challenge, even the lower cost alternative of Massive Parallel Processing (MPP) still had a market price of about $22,000 a terabyte. At the time, they were generating an average of ¾ of a terabyte a day. Knowing data volumes were only going to grow, and they needed to keep 7 years on hand, storage of the data alone was a crippling prospect.
Also, actually using the data was problematic. Typical queries had to be performed by experienced DBAs so as to not upset the infrastructure and average query time took 7 hours, and frequently required additional filters and queries. NYSE Euronext executives frequently waited up to 3 weeks to receive the results of their queries. With data not being ubiquitous and available in even near real-time, the NYSE knew it was missing opportunities—big opportunities as it turns out.

The Solution

Their first attempt to solve their data challenges started in 2007. NYSE Euronext set out to develop a solution built on our Greenplum technology (inherited from EMC in our spin-out) and IBM Netezza. The idea was to build a Data Lake, where active and historical data could be self-provisioned on-demand by the data analysts. Separate from real-time operations, data analysts were free to use the data at will.
Data volumes continued to grow and by 2010, NYSE Euronext was collecting 2 TB a day. While the cost of MPP processing was becoming cheaper, the growth rate was not affording any further savings. Their estimated costs were approaching $4.5 million a year to just store data. However, since the data could be federated across commodity hardware, this solution was still estimated to be 1/40th of a traditional data storage in a analytics environment, and provided broader access to analysts—something that provided enormous value to daily operations.
With that in mind, NYSE Euronext took a leap that many organizations to date have not. They went big on big data. With help from vendors like Pivotal, they built a system, now publicly available called Data Dispatch, that should be treated as a model for the enterprise.

Key Benefits

For full financial disclosure on the economics, please read the full report by Nucleus Research. However here are some notable highlights from the report itself speak volumes on this aggressive approach to big data, and why big data leaders like NYSE Euronext are showing the way to the market that investing in strategies to make it ubiquitous and real-time really pay off:

• Power user productivity. With IT eliminated from the active process of harvesting data, business users are not only in control, they are empowered to use data daily to fully understand their markets. With the data available, they tend to use it more and improve business decisions.
• Increased productivity. With the back and forth between IT and the business eliminated in every data request, both the business and IT can focus on their own areas of expertise. Data requests are fulfilled more quickly with the person who knows what they are actually after and what the data means in the drivers seat. IT also manages to service the business more effectively, while reducing the amount of direct help. This is a win-win for everyone.
• Reduced IT labor costs. The Pivotal Data Dispatch tool services about 2000 data requests each month. Historically each request took a DBA about 1 hour, so this means approximately 2000 hours or over 83 man days of DBA work can be refocused into other areas of their massive data infrastructure.
• Reduced decision latency. With the data request cycle compressing from 3 weeks to hours, the nearly 250 data analysts at NYSE Euronext are by default working on fresher data. This results in reduced decision latency, allowing them to use near real-time data to make important inferences and prove imperically what their markets need.

More on NYSE Euronext and Pivotal Data Dispatch

• Product page for Pivotal Data Dispatch including how to get it for your enterprise.
• Pivotal and NYSE Euronext announce Pivotal Data Dispatch is available. Also see the press release from that day.
• Pivotal and NYSE kicked off our partnership in providing this solution by ringing the bell at NYSE. Read more about that event or see the video.
• See the video why NYSE Euronext chief data officer Steve Hirsch made his technology choices.
• Read the blog post on how Making Big Data Real-time and Ubiquitous Is Key to Enterprise Adoption.

- See more at: http://blog.gopivotal.com/case-studies-2/independent-report-pivotal-data-dispatch-reached-payback-in-3-months-with-377-annual-roi-at-nyse-euronext#sthash.lCt6gLwH.dpuf

Data is Worthless if You Don't Communicate It

by Tom Davenport  |   8:00 AM June 18, 2013

There is a pressing need for more businesspeople who can think quantitatively and make decisions based on data and analysis, and businesspeople who can do so will become increasingly valuable. According to a McKinsey Global Institute report on big data, we'll need over 1.5 million more data-savvy managers to take advantage of all the data we generate.

But to borrow a phrase from Professor Xiao-Li Meng — formerly the Chair of the Statistics Department at Harvard and now Dean of the Graduate School of Arts and Sciences — you don't need to become a winemaker to become a wine connoisseur. Managers do not need to become quant jocks. But to fill the alarming need highlighted in the McKinsey report, most do need to become better consumers of data, with a better appreciation of quantitative analysis and — just as important — an ability to communicate what the numbers mean.
Too many managers are, with the help of their analyst colleagues, simply compiling vast databases of information that never see the light of day, or that only get disseminated in auto-generated business intelligence reports. As a manager, it's not your job to crunch the numbers; but — as Jinho Kim and I discuss in more detail in Keeping Up with the Quants — it is your job to communicate them. Never make the mistake of assuming that the results will "speak for themselves."
Consider the cautionary tale of Gregor Mendel. Although he discovered the concept of genetic inheritance, his ideas were not adopted during his lifetime because he only published his findings in an obscure Moravian scientific journal, a few reprints of which he mailed to leading scientists. It's said that Darwin, to whom Mendel sent a reprint of his findings, never even cut the pages to read the geneticist's work. Although he carried out his groundbreaking experiments between 1856 and 1863 — eight years of painstaking research — their significance was not recognized until the turn of the 20th century, long after his death. The lesson: if you're going to spend the better part of a decade on a research project, also put some time and effort into disseminating your results.
One person who has done this very well is Dr. John Gottman, the well-known marriage scientist at the University of Washington. Gottman, working with a statistical colleague, developed a "marriage equation" predicting how likely a marriage is to last over the long term. The equation is based on a couple's ratio of positive to negative interactions during a fifteen minute conversation on a "difficult" topic such as money or in-laws. Pairs who showed affection, humor, or happiness while talking about contentious topics were given a maximum number of points, while those who displayed belligerence or contempt received the minimum. Observing several hundred couples, Gottman and his team were able to score couples' interactions and identify the patterns that predict divorce or a happy marriage.
This was great work in itself, but Gottman didn't stop there. He and his wife Julie founded a non-profit research institute and a for-profit organization to apply the results through books, DVDs, workshops, and therapist training. They've influenced exponentially more marriages through these outlets than they could possibly ever have done in their own clinic — or if they'd just issued a press release with their findings.
Similarly, at Intuit, George Roumeliotis heads a data science group that analyzes and creates product features based on the vast amount of online data that Intuit collects. For his projects, he recommends a simple framework for communicating about each analysis:

1. My understanding of the business problem
2. How I will measure the business impact
3. What data is available
4. The initial solution hypothesis
5. The solution
6. The business impact of the solution

Note what's not here: details on statistical methods used, regression coefficients, or logarithmic transformations. Most audiences neither understand nor appreciate those details; they care about results and implications. It may be useful to make such information available in an appendix to a report or presentation, but don't let it get in the way of telling a good story with your data — starting with what your audience really needs to know.

More blog posts by Tom Davenport

Are You Data Driven? Take a Hard Look in the Mirror.

by Thomas C. Redman  |  11:00 AM July 11, 2013

The term "data driven" is penetrating the lexicon ever more deeply these days. Data Driven was the title of my latest book, and recent academic work shows that companies that regard themselves as "data driven," are measurably more profitable than those that aren't. So becoming data-driven is clearly a worthwhile endeavor. Yet for all the attention, I've yet to see any clear criteria by which leaders can benchmark themselves and their organizations to figure out what they need to do better.
In my view, the essence of "data-driven" is making better decisions up and down the organization chart. Over the years I've had the good fortune to work with plenty of individual decision-makers and groups, some terrific and some simply awful. From that work, I've distilled twelve "traits of the data-driven." (It bears mention that the data-driven also avoid some self-destructive traits; I'll take these up in another post.)
Traits of the Data-Driven
The data-driven:

• Make decisions at the lowest possible level
• Bring as much diverse data to any situation as they possibly can.
• Use data to develop a deeper understanding of their worlds.
• Develop an appreciation for variation
• Deal reasonably well with uncertainty
• Integrate their ability to understand data and its implications and their intuitions.
• Recognize the importance of high-quality data and invest to improve.
• Are good experimenters and researchers.
• Recognize that decision criteria can vary with circumstances.
• Recognize that making a decision is only step one.
• Work hard to learn new skills and bring new data and new data technologies (big data, predictive analytics, metadata management, etc) into their organizations.
• Learn from their mistakes.

All of these traits are important. And most are self-evident. Only a few require further explanation. First, data-driven companies work to drive decision-making to the lowest possible level. One executive I spoke to described how he thought about it this way: "My goal is to make six decisions a year. Of course that means I have to pick the six most important things to decide on and that I make sure those who report to me have the data, and the confidence, they need to make the others." Pushing decision-making down frees up senior time for the most important decisions. And, just as importantly, lower-level people spend more time and take greater care when a decision falls to them. It builds the right kinds of organizational capability and, quite frankly, appears to create a work environment that is more fun.
Second, the data-driven have an innate sense that variation dominates. Even the simplest process, human response, or most-controlled situation varies. While they may not use control charts, they know that they have to understand that variation if they are going to understand what is going on. One middle manager expressed it to me this way, "When I took my first management job, I agonized over results every week. Some weeks we were up slightly, others down. I tried to take credit for small upturns and agonized over downturns. My boss kept telling me to stop — I was almost certainly making matters worse. It took a long time for me to learn that things bounce around. But finally I did."
Third, the data-driven place high demands on their data and data sources. They know that their decisions are no better than the data on which they are based, so they invest in quality data and cultivate data sources they can trust. As a result, when a time-sensitive issue comes up they are prepared. High-quality data makes it easier to understand variation and reduces uncertainty. Success is measured in execution, and high-quality data makes it easier for others to follow the decision-makers logic and align to the decision.

Further, as one executes, one acquires more data. So the data-driven are constantly re-evaluating, refining their decisions along the way. They are quicker than others to pull to plug as when the evidence suggests that a decision is wrong. To be clear, it doesn't appear that the data-driven "turn on a dime"; they know that is not sustainable. Rather, they learn as they go.
Now take that hard look in the mirror. Look at the list above and give yourself a point for every trait you follow regularly and half a point for those you follow most — but not all — of the time. Be hard on yourself. If you can only cite an instance or two, don't give yourself any credit.
Unless you're one of the rare few that truly score seven or more, you need to improve. While each person and organization is different, I'd first recommend starting by pushing decision-making down the organization. I've already noted the benefits. It may be tough and counterintuitive, especially for managers that want to feel in control, but it's worth the effort.
Second, invest in quality data. Frankly, you simply cannot be data-driven (or do anything consistently well for that matter) without high-level of trust in your data and data sources. You're reduced to your intuition alone, the antithesis of the goal here. Quality data is a necessity.
Now, take one more step. You've taken a hard look at yourself. Engage your management team in doing exactly the same thing for your organization.

More blog posts by Thomas C. Redman

To Go from Big Data to Big Insight, Start with a Visual

by Sinan Aral. Visualization by Nikolaos Hanselmann  |   8:00 AM August 27, 2013

Although data visualization has produced some of the most captivating artistic displays in recent memory, some of which have found their way into exhibits at the New York Museum of Modern Art and countless art installations around the world, business leaders are asking: is data visualization actionable?
I think so. In my role as the Scholar-in-Residence at The New York Times R&D Lab, I am collaborating with one of the world's most advanced digital R&D teams to figure out how we can draw actionable insights from big data.
How big? Massive: We are documenting every tweet, retweet, and click on every shortened URL from Twitter and Facebook that points back to New York Times content, and then combining that with the browsing logs of what those users do when they land at the Times. This project is a relative of the widely noted Cascade project. Think of it as Cascade 2.0.
We're doing this to understand and predict when an online cascade or conversation will result in a tidal wave of content consumption on the Times, and also when it won't. More importantly we are interested in how the word-of-mouth conversation drives readership, subscriptions, and ad revenue; how the Times can improve their own participation in the conversation to drive engagement; how we can identify truly influential readers who themselves drive engagement; and how the Times can then engage these influential users in a way that complements the users' own needs and interests. Do it, and we can turn that statistical analysis, as you'll see below, into elegant, artistic real time data streams.
Handling the streams, archiving the sessions and storing and manipulating the information are in themselves herculean tasks. But the even bigger challenge is transforming beautiful, big data into actionable, meaningful, decision-relevant knowledge. We've found that visualization is one of the most important guideposts in this search for knowledge, essential to understanding where we should look and what we should look for in our statistical analysis.
For example, here are three visualizations that have helped us gain knowledge. They show cascades of the tweets and retweets as lines and dots about three different Times articles over time, combined with the click-through volume on each article synced in time and displayed as a black graph under each cascade. Each panel tells a different story about engagement with the content.

For the first article, there is a sizable Twitter conversation and several large spikes in traffic. But the click-through volume seems independent of the Twitter conversation: The largest spike in traffic, highlighted in blue, occurs when there is very little Twitter activity. In this case, a prominent link on a blog or a news story that referred to the story, rather than the Twitter conversation itself, is probably driving the traffic.

On the second article, the Twitter conversation is intense. There are many, tweets and retweets of the article — yet the article itself gets very little traffic. People are talking about the article on Twitter, but not reading it. This sometimes happens when the main message of an article sparks a debate or a conversation that can happen without the content of the article being that important, for example, when a timely piece of news contains little analysis or editorial content, or when the conversation or debate gets away from the article and evolves its own independent content.

In the third and final article, an intense Twitter conversation moves in lockstep with engagement. As people tweet and retweet the article, their followers are clicking through and engaging with the content itself. This tight relationship between the online conversation and the website traffic is most pronounced when the three "influencers" tagged in the figure inspire the two largest spikes in traffic over the engagement lifecycle of the article.
With just these three data visualizations, we've gained understanding in important nuances about so-called virality. The relationship between online word-of-mouth conversations and engagement isn't as simple as something just "going viral." Different patterns emerge with different types of content.
Still, the visuals cannot tell the whole story. We see some clear correlations here, but complex conditional dependencies and temporal and network autocorrelation make it necessary to build more sophisticated causal statistical models that will generate true, reliable insights about word-of-mouth influence.
What these visuals do help with is getting us to know where to look and what questions to ask of the data. That is, we can't build the more complex models until we know the most suitable places for building them. These visuals give us some of that inisght.
Cascade 2.0 will be built on sophisticated analytics, and it will require data visualization. Asking important questions and avoiding unnecessary ones is essential to moving forward effectively and efficiently with big data. Without visualization, we are much less efficient in getting to the questions whose answers teach us something. That's why visualizing data must be one of the most important tools for data scientists. It is our torch in a thick, dark forest.

More blog posts by Sinan Aral. Visualization by Nikolaos Hanselmann

http://blogs.hbr.org/cs/2013/08/visualizing_how_online_word-of.html

NSA Code Cracking Puts Google, Yahoo Security Under Fire

By Allan Holmes - Sep 6, 2013 4:19 PM ET

Matthias Balk/picture-alliance/dpa/AP Photo

The agency has fulfilled a decades-long quest to break the encryption of e-mail, online purchases, electronic medical records and other Web activities, the New York Times, the U.K.’s Guardian and ProPublica reported yesterday.

Disclosures that the U.S. National Security Agency can crack codes protecting the online traffic of the world’s largest Internet companies will inflict more damage than earlier reports of complicity in government spying, according to technology and intelligence specialists.

Sept. 6 (Bloomberg) -- Full episiode of "Bloomberg West." Guests: Veracode VP of Research Chris Eng, CGV Capital Partner’s Glenn Solomon and Bloomberg's Allan Holmes, Stephen Engle, Willem Marx and Nela Richardson. (Source: Bloomberg)

The agency has fulfilled a decades-long quest to break the encryption of e-mail, online purchases, electronic medical records and other Web activities, the New York Times, the U.K.’s Guardian and ProPublica reported yesterday. The NSA also has been given access to -- or found ways to enter -- databases of major U.S. Internet companies operating the most popular e-mail and social-media platforms, the news organizations reported.
The reports, based on documents from former intelligence contractor Edward Snowden, emerged amid an expanding debate over whether NSA surveillance activities undermine civil liberties. The revelations raise fresh questions about the security of data held by companies including Google Inc. (GOOG), Facebook Inc. (FB) and Microsoft Corp. (MSFT) just as more commerce shifts online.
“This is a fundamental attack on how the Internet works,” Joseph Lorenzo Hall, senior staff technologist at the Washington-based policy group Center for Democracy & Technology, said in an interview. “Secure communications technologies are the backbone of e-commerce” including the transfer of medical records and financial exchanges.
“People in business will either not engage in those activities, or find other ways,” Hall said.

Snowden Revelations

The reports in the Guardian, the Times and the non-profit ProPublica news website said that NSA spends more than $250 million a year on a program working with technology companies to “covertly influence” product designs. The reports didn’t name the companies cooperating with the NSA and didn’t describe the extent to which the agency was using its code-breaking capability on the Internet.
The classified documents are the latest that Snowden has exposed revealing previously secret NSA programs. The 30-year-old former employee of government contractor Booz Allen Hamilton Holding Corp. (BAH) faces espionage charges in the U.S. and is in Russia under temporary asylum.
President Barack Obama’s administration has been coping with increasing public backlash over U.S. spying activities since top-secret documents leaked by Snowden began emerging in June. Foreign governments, including Brazil and Germany, have objected to U.S. surveillance and spying operations.

Obama Remarks

Brazilian authorities canceled a trip to Washington this week to prepare for President Dilma Rousseff’s state visit in October to protest allegations the U.S. spied on communications between officials in Latin America’s largest economy.
Obama told reporters at a news conference today in St. Petersburg, Russia, that “what we do is similar to what countries around the world do with their intelligence services.” He said that he had met with Rousseff and Mexican President Enrique Pena Nieto during the G-20 summit to “discuss the allegations made in the press about NSA.”
The U.S. president also said that the nation should review the spy programs to determine if they should continue. “The nature of technology and the legitimate concerns around privacy and civil liberties means that it’s important for us, on the front end, to say, all right, are we actually going to get useful information here,” he said. “And if not, or how useful is it, if it’s not that important, should we be more constrained in how we use certain technical capabilities.”

Lost Business

U.S. companies that are “household names” gave the NSA access to all communications, said Cedric Leighton, a former Air Force intelligence officer and a former NSA training director. Companies gave easy access to NSA because their managers believed it was necessary and they trusted that the government agency wouldn’t do anything wrong, Leighton said.
“But this takes the cake,” he said. “This has done a lot of damage to our ability to collect intelligence.”
Even before the latest reports, U.S. technology companies offering network infrastructure services such as cloud computing and popular social-networking applications were facing the prospect of losing business overseas.
Industry groups sounded alarm at the revelations. “This is a tragic case of myopia on the part of the NSA, and the surveillance infrastructure throughout the government,” said Ed Black, president of the Computer & Communications Industry Association, a Washington trade group, in a statement today. “By secretly embedding weaknesses into encryption systems in order to create a ’back door’ for surveillance access, the NSA creates a road map for similar cyber-incursions by others with less noble intentions.”

‘Hugely Disappointing’

Companies offering cloud services -- in which businesses pay a third party to provide databases, storage and computing power -- may lose as much as $35 billion by 2016 as foreign companies avoid U.S. solutions because of the fear the NSA may have access to the data, according to a study released last month by the Information Technology & Innovation Foundation.
“This is a hugely disappointing revelation,” Daniel Castro, author of the Washington-based group’s study, said in an e-mail. “This most recent news will certainly contribute to the perception that U.S. Internet companies cannot be trusted.”
Michael Birmingham, a spokesman for the Office of the Director of National Intelligence, which oversees U.S. intelligence agencies, declined to comment on the reports.
“Anything that yesterday’s disclosures add to the ongoing public debate is outweighed by the road map they give to our adversaries about the specific techniques we are using to try to intercept their communications in our attempts to keep America and our allies safe,” according to a statement posted to the national intelligence office’s website today.

More Legal Protection

Obama and officials from intelligence agencies have defended the NSA’s surveillance programs as essential to thwarting possible terrorist attacks. U.S. officials have told lawmakers that the programs are legal and subject to oversight by a federal court and members of Congress.
Sixty-six percent of U.S. Internet users polled believe current laws aren’t good enough to protect people’s privacy online, according to a survey released yesterday by the Pew Research Center. That compared with 24 percent who believe current laws provide reasonable protections, Pew said. The July 11-14 telephone survey of 792 Internet users has a margin of error of plus or minus 3.8 percentage points.

Obama Measures

Amid increasing public unease over the surveillance programs, Obama said Aug. 9 he would ask Congress to change the section of the Patriot Act allowing collection of telephone records, to increase oversight and transparency.
The president also said he’ll propose a legal advocate to serve as an adversary when spy agencies make requests in the secret sessions of the Foreign Intelligence Surveillance Court, which vets requests for electronic eavesdropping. Last week, he met for the first time with a panel he requested to review U.S. surveillance initiatives.
Leslie Miller, spokeswoman for Google, said in an e-mail that the company doesn’t “provide any government, including the U.S. government,” access to its systems.
“As for recent reports that the U.S. government has found ways to circumvent our security systems, we have no evidence of any such thing ever occurring,” Miller said. “We provide user data to governments only in accordance with the law.”

‘Legally Obligated’

Microsoft provides the U.S. government information when “legally obligated to comply with demands,” according to a July 15 blog post by Brad Smith, general counsel for the Redmond, Washington-based company. “To be clear, we do not provide any government with the ability to break the encryption, nor do we provide the government with the encryption keys.”
Smith’s comments apply to yesterday’s reports, said Dominic Carr, a spokesman for Microsoft.
“We are unaware of and do not participate in such an effort, and if it exists, it offers substantial potential for abuse,” Suzanne Philion, spokeswoman for Sunnyvale, California-based Yahoo! Inc. (YHOO), said in an e-mail today. “Yahoo zealously defends our users’ privacy and responds to government requests for data only after considering every applicable objection and in accordance with the law.”
Facebook’s spokeswoman Sarah Feinberg didn’t respond to requests for comment.
Google, Microsoft, Apple Inc. (AAPL) and 19 other technology companies sent a letter in July to Obama and congressional leaders urging that the companies be allowed to report statistics concerning requests for user data received from intelligence agencies.
To contact the reporter on this story: Allan Holmes in Washington at aholmes25@bloomberg.net
To contact the editor responsible for this story: Bernard Kohn at bkohn2@bloomberg.net

Big Brother Apparently Needs Glasses

By Megan McArdle Sep 4, 2013 3:28 PM 

Periodically, I read these stories about how marketers are really Big Brother, and I wonder how much they know about me. I assume you’ve read the stories, too, and seen the Very Special Episodes of "Law and Order" where we learn that companies know everything about us, from how much we overspent on wine at our anniversary dinner to the sexual problems we’re doing Google searches for.

In practice, I’ve found marketers’ predictive abilities underwhelming. Between my leaving Newsweek and starting at Bloomberg View, my husband and I spent a week touring Virginia, including a stay at Colonial Williamsburg. For months after that trip, it seemed as if every other Google ad I saw was for … Colonial Williamsburg. I mean, we liked it and everything. But frankly, one overpriced meal at the King’s Arms Tavern is enough to do you for quite a while.

Yet all the rumors on the Internet whisper of more powerful marketers, ones who could tell you your shoe size, recommend the perfect cheese to serve with those grapes you just purchased and remind you that you really need to call your grandmother, because it’s been 21 days. Perhaps the ads for Colonial Williamsburg were some sort of elaborate feint, the Operation Quicksilver of modern marketing. Wheels within wheels, my friend, wheels within wheels.

So it was with excitement and some trepidation that I learned of AboutTheData.com, which lets you see what marketers know about you. Here’s the New York Times description of the sort of data available to the shadowy forces behind the credit card offers and catalogs piling up in your mailbox:

The data on the site, called AbouttheData.com, includes biographical facts, like education level, marital status and number of children in a household; homeownership status, including mortgage amount and property size; vehicle details, like the make, model and year; and economic data, like whether a household member is an active investor with a portfolio greater than $150,000. Also available will be the consumer’s recent purchase categories, like plus-size clothing or sports products; and household interests like golf, dogs, text-messaging, cholesterol-related products or charities.

In the New York Times article, the site is rather too familiar with the personal details of its test subject:

The Acxiom Corporation, a marketing technology company that has amassed details on the household makeup, financial means, shopping preferences and leisure pursuits of a majority of adults in the United States, knows that Mr. Howe is 45, married with children, the owner of a house in the 2,500-square-foot range, and is interested, among other things, in tennis, domestic travel, cooking, crafts, sweepstakes and contests. Those intimate details, Mr. Howe says, are entirely accurate.

My heart pounding like a junior invited to senior prom, I went to the site and entered my own details. And all I can say is, the Colonial Williamsburg ads seem sophisticated in comparison. At least I actually went there.
According to AboutTheData.com, I’m between 40 and 41 years old. This would be more impressive if I hadn’t given them my birthday to register. It also knows that I’m a woman. A single Italian woman whose highest level of completed education is high school. (Ethnicity, the site helpfully notes, is inferred from last name. With a name like “McArdle,” you can see what an easy mistake that was to make.)

As far as AboutTheData.com is concerned, I don’t own a home -- even though I had to give them an address to register. An address where my husband and I are titled on a house and a mortgage. It’s also unaware that I own a car, presumably because I don’t have a loan on it.

What about the economic data? AboutTheData.com believes my household income to be about half of my actual salary, possibly because we’re kinda cheap. Though not quite as cheap as AboutTheData.com thinks; according to them, in the last 24 months, I’ve spent a total of $1,126 on online and offline purchases.
And here’s what AboutTheData.com thinks I’ve bought:

Source: Aboutthedata.com

I know a cold shiver is supposed to go up my spine at finding out someone knows I’ve been pricing window treatments. (Though not, contra this graphic, actually buying any.) But any icy tendrils are immediately melted by the conclusion that I’ve been buying art. I have no idea where the site got this idea, unless a new ice cream machine counts. And now that I mention it, how come it doesn't know I’ve got a new kitchen appliance? If there’s one thing that the Internet ought to know about me, it’s that I’ve always got a new kitchen appliance. Frankly, this list could practically have been assembled just knowing my age and gender.
So time to put away my panic about the Panopticon Society, at least for a while. Big Brother may be watching, but he apparently needs glasses. 

The Most Expensive Countries for IPhones - Brazil #1

GLOBALTECH TODAY Covering the 57 million square miles outside Silicon Valley.

September 6, 2013

For more, visit Bloomberg.com/global-tech >>

The Most Expensive Countries for IPhones By Bloomberg Rankings AROUND THE WORLD Lenovo Aims to Beat HP in Europe PCs by 2015 Apple Examining Claims of Labor Violations Google London Office With Track Approved Panasonic Sees Sales Growth in Europe Read more news at Bloomberg Global Tech >> If you think you're already paying too much for an iPhone, be glad you're not an Apple fan in France or Germany. The price of a 16-gigabyte iPhone 5 in those countries is about 40 percent more expensive than one sold in the U.S. Bloomberg Rankings analyzed prices in 34 international stores Apple lists on its website. The prices were then converted to U.S. dollars on Sept. 3. In the U.S., Apple sells an unlocked iPhone 5 without a mobile contract for $649. So which country was the priciest? Try Brazil, where you'll pay about $368 more than in the U.S.    To see the full rankings of the 20 most expensive countries, check out our slideshow.

online gambling $14bi profits in Europe

 BIG DATUM

$14	billion is about how much in revenues online gambling generated in Europe last year. The U.S. market may reach about half that amount by 2017.

SEPTEMBER 7, 2013, 2:13 PM Comment

A Brief History of Data Revolutions in Economics

By STEVE LOHR

Peter Morgan/ReutersHollerith tabulating machines, which were first used in the 1890 census, were the founding technology of the company that became I.B.M.

• FACEBOOK
• TWITTER
• GOOGLE+
• SAVE
• E-MAIL
• SHARE
• PRINT

Measurement is the lifeblood of economics. And today, economists are optimistic that all the new digital sources of information — Big Data, in the popular shorthand — will open the door to breakthroughs in the ability to measure, monitor and perhaps guide the economy. That is the subject of an article I wrote, which was published on Sunday.

Advances in technology have long fueled improvements in the measurement of populations and economic activity. Herman Hollerith’s punched-card tabulating machines, first used in the 1890 census, made it possible to efficiently count and categorize America’s surging population at the end of the 19th century. (And the Hollerith tabulating machines were the founding technology of the company that became I.B.M.)

In the early 20th century, the spread of automobiles and telephone service made it possible to expand and increase the frequency of the household surveys — done in person and by telephone — used to track employment and job losses.

In the 1960s, mainframe computers made it possible to mine and process larger and larger amounts of data. And the adoption of affordable digital tape storage was another vital technology, said James Poterba, president of the National Bureau of Economic Research and a professor at the Massachusetts Institute of Technology.

Vast amounts of government data — the equivalent of truckloads of punched cards — could then be stored on a few computer tape reels and easily copied. Those portable tapes could then be fairly easily sent off to curious economists and social scientists at universities across the country. “That opened the floodgates to wholesale statistical analysis of government data sets,” Mr. Poterba said.

Technology may provide tools, but the incentive for innovation in measurement is often needed. In June 1930, based on scattered reports of improvement, Herbert Hoover declared, “The Depression is over.” At the time, there was no measure of national economic activity, which we now call the gross domestic product, or G.D.P. And there was no reliable information on unemployment. Policy-making was based on guesswork, gut feel and wishful thinking.

To address the gap, the government hired economists and statisticians to come up with a scientific method for measuring the national economy. In 1934, a team led by Simon Kuznets published their report on how to calculate national income, and the field of econometrics took a giant step forward. “That work was propelled almost entirely by the necessity of the Great Depression,” said Claudia Goldin, an economist and historian at Harvard.

Economists, both in academia and in business, see a pressing need these days behind the flurry of Big Data research. After all, they note, the United States is emerging fitfully from the worst recession since the Depression. “We have more data, better tools and an urgent demand for greater understanding,” said Jed Kolko, chief economist at a Trulia, an Internet start-up that mines residential real estate data.

More Data Can Mean Less Guessing About the Economy

By STEVE LOHR

DAVID TABOR, 37 and a civil engineer, runs an eight-person home-inspection company in Hurricane, W.Va.

Other than their owners, most small businesses have no employees. His is one of the 4.3 million that do have them, and that employ fewer than 20 people each. And while these companies collectively produce roughly 15 percent of the nation’s economic output, their activities aren’t captured by the official numbers in a timely or detailed way.

Yet this measurement shortfall in the small-business sector, and a series of other information gaps in the economy, may be overcome by what experts say is an emerging data revolution — Big Data, in the current catchphrase. The ever-expanding universe of digital signals of behavior, from browsing and buying on the Web to cellphone location data, is grist for potential breakthroughs in economic measurement. It could produce more accurate forecasting and more informed policy-making — more science and less guesswork.

“We’re seeing the emergence of new sets of data and knowledge that we’ve never had before,” says James Poterba, president of the National Bureau of Economic Research and a professor at the Massachusetts Institute of Technology. “It’s a real opportunity for policy makers.”

In the small-business case, Mr. Tabor’s company, American Home Inspectors and Engineering Assessments Inc., is one of more than 200,000 that have allowed Intuit, the software maker, to gather data on their use of Intuit’s online payroll or online accounting products for research on employment and sales trends. The data are stripped of identifying information, and, in asking permission, Intuit also emphasizes that it uses the data to improve its products.

Mr. Tabor said he had no qualms about contributing his anonymized data. “I’m happy they use the data for research and to make their services better,” he says.

Intuit began its research in 2004 on small businesses and has expanded its scope, including many more companies and becoming more fine-grained in its data tracking as it has added products, services and customers.

Researchers at the Bureau of Economic Analysis, the government’s statistical scorekeeper of economic activity, are now experimenting with the Intuit data, seeking to tap it to improve the official estimates.

“The promise of new data sources, like Intuit and others, is more accurate, more timely and cheaper data for monitoring the economy,” says Steve Landefeld, director of the bureau. “That could be a really big step forward.”

National income accounting emerged in the Great Depression, in an effort to bridge the economic information gap of its day. In June 1930, based on scattered reports available to him, President Herbert Hoover declared, “The Depression is over,” when in reality conditions were quickly worsening.

The main tool for government statistics remains telephone and in-person surveys of households and businesses — surveys that are costly and time-consuming.

Tracking behavior online can pull in far more data, more quickly — so that governments should be able to see signs of inflation, deflation and employment trends sooner and adjust policy faster.

For example, the Intuit monthly employment data, based on online monitoring, is current. That is eight months to a year ahead of the government’s best statistical look at the health of small business, which is culled from quarterly surveys, state unemployment records and tax returns. The monthly Intuit survey data have also proved accurate, almost mirroring the government results when they are finally reported, according to Susan Woodward, a consulting economist for Intuit.

“Whatever is happening, it is better to know sooner,” Ms. Woodward says.

Yet whether more data, collected faster, will improve economic forecasting is uncertain. So far, the results are mixed. An encouraging study, begun in 2009 and repeatedly updated, has used Google searches to predict home sales and prices three months into the future. In the study, the higher the frequency of search terms like “house prices,” “real estate agent” and “mortgage rates,” the more likely the national housing market would heat up.

The results of the study, “The Future of Prediction,” by Lynn Wu, an assistant professor at the Wharton School of the University of Pennsylvania, and Erik Brynjolfsson, a professor at the M.I.T. Sloan School of Management, have held up over time. In the most recent version, their model using search data predicted future home sales 24 percent more accurately than the forecasts by experts from the National Association of Realtors.

But another major project using Google search terms points to the limits of such techniques: Google Flu Trends uses the same methods as Big Data-style economic prediction, although it focuses on public health.

The service monitors flu-related search terms and seeks to predict the incidence of flu, ahead of official statistics based on doctors’ reports to the Centers for Disease Control and Prevention. In 2009, with the outbreak of H1N1 flu, Google Flu Trends was prescient.

Yet this past flu season, Google’s algorithms stumbled. As an article in the science journal Nature noted in February, Google Flu Trends estimated that nearly 11 percent of Americans were ill at the January peak — nearly double the 6 percent reported later by the C.D.C. Apparently, the Google algorithms were unable to sift out news reports and social media messages warning of a harsh flu season, which sent flu-related searches spiking.

Google is tweaking its algorithms accordingly. But experts say the episode underlines why the new technology will most likely supplement official economic statistics rather than replace them for the foreseeable future.

“The potential is extraordinarily exciting, but I worry that the expectations are way ahead of the likely reality for the next several years,” says Mr. Landefeld of the Bureau of Economic Analysis.

THE economics profession is gearing up to exploit new sources of digital data. In a recent paper, “The Data Revolution and Economic Analysis,” two Stanford economists, Liran Einav and Jonathan Levin, concluded that “there is little doubt, at least in our minds, that over the next decades ‘big data’ will change the landscape of economic policy and economic research.”

At Intuit, the small-business data portray a sector that was “hurt much more than big business by the recession and its recovery has been far worse,” says Ms. Woodward, the economic consultant. Over the last three and a half years, payroll employment for all companies has increased 6.9 percent, while small-business employment has risen far less, just 1.9 percent. Hiring among the small companies, though still sluggish, has inched ahead in the last three months.

In West Virginia, Mr. Tabor’s company did better than many, but his story echoes the national trend. He bought his home-inspection business in 2007, when times were good. But later, when the economy slowed, hours would go by without the phone ringing. “I thought, ‘Oh dear, what have I done,’ ” he recalls.

Mr. Tabor modernized the business, shifting from paper to laptops and smartphones, and expanded into doing valuations for insurance companies and serving as an engineering expert in litigation. Revenue, he said, is up 15 percent this year.

“We are going to be looking to hire, probably next year,” he says.

http://www.nytimes.com/2013/09/08/business/more-data-can-mean-less-guessing-about-the-economy.html?ref=technology

the mathematics lives and breathes in the mind of the reader

These days, mathematics books tend to be awash with symbols, but mathematical notation no more
is mathematics than musical notation is music. A page of sheet music represents a piece of music; the
music itself is what you get when the notes on the page are sung or performed on a musical instrument. It
is in its performance that the music comes alive and becomes part of our experience; the music exists not
on the printed page but in our minds. The same is true for mathematics; the symbols on a page are just a
representation of the mathematics. When read by a competent performer (in this case, someone trained
in mathematics), the symbols on the printed page come alive | the mathematics lives and breathes in the
mind of the reader like some abstract symphony.
- excerpt from What is mathematics?

NSA “touches” more of Internet than Google / In deep packet inspection, it's not the size of the data that matters.

NSA “touches” more of Internet than Google

In deep packet inspection, it's not the size of the data that matters.

by Sean Gallagher - Aug 13 2013, 9:30am EDT

Big Data
National Security
Networking

Equinix's co-location facility in San Jose, California, one of the network exchange sites likely tapped by the NSA's "one-side foreign" surveillance.

Photo: Peter McCollough/Wired.com

NSA leaks

• New profile of Snowden’s trusted ally illustrates importance of opsec
• Obama’s “reform” panel to be led by Clapper, who denied spying to Congress
• Lawmakers tasked with overseeing NSA surveillance programs feel “inadequate”
• Don’t worry, NSA says—we only “touch” 1.6% of daily global Internet traffic
• New leak: NSA can search US e-mail data but theoretically won’t

View all…

In a memo issued last Friday, the National Security Agency (NSA) provided details of its ongoing network surveillance operations intended to assuage concerns about its scope, content, and oversight. As Ars' Cyrus Farivar reported, the NSA tried to set the context of its activities with a Carl Sagan-like metaphor:

According to figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6 percent of that. However, of the 1.6 percent of the data, only 0.025 percent is actually selected for review. The net effect is that NSA analysts look at 0.00004 percent of the world's traffic in conducting their mission—that's less than one part in a million. Put another way, if a standard basketball court represented the global communications environment, NSA's total collection would be represented by an area smaller than a dime on that basketball court.

The numbers are no real surprise—we've already discussed how the laws of physics would make it impossible for the NSA to capture everything, or even a significant portion of everything, that passes over the Internet. But they're also misleading. In the world of deep packet inspection, verbs like "touch," "select," "collect," and "look at" don't begin to adequately describe what is going on or what information is extracted from traffic in the process. Considering all that's within what flows across the Internet, 1.6 percent could hold a significant portion of the metadata describing person-to-person communications.

How much is 1.6 percent?

The dime on the basketball court, as the NSA describes it, is still 29.21 petabytes of data a day. That means the NSA is "touching" more data than Google processes every day (a mere 20 petabytes).

While 29.21 petabytes is a fraction of the overall traffic on the Internet, it is the equivalent of the traffic that passes through several major Internet exchanges each day. It amounts roughly to 2.77 terabits per second—more than the average throughput of the Equinix exchange network, the CoreSite Any2 Exchange, New York International Internet Exchange (NYIIX), and Seattle Internet Exchange (SIX) combined. In other words, the 1.6 percent of the total of Internet traffic "touched" by the NSA could easily contain much of the traffic passing through the US' core networks. It can certainly include all the traffic inbound from and outbound to other nations.

Those exchanges are likely the primary targets of the NSA's Special Source Operations "one-end foreign" (1EF) network tap operations. The remaining sources are overseas taps, including "FORNSAT" satellite communications intercepts and data shared by friendly foreign governments' own network surveillance—such as Germany's foreign intelligence agency, the Bundesnachrichtendienst (BND), as detailed in a report published today by Der Spiegel. There are also covert sites set up by the NSA's Special Collections Service, likely including targeted taps of networks in even "friendly" countries.

The NSA has approximately 150 XKeyscore collection points worldwide. To reach 29.21 petabytes per day, XKeyscore sites pull in around 190 terabytes a day. And to keep the three-day "buffer" XKeyscore holds of captured traffic, that would mean the sites have an average of about 600 terabytes of storage—the equivalent of a fairly manageable 150 4-TB drives.

Pick a peck of packets

Regardless how much data flows through the NSA's tap points, all of it is getting checked. While the NSA may "touch" only 29.21 petabytes of data a day, it runs its digital fingers through everything that flows through the tap points to do so.

The NSA's XKeyscore uses packet analyzers, the hardware plugged into the network that diverted Internet data is routed down, to look at the contents of network traffic as it passes by. The packet analyzers use a set of rules to check each packet they "see" as it is read by the analyzers' software into memory.

Packets that don't meet any of the rules that have been configured are sent along unmolested. In a "normal" network filtering situation, these packets would then be forwarded down the wire to their recipient, but in the NSA's case the packets are just clones of the packets that have already passed onto their intended destination. They are just sent to /dev/null—flushed away forever.

Packets that match one or more of the rules get routed to processing servers for further analysis. Those rules can be very broad—"grab everything with an IP address in its header that is outside the United States," for example—or they can look for very specific patterns within packets, such as those of VPN and website log-ins, Skype and VoIP traffic, or e-mails with attachments. In some cases, a filter may capture only the initial three-part TCP handshake of a connection between two systems, or it may only look for specific patterns of Web requests from clients. The rules could also include "if-then-else" logic: "If this is a packet that is part of an e-mail message I saw going by earlier and it includes attachment data, then grab it."

Enlarge / A single IP packet, snatched from the stream, can tell a lot. This HTTP request tells anyone who reads it where I am, what I'm looking for, where on the Web I'm coming from, and a collection of cookies that can be used to track me later.

Sean Gallagher

As a result, if properly tuned, the packet analyzer gear at the front-end of XKeyscore (and other deep packet inspection systems) can pick out a very small fraction of the actual packets sent over the wire while still extracting a great deal of information (or metadata) about who is sending what to whom. This leaves disk space for "full log data" on connections of particular interest.

How I learned to stop worrying and love packet capture

There's a lot of chaff that gets ignored by XKeyscore sites. They ignore Web server responses, DNS traffic, content distribution network traffic, and the other administrivia and overhead that is involved in making the Internet work. At least in theory, they largely ignore domestic Internet traffic that doesn't transit outside the US. Though depending on where you live in the US and where you're connecting to, a significant portion of your network traffic may pass through Canada or follow other paths that could expose you to surveillance.
And, at least up until this point, all of the processing is being done without human intervention or human eyeballs being involved. The data is kept in buffer for three days and heavily indexed for search, with metadata extracted and held for a month. Still, unless the content matches a cross-referenced search like "all Excel documents e-mailed in Iraq," it will probably avoid human eyes.

XKeyscore is also integrated into Marina, the NSA's phone call metadata database.  That allows for the trolling of Internet traffic for phone numbers of interest and for quick searches of raw data in XKeyscore's cache by analysts.

After XKeyscore's processing servers churn through the raw captured data, they forward extracted information such as metadata, attachments, and other content related to cases assigned a National Intelligence identifier over the wire to PINWALE, the NSA's in-house "big data" analysis platform for Internet intercepts (believed to be based on the Accumulo data analysis engine). This has to be done with a good deal of care not to overwhelm the NSA's private network backhaul to its data centers and reduce the performance of XKeyscore searches. So by the time the data has gone through this many levels of refinement, the NSA says that only 0.025 percent of the data  "touched" by its systems each day is "selected for review" and sent back.

That's 7.47 terabytes a day of connection metadata, cyber-attack targeting data and virtual private network intercepts, e-mail attachments, and instant messages. Really, that's nothing, right? The 2.66 petabytes a year of analytics that get rolled up in front of the eyeballs of analysts at NSA, in the DOD, and various other intelligence and law enforcement agencies is but a pittance.

Of course, that doesn't cover the fact that the NSA is, in effect, collecting 10.411 exabytes of short-term searchable content in XKeyscore. They extract information from it that is much more valuable (and potentially more intrusive) than the raw data.

Promoted Comments

• nrduren Smack-Fu Master, in training

   
  This also doesn't take into account that the vast majority of traffic in terms of size is now video and audio streaming from sites like Youtube and Netflix. These two alone account for just under half of peak internet usage. Take away all of the stuff that no one could ever care about and that percentage will certainly grow.
    - registered Jul 2, 2012

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.@thepacketrat