The theory underlying "The End of Theory" in Big Data. (updated)

Periods of social and economic change ignite our imaginations. New technologies and socioeconomic phenomena elicit both promise and dread.¹ We can, at the same time, see promises of utopia and specters of dystopian "Brave New Worlds."

Reality usually falls somewhere in between these extremes. Many anticipated that the Internet would "kill" distance and make the world flat.² Our planet nonetheless remains round³ — both geographically and socioeconomically. Modern transportation networks may reduce distance's relative importance. Other dimensions of culture, polity, and economics still matter as much as ever.⁴

The Internet was to lead to a "new economy." Website traffic rates supplanted net-free cash flow as the principal basis for valuing many firms. At its peak, America Online's market capitalization once exceeded that of Boeing. Corrections in the end returned familiar approaches to business valuation to a more-conventional place.⁵^,⁶

Genomics promised to revolutionize medicine. We now face however the reality of that science's limitations.  A recent study "confirms that genetics are a poor, if not purposeless, prognostic of the chance of getting a disease."⁷ Combined effects of nature and nurture prove difficult to unwind.

Big-Data analytics is following a similar pattern. Industries — particularly those based on technology — tend to follow lifecycle trajectories.⁸ Gartner uses a "hype-cycle" method to track these trajectories. The consultancy's most-recent analysis — in Figure 1 — shows gravity's affect on the Big-Data movement.¹¹^,¹²  The mainstream press even recognizes this trend.¹²

Figure 1 — Gartner's 2014 "Hype cycle for emerging technologies" report shows Big Data sliding down from the "Peak of Inflated Expectations" towards the trough of disillusionment."¹⁰

The inaugural installment of Quant's Prism showed Big Data at the "peak of inflated expectations" on Gartner's curve. This blog's narrative consistently emphasizes the importance scientific discipline to business analytics. Value is derived through data science. Big Data's slide may be attributable in part to occasional lack of scientific rigor in its application.

Naïveté underlies the irrational exuberance accompanying highly-hyped, "cool" new technologies. This naïveté often involves discounting fundamental principals. Mother Nature nonetheless gets to vote — and her's counts. Mathematics — particularly statistics — is the science to which Big Data is subject.

Financial Times columnist Tim Harford — the "Undercover Economist" — describes four fundamental precepts of statistics that Big Data enthusiasts sometimes set aside.¹³  This installment of Quant's Prism summarizes Hartford's observations. I provide my own illustrations.

Setting science aside.

Harford summarizes four points of view comprising an integrated thought system. They appear in different places. Each idea tends however to be interconnected with the others. 

Theory doesn't matter.

Exuberant claims about applying machine-learning to large, complex data sets largely originate from the tech industry. Tech-industry evangelist Chris Anderson first suggested "The end of theory" hypothesis in Wired.¹⁴ Anderson grounds his rationale on statistician George Box' observation, "All models are wrong, but some are useful."

Google's experiment¹⁶ in the use of its search engine to tack the 2008 flu season is offered as the demonstration that proves the "end-of-theory" assertion. This anecdote regrettably failed the repeatability criterion. Scientific conclusions are only validated if they can be independently verified. Google's search-engine experiment failed to deliver the same success during subsequent years.¹⁷ Its results were neither repeatable, nor independently verifiable.

The absence of an underlying theory likely explains the absence of repeatability with Google's flu-tracking experiment. Google assumed relationships between users' medical conditions and their Internet searches. These relationships might change over time. Any such tracking model requires validation of the relationships and precise tracking of their change over time. In short, it takes a theory!

The sample can include the whole population (N → all).

"N →all" assertions largely occur related to Internet and social-media data. They use what Harford calls data "exhaust." Data exhaust includes things observed by tools that track website and smartphone use. These are the footprints connected users leave wherever they go in cyberspace. 

Some "N →all" enthusiasts conveniently assume that observations about online-exhaust data serve as a proxy for the population as a whole. These data in fact provide a biased view of the population. Adoption of exhaust-producing online technologies is far from uniform. 

Figure 2¹⁹ shows the unevenness of home broadband-access penetration within the U.S. Penetration by smartphones — another source of online-exhaust data — similarly varies by age²⁰ and ethnicity.²¹ Analyzing exhaust data only tells us about users of particular exhaust-producing technologies. Exhaust data tell us nothing about non-users. This is not equivalent to "N →all."

Figure 2 — Large samples of data derived from exhaust from home broadband internet will be biased according to the proportion of households with the service. (from Financial Times.¹⁹)

Analyzing the data alone can explain it all.

Leading statistician David Spiegelhalter observes, “There are a lot of small data problems that occur in big data. They don’t disappear because you’ve got lots of the stuff. They get worse.”¹³ This blog has already briefly explored the issue of dimensionality in large data sets. Analytics practitioners frequently battle "the curse of dimensionality."²¹

Simply throwing a pile of data at a canned algorithm rarely produces valuable insight. Systematic methods are required. Understanding and preparing the data typically requires 75% to 80% of the effort in a modeling project. Preparation is always guided by analysis of the business stated as a hypothesis to test a theory about what the data mean.²² These hypotheses systematically link possible explanations about the business problem to a mathematical description.²³

Correlation is enough, and causality doesn't matter.

The "causality doesn't matter" assertion appears in the May 2013 article of Foreign Affairs.¹⁵ Authors Cukier and Mayer-Schoenberger similarly base their arguments in the culture of the Internet. Anderson's, Cukier's, and Mayer-Schoenberger's common link to British news weekly Economist is interesting. This correlation, incidentally, does not predict that all Economist staffers hold this view.

Ars Tecnica Senior Science Editor John Timmer places correlations in their place: "Correlations are a way of catching a scientist's attention, but the models and mechanisms that explain them are how we make the predictions that not only advance science, but generate practical applications."²⁴ Correlations provide opportunities for further exploration involving hypothesis testing to infer underlying causalities. 

"False-positive" correlations frequently occur, particularly in high-dimensional data sets.²¹ An unexplained correlation at best provides a tenuous basis for action. This is especially true when substantial value is at risk. Basing big decisions on causal understandings is the safest path.

The health sciences offers a current example of shortcomings of correlation-only decision-making. The federal government's nutritional policies have been based largely on epidemiological studies, which are "observational" in nature.²⁶  That is, they attempt to draw conclusions solely from correlations.  The continuously-varying nutritional guidelines illustrate the challenge of Big-Data analysis that doesn't explain causality.

Approaching Big Data scientifically.

The bloom may be falling off the Big-Data rose. This is a completely normal — and healthy — phase of Big-Data evolution. Gartner's "Hype-Cycle" methodology anticipates this. A "trough of disillusionment" following a "peak of inflated expectations" corresponds to an "awkward adolescent" stage of life. 

Moore's (not to be confused with G. E. Moore of "Moore's Law" fame) lifecycle⁸ uses the terms "chasm," "tornado," and "bowling alley" to characterize the volatility of the early stages of technology life-cycles. Would-be gold-rushers exuberantly rush in, seeking alchemic payoffs. Most are disappointed.

Business leaders looking for competitive differentiation from Big Data must remain patient. Big Data will reach a "plateau of productivity." Proven, repeatable practices will emerge. These practices will be based — like all mature business practices — on systematic, scientific approaches. Early adopters must keep their heads. Sketches of pragmatic, deliberate approaches — exemplified by Davenport²² and Lavalle²⁵ — have already been in the literature for years.

References

¹ D. Bollier, The Promise and Peril of Big Data, The Aspen Institute, January 1, 2010, http://www.aspeninstitute.org/publications/promise-peril-big-data.

² T. L. Friedman, The World Is Flat, New York: Macmillan, 2007, http://goo.gl/q2UdPd. 
³ L. Prusak, "The world is round," Harvard Business Review, April 2006, https://hbr.org/2006/04/the-world-is-round. 
⁴ P. Ghemawat, World 3.0: Global Prosperity and How to Achieve It, Boston: Harvard Business Press, 2011, http://goo.gl/QLkVOK.
⁵ M. Porter, "Strategy and the Internet," Harvard Business Review, March 2001, https://hbr.org/2001/03/strategy-and-the-internet.
⁶ C. M. Reinhart and K. Rogoff, This Time Is Different: Eight Centuries of Financial Folly, Princeton, NJ: Princeton University Press, http://goo.gl/C5Zfnc
⁷ D. Shenk, "The Limits of Genetic Testing," The Atlantic, April 3, 2013, http://www.theatlantic.com/health/archive/2012/04/the-limits-of-genetic-testing/255416/. 
⁸ G. Moore, "Darwin and the demon: Innovating within established enterprises," Harvard Business Review, July 2004, https://hbr.org/2004/07/darwin-and-the-demon-innovating-within-established-enterprises. 
¹⁰ J. Rivera, "Gartner's 2014 Hype Cycle for Emerging Technologies Maps the Journey to Digital Business," The Gartner Group, http://www.gartner.com/newsroom/id/2819918.
¹¹ M. Wheatley, "Gartner’s Hype Cycle: Big Data’s on the slippery slope," SiliconANGLE, August 19, 2014, http://siliconangle.com/blog/2014/08/19/gartners-hype-cycle-big-datas-on-the-slippery-slope/.
¹² G. Langer, "Growing Doubts About Big Data," ABC News, April 8, 2014, http://abcnews.go.com/blogs/politics/2014/04/growing-doubts-about-big-data/.
¹³ T. Harford, Big data: are we making a big mistake?" Significance, The Royal Statistical Society, December 2014, pp. 14 - 19,  http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2014.00778.x/abstract; Financial Times, March  28, 2014, http://www.ft.com/intl/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html.
¹⁴ C. Anderson, "The end of theory: The data deluge makes the scientific method obsolete," Wired, June 23, 2008, http://archive.wired.com/science/discoveries/magazine/16-07/pb_theory.
¹⁵ K. N. Cukier and V. Mayer-Schoenberger, "The rise of big data," Foreign Affairs, May/June 2013, pp. 28-36, http://goo.gl/3tGMZc.

¹⁶ J. Ginsberg, et al, Detecting influenza epidemics using search engine query data," Nature, November 19, 2008, http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html, (republished by Google to http://goo.gl/Ly0bwR).
¹⁷ H. Hodson, "Google Flu Trends gets it wrong three years running," NewScientist — Health, March 13, 2014, http://www.newscientist.com/article/dn25217-google-flu-trends-gets-it-wrong-three-years-running.html#.VOQZq7DF-oU.
¹⁸ D. Crow, "Digital divide exacerbates US inequality," Financial Times, October 28, 2014, http://www.ft.com/intl/cms/s/2/b75d095a-5d76-11e4-9753-00144feabdc0.html#axzz3J94dMqob.
¹⁹ "US Smartphone Penetration by Ethnicity, Q12012," Neilson, June 1, 2012, accessed from Beehive Group, http://beehive.me/2012/06/us-smartphone-penetration-by-ethnicity/

²⁰ "U.S. smartphone penetration by age group, 2Q2014," MarketingCharts, http://goo.gl/VCvjDt.
²¹ T. Hastie, et al, The Elements of Statistical Learning, second edition, Springer, 2009, http://goo.gl/ipNrMU, http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf.

²² T. H. Davenport, "Keep up with your quants," Harvard Business Review, July 2013, http://goo.gl/BrWpD1.

²³ D. R. Cox and D. V. Hinkley, Theoretical Statistics, Boca Raton, FL: CRC Press, 1996, http://goo.gl/1zw8H0.

²⁴ J. Timmer, "Why the cloud cannot obscure the scientific method," Ars Tecnica, June 25, 2008, http://arstechnica.com/uncategorized/2008/06/why-the-cloud-cannot-obscure-the-scientific-method/.

²⁵ S. Lavalle,  et al, "Big data, analytics and the path from insights to value," MITSloan Management Review, Winter 2011, http://goo.gl/8RSn5H.
²⁶ N. Teicholz, "The government’s bad diet advice," Washington Post, February 20, 2015, http://goo.gl/aeyNwC.

© The Quant's Prism, 2015

Remain skeptical when looking at slick data-mining reports.

Nobel-Prize economist Ronald Coase warned, "If you torture the data long enough, it will confess."  Our current infatuation Big Data sometimes sometimes leads us down this road of unjustified exuberance. Large data sets are more accessible than ever. And tinkering with them can be really fun!

The Sunday Review section of today's New York Times provides a timely illustration. Researchers for the Times went out looking for correlations between street name and home value.¹  They package their findings behind slick, interactive graphics. We can type in our street name and see how the price of homes located on streets with that name compare with national averages.

This topic entices us immediately. Our home — for many of us — stores most of our household wealth. We are also less than a dozen years removed from the roller-coaster mass hysteria of a nationwide housing bubble. Many of us might still be more nervous than ever as we watch our prize assets' unsteady recovery of pre-crash value. 

The authors' first example gives us a red flag. They assert, "On average, homes on named streets are 2 percent more valuable than those on numbered streets." We should immediately ask ourselves, "How significant is this?" Should the street name influence my next home-buying decision?

The street-name to home-value analysis stumbles into three common Big-Data pitfalls. I mostly focus here on the first: They only tell us about averages. Averages — or means — are among the most-abused of all statistics. They can become outright obfuscatory when offered in absence of other statistics. "What is the standard deviation," should be our reflexive response whenever someone offers an average during a serious conversation.

The Times authors don't give us any standard deviations. So what can we do?  We can ask, "How significant is a 2% difference in home values in your study?"  This is easy to test. We can get a pretty good idea on our own, even without access to their data.

I did exactly this to produce Figure 1. I am trying to answer the question,"At what home-price difference might different street names become different?" To do this, I assumed that the house prices fit a normal distribution and that their standard deviations are the same. These assumptions can in themselves be perilous? But they give me a good sanity check.

I used a statistical test called a "Kolmogorov-Smirnov" test for my little experiment here. This is a statistical method that tells whether two data sets are from the same or different statistical distributions. Rascoff and Humpries — authors of the Times article — give us an example in which houses on numbered streets are — on the average — worth two percent less than those on named streets.  

I test this with two 1,000-point random normal data sets. One data set has a mean of zero. The other has a mean of 0.02. I run this test 1,000 times. I calculate the Kolmogorov-Smirnov statistic for each of these 1,000 trials. I then calculate the average for the 1,000 trials. I also do this for 999 other cases in which the second data set has a different mean. This amounts to a million tests using a total of two billion data points.

Figure 1 shows the result. I believe that street name might make a difference if the "p-value" from my Kolmogorov-Smirnov test is less than about 0.05. What does Figure 1 show? At a difference of two percent my p-value is much closer to 0.5 than to 0.05. 

Street name should only really matter if average house prices differ by about 14%. Otherwise, the street name does not matter!  There is no evidence that the statistical distribution for house prices on named streets is different from that for house prices on numbered streets. A two-percent difference is well within what I expect from random chance!

Figure 1 — p-value results from Kolmogorov-Smirnov test comparing random normal samples. The plot contains the averages of 1,000 trials each at 1,000 distinct means.

So, Rascoff and Humphries commit three primary Big-Data offenses. I show here their abuse of means. Their tool also doesn't account for problems with small samples. I type in my street name and receive the happy news that houses on my street name are worth on average 105% more than the national average!  But there are only two homes per 100,000 with that address. The sample size cannot be significant.

Finally, the authors' admittedly fun exercise fails to consider other factors. No rational home buyer would use street name as a determining factor when shopping. They consider school districts, crime rates, tax rates, and a myriad of other factors. While their exercise is entertaining, it illustrates the pitfalls we must consider when using analytics for more serious work.

References

¹ S. Rascoff and S. Humphries, "The Secrets of Street Names and Home Values," Sunday Review, The New York Times, January 24, 2015, http://www.nytimes.com/2015/01/25/opinion/sunday/the-secrets-of-street-names-and-home-values.html?ref=opinion.

© The Quant's Prism, 2015

Clustering analysis demonstration: Analytics results sometimes require us to examine our presuppositions.

Business managers often encounter situations that require them group things together. Typical questions include:

• How can I segment or sub-segment my customers so that I can appropriately tailor my offerings to them?
• Which students in my school or district share similar learning challenges requiring specialized approaches to help them learn?
• Which of my customer-service cases share things in common that might result from some common root cause that I haven't yet discovered?

Cluster analysis is a family of Big-Data Analytics methods that precisely seeks to address these kinds of questions. 

I provide a practical demonstration in this installment. I write as usual for business users of analytics. Arming them with enough knowledge to ask good questions of their analysts is my objective. 

I continue here my tinkering with a data set from the Bureau of Labor Statistics (BLS) O*Net service. O*Net provides free, online career counseling to the general public. O*Net users complete a brief survey about interests, experiences, and preferences. It turns these inputs into recommendations about occupations.

O*Net uses a model based on Holland's Occupational Themes. This model uses 258 features to predict how well-matched a O*Net user might be to one of 923 distinct occupations. The 258 features measure interests, knowledge, abilities, and skills. 

I showed in a previous Quant's Prism installment that these 258 features can be boiled down to about four "abstract" features containing unique information. Most of the Holland model's 258 features are redundant to each other. I show here that most of the 923 occupations fit within one of about 22 clusters.

The following discussion begins by elaborating on this result. I then provide an overview of cluster analysis, its challenges, and pitfalls. Finally, I offer suggestions as to the significance of the results.

Occupational clusters from Holland's model.

Figure 1 shows graphically the results of a cluster analysis of the 923 occupations in O*Net's holland model. Figure 1(a) shows how the data are scattered. Figure 1(b) shows the shapes, sizes, and locations of the clusters into which they fit. The 923 occupations fit mostly into 22 occupational clusters. These clusters are distinguished by abilities, skills, interests, and knowledge.

Figure 1 — Best-practice clustering techniques group the 923 distinct occupations from the Holland model underlying O*Net 18.1 into 22 occupational clusters. The plots were constructed using the RGL package¹ in R.² 

This is the same data set I studied previously. I performed clustering using four of the 258 knowledge, skills, abilities, and interests features. The four features result from a principal-component analysis (PCA). These four features — or dimensions — account for about 95% of all of the variation between the 258 features.

The cluster regions in Figure 1(b) were produced using a model-fitting cluster method. The routine is implemented in the mclust routine³ for the open-source statistical tool R.² The mclust tool attempts to fit the data to geometric models. Well-known linear regression works in much the same way.

Figure 2 zooms in on two adjacent clusters from Figure 1. It shows the scatter points superimposed on into the ellipsoidal regions. These data correspond to the sixth and seventh clusters. We see that the cluster regions partially overlap. Some of the data points also fall outside of the cluster regions. Handling outliers is a challenge for cluster analysis as much as it is for other statistical techniques.

Figure 2 — Scatter points superimposed on ellipsoidal regions from the model give some insight into how well the cluster data fits the model. The data correspond to the sixth and seventh clusters.

Table 1 summarizes the results for the sixth cluster. This cluster contains the statisticians occupation. It includes a total of 48 distinct occupations. These assignments are the statistically best assignments to the statistically best set of clusters identified by the algorithm. The algorithm makes assignments based on 258 features characterizing interests, abilities, knowledge, and skills.

Table 1 — The clustering model assigns occupations to clusters with varying degrees of confidence.

Ten of the 48 occupations in Table 1 are highlighted in red. These are occupations for which the cluster-assignment confidence is less than or equal to 80%. Not every occupation fits well into the 22 clusters depicted in Figure 1(b). This set of clusters and assignments is, statistically speaking, the best choice. They are nonetheless not perfect . Adding more clusters does not improve things.

Cluster analysis — Approaches and pitfalls.

Cluster analysis represents one of the less scientific families of big-data methods. "Most clustering done in practice is based largely on heuristic but intuitively reasonable procedures...."⁴ Researchers — many working within a field referred to as "machine learning" — used common sense to derive algorithms that group things together. Many of them extensively use geometry to identify clusters.

Statistical methods were introduced to cluster analysis more recently. "Model-based" clustering involves fitting the data to a model — in this case a set of clusters. This works much like linear regression, where we fit data to curves. Fitting data to clusters is, of course, more complicated.

Analysts must first answer a number of particularly tricky questions, whether they use a machine-learning or a model-based approach. These include:

• How many clusters are there in my data?
• What are the shape, size, and orientation of each cluster? 
• Are they the same?  Or different?
• What is the best way to measure the "goodness of fit" to my set of clusters.

There are no known algorithms that directly answer these questions. Analysts start by making educated guesses.  They then do lots of iteration and trial and error. Most packaged clustering algorithms perform a blind search for the best cluster. They compute a score for each trial and then report back which cluster provides the best score.

In the end, my clusters are simply a model for what reality might look like. A famous statistician wrote that, "Essentially, all models are wrong, but some are useful."⁵ Much of the time there may be no natural, underlying grouping that corresponds to a model. Many points in a data set simply might not fit any cluster model very well.

Figure 2 and Table 1 highlight some of these outlier data points. The algorithm I used here tried 110 different cluster models. Each trial involved 50 different guesses about which cluster each occupation belonged to. The 110 models differe in terms of numbers of clusters, shapes, and varieties of shapes.  The best result from these 110 trials simply doesn't do a perfect job of assigning all of the points to my clusters.

The "mclust"³ algorithm tells me the probability that each of the 923 occupations might fit into each of the 22 clusters. Figure 3 summarizes these results with histograms.  I show the probabilities that data points fit in the most likely, second-most likely, or other cluster. Figure 4 shows the same information using a scatter plot.

Figure 3 — Histograms showing the probabilities that data points fit the most-likely, second-most-likely, and other clusters.

Figure 4 — Scatter plot showing probability that each of my 923 data points fit in the most likely versus second-most likely, versus other of the 22 occupational clusters clusters in my model.

Most of the data points fit in the best or second-best cluster with high probabilities. Figure 3(c) reveals nonetheless that lots of the O*Net data points don't fit well with any of my clusters. My model — the best one I can come up with using the best methods — is not a perfectly accurate characterization of the data.

What this all means.

Three obvious takeaways from this exercise occur to me. There are undoubtedly more. I offer these to non-quants seeking to become more-astute users of analytics.

Formidably large data sets don't always tell us as much as we expect them to.  I started with 923 data points spanning 258 dimensions. I previously showed that those 258 dimensions really only contain 4 dimensions of useful data. I show here that these 923 data points are not even as diverse we might expect. Most of the points fit into one of 22 categories.

Managers should not be intimidated by the apparent size and complexity of data sets. Analysts sometimes get lost in data sets' complexities. Users of analysts' work products should keep them focused on the questions for which answers are sought.  Leading analytics consultant Davenport⁶ makes specific recommendations on how to keep analysts on track and honest.

HR managers may have more flexibility than they realize.  Many occupations — characterized in terms of interests, abilities, knowledge, and skills — are more similar than we suspect. Opportunities may exist to move employees — or prospective hires — between occupations within the same or adjacent clusters. This flexibility applies to employees as well.

This analysis in no way repudiates the Holland model on which the data are based. The Holland model contains 258 distinct features. Differences between these features may be significant within the occupational-psychology and labor-economics disciplines that defined them. That 923 occupations fit into 22 clusters may in fact be limitations in the system of measurement.

Cluster analysis — in the end — is an essential part of the big-data analytics toolkit. It however suffers from the same limitations of all analytics methods. We seek through analytics to fit data to a model that only approximates reality. The fit is probabilistic. A good model will — at best — fit most of the data pretty well. Managers using analytics products must still decide how valid the model is, as well as how to deal with the outliers.

References

¹ D. Adler, et al, "RGL: A R-library for 3D visualization with OpenGL," http://rgl.neoscientists.org/arc/doc/RGL_INTERFACE03.pdf
² A. Vance, "Data Analysts Captivated by R’s Power," New York Times, January 6, 2009, http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html?_r=2&
³ C. Fraley, et al, "mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation," The University of Washington, June 2012, https://www.stat.washington.edu/.../tr597.pdf.
⁴ C. Fraley and A. E. Fraftery, "Model-Based Clustering, Discriminant Analysis, and Density Estimation," Journal of the American Statistical Association, June 2002, p. 611.
⁵ Box, G. E. P., and Draper, N. R., (1987), Empirical Model Building and Response Surfaces, John Wiley & Sons, New York, NY, p. 424, http://en.wikiquote.org/wiki/George_E._P._Box.
⁶  T. H. Davenport, "Keep Up with Your Quants," Harvard Business Review, July 2013, http://goo.gl/BrWpD1.

© The Quant's Prism, 2015

Being scientific about the business requires more than treating data scientifically.

Extracting value from big-data analytics is about changing the way in which business is done.¹ Intuition and tradition must give way to evidence and rigor. Kotter (Leading Change²) may actually tell us at least as much about how to achieve this as Bartlett (Practitioner's Guide to Business Analytics³).

Some analysts assert that "Big data business benefits are hampered by 'culture clash'."⁴ Allegedly, traditional "requirements-driven" approaches to managing enterprise IT systems impede "opportunistic analytics and exploring answers to ill-formed or nonexistent questions." Traditional IT bureaucracy purportedly suffocates entrepreneurial creativity of a "new breed" of big-data specialists.

I previously wrote that extracting value from big-data analytics requires us to simultaneously apply scientific methods to both our data and the business questions we seek to answer. I observed that data-science methods — by which big-data analytics yield information bearing economic value — can be classified either as data-centric or business-centric. The business-centric methods emphasize scientific methods for business questions. Data-centric methods narrowly focus on the data.

Guiding an organization's ubiquitous adoption of data-driven practices leads us beyond the scientific method for business questions. We must be systematic and methodical about the business itself. We consider the entire span of businesses attributes over which we can reasonably assert control. The following discussion summarizes a pair of frameworks by which we guide the application of opportunistic analytics by entrepreneurial big-data specialists.

I first describe a framework originally formulated to guide the adoption of Service-Oriented Architecture (SOA).⁵ We decompose the business into separable components. I illustrate using Michael Porter's Value-Chain Framework⁶ as a starting point. A second framework — characterizing business functions' operational criticality — focuses our attention on the risk of insufficiently-managed change. 

These two methods establish boundaries and conditions under which experiments in opportunistic analytics may be applied to core-business functions. They give decision makers control points by which to manage the pursuit of big-data opportunities. 

Business-capabilities analysis — identifying targets for big-data experimentation.

The scientific method makes extensive use of models. Developing a model for the organization often represents the first step in identifying and prioritizing opportunities for big-data analytics. Academia and the consulting industry have produced a variety of useful approaches to the modeling of organizations. Examples include:

• Enterprise architecture methods (e.g., The Open Group Architecture Framework (TOGAF)⁸) characterizes organizations in technology-centric terms;
• Business-Process Modeling takes a functional view of organizations;
• Lean/Six-Sigma emphasizes quality and consistency of business outputs; and
• Quantitative business models characterize financial and economic attributes of business functions.¹⁰

Figure 1 illustrates a functional model for a manufacturing business. It decomposes a value-chain⁶ view into increasingly more-granular functions. This approach was developed to identify and prioritize opportunities for technology investments leading to improved business performance. Consulting organizations, exemplified by IBM,⁹ have more formally developed similar methods.

Figure 1  — A Business-Capabilities Analysis (BCA) prioritizes and establishes the scope for opportunistic experimentation with big-data analytics. (From Merrifield.⁷) 

The business' functional breakdown in Figure 1 is annotated with two pieces of information. A label at the top of each box indicates the importance of the function to the business. This rating might be assigned based on contribution to competitive advantage, essentiality to business operations, or other criteria.

The boxes themselves are color-coded. The coloring indicates how well the function is performing. This performance characterization could compare the individual business against industry benchmarks. It might also indicate disproportionate consumption of management attention or other resources.

Priorities for experimentation with big-data analytics are assigned based on these two criteria. Big-data analytics resources — of which scarce, high-priced talent is often the principal component — are apportioned judiciously. High-value business functions requiring attention are usually assigned the highest priorities.

Characterizing the risk of change associated with new applications of big-data analytics.

Change to an organization is often difficult — even, frequently, turbulent. Introducing new big-data analytics business functions requires organizations' members to change how they think about the business. The maxim "If it ain't broke, don't fix it" usually represents good advice. 

Figure 2 contains a framework for thinking about how business functions contribute to an organization's mission. This framework was originally developed to characterize IT's contribution to a business en total. It is equally valid for thinking about business analytics. 

Figure 2 — A strategic-impact grid gives us a rigorous framework for characterizing the risk of experimentation and change associated with introducing new big-data analytics to a business function. (From Nolan.¹¹)

Information quality provides the big-data-analytics equivalent to the IT-reliability factor. Factory- and strategic-mode business functions demand high-confidence, relevant data. Figure 2 emphasizes the availability of IT services in general. Operational capabilities using outputs from big-data analytics must also provide high-confidence, actionable information.

I provided another 2×2 grid for characterizing data quality in Figure 2 of my previous installment on treating business questions scientifically. Factory- and strategic-mode functions demand outputs from big-data analytics that fall in the "Differentiating Information" quadrant. Even of the service based on big-data analytics is continuously available, lapses in information quality may lead to catastrophically flawed decisions.

Characterizations based on Figure 2 in this installment might also inform characterization of business functions the framework of Figure 1. The vertical axis in Figure 2 might inform our business-value assignment. The need for new business analytics — the horizontal axis of Figure 2 — might inform the performance-spectrum assignment.

Experimentation with big-data analytics as the launchpad for business innovation.

We began this discussion with the observation about an alleged conflict⁴ between traditional enterprise-information management — data warehouses and business intelligence — and and the "new breed" of opportunistic big-data specialists. In reality, both are required. Innovation based on big-data analytics requires a disciplined, managed approach to change. 

Changes to factory- and strategic-mode business functions from Figure 2 must be made cautiously. The business depends upon their reliability. Thomke and Manzi¹² describe a disciplined approach to business experimentation. Their examples extend beyond opportunistic big-data analytics. Nonetheless their approach also applies here. 

We perform big-data analytics experiments through controlled, non-disruptive pilots. Experimenters are free to explore new opportunities using business-representative data. Their activities are isolated from core business operations, particularly those that are factory- or strategic-mode in nature.

But what about scenarios in which big-data analytics identify opportunities for disruptive innovation? 

This question addresses scenarios leading to extreme change. The innovation opportunity may require a business to begin cannibalizing existing revenue streams before competitors exploit the opportunity first.¹³ 

Two principles guide the decision to embark on such a radical course. First, we previously observed that the information from big-data analytics assumes value commensurate with the economic returns from its exploitation. This is a foundational tenet of Business Analytics. 

Capturing this value may require re-orchestration of activities by the entire organization. Operationalizing new big-data analytics capabilities may lead to reconfiguring the business model exemplified in Figure 1. "New-breed" big-data specialists cannot do this by themselves. Taking action to capture value resides beyond the scope of most analytics specialists' activities.

Second, big-data specialists do not by themselves make the decision to embark on major organizational change. This is particularly the case for decisions about whether to divest or unwind revenue streams that are highly vulnerable to market disruption. These decisions are made by the organization's leadership.  

"New-breed" big-data specialists aid in identifying these scenarios. They serve as trusted advisors to the organization's leaders about how best to exploit the opportunities. They also assist in working through practical details of operationalizing the opportunities they discover. The results of their work — particularly when applied to factory- and strategic-mode business functions — ultimately however become subject to the disciplined governance of "traditional" data-warehouse and business-intelligence functions.

References

¹ S. LaValle, et al, "Big Data, analytics, and the path from insights to value," MITSloan Management Review, Winter 2011, pp. 21 - 31, http://goo.gl/MY5VK6. 

² J. P. Kotter, Leading Change, Boston: Harvard Business Review Press, 1996, http://goo.gl/EqglOJ. 

³ R. Bartlett, A practitioner's guide to business analytics, New York: McGraw-Hill, 2013, http://goo.gl/o6dTOS.

⁴ F. Buytendijk and D. Laney, "Big data business benefits are hampered by 'culture clash'," The Gartner Group, Report G00252895, September 12, 2013, http://goo.gl/8GLRgk.

⁵ "New to SOA and web services," IBM DeveloperWorks, https://www.ibm.com/developerworks/webservices/newto/.

⁶ M. Porter, Competitive Advantage of Nations, New York: The Free Press, 1990, Figure 2-3, p. 41, http://goo.gl/Qn1xGn.

⁷ R. Merrifield, et al, "The next productivity revolution," Harvard Business Review, June 2008, https://hbr.org/2008/06/the-next-revolution-in-productivity.

⁸ "The Open Group Architecture Framework," version 9.1, The Open Group, http://www.opengroup.org/togaf/. 

⁹ "Component business models: Making specialization real," IBM Institute of Business Value, http://goo.gl/QhucjH.

¹⁰ J. Tennant and G. Friend, Guide to Business Modelling, third edition, London: The Economist Newspaper, 2011, http://goo.gl/b36blO.

¹¹ R. Nolan and F. W. McFarlan, "Information technology and the board of directors," Harvard Business Review, October 2005, http://goo.gl/AqCdLK.

¹² S. Thomke and J. Manzi, "The discipline of business experimentation," Harvard Business Review, December 2014, pp. 70 - 79, https://hbr.org/2014/12/the-discipline-of-business-experimentation.

¹³ C. M. Christensen, The Innovator's Dilemma, Boston: Harvard Business School Press, 1997, http://goo.gl/kg2u9Y.

© The Quant's Prism, 2014

Getting value from Big Data requires the same discipline as any other business investment.

Reports on the surface might lead us to believe that the Big Data "revolution" brings alchemy in our time. A look below the surface suggests however that the path to Big Data value is not without challenges.  For example:

• Information Week recently reported that "the average company right now is getting a return of about 55 cents on the dollar" invested in Big Data analytics.  
• A related survey by Wikibon found that "46% of Big Data practitioners report that they have only realized partial value from their Big Data deployments," while, An unfortunate 2% declared their Big Data deployments total failures..." 
• Information Week also summarized a survey by IDG and Kapow Software revealing that, "...just 23% of <business> leaders see big data projects as 'successes' thus far."

Moreover, organizations find that even measuring Big Data return on investment (RoI) remains challenging.

Gartner's "Emerging Technologies" Hype Cycle Report places Big Data at the Peak of Inflated Expectations. The consultancy also predicts that five to ten years remain before Big Data reaches the "Plateau of Productivity" stage. This positioning is telling.

What then are the obstacles to big-data value? Analysts suggest that talent scarcity, technology maturity, and less than perfect alignment between Big Data initiatives and business priorities lie among the culprits.  I focus here on thee other barriers:

• Failure to treat business issues scientifically;
• Limitations in data itself; and
• Cultural disinclinations to fact-based decision making.

Failure to treat business issues scientifically.

Big data simply represents the latest wave of business technology. Authors of the IT Infrastructure Library™² observed the influence of organizations' technology management style on their ability to derive value. Organizations that focus on technology's strategic contribution get more out of it. Ross, et al, made similar observations.  Technology can either "fossilize" an organization, or it can provide a platform for "dynamic capabilities."⁴

The same is true for analytics.  Lavalle, et al,⁵ published what remains one of the more useful roadmaps to Big Data value.  Davenport, et al,⁶ echo similar concepts. Three of Lavalle's five central recommendations apply here:

• First, think biggest — Focus on the highest-value opportunities; and
• Start in the middle — Within each opportunity, start with questions, not data; and
• Build the parts, plan the whole — Define an information agenda to plan for the future.

Lavalle and Davenport separately portray analytics — in not so many words — as a scientific approach addressing business challenges.

What is the state of the practice of Big Data analytics today?  A twenty-year-old issue of Scientific American⁷  gives us a thought model. Technology professions proceed through stages of maturity from craft, to commercialization, to professionalism.  The emergence of standard, repeatable practices represents a key milestone.

Sets of best practices for Business Analytics have begun to emerge. Figure 1 summarizes four of them.  Three are data-centric methods: Patterns for applying the scientific methods to data. Two of these were developed by vendors to aid in use of their analytics software. The third originated from academia.  Davenport¹¹ describes a fourth, business-centric method.

Figure 1 — A series of best practices methods for data analytics have emerged.  The data-centric methods at best address business problems only obliquely.

What does this say about the state of the Business Analytics profession? Many of the leading practices — only incidentally at best — focus analysts on business problems. Would-be adopters of Big Data analytics apply scientific approaches to data, but not always necessarily to business questions they are asked to answer.

How does lack of scientific approaches to business impede RoI from Big Data investments? Figure 2 illustrates. Haphazard treatment of business priorities by analytics practitioners leaves organizations in the "Sandbox" or "Science Project" modes.

Figure 2 — RoI from Big Data analytics investments depends on scientific treatment of business as well as the data.

Getting value from Big Data depends on scientific treatment of the business as well as the data. Organizations' due diligence in their Big Data investments demands the the scrutiny and discipline as for any other major business investment. No organization would launch a marketing strategy without first considering how it intends to differentiate itself. Investing in Big Data demands the same focus.

Limitations in the data themselves.

A recent contributor to Harvard Business Review postulates that "...advanced algorithms can take a near-unlimited number of factors into account...."¹² This perception about Big Data walks a tenuous fine line.  The temptation to attribute alchemic abilities to Big Data analytics sometimes appears pervasive. Fundamental limitations underly data science — on which Big Data analytics is based — in the same way that weight and drag constrain aerodynamics.

But what about the Big Data "revolution"? Experiences with and perceptions of consumer data mining might produce misleading intuition. Consumer analytics is not a haphazard "search for serendipity" in a random "stack" of Big Data. Our consumer devices and applications are carefully instrumented to collect precisely the information that predicts our spending behaviors.

What kind of data limitations can impede analytics RoI?  This blog has already considered two. I illustrated how simply adding more data does not increase the information analytics can produce. Adding more factors to a model only helps if they contain useful information that the previous ones lacked.

Also, the lack of data quality degrades analytics outputs.  Analytics models are subject to the "Garbage-in/Garbage-out" syndrome.  I looked at five data quality attributes that influence the usefulness of analytics outputs. Neglecting data quality potentially lands organizations in the "Lies, damn lies, and statistics" quadrant of Figure 2.

Finally, analytics may at best yield information that is already known. Organizations may not need Big Data to improve operational optimization, for example. Responding to competitive market forces may have already driven them to the optimize their operations. Analytics would — under such a scenario — simply validate that the organization has already achieved optimality.

What then do we do about our data?  We adopt an approach that applies the scientific method jointly to our business and our data. A disciplined, scientific method leads us to the truth about the value of opportunities hidden in our data. It also reveals new information-collection needs.

Cultural disinclinations to fact-based decision making.

Lavalle⁵ observes that "the adoption barriers that face most organizations are managerial and cultural rather than related to data and technology."⁵ Deploying technology is the investment side of the Big Data RoI equation. Driving return often involves changing the business.

The Data Warehouse Institute — a an organization of technical professionals dedicated to Enterprise Information Management — surveyed its membership about the closely related imperative of aligning management of data to strategic business priorities.¹³ The first technology related issue did not appear until fifth place among their responses.  The four highest-ranking responses were:

• Data ownership and other politics;
• Lack of governance or stewardship;
• Lack of business sponsorship; and
• Unclear business goals for data.

Effectively executed Big Data analytics — through joint scientific management of the business as well as the data — can provide the "What" part of answers to pressing business questions. I illustrated by showing how customer analytics might provide the basis for a targeted marketing campaign.

The "So What" part of business questions' answers addresses how the organization responds. Information's economic value is based on the value of the opportunity arising from its optimum application. Optimally applying information often requires the people in an organization to change how they operate. They must change what they measure about their business, and what they do with those measurements. These changes are scary and hard!

Kotter¹⁴ may in the end tell us as much about driving RoI from Big Data as Hastie¹⁵ or any other technical text. Harvesting analytics' value depends as much on changing an organization's behavior as it does on tools and technology. 

References

¹ Gartner Group's 2013 installation of "Hype Cycle for Emerging Technologies,"  http://goo.gl/a4xlEY.

² ITIL™ Service Operation, U.K. Office of Government Commerce, 2007, Figure 5.1, p. 81.

³ J. W. Ross, Peter Weill, D. C. Robertson, Enterprise architecture as strategy, Boston:  HBR Press, 2006, http://goo.gl/B7J5P8. 

⁴ C. E. Helfat, et al, Dynamic capabilities, Wiley, 2007, http://goo.gl/Gn6cjC.

⁵ S. Lavalle, et al, "Big data, analytics, and the path from insights to value," MITSloan Management Review, Winter 2011, http://goo.gl/8RSn5H.

⁶ T. H. Davenport, J. G. Harris, and R. Morison, Analytics at work, Boston: HBR Press, 2010, http://goo.gl/olZkKm.

⁷ W. W. Gibbs, "Software's chronic crisis," Scientific American, September 1994, pp. 86 - 95.

⁸ IBM SPSS Modeler CRISP-DM Guide, IBM Corporation, http://goo.gl/4Gg7Pa.

⁹ Enterprise Miner™ SEMMA Method, SAS Corporation, http://goo.gl/8ig4RX.

¹⁰ "The KDD Process for Extracting Useful Knowledge from Volumes of Data," Communications of the ACM, November 1996, http://goo.gl/s4gvDd.

¹¹ T. H. Davenport, "Keeping up with your quants," Harvard Business Review, July-August 2013, http://goo.gl/BrWpD1.

¹² T. C. Redmond, "Algorithms make better predictions — Except when they don't," HBR Blog Network, September 17, 2014, http://goo.gl/n0kPJd.

¹³ "TDWI Technology Survey: Enterprise Data Strategies," Business Intelligence Journal, Vol 18, No. 2, March 2013.

¹⁴ J. P. Kotter, Leading change, Boston: HBR Press, 1996, http://goo.gl/EqglOJ.

¹⁵ T. Hastie, R. Tibsharini, and J. Friedman, Elements of statistical learning, NY: Spinger, 2009, http://goo.gl/23tclz.

© The Quant's Prism, 2014