I was reading the Sunday New York Times this past weekend and came upon the great article by Steve Lohr – “Sure, Big Data Is Great. But So Is Intuition.”
(http://www.nytimes.com/2012/12/30/technology/big-data-is-great-but-dont-forget-intuition.html)
This is one of my favorite topics so it was exciting to see this featured under “Bright Ideas.” Lots of great tips like the reference to Thomas H. Davenport stating that a major part of managing Big Data projects is “asking the right questions” and the line “After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?”
Context is so important around data. Without it bad decisions can be made and time/money can be wasted. Ironically enough (or maybe the stars are aligning) I started the “Creating a Data-Informed Culture” chapter of Measuring The Networked Nonprofit (Kanter & Paine) just after seeing this NYT article and came across the line “Being data driven sounds great – in theory. But, because it doesn’t acknowledge the importance of basing decisions on multiple information sources, it can doom an organization to epic failures.”
**Side note – I highly recommend the above book by Beth Kanter and Katie Delahye Paine if you are involved with data and measurement.
This article also reminded me of a couple blog posts I wrote on the same topic. Here they are FWIW –
Originally published on Sports Data Hub in November 2007. The post was for Fantasy Football analytics which Sports Data Hub was focusing on back then (they now focus on sports analytics data management and visualization) but it revolves around my long time working with data and decision making.
The most important factor in winning you fantasy playoffs is your draft. However, the second most important factor is adding key players that will carry your team to the championship. Last season I won our league championship. Getting to the playoffs was primarily due to my favorite player, L.T. But winning was due to picking up Ron Dayne.
I recently came across an interesting blog by Mac McMann called "Want New Results? Trust Your Gut" (http://motivation-works.com/1504/want-new-results-trust-your-gut/). **That link is no longer working but I did find the article here ** It is one of many entries in the ever present intuition vs. data discussions. Mr. McMann follows the path of Malcom Gladwell in saying that we do not do enough trusting of our guts. While I do agree with this view, I also think that good intuition that you can trust comes only after solid experience and knowledge. Knowledge comes from access to, and proper use of, factual data.
Below are three quotes from Mac McMann’s blog that stood out to me:
“The problem with this is two fold. One all this information does not make us more productive. It makes us more comatose, paralyzed by the fear of acting prematurely, and let’s face it premature anything is bad. We receive information that counters previous information, than we must synthesize both inputs into one valid hypothesis.”
“The other problem, and the bigger problem with data overload, is we no longer trust our judgment.”
“Today it is very popular to have your guru of whatever subject you are following tell you what to think. And from finance to internet to football to celebrity gossip, the world has plenty of them.”
I agree 100% with information overload. I have found myself many times more confused after digging into information resources than I was before I started. There is so much information out there, often times conflicting, that you can easily reach frustration and stagnation. At this point you have to trust your gut feel and pick the option that seems right to you. This has become challenging for many people though. It also is putting people more into reactive survival mode. As Mr. McMann points out many people seem to have gotten lazy, or fearful, of their own thoughts and now just rely on the many “gurus” out there to make their thoughts for them. There are some gurus who really do have good thoughts based on facts but unfortunately for every one of those there are many others who are really nothing more than talking heads who have been able to get into situations where we think they are smarter than we are.
Mac mentions Gladwell’s Blink book. This is indeed a good book on the topic of intuition. I would also offer up a read of “Competing on Analytics” by Thomas H. Davenport and Jeanne G. Harris to show why properly used analytics are critical to support your intuition. Their focus is mainly on data and analytics in a corporate environment but I think their approach applies outside of a corporation as well. Below are some quotes.
“Good decisions usually have systematically assembled data and analysis behind them. Analytical competitors, then, are organizations that have selected one or a few distinctive capabilities on which to base their strategies, and then have applied extensive data, statistical, and quantitative analysis, and fact-based decision making to support the selected capabilities.” (Competing on Analytics, p. 9)
“What is necessary is a willingness to delve into analytical approaches, the ability to engage in discussions with quantities experts, and the fortitude to push others to think and act analytically.” (Competing on Analytics, p. 31)
“You absolutely cannot make a series of good decisions without first confronting the brutal facts.” (Competing on Analytics, p.45, quote from Jim Collins of Good to Great referenced)
“The challenge, then, is not simply to identify internal applications of business analytics but to find some that are clearly strategic and involve competitive advantage.” (Competing on Analytics, p.57)
In other words, you cannot just measure everything. You need to pick things to measure and analyze that nobody else is looking at.
“Gladwell is undoubtedly correct, for example, than human beings evolved a capability to make accurate and quick decisions about each other’s personality and intentions, and it's rare for formal analysis to do that as well. Yet even Gladwell argues that intuition is a good guide to action only when it's backed by many years of expertise. And many of Gladwell’s examples of intuition are only possible because of years of analytical research in the background…” (Competing on Analytics, p.14)
If you really want to get the edge, get past reliance on others to make your decisions for you, and really put yourself ahead of competition in whatever you do then you have to find a way to get to the raw data or facts, have some key measurements on the data that are different than what others are doing (this will weed out the noise disguised as valuable information), then put it into your gut for analysis on what you feel is best to do with the information you have.
I did an updated perspective on my old blog (http://duncaninfo.com/?p=147) in 2009 around preparing for a bike race:
A couple weeks back on a cool Saturday morning I was on my rollers at the base of Lookout Mountain in Golden Colorado getting ready for the Lookout Mountain Hillclimb race. This race requires a short but intense amount of energy especially at the start. During the warm-up I had a gut feel to get off the rollers and do some actual climbs up some of the smaller hills near the start to shake out my upper body. I ignored my gut and stayed on the trainer. Big mistake. A few minutes into the surge my upper body became almost useless with fatigue and lactic acid. This adversely impacted my performance and lead me to a slower than desired time.
After the race as I was looking back on what I could have done better I realized that I should have trusted my gut. We all have experiences, sometimes more than once in a day, where in hindsight we realized that we ignored our gut feel. On the flip side we sometimes also act too quickly on gut feel alone when the facts may have had us take a different action.
In my simple scenario above I should have done a better job with my pre-race checklist to have the step of getting off my rollers for an upper body shake out. This would have balanced out my gut feel (and subsequent ability to ignore it) with some data on my personal race warm-up experience.
On the drive home I remembered an old blog I wrote for Sports Data Hub on this very subject. See the post at http://ww2.sportsdatahub.com/index.php/2007/11/18/trust-your-gut-data-or-both/. **Link no longer working**
The long and the short of it is that we need both our gut and data (which can sometimes be robust analytics or a simple checklist) to make the best decisions.
