Jump to content
Click here to know about ONLINE Lean Six Sigma Certiifcation ×
  • Data Driven Decision Making - A Holistic Approach To Decision Making


    “A point of view can be a dangerous luxury when substituted for insight and understanding. – Marshall Mcluhan â€

    We all have come a long way from industrial revolution where mass production and consumption was the order of the day. Over the years, the once not so knowledgeable consumer has transformed into a living storehouse of information. What has changed over the years? Why did the companies who were the industry leaders at one point of time failed miserably later on? What changed? The answer to all this lies in data. In our thirst to find newer markets for products and differentiate them from our competitors, we pushed the consumer towards seeking knowledge and then when he demanded, we failed to live up to standards.


    We pushed the customer to give more importance to value i.e. value today has become emergent as compared to the old times when it was measurable. Now the new age customer wants more and he/she will switch, the moment he/she finds more value from another product or another competitor. How do we live up to this? Given the rapid changes in technology and the rapid adoption of management information systems by almost all industries, we need not look further. We have the solutions but we are too thick skinned to look at it. Today’s customer is like a walking hard drive with copious amounts of data. What we need to do is analyse it, find patterns where none exist, combine it with the experience of mankind and predict the needs and demands of future backed by strong data analysis.


    If we stop and look back into the history what will be distinctly visible to us is the fact that companies that were present in the first list of Standard & Poor’s 500 only 74 remain in operation today. Almost half of the 25 companies that passed the rigorous test for inclusion in Tom Peters and Robert Waterman’s 1982 book “In Search of Excellenceâ€, have gone bankrupt or have shut down due to poor performance.


    The reason being that when organisations are thriving and successful, it creates an illusion that whatever gave them success in the past will work in the present and still remain valid in the future. They forget the fact that the world is constantly changing, constantly evolving and to survive they need to evolve with it too. Today there is no dearth of examples of companies that failed after achieving success on a grand scale, to quote a few, Wang Labs, Digital Equipment, Borders and Blockbusters and many more; of which many either fell after achieving industry leadership position or shrunk in size.


    The reason is not that they failed to act or didn’t know what was happening but the fact that they had all the data but didn’t analyse it. The patterns were there, had they analysed the data they already possessed they would have seen it. They believed that experience can upstage data but they failed to comprehend that with today’s rapidly changing global economy, it’s impossible for the human mind to absorb all factors that go into making sound decisions. Statistical analysis often uncovers correlations that no one could have predicted on the basis of their past experiences. We won’t be wrong in accepting the fact that data and statistical analysis isn’t perfect but who hasn’t made poor decisions on pure gut instinct!! It is said that sustainability is based on transforming data into analyzable information for insights and decision-making. This is where data driven decision making comes in and plays its part. (Cokins)


    Take for example the case of Wal-Mart as reported in New-York Times. This occurred when Hurricane Frances was moving across the Caribbean, threatening to directly hit on Florida’s Atlantic coast. Residents were making a rush for the higher ground, but at the same time, far away, in Bentonville, Ark., executives presiding over the Wal-Mart Stores decided that the situation opened a great window of opportunity for their newest data-driven weapons, Predictive technology. A week ahead of storms predicted landfall date, Linda M. Dillman Wal-Mart’s CIO at that point of time asked her staff to come up with forecasts based on what happened when Hurricane Charley struck several weeks earlier. Their experts mined for data and found that the stores would need certain products more than the usual batteries, flashlights, and bottles of water.


    All this brought out some very interesting facts about the pre-storm behaviour of consumers. Strawberry pop tarts and beer were the top-selling items. By predicting what was going to happen rather than waiting for it to happen, Wal-mart was able to deal with this unexpected situation in a holistic manner.


    It is said that if we can improve our ability to estimate for a given customer, we stand to gain by applying the same logic to millions of customers. This applies to many other areas where intense application of data science and data mining is seen like direct marketing, credit scoring, financial trading, help-desk management, fraud detection, search ranking, product recommendation and many more. An example for this would be the instant recommendations of Amazon.com and Netflix which are a result of split-second advertising decisions based on the items a user has previously seen or is viewing right now, which has resulted in increased revenues for both of them.


    The increasing amount of data generated by humans while doing their daily chores has lead us into a world where the purchases and habits of a consumer give us an insight into the behaviour of the consumer, how he reacts to various offers , what are his preferences, Is the customer a sort of person who will pay back loans. The algorithms have been designed and implemented which predict accurately how likely a person is to get diabetes. All this has a potential to give rise to platform of customization for every consumer i.e. mass customization.


    A recent study by MIT professor Erik Brynjolfsson has brought out the fact that most revolutions in science begin with better methods of measurement. When we can see new things, we are driven to seek answers and thus build new ways of thinking and operating. The main underlying part of his findings was the fact that organisations that relied on data driven decision making, enhanced their performance by 4 – 6 % viz–a-viz their peers in the same market. This figure has remained robust despite taking into account the contributions for labour, capital, purchased services and traditional IT investment. (Lohr)


    Another research conducted by Mr. Erik along with Mr. Lynn Wu, an assistant professor from Wharton Business School used publicly available web search data to predict housing-price-changes in metropolitan areas across the United States. They didn’t possess any special knowledge of the housing market when they began their study, but they reasoned that virtually real time search data would enable good near term forecasts about the housing market- and they were right. In fact their predictions proved to be more accurate than the official ones generated by the National Association of Realtors which had far more complex model but relied on slow changing historical data. (Brynjolfsson)


    Now if we analyze the business that exist today and have been running on intuition, tradition and convenience, we observe that they have scattered staff development programs, the budgetary decisions are based on prior practice and programs, Staff assignments are based on interest and availability and goal setting is based on votes , favourite initiatives or fads. In contrast, the companies that have adopted data driven decision making for the staff development programs are more focussed as an improvement strategy to address the problems identified by the analysis of data. Staff assignments are based on skills required for future as well as the present requirement. Goal settings are done on the basis of problems identified by data analysis and their possible explanations.


    One factor that we must consider is that with the ever increasing rapidly changing pace of technology, companies today have multiple points of contact with the customer i.e. blogs, call centres , customer reviews or simple comments on social media platforms which are all a valuable source about client segments , insights and behaviour. Today it can be observed that increasing competition, decreasing margins and availability of information has left many companies struggling for survival. The only way out seems to stem from the fact that it is very important today to analyse the vast amount of data being generated daily and convert it into actionable knowledge so that client needs are addressed adequately. (Mike Lynch)


    An example worth mentioning here is that of researchers at John Hopkins School of Medicine, who provide a classic example of how simple models and big data win over more elaborate analytic approaches. They used the data from Google Flu Trends, which is publicly available, to forecast surges in flu related emergency visits a week before similar warning were received from the Centres of Disease Control. Similarly, twitter updates provided accurate records similar to the official reports that were tracking the spread of cholera in Haiti after the 2010 earthquake. (Brynjolfsson)


    From the numerous examples quoted above, it can be substantiated that data analytics unlock huge amounts of useful information, which can be made more transparent in nature by use of Big Data. It is often seen that we search for data in forms other than electronic and then transfer it to electronic form. This effort represents a huge inefficiency; if we could the information directly from other forms to electronic form without human intervention it could save us a lot of resources and time.


    Our online financial transactions or any transaction for that matter, if and when collected over a period of time can bring out interesting facts about our choices and behaviour. Now if we apply the same to a huge organization, what we have is copious amount of data that can provide us with very accurate and detailed information about various aspects of the organization. This further helps the middle level and upper level management make meaningful and informed decisions regarding day-today decisions to designing long term plans for the organizations along with requisite data to support it.


    Data collected from consumer transactions can help us predict the behaviour of the customers and help the organisations in maintaining requisite inventory levels. They can also tailor and customize the products and services according to the needs of an individual customer without the customer making an additional effect in telling the provider about it. It also leads to narrower segmentation of customers.


    Data obtained when combined with power analytics techniques can be used to find hidden patterns which otherwise would remain hidden and help in predicting future needs , improvise after-sale services and meet the expectations of the consumers in a holistic manner. In short the transfer of information between consumer and manufacturer can be defined as the art of communicating; without actually communicating; as no formal/informal communication takes place between the parties’ involved; just observation, which leads to predicting the needs and aspirations of consumers. We need to understand that the data is not the only tool for the decision making. If applied in isolation without experience, the end result is a Frankenstein’s monster of business process and best practices woven together. As value increases its dependency on unique knowledge and human experience, objective data sources must be supplemented with perspectives from organizational cultures, communities of interest and other relationships. Only then can we take holistic and sound decisions.


    Bibliography (Debates, 2011)

    Bonabeau, E. (n.d.). Harvard Business Review . Retrieved 11 21, 2013, from hbr.org : http://hbr.org/2003/05/dont-trust-your-gut/ar/5

    Bono, E. D. (n.d.). Search Quotes. Retrieved 11 21, 2013, from searchquotes.com: http://www.searchquotes.com/quotes/author/Edward_De_Bono/3/

    Brynjolfsson, A. M. (n.d.). Harvard Business School . Retrieved 11 21, 2013, from Harvard Business School Website : http://hbr.org/2012/10/big-data-the-management-revolution

    Christensen, C. (n.d.). Design Intelligence. Retrieved from di.net: http://www.di.net/articles/the-innovators-dilemma-when-new-technologies-cause-great-firms-to-fail/

    Cokins, G. (n.d.). analytics-magazine. Retrieved 11 21, 2013, from analytics-magazine.org : http://www.analytics-magazine.org/may-june-2012/577-corporate-decision-making-why-do-large-once-successful-companies-fail

    Debates, D. (2011). When it comes to important business decisions , should you trust your gut or follow the numbers . Deloitte Development LLC.

    Lohr, S. (n.d.). New York Times . Retrieved 11 21, 2013, from New York Times Website : http://www.nytimes.com/2011/04/24/business/24unboxed.html?_r=0

    Mike Lynch, c. e. (n.d.). Financial Times . Retrieved 11 21, 2013, from Financial Times Website : http://www.ft.com/cms/s/0/62f37a4a-931c-11de-b146-00144feabdc0.html#axzz2lITs0aA5

    Plisch, B. G. (2004, 02 04). WTN News . Retrieved 11 21, 2013, from WTN News Website : http://wtnnews.com/articles/557/

    Tim McGuire, J. M. (n.d.). Ivey Business Journal . Retrieved 11 21, 2013, from iveybusinessjournal.com: http://iveybusinessjournal.com/topics/strategy/why-big-data-is-the-new-competitive-advantage#.Uo47PsXlbrN

    Wladawsky-Berger, I. (n.d.). Irving Wladawsky-Berger. Retrieved 11 21, 2013, from Irving Wladawsky-Berger blog: http://blog.irvingwb.com/blog/2013/09/data-driven-decision-making-promises-and-limitations.html

    Zwilling, M. (n.d.). Forbes . Retrieved 11 21, 2013, from Forbes.com: http://www.forbes.com/sites/martinzwilling/2012/03/25/too-many-leaders-still-rely-on-their-golden-gut/


    Note - Visitors shall not be able to comment on this article until they are logged in.

    User Feedback

    Recommended Comments

    Excellent article. I have a question - How do you suggest use of data in decision making when the product belongs to an entirely new category with no antecedents? Forecasts are unlikely to work and gut feel probably reigns supreme. The question is important as the biggest profits in the corporate world are derived from innovations. A response with examples will be appreciated. 

    Link to comment
    Share on other sites

    Hi Vishwadeep 


    Thank You for the review


    In today’s world , innovation is the key to success but how far it will be successful depends on the amount of under served needs of the customer it address to. We all are capable of innovation but the key differentiator is the amount of success the individual innovation is capable of achieving. This is where data comes into picture. Data analytics can uncover patterns where none exist and help us in giving a direction towards the same. Every innovation we make can only be successful if it addresses the customer’s needs and wants. With copious amount data available today, it is not impossible for us to uncover the under served needs of customers. Data driven decision making can also be used in developing real world models virtually on which multiple scenarios can be tested with the help of predictive analytics.

    The data being generated today has created a transparency between the various stakeholders involved and usable at high frequency.  Data has helped in bridging the gap between customer and manufactures. This can be seen from the fact the today companies are being increasingly focused on collection of data which further helps them in customizing their product for the customer. What we are looking at is customization on a massive scale given the fact that big data allows even narrower segmentation of customers and thereby enabling development of much more precisely tailored products or services.

     Another example of innovation and data analytics is Amazon has recently come up with an innovative of delivering goods within 30 minutes of their purchase. A futuristic mini-drone delivering plan, which are programmed to operate autonomously and drop items at the target locations according to GPS coordinates fed into them. Now how did they come up with such an innovation? The answer to that lies in predictive data analytics. They analysed data and came up with such an idea.

    Today organizations can create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days, and thereby exposing variability and boosting performance and innovation accordingly.

    Big data can be used to improve the development of the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance (preventive measures that take place before a failure occurs or is even noticed) and also offering products similar to the ones purchased by the customer. For example Tesco has come with an innovative idea by merging innovation and predictive analytics. It has initiated offers, in real time, products related to existing purchases. A loyalty card is inserted into a device that is fixed to a shopping trolley, through which tailored offers are made to the shopper as they place items in their trolley. For example, a shopper may place a barbecue chicken in the trolley and then receive a 3-for-2 offer related to the item, such as charcoal for barbecues. By recognizing who the customer is and what purchases they have made, Tesco is attempting to predict what they are going to do next and intervene with relevant offers. It is a simple premise – and one that, if successful, could attract customers and lead to greater profits.

    Some financial companies have started recognising that a certain type of customer responds negatively to payment reminders – if left alone they will make the payment, but if nudged they will dig their heels in and delay payment. By identifying these behavioural trends, companies are able to optimize their operations and improve customer relations. Now who would have thought that customer relations, which involve human contact like face to face communication, could be merged with data analytics and used to improve the same? Another example would be of the Gaming company Bigpoint , creator of Battlestar Galatica using predictive analytics to monetize players and increase revenue by a projected 10 to 30 percent every year.

    Seeing all these examples we can see that innovation in whatever form today needs help of data driven decision making to make it successful. It is responsible for increasing the success rate of an innovation by helping the maker in deciding how to market the innovation and reap huge profits.

    Link to comment
    Share on other sites

    Hello Deepak


    A very well written article , I did like to know exactly how is it that people who have created industries and conglomerates from scratch never bothered about data and today you say that gut based decision making is wrong and people based on gut based decision making stand to lose.



    Link to comment
    Share on other sites

    Hi Utkarsh 


    Thank you for the response 


    Utkarsh  allow me to bring to your notice the fact that we have moved from a world where mass consumption and mass production was the order of the day and the consumer was more obsessed with consuming the product rather than worrying what more it could derive the product . Today a customer looks more at what intangible value the product provides along with intangible value. People today want value for money, more for less, mass customization is the order of the day. There was a time when a single product satisfied a large customer base but today every customer wants a customized product for himself/herself. 


    Today you cannot afford to make decision based on gut alone , you need statistical data to support your ideas, innovation. With gut based decision making  you might succeed or fail but with the help of data analytics esp. predictive data analytics you will be able to forecast future demands, initiate innovation where none exists because data analytics are able to find a pattern within customer behaviour, which might end up increasing your chances of success. Data driven decision making is not fool proof but is tool to aid your decision making. So that you make an informed decision rather than going solely on the basis of your gut. Companies have been forced to shut down because they did not embrace data driven decision making. Data driven decision making is what has made companies like Tesco, Wal-Mart, and Amazon to name a few. How did they beat competition??? , How did they increase their sales? The answer to these two questions lies in the fact that they collected customer data, quality control data and used it to improve their processes and marketing techniques. Today with the rapid pace of technological development, a person surfing the net leaves his/her footprints all over the internet in the form of small amounts of data. Companies collect this data, process it and then use it target customers. For example, by recognizing who the customer is and what purchases they have made, Tesco is attempting to predict what they are going to do next and intervene with relevant offers. It is a simple premise – and one that, if successful, could attract customers and lead to greater profits.


    Consider retailing for example. Traditionally, booksellers in physical stores could always keep a track which books sold and which did not. If they had a loyalty program, they could tie some of those purchases to individual customers. And that was about it. Once shopping moved online, though, the understanding of customers increased dramatically. Online retailers could track not only what customers bought, but also what else they looked at; how they navigated through the site; how much they were influenced by promotions, reviews, and page layouts; and similarities across individuals and groups. Before long, they developed algorithms to predict what books individual customers would like to read next—algorithms that performed better every time the customer responded to or ignored a recommendation. Traditional retailers simply couldn’t access this kind of information, let alone act on it in a timely manner. It’s no wonder that Amazon has put so many brick-and-mortar bookstores out of business.

    Big data has the potential to transform traditional business also as it can offer them even greater opportunities of competitive advantage. We can measure therefore can manage more precisely than ever before. Data driven decision making helps us by making better predictions and smarter decisions. Today we can target more effective interventions, and can do so in areas that have been so far dominated by gut and intuition rather than by data and rigor.

    I hope I have been able to satisfy your query


    Link to comment
    Share on other sites

    Dear Deepak Dont you think using data based decision making shows no regard for people’s privacy. I mean you are saying that collecting data of an individual, I wonder how can we say any more that we have privacy , it looks like big brother is always watching us. what is your opinion on that.

    Link to comment
    Share on other sites

    Hi Deepak


    A very well written article Can you shed some light on how data driven decision making can be used in education??


    Hi Rupak

    Thanks for the appreciation

    It is known fact that educators and education system have forever worked under the assumption that there are only certain problem areas. These problem areas were not backed by actual data or statistics but were based on the assumptions and perceptions of our educators. Often the team responsible for school improvement would end up addressing up a problem that may or may exist, thereby wasting precious resources and time. Data driven decision making here comes to the rescue. Data provides the educators with an overview of the strengths and weakness in a targeted area.


    Once the targeted areas are identified , educators can commit their resources towards producing more efficient and effective programs. With the help of data analytics programs can be streamlined towards the specific areas. For example, in a school that does not use data-driven decision making, professional development may look scattered and unintentional.  The staff may seem uninterested, or find the information not useful because they see no use to their particular area.  If the school uses data-driven decision making to determine professional development, the staff will benefit because professional development will be focused toward their needs.


    When educators are able to draw inferences from data, they can, not only see the need for change, but can also identify the direction in which the change is needed, pinpoint the students needing intervention, and identify approaches offering promising solutions to help the respective students succeed. The use of multiple, and sometimes creative, sources of data enables school leaders to make mid-course corrections and continuous improvement toward academic success by their students.


    School that engage in data-driven decision making, have the information that not only measures students' progress in meeting standards, but also enables them in assessing current and future needs of students, parents, staff and the community; determine if goals are being met; ensure that students are not falling through the cracks; improve instructions; identify the root cause of problems; and, engage in continuous school improvement and help the students in their overall and all round development. To summarize I can say that data driven decision making can help in the following ways:-


    ·         Narrow achievement gaps

    ·         Improve teacher quality

    ·         Improve curriculum-development

    ·         Find root cause of problems

    ·         Share best practices

    ·         Communicate more effectively with key stake holder

    ·         Motivate students and increase parental improvement


    For example:-

    St. Paul is an urban district serving 43,000 students in Minnesota. With 34 percent of its students identified as Limited English Proficient and a high mobility index of 22 percent, analyzing student achievement proved to be difficult. To better meet the needs of its students, the district piloted a data-driven decision making system at one of its schools, Arlington Senior High School. The trial was set up as a targeted assistance model, serving the “highest” needs students who receive the most assistance.


    To do that, staff needed to examine test data on a regular basis and use it to identify the most academically challenged students in order to improve their test scores and provide them with an opportunity to graduate.



    Prior to implementing the data-driven decision making system, teachers would start the year with little or no knowledge of their students’ past performance. Data was only available for students who had taken tests in their building the previous spring, and it was not easy to use and required extra time for the teacher to retrieve. As a result of these obstacles, teachers used only overall test scores and pass/fail status.


    Data-driven decision making changed all of that. Upon implementation of data analysis techniques, teachers were able to look at scale score improvements on standardized tests to evaluate their curriculum, instruction and services. They could look at strands within a test to determine trends in performance and adjust curriculum and instruction accordingly. They also looked at strands to identify and assist students through the school’s comprehensive tutoring program.


    Rather than having to key in data manually, the student information was updated nightly, giving teachers real-time information about the students sitting in front of them. They could review classes by overall test score/performance and drill down to the strand/sub skill level to see how students performed. As a result of looking at data by class and grade level, Arlington began curriculum mapping and a building-wide literacy initiative. Seeing the data in "black and white" and taking the time to understand and interpret the data helped Arlington solve critical building-wide curriculum and instructional issues.


    The project was so successful that efforts are underway to expand the data-driven decision making system’s use across the district.


    Hope this address your query 


    Link to comment
    Share on other sites

    Hi Deepak,


    You seem to have struck a chord with the business managers, with the logic of using data instead of intuition to make these decisions. It is one of the most fundamental principles of decision making,one that should be kept in mind at all costs. Unfortunately, a lot of times, people tend to overlook this fact, and end up making wrong decisions. You have reminded, with relevant examples, how data driven decisions can minimize the risk of a decision. A very well written article indeed, complete with the right flow of thoughts and ideas.

    Link to comment
    Share on other sites

    Hi Deepak,


    You seem to have struck a chord with the business managers, with the logic of using data instead of intuition to make these decisions. It is one of the most fundamental principles of decision making,one that should be kept in mind at all costs. Unfortunately, a lot of times, people tend to overlook this fact, and end up making wrong decisions. You have reminded, with relevant examples, how data driven decisions can minimize the risk of a decision. A very well written article indeed, complete with the right flow of thoughts and ideas.


    Thank you Prateek Chaturvedi

    Link to comment
    Share on other sites

    As its commonly said "If you're not using data to make decisions, you're flying blind". D3M process also always begin with these questions: "How's business? How do you know it? And what can you do to improve it? Deepak you have put forward this in a very lucid way. Nice article!!

    Link to comment
    Share on other sites

    As its commonly said "If you're not using data to make decisions, you're flying blind". D3M process also always begin with these questions: "How's business? How do you know it? And what can you do to improve it? Deepak you have put forward this in a very lucid way. Nice article!!


    Thank You Mukul Chandola 

    Link to comment
    Share on other sites

    This is now closed for further comments

  • Create New...