Jump to content
  • Sign in to follow this  

    Data Driven Decision Making - A Holistic Approach To Decision Making


    “A point of view can be a dangerous luxury when substituted for insight and understanding. – Marshall Mcluhan â€

    We all have come a long way from industrial revolution where mass production and consumption was the order of the day. Over the years, the once not so knowledgeable consumer has transformed into a living storehouse of information. What has changed over the years? Why did the companies who were the industry leaders at one point of time failed miserably later on? What changed? The answer to all this lies in data. In our thirst to find newer markets for products and differentiate them from our competitors, we pushed the consumer towards seeking knowledge and then when he demanded, we failed to live up to standards.


    We pushed the customer to give more importance to value i.e. value today has become emergent as compared to the old times when it was measurable. Now the new age customer wants more and he/she will switch, the moment he/she finds more value from another product or another competitor. How do we live up to this? Given the rapid changes in technology and the rapid adoption of management information systems by almost all industries, we need not look further. We have the solutions but we are too thick skinned to look at it. Today’s customer is like a walking hard drive with copious amounts of data. What we need to do is analyse it, find patterns where none exist, combine it with the experience of mankind and predict the needs and demands of future backed by strong data analysis.


    If we stop and look back into the history what will be distinctly visible to us is the fact that companies that were present in the first list of Standard & Poor’s 500 only 74 remain in operation today. Almost half of the 25 companies that passed the rigorous test for inclusion in Tom Peters and Robert Waterman’s 1982 book “In Search of Excellenceâ€, have gone bankrupt or have shut down due to poor performance.


    The reason being that when organisations are thriving and successful, it creates an illusion that whatever gave them success in the past will work in the present and still remain valid in the future. They forget the fact that the world is constantly changing, constantly evolving and to survive they need to evolve with it too. Today there is no dearth of examples of companies that failed after achieving success on a grand scale, to quote a few, Wang Labs, Digital Equipment, Borders and Blockbusters and many more; of which many either fell after achieving industry leadership position or shrunk in size.


    The reason is not that they failed to act or didn’t know what was happening but the fact that they had all the data but didn’t analyse it. The patterns were there, had they analysed the data they already possessed they would have seen it. They believed that experience can upstage data but they failed to comprehend that with today’s rapidly changing global economy, it’s impossible for the human mind to absorb all factors that go into making sound decisions. Statistical analysis often uncovers correlations that no one could have predicted on the basis of their past experiences. We won’t be wrong in accepting the fact that data and statistical analysis isn’t perfect but who hasn’t made poor decisions on pure gut instinct!! It is said that sustainability is based on transforming data into analyzable information for insights and decision-making. This is where data driven decision making comes in and plays its part. (Cokins)


    Take for example the case of Wal-Mart as reported in New-York Times. This occurred when Hurricane Frances was moving across the Caribbean, threatening to directly hit on Florida’s Atlantic coast. Residents were making a rush for the higher ground, but at the same time, far away, in Bentonville, Ark., executives presiding over the Wal-Mart Stores decided that the situation opened a great window of opportunity for their newest data-driven weapons, Predictive technology. A week ahead of storms predicted landfall date, Linda M. Dillman Wal-Mart’s CIO at that point of time asked her staff to come up with forecasts based on what happened when Hurricane Charley struck several weeks earlier. Their experts mined for data and found that the stores would need certain products more than the usual batteries, flashlights, and bottles of water.


    All this brought out some very interesting facts about the pre-storm behaviour of consumers. Strawberry pop tarts and beer were the top-selling items. By predicting what was going to happen rather than waiting for it to happen, Wal-mart was able to deal with this unexpected situation in a holistic manner.


    It is said that if we can improve our ability to estimate for a given customer, we stand to gain by applying the same logic to millions of customers. This applies to many other areas where intense application of data science and data mining is seen like direct marketing, credit scoring, financial trading, help-desk management, fraud detection, search ranking, product recommendation and many more. An example for this would be the instant recommendations of Amazon.com and Netflix which are a result of split-second advertising decisions based on the items a user has previously seen or is viewing right now, which has resulted in increased revenues for both of them.


    The increasing amount of data generated by humans while doing their daily chores has lead us into a world where the purchases and habits of a consumer give us an insight into the behaviour of the consumer, how he reacts to various offers , what are his preferences, Is the customer a sort of person who will pay back loans. The algorithms have been designed and implemented which predict accurately how likely a person is to get diabetes. All this has a potential to give rise to platform of customization for every consumer i.e. mass customization.


    A recent study by MIT professor Erik Brynjolfsson has brought out the fact that most revolutions in science begin with better methods of measurement. When we can see new things, we are driven to seek answers and thus build new ways of thinking and operating. The main underlying part of his findings was the fact that organisations that relied on data driven decision making, enhanced their performance by 4 – 6 % viz–a-viz their peers in the same market. This figure has remained robust despite taking into account the contributions for labour, capital, purchased services and traditional IT investment. (Lohr)


    Another research conducted by Mr. Erik along with Mr. Lynn Wu, an assistant professor from Wharton Business School used publicly available web search data to predict housing-price-changes in metropolitan areas across the United States. They didn’t possess any special knowledge of the housing market when they began their study, but they reasoned that virtually real time search data would enable good near term forecasts about the housing market- and they were right. In fact their predictions proved to be more accurate than the official ones generated by the National Association of Realtors which had far more complex model but relied on slow changing historical data. (Brynjolfsson)


    Now if we analyze the business that exist today and have been running on intuition, tradition and convenience, we observe that they have scattered staff development programs, the budgetary decisions are based on prior practice and programs, Staff assignments are based on interest and availability and goal setting is based on votes , favourite initiatives or fads. In contrast, the companies that have adopted data driven decision making for the staff development programs are more focussed as an improvement strategy to address the problems identified by the analysis of data. Staff assignments are based on skills required for future as well as the present requirement. Goal settings are done on the basis of problems identified by data analysis and their possible explanations.


    One factor that we must consider is that with the ever increasing rapidly changing pace of technology, companies today have multiple points of contact with the customer i.e. blogs, call centres , customer reviews or simple comments on social media platforms which are all a valuable source about client segments , insights and behaviour. Today it can be observed that increasing competition, decreasing margins and availability of information has left many companies struggling for survival. The only way out seems to stem from the fact that it is very important today to analyse the vast amount of data being generated daily and convert it into actionable knowledge so that client needs are addressed adequately. (Mike Lynch)


    An example worth mentioning here is that of researchers at John Hopkins School of Medicine, who provide a classic example of how simple models and big data win over more elaborate analytic approaches. They used the data from Google Flu Trends, which is publicly available, to forecast surges in flu related emergency visits a week before similar warning were received from the Centres of Disease Control. Similarly, twitter updates provided accurate records similar to the official reports that were tracking the spread of cholera in Haiti after the 2010 earthquake. (Brynjolfsson)


    From the numerous examples quoted above, it can be substantiated that data analytics unlock huge amounts of useful information, which can be made more transparent in nature by use of Big Data. It is often seen that we search for data in forms other than electronic and then transfer it to electronic form. This effort represents a huge inefficiency; if we could the information directly from other forms to electronic form without human intervention it could save us a lot of resources and time.


    Our online financial transactions or any transaction for that matter, if and when collected over a period of time can bring out interesting facts about our choices and behaviour. Now if we apply the same to a huge organization, what we have is copious amount of data that can provide us with very accurate and detailed information about various aspects of the organization. This further helps the middle level and upper level management make meaningful and informed decisions regarding day-today decisions to designing long term plans for the organizations along with requisite data to support it.


    Data collected from consumer transactions can help us predict the behaviour of the customers and help the organisations in maintaining requisite inventory levels. They can also tailor and customize the products and services according to the needs of an individual customer without the customer making an additional effect in telling the provider about it. It also leads to narrower segmentation of customers.


    Data obtained when combined with power analytics techniques can be used to find hidden patterns which otherwise would remain hidden and help in predicting future needs , improvise after-sale services and meet the expectations of the consumers in a holistic manner. In short the transfer of information between consumer and manufacturer can be defined as the art of communicating; without actually communicating; as no formal/informal communication takes place between the parties’ involved; just observation, which leads to predicting the needs and aspirations of consumers. We need to understand that the data is not the only tool for the decision making. If applied in isolation without experience, the end result is a Frankenstein’s monster of business process and best practices woven together. As value increases its dependency on unique knowledge and human experience, objective data sources must be supplemented with perspectives from organizational cultures, communities of interest and other relationships. Only then can we take holistic and sound decisions.


    Bibliography (Debates, 2011)

    Bonabeau, E. (n.d.). Harvard Business Review . Retrieved 11 21, 2013, from hbr.org : http://hbr.org/2003/05/dont-trust-your-gut/ar/5

    Bono, E. D. (n.d.). Search Quotes. Retrieved 11 21, 2013, from searchquotes.com: http://www.searchquotes.com/quotes/author/Edward_De_Bono/3/

    Brynjolfsson, A. M. (n.d.). Harvard Business School . Retrieved 11 21, 2013, from Harvard Business School Website : http://hbr.org/2012/10/big-data-the-management-revolution

    Christensen, C. (n.d.). Design Intelligence. Retrieved from di.net: http://www.di.net/articles/the-innovators-dilemma-when-new-technologies-cause-great-firms-to-fail/

    Cokins, G. (n.d.). analytics-magazine. Retrieved 11 21, 2013, from analytics-magazine.org : http://www.analytics-magazine.org/may-june-2012/577-corporate-decision-making-why-do-large-once-successful-companies-fail

    Debates, D. (2011). When it comes to important business decisions , should you trust your gut or follow the numbers . Deloitte Development LLC.

    Lohr, S. (n.d.). New York Times . Retrieved 11 21, 2013, from New York Times Website : http://www.nytimes.com/2011/04/24/business/24unboxed.html?_r=0

    Mike Lynch, c. e. (n.d.). Financial Times . Retrieved 11 21, 2013, from Financial Times Website : http://www.ft.com/cms/s/0/62f37a4a-931c-11de-b146-00144feabdc0.html#axzz2lITs0aA5

    Plisch, B. G. (2004, 02 04). WTN News . Retrieved 11 21, 2013, from WTN News Website : http://wtnnews.com/articles/557/

    Tim McGuire, J. M. (n.d.). Ivey Business Journal . Retrieved 11 21, 2013, from iveybusinessjournal.com: http://iveybusinessjournal.com/topics/strategy/why-big-data-is-the-new-competitive-advantage#.Uo47PsXlbrN

    Wladawsky-Berger, I. (n.d.). Irving Wladawsky-Berger. Retrieved 11 21, 2013, from Irving Wladawsky-Berger blog: http://blog.irvingwb.com/blog/2013/09/data-driven-decision-making-promises-and-limitations.html

    Zwilling, M. (n.d.). Forbes . Retrieved 11 21, 2013, from Forbes.com: http://www.forbes.com/sites/martinzwilling/2012/03/25/too-many-leaders-still-rely-on-their-golden-gut/


    Note - Visitors shall not be able to comment on this article until they are logged in.

    Sign in to follow this  

    User Feedback

    Recommended Comments

    Dear Deepak Dont you think using data based decision making shows no regard for people’s privacy. I mean you are saying that collecting data of an individual, I wonder how can we say any more that we have privacy , it looks like big brother is always watching us. what is your opinion on that.


    Hi Adit


    Thanks for your appreciation


    Almost everyone today considers data collection by organizations as a breach of privacy. As a result of which data governance is considered to be emerging discipline with an evolving definition. The discipline is a converging point for data management, data quality, data policies, business process management and risk management surrounding the handling of data in an organization. It’s through data governance that organizations and governments are looking to exercise positive control over the process and methods in use by their own data steward and data custodians.


    The Personal Data Protection Act has also been implemented by various countries. The PDPA establishes a data protection law that comprises various rules governing the collection, use, disclosure and care of personal data. It recognises both the rights of individuals to protect their personal data, including rights of access and correction, and the needs of organisations to collect, use or disclose personal data for legitimate and reasonable purposes.


    The PDPA provides for the establishment of a national Do Not Call (DNC) Registry. The DNC Registry allows individuals to register their telephone numbers here to opt out of receiving marketing phone calls, mobile text messages such as SMS or MMS, and faxes from organisations.


    A process know as de-identification(DeID-T) has also been implemented by many organizations and government agencies where in the organizations remove or obscure links between an individual’s identity and the individual’s personal information. This process involves deleting or masking personal identifiers, such as names and social security numbers, and suppressing or generalizing quasi-identifiers, such as dates of birth and zip codes. By using technical de-identification, organizations can transform sensitive information from being fully individually identifiable to being unconnected to any particular person. With publicly disclosed datasets, DeID-T provides the sole line of defense protecting individual privacy.


    De-identification is a process used to prevent a person’s identity from being connected with information. Organizations de-identify data for a range of reasons. Companies may have promised “anonymity” to individuals before collecting their personal information, data protection laws may restrict the sharing of personal data, and, perhaps most importantly, companies de-identify data to mitigate privacy threats from improper internal access or from an external data breach. Hackers and dishonest employees occasionally uncover and publicly disclose the confidential information of individuals. Such disclosures could prove disastrous, as public dissemination of stigmatizing or embarrassing information, such as a medical condition, could negatively affect an individual’s employment, family life, and general reputation. Given these negative consequences, industries and regulators often rely on de-identification to reduce the occurrence and harm of data breaches.


    Though I agree with you many laws have been enacted by various governments but are yet to be implemented in a comprehensive manner also what is required is two way understanding between people and organizations so that every stakeholder’s concerns are taken care of.



    I hope this helps


    Share this comment

    Link to comment
    Share on other sites

    Hi Utkarsh 


    Thank you for the response 


    Utkarsh  allow me to bring to your notice the fact that we have moved from a world where mass consumption and mass production was the order of the day and the consumer was more obsessed with consuming the product rather than worrying what more it could derive the product . Today a customer looks more at what intangible value the product provides along with intangible value. People today want value for money, more for less, mass customization is the order of the day. There was a time when a single product satisfied a large customer base but today every customer wants a customized product for himself/herself. 


    Today you cannot afford to make decision based on gut alone , you need statistical data to support your ideas, innovation. With gut based decision making  you might succeed or fail but with the help of data analytics esp. predictive data analytics you will be able to forecast future demands, initiate innovation where none exists because data analytics are able to find a pattern within customer behaviour, which might end up increasing your chances of success. Data driven decision making is not fool proof but is tool to aid your decision making. So that you make an informed decision rather than going solely on the basis of your gut. Companies have been forced to shut down because they did not embrace data driven decision making. Data driven decision making is what has made companies like Tesco, Wal-Mart, and Amazon to name a few. How did they beat competition??? , How did they increase their sales? The answer to these two questions lies in the fact that they collected customer data, quality control data and used it to improve their processes and marketing techniques. Today with the rapid pace of technological development, a person surfing the net leaves his/her footprints all over the internet in the form of small amounts of data. Companies collect this data, process it and then use it target customers. For example, by recognizing who the customer is and what purchases they have made, Tesco is attempting to predict what they are going to do next and intervene with relevant offers. It is a simple premise – and one that, if successful, could attract customers and lead to greater profits.


    Consider retailing for example. Traditionally, booksellers in physical stores could always keep a track which books sold and which did not. If they had a loyalty program, they could tie some of those purchases to individual customers. And that was about it. Once shopping moved online, though, the understanding of customers increased dramatically. Online retailers could track not only what customers bought, but also what else they looked at; how they navigated through the site; how much they were influenced by promotions, reviews, and page layouts; and similarities across individuals and groups. Before long, they developed algorithms to predict what books individual customers would like to read next—algorithms that performed better every time the customer responded to or ignored a recommendation. Traditional retailers simply couldn’t access this kind of information, let alone act on it in a timely manner. It’s no wonder that Amazon has put so many brick-and-mortar bookstores out of business.

    Big data has the potential to transform traditional business also as it can offer them even greater opportunities of competitive advantage. We can measure therefore can manage more precisely than ever before. Data driven decision making helps us by making better predictions and smarter decisions. Today we can target more effective interventions, and can do so in areas that have been so far dominated by gut and intuition rather than by data and rigor.

    I hope I have been able to satisfy your query


    Thank you so much


    Share this comment

    Link to comment
    Share on other sites

    Hi Rupak

    Thanks for the appreciation

    It is known fact that educators and education system have forever worked under the assumption that there are only certain problem areas. These problem areas were not backed by actual data or statistics but were based on the assumptions and perceptions of our educators. Often the team responsible for school improvement would end up addressing up a problem that may or may exist, thereby wasting precious resources and time. Data driven decision making here comes to the rescue. Data provides the educators with an overview of the strengths and weakness in a targeted area.


    Once the targeted areas are identified , educators can commit their resources towards producing more efficient and effective programs. With the help of data analytics programs can be streamlined towards the specific areas. For example, in a school that does not use data-driven decision making, professional development may look scattered and unintentional.  The staff may seem uninterested, or find the information not useful because they see no use to their particular area.  If the school uses data-driven decision making to determine professional development, the staff will benefit because professional development will be focused toward their needs.


    When educators are able to draw inferences from data, they can, not only see the need for change, but can also identify the direction in which the change is needed, pinpoint the students needing intervention, and identify approaches offering promising solutions to help the respective students succeed. The use of multiple, and sometimes creative, sources of data enables school leaders to make mid-course corrections and continuous improvement toward academic success by their students.


    School that engage in data-driven decision making, have the information that not only measures students' progress in meeting standards, but also enables them in assessing current and future needs of students, parents, staff and the community; determine if goals are being met; ensure that students are not falling through the cracks; improve instructions; identify the root cause of problems; and, engage in continuous school improvement and help the students in their overall and all round development. To summarize I can say that data driven decision making can help in the following ways:-


    ·         Narrow achievement gaps

    ·         Improve teacher quality

    ·         Improve curriculum-development

    ·         Find root cause of problems

    ·         Share best practices

    ·         Communicate more effectively with key stake holder

    ·         Motivate students and increase parental improvement


    For example:-

    St. Paul is an urban district serving 43,000 students in Minnesota. With 34 percent of its students identified as Limited English Proficient and a high mobility index of 22 percent, analyzing student achievement proved to be difficult. To better meet the needs of its students, the district piloted a data-driven decision making system at one of its schools, Arlington Senior High School. The trial was set up as a targeted assistance model, serving the “highest” needs students who receive the most assistance.


    To do that, staff needed to examine test data on a regular basis and use it to identify the most academically challenged students in order to improve their test scores and provide them with an opportunity to graduate.



    Prior to implementing the data-driven decision making system, teachers would start the year with little or no knowledge of their students’ past performance. Data was only available for students who had taken tests in their building the previous spring, and it was not easy to use and required extra time for the teacher to retrieve. As a result of these obstacles, teachers used only overall test scores and pass/fail status.


    Data-driven decision making changed all of that. Upon implementation of data analysis techniques, teachers were able to look at scale score improvements on standardized tests to evaluate their curriculum, instruction and services. They could look at strands within a test to determine trends in performance and adjust curriculum and instruction accordingly. They also looked at strands to identify and assist students through the school’s comprehensive tutoring program.


    Rather than having to key in data manually, the student information was updated nightly, giving teachers real-time information about the students sitting in front of them. They could review classes by overall test score/performance and drill down to the strand/sub skill level to see how students performed. As a result of looking at data by class and grade level, Arlington began curriculum mapping and a building-wide literacy initiative. Seeing the data in "black and white" and taking the time to understand and interpret the data helped Arlington solve critical building-wide curriculum and instructional issues.


    The project was so successful that efforts are underway to expand the data-driven decision making system’s use across the district.


    Hope this address your query 


    Thank You Deepak.  Well explained, got it now.

    Share this comment

    Link to comment
    Share on other sites

    Very nice article Deepak...you seem to have got the secret for correct decision making, and have presented it in a very easy yet understandable way!!!

    Thank You Kshitij Shandilya

    Share this comment

    Link to comment
    Share on other sites

    This is now closed for further comments

  • Create New...