Jump to content
  • My Experiments With Data Driven Decision Making (D3M)


    decision-making-skills.jpg

    As a Computer Illiterate growing up in the new millennium, I had tremendous difficulty making sense of how to do most of my stuff. Most of my decisions were snap, on-the-moment and intuitive ones. By the time I was in my teens, I had learnt my way around the big computer problem. My experiences and the Great Indian “Jugaad†mind-set had provided me with a sufficient enough method wherein I only worked on the computer when absolutely necessary. If there was any other way to do the task, any escape route, I gladly followed it.

    By now you must have realised that I was and am sceptic and data-averse. Data to me was always those unreadable files that eat a hell lot of my storage space. I was gaining proficiency in getting around this data problem and along came Big Data. Everyone from Google to Barrack Obama was using it.[1] As a student at one of the most prestigious B-Schools in the knowledge market, not only was I supposed to know what it was, but was expected to be able to tune in and utilise it to make a difference. Life has strange ways of getting back at us, mere mortals, and here was my customised gut wrenching sucker punch. The fighter in me knew I had to do this, but I did not find a way I could.

     

    A 3 month long research in the field led me to Data Driven Decision Making or D3M. Simply put, it is the pleasure of sipping a coffee in your office chair while you watch your computer work up some algorithms and provide you invaluable decision making tips to face some of your most frequently encountered problems.

     

    Immediately after, I experimented with the different sorts of decisions that can be made with D3M. The results were a true eye-opener. If we think of decision making as a broad spectrum from operational decisions at one end to strategic decisions at the other. Operational decisions can be characterized as highly structured, routine, short-term oriented and increasingly embodied in sophisticated software applications. On the other hand strategic decisions are taken by the top management and serve to set the long-term directions, policies and procedures of an organization. They tend to be complex and unstructured because of the uncertainty and risks that generally accompany longer term decisions. In between these two extremes, we can have varied decisions including non-routine ones in response to new or unforeseen circumstances beyond the scope of operational processes, and tactical decisions dealing with the necessary adjustments required to implement longer term strategies. [2]

    Evaluation2.jpg

     

    Half a decade ago, D3M could have only helped you with the more structured forms of decision making but no more. With the advent of Big Data, machines know much more about humans and human behaviour than humans themselves. Sample this, personal analytics can actually allowed me to analyse my Whatsapp chat history and find why most of the girls I chatted with refused me a date. The word cloud usage showed me that my texting skills were uninspiring to say the least with the most common words being “ok†and “yaâ€. I realised I needed to be more creative and engage better. What did not help was that most girls chatted in the window of 8-12pm whereas I had the habit of taking a short nap at the time. Also it helped me identify my most productive work hours and helped me plan out my work better.

     

    Thus far, all seemed well with D3M but on further research most early moving corporates into the field were still undecided on its benefits. A closer look exposed a distinctly similar pattern in them. Most early moving managers thought that D3M will help them save money or time or both. Research shows that they are misguided to say the least. D3M does neither, atleast not in the short term. What D3M does allow though is discovering solutions you never knew existed. What it will do though is finding needles in the haystack consistently. Another interesting insight was that D3M depends a lot on the data collection. Great collection leads to great results. All we are required to do is ensure data cleanliness, variety and velocity.

    In his book, Data Driven: Profiting from Your Most Important Business Asset, Prof. Thomas Redman summarizes the whole decision making via data as “Good decision makers follow at least three Bayesian principles. First, they bring as much of their prior experience as possible to bear in formulating their initial decision spaces and determining the sorts of data they will consider in making the decision. Second, for big, important decisions, they adopt decision criteria that minimize the maximum risk. Third, they constantly evaluate new data to determine how well a decision is working out, and they do not hesitate to modify the decision as needed.â€

     

    After months of thorough investigation and experimentation, I have arrived at this conclusion, “To stay a step ahead in this ever competitive world, using Data Driven Decision making is a must. So let D3M take care of all the external data that you need to work with and let your mind focus on understanding “the data from insideâ€. That will surely lead you to not just success but contentment.

     

    References

    1. http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/

    2. http://blogs.wsj.com/cio/2013/09/27/data-driven-decision-making-promises-and-limits/

     

    Note - Visitors shall not be able to comment on this article until they are logged in.


    User Feedback

    Recommended Comments



    Glad that you liked the article, Sir. As you may have guessed correctly, the above statement is Data Driven. 

     

    Data Driven Decision Making has been extensively used in the field of Education in the US and a GilfusEducation Research in 2012 found out that for Schools using D3M the average monthly spending went up by 10 percent and the average Teacher-Student interaction time decreased by 7 percent due to increased workload of feeding data.

     

    One may counter argue that the same may not be true of instances wherein the focus is on saving money and not improving the 

    quality of education but setting up a good Data Warehouse with efficient Decision support tools will always involve substantial 

    initial outlay.  

     

    It's not all gloom though as a 2011 Study by MIT found that organizations that are driven by D3M have 4% higher productivity and 6% higher profits in the long run.

    Link to comment
    Share on other sites

    nice article nachiketa,, would like to know more about D3M,,,but as you mentioned D3M depends A LOT on data collection ,so  how do we ensure data cleanliness as it must be a hell of a job?

     
    Thanks Hila Jimosaya, 
     
    As the age old adage goes, "Cleanliness is next to godliness"
    I have already pointed out Data cleansing is one of the most important steps in Data Driven Decision Making.
     
    In a Decision support System, Data needs to be consistent and accurate with maximum coverage.
    Most Such systems source Data from a Transaction Processing System and a CRM system with some other external sources
    if needed be. In such a case we need to ensure data integrity, consistency and accuracy. We need to check for Redundancies and duplication. Also certain checks and monitors need to be placed for finding and correcting errors.
     
    Regular Maintenance of the Data warehouse with Quarterly Data Quality Audits can ensure cleanliness.  
    Link to comment
    Share on other sites

    ok,,,could you give me some examples from the past on how accurate and successful the decisions have been by using D3M??....as the link which you provided showed how obama's campaigner used D3M which led him to victory but with only a little more than half of the country's vote going to him(51.1%),,and not to mention that the other half hated him.,so this made me conclude that though i may win using D3M but with only a little more than 50% probabilty ,,what do you say about this??


    Link to comment
    Share on other sites

    Hi Hila,


     


    Some of the recent success stories that I have come across include,


     


    1.  A major cable-TV channel uses viewer and other data to create a highly detailed list of its viewers’ core values and beliefs. This list is central to everything it does, even when developing concepts for new shows.   The company weighs almost every decision it makes against these values. Whatever doesn’t fit in won’t get the green light to continue, even if it has terrific commercial potential.


     


    2. A radio network that produces original reporting uses listener and marketing data to create distinct listener profiles. These profiles shape all programming decisions for both news and feature stories. Any content that falls outside the profiles won’t make it to air.


     


    You can look up any research site for scholarly articles and case studies on successful implementation of Data driven Decision Making. 


     


    I would also suggest that you go through the MIT Centre for Digital Business at the following link. http://searchbusinessanalytics.techtarget.com/news/2240035852/MIT-study-Data-driven-decisions-mean-higher-productivity-profits

    Link to comment
    Share on other sites

    Great work Nachiketa! 


    You presented your argument in a very simple and reader friendly language.


    But can you provide few more technical details, like what is the minimum amount of data that would give us a fairly accurate result and the details of machines that would be required to process the data..

    Link to comment
    Share on other sites

    Very well written Nachiketa. I appreciate it.


     


    Question - Your article described how you transformed from a intuitive to a data driven person. Even in the modern world, sometimes intuitive decisions prove to be far better than data driven ones. In your opinion, where should we draw the line between data driven and intuitive approaches?


    Link to comment
    Share on other sites

    A very interesting read Nachiketa.

    However the basic difference between a  Data Driven approach and Intuition is the ability to take a wild guess.Logic can  only go so far amidst real world noise. We have to take a wild guess occasionally, which is random.

     A significant amount of information we incorporate and process, we just run into it. I wonder how cliched and our restricted our information will be if we get exactly what we are looking for. A solution finds a problem , not the other way around. There are many solutions for questions we might never pursue. Isn't that compromising on innovation and intuition.

    Is Data Driven Decision making really required ?

    Link to comment
    Share on other sites

    Great work Nachiketa! 

    You presented your argument in a very simple and reader friendly language.

    But can you provide few more technical details, like what is the minimum amount of data that would give us a fairly accurate result and the details of machines that would be required to process the data..

    Thanks for the praise Aishwarya. 

     

    I would like to answer your questions one by one.

     

    Firstly, you want to know what is the minimum amount of data required to implement Data Driven Decision Making on a Decision support System. Like in the case of real life, taking a decision is a result of weighing in the pros and cons of every possible state. Similarly in Data Driven Decision Making we try to look into as many aspects as possible and weigh in their pros and cons. Weighing in an aspect/state usually entails quantifying the possible states in positive and negative values. The more data we have, the better and more informed decisions we can take. As I had stated earlier we usually need data from a CRM and the regular system the organization works with along with some Industry Specific or Market Data. Historical records can also be scanned and utilised. There is no specific minimum or maximum limit to the data. If the Decision to be made is simple then the amount of data required is less and so is the accuracy of the decision. If we need accurate results we need to weigh in as many data points as possible.

     

    To answer your second question. Nowadays with the advent of high speed powerful micro processors Data Driven Decision Making does not depend on specialized hardware anymore. Though what is required is getting a specific Decision Support software built for your industry/company by a vendor. Also whats required is specific hardware for the Data Warehouse (the size of which is proportional to the data you'd be using) . Such an investment in a Data warehouse though has additional benfits as it can be used in Data Mining and OLAP systems as well. Out of the three main components of a DSS setting up a knowledge base is more hardware dependent than the other two namely user interface and the model(decision context and user criteria).

     

    I'd suggest that you go through the following links for better clarity on the topic as well as specific hardware requirements calculations.

    Link to comment
    Share on other sites

    Very well written Nachiketa. I appreciate it.

     

    Question - Your article described how you transformed from a intuitive to a data driven person. Even in the modern world, sometimes intuitive decisions prove to be far better than data driven ones. In your opinion, where should we draw the line between data driven and intuitive approaches?

    Dear Ganesh Venkat,

    What you said is absolutely true. In the real world we have so many examples of intuitive decision makers being successful. Two of the best examples that come to my mind are IBM's Deep Thought Computer losing to Kasparov and even its improved version Deep Blue losing the opening game to the former World Champion. One more example could be the continued success of Mr. Warren Buffet in the stock market. 

    But, what we need to look into is the question that are these truly intuitive decisions? 

    Most of the decisions that we humans take are not intuitive, though we might like to think that they are.

    As soon as we are confronted by a situation our mind starts working on it. The processing done by the mind is so fast that we often mistake it for intuition. Even in cases where it is intuition, it is based on some past experiences/memories that we have. 

    In 1921, Carl Jung defined intuition as  "perception via the unconscious: using sense-perception only as a starting point, to bring forth ideas, images, possibilities, ways out of a blocked situation, by a process that is mostly unconscious".

     

    The unconscious as stated above is slightly dependent on our past experiences and prevailing thoughts. The reliability of one's intuition depends greatly on past knowledge and occurrences in a specific area. For example, someone who has had more experiences with children will tend to have a better instinct or intuition about what they should do in certain situations with them. This is not to say that one with a great amount of experience is always going to have an accurate intuition (because some can be biased); however, the chances of it being more reliable are definitely amplified. 

     

    Data driven approach to decision making is more of a left brain type of activity whereas intuitive decision making is a right brain type of activity. The best systems are those that make use of both of them. As in the Case of Deep Blue, Kasparov alleged that some experts controlled some of the moves the computer made. Nowadays, integrating the ingrained knowledge of experts of a particular field while making a DSS for the field is not uncommon. Also nowadays, we can utilize Recognition Primed Decision(RPD) and Intuitionistic Logic while designing D3M systems.  

     

    To sum up my argument, I can say that making intuitive decisions may lead you to runaway success in a situation but using D3M systems would lead you to continual and consistent success. A DSS that can integrate both these approaches is obviously the best choice.

    Link to comment
    Share on other sites

    Dear Ganesh Venkat,

    What you said is absolutely true. In the real world we have so many examples of intuitive decision makers being successful. Two of the best examples that come to my mind are IBM's Deep Thought Computer losing to Kasparov and even its improved version Deep Blue losing the opening game to the former World Champion. One more example could be the continued success of Mr. Warren Buffet in the stock market. 

    But, what we need to look into is the question that are these truly intuitive decisions? 

    Most of the decisions that we humans take are not intuitive, though we might like to think that they are.

    As soon as we are confronted by a situation our mind starts working on it. The processing done by the mind is so fast that we often mistake it for intuition. Even in cases where it is intuition, it is based on some past experiences/memories that we have. 

    In 1921, Carl Jung defined intuition as  "perception via the unconscious: using sense-perception only as a starting point, to bring forth ideas, images, possibilities, ways out of a blocked situation, by a process that is mostly unconscious".

     

    The unconscious as stated above is slightly dependent on our past experiences and prevailing thoughts. The reliability of one's intuition depends greatly on past knowledge and occurrences in a specific area. For example, someone who has had more experiences with children will tend to have a better instinct or intuition about what they should do in certain situations with them. This is not to say that one with a great amount of experience is always going to have an accurate intuition (because some can be biased); however, the chances of it being more reliable are definitely amplified. 

     

    Data driven approach to decision making is more of a left brain type of activity whereas intuitive decision making is a right brain type of activity. The best systems are those that make use of both of them. As in the Case of Deep Blue, Kasparov alleged that some experts controlled some of the moves the computer made. Nowadays, integrating the ingrained knowledge of experts of a particular field while making a DSS for the field is not uncommon. Also nowadays, we can utilize Recognition Primed Decision(RPD) and Intuitionistic Logic while designing D3M systems.  

     

    To sum up my argument, I can say that making intuitive decisions may lead you to runaway success in a situation but using D3M systems would lead you to continual and consistent success. A DSS that can integrate both these approaches is obviously the best choice.

     

    Nice answer Nachiketa :)

    Link to comment
    Share on other sites

    Nice work Nachiketa!


     


    Just one question haunting my mind is that how will D3M help me in unpredictable, unordered and complex context where right decision cannot be made through persistent investigation or understanding of available data/information?


    Link to comment
    Share on other sites

    Great work Nachiketa .

    Can you elaborate more on how D3M allows in discovering solutions you never knew existed ?

    Dear Sakshi, thanks for the praise. now moving on to your question.

     

    Humans make decisions on the basis of their perceptions of reality and not reality itself. These perceptions are based on their sensory organs, experiences, emotions and motivation amongst other factors. These perceptions also suffer from camouflage, recency effect, contrast effect and grouping along with a host of arbitrary errors. These perceptions differ from the reality in varying degrees and in worst case scenario it can be unrecognizably different from the reality. Thus the question that really comes to all our minds is whether it is safe to take decisions on such false premises???

     

    The answer is no. Human Decision making is blinded by our perception which acts a screen between the outside world. This perception is riddled with biases and errors.

     

    There have been cases where the best possible solution to a given problem has been ignored completely by the human mind or camouflaged or grouped along with irrational solutions. Also human decision making stops when we arrive at an optimal solution. 

    What we fail to recognize is that there maybe more than one optimal solutions to a problem.

     

    On the other hand DSS systems start with an exhaustive list of possible solutions and evaluate each of them on their merits and demerits. Thus they can arrive at multiple optimal solutions and never miss the best solution.

    Link to comment
    Share on other sites



    Guest
    This is now closed for further comments

  • Member Statistics

    55,093
    Total Members
    888
    Most Online
    Mahtab Khan
    Newest Member
    Mahtab Khan
    Joined
  • Who's Online   0 Members, 0 Anonymous, 37 Guests (See full list)

    There are no registered users currently online

×
×
  • Create New...