Jump to content
  • My Experiments With Data Driven Decision Making (D3M)


    decision-making-skills.jpg

    As a Computer Illiterate growing up in the new millennium, I had tremendous difficulty making sense of how to do most of my stuff. Most of my decisions were snap, on-the-moment and intuitive ones. By the time I was in my teens, I had learnt my way around the big computer problem. My experiences and the Great Indian “Jugaad†mind-set had provided me with a sufficient enough method wherein I only worked on the computer when absolutely necessary. If there was any other way to do the task, any escape route, I gladly followed it.

    By now you must have realised that I was and am sceptic and data-averse. Data to me was always those unreadable files that eat a hell lot of my storage space. I was gaining proficiency in getting around this data problem and along came Big Data. Everyone from Google to Barrack Obama was using it.[1] As a student at one of the most prestigious B-Schools in the knowledge market, not only was I supposed to know what it was, but was expected to be able to tune in and utilise it to make a difference. Life has strange ways of getting back at us, mere mortals, and here was my customised gut wrenching sucker punch. The fighter in me knew I had to do this, but I did not find a way I could.

     

    A 3 month long research in the field led me to Data Driven Decision Making or D3M. Simply put, it is the pleasure of sipping a coffee in your office chair while you watch your computer work up some algorithms and provide you invaluable decision making tips to face some of your most frequently encountered problems.

     

    Immediately after, I experimented with the different sorts of decisions that can be made with D3M. The results were a true eye-opener. If we think of decision making as a broad spectrum from operational decisions at one end to strategic decisions at the other. Operational decisions can be characterized as highly structured, routine, short-term oriented and increasingly embodied in sophisticated software applications. On the other hand strategic decisions are taken by the top management and serve to set the long-term directions, policies and procedures of an organization. They tend to be complex and unstructured because of the uncertainty and risks that generally accompany longer term decisions. In between these two extremes, we can have varied decisions including non-routine ones in response to new or unforeseen circumstances beyond the scope of operational processes, and tactical decisions dealing with the necessary adjustments required to implement longer term strategies. [2]

    Evaluation2.jpg

     

    Half a decade ago, D3M could have only helped you with the more structured forms of decision making but no more. With the advent of Big Data, machines know much more about humans and human behaviour than humans themselves. Sample this, personal analytics can actually allowed me to analyse my Whatsapp chat history and find why most of the girls I chatted with refused me a date. The word cloud usage showed me that my texting skills were uninspiring to say the least with the most common words being “ok†and “yaâ€. I realised I needed to be more creative and engage better. What did not help was that most girls chatted in the window of 8-12pm whereas I had the habit of taking a short nap at the time. Also it helped me identify my most productive work hours and helped me plan out my work better.

     

    Thus far, all seemed well with D3M but on further research most early moving corporates into the field were still undecided on its benefits. A closer look exposed a distinctly similar pattern in them. Most early moving managers thought that D3M will help them save money or time or both. Research shows that they are misguided to say the least. D3M does neither, atleast not in the short term. What D3M does allow though is discovering solutions you never knew existed. What it will do though is finding needles in the haystack consistently. Another interesting insight was that D3M depends a lot on the data collection. Great collection leads to great results. All we are required to do is ensure data cleanliness, variety and velocity.

    In his book, Data Driven: Profiting from Your Most Important Business Asset, Prof. Thomas Redman summarizes the whole decision making via data as “Good decision makers follow at least three Bayesian principles. First, they bring as much of their prior experience as possible to bear in formulating their initial decision spaces and determining the sorts of data they will consider in making the decision. Second, for big, important decisions, they adopt decision criteria that minimize the maximum risk. Third, they constantly evaluate new data to determine how well a decision is working out, and they do not hesitate to modify the decision as needed.â€

     

    After months of thorough investigation and experimentation, I have arrived at this conclusion, “To stay a step ahead in this ever competitive world, using Data Driven Decision making is a must. So let D3M take care of all the external data that you need to work with and let your mind focus on understanding “the data from insideâ€. That will surely lead you to not just success but contentment.

     

    References

    1. http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/

    2. http://blogs.wsj.com/cio/2013/09/27/data-driven-decision-making-promises-and-limits/

     

    Note - Visitors shall not be able to comment on this article until they are logged in.


    User Feedback

    Recommended Comments



     

    Thanks for the praise Aishwarya. 

     

    I would like to answer your questions one by one.

     

    Firstly, you want to know what is the minimum amount of data required to implement Data Driven Decision Making on a Decision support System. Like in the case of real life, taking a decision is a result of weighing in the pros and cons of every possible state. Similarly in Data Driven Decision Making we try to look into as many aspects as possible and weigh in their pros and cons. Weighing in an aspect/state usually entails quantifying the possible states in positive and negative values. The more data we have, the better and more informed decisions we can take. As I had stated earlier we usually need data from a CRM and the regular system the organization works with along with some Industry Specific or Market Data. Historical records can also be scanned and utilised. There is no specific minimum or maximum limit to the data. If the Decision to be made is simple then the amount of data required is less and so is the accuracy of the decision. If we need accurate results we need to weigh in as many data points as possible.

     

    To answer your second question. Nowadays with the advent of high speed powerful micro processors Data Driven Decision Making does not depend on specialized hardware anymore. Though what is required is getting a specific Decision Support software built for your industry/company by a vendor. Also whats required is specific hardware for the Data Warehouse (the size of which is proportional to the data you'd be using) . Such an investment in a Data warehouse though has additional benfits as it can be used in Data Mining and OLAP systems as well. Out of the three main components of a DSS setting up a knowledge base is more hardware dependent than the other two namely user interface and the model(decision context and user criteria).

     

    I'd suggest that you go through the following links for better clarity on the topic as well as specific hardware requirements calculations.

     

     

    Thanks for the response buddy :)

    Link to comment
    Share on other sites

    Nice article Nachiketa.


    BTW i have just one question for you.


    'How you can correlate cognitive decision processes with D3M?'


    or i can reframe my question to


    'Can we replace cognitive decision making with D3M's huge data analyzing methodologies in the near future? and how feasible the solutions will be compared to the organic decision making processes?'


    Link to comment
    Share on other sites

    Great article Nachiketa. Well potrayed with personal examples which is facilitating readers to relate and grasp easily.


    Your last paragraph says "After months of thorough investigation and experimentation, I have arrived at this conclusion" , on this i would like to put my views forward , you should continue with your investigation and experimentation with same enthusiasm as there are always unexplored areas which when found makes a Research .


    All the best.


    Link to comment
    Share on other sites

    Brilliant work Nachiketa!
    I just have one query, is it possible to make use of this model to take the big decisions, like where too many factors (extraneous and non extraneous) are at play and you can't quite decide which one effects and by how much and for how long. Or maybe, say a decision which on one hand requires exhausting more of A than B resources and gaining C points, on the other exhausting more of B than A resources and gaining D points...and so on and so forth, everything that you might seem to want equally because you can't decide what might be good to lose or gain in the longer run, something that can't easily be forecasted.

    Link to comment
    Share on other sites

    Great article Nachiketa. Well potrayed with personal examples which is facilitating readers to relate and grasp easily.

    Your last paragraph says "After months of thorough investigation and experimentation, I have arrived at this conclusion" , on this i would like to put my views forward , you should continue with your investigation and experimentation with same enthusiasm as there are always unexplored areas which when found makes a Research .

    All the best.

    Thanks Chahat, 

    I hope this article was of some help to you and thanks for your motivational words.

    Link to comment
    Share on other sites

    Nice study Nachiketa. It is great food for thought for people who believe in data driven decision making. D3M seems to be a great technique but if you could please elaborate it with an example would make things much clear. Also i would like to know more about about the calculations for the consistency and reliability of the data provided by the sources under this technique.


    Link to comment
    Share on other sites

    Its a good write up and yes does give quite an insight on D3M. You have put it across in a way most of us can easily accept and understand.


    In my opinion D3M has existed as long as man, earlier as well any decision depended on the something, i.e. data.


    Yes with advent of more data the complexity of data analysis has increased and here D3M is a good approach.


    Link to comment
    Share on other sites

    Nice flow of thoughts in your article Nachiketa. Your article reminds me of the scene from Swades (by Ashutosh Gowariker and Shah Rukh Khan) when a local farmer predicts about the rains in their village seeing the clouds, leaving SRK stunned (as he plays a data-crunching scientist from NASA).


     


    You have also answered to the previous queries very well citing examples of the Machine being beaten by Grandmasters. So what I conclude from this discussion is that Data analysis is very important but at times things also have to be thought beyond the numbers, just like a long term judgmental and expert opinion after you have calculated the NPVs and IRRs of projects.


     


    Could you please tell me if Data- and Gut- conclusions are always two sides of the same coin or whether one holds a slight edge over the other in specific fields?


    Link to comment
    Share on other sites

    I knew that every decision is made by my mind, but now I wonder whether it is backed by intuition or data. I feel that each of my decision is backed by data which comes as a result of intuitive thoughts.


    I look forward for an even strong merger of data of my mind with my intuitive powers.


    Link to comment
    Share on other sites

    Its a good write up and yes does give quite an insight on D3M. You have put it across in a way most of us can easily accept and understand.

    In my opinion D3M has existed as long as man, earlier as well any decision depended on the something, i.e. data.

    Yes with advent of more data the complexity of data analysis has increased and here D3M is a good approach.

    Thanks Swati for the compliments. I totally agree with you on the fact that Decision making has been based on reasoning and logic since time immemorial. Intuitive decision making has always been a smaller set. And in the age of Big Data, we realize that we have the golden opportunity to tune into this data and utilize it to make better and more sound decisions...

    Link to comment
    Share on other sites

    A very interesting read Nachiketa.

    However the basic difference between a  Data Driven approach and Intuition is the ability to take a wild guess.Logic can  only go so far amidst real world noise. We have to take a wild guess occasionally, which is random.

     A significant amount of information we incorporate and process, we just run into it. I wonder how cliched and our restricted our information will be if we get exactly what we are looking for. A solution finds a problem , not the other way around. There are many solutions for questions we might never pursue. Isn't that compromising on innovation and intuition.

    Is Data Driven Decision making really required ?

    Thanks Himanshu Lohani..

    Your question is multi-fold and I would like to answer it byte-by-byte. 

    I agree with you to the extent that yes in times when we don't have sufficient data at hand we have to take wild guesses. But taking such wild guesses repeatedly bequeath logic. We have so much data nowadays. The world today is home to 5 zetta bytes of data. Facebook alone collects upto 15 TB of data each day. Needless to say, we now need to come out of our hollow cocoons of intuitive decision making and face the brave new world of data. 

     

    Today Data is everywhere. We have enough data to find solutions to most of our problems however structured or unstructured they may be. One of the key features of a DSS is the ability to solve arbitrary and unstructured problems and that will be the key to success in the future.

     

    This feature will also help us seek newer problems and apply all our pre-obtained knowledge on them to solve the, This is the power of Data Driven Decision Making.

    Link to comment
    Share on other sites

    Well written article Nachiketa

    If you could enlighten more on the data collection part and of how much impact has it been in the modern world at a successful rate...

     

    Nice Nachi!!!! ...very well written, wat are the techniques ideas are their for data collection, at least to the level to make decisions.

     

    Thanks for the appreciation Stacy Angom and Karan Kamra. Since both of you have asked a similar question I'd like to answer both of them together.

     

    I concur with you on the importance of data collection and it's truly a wonderful question. 

    As I have stated repeatedly in the comments Data in the Knowledge Base is basically from three sources.

    1. Legacy System/Core Transaction Processing System - The data from these sources is saved in the form of a core dataset and incremental snapshots of the data at fixed intervals of time.

    2. Customer Relationship Management system - The data from the CRM system is cleansed for inconsistencies and added to the data from Stage 1 wherever needed.

    3. Market, Industry and Environment data - The Market, Industry and environment data is obtained by evaluating the environment variables and internal and external reports and market analysis.

     

    In addition to the aforementioned types historical data, expert opinion and industry specific models may also be a part of the knowledge base.

    Link to comment
    Share on other sites

    Commendable work Nachiketa. Most of my queries have already been answered in your previous posts and I must say, this thread can teach a layperson a lot about D3M. 


    D3M seems to be quite an interesting tool. I was wondering if it can be used for various HR functions such as recruitment and talent management. A lot of times, there lies a lot of subjectivity in the kind of data the HR department of an organization deals with. Can you suggest some ways in which HR managers can employ 'data-driven-decision making'?


    Link to comment
    Share on other sites

    A good write-up.. but isn't it too early to say that D3M is 'THE' thing for the future. I mean, we still don't know the reliability of the information. The information might be as good as predicting the next movement of stock market. Moreover, it will be available to almost all the companies. So, how do we then use it to gain competitive advantage?


    Link to comment
    Share on other sites

    Loved the article Nachiketa. The best part is the gradual transformation from great Indian juggad-gut decision making process to only D3M and to last the holistic D3M. You have given a personal touch to whole article by giving your own experience, which really explains the real time application of these decision making process.

    I agree with you that just D3M is not going to help in long run but D3M with right perspective can yield long term profit. Only with gut decision, right perspective and D3M gave us shiny Apple gizmos.

    As you say "let's Live with sense not just data."

    Link to comment
    Share on other sites



    Guest
    This is now closed for further comments

×
×
  • Create New...