Q 505. What is the difference in Bayesian and Frequentist approach for hypothesis testing? Also explain Bayesian way of thinking and the Frequentist's way of thinking with simple examples. 

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Frequentist statistics, which could also be described as experimental or inductive, rely on the law of observations.

In a frequentist model, probability is the limit of the relative frequency of an event after numerous trials. In other words, this system calculates the probability that the trial would have the same issues if you were to replicate the same conditions again. This model only uses data from the current trial when assessing issues.

 When applying frequentist statistics or using a tool that uses a frequentist model, you'll presumably hear the term p- value. A p- value is the advised probability of carrying an effect at least as extreme as the one in your sample data, assuming the verity of the null thesis. For illustration, a small p- value means that there's a small chance that your results could be fully arbitrary. A large p- value means that your results have a high probability of being arbitrary and not due to anything you did in the trial. In short, flash reverse that the lower the p- value, the more statistically significant your results.

 Unfortunately, people constantly misinterpret what p- value represents. P- value is basically the probability of a false positive rested on the data in the trial. It doesn't tell you the probability of a specific event actually passing and it doesn't tell you the probability that a variant is better than the control. P- values are probability statements about the data sample not about the thesis itself. So if you ran an A/ B test where the conversion rate of the variant was 10 advanced than the conversion rate of the control, and this trial had a p- value of0.01 it would mean that the observed result is statistically significant.

 

Bayesian statistics, which is theoretical/deductive, enables us to combine the information provided by data with a priori knowledge from previous studies or expert opinions

Bayesian statistics are named after champion Thomas Bayes who believed that “probability is orderly opinion, and that conclusion from data is nothing other than the modification of similar opinion in the light of applicable new information.  

 With Bayesian statistics, probability indicates a degree of belief in an event. This system is different from the frequentist methodology in several ways. One of the big differences is that probability expresses the chance of an event end. Although the computation can be extremely complex, this system seems to be a simpler and further intuitive approach for A/ B testing. fairly simply, a Bayesian methodology will tell you the probability that a variant is better than an original or vice versa.

 

 The Bayesian generality of probability is also further conditional. It uses former and posterior knowledge as well as current trial data to prognosticate issues. Since life doesn’t be in a vacuum, we constantly must make hypotheticals when running trials. But the Bayesian approach attempts to regard former knowledge and data that could impact the end results.

 

 

Bayesian approach for hypothesis testing	Frequentist approach for hypothesis testing
Bayesian approach   How unusual is the observed result under the given hypothesis is Bayesian approach.   p-value for a Bayesian: The p-value is the probability is an expression of a degree of belief of an event, based on prior knowledge (previous experiments) or personal belief. P-value is the probability of the hypothesis given the data, P(Hypothesis|Data).	Frequentist approach   What is the probability that the hypothesis is true given the observed result is Frequentist approach.   p-value for a Frequentist: The p-value is the probability of the observed data, or more extreme data under the assumption that the null hypothesis is true, P(Data|Hypothesis).
Bayesian way of thinking   ·         Bayesian statistics relates with subjective belief. ·         Bayesian statistics uses the idea of updating beliefs with new information when testing a hypothesis. ·         Prior belief x Bay’s factor = Posterior belief (= updated, new belief). ·         Bayesian talks about the observed data been fixed and the varying the model around. ·         Given the observed data, there is 95% probability that the true value of the parameter lies within the credible region.	Frequentist way of thinking   ·         Frequentist statistics is about the absolute truth and care about the true answer. ·         Frequentist statistics is not involving the opinion. ·         Frequentist talks about models been fixed and the data varying around them. ·         If the experiment is repeated multiple times, in 95% of the cases the computed confidence interval will contain the true value of the parameter.

There are two common statistical approaches that are being followed when it comes to statistical testing i.e. The Frequentist Approach, which is based on the observation of data at a given moment or instance & The Bayesian Approach, which is basically a forecasting approach & it involves analyzing prior information.

 

The frequentist approach is also described as experimental or inductive as it relies on observations while the bayesian approach is theoretical or deductive as it enables to combine the information provided by data with a priori knowledge from previous studies or expert opinions.

 

Let us take a very simple example to understand both the concepts:-

 

Let us toss a coin 10 times, now when it comes to frequentist approach, the probability of getting either a head or a tail is 0.5, now let’s say we get heads on 7 out of 10 tosses, then the probability of getting the heads will be 7/10 i.e. 0.7.

 

Now let’s say we have a prior information through previous experiments of expert experience that heads will come 6 out of 10 times thus we have a prior probability of 0.6, now we will compare the outcome of the experiments with this prior probability.   

 

Thus we can say that the objective of the frequentist approach is to explore the data collected in order to identify a significant effect that could only be explained through by the hypothesis of the experiment & for the bayesian approach the focus is on comparing two hypothesis by comparing the data collected at the time of the experiment with the prior information available therefore assessing the chances that one was true comparison to other.

 

As an organization performing experiments & relying on statistical analysis for analyzing the results of these experiments, it is thus important to understand the difference between the above two approaches on the basis of different parameters which are as shown below:-

 

In terms of analyzing the test data :-

 

Frequentist approach requires the experiment to be completed first by collecting sufficient samples before analyzing the data, this limits the test to be an offline experiment.

 

Bayesian approach analysis can be performed during the experiment while collecting the data. Also it is an online experiment as the analysis results get updated when new batch of data gets ingested.

 

Sample Size :-

 

Frequentist approach requires calculating the sample size prior to conducting the test, also the number of samples among test groups needs to be balanced.

 

Bayesian approach does not require a pre-defined sample size & also there is no need to have same number of samples amongst the test groups thus allowing an imbalanced sample size.

 

Test results explanation :-

 

For the frequentist approach, conclusions can be made like “We reject/ fail to reject the hypothesis that group A is better then group B. This conclusion is based on the observation of the historical data collected during the test. This approach uses p-value in order to quantify the confidence of the business conclusions.

 

For the bayesian approach, we introduce the element of probability while making an interpretation of results such as “ There is a 98% probability that group A is better than group B”. Thus this probabilistic result quantifies the confidence of the business conclusions.

 

Leveraging Test Results :-

 

Frequentist approach gives summary statistics of the samples collected during the experiment period, thus cannot be used for making any conclusions about the future unseen data.

 

Bayesian approach leverages the parameters of the distribution from the data & gives the posterior predictive distribution for unobserved, future values on the observed data.

 

Duration of the Test :-

 

In the frequentist approach, the duration of the experiment can be estimated basis the designed sample size as it is easy to estimate how long an experiment will be conducted.

 

In the bayesian approach, the duration of the experiment cannot be estimated as more samples coming every day helps to get more confidence conclusions, but cannot estimate how long a specific experiment would take.

 

Granularity of input data :-

 

In the frequentist approach, the level of granularity of the input data is at the very base level for eg: data collected basis each user / ID & also it depends on the duration for which the test is conducted.

 

In the bayesian approach, the level of granularity of the data depends on the frequency of the updating the test results, for eg : in case you are testing the Click through rate & the results are updated every 24 hours, one needs to calculate the number of total seen events & number of click events every day in order to arrive at the daily click through rate.

 

Performing Multiple Comparison :-

 

Frequentist approach leverages bonferonni adjustment in case when multiple variants are required to be tested at the same time.

 

Bayesian approach uses hierarchical bayesian methods for cases involving multiple variants.

 

Testing Approach :-

 

The frequentist approach recommends different tests based on the distribution(s) that a variable of variable(s) follows.

 

The bayesian approach leverages conjugate families for variables following different distributions for eg : Click through rate would leverage the beta distribution conjugate wherein prior parameters need to be set for the beta distribution, collected data is updated basis the baye’s rules in order to get the posterior of the parameters, then samples are taken from the posterior distribution & inferences are made on the test results accordingly. 

Frequentist Methodology

 

In frequentist model, probability is the limit of the relative frequency of an event after many trials. This method calculates the probability that is from the current experiment when evaluating outcomes which would have the same outcomes and would replicate the same condition again.

 

When applying frequentist statistics that uses a frequentist model, we will come across the p- value. Which is the calculated probability of obtaining an effect at least as extreme as the one in your sample data when we assume the truth is the null hypothesis. For example, a small p-value means that there is a small chance for the results to be completely randomly. A large p-value means that the results have a high probability of being random. In short, the smaller the p-value is more statistically significant.

 

Often p-value is misinterpreted differently. P-value is the probability of false positive based on the data in the given experiment. It does not tell the probability of a specific event actually happening and it does not the probability that a variant is better than the control. P-values is the probability statement about the data sample and not about the hypothesis itself. So if an A/B test where the conversion rate of the variant is 10% higher than the conversion rate of the control, then in this experiment had a p-value of 0.01 would mean the observed result is statistically significant in the given experiment

 

Bayesian Methodology

 

Bayesian statistics is named after philosopher Thomas Bayes where the probability simply expresses a degree of belief in an event.

 

Bayesian method is different from the frequentist methodology in a number of ways. One of the biggest differences is the probability actually expresses the chance of an event happening. Although the calculation can be extremely complex, Bayesian method seems to be a simpler and more intuitive approach. In simple words, a Bayesian methodology will tell the probability that a variant is better than an original and vice versa

The Bayesian concept of probability is more conditional which uses prior and posterior knowledge and current experiment data to predict outcomes. Since we often have to make assumptions when running experiments, the Bayesian approach attempts to account for previous learnings from the experiments already done and data that could influence the end results of all the experiments

 

At this point, many experimentation platforms are using proprietary, hybrid models that would combine a traditional statistical model which can either be Bayesian or frequentist model with some other technology such as machine learning. It is certain to have at least a basic understanding of the methodologies., when it comes down to it, what actually matters is how well we understand the results we have got in the experimentation platform of the choice that we have taken. This understanding leads to a more data-driven approach for assessing risk and what the organization is willing to accept and the predicted improvement to business outcomes could be.

 

When we are debating the pros and cons of Bayesian and Frequentist statistical methodologies. We may have experimentation stakeholders from multiple departments simply wanting a decision and often there would be no regard for the statistical methodology used.

 

Examples:

 

Bayesian way of thinking example: Bayesian way is used in various occasions in our daily life which includes a medical testing for a rare disease. With this we can estimate the probability of actually having the condition given the test coming out positive

 

Frequentist's way of thinking: The frequentist way is probability if there is the long-run frequency of repeatable experiments. For example, saying that the probability of a coin landing heads 0.5 means that if we were have to flip the coin enough times the we would see heads 50% of the time.

 

Bayesian Vs Frequentist main difference of both thought process is reasoning of probability

 

Bayesian	Frequentist
Bayesian thinking see the probability based on their certainty and uncertainty of trail. It’s based on belief of an event based on prior information	Frequentist thinking see the probability based on frequencies of the repeated trail
P Value based on the probability of the hypothesis. Which is inverse of Frequentist	P value is probability of more extreme data with the assumption that null hypothesis is true
Hypothesis is based on no variation in the data but the variation in the Model/Parameter	Hypothesis is based on Variation of the data and their derived quality based on the repeated measurement with fixed Model/parameter
Assuming 95% of true value of a model lies with in credibility region	95% of cases confident interval will have true value of the model
Varying True value and Fixed credible region	Varying confidence interval and fixed true value
Process to understand possibility that Hypothesis it true based on observed result	Process to understand how extreme the observed result under the Hypothesis
Example: Playing Card with friend which has 52 card and Friend drawn a card and seeing card in hand asking possibility of card in his hand is Diamond card, On Bayesian way of thinking possibility of getting Diamond is out 13 out of 52 cards which is 25% its Diamond	Example: Since the card has been drawn and the result is known its either Diamond or others, so this can be either 0 or 1

 

Bayesian way of thinking is preferred in term of clinical trails based which take more result change based on prior data and new state. Also, current AI model use more Bayesian theorem to learn based on prior data and change the result. Which allow to correct Bais and Noise level based on prior data to correct make it more accurate. But in uncertain case when there is no prior data (non-informative prior) consider all data are equally likely which create a Bias hypothesis in given data will not be always correct.

Both Bayesian and Frequentist approach of hypothesis testing are important and relevant method to facilitate determination of an event. Dependent on the approach for decision making, a choice can be made between the two well known ways of hypothesis

Bayesian Hypothesis testing is a method of assigning probability to unknown parameter, compared with available historical trend and gathered knowledge and later extended with most recent information about the unknown parameter in consideration. It is like simulating with the in-scope data multiple times which will ideally provide more details on the alternatives and firm up the probability factor on occurrence, with time. In simple terms, it is somewhat how we normally think and affirm an opinion. Start with a prior belief and keep improving this in light with new evidence. We regularly update our knowledge in light of the known facts – focus on what is known via knowledge and existing data, identify an issue with unknown fact that is in scope to decision making and carry out multiple / repeated actions that allows us to evolve and firm up a decision.

Scientifically, It has 3 stages, specifying a prior probability distribution on unknown parameter, observed data summarized using Likelihood function hypothesis and posterior distribution also known as updated knowledge. 

Frequentist Approach is more of making predictions basis Data from the current experiment and is driven by what is known at a given point of time. The Hypothesis test (Null and alternative) based on applying statistics conclusion to identified data and when compared to P Value, will either recommend acceptance or denial of the specific outcome. In simple terms, Frequentist approach confirms the probability of having the same outcome if the condition is repeated again and again. This model only uses data from the current experiment when evaluating outcomes.

Statistically it has 4 stages, Defining the assumption (Model), Null and Alternative Hypothesis, Test data basis available tools and use outcome to from mathematical conclusion.

Despite of having same intent on outcome, the basics around theory and independent characteristics differentiates them from each other, some of them are noted below:

S. No	Bayesian Hypothesis	S. No	Frequentist Approach
1.	Derives Probability by inferencing past knowledge combined and upgraded with outcome from current experiment.	1.	Makes predictions purely basis data from current experiment, it is long run frequency of repeated experiments.
2.	Parameters are random variables and data is fixed	2.	Parameters are fixed and data is replicated
3.	Probability is assigned to both hypothesis and past data	3.	No probability is assigned to the hypothesis
4.	Performs well with small data set, one can start with as small as one data set	4.	Gives confidence in large data sets, since the later is randomized
5.	Driven by ability to form prior model and relate to the difference in the answers	5.	Easy to calculate and formulate hypothesis – statistical analysis based
6.	Inferences here will lead to better communication of uncertainty	6.	Based on fixed data and hence lacks the flexibility to adapt uncertainty
7.	Easy to relate to the outcome, since the advantage is of having a prior parameter and knowledge	7.	Mostly difficult to interpretate P Value and hence somehow keeps the confusion on absolute interpretation
8.	Comparison is with P value with prior probability and prior is subjective	8.	Hypothesis testing uses comparison with P Value that has never been observed

 

Let’s also look at unique preposition around Bayesian way of thinking compared to a Frequentist way with an example, If someone asks us what is the probability of getting King of hearts in the given image, when picked from blind side of the cards?

Most often than not, the response will be ½ or 50%. Absolutely right!!

But if the cards are jumbled again and if I ask you to choose one of the cards and raise the same question.. what is the possibility of getting king of heart? Will the answer be different now?

 

Some of us will say, now since I have picked one card, either the King of heart has been picked 100% or not picked which is 0%. Obviously, now there isn’t a choice anymore between two options to be considered as Heart or Club? That’s how Frequentist will think.

Some of you will record the previous hold knowledge on probability and say it is still 50% and unless proved otherwise with series of trials. We win some and we lose some. You just choose Bayesian way of thinking.

Similar example from real life may be how doctor examines the health condition of a patient. In a typical Bayesian way of thinking, doctor will take weightage around prior diagnostics, do fresh investigation and later recommend meditation basis fresh assumptions and result. But a Frequentist way would lead basis current diagnostic results and prescribe related mediation.

 

These examples will direct us to think of below characteristics …

·         Bayesian way will tell you what you want to know, which one is better

·         Frequentist will keep It difficult for you to interpretate uncertainty since comparison is with P Value

·         Frequentist do not explicitly call out assumptions

·         Bayesian method is immune to data peeks, whether you update prior parameter with every experiment or at any given point of experiment – makes no difference

 

 

 

Frequentist approach

It’s the model of statistics taught in most core-requirement and its approach most often used by A/B testing.

Making predictions on the underlying truths of the experiment using data from the current experiment in the Frequentist method

Example: Is this variation different from the control in a t-test or the probability of a coin landing heads being 0.2 means that if we were to flip the coin enough times, we would see heads 20% of the time

 

Bayesian Approach

Bayesian approach is a more bottom-up approach to data analysis. In this approach past knowledge of similar experiments is encoded into a statistical device known as a prior, and this prior is combined with current experiment data to make a conclusion on the test.

Example: if X-company knows that by 5 PM there are 50 reservations, then they can predict that there will be around 250 covers for the night.

 

Major Difference Between the Frequentist and Bayesian Approach are

Frequentist statistics never uses or calculates the probability of the hypothesis, while Bayesian uses probabilities of data and probabilities of both the hypothesis.

In the frequentist approach, they are fixed variables. Bayesian approach, trying to estimate are treated as random variables.

Bayesian view, a probability is assigned to a hypothesis whereas in the frequentist view, without being assigned a probability a hypothesis is tested.

On 9/16/2022 at 4:34 PM, Vishwadeep Khatri said:

Q 505. What is the difference in Bayesian and Frequentist approach for hypothesis testing? Also explain Bayesian way of thinking and the Frequentist's way of thinking with simple examples. 

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Bayesian Test interprets probability as a measure of the belief or confidence that an individual might hold regarding the likelihood of an event occurring and the prior beliefs about an event will likely change when new information is revealed. It not only considers the likelihood of the occurrence but also consider the beliefs and experiences that an individual may hold which may be fair. This helps in arriving at a hypothesis basis historic trend and real life experiences and beliefs which is more practical.

 

The Frequentist inference interprets probability as the frequency of repeatable experiments and the gathering of information. This may be used when we have existing data however the inferences may be close to the sample population and would be dependent on the current data sample that the person may have selected. 

Frequentist inference relies on P-value and it is assumed that null hypothesis is true however Bayseian approach is based on beliefs based on new information derived at the time of conducting the hypothesis.

 

Lets take the example of tossing a coin

As per frequentist approach the likelihood of heads on the coin will depend on the sample and heads being received as per sample data v/s actual hypothesis.

Parameters remain fixed and data is random since its based on frequency of repeated events

As per Bayesian approach the likelihood of heads repeating 90 out of 100 times could also mean that there is something to do with the flipping of the coin or the coin itself. Hence the parameters are random but the data is fixed and probability depends on degree of certainty about values.

 

Whilst frequentist method may be used for hypothesis testing, Bayesian method can be alternative method since the inferences are not based only on sample data but it also takes into account observations during hypothesis and considers both the null and the alternate.

This was a tough one to answer. Rahul Arora has given the best answer to this question.

 

Read through the other answers to get some more examples that highlight the differences between Bayesian and Frequentist approach.