Q 510.  Compare the three different types of experimental approaches to process change - Trial and Error, One Factor at a Time, Multiple Factors at a Time. Are there any situations where Trial and Error will be preferred over the other 2 more structured approaches?

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Below are the key differentiating aspects amongst the three experimental approaches:-

Differentiating Aspect	Trial & Error	One Factor at a Time (OFAAT)	Multiple Factors at a time (Factorial Design)
Concept	Subject matter experts hypothesize the critical independent factors that will create a desired outcome or response. The experiments are done with these factors. If the experiments are not successful, another set of factors is selected & the experiments are repeated on the another set of factors until the hypothesized factors are not validated to impact the outcome	The experiments are designed in such a way that one factor is varied throughout its normal range while other factors are kept constant. The factor is set at the optimal setting & the next factor is selected & varied throughout its normal range in order to determine the optimal setting. The process continues until all the factors are tested by varying one factor while keeping all other factors constant	This experimental approach consists of two or more factors each with discrete possible values known as levels, the experimentation takes into account all possible combination of these levels of all the factors involved, the experiments are conducted by taking different levels of the factors simultaneously
Suitability	Appropriate where there is a specific goal or response that is desired from the dependent factors & there are subject matter experts who can confidently select the appropriate independent factors for conducting the experiment	Best suited for basic research projects for testing new technologies or inventions. This allows the researchers to define the relationships between the factors & the system performance	Often used to create a statistically valid equation for the system performance based upon the input values of the selected factors being studied. It determines the optimal level of performance basis the multiple level factors that are used
Advantages	This approach is the fastest & lowest cost experimental design approach. By leveraging the expertise of the subject matter experts & focusing the experiments on a specific goal, the number of tests can be held to a minimum	The OFAAT methodology is very efficient when it comes to characterizing how the selected factors impact the system performance i.e. product, service or process. By varying each factor at a time through its normal range, one can efficiently study the magnitude of impact of each factor on the output & this will aid in better decision making in terms of factor selection for further optimization	This is the most comprehensive approach of experimental design as we can easily perform a comprehensive analysis of the design space for the system being analyzed. The final result of all the experimentation is an equation that can be leveraged to predict performance & this equation camn be used to identify the factor settings that will yield optimal performance, thus making this approach a go-to tool for prescriptive analytics
Limitations	This approach is highly dependent upon the knowledge & experience of the subject matter experts. Also this is a difficult methodology to estimate, this is because if the estimates of the factors during experiment turns out to be not true, then additional unplanned tests are needed & this can create massive delays & overruns	This approach works best if the impact of the factor is linear, however if the effect is non-linear or curvi-linear then the order of factors can impact the final setting & performance. Also this approach only studies the effect of one factor on the outcome however does not take into consideration the intercation effects or two or more factors simultaneously on the overall outcome	All the tests basis the possible combination of levels of all the factors must be conducted in order for the statistics to be valid, thus making this approach both time & resource intensive

 

 

In cases where exploration of the potential factors that can impact the outcome needs to be done, trial & error approach will be more beneficial compared to OFAAT or Multiple Factors at a time. Here we can conduct preliminary experiments to check whether the selected factors are having a significant impact on the outcome or not. If not we can further identify another set of factors in order to validate their impact on the outcome.

 

From a cost, time & resource perspective also, we want to make sure that the factors that are being considered for study have at-least gone through a preliminary validation & now we are sure that the factors being studied through OFAAT or Multiple factors at a time approach are being identified correctly. In such scenarios, trial &b error approach will play a very important role.

 

One Factor At a time (OFAT)	Multiple Factor	Trail & Error Method
This is called Controlled Experiment. Where we keep one controlled factor and other experimental Factor varied to get best settings	In Multifactor all are considered as experimental factor varied to identify the best settings	Try multiple iteration based by making changes to settings based on previous failure to reach best settings
Number of runs to find setting with controlled factor is less	Number of runs to is high due varying factors	No fixed number. Depend on learning from previous error
Experimental error due another factor is less. No interaction factor. Possibility of missing optimal setting	Possible of experimental error based on other factors. There is interaction factor which provide best possible setting for optimum result	Prolonged experiment if more failure
Since controlled experiment data setting is very vital and need to be handled with care	Multi factor allows liberty on the data setting since we are varying all the factor	Its based on 3 main law of experiment. ·       Law of readiness ·       Law of effect ·       Law of exercise
There are has been many instances that OFAT & Multi factor are being preferred to identify optimal setting based on data and process, due to structured approach.   Inventions are born from failures. When correcting the failure and re-trying your experiment many inventions are born, which is a very less percentage but more effective and leads to many innovations. Even though Trail and Error is not structured but it’s the approach that can bring new inventions. If the trial and error approach not been followed, we wouldn’t be in this Digital world and debating this approach..

 Trial and Error – Randomly changing inputs to observe changes to output without any specific plan

One-Factor-At-A-Time (OFAT) – Varying each factor one at a time to understand the impact of that variable on the output.

 Full Factorial Design – Running all possible combinations of factors in order to develop a complete model of the system.

 Fractional Factorial Design – Running only a subset of the tests and still being able to derive useful information from the tests.

 

The Trail Error Experiment is also known as S-R bond Theory and Learning Theory

This method was introduced by Edward Lee Thorndike. He first stated the element of this theory of learning in 1913 that connections are formed in the nervous system between stimuli and response illustrated by the symbols S-R another word used to describe these connections is the word “ bond” and hence this theory is sometimes called a bond theory of Learning.

 

Trial and error is a problem-solving tool and learning tool where multiple attempts are performed to get the right response / Solution. Learning takes place by trial-and-error theory. The learner makes random activities and finally reaches the goal accidentally.  

The scientific method can be thought of as an elemental trial and error in the formulation and testing of hypotheses. Also compare genetic algorithms, simulated annealing , and reinforcement learning – all varieties for search which apply the basic idea of trial and error.

Trial and error mostly use for learning, traditionally been used for new medicines and clinical trials, for example, Corvid vaccine, Corvid Drugs, etc. Chemist simply selects chemicals randomly and try to find desired effects. Trail and error experiment methods are mostly used in developing antibiotic drugs the saving human life.

Trial and Error:

  Trial and error is a problem solving method in which multiple attempts are made to reach a solution and learning from the mistakes. It is basic method of learning of new behaviors. This method is repeated until success or a solution is reached.

Example- climbing of a spider on wall is one of the best examples as spider use to climbs up and falls down and keeps trying till to reach the top or destination.     

One Factor at a Time:

OFAT is a method in which the impact of change in one factor is studied on the output when all the other factors are kept constant.

In OFAT, only one factor can be changed and can be used for screening of critical factors only.

OFAT tells the main effect of the of the factor on the output.

 In OFAT, the project lead can decide the number of experiments that want to do

It is statistical technique and it requires experiments to be conducted. Solution identified from OFAT need to be checked for practical or business sense.

Example- Assume the mileage of a car as the output and there are multiple inputs for this like:

Car condition, road condition, fuel type way you drive and resistance between tyres and road.

If I have only one car which is 10 yrs old, car condition is good, road condition is good, it is petrol car, use to take the same route to go to office every day, I have fixed the driving style and the tyres are in good condition. The given things mean that except of fuel type other factors are almost constant, for this I can use OFAT.

Multiple Factors at a Time:

MFAT is a method in which the impact of change in two, more or all the factors are studied on the output.

It is also a statistical technique and it requires experiments to be conducted. Solution identified from MFAT need to be checked for practical or business sense.

Example- Taking the same example as above of car mileage issue and assume the same as the mileage of a car as the output and there are multiple inputs for this like:

Car condition, road condition, fuel type way you drive and resistance between tyres and road.

If I have only one car which is 10 yrs old, car condition is good, road condition is not good, it is petrol car, I use to take different routes to go to office, I have fixed the driving style and the tyres are not in good condition. The given things mean that multiple factors or variables are responsible here for the mileage issue, for this I can use MFAT.

 

*Trial and error can be preferred over OFAT and MFAT in new product development where all the raw-material are available as per the requirement but the person/chemist/engineer is not making it as per the benchmark that why min three trial batches use to be prepared before sending to the market so trial and error can make perfection here, practice/trial can make the perfection after learning from the errors.

Trial and error can be preferred over OFAT and MFAT in learning the driving also. So the trials and errors can be preferred for new learning.

 

Experiment in statistical terms will refer to a method of conducting test in order to learn or identify something. It's like performing series or unique composition of input to observe outcome, an expected outcome. Most common example is measuring a cover box that needs to be in exact dimension to carry an expensive machine part. A mechanic will continue to focus on getting most accurate measure and may need some attempts to get a perfect 10. Experiments are essential to decide nature of the relationship between independent and dependent variables. The intent is always to design a process that reduces and restricts the variable impact on output.

Trial and Error – Repetitive attempts made until the perfect expected outcome is attained. In simple words, continue to practice till the desired result is obtained. Often you will notice a Cricket Batsman will continue to practice hook shots, practice till the time he is most certain to hit a six with least chances of getting caught. Ravi Shastri derived a unique way of pulling the ball and still keep the ball grounded. He surely used Trial and Error method to formulate the shot. The idea is to keep learning and adapting basis learning to get close to better than last time, eventually get to a solution.

Trial and Error method is most applicable where the risk or damage due to failure is less and affordable, where the chances are often and in some cases in abundant. You may want to keep trying drive a motorbike and fall but would not put same efforts for landing an airplane. Effectively used in Lean problem solving, Repairs, tuning an engine and obtaining knowledge.

 

One Factor AT A Time also known as one variable at a time that involves testing of one cause or factor at a time rather having multiple factors contributing to the outcome. In absence of historical trends providing expertise, it is viable to use One Factor and track the consequences. It is a simple way to notify how much has the result changes due to one factor. This test is much more controlled and helps in moving the decision logically and within a defined structure. Obviously because of its nature, one cannot predict the relationship between multiple factors and hence optimizing the solution by getting the perfect combination is missed. Also, OFAT may result in multiple runs and basis causes list. Example studying the impact on the strength of a glass bottle by changing Temperature, later Silicon dioxide composition and later funnel to hold the base.

Multiple Factors at a Time also called as Fractional Experiment is a design of experiment where two or more factors are involved, each having a specified contribution (levels) and all possible combinations across different levels generates a model. These experiments allow the study of each factor and also their interactions. These designs are more efficient and informative, low on cost and fast in deriving the optimal combination and have a better coverage when it comes to factors and at their unique combinations. Example, a motor mechanic may want to observe power consumption of two machines running at different speeds, carrying different weights and on different terrains. 

Trial and Error method for a simple reason of being fast, agile, less complex and investments is widely preferred over other experiments. Example in preparing new Combinations of drugs and medicine and antibiotics that will largely evolve with Learn Apply Learn Method and over time. This is also called Structure Activity relationship where relationship is built with what chemical will work best with antibiotic. Success of medicines in controlling covid was largely run with test and error experiments before Vaccination got evolved. Someone did the experiment with multiple paracetamol combination and eventually concluded DOLO 650 to be the best.

Similarly, most of the sports team will use the Trial and Error method over other options, especially when the opponent in new team and what may click and work in one circumstance may vary for the next time, these sequences best fit in Trial and Error Method.

 

 

Trial and error is the simplest experimental approach. Currently, across many industries, this approach is often coupled with fail fast, fail often concept to promote aggressive experimentation to learn and explore new ways of designing products/processes. Gordon Moore, founder of Intel, noted: “With engineering, I view this year's failure as next year's opportunity to try it again. Failures are not something to be avoided. You want to have them happen as quickly as you can so you can make progress rapidly.”

In the trial and error method, a 4 step approach is typically followed, wherein the experts start by first observing a problem, then make assumptions on what could or could not possibly work, devise & deploy a solution based on the assumptions, learn valuable lessons from the outcome of the deployed solution. This approach is repeated again and again, each time with adjustments made to the design based on previous learnings, until a convincing solution is arrived at (or a decision is made to simply stop the experiment based on several other factors).

 

Trial and error has many advantages compared with the other experimental setups:

a) it is easy and quick to setup

b) it does not need special expertise and can easily be implemented by the specific subject matter experts

c) it promotes a culture of learning by doing and observing

 

Trial and error should be the last resort if other approaches cannot be used due to the following disadvantages:

a) Not rule based and hence, does not provide a structured thinking on all the factors that could affect the experiment

b) Its best suited only for simple problems, whereas typical real world problems have a wide variety of variables as inputs

c) It cannot be used to predict an outcome as it does not provide enough clues on why a solution works

d) It cannot be used to find the best possible solution or all possible solutions

 

Examples to understand trial and error: Our day Is filled with instances where we continuously experiment with new things. Trial and error seen in nature: A monkey figuring out to open a soda bottle with a bottle opener; Trial and error seen in our everyday life: A person trying to figure out the best transportation to commute to work, a baby trying to stand up and walk; Trial and error seen in industry: Supervised machine learning.

 

One factor at a time: As the name implies, an experimental approach in which the factors are tested one at a time. This approach is also known by various other acronyms such as OVAT, OVaaT (one variable at a time). Suppose we have a process with inputs X1, X2 that are processed to provide an output Y. In OFAT, we keep X2 constant and vary X1 to find the optimum result. Once the optimum output is achieved, X1 is fixed at that point and we then vary X2 to find its optimum value. In this way, one factor or variable (ie)., X1 or X2, is varied at a time till all the input factors are exhausted.  

 

The main advantage of OFAT is that it serves as a good alternative to multi factor analysis when collecting data for multi factor is cumbersome or not economically viable.

 

OFAT has the following disadvantages:

a) Cannot figure out interaction effects between the variables under analysis. This means, OFAT cannot be used if the input variables to a process interact with each other and the interaction also has an impact on the output.

b) In certain cases, depending on the number of variables, can lead to large number of experimental runs with a negative impact on cost and time.

 

If you want to understand adding 2 strawberries or 4 strawberries (factor / variables) to 1 glass of milk, with 2 spoons of sugar, mixed for 3 mins in the same mixer (fixed entities) gives the best strawberry smoothie, we can use OFAT.

 

Multiple Factors at a time: When multiple factors are manipulated to study their effect on the response/output, its called multi factor at a time experimental approach. This solves the major disadvantage with OFAT pertaining to interaction effect as the response pertaining to changes on multiple factors at the same time are observed to find the optimum response. In case of random selection of input variables as followed in trial and error, there is no structure which gives clear understanding if all possible combinations of inputs have been analyzed in an orderly way. Hence, with trial and error, its not possible to quantify if the solution finalized is the best optimum. With multi factor analysis, as the experiment can be designed to include all the interactions that is of interest, probability of achieving the best optimum solution with an orderly experimentation is high. Further, the multi factor analysis also provides key insights on the key factors to be considered, the optimal settings for each input variable to be set that would result in the least variation in the process output.

 

Despite the advantages, the subject matter experts in coordination with the management should determine the right use cases for doing a multi factor analysis by taking the complexity, cost of experimentation, multiple runs, practical feasibility for simulation and time needed.  

 

An OEM that manufactures Turbines for power generation can use a multi factor analysis experimental approach to find which operational conditions such as ambient temperature, fuel used, fuel gas temperature, etc will result in the maximum power efficiency.

Trial and Error	One Factor at a Time	Multiple Factors at a Time
This is problem solving method for relatively simple problem	This is problem solving method relatively complex method	This is problem solving method relatively complex method
Multiple attempts to be made to get a solution	Only one factor can be changed at a time, but others remain unchanged.	Multiple factors can be changed at a time
Trial and error doesn't try to apply a solution to other issues in a general way.	This can be used to generalize a solution to other problems.	This can be used to generalize a solution to other problems.
This is a non-optimal method to find solution.	This is an optimal method to find solution.	This is an optimal method to find solution.
This is not for finding optimal solutions	This is relatively slower than the Multiple factors at a time to find optimal solutions	In addition, it can speed up the finding of optimal solutions compared to OFAT experiments.

 

 

Over the other two more structured approaches, trial and error will be preferred for the following conditions

1.When you have more test subjects than you require for each test, trial, and error works best.

2.When optimal solution is not required

3. When there are a testably finite number of potential solutions, it is possible to use trial and error to find all solution or the best solution

Interesting answers from all participants. Structured experiments are always better than trial and error. Situations where Trial and Error could potentially be used are

a. Cost is a big constraint and hence structured experimentation is not possible

b. Complexity is high and all factors cannot be controlled

 

There are 2 solutions which hinted towards these concepts - Ashish Kumar Sharma and Godwin Thomas. Hence they are the joint winners for this question