AnshulVaidya

Harada Method is a lean improvement method, developed by Takashi Harada, that targets “Enterprise Performance Improvement”, through daily employee coaching initiatives and, augmented motivation routines.
 
Lean Methods attempt to preempt waste removal before actually being produced in process. The Harada Method helps individuals leap ahead towards self-reliance, by targeting reduction of the eighth lean waste, “under-utilized talent”, through “a five-step approach”.
 
To initiate a foundation, let’s start with two Japanese terms, associated with the method--
 
Monozukuri and Hitozukuri
 
Monozukuri: refers to sense of craftsmanship. The method propagates that employees can reach excellence in work by taking pride in their work. An endeavour should be made by employees to strive for continuous improvement.
 
Hitozukuri refers to continuous process of helping people excel in work and achieving excellence in tasks and skills. Hitozukuri advocates that people need to be trained on skills, tasks required to set their own work targets. The employees then are encouraged to attain the performance targets, set by employees for themselves.
 
Harada method propounds that the employees can achieve Monozukuri through Hitozukuri.
 
The method envisions a five-step approach to plan for skill achievement, initiate improvement in task and reach excellence in a skill:
 
1.      Premeditation: Defining the skill, a worker would like to improve upon.
 
2.      Personal excellence: meeting excellence in desired skill and task.
 
3.      Goal setting: participating in goal planning sessions with mentor, coaches, managers to reach mastery in a skill.
 
4.      Selfless service: applying new learning with utmost dedication to production.
 
5.      Self-reliance: exert self-effort into production, and develop self-reliance with competency.
 

 
Above five action-points are realized by starting with an introspection session on the purpose of work, between employee and coaches/trainer. A task, skill and goal are identified, where competency may be mastered by the company employee. An in-depth analysis of strengths and weakness helps deconstruct preparedness required, to attain mastery. A worker or participant is required to list out 64 small steps to gain expertise the requisite work-domain, in precise time horizon.  A 64 charts box is constructed to “list and grid” 64 key identified ideas in a 3*3 matrix depicted here-under:
 

 

 
In training interaction, employees are made to introspect about meaning for self-reliance for themselves.  An employee is made aware about Mental Wellness, Skill technique n style, Physical Health endurance, Life aspects at work and outside of work. These four frontiers of human existence are discussed, with employee’s past performance and future expected performance on paper, during learning sessions. Basis the discussion, goal, purpose and targeted goal completion time line as a part of training plan are discussed and, finalized for the employee.  
 
New routines are identified, prioritized in order of importance and urgency. An employee is made aware about new daily routines and “time framed goals”, that an employee is expected to complete in sequenced time duration. The completion timeline is estimated for each routine, with completion date defined for each targeted goal, actionable idea and new routine. Managers and employees are encouraged to record their daily progress on trajected improvement plan and training sessions, in a progress dairy maintained by the employee.
 

 

 

 
 
Harada Method is keenly followed by Toyota Motors through adaptation of logo “making things is making people” (Monozukuri wa hitozukuri) or “develop people and then build products” in Toyota Production System. Harada Method is ingrained in Toyota Lean methodology as Leader standard work. The structure encompasses a shift from “thinking results” to “results and process” by translating focus on process, into a concrete expectation from leader’s job performance.
 
 

While terms special causes and outliers are used interchangeably, the definition, occurrence, method of detection differ for these two phenomenon. Hence, it’s imperative to study and consider them as different elements, when these are used to denote specific data points; away from common distribution of sample universe.
 
Definition
 
A Special Cause variation is a variation which is assigned due to special assignable cause such as accident, breakdown, defect, delay, fault, mistake, and/or shortage in the process. The term was first introduced by W. Edwards Deming, and used to denote an unexpected glitch that is “unusual, sporadic & non-quantifiable” in nature. Examples of Special Cause variation includes computer crash, machine failure, Operator falls asleep, Insufficient awareness, irregular click through rate of Google Ad Words, Deficient batch of raw material.
 
An Outlier is attribute assigned to data point that is distantly away and differs significantly from other observation. Outlier occurrence is assigned to variability in measurement or/else to an experimental error. An Example of Outlier includes a set of lower magnitude values (10,15,25) or higher magnitude values (150,200,225) in a set of natural numbers between 50 to 100.
 
Detection
A. Detection of Special Cause Variation
 
Special cause variation are random unexpected variations occurring due to unusual occurrences.  Control charts are used to identify special cause variation. A stable process is represented on control chart as given hereunder:
  
 Control Chart for a Stable Process
 
A special cause can be identified by looking for presence of plotted point located outside the control limits or having presence of a non-random pattern of variation on control charts specified with in the control limits.

Control Chart for Special Causes
 
B. Detection of Outliers
 
1. Sorting method In Sorting methods data variable are sorted in lower to higher order or vice-versa to identify and eliminate extreme small or larger magnitude numeric variables.
  
2. Use of Graphs The Data values are plotted using Histogram, Scatter-charts and box plots to identify outliers in schematic charts.
Any outlier is represented by taller pillar in histogram plot or a smaller pillar of lower magnitude, distinct from other data values.

 
Similarly, data outlier may be represented using box plot where percentile data and quartile values may be used to represent outlier distinct data point as point distinctly located from main quartile box plot or located as distinct data point away from box plot of category of different category of data values.
 
 

 
Scatter plot for regression between two variable is represented below with most of the points fitting the model however circled outlier represents points that does not fit the regression slope line plotted hereunder:
 

 
 
3. Z-Score  Z-Score is plotted for the numeric data values and distance of numeric data values from mean value of the sample is determined. Values with too small or too high Z-Score is considered an outlier value. Here, as a rule of thumb numeric values with Z-Score higher than 3 and lower than -3, are considered as outlier.
 
Z-Score = (X-µ)/α 
i.e., Data value minus mean value divided by standard deviation

 
1.      If the data is not following normal distribution, Z-Score based identification may not be useful in identifying outliers.
2.      For smaller data sets, Z-Score may not provide valid identification of outlier since maximum Z-Score value is limited to (n−1) / √ n
 
4. Interquartile Range
Interquartile Range is a measure of statistical dispersion of data. The IQR is used to describe middle 50% of value residing between Quartile Three Q3 and Quartile One Q1; i.e. IQR= Q3-Q1, indicating difference between 75th and 25th percentile of data. IQR is also represented with terms mid-spread, middle 50%, fourth spread, or H‑spread.
 The valuation of IQR, quartile values and adjustment factors are used to determine the minor and major outliers in data.
IQR estimated is multiplied by 1.5 and 3.0 respectively and resultant values are further used to estimate minor inner fence outlier, minor outer fence outlier along with major inner fence and major fence outlines.
Let us consider hypothetical value of Q1 as 2.354 and Q3 as 3.055 that results in IQR =0.701
Multiplying IQR with 1.5 and 3.0 results into 0.701*1.5= 1.0515 & 0.701*3.0= 2.103
To calculate minor inner fence outlier, minor outer fence outlier, subtract the two values obtained above from Q1-->2.354
2.354-1.0515=1.3025-->minor inner fence outlier.
2.354-2.103 =0.251--> minor outer fence outlier.
To calculate major inner fence outlier, major outer fence outlier, add the two values obtained above from Q3-->3.055
3.055+1.0515=4.1065--> major inner fen ce outlier.
3.055+2.103=5.158-->major outer fence outlier.
By comparing data point values with values obtained above points lying beyond major fence outlier value 4.1065 in this case are considered as major outlier.
 
5. Hypothesis Testing
 
Hypothesis testing may be used with constructing Null Hypothesis and Alternate Hypothesis as
 
Null Hypothesis: Ho: All data points in sample are collected from same sample following normal distribution.
Alternate Hypothesis: Ha: One value in the sample is not collected from sample with all other values of sample following normal distribution.
 
In case of p-value being lower that significance value of 0.05, it is concluded that Alternative Hypothesis is true, that one data value in sample universe is outlier; and not following normal distribution unlike rest of other sample values in the study.
 
 

Lean Accounting, refers to the use of Lean thinking in financial practices, followed by a company. Lean Accounting is centred around the idea of improvement in value delivered to clients and waste elimination targeted through better workflow & material management. Core outcome of lean accounting includes improvement in sales, cost reduction, growth in company business and improvement in company’s bottom line operations.
Lean Accounting may be implemented through adaption of key five ideas, as given hereunder:
1) defining value, 2) mapping the value stream, 3) creating flow, 4) using a pull system, and 5) pursuing perfection
1.       Defining Value: Value is benefits associates with product/service that the customer is inclined/willing to pay for. Ensuring that the value is defined as, “the exact benefit”, catering to a defined customer need, is important to establishing value of product/product group. The use of primary market research techniques such as Face-to Face Interview, surveys, customer-polls help in establishing value of product/service for the customer.  The qualitative and quantitative techniques help in estimating the product mix including product, product usage, price and product availability in niche market.
2.       Mapping the value stream: Value stream is sum total of activities that add value for the customer from initiation step to final realization of value by customer. The activities that do not add value to customer are defined as waste. Waste may be further differentiated as
a.       non-valued added but necessary: should be reduced as far as possible.
b.      non-value & unnecessary: should be eliminated.
Different steps and factors in production add to the cost of product and any increment in these register an impact on value stream profit and loss statement in lean accounting.

 
Cause and Effect Diagram: “Value Stream mapping in Lean Accounting”
c.       Creating flow: Generating a flow in production is necessary to streamline production. The use of sequential steps in production, realignment of activities in production, bifurcation of production space into cross-functional departments, generates steady flow in production. The workload levelling by reducing Mura (unevenness) in production (production of intermediate goods at constant rate), mitigates fluctuations in consumer demand & generated flow in production.
d.      Using a pull system: A pull based system is established by maintaining a minimal inventory of stocks of raw material, goods, work in progress items required for production. Thus, Just in Time systems, thus generated, ensure that the product/product group is manufactured at the right time and, in the specified quantities, as defined in customer order. The over-production of goods is eliminated as a consequence of maintaining a minimal inventory of stocks of raw material, goods, work in progress items.
e.       Pursuing perfection: Pursing perfection ensures that company is a learning organization and finds ways to implement little improvements every day, in its operations.

 
Lean Accounting
A Lean Accounting System is believed to initiate changes in Financial Management, Financial Leadership, Accounting methods- variables, target, process and stakeholders, along with use of Shewhart cycle or Deming Cycle or PDCA cycle to improve value delivered to customer sustained through waste reduction. With Lean Accounting, the accounting process is shared with Accountants and Value Stream professionals instead of the controllers. The accounting charts are now differently parameterized from standard cost accounting variables-- product cost, standard cost & variance analysis, to value stream profit and loss. The use of cause-and-effect diagram enables prediction of the likely impact of changes in production variable, to value stream of the production process.
Anchoring Training and Education about value stream for mentoring employees, assessment about current and present state of industrial production, pilot studies to design value stream improvements, & use of PDCA (acronym for Plan, DO, Check, Act or Deming Cycle) in continuous improvement measures, redefines baseline effectiveness, in the entire accounting process.
In PDCA cycle, the alphabet P stands for plan, alphabet D stands for DO, alphabet C stands for Check and alphabet A stands for Act; can be utilized to stream-line the process through examination for weakness and threats, that retard the production. Daily hurdles, visual boards, status checks, Team problem-solving meetings are planned and executed to implement continuous improvement through Shewhart Cycle. Quick wins, are established, that list non-value added and unnecessary activities, which when stopped have no impact on production. Improvement ideas in accounting domain are tested for one to two weeks, before incremental plan is devised in short daily meeting, through continuous monitoring and analysis of the functional data, related to devised improvement idea. The Process improvements are planned in advance to reach specific improvement in performance metrics over a period of time. A process improvement may be discussed and initiated over a period of time to tackle a significant, sudden disruption in process performance. A hypothesis is developed and tested around production and operational parameters, to accept or reject the contribution of baseline performance metrics, towards waste elimination & operational excellence. The results are further scrutinized by experts and organization higher management, to initiate a cycle of activities and schedules, are that repeated periodically in-phase, to improve effectiveness, in industrial production.
An existing lean set up in an organization, is qualifying criteria to proceed ahead with Lean Accounting. Next, a provision for a lean budget or hoshin kanri, is pre-requisite to lean accounting. A continuous improvement focus through Kaizen practices, is essential to identify the performance parameters to be monitored, and, tracked through lean accounting. The performance parameters used in lean accounting are represented with the use of box scores. The box scores are tools used for short term decision making, basis the assumption that, the company costs and company consumption is fixed. In medium-term, decision-making box score are developed, assuming that company costs and company’s cost are not fixed. The use of box score enables value stream to publish a ‘weekly P&L’ in terms of actual costs, actual production units. Additionally, it is possible to plot real cost drivers of conversion margin and conversion cost, which is not possible in case of traditional accounting. The box scores are used to shows the performance of the financial results, operational results, value streams and the capacity usage.

 
Financials:
Capacity:
Operational metrics:
Duration:
Revenue
Available capacity
Average cost
3 days SCO
Return on sales
Productive capacity
Dock to dock days
10 days RUN
Value stream profit
Non productive capacity
Sales per person
3 days Evaluate
Total costs
 
Stock outs
 
Material costs
 
Scrap
 
Employee costs
 
 
 
Machine costs
 
 
 
Other costs
 
 
 
Utilities
 
 
 
Facility
 
 
 
Inventory value
 
 
 
Cash flow
 
 
 
 
Lean Accounting Metrices

Box Scores
 Performance Measure
6/2
6/9
6/16
6/23
 6/30
Goal
 Operational
 Units Per Person
15.10
15.63
14.7
15.91
15.90
20.7
 On-Time-Shipment, %
 100
100
100
100
100
100
 Average Cost, $
 343
337
362
338
337
262
 Capacity
 Productive, %
 29
29
29
28
28
40
 Non-Productive, %
 54
54
54
52
52
 33
 Available, %
 17
17
17
20
20
27
 Financial
 Revenue, $
471
485
456
490
488
576
 Material Cost, $
 123
125
129
132
135
139
 Fixed Costs, $
 120
120
118
116
116
108
 Return On Sales, $
 38
39
35
38
33
48
 

 


 
 
Types of Box Scores
The data for operational performance in box score is estimated  using value stream visual management boards. The data for the financial performance information is derived from value stream P&Ls and supporting schedules. The data on capacity is developed as link between operational and financial performance. With implementation of lean initiatives, an improvement in capacity is registered, as non productive capacity is aligned as available capacity.
Further the individual estimate of direct cost factors-- labour, material and other factors associated with the industrial production of product/product group. Fixed factors of production-- equipment tool and machinery, insurance, rent, taxes are calibrated to the estimate to reach total costs. The total cost estimated for production of product/product group is then divided by by the number of units to arrive at unit cost for the product/product group in industrial production. Thus a reliable estimate of Direct Cost, Occupancy Cost and Contribution margin and box scores for performance metrices for each product/product group is achieved, that helps in informed decision making, about the product.
Lean Accounting  in nutshell                                          
Actionable                                                     
                
                Targeted  
                Impact     
               Level
Create Awareness
Build Desire
Demonstrate
Sustain System
Activities
Training and Education
Assess Current State.
Define Future State.
Conduct Pilots
Standardize work.
 Practice Routines. P.D.C.A.
Stakeholders
Senior Leaders
Finance and Accounting
Functional Managers
Core Transition Team
Lean Financial Coaches and Entire Organization
Outcomes
Training onsite or online
Assessment and Design Services
Onsite Consulting
Blended consulting on-site/online
 
Traditional Accounting & Lean Accounting
Traditional accounting practice differs from Lean Accounting on tenants such as inventory management, use of simpler accounting variables, generation of simplified accounting reports, incorporation of value stream & continuous improvement rather than product as accounting objective. Lean accounting encompasses Lean-focused performance measurements to generate correct understanding of the financial impact of lean change. The lean accounting relies up-on direct costing of the value streams & does not support the use of traditional accounting variables such as standard costing, activity-based costing, variance reporting, cost-plus pricing & complex transactional control systems. This in-effect eliminates budgeting through monthly sales, operations and financial planning processes.
Lean Accounting companies are expected to have lesser stock items in inventory, to achieve Just-In-Time specification in production, with specific mention in balance sheets as the total value of all inventory. A noted difference between traditional accounting and lean accounting is compliance to Accounting Standards General Acceptable Accounting Practices GAAP, an established accounting standard in United Kingdom. Lean Accounting, as a practice, is not compliant to GAAP requirements & Enterprise Resource Planning ERP software, that make it less prevalent, at many organizations. Further legal provisions may mandate maintaining accounting books, in both the traditional book-keeping and lean accounting formats.

 
                                      
 
 
 
 
 
 
 

CATWOE Checklist
CATWOE is an acronym for Customers, Actors, Transformation, Weltanschauung or World View, Owners, and Environment given first by David Smyth in 1975, as combination of six elements- problem solving.
C – Customers:  Customers includes a set of users and stakeholders of a system. Customer benefit from changes within a process & at resolution of the problem impacting system. An early categorization of winners and losers in customer group, wins apostles and prevent future loss to the organization.
A – Actors: Actors refers to employees within an organisation. Actors are responsible to create output and implement changes in the system. An activity measuring actor attributes-- abilities, qualities and interests acts as a tool to determine actor’s influence and resistance to change management in the system.
T – Transformation Process: Transformation is the change process influencing the system or the change effected as result of process activity. As such transformation may be actively introduced through human design to improve business, by estimating and arranging for volumes of input required; estimating volumes of output generated and planning for the market access, & revaluating production steps.  
W – World view: World view refers to opinion of different stakeholders and interested parties acting in organization’s market environment. Stakeholders may be opinionated differently about system/process/organization and bring conflicted view to operational desk. CATWOE analysis attempts to shares explicit contract in opinion, while deciding upon best approach for organization to follow. Consideration about the Unique Selling Preposition USP & the bigger industry picture, real market scenarios and likely impact of solution in world-view, need to be considered, while designing success strategies for the business.
O – Owners:  Owners includes the top management, entrepreneur, investor in a business that hold final say on “GO or NO-GO decisions” related to organization affairs. Key considerations on cost, profit management and ethical consideration for business are decided by these highest decision makers for the organizations.
E – Environmental Constraints:  Environment includes the actual environmental elements influencing business and may curtail strategic decisions for the organization. Political, Ethical, Social, Technical, Legal, Finance Leverage &funding and Environment factors act as environment constraints for organization and business activities. Due consideration is given to chalk best action plan for organization using CATWOE framework.

 
 In a conflicting bear economy, as in low profit market, an investor may be interested in selling investment to close losses, an entrepreneur may declare lay-off to balance losses in account accounts & to optimise process efficiency, employees may favour increasing price of produced goods & services, and community may expect new hiring to improve market sales. The wider contract of perception-based differences among the organization stakeholders, in-effect lowers the effectiveness in decision-making and problem solving. Observed series of conflict of opinion between stakeholders, may bubble into stakeholder conflicts & serious challenges later for the organization.
Consequently, CATWOE checklist, may be used to define and analyse business stakeholder’s perspectives, to reach best achievement.
CATWOE checklist enables problem identification, reviewing company’s targeted achievements and, designing bouquet of potential solutions to influence stakeholders. The techniques offer resolution to complex business situations, conflict of interests, standpoints & backlog, due contrasting interests of multiple stakeholders.
CATWOE checklist, is emphasised as “ROOT DEFINITION CHECKLIST”, implying a list structured— “to enlist stakeholders and represent stakeholder’s positioning” towards segmentation, targeting and promotion of product or service in market.
A competitive market scenario would require stepwise intricate analysis of stakeholder contribution, to develop strategic direction for product or service in a niche market.

CATWOE Stakeholder Analysis
CATWOE allows managers/problem solvers to develop holistic encompassing view approach to review system and its elements in multiple perspectives.
CATWOE analysis attunes clarity to feasible outcome to an issue with multiple perceptions. A converged solution document bearing explicit research about feasible targets and likely achievement of each stakeholder, is generated through CATWOE analysis.
The varied perceptions of different stakeholders are collated at a common platform, through holistic understanding. Facts are assembled to evaluate authority positions of stakeholders, domain-based assertions, baseline assumptions theorizing integrity of the data and information. The emerging solution is scrutinized to test ethical angle in each stakeholder analysis.   The practice forms basis to integration between two or more perspectives, assigning priority to the merits of the respective perspective, and final selection of one perspective while overriding other perspectives.
Thus, the business model is studied, stakeholder’s expectations and targets are evaluated at micro and macro levels; to share projected estimate of business projections, valuation about bottlenecks to be eliminated and final review of ethical consideration is held in decision-room. The process culminates into a correct approach to be followed by organization, at different stakeholder levels.
 
 C.A.T.W.O.E. In New product Development
C – Customers.
Real consumers and beneficiaries of transformation--Company customers and marketing executives.
A – Actors.
Persons responsible for transformation activities--Researchers, engineering executives, marketing executives, HR executives, supplier’s engineers, Customer’s engineers, production executive, P/E executive.
T – Transformation Process.
Brief about changes in high-level business process -- VOC + tangible + Intangible assets  à First time right products and services meeting customer requirement at the lowest cost.
W – Weltanschauug or World view.
Stakeholder view/World view on Company positioning -- Strategic business positioning in global market where cost effect NPD is must for the survival.
O – Owners.
Person with authority to direct transformation activities -- Company’s Top-management
E – Environmental Constraints.
Impact on condition/rules under which the business operates --Manufacturing facility, technological development, budget allocation.
 
 
 
 

A/B Testing
 
Let’s consider a test goal of designing web pages using marketing strategy, to get maximum online audience & placement for marketing ad creative. (i.e., groups of words used in text promotion.)  
 
A/B testing or bucket testing or split-run testing is utilized to test “likeness performance of a text/ ad-message” by control group and test audience, for two finalized designs of promotional content; for the vector under consideration.
 
At ideation stage, copywriter/marketer may narrow down two final designs to be creatively scaped on landing page of website, email template, webpage advertisement or ad-word ad, pop-up message on end user screen. A set of reliable measurable data on Key Performance Indicators KPIs, “Return on Investment”, “Click Through Rate”, “Average Revenue Per Session”, “Revenue Per Visitor”, is put to scanner through “A/B testing”; to select better design for online promotion: as perceived by Control & Test audience.    
 
ROI: Return-On-Investment, is defined as “ratio of Profit earned by investment divided by total cost of investment.”
 
CTR: Click-Through-Rate, is defined as ratio of “select number of users clicking on hyperlink against total number of users reaching webpage, email, ad-creative.” 
 
ARPS: Average Revenue Per Session, is defined as, “average amount of earning per web session or earning registered in visits to the website”, and calculated by “dividing the total revenue of in web session by the total number of website visits.”
 
RPV: Revenue Per Visitor, is defined as, “the amount of money generated each time a customer visits a website”, and calculated by “dividing the total revenue by the total number of visitors to a website”.
The two variants— “Variant A” or “Variant B”, are put to test using one or more method-- “AB Lift Test”, “Z-Score test” & “statistical hypothesis testing” or "two-sample hypothesis testing". The scores are analysed to decide about effective outlook to be shared in production.
 
 
One may complete A/B testing by following these steps:
1.       Think and decide on objective to be achieved from online promotion, including learning, training-needs to-be-met, & promotion to be shared.
2.       Determine Key Performance Indicator of customer success to be utilized Return on Investment, Click Through Rate, Average Revenue Per Session, Revenue Per Visitor.
3.       Decide on ad-timeslot and web-locations with highest traffic volume and that most-cost to improve optimization process. It is important to note that spots with lower traffic volume require longer campaign run periods to get standard outcomes.
4.       Decide on elements of campaign design to be tested in A/B testing. Usually, a change in single element is recorded in A/B testing to get authentic results. Changes in banner- header or footer, different placement of message with varied selection of words- short precise or simple statements in user comprehendible language, may be put to test.
5.       Monitor data generated in the test run and highlight trends resulting in better KPIs higher traffic, higher conversions, more revenues and better ROI%.
 
Precautions in A/B Testing
1.     Sample Size: Sample size used in A/B testing should be sufficiently large, to generate authentic user opinions, on changes in content design; as means to effective communication. A sample size between 100 to 1000 user observation be considered a good fit. Further confidence level, statistical significance may be observed to interpret finding from statistical tests used in A/B testing.
2.     Blocking: Blocking is act of accounting biases in sample group to maintain randomness in the sample. The first step is to form homogenous columns (basis gender, age, geographic location, occupation) in groups, followed by a randomized test within each group to check trends within the group. Blocking adjusts randomness biases originating from homogenous similar groups of respondents, in sample- control group (responding to campaign design without altered design variable) and test group (responding to campaign design with altered design variable), with fresh randomized testing. Blocking is effective for 2-3 variable and re-randomization is preferred in A/B Testing, when dependent covariate is higher in number in sample.
3.     Adjust Test run duration with traffic volume, to generate authentic trends. A/B test run with lower traffic need to be continued for a longer duration of time to result in authentic customer preferences insights on campaign design.
4.     Testing for multiple changes in campaign design, using A/B testing, may lead to spurious correlation in results.
 
Tests used in A/B Testing
A/B Lift: Lift in A/B Testing is defined as, “the percentage difference in conversion rate of control design and a successful test treatment.”
Application:
1.       Incremental Lift in Revenue Per Session in A/B Testing: may be calculated by initially subtracting revenue per session of the control design from the test treatment campaign. The resultant figure may be divided by the revenue per session of test treatment and then multiplied by 100, to get percentage estimate of Incremental Lift in Revenue Per Session in A/B Testing.
2.       Cost of Running the Test: Since, Control and Test group form basis to A/B testing, and are implemented on 50%-50% capacity: it can be inferred that control, did not generate revenue in the A/B testing. Cost of Running the test, is calculated by, “multiplying the revenue generated for control treatment by the incremental revenue per session (%).”
3.       Identification of Multiplier for Test Distribution: A multiplier is required to be identified, to estimate of value required to get value of test distribution (Test/Control) equal to 100% traffic volume. Basis Test and Control group used in 50%-50% campaign runs during A/B Testing, multiplier of 2.0 is considered for evaluation purpose.
4.        Estimate the Value gained if Test Campaign is rolled to a 100% traffic-users:  The estimate of value gained on idealizing winning test campaign to 100% traffic user is calculated by, “multiplying cost of test campaign with value of Multiplier realised for Test Distribution in earlier step.”
5.       Value Gained from Testing Period: To calculate the value gained by the winning test only during the testing period, “subtract the cost of running the test from the value gained if rolled out to 100% traffic volume.
6.       Forecasting Incremental Revenue Gain: It can be safely concluded from above steps that treatment design wins and is rolled out at 100%. Further the winning treatment campaign would be earning an incremental revenue for a set period of time known as forecasted period for successful run decided using market condition, nature of product or service and campaign design. Forecasting Incremental Revenue Gain can be done by, “dividing the value of the selected winning test design at 100% traffic volume, by the number of days of test design run. Further, multiply the daily gain by the forecasted number of days.”
7.       Overall Program Value:  Overall Program Value need to be calculated by estimating the total value of A/B testing program including cost of all test design campaigns and statistical testing including hypothesis testing implemented for programme. The operational cost is then reduced from program cost achieved above to reach the estimate cost of A/B Testing.
 
Z-Score in A/B Testing:
Z-Score is standardized score used for data under normal distribution, that describes the number of standard deviations a sample element is from its mean. In case of A/B testing website or online respondent is considered as observation parameter. The Control group are represented through bell-curve in plot and Test group & sample set of all website visitors represent the second bell curve or the shift in the bell curve from sample mean.   
 
Hypothesis Testing
Two alternative hypotheses are tested for statistical significance and impact of A/B Testing, assuming:
i.                     Null Hypothesis: H0: Default campaign design generates a conversion rate equal to x% (default).
ii.                   Alternate Hypothesis: H1: New variable in campaign design generates a conversion rate greater than x%.
Here, critical observation is expressed using
p-value= statistical significance of test=probability of negating Null hypothesis when Null Hypothesis is true.
Alpha(α)= false positive rate; or significance level; or type I error.
Beta(β)= false negative rate; or type II error.
power: true positive rate; (1 – beta).               1 - alpha: true negative rate.0
 
Likely outcome of Hypothesis testing:
No error
Type I error: In-correctly rejecting Null Hypothesis, when H0 is true and accepting alternative hypothesis as likely outcome of Hypothesis testing.
Type II error: In-correctly accepting Null Hypothesis, when H1 is true and rejecting alternative hypothesis as likely outcome of Hypothesis testing.
 
Here, key consideration is to determine, whether the difference between two sample distribution; i.e., the original default control design & the new test design with change in one variable; has occurred as random chance: or, there is a presence of bias, accounting for the difference. Statistical reasoning using three critical parameters Confidence, Statistical significance and Significance may be utilized to remove speculation about bias in observation of control and test design.
 
Confidence intervals measure the degree of uncertainty or certainty in a sampling method. Or a confidence level refers to the percentage of all possible samples that can be expected to include the true population parameter. Or, a confidence interval is how much uncertainty there is with any particular statistic. Confidence intervals are often used with a margin of error. Normally confidence level of 95% to 99% is desirable for test statistics to reach meaningful interpretation.
 
Statistical significance is validation of a result to be attributable to a specific cause, after testing or experimentation, is achieved on sample data. In statistical hypothesis testing, a result has statistical significance when result is an unlikely outcome, given the null hypothesis. Statistical Significance is measured in p-value and normally p-value for valid statistical significance is estimate as p-value=0.05.  The result hold statistical significance, when p ≤ α.
 
The significance for a hypothesis test is a value, for which a P-value, less than or equal to, is considered statistically significant. The significance value may be shared as 0.1, 0.05, 0.01 depending on nature of sample universe and study type. The significance values correspond to the probability of observing an extreme value in sample by chance.