I have found many people struggling with the question -

"what should be the appropriate sample size for the kind of data I have?"

Let us initiate this as a discussion string for everyone interested to participate in it.

We shall be in the discussion all the while to take it forward.

Is there a way to calculate the appropriate sample size to do internal audit in a data entry process? Avg. production per day is 4000 records.  Team size 75.

Client SLA - 98% - DPMO.

Thanks,

Fowzieh

Fowzieh,

To calculate sample size in your example, we can proceed like this -

• Margin of error - Let us assume 1%. This means when you report 98% as accuracy, it could actually vary between 97% and 99%
  
• Confidence Level - 95%. This means that 95% of the case shall lie within margin of error.
  
• Population size - 4000
  
• Response Distribution - 95% This means you expect 95% accuracy normally. If you do not know what to expect this should be taken as 50%.
  

Now use http://www.raosoft.com/samplesize.html to get sample size. You can use a formula alternatively.

You shall get a sample size of 1254. If you are willing to accept 2% margin of error, the sample size reduces to just 410.

If you do not know response distribution (and assume 50%) you shall get large sample sizes. In this case, instead of 1254, you shall get a size of 2825.

You can notice that if you are willing to live with higher margin of error (say 5%), you can manage with smaller sample sizes.

Let me know if you have any questions.

Regards,

VK

Hello Vishwadeep,

Thanks for your response. The link is very helpful.

If I consider the Average Total Volumes processed in a month as the population which is let's assume 80000.

Margin Error - 1%

CI - 95%

Response Distribution - 95%

The sample size is 1785. - Since the population is for 1 month, can the target for the Quality analyst be - 1785 / 20 [ considering 20 working days in a month] which is 89.25 ~ 90 records a day?

Also - Can the previous month's volume processed can be considered as the population to calculate the sample size?

Or Is it always better to take a day wise sample size based on the volume cleared on the previous day?

With the following population size and the rest of the numbers as mentioned above - the sample sizes are:

Day Wise Volume processed - 4000 - 1254

Week Wise Volume processed - 20000 - 1673

Monthwise Volume Processed - 80000 - 1785

I am guessing if I freeze on 1785 as the sample, I could divide that into no. of days in a month and the result (90) can be the target for the Quality Analyst.

Please let me know your thoughts.

Thanks,

Fowzieh

Fowzie,

My views are -

If you report out every day, you will need sample size based on daily volumes.

In case you report out on a monthly basis and data is fairly randomized (similar call types on different days of the week), it may be OK to take 1800/20 = 90 daily if 1800 is your monthly sample size.

Any comments by others on this opinion are welcome.

That answers my question.  Thanks Vishwadeep.

Regards,

Fowzieh

 

nice to see the feedback exchanges at the forum.

sanjay bishnoi

mumbai

Dear VK,

This is my first response to this forum. (I did the GB in Frb 2005 at Chennai, if you recall).

Some observations to you and Fowzieh :

1. Why should you be doing a monthly data entry audit (at end of month I suppose) if you are handling such high transactions?

2. If so, How could you cope up with timely data corrections and taking timely CAPA actions to solve those defective records

3. Surely you would prefer to do the CAPA on a daily basis with less transactions (4000) and daily sample lots (even if higher at 1254 records). Surely you want to correct any defectives at least by the next day (hopefully) isnt?

4. VK, you didnt comment on the type of data distributions in this case : is it binomial, or a hypergeometric application, since sample sizes are huge. The referrred Calculator assumes which type in these cases? (HG, B or Normal approximations?)

Awaiting your inputs and clarifications

George Varkey

Chennai

Dear George,

Good observations.

The decision on how often to sample and at what periods we need to take action based on sample results are business decisions a company has to make. During the initial phases, this data collection and action period may be long but over a period of time, I would expect that it would get shorter and shorter so that companies can respond quickly to changing performance levels or requirements. Would invite other's comment on the first three questions...

Let me answer your last question:

The formula in www.raosoft.com uses a normal approximation to the binomial distribution. This is a reasonable approximation if n is large and the value of p is not close to 0 or 1. This approximation is valid if n*p > 5 and n*(1-p) > 5. In the above case discussed by Fowzieh, this is satisfied.

Note that the formula quoted on the raosoft website is calculated at a power of 50%. If you desire a higher power of test (1-beta), you need to use a formula that includes power in the sample size calculations. Attached is an excel spreadsheet that you can use to calculate the sample size for different values of power.

sample_size_calculator_v1.xls

Dear SJ,

Thanks you for the Sample size calculator, would like some clarification as when to choose Mean and when to choose Proportion while calculating the sample size.

Thanks,

Nitin.

Hi George -

Please find my response to the following question-

1. Why should you be doing a monthly data entry audit (at end of month I suppose) if you are handling such high transactions?

A: The audit is done on a daily basis for the process that I handle. The avg. volume per day is 4000 out of which if I had non-production resource/ non-revenue generating resource ( QA ) do the audit, 1254 is a huge sample to check on a daily basis. However, with the assumptions that were mentioned earlier - with an overall Vol received avg for the month - 80K - the sample size was 1785 - which is much more doable and agreeable from business perspective.

Having said that my team is at 99.6% accuracy level which is pretty good at a transaction level. I was more curious about the way Sample Size is calculated and I guess as SJ mentioned its a business decision that the co. has to make.

Thanks for throwing light on the type of data distributions.

Regards,

Fowzieh

Fowzieh / Suresh

Pl clarify :

1. You decided to do daily audits now after seeing the sample size requirements of 1254, or was it the prevalent practice ?

2. Pl elaborate on this : “if I had non-production resource/ non-revenue generating resource ( QA ) do the audit” Otherwise is it a self-validation by you daily?

3. Or you directly doing the data-entry yourselves, and correcting them on-line (ie. Live – in this case its not audit ! ), or checking them at end of day (this is extra-audit)?

4. Are you now concluding that since monthly audit sample size being n=1785 (N= 80,000), you are now opting for this in lieu of daily audit n= 1254 (N= 4000)?

5. If that being so, you will be postponing your corrective actions till end of month? . I don’t know the implications of such delay from customer perspective , or cost implications, etc… Pl clarify the nature of transactions ; this may be a huge COPQ issue.

6. the sample sizes were generated assuming your response expectation accuracy is 95% (as per VK) , with 95% CI. and with ±1% Error margin. Remember customer SLA is 98% desired. He wants this to be your min.response level, in which case your margin of error should be ± 2% ( NOT ± 1% as per VK assumptions) . VK to clarify this, please. Am I missing something here? If 98% is min desired, then customer could expects ( 100% to 98%), ie 99% ±1%

7. In view of pt 6 above sample sizes will change

8. Fowzieh, you mentioned your team accuracy now is 99.6%, pretty good, then your sample sizes will come down?

9. At the end of the day , I still don’t get it why should you do a monthly audit because of your high volumes ?

10. Fowzieh : How do you search and track your Data entry errors whether on a daily or monthly basis? Do you red tag them for later corrections

11. Mr. Suresh : you mentioned the Calculator did the sampling at 50% “power”. Is this the 50% response accuracy referred above? (later changed to 95% per VK). You mentioned “If you desire a higher power of test (1-beta), you need to use a formula that includes power in the sample size calculations” . Does this imply the raosoft formula does not build-in (1-beta) power , and we have to resort to manual formula calculations?

12. Regarding the distributions, it is Hypergeometric not Binomial but ultimately the Normal approx are being used in the formula : Comments

Regards

George Varkey

Fowzieh / Suresh

Pl clarify :

1. You decided to do daily audits now after seeing the sample size requirements of 1254, or was it the prevalent practice ? --It is a prevalent practice to do a daily audit. The internal Quality check is done on a daily basis and the external audit scores are sent by the client on a weekly basis. All the errors that are identified by the Internal/external QA are sent to the users to either rebut or rectify the errors keeping the respective supervisor in loop ( real time correction )

2. Pl elaborate on this : "if I had non-production resource/ non-revenue generating resource-(NRGFTE) ( QA ) do the audit" Otherwise is it a self-validation by you daily? - The NRGFTE is a resource identified within the specific process. They are responsible to do the audit for the entire team and release the error list / accuracy scores on a daily basis to take corrrective action. It is overhead cost, would have to bear it to meet the Customer SLA and have an internal control.

3. Or you directly doing the data-entry yourselves, and correcting them on-line (ie. Live - in this case its not audit ! ), or checking them at end of day (this is extra-audit)? - No

4. Are you now concluding that since monthly audit sample size being n=1785 (N= 80,000), you are now opting for this in lieu of daily audit n= 1254 (N= 4000)? - Yes

5. If that being so, you will be postponing your corrective actions till end of month? . I don't know the implications of such delay from customer perspective , or cost implications, etc... Pl clarify the nature of transactions ; this may be a huge COPQ issue. - Based on the sample size which is 1785 for the month - that is distributed over the number of days in the month = 1785/21 and 85 becomes the target for the QA on a daily basis. So eventually the errors that are identified on a daily basis are rectified and the accuracy scores shared with the respective team members.

6. the sample sizes were generated assuming your response expectation accuracy is 95% (as per VK) , with 95% CI. and with ±1% Error margin. Remember customer SLA is 98% desired. He wants this to be your min.response level, in which case your margin of error should be ± 2% ( NOT ± 1% as per VK assumptions) . VK to clarify this, please. Am I missing something here? If 98% is min desired, then customer could expects ( 100% to 98%), ie 99% ±1%

7. In view of pt 6 above sample sizes will change

8. Fowzieh, you mentioned your team accuracy now is 99.6%, pretty good, then your sample sizes will come down? - We normally do a total of 4000 or more samples a month which I could reduce and thereby reduce my overhead cost based on the performance of the team

9. At the end of the day , I still don't get it why should you do a monthly audit because of your high volumes ? - Like I said it is not a monthly audit, it is done on a daily basis

10. Fowzieh : How do you search and track your Data entry errors whether on a daily or monthly basis? Do you red tag them for later corrections -- Searching the errors is done by the QA. Tracking the errors is on the excel spread sheet report and the erros are sent to the user for correction / rebuttal. The rebuttals are verified by the supervisor for corrective action.

11. Mr. Suresh : you mentioned the Calculator did the sampling at 50% "power". Is this the 50% response accuracy referred above? (later changed to 95% per VK). You mentioned "If you desire a higher power of test (1-beta), you need to use a formula that includes power in the sample size calculations" . Does this imply the raosoft formula does not build-in (1-beta) power , and we have to resort to manual formula calculations?

12. Regarding the distributions, it is Hypergeometric not Binomial but ultimately the Normal approx are being used in the formula : Comments

Regards

George Varkey

Dear George,

With respect to 11 & 12 above,

11. Yes, Raosoft calculations are inaccurate. They do not consider the power values when they recommend a sample size. You can use Minitab and show that if you set the power of test at 50%, you will get the same answers as recommended by Raosoft. You don't have to do the manual calculations by hand. Of course, you are welcome to use the excel spreadsheet that I provided if you don't have access to Minitab. This is not the same as 50% response accuracy. That refers to the null hypothesis Ho: p = 50% (where p stands for proportion and not the p-value).

12. If sampling is done with replacement, then it follows a binomial distribution and if sampling is done without replacement, then if follows a hypergeometric distribution. So, the question you need to ask is does the % accuracy numbers change as you sample the data. This is a moot point usually, when the population sizes are very large and both of these distributions will give you similar answers. As you have correctly pointed out, the formulas are using normal approximations anyway. I also derived my formulas that I used in the excel spreadsheet using the normal approximation.

Best Regards,

SJ

Dear Nitin,

If you are working with continuous data and based on the sample you want to infer what is happening to the population mean, you would use the mean formula. If you are working with discrete data (defects/defectives), we usually work with proportions.

Example:

If you want to compute the average height of a person in India, then you would use the mean formula.

If you want to find out what percentage of the population is greater than 6 feet, then you would use the proportion formula.

Other comments?

SJ.

Dear All :

Fowzieh / VK / Suresh Jayaram

There is something wrong in accepting these sampling conclusions : Please comment on the individual observations :

1. Fowzieh says it’s the prevalent practice to do daily internal audit + weekly external audit by client themselves. So I guess by the time the users actually correct them, we are into the 2nd/ 3rd week.

2. Having decided to do that, the reporting pattern is already “pre-defined” a priori as “daily” . Hence our Pop N = 4000 (Daily volume basis) , and

3. we SHOULD NOT reckon the monthly N = 80,000 to calculate Sample size. We therefore cannot select n = 1785/ 21 to arrive at QA’s daily audit schedule target !!

4. Audit target MUST be 1254 for the QA iff you do a constant daily N of 4000 !

5. Fowzieh, since your daily production (data entry) is fluctuating ( 4000 ± something) your Sample size itself is going to be a Variable.

- – -

6. You may need to track the (historical) daily volumes on an Attributes Control Chart based on another sampling size n0 (p chart for proportions non-conforming) , arrive at an average p value, provided your current process is stable in operations.

7. Fowzieh : As an aside, in response to your comments vide point #10 above wherein you “track” those errors, are you going any further, say following any Control charts in your operations??

8. Then you may use this average p (from Control chart ) and input this info into the www.roasoft Hypergeometric/ Normal formula to derive an average Daily sample size which could be more apt than considering the previous sample figure of n= 1254

9. In other words the sampling requirements arise on account of 2 reasons/ purposes :

(a.) sample size n0 to prepare the p-Control chart (not being done?)

(b.) sample size n to just get random records for audit purposes ONLY. (being attempted now ? )

10. This query of Sampling is not that simple as it looks. VK / Suresh , kindly whet if the above line of thinking is ok (based on Control charts) , or we may simply do all this on approximations. ( ie. upto point #4 )

Inviting comments from everyone please.

George Varkey

Dear Suresh,

Thanks for input. Appreciate that.

With respect to 11 & 12 above,

11. Yes, *Raosoft calculations are inaccurate*. They +do not+ consider the power values when they recommend a sample size. You can use Minitab and show that if you set the power of test at 50%, you will get the same answers as recommended by Raosoft. You don't have to do the manual calculations by hand. Of course, you are welcome to use the excel spreadsheet that I provided if you don't have access to Minitab. *This is not the same as 50% response accuracy.* That refers to the null hypothesis Ho: p = 50% (where p stands for proportion and not the p-value).

A. So I take it that raosoft just doesnt bother about Power. The Response accuracy referred in raosoft is our expected proportion p = 0.95 ? Can I assume that since SLA is fixed at 98% accuracy, we may input p=0.98 and redo the sample calculation?

However your statement "This is not the same as 50% response accuracy" is with ref to the Minitab or your sample calculator ? (have not tried yours yet)

"That refers to the null hypothesis Ho: p = 50%" is with ref to Raosoft calculator?

12. If sampling is done with replacement, then it follows a binomial distribution and if sampling is done without replacement, then if follows a hypergeometric distribution. So, the question you need to ask is does the % accuracy numbers change as you sample the data. This is a moot point usually, when the population sizes are very large and both of these distributions will give you similar answers. As you have correctly pointed out, the formulas are using normal approximations anyway. I also derived my formulas that I used in the excel spreadsheet using the normal approximation.

A. In the case of sampling audits for corrective actions, you are only searching for erroneous records, and you SHOULD'NT count the same error twice in the day/ monthly for reconciliation. Hence this has to be without replacement case : hence Hypergeometric.

The present case of Audit by Fowzieh for large population sizes will apply even for much smaller sizes say more than 30 too, whence the Hypergeometric will always be used for "without replacement" .

Please comment. Thanks

George Varkey

Dear George,

With respect to the first question (#11 above), yes you should use 98% if your historical process is performing at 98%.

With respect to the second question (#12 above), technically you are right, it is a hypergeometric distribution. However, when the population size is large, then it is usually convenient and recommended to work with the binomial distribution.

Ref: http://www.quantitativeskills.com/sisa/distributions/hypghlp.htm  

Best Regards,

SJ