Hi All

 

I need advise on a project that I am about to start. I have undergone GB training.

 

Background: I work for an NGO BPO called Samasource that provides digital work to people across the world. We mostly deal with discrete data. Tasks are provided to agents and then these tasks are reviewed by reviewers who mark them correct or incorrect. 

 

The project: Currently, the reviewers check  100% of the tasks completed by the agents. Our quality threshold is 95%, We would like to know by how much we can reduce the review % but still achieve 95% quality. 

 

Assumptions: All background work is done - reviewers are calibrated, agents are well trained, non-performers are removed. With all things set, how will I know how much review the reviewers need to do in order to achieve the 95% threshold? 

 

Regards

Surabhi

Dear Surabhi,

 

If your data is discrete, you can use below formula to arrive at right sampling for review.

 

n = [ (z² * p * q ) + ME² ] / [ ME² + z² * p * q / N ]
 

Z = 1.96 (for 95% confidence level)

p = proportion defective

q = 1-proportion defective

ME = Margin of error that you can accept in your sample

N = Population

n= Sample

 

For example,

 

Your team has processed 1000 works with 95% of accuracy as you specified.

You are willing to accept 1% margin of error in your sampling.

 

Then calculation would be,

 

n= (Square of 1.96 * 0.05*0.95+0.0001)/(0.0001+square of 1.96*0.05*0.95/1000)

 

n = 0.182576/0.000282

n= 646

 

Reviewer has to audit 646 works out of 1000 works done.

 

There are many articles available on sample calculations. I suggest you to go through those for better understanding.

 

Hope this helps.

 

Regards,

Suri

 

Hi Suri

 

Thank you for the detailed explanation. However, it is not a sample that I am looking for. The reason is that a sample will only tell me "what" the quality score is. I don't want to know just quality score.. I want to ensure it is at 95%. Remember that the reviewers are correcting tasks as they go along and hence improving quality as they go along. How much should they correct to ensure the the 95% threshold?

 

Let me explain further -

 

This is the current process we follow:

As our reviewers review the work, they are also expected to correct any mistakes. So 2 things happen when a reviewer corrects a task - 1)That task goes into the error bucket for the project. 2) It also goes into the correct tasks for client delivery

Based on the above, the system calculates an actual accuracy rate and a predictive accuracy rate. Eg, there are a total of 100 tasks. The reviewer has reviewed 10 and corrected 4 of those. So the actual accuracy rate is 60% and predictive accuracy rate is 64%. (Assuming the 10 tasks corrected by the reviewer are correct and 60% of the rest of the 90 tasks are also correct)

Now based on the two figures (actual and predictive accuracy rate), the system also calculates a confidence level (i dont know what formula it uses - Please help me here if possible)

So, we tell the reviewers that continue to review the tasks until the predictive accuracy rate and confidence level both reach 95%

 

I would like to know

 

1. Is this iterative model correct?

2. Is there any other way to see "how much" should be reviewed and corrected.

3. How is the confidence level calculated?

 

Regards

Surabhi