Hello,

I analyze monthly transactional CSAT responses from customers. These responses are on a scale of 1 to 5 (Discrete). The company's target is to maintain 80% of satisfactory responses and all reponses of 4 or 5 are classified as satisfactory responses.

There are about 25000 such transactions that happen every month. A survey is sent out to each of these customers. We receive about 230 responses everymonth, which is less than 1% of the total transaction volume.

How do I determine the sample size necessary for me to make a statistically significant statement about the company's performance on the transactional CSAT front?

I did try and use 1-p power and sample size determination test but I am really not sure, how do determine the sample size, which formula/test to use and why?

Kindly help.

Hi Shiva,

In my opinion taking sample size on the sample of 25000 would not be appropriate you good results.  In case of a C-SAT survey, there is a probability that the customers respond only 5-10%.

As of now the number of sample you are reciving is mererly 200.  It is not good to predict any thing on such small sample.  PLease shre the data and trends for last three monhts then only it will be feasible to analyse it.

Hi Shiva,

        The formula for sample size for estimating the population proportion is :-

n = (z^2)*p*(1-p)/(E^2), where E = Margin of error. 

       Alternatively, you can also use the sample size calculator uploaded by SJ sir in the file cabinet herein, for your perusal.

Thanks & rgds.

Hi again Shiva,

                Per the information given by you, the desired sample size for a population of 25000 comes to 1168 which can be rounded off to 1200.

      The following assumptions have been made in arriving at the above.

1) Hypothesis proportion is kept at 50%, as not known.

2) Margin of error is kept at 4%, as not known.

3) Confidence level is kept at 95%, as normally expected.

3) Power of test is kept at 80%, per your company's target.

4) Alternate Hypothesis is kept as ' not equal to '.  

    You may please cross check the same at your end. The sample size would certainly vary for different assumptions that you make. 

Thanks & rgds.

Dear Shiva Kumar,

I agree with Nirankar Trivedi that you will not be able to predict customer satisfaction scores with such a low response rate (1%). You will only hear the opinion of people who are totally satisfied or totally dissatisfied. You will not hear the opinion of those in the middle.

If you truly want to understand your true CSAT score, then you will have to achieve a response rate of 50% or greater. Of course, you don't need to poll all 25,000 of your transactions (you can use random sampling and determine the minimum number of people you need to talk to based on sample size calculations).

You can continue to do what you are doing and track CSAT trends over time with a 1% response rate (but you will not be able to compare it with 80% requirement).

SJ

Hello Suresh Sir, Manian and Nirankar,

Thanks for the information. I still have a question though. currently thourgh the response rate is less than 1%, 80% of the respondents are 'Satisfied'.

So, when I use the formula for determining the Margin of error with the below data, I get a response of 5.13%. (I have used the sample size calculator available as freeware on the net for random samples)

Sample Size: 231

Population: 25000

% of respondents giving one specific response (satisfied): 80%

Confidence Level: 95%

From this why should we not infer, "With a confidence level of 95%, we can conclude that 75% to 85% of the population (25000 users) are satisfied with the service provided".

I am not sure, where am I lost in undestanding how to interpret this response, kindly help.

Manian: Can you kindly confirm which test did you use to arrive at 1200?

Hi Shiva,

         I used the sample size calculator (for proportions) uploaded by SJ sir in the file cabinet herein with the assumptions in my previous response to arrive at the figure of 1200. It's a valid statistical sample size for a population of 25000 for further statistical analysis (any job), irrespective of the % of responses that you get from customers for assessing CSAT.

        Per your contention & question, as rightly mentioned above by Nirankar & SJ sir , " it will be difficult to predict customer satisfaction scores with a response rate as low as 1% ". As further mentioned by SJ sir, using random sampling for the sample size calculation, best representing the desired/sought population charateristic (CSAT) would be appropriate.

       Also, you need to see, if 1% of the customers giving their feedback on CSAT everymonth are repeat (same) customers or different. Also different customers belonging to / having same demographics (eg same tastes, income, area of residence, etc) are likely/tend to opine in a similar way. If its possible to check on these for a few customers (as no of respondents is very less) you will have a better idea if you are having the opinion of a population that is spread diversely without chance of repetition (same customers or different customers falling under some common characteristics / features) in subsequent feedbacks to give further impetus to the inference purported by you above.

Thanks & rgds.

     

Hi Mani,

Using the below data in the same sample size calculator from the file cabinet, I obtained the sample size of 232.

Population: 25000

Hypothesized proportion: 80% (as I have already obtained responses)

Margin of error: 5%

Confidence level: 95%

Power of test: 50%

Alternate Hypotheses: Not Equal to

The required sample size is 232.

So my inference comes from working backwards from here:- I have surveyed 231 users and have received one specific response from 80% of respondents.

As you pointed out, one thing I need to ensure is that the responding sample is random and not biased. So, assuming randomness in the sample, the above inference should hold good?

Suresh sir, any comments?

Hi once again,

              Going by the assumption of randomness in the responding sample at present, your conclusion over the inference in the previous response may hold good, provided consistency in the desired (80% satisfactory) response has been seen over a reasonable period of time (again with randomness in responding sample every time in the past), say last 6-8 months (point on assessment of past data also raised by Nirankar, before), based on which extrapolation (taking a trend forward & assuming that the trend will continue) can be done, as the response rate is very low (less than 1%). 

          I would like to hear from others on this & more.

Thanks & rgds. 

PS: Extrapolations need to be treated with caution. There may be some element of truth in them but they are unlikely to work out as claimed. Very often, counter forces come in which oppose the trend.