Measures of statistical validity give us a sense of how confident you can feel in our survey results. The most common measure of statistical validity is a margin of error, which is used to create confidence intervals. These are invoked when you conduct a survey with a random sample of residents (i.e., you have a list of residents that you randomly choose from so everyone has an equal probability of being selected). There are two key parts to this: the margin of error and the level of confidence.
The margin of error is dependent on the number of responses you get to your survey; as the number of respondents increases, the margin of error goes down.
The level of confidence indicates how confident you are that the true rating, if the whole population were surveyed, would be within your margin of error.
For example, with 600 responses, the margin of error at a 95% level of confidence is plus or minus 4%. This means that, if 75% of residents rate a service as “excellent” or “good,” then if the entire community responded to a question, between 71% and 79% would rate the service as “excellent” or “good.” The 95% level of confidence means that, if you conducted this survey 100 times with different random samples, in 95 of the 100 implementations, the average rating would be within the margin of error (i.e., between 71% and 79%).
When your respondents are not randomly sampled—like when you share a link to the survey across multiple communication channels—other measures, such as credibility intervals, are used. The methods are complex, but practically speaking, the error bands are wider than a margin of error from a probability-based sample of a similar size.
You may not always need to measure the precision of your results, or, even if you do measure, you may be comfortable with less precision in some cases. Generally there are increasing costs to getting more precise results; higher-stakes decisions may warrant more investment than lower-stakes ones. In each case, you want to consider the tradeoffs of cost, timeliness and level of accuracy.
|
|