Determining Survey Sample Size Needs
We get a common question all the time: what should my survey sample size be?
It’s a very reasonable question. You want enough people in a survey that you feel confident about the results. But, you don’t want to break the bank and pay for too many respondents.
What’s the sweet spot? It depends. Let’s look at a few scenarios below to help determine what your survey sample size should be.
Scenario One: You want a survey sample size that minimizes sampling error
First off, we need to assess why you’re doing a survey in the first place. Some people need to understand general trends within a category or market to gauge overall preferences. This could include things like product concept screens, ad and creative testing, or buyer journey research.
In contrast, you may need to hone in on very specific metric about a population. One great example is a brand awareness study. Since these studies focus on isolating a number to use for benchmarking, you need far greater accuracy. A general read is not enough.
This gets us to one key metric to use when determining a survey sample size: margin of error.
Margin of error is how confident you can be that the metric is accurate. For instance, a margin of error of 5% means that the actual number is within plus or minus 5% of the number in your survey.
When specificity matters, like with awareness studies, we need a lower margin of error. This means a larger survey sample size. When general trends matter more, we can accept more error and therefore smaller sample sizes.
We like this margin of error calculator to back into the final error numbers.
Scenario Two: You want a survey sample size that lets you read differences over time
Another common scenario that results in asking about survey sample size is the case where you want to look at longitudinal changes. That is, you are doing research work to get a benchmark number. And, you plan to measure that benchmark over time.
Brand awareness is one typical metric in this category, but so are complaint rates, satisfaction rates, and any other measure you’re looking to improve as time goes on.
In this case, we care about something new: statistically significant differences.
Statistical significance measures how confident we can be that differences we see in our data are actually differences and not the result of sampling error. When measuring differences over time really matters, we want to increase the chances of being able to read differences. This means larger sample sizes. In contrast, when we are looking for a point-in-time measurement, we can accept lower confidence and therefore lower sample sizes.
Here’s an online calculator we like to measure statistical differences.
Scenario Three: You want to look at data from sub-populations
Now, let’s look at another common situation that pops up when looking at survey sample sizes. This is where you want to gather data from specific groups within your larger sample.
For instance, let’s say you start with recruiting dog owners. However, you then want to ask a series of questions to people who own small dogs. You now need to think about having enough small dog owners in your sample population.
If this is a scenario you’re facing, then you need a bigger sample size. Essentially, you need enough people to do a broad sweeping review of the category and then also enough within your sample to look at this smaller group.
How Many People Should You Have In Your Survey
We’ve now taken you through the different scenarios we think through when considering survey sample size. That takes us to the most important question: how many people do you actually need?
For starters, we have one rule of thumb: Aim for a minimum sample size of 100 for any single population you want to explore. This is for two reasons:
- Captures Red Flags: This is a large enough sample size that if there are any unexpected issues with what you are testing, you will see that in the data.
- Shows Directional Measures: While it isn’t a large sample size, it is enough to capture general trends or skews.
From here, use your discretion based on the scenarios above. Increase your sample size until you get a margin of error that you’re comfortable with. Or, increase the survey sample size to more confidently read differences across populations.
At the end of the day, there is no hard and fast rule you must follow. Instead, use key inputs — risk aversion, learning objectives — to hone in on the right survey sample size.