A question I’m always asked by our local government clients is ‘how many people do we need to survey for our results to be representative?’ While survey method plays a big role in answering this question, sample size is also an important consideration. In this short post we talk through some of the things that local government professionals should consider when contemplating a survey sample size.
Establishing a reasonable sample size
When conducting a survey, asking a set of questions to the entire population of interest (all residents from your LGA) is impractical, unfeasible and really not required. Probability theory establishes that provided that a population of interest (e.g. a total LGA population) is greater than 1,000 residents, the resultant margin of error will not be affected substantially. While that’s all good and well, the question still remains: well, how many people do I need to survey?
When designing the sample size for our local government clients who participate in Living in Place, we find that asking two key questions can be helpful in reaching a resolution:
- What local areas within your LGA do you want to represent discretely?
- How much potential error are you willing to accept when representing the views of these residents?
Let’s answer those questions using the example of our Living in Place project we delivered for the Sunshine Coast Regional Council in April 2021.
What local areas do you want to represent?
While LGAs are some of the smallest geographies used to inform government decision making and service delivery, we know that they can be quite diverse spatially in and of themselves. Taking this into consideration, we firmly believe that building your target sample size from local areas up (making sure you collect the participants’ suburb / town in the survey to provide maximum flexibility when analysing and reporting) is a better approach than aiming for an arbitrary overall LGA number and then crossing your fingers that you’ll be able to analyse the results at meaningful sub-geographies once the survey has been completed.
In example, while we ended up surveying a total of 1,628 residents for the Living on the Sunshine Coast project, our main focus was on being able to credibly represent their community’s views across six discrete local areas – aligned with Council’s planning areas – with a defendable level of confidence.
How much potential error are you willing to tolerate?
Our ability to say that survey results reflect the views of the broader population is informed by sample size and, while that rule is relatively objective and formula led, choosing when to report on sample sizes to represent the views of your community is subjective, requires judgement and operates on a continuum from representative (larger samples) to wildly misleading (smaller samples). While there are some broad theories around minimum sample sizes, and associated privacy considerations, the critical thing is to understand the relationship between sample size and error and then being really transparent in your reporting about the sample size benefits and limitations in absolute terms, but also in terms of how well it reflects the demography of the LGA as well. This calculator is a really good resource to help you understand the relationship between sample size and confidence.
To understand this calculator in a relevant context, one of the six planning areas we reported on from the Living on the Sunshine Coast project is home to more than 80,000 people. We surveyed n=394 people to represent the views of these residents and, as such, our sample returned an error margin of +/-4.9%. at a 95% confidence level. This means that when 75% of these residents selected “access to the natural environment” as one of the top five things that makes somewhere a good place to live, we were able to say with high conviction (95% confidence) that between 70% and 80% of all residents from that planning area shared that view. Had we only surveyed 50 residents from this planning area the error margin would have been +/-14% and, in our judgement, not large enough to truly represent the views of that community.
In summary, while there are a host of considerations when determining an appropriate sample size, and judgement required when reporting on survey results, we hope this little post provides some food for thought around the things that local government professionals need to think about when making this decision.
Links to more information…
Visit our dedicated website to learn more about how .id is using survey research to help councils make more resident centric decisions. We’ve also written a blog discussing our deeper thinking the need for Living in Place and a series of articles exploring the top five things that Australians believe make somewhere a good place to live. Feel free to book a meeting if you’re keen to learn more.
