How our SAFi population forecasts performed against State Government

Forecasting the future population is a highly complex undertaking that incorporates a large number of inputs, detailed underlying assumptions, sophisticated modelling, and a huge amount of knowledge and experience. We often get asked about the accuracy of our Small Area Forecast information (SAFi) so, this blog summarises the performance of SAFi compared with the population projections produced by various state agencies.

Please note: .id’s Local Government population forecasts (forecast.id) are separate from this assessment as they involve different coverage and production methodology.

Currently, SAFi covers cover 70% of Australia’s population, providing detailed population and dwelling forecasts for New South Wales, ACT, Victoria and Western Australia.

This analysis is from .id’s Head Population Forecaster, Matthew Deacon, and assesses the performance of State Government and SAFi forecasts in terms of the numeric (number of LGAs that performed better in each state) and the percentage of LGAs in the state that performed within a tolerance level.

Each state produces projections or forecasts on a variety of geographic bases. As all states produce forecasts at a Local Government Area (LGA) level, this analysis focuses on a comparison of the total population at LGA level. It should be noted that SAFi forecasts are produced at a micro-geographic level (sometimes sub-SA1) which are designed to be aggregated to form custom catchments to meet client’s strategic planning needs.

Performance comparison

We compare all states where SAFi is currently produced (New South Wales, Victoria, and Western Australia) and use the most recently produced state projections – all of which were produced using the 2011 Census as a ‘base’. The same comparison method is used to compare SAFi forecasts with the relevant State Government forecasts or projections.

Percentage difference

The tables below present a comparison of. id’s SAFi population forecasts compared with each state’s population projections. For each state, the tables show the number and percentage of LGAs where the population forecasts came within a set percentage variation from the ABS 2016-base Estimated Resident Population (ERP).

To select an appropriate percentage threshold to the ERP to run the comparison we have made the below observations:

• Within 1% is presented to demonstrate the amount of LGAs where the forecasts performed exceedingly well.
• Within 3%’ is generally considered to be an acceptable level of quality to enable confident strategic planning.
• Within 5% is included as an indication of lower acceptable level of useable quality, ideally all forecasts would fall into this range.

For this blog, we have provided the results based on a tolerance of 3% that enables confident strategic planning. If you would like to understand more please get in contact with us.

Colour-coding within the table compares SAFi forecasts to the forecasts or projections of each state, highlighting which entity performed better, the same, or worse than the other by the number of LGAs that were forecast within the percentage threshold to the ERP.

Absolute difference

Performance is also measured in terms of how much the forecasts vary from the 2016 rebased ERP, essentially showing who came cl0ser to the ERP. Analysis is provided at an LGA level for each state overall and by its metropolitan and regional LGAs. For example, in Central Coast Council area of NSW, the 2016 ERP was 335,309. The difference between SAFi and the 2016 ERP figure is -1,536 (-0.5%) while the difference for the NSW Planning Department and the 2016 ERP is 4,241 (1.3%). SAFi shows a smaller difference and so is considered the better performing forecast in this instance.

SAFi versus New South Wales Planning Department

In general, both sets of population forecasts for NSW performed well.

Percentage difference

• Total LGAs: SAFi forecasts outperform the Planning Department with 103 LGAs (79.2%) falling within 3%, compared with the Planning Department’s 94 LGAs (72.3%).
• Metropolitan LGAs: SAFi forecasts outperform the Planning Department with 32 LGAs (91.4%) falling within 3% compared with the Planning Department’s 31 LGAs (88.6%).
• Regional LGAs: SAFi forecasts outperform the Planning Department with 71 LGAs (74.7%) falling within 3% compared with the Planning Department’s 63 LGAs(66.3%).

Absolute difference

• Total LGAs: For 71 out 130 LGAs in NSW (54.6%), SAFi forecasts outperformed NSW Planning Department projections.
• Metropolitan LGAs: In 19 out of the 35 (54.3%) SAFi forecasts had a smaller absolute variations than the NSW Planning Department LGA forecasts.
• Regional LGAs: SAFi was more accurate in 52 out of 95 LGAs (54.7%).

SAFi versus Victoria Planning Department

Overall both SAFi and the Victorian Planning Department (VIF) performed very well. VIF performed better in metropolitan LGAs, while SAFi performed better in regional LGAs and was slightly more accurate overall. The recent rate of growth in metropolitan Victoria and the age of SAFi forecasts had an impact on results. Our team is currently in the final stages of completing the latest update of SAFi for Victoria.

Percentage difference

• Total LGAs: SAFi forecasts performed better with 72.5% (58) of Victorian LGAs falling within 3%, compared with 60% (48) for VIF.
• Metropolitan LGAs: VIF tended to perform better in metropolitan areas with 26 of Melbourne’s 31 metropolitan LGAs (83.9%) falling within 3% of the rebased-2016 ERP.
• Regional LGAs: SAFi forecasts performed better in regional Victoria with 75.5% (37) of regional LGAs falling within the 3% threshold.

Absolute difference

• Total LGAs: In 56.3% of Victorian LGAs, SAFi forecasts outperformed Victoria in Future (VIF) population projections.
• Metropolitan LGAs: SAFi forecasts performed worse in metropolitan LGAs, only performing better than the VIF LGA forecasts in 11 (35.5%) of the 31 metropolitan LGAs.
• Regional LGAs: SAFi was more accurate in regional Victorian LGAs, with a smaller variation than VIF in 34 of the 49 regional LGAs (69.4%).

SAFi versus Western Australian Planning Commission (WA Tomorrow)

In all cases analysed, .id SAFi forecasts outperform WA Tomorrow forecasts. Population forecasting in WA is particularly challenging due to substantial recent fluctuations in the number of births, deaths and migration, which makes gazing into the future particularly problematic. There is also inherent volatility when forecasting for small regional populations, particularly in mining areas.

Percentage difference

• Total LGAs: SAFi forecasts were within 3% of the rebased ERP in 40.1%(55) of LGAs, compared with 27.7% (38) in WA Tomorrow.
• Metropolitan LGAs: SAFi outperformed WA Tomorrow with 56.3% (18)of metropolitan LGAs falling within 3% compared with 25% (8).
• Regional LGAs: SAFi was within 3% for 35.2% (37) of regional LGAs compared with WA Tomorrow’s 28.6% (30).

Absolute difference

• Total LGAs: In 61.3% of LGAs SAFi forecasts outperformed the medium term population forecasts (Band C) produced by the WA Planning Commission (“WA Tomorrow”).
• Metropolitan LGAs: SAFi forecasts had a smaller absolute variation from the rebased-2016 ERP than WA Tomorrow’s LGA forecasts in 81.3% of metropolitan LGAs.
• Regional LGAs: For regional LGAs, SAFi was also more accurate (55.2%).

Factors influencing the performance of population forecasts

• Input statistics. The quality and accuracy of input statistics are a major factor in how well any population forecast performs. All population forecasts are based on assumptions about how the population will change in the future, based on data and knowledge of the recent past. Forecasts are only as good as the assumptions and input data that they are based on. Forecast methodology usually involves using the most recently published Estimated Resident Population (ERP) figures from the ABS at a state level as a constraint for sub-state population forecasts. How closely forecasts are tied to these official population counts is a major factor that shapes the extent to which population forecasts can anticipate the future population accurately. You can read more about the impact of using 2011-based ERPs in these related blogs: How close did we get? Comparing .id’s Small Area Forecast information with ABS population estimates and Forecasting the future: what we learned from the 2016 Census.
• Base year and timing of updates. The start year of a forecast period, and the assumptions developed at that time are another important factor in the performance of population forecasts. Where there is a high pace of change in the components of population change (births, deaths and migration) forecasts are less likely to perform well. When assessing the performance of population forecasts in Western Australia which has recently seen a dramatic slowdown in the mining boom, for example, we see worse performance overall compared with less volatile states.
• Level of population analysis. Analysis of the total population of any geography is likely to perform better than a sub-set of the population such as a breakdown by age group or gender. However, exploring the performance of a population forecast by age structure can provide a valuable way of highlighting issues associated with the forecast assumptions such as fertility rates, mortality rates or migration profiles.
• Level of geography used for comparison. Much like the level of population analysis, forecasts for larger geographies such as states and territories generally perform far better than when drilling down into the detail of smaller geographies.

The latest 2016 Census counts provide an opportunity for us to update our population forecasts with the latest information available. The SAFi team are currently updating forecasts for Victoria, analysing and incorporating dwelling counts from the 2016 Census for small area geographies (usually SA1-based or sub-SA1 geography). Our research and analysis will be ongoing as we finalise Victoria, and move on to update New South Wales, ACT, and Western Australia.

Stay tuned for future blogs where we look into what the Census tells us about the population living in non-private dwellings, like mining camps in Western Australia.

.id is a team of demographers, population forecasters, spatial planners, urban economists, IT and data experts who use a unique combination of online information applications and consulting services to help governments and organisations understand people and places for evidence-based planning.