How to interpret the data is a guide for users on the information presented on the Closing the Gap Dashboard. It summarises the key terms and provides an overview of the measurement and statistical concepts used on the Dashboard.
Click on the headings below to open the supporting material to assist with interpreting the data.
Performance measurement concepts in the National Agreement on Closing the Gap
The National Agreement on Closing the Gap (the Agreement) is built around Priority Reforms aimed at changing the way governments work with Aboriginal and Torres Strait Islander people, organisations and communities, so as to accelerate improvements in the lives of Aboriginal and Torres Strait Islander people. There are four Priority Reform areas in the Agreement.
The socioeconomic outcome areas relate to the social and/or economic areas of life identified as being important to the wellbeing of Aboriginal and Torres Strait Islander people. There are currently 17 socioeconomic outcome areas in the Agreement.
Targets are defined in the Agreement as ‘specific, measurable goal[s] that Parties are accountable to meet’. Targets focus on an ‘end point’ and are a way for us to see if the outcome wanted for a Priority Reform or socioeconomic outcome area has been achieved.
While a target is the agreed ‘end point’ or goal, the target trajectory shows us the potential pathway to get there. The trajectory begins from a baseline year or ‘start point’. It shows us the direction and speed of change needed to meet the target in future. It is not a prediction of the progress we expect each year or the actual pathway to the target that may eventuate, but it can indicate whether the Parties are ‘on track’ to meet the target, that is, whether the current direction and speed of change will allow the target to be met in the future.
Assessments of progress are presented for each of the targets where new data are available since the baseline year.
- At the national, state and territory level an assessment is made on whether the trend in the data represents no change, an improvement or a worsening since the baseline year.
- Where there is an improvement at the national level, a further assessment is done to see if this improvement was on track (or not on track) to meet the target in future (in the relevant target year, for example 2031).
These assessments of progress need to be considered with caution. They are based on trends estimated using a very limited number of data points. As more data points are included, the estimation of the trend will become more reliable, especially if there is limited variability in the data. When the assessment is based on five or more points (currently one target only), statistical techniques are used to provide an indication of reliability of these assessments; that is, they are provided with a High or Low level of confidence.
The method for deriving the assessments of progress and levels of confidence will continue to be reviewed and refined over time. Details on the current method are provided in Method for estimating the assessments of progress and their reliability.
Indicators are the concepts, experiences, or activities that are being measured, including for each of the targets. For example, for the target ‘By 2031, there is a sustained increase in number and strength of Aboriginal and Torres Strait Islander languages being spoken’, the indicator is ‘the number and strength of Aboriginal and Torres Strait Islander languages being spoken’.
In addition to indicators for targets, there are supporting indicators that:
- relate to factors that are likely to significantly affect whether a target will be met − these are called drivers – for example, the number and age profile of speakers of Aboriginal and Torres Strait Islander languages
- provide information on the experiences of Aboriginal and Torres Strait Islander people under each outcome − these are called contextual information – for example, the number of Aboriginal and Torres Strait Islander people accessing Commonwealth funded language centres to maintain and preserve languages1.
For each indicator, there are measures that show how the indicators are to be calculated (the ‘computation’ rules) and state where the data are to come from (the ‘data sources’).
Disaggregations of measures, indicators and targets recognise that the experience of Aboriginal and Torres Strait Islander people is likely to be different across groups and locations (for example males/females, or geographical areas). Disaggregations allow us to understand where improvements are being made and where greater effort is needed.
- Supporting indicators under the Priority Reform areas are referred to as ‘Indicators’ and ‘Outcome indicators’. Locate Footnote 1 above
Identifying Aboriginal and Torres Strait Islander people in data
Aboriginal and Torres Strait Islander people are usually identified in data sets through questions that invite them to self-identify as an Aboriginal and Torres Strait Islander person. For example, a person completing a survey (or form) may be asked the question:
Are you of Aboriginal and/or Torres Strait Islander origin?
There is standard wording used to ask this question across ABS collections and by many government agencies and Aboriginal and Torres Strait Islander organisations. Using this standard wording, a person can be recorded as:
Aboriginal and Torres Strait Islander
- Torres Strait Islander
- Both Aboriginal and Torres Strait Islander
- Not stated.
The data recorded for this question is generally referred as a person’s Indigenous status.
The Indigenous status of a person may be recorded as ‘not stated’ for a range of reasons, such as the person’s Indigenous status was not collected or they chose not to respond to the Indigenous status questions.
If a relatively large number of people in a dataset have a ‘not stated’ Indigenous status this can mean that Aboriginal and Torres Strait Islander people are being undercounted. However, it is often unclear how this will affect the measurement of indicators or targets. The potential effect depends on the number of people with a ‘not stated’ status and the nature of the measure (such as, does it use one dataset for both numerator and denominator or different datasets). Where possible, reporting includes information on ‘not stated’ rates so it is possible to understand how the relevant measure may be affected.
Rates and proportions
We express many of the targets, indicators and measures as rates or proportions. We do this to account for:
- changes in the size of the Aboriginal and Torres Strait Islander population over time – for example, the number of Aboriginal and Torres Strait Islander people has increased over the past decade
- differences in population size across jurisdictions, regions or groups – for example, there are more non-Indigenous people than Aboriginal and Torres Strait Islander people in most areas.
If we do not account for these changes or differences, we will not know if things are getting better or worse for Aboriginal and Torres Strait Islander people overall. For example, if we want to know if Aboriginal and Torres Strait Islander children are more likely to attend preschool compared to non-Indigenous children, we need to adjust for differences in the number of Aboriginal and Torres Strait Islander children and non-Indigenous children of preschool age and not just compare the number of children in these populations attending preschool (as there are considerably more non-Indigenous children than Aboriginal and Torres Strait Islander children in Australia).
Rates compare the subject (or the numerator) to a standard comparator (the denominator). For rates, the subject and comparator may be different types – for example, the number of qualifications per person. Several target measures are expressed as a rate, where the subject is sourced from administrative datasets (such as the number of children in preschool) and the population (the denominator) is sourced from official ABS population estimates.
Proportions are a specific type of rate that presents the subject (the numerator) as a part of the whole (the denominator). Many of the target measures that are proportions are expressed as a percentage (that is, per 100 people), but a proportion may also be expressed as a fraction, or as a ratio (such as per 100,000 people), particularly where the counts of the subject of interest are small.
Expressing data as a rate makes it easier to compare different population groups or comparing change over time. Age-standardised rates (also called age-adjusted rates) are recommended when these populations have very different age structures and the topic we are interested in varies considerably with age. For example, we know that the Aboriginal and Torres Strait Islander population has a younger age structure than the non-Indigenous population in Australia and that younger people are more likely to interact with the criminal justice system (as young adults are more likely to engage in risky behaviours than elderly people). Therefore, if we want to answer the question of whether the Aboriginal and Torres Strait Islander population is more likely to be involved with the criminal justice system than the non-Indigenous population, we need to adjust for age differences across these populations.
Age-standardised rates can remove the effect of different age structures when we are considering outcomes. They show what the rates would be if Aboriginal and Torres Strait Islander population and non-Indigenous population had the same age distribution.2
Crude rates have not been adjusted for differences in age structures across populations. Using crude rates is recommended when we are:
- comparing between narrow age ranges (for example, people aged 25–34 years)
- interested in a subject where age is not a factor that affects outcomes
- interested in the overall results, irrespective of age – for example, the proportion of the population living in adequate housing.
- The standard population against which each population is age-standardised is the total Australian Estimated Resident Population at 30 June 2001. Age-standardisation is done in accordance with the agreed principles for direct age-standardisation. See AIHW 2011, Principles on the use of direct age-standardisation in administrative data collections: for measuring the gap between Indigenous and non-Indigenous Australians, Cat. no. CSI 12, Canberra. https://www.aihw.gov.au/reports/indigenous-australians/principles-on-the-use-of-direct-age-standardisatio/contents/table-of-contents Locate Footnote 2 above
To help us compare Aboriginal and Torres Strait Islander people’s and non-Indigenous people’s outcomes, some target measures are expressed as a rate ratio or a rate difference.
The rate ratio is the Aboriginal and Torres Strait Islander rate divided by the non-Indigenous rate. The rate ratio helps us understand the extent to which Aboriginal and Torres Strait Islander people are more or less likely to have had something occur; this is often referred to as being over- or underrepresented compared to non-Indigenous people. A rate ratio:
- greater than one – indicates that Aboriginal and Torres Strait Islander people are overrepresented. A rate ratio of two indicates that Aboriginal and Torres Strait Islander representation is twice the non-Indigenous representation, that is twice as likely to have had something occur. A rate ratio of three is three times the rate … and so on.
- less than one – indicates that Aboriginal and Torres Strait Islander people are underrepresented. A rate ratio of 0.5 indicates that Aboriginal and Torres Strait Islander representation is half the non-Indigenous representation, that is half as likely to have had something occur.
- equal to one – indicates that Aboriginal and Torres Strait Islander representation is proportionate to non-Indigenous representation, that is the same.
The rate difference is defined as the Aboriginal and Torres Strait Islander rate minus the non-Indigenous rate. The rate difference is a measure of the ‘gap’ between the Aboriginal and Torres Strait Islander and the non-Indigenous rate.
The rate ratio or rate difference can be calculated for crude rates and/or age-standardised rates.
Population estimates and projections
Many of the target measures use populations in the calculation of rates (see ‘Rates and proportions’). It is therefore important to understand that these population data are not exact counts but are estimates.
Aboriginal and Torres Strait Islander population data are sourced from the ABS collection: Estimates and Projections, Aboriginal and Torres Strait Islander Australians, 2006 to 2031. The ABS uses different methods for measuring the population across years. The Aboriginal and Torres Strait Islander populations for:
- 30 June 2016 are estimates from the count of Aboriginal and Torres Strait Islander people in the 2016 Census, adjusted to consider any undercount as measured by the Census ‘Post Enumeration Survey’ (discussed further below)
- 30 June 2006 to 30 June 2015 are estimates using the information provided by the 30 June 2016 estimates
- 30 June 2017 to 30 June 2031 are predictions (or projections) of the population based on different scenarios, using demographic assumptions on future levels of fertility, paternity, migration, and life expectancy.
Non-Indigenous population data are available from the ABS for Census years (for example, 30 June 2016). The ABS does not construct official estimates of the non-Indigenous population for non-Census years. In non-Census years, non-Indigenous population counts are derived by subtracting the estimated or projected Aboriginal and Torres Strait Islander population from the estimate of the total population.3
These population data are the best available estimates. They are based on Census population counts adjusted for the undercount of people who were unable to and/or did not complete the Census (as estimated through a Post-Enumeration Survey). Adjusting for this undercount is particularly important for the Aboriginal and Torres Strait Islander population, where the estimated undercount is 17.5 per cent, substantially higher than for the non-Indigenous population (about one per cent).
However, the accuracy of population estimates tends to decrease the further away the year from the Census upon which they are based (currently the Census base is 2016).
- For the total population, the ‘first preliminary’ estimated resident population data (the first population estimate the ABS produces for the reference date) are used wherever possible and are replaced with ‘final’ population data after each Census when the final data become available. Locate Footnote 3 above
Accuracy of the data
Accuracy refers to the closeness of the estimated data value and the (unknown) true value. Assessing the accuracy of the data involves assessing the potential sources of error associated with an estimate.
Data for several of the target measures are based on information collected from a random subset of the population (these are known as samples). This subset can come from a survey, or a selection of people who are included in an administrative dataset.
Sampling error occurs because when data are collected from a sample (or subset) the results are different to those that would be seen if we could collect data from everyone in the population. The extent of the error is affected by two factors:
- the size of the sample – the larger the sample, the lower the potential sampling error.
- the variation in people’s responses – the more people that respond to the survey in a similar way, the lower the sampling error.
To help understand the possible extent of the sampling errors, results are often reported with relative standard errors (RSEs) and/or confidence intervals (CIs). Larger sampling error is associated with higher RSEs and wider CIs. Within a sample survey, usually the greater the level of disaggregation the greater the sampling error. This can mean that data that meets quality benchmarks at one level (such as the national level) may not meet those same benchmarks at a more disaggregated level (such as the regional level).
- RSEs provide a measure of sampling error, expressed as a percentage of the estimate. Estimates with a low RSE have a low sampling error. Estimates with larger RSEs (between 25 per cent and 50 per cent) have a larger sampling error and should be used with caution. Estimates with RSEs of 50 per cent or more are considered too unreliable for most purposes.
- CIs are the range where the ‘true’ result (the result we would get if we asked everyone) is very likely to be found for a given level of probability. The CIs in the Dashboard data use a 95 per cent level of probability. This means we are 95 per cent confident that the true result lies in the reported range.
CIs can be used to provide a simple test as to whether the results reported for two separate estimates are different. We can use CIs to test if there has been a ‘real’ change over time for one group of people or between different groups at a point in time. For example, we often want to know whether the rate for Aboriginal and Torres Strait Islander people for the current year is different to the rate for the baseline year, or if there is a real difference between two groups of people (such as between Aboriginal and Torres Strait Islander and non-Indigenous people). If the CIs for the rates do not overlap, then we can be confident that the rates are different.4
For the life expectancy estimates (reported under the first socioeconomic outcome area), the reported confidence intervals represent the estimates’ sensitivity to several assumptions, including sample error. For more information see: Life Tables for Aboriginal and Torres Strait Islander Australians methodology, 2015 – 2017, https://www.abs.gov.au/methodologies/life-tables-aboriginal-and-torres-strait-islander-australians-methodology
- In some scenarios where the CIs do overlap, the estimates may yet be different. For further information about sampling errors and how to test for accuracy, see the ABS National Aboriginal and Torres Strait Islander Health Survey methodology: Technical note – reliability of estimates, (https://www.abs.gov.au/methodologies/national-aboriginal-and-torres-strait-islander-health-survey-methodology/2018-19#technical-note-reliability-of-estimates) Locate Footnote 4 above
Rates derived from administrative data counts are not subject to sampling error but might be subject to natural random variation, especially for small counts. For some target measures sourced from administrative data (for example, the proportion of babies of healthy birthweight) variability bands are provided by statistical agencies (such as the AIHW) to account for this variation. Variability bands are similar to sampling CIs, in that they provide a specified range for an estimate which is very likely (95 times out of 100) to contain the ‘true’ unknown value.
Variability bands can be used to compare the results for two groups of people within one jurisdiction at a point in time (such as between Aboriginal and Torres Strait Islander and non-Indigenous people) and for people within a jurisdiction over time. Where the variability bands for two estimates do not overlap it can be concluded that there is a statistically significant difference between the two estimates. Variability bands should not be used for comparing results between jurisdictions as they do not take into account the differences in under-identification of Indigenous status between jurisdictions.
Other sources of errors can also affect the quality of data sets, and it is generally not possible to quantify or adjust for these errors. Some of these sources of error can particularly affect data on Aboriginal and Torres Strait Islander people, including the following:
- Under identification of Aboriginal and Torres Strait Islander people in data sets.
- Difficulty in collecting data from people in remote and very remote locations (where a higher proportion of Aboriginal and Torres Strait Islander people live compared to non-Indigenous people), leading to an undercount in these areas.
- Data collection forms that are not designed well for Aboriginal and Torres Strait Islander people, leading to missing or low quality data.
Compounding these issues is the possible impact of ‘response bias’ in the collection of data for Aboriginal and Torres Strait Islander people. In most data sets, it is assumed that the people who respond to data collections reflect the population as a whole. This may not be the case for Aboriginal and Torres Strait Islander people who are more likely to experience racism, or fear discrimination, which can create a barrier to them providing access to their personal information. When this occurs official datasets can be skewed (or biased), as they do not represent the full range of experiences and outcomes of Aboriginal and Torres Strait Islander people.
Although not an ‘error’, the small number of Aboriginal and Torres Strait Islander people recorded in data sets can make time series analysis difficult. Outcomes measured with small numbers can be volatile, as a change of only a few people can affect the results substantially. For results derived using small numbers, year-on-year movements in the results should be interpreted with caution. Variability bands (see above) have been provided for some indicators to provide a measure of this ‘natural’ volatility.
Data providers (such as the ABS and the AIHW) provide quality statements and/or explanatory notes for their collections to aid data analysis, and where available, links are included in the data specifications for each of the socioeconomic targets on the Dashboard.
Revisions to data reported for targets and indicators
Data reported for targets and indicators may change over time as additional knowledge and expertise leads to improvements in the data quality.
For some data changes, it is possible to revise the time series data for a target baseline and its trajectory to the end point. This is possible where the data changes relate to:
- administrative data to incorporate additional data and/or corrections to historical data (for example, where counts weren’t included or were included incorrectly and are able to be fixed for previous years data)
- population data (population estimates and projections), to incorporate information from the most recent Census which occurs every five years.5
Data collection and compilation methods can also change over time. These changes are usually made to improve the accuracy or relevance of the data being collected. In these cases, a break in time series can arise that means it is not possible to track progress for a target from the baseline year.
- For the Closing the Gap targets, the population estimates first produced by the ABS are used wherever possible, and these are replaced with the ‘final rebased’ population data from the latest Census when available. Locate Footnote 5 above