Method for estimating the assessments of progress and their reliability

The method will continue to be reviewed and refined over time.

Overview of key steps and statistical techniques

The method has four key steps, with step 3 currently relevant at the national level only.

  • Step 1: Assess if the trend in the data represents a ‘no change’ or ‘change’ scenario (that is, whether there is change or not from the baseline year to the current year)
  • Step 2: If change is identified, assess if the change is an ‘improvement’ or a ‘worsening’
  • Step 3: If the change is an improvement at the national level, assess if the improvement is sufficient to ensure the target is ‘on track’ or ‘not on track’ to be met in the future target year
  • Step 4: Assess the reliability of (that is, our confidence in) the derived assessments of progress.

Within these steps three key statistical techniques are used:

  1. Regression analysis, with ordinary least squares regression used. This analysis simply calculates a line that will best fit the data and then describes the line using an equation. For example, the equation for a straight-line is:

    𝑥time = α + β time

    where:

    • 𝑥time and time are data-pairs at baseline, Time 1, Time 2 and so on. 𝑥time is the actual data for the indicator and time is the predictor variable (e.g. year)
    • α and β are estimated using regression analysis: α is the intercept and β is the ‘slope’ or the annual change in the indicator per year.
  2. Likelihood ratio test comparing nested regression models to determine which model best fits the data (linear models only have been used to date) and assess if the trend in the data represents a ‘no change’ versus a ‘change’ scenario.
  3. Confidence intervals around the regression line and significance testing of the likelihood ratio test to understand the reliability of the assessments (once five or more datapoints are available).

Steps for assessing progress

Step 1: Assess if the data reflects a ‘no change’ or ‘change’ scenario.

A likelihood ratio (LR) test is used to test the ‘no change’ scenario comparing nested regression models1 in two stages:

  1. Perform the following regression models:
    • flat line that represents the ‘no change’ scenario. This is an intercept-only model that restricts the parameter β to be zero.
    • full model that best fits the data:
      • for this Dashboard update (31 March 2022): the linear model shown above (i.e. intercept model nested with a fitted model that includes the time parameter) is the default for all targets
      • for future Dashboard updates: for less than five data points, the linear model will be the default, but for five or more data points the assumption of whether a linear regression model is the best fit will be tested against alternative non-linear models (quadratic, exponential). The alternative models tested will be based on how the data appears when plotted.
  2. Once a full model is chosen, a ‘likelihood ratio test’ is performed to determine whether the flat line or the full model is the best fit for the data. The likelihood ratio test provides a statistical basis to reject the null hypothesis H0 in favour of the alternative hypothesis Ha :
    • H0 : The full model (linear, quadratic or exponential) and the nested (intercept-only) model fit the data equally well.
    • Ha : The full model fits the data significantly better

    The LR test statistic is expressed as a difference between the log-likelihoods of the two models:

    LR (chi2(k − 1)) = 2(L1 − L0)

    where:

    • L1 is the log-likelihood value for the full model.
    • L0 is the log-likelihood value for the intercept-only model.

    The (LR) test has a chi2(k-1) distribution, where (k-1) is the number of parameters estimated (other than the intercept).

    The p-value of the test is used to reject the null hypothesis and conclude that the full model better fits the data. If the p-value of the likelihood ratio test is:

    • … greater than 0.5, the assessment is no change
    • … less than or equal to 0.5 the assessment is change.

Step 2: If change is identified, assess if the change is an improvement or a worsening

Once ‘no change’ is ruled out, the values on the regression line for the baseline year to the current or target year are compared. A change is identified as an improvement or worsening using the conditions in table 1.

Table 1 Conditions for improvement or worsening, state and territory and national
State and territory assessment On the regression line, the difference between the most recent year and the baseline year is
Improvement Positive (for indicators that are expected to increase over time, such as life expectancy) and negative (for indicators that should decrease over time such as incarceration rates)
Worsening Negative (for indicators that are expected to increase over time, such as life expectancy) and positive (for indicators that should decrease over time such as incarceration rates)
National assessment On the regression line, the difference between the estimate for the target year and the baseline year is
Improvement Positive (for indicators that are expected to increase over time, such as life expectancy) and negative (for indicators that should decrease over time such as incarceration rates)
Worsening Negative (for indicators that are expected to increase over time, such as life expectancy) and positive (for indicators that should decrease over time such as incarceration rates)

Step 3: If the change is an improvement at the national level, assess if the improvement is sufficient to ensure the target is ‘on track’ or ‘not on track’ to be met in the future target year

The regression trend line is projected out to the target year and compared with the value of the target in the target year, according to the conditions in table 2.

Table 2 Conditions for improvement and ‘on track’ or ‘not on track’, national

Assessment

At the target year, the regression line estimate is

Good improvement and target is on track to be met

The target or higher (for indicators that are expected to increase over time, such as life expectancy) and the target or lower (for indicators that should decrease over time such as incarceration rates)

Improvement but target is not on track to be met

Below the target (for indicators that are expected to increase over time, such as life expectancy) above the target (for indicators that should decrease over time such as incarceration rates)

Step 4: Assess the reliability of (that is, our confidence in) the derived assessments of progress.

Step 4A: For targets, with less than five data points note that the assessments of progress should be used with caution

The following note will be added when assessments of progress are provided.

  • Note: These assessments of progress should be used with caution as they are based on a limited number of data points.

Step 4B: For targets with five or more data points2, the level of confidence in the assessment will be provided

For the state and territory assessments, the level of confidence (High or Low) is based on table 3 conditions, using the likelihood ratio test from step 1 (point 2) that compares the likelihood of the full model to the no change model. These conditions will also be used for determining the level of confidence for the national assessment for no change or worsening:

Table 3 Conditions for assigning confidence levels (once no change ruled out) – State and territory assessments and national worsening

Condition based on p-value of LR test

Assessment

Confidence

p-value >= 0.95

No change

High

0.5 < p-value < 0.95

No change

Low

0.05 < p-value <= 0.5

Improvement/ Worsening

Low

p-value <= 0.05

Improvement/ Worsening

High

Confidence intervals around the regression trend line estimates are provided at the state, territory and national levels. However, these confidence intervals are only used for the assessments of progress at the national level to assign a level of confidence for assessments that are an improvement (on track and not on track). The conditions for assigning a level of confidence are in table 4.

Table 4 Conditions for assigning confidence in improvement and ‘on track’ or ‘not on track’, nationala

Conditions apply to indicators where there is improvement

Assessment

Condition for assessment (as per table 2)

Confidence level

Condition for confidence level: confidence intervals around the regression line at the target year

Good improvement and target is on track to be met

The regression line estimate is the target value or

  • higher (for indicators that are expected to increase over time)
  • lower (for indicators that should decrease over time such as incarceration rates)

High

Do not overlap with the target value

Low

Overlap with the target value

Improvement but target is not on track to be met

The regression line estimate is

  • below the target (for indicators that are expected to increase over time)
  • above the target (for indicators that should decrease over time such as incarceration rates)

High

Do not overlap with the target value or the baseline year value of the regression

Low

Overlap with the target value and/or overlap the baseline year value of the regression

a All conditions are based on comparisons made at the target year.


Footnotes

  1. Nested models are those where the full model can be transformed into the simpler model by imposing constraints on the parameters of the full model. Locate Footnote 1 above
  2. To assign a level of confidence five valid data points are needed. This means for non-annual infrequent data sets levels of confidence may not be able to be assigned even up to the final target year. For the purposes of statistical regression, valid data points are numeric only, with ‘not applicable’, ‘not available’, or ‘not published’ considered invalid and are treated as missing values. Locate Footnote 2 above

The Productivity Commission acknowledges the Traditional Owners of Country throughout Australia and their continuing connection to land, waters and community. We pay our respects to Aboriginal and Torres Strait Islander Cultures, Country and Elders past and present.

Aboriginal and Torres Strait Islander people should be aware that this website may contain images, voices or names of people who have passed away.