7.2 Development of a monitoring and evaluation framework

On this page:

To ensure the success of the proposed Aboriginal and Torres Strait Islander higher education strategy, a robust monitoring and evaluation framework will be required. In conducting this Review, the Panel required accurate data and evidence to assess the current situation, what needs to change and what is working well for Aboriginal and Torres Strait Islander people in the higher education sector. While considerable data was available through departmental program-based reporting to monitor progress, there was not always sufficient evidence to assess the overall success or otherwise of specific programs. In some cases, there were no independent evaluations of programs for the Panel to draw on.

The importance of research and quality evidence was emphasised in Warrior’s research paper to the Review, Native higher education in the United States and Canada: a report for the Review of Higher Education Access and Outcomes for Aboriginal and Torres Straits Islanders. He noted the ‘descriptive nature’ of much of the literature on Indigenous higher education in the United States and Canada and argued that:

[w]hatever and however governments decide to intervene in Indigenous higher education, a critical component of that engagement is sustained research that tests the bases of programs (Warrior 2012, p. 23).

Universities currently provide considerable amounts of data to the government through student-based and program-based reporting; however, there does not appear to be a strategic approach to how this information is being collected and used for monitoring and evaluation of Aboriginal and Torres Strait Islander outcomes. As noted earlier, there is duplication of effort across some program reporting by universities to government. COAG-based reporting is still being developed and there is limited data currently collected at this level that relates specifically to Aboriginal and Torres Strait Islander peoples’ success in higher education.

An Aboriginal and Torres Strait Islander higher education monitoring and evaluation framework could complement the strategy and provide a more coherent and purpose-driven mechanism for data collection to feed into monitoring and assessing overall performance and success across the sector. The development of the framework would require close collaboration among universities, governments and COAG officials to reduce the risk of duplication of effort and to agree on purpose, appropriate sources and collection methods for data.

Current situation

COAG-based reporting

Almost all national-level reporting of outcomes for Aboriginal and Torres Strait Islander people is conducted within a framework originally endorsed by COAG in 2003 and subsequently revised in line with the 2007 National Indigenous Reform Agreement ( Closing the Gap). The reporting framework is a ‘living’ structure in the sense that it is subject to revision in light of both new demands for monitoring and new evidence about the impact that government policies and programs are having on outcomes for Aboriginal and Torres Strait Islander people.

Also since 2003 the Steering Committee for the Review of Government Service Provision has published a series of biennial reports, Overcoming Indigenous disadvantage: key indicators. The reports contain information on progress towards the six COAG Closing the Gap targets, together with reports on a further six headline indicators of disadvantage and seven building blocks—strategic areas for action that are seen as underpinning any sustainable improvements in outcomes for Aboriginal and Torres Strait Islander people.

The Overcoming Indigenous disadvantage reports indicate that ongoing, coordinated work on data development continues to be done by Commonwealth and state and territory government agencies in consultation with stakeholders. Much of the focus of this data development work is on improving the quality of existing measures and on addressing recognised gaps in data coverage.

In the sphere of education, most of the indicators under the National Indigenous Reform Agreement relate to the primary COAG targets of early childhood education; reading, writing and numeracy; and Year 12 attainment. Although many of the indicator variables for Closing the Gap targets are related to later participation in higher education, participation itself is captured in the headline indicator ‘Post-secondary education—participation and attainment’.

COAG also recognises that the indicators are interdependent and that ‘[f]ew of the COAG targets or headline indicators are likely to improve solely as the result of a single policy or a single agency’ (SCRGSP 2011b, p. 4.2).

Departmental reporting

Universities currently report substantial additional information on their students to the Department of Industry, Innovation, Science, Research and Tertiary Education. For each provider, the Higher Education Information Management System (HEIMS) records information including, but not limited to:136 Commonwealth Higher Education Student Support Number; level of study; course of study; basis of admission; prior study; permanent address; and whether students identified themselves to the university as being of Aboriginal or Torres Strait Islander origin.

Based on the information reported, the department can derive other data regarding Aboriginal and Torres Strait Islander participation in higher education, including enrolments, retention and completions. These can be disaggregated further, for example, for specific time periods and geographic regions.

Data is also collected on staff employed by universities; and student applications, offers and acceptances to university through the Higher Education Information Management System.137

Additional information, much of it qualitative, is available from universities’ annual Indigenous Education Statements.

What needs to change?

Government should develop an Aboriginal and Torres Strait Islander higher education monitoring and evaluation framework

While a substantial amount of high-quality data is already collected from universities on a variety of outcome measures, data is not collected with a strategic focus on the specific outcomes for Aboriginal and Torres Strait Islander people. Most data that relates to outcomes for Aboriginal and Torres Strait Islander students is collected as part of a broader data collection process in which respondents or students are simply recorded as having identified themselves as being of Aboriginal or Torres Strait Islander origin. This collection approach may mean under-reporting by Aboriginal and Torres Strait Islander students, particularly if they do not see any relevant purpose to the data collection.

As a result, it may be difficult to determine in any detail the specific experiences of higher education of Aboriginal and Torres Strait Islander students and how it may differ from that of non-Indigenous students.

Therefore, the Panel recommends that a monitoring and evaluation framework be developed for assessing the progress in achieving outcomes for Aboriginal and Torres Strait Islander people in higher education. The framework would incorporate quantifiable measurement of progress on the specific targets referred to throughout this report, that is, the parity targets for Aboriginal and Torres Strait Islander students, staff and researchers. Progress would be measured at both a sector-wide level and institution level. Qualitative information could also be collected on issues affecting achievement of these targets and on the experiences of students and staff within the sector. The actual variables collected would need to be determined through consultations with the higher education sector. It would be important to ensure that the collection of any additional data was based on consistent definitions that could be applied across all universities.

The Panel notes that the Aboriginal and Torres Strait Islander Health Performance Framework that was endorsed by the Australian Health Ministers’ Advisory Council in 2006 may provide a useful model. It includes information on health status and determinants of health for Aboriginal and Torres Strait Islander people and also on the performance of the health system.

Existing Indigenous Education Statements, which are currently used as the annual activity and financial reporting mechanism for the Indigenous Support Program, could be amended in collaboration with universities for use as a mechanism for annual reporting against performance targets. The statements currently provide the department with annual information on an institution’s:

  • objectives to improve higher education outcomes for Aboriginal and Torres Strait Islander Australians
  • policies, strategies and activities in relation to the goals of the National Aboriginal and Torres Strait Islander Education Policy
  • constraints that impact on the ability to achieve each goal
  • plans for future improvement against each goal.

The Panel also notes the need to ensure that cultural and other barriers to the effective enumeration of Aboriginal and Torres Strait Islander people are identified and addressed. For example, universities should be encouraged to:

  • align their questions about identification as Aboriginal or Torres Strait Islander with the Indigenous identification question in the national census
  • explore different approaches to eliciting self-identification information (for example, emphasising confidentiality and outlining limits on the use of the information).

The Panel suggests, in the development of the monitoring and evaluation framework for Aboriginal and Torres Strait Islander higher education outcomes, that:

  • suitable existing data collections be incorporated into the performance framework rather than duplicating these collections
  • only essential new data be collected.

Recommendation

Recommendation 35

That the Australian Government and universities work together to:

  • develop a national Aboriginal and Torres Strait Islander higher education monitoring and evaluation framework
  • develop a set of standardised words to be used by universities, based on the national census’s Indigenous identification question, when asking whether a person identifies as Aboriginal or Torres Strait Islander
  • encourage Aboriginal and Torres Strait Islander students to identify themselves as such to their university.

136 A complete list of the data items collected by HEIMS is beyond the scope of this document. HEIMS-HELP has complete information by year of data collection at heimshelp.

137 Around 120 HEIMS data elements are collected for each enrolled student, 22 for each staff member, and 48 for each applicant for admission.