We have developed a powerful productivity measure called the Research and Education Efficiency Frontier (REEF) Index. The methodology behind REEF has its origins in data envelopment analysis (DEA) and is easily represented graphically using a frontier technique. It is uniquely suited to universities. We explain why and how below.
The REEF methodology and universities
All Australian universities have dual key missions of education and research. They cannot be accredited as universities unless they maintain minimum levels of each. This is true of many other university systems around the world.
However, the costs of these two missions are not separated out into independent businesses. If an organisation had only one activity or product and if the only motive were profit, then the efficient choice would always be the one that maximises profit. There would be no need to see a frontier spanning alternative outcomes. But with organisations like universities with more than one mandated activity, and more than one measure of success, using the REEF methodology enables us to see where each output is efficiently maximised in relation to the other output, and how that university compares with others.
This is tricky to understand, but when you see the graphs below of REEF in action it will become clearer.
First, think about the numerous allocations of time and money that are made independently of each other throughout a university’s academic year:
- One academic may be given a teaching load and a target for research output.
- Another might be relieved of teaching for a period but be expected to produce more research.
- A budget might fund some research-only or teaching-only staff, working alongside those expected to do both.
- A restructure might create research institutes with no expected education output.
- A decision might be made to hire more administrators, in order to relieve academics of administration and make them more productive academically.
- Or a different view could be taken, investing in automation of processes and self-service by academics, in order to reduce spending on administrators.
All of these decisions can take place simultaneously within a university, and sometimes even within a single faculty! Given this complexity, one needs a way of seeing what the ultimate outcome is and whether it is taking the university where it wants to be: which is where the frontier presentation technique underpinning REEF comes in.
The REEF method provides a way of seeing where each university stands in relation to the efficiency of all others, or a group with which it normally compares itself.
An example of its application was in an article by Keith Houghton and Stephen Parker in The Australian in August 2020 where the following historical graph about the university sector as a whole appeared:
As with all REEF method graphs, this one measures research efficiency on the vertical axis, in this case using the metric of publications per million dollars of expenditure, and education efficiency on the horizontal axis, in this case using the metric of students taught per million dollars of expenditure.
Each dot is a university and because all universities must teach as well as research, it shows where each is in both outputs. The frontier lines for the two years being compared – 2011 and 2016 – show the most productive universities in those years.
The colour coding in this case is by university grouping. For example, the Group of Eight universities (Australia’s most research-intensive institutions) are orange dots; and not surprisingly there is a Group of Eight dot at the research productivity frontier in each year (2.5 publications per million dollars in 2011 and 3.5 publications per million dollars in 2016). We chose not to identify any university specifically.
The same technique can be used for a single university, showing its progress over time. This means one can also compare the trajectory of two or more named universities. We did this in an article in July 2021, on the occasion of the vice-chancellor of The University of Technology Sydney being appointed to become the vice-chancellor of The University of New South Wales.
Here is UTS’s productivity between 2001 and 2019.
This graph shows a vertical increase in research productivity over the period, but no real net change in education productivity: the dots do not end up further to the right or left in 2019 than in 2001.
Here is UNSW’s.
This graph shows an increase in research productivity to 2016 but then a decline to 2019. There is, however, a net increase in education productivity over the whole period of 2001 to 2019, because the 2019 dot is further to the right than the 2001 dot.
This is how the two universities stood in relation to each other, to the efficiency frontier and to universities generally in 2019.
Efficiency need not be measured only in terms of outputs for dollar expenditure. For example, we can produce analyses of output per academic. We also demonstrated this in the UTS versus UNSW piece.
Our method can be applied deep within a single institution; for example, at department level, or Field of Education / Field of Research, or even at the level of individual academic. And it can be applied to contrast dollars spent on administration versus academic activity. Another application of REEF is to provide support for evidence-based workload models.
No other performance measure can convey these kinds of pictures.
Activity-based costing (ABC) purports to show what individual activities cost, but in practice there always seems to be argument over the detail and the validity of the results, particularly over the use of ‘cost drivers’.
There are also issues around ABC’s ability to validly deal with ‘joint and common costs’ that are naturally occurring in higher education. We have a separate Explainer on joint and common costs, which are prevalent in universities because, as far as we know, there are no universities where teaching and research are separated into different legal entities, thus keeping residual common overheads limited to a separate holding entity.
Even when one thinks these problems have been catered for, the implications of ABC results for management are not clear. If the reaction is to “get costs down”, this might actually be inefficient from an output perspective.
Another common measure is benchmarking. Benchmarking contrasts inputs or outputs, but doesn’t show outputs for a given level of input. Every institution has a different context: a different level of funding, a different balance of teaching and research, and a different view on spending on academic versus administrative functions. Benchmarking does not tell us which comparisons between institutions are valid and useful.
The REEF methodology, however, starts with audited financial statements and other verified public and private data. The curation process requires certain adjustments, for example we exclude some non-cash expenditure items such as investment losses and asset impairments.
The REEF methodology assumes that an Australian university must be performing at threshold levels of quality in education or research. Otherwise, the regulatory body TEQSA would be intervening, and alarm bells would be ringing from quality data such as in the Government’s QILT data, or the Australian Research Council’s ERA assessments, and so on.
Whatever it is a management team might tell itself about what it is doing, and how well, at the end of the day, research output and students educated can always be calculated, validated and divided into the expenditure totals audited by an Auditor-General or by staff numbers as reported to Government. Frontier analysis allows us to see the outcomes comprehensively, in one measure and clearly.
We argue that frontier analysis, as shown in the REEF methodology and its numerous possible variants, should be the primary level of analysis, telling the organisation how it is tracking in the relative productivity of its education and research efforts.
If the organisation is unhappy with its performance, as against others, or as against its earlier self or within specific parts of the organisation, it could then resort to ABC or benchmarking to dig deeper, although our frontier analysis also generates important insights at granular levels.
A public university with the mandatory joint mission of education and research does not have a set of shareholders. It does not have the pressure of an AGM or a share price to tell it where it should be putting its efforts. And because its two missions of teaching and research involve many “joint and common costs”, it is often hard to see the end result just by looking at long lists of KPIs. Our use of frontier analysis in REEF is uniquely suited to universities and tells all stakeholders – governments, governing bodies, management teams and the public – how efficient each institution is at delivering its important, complex dual mission according to the choices it has made.
Numerous examples of the REEF methodology in action can be found on this website and elsewhere. We have a 13 minute YouTube video and various blog posts, You can also watch a 23 minute slideshow presentation with voice-over given to the Australian Higher Education Industry Association in May 2021.