PISA assessments are flawed

A number of prominent public figures in highly salaried positions in Northern Ireland constantly make use of OECD pronouncements to undermine our education system. This is surely damaging to inward investment which could bring much-needed employment to Northern Ireland.

Central to the OECD project is a model which purports to compute ability estimates which are independent of the test items taken by young people.

I know of no serious thinker who believes this is possible, and certainly no respected mathematician or physicist.

To offer an illustration, consider a simple numeracy question involving the even numbers.

For the OECD model to work, the individual taking the item must have “in mind” ALL the even numbers. It isn’t enough that they merely have a simple rule in mind from which these numbers might be generated. The OECD ranks will only work if there exist mental states with highly exotic properties: they must be infinite, timeless and capable of anticipating the future.

According to the OECD, to possess an ability is to be in a state which is hidden, infinite, timeless and future-anticipating.

This impossible scenario is what is needed to justify the neat list of countries in rank order. Ask yourself how many things in life can be represented by a single number. While the location of the end of your nose requires three numbers, the OECD claims to summarise the quality of a continent’s education system in a single number.

The OECD statistical model is what is known as a “reflective” model which treats these infinite states as more fundamental than the actual responses of the test-taker. The truth is the exact reverse of the OECD approach: our criteria for ascribing abilities to test-takers are derived from their test responses which happen in real time. Third-person ascriptions of ability derive from responses to test items, and these ascriptions are defeasible.

The OECD invites policy-makers to believe that their tests measure some inherent quality of the test-taker, namely, their “literacy” or “numeracy” skills.

These are modelled as inner states hidden behind behaviour.

In reality, however, information is merely being gathered on the test-taker’s ability to respond to something the OECD has fancifully called a literacy or numeracy measure. It is impossible, in principle, to demonstrate that

intrinsic states of the test-taker are being measured by PISA tests.

In effect, the OECD completely confuses two profound questions: “what is ability?” and “what can we say about ability?” The former has no answer and the second completely rules out numerical ranking.

In the last few months, British universities have rejected PISA-type assessments in higher education. No doubt these institutions want to be spared the sort of judgement which appeared on the front page of THE TIMES of 29.01.2016: “Young people in England are the most illiterate in the developed world.” It’s time to follow higher education’s lead in respect of our schools.

Dr Hugh Morrison, Queen’s University (retired)