Should we eat more fish or more ice-cream to boost PISA scores?

John Jerrim.

If anyone has ever read one of the international PISA reports or seen Andreas Schleicher present they will know that the OECD is rather fond of cross-national scatterplots. These illustrate the relationship between two variables measured at the country level.

Take, for instance, the chart below. This has been taken from one of Mr Schleicher’s blogposts, and illustrates the relationship between a country’s test scores and its rate of economic growth. It has been interpreted by the OECD as showing “that the quality of schooling in a country is a powerful predictor of the wealth that countries will produce in the long run”.

Source: Research by Hanushek and Woessmann, via the OECD Education and Skills Today blog

Sounds convincing, right?

The trouble is, correlation does not equal causation. And, despite the OECD’s obsession with such cross-country relationships, they can often deceive.

Should we eat more fish?

Of course, it is not only the OECD that produces such analyses. Academics do as well.

One of the more unusual examples I have found is this study by a group of epidemiologists. They look (I kid you not) at the relationship between average fish consumption and average PISA test scores.

Somehow, the study manages to find a strong positive correlation of around 0.6! It then goes on to conclude that a “linear association between fish consumption and PISA scores is likely” and that “it is plausible to assume that a higher level of fish consumption drives improvement at school level”.

As a nation of fish-and-chip lovers, this should be good news for our PISA scores…

But isn’t eating ice-cream just as important?

Getting kids to eat more fish is difficult. My little boy hates the stuff! He does, however, like ice-cream – and lots of it (who doesn’t?). If only eating ice-cream were as good for PISA scores as eating fish.

Well, apparently it is.

The chart below provides one of my all-time favourite PISA graphs. It plots the level of ice-cream consumption of a country against average PISA reading scores. There is actually quite a strong positive relationship, with around half the variation in PISA scores explained.

It also seems that the relationship is exponential (i.e. the line of best fit is curved upwards). Not only should we eat more ice-cream, but we should be eating lots of it.

The take-away message (no pun intended)

Hopefully, the point of this blogpost has become clear.

When the PISA results get released at the start of December, the international report and presentations given by the OECD are bound to include this kind of graph, along with stories about how ‘high-performing countries’ all do X, Y or Z.

Clearly, we should be treating any such interpretation of the PISA results with caution. There are likely to be hundreds, if not thousands, of reasons why some countries do well on PISA and others don’t. In reality, it is almost impossible to separate these competing reasons out.

What we do know is that overly simplistic “explanations” for the PISA results must be avoided. Organisations like the OECD have their own agenda, and it is just too easy for such groups to use the results to promote their own hobby-horses.

Other posts in this series can be found hereherehereherehereherehere and here.

Photos: fish by tkcrash123, ice cream by J Lippold via Creative Commons

Tagged with: ,
Posted in accountability and inspection, International comparisons
9 comments on “Should we eat more fish or more ice-cream to boost PISA scores?
  1. John, super post, as are all in this series!

    I am just intrigued to know that despite us eating so much fish, whether or not there is a negative correlation with chips….. so much research still to do (sigh!).

  2. “If anyone has ever read one of the international PISA reports or seen Andreas Schleicher present they will know that the OECD is rather fond of cross-national scatterplots. These illustrate the relationship between two variables measured at the country level.”

    This is true, but the really important link with national IQ is ignored. I did the statistics myself for PISA maths scores and you can find it here.

    There is a correlation of 0.89, which is very similar to that which results when the GCSE scores of schools within an LA area are plotted against mean intake Cognitive Ability Test (CATs) scores. Whereas causation looks unlikely in the case of fish & ships and ice-cream this is not the case in respect of IQ and it is indeed clear from national CATs data going back decades that CATs scores are the strongest predictor of GCSE results. Indeed when CATs scores are taken into account social and economic disadvantage effects largely disappear.

    My study has been updated with better national IQ data and can be found here.

    Surely there should be no great surprise that OECD’s plot of national economic growth against national IQs follows the same pattern as PISA scores.

    Virtually every article on my website hammers away at these links, the obvious point being that this being the case schools should be focused primarily on exploiting the proven plasticity of cognitive ability/IQ to prioritise its development over jumping the arbitrary GCSE school attainment thresholds need to drive school league tables and the marketisation ideology produced perverse incentives that are actually making our students dimmer rather than brighter.

  3. africanschoolclapham says:

    Great article thanks!
    I wrote a paper on PISA while at the IoE. I visited the offices of the OECD in Paris and had to plead and go through 3 levels of security just to go to the bookshop pick up the 20 pound PISA report.
    Also what Schleicher et al put forward from the report is a mini-sample from the research. If you read more deeply you will see a large part of the findings contradict what the OECD is promoting. So for ex the OECD promotes the publication of student results in league tables but its own research shows that countries that don’t do this have the highest scores (and generally best regarded educational systems). Hmmmm

  4. […] form of relationship. However, as Edward Tufte has famously illustrated, a a recent article from John Jerrim at University College London beautifully illustrates, correlations can be somewhat absurd. As a warning about the upcoming release of PISA scores, […]

  5. […] Other posts in this series can be found here, here, here, here, here, here, here, here, here and here. […]

  6. […] Other posts in this series can be found here,  here, here, here, here, here, here, here, here, here and here. […]

  7. […] correlación y causalidad. No hace falta ser psicómetra para adivinar que la correlación entre el consumo nacional de helados y el desempeño en PISA no es causal (ojalá…). Las ILSA son estudios que retratan el momento, no están diseñados para […]

  8. […] and causation. You don’t need to be a psychometrician to guess that the correlation between national ice cream consumption and PISA performance is not causal (if only…). ILSAs are snapshot studies, not designed to establish causal […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

UCL Institute of Education

This blog was written by academics at the UCL Institute of Education (IOE), for anyone interested in current issues in education and related social sciences.
IOE Tweets
%d bloggers like this: