The pupil premium is not working (part I): do not measure attainment gaps

On Saturday 8 September 2018 I gave a talk to researchED London about the pupil premium. It was too long for my 40-minute slot, and the written version is similarly far too long for one post. So I am posting my argument in three parts [pt II is here and pt III is here]. The IOE London Blog is re-posting Part 1 and recommends following the links to the other two on rebeccaallen.co.uk for the full analysis.

Every education researcher I have met shares a desire to work out how we can support students from disadvantaged backgrounds as they navigate the education system. I wrote my PhD thesis about why school admissions help middle class families get ahead. No politician is crazy enough to do anything about that; but they have been brave enough to put their money where their mouth is, using cash to try to close the attainment gap. This series of blog posts explains why I think the pupil premium hasn’t worked and why it diverts the education system away from things that might work somewhat better. I suggest it is time to re-focus our energies on constructing classrooms that give the greatest chance of success to those most likely to fall behind.

Money, money, money…

We think about attaching money to free school meal students as a Coalition policy, but the decision to substantially increase the amount going to schools serving disadvantaged communities came during the earlier Labour Government. The charts below come from an IFS paper that shows how increases in funding were tilted towards more disadvantaged schools from 1999 onwards. The subsequent ‘pupil premium’ (currently £1,320 for primary and £935 for secondary pupils) really was just the icing on the cake.

1Funding

However, the icing on the cake turned out to have a slightly bitter taste, for it came with pretty onerous expenditure and reporting requirements:

  1. The money must be spent on pupil premium students, and not simply placed into the general expenditure bucket
  2. Schools must develop and publish a strategy for spending the money
  3. Governors and Ofsted must check that the strategy is sound and that the school tracks the progress of the pupil premium students to show they are closing the attainment gap

The pupil premium does not target our lowest income students

Using school free school meal eligibility as an element in a school funding formula is a perfectly fine idea, but translating this into a hypothecated grant attached to an actual child makes no sense. The first reason why is that free school meals eligibility does not identify the poorest children in our schools. This was well known by researchers at the time the pupil premium was introduced thanks to a paper by Hobbs and Vignoles that showed a large proportion of children eligible for free school meals (between 50% and 75%) were not in the lowest income households (see chart below from their paper). One reason why is that the very act of receiving the means-tested benefits and tax credits that in turn entitle the child to free school meals raises their household income above the ‘working poor’.

7FSMpoverty

Poverty is a poor proxy for educational and social disadvantage

Even if free school meal eligibility perfectly captured our poorest children, it would still make little sense to direct resources to these children since poverty is a poor proxy for the thing that teachers and schools care about: the educational and social disadvantage of families. Children who come from households who are time-poor and haven’t themselves experienced success at school often do need far more support to succeed at school, not least because:

  • Their household financial and time investment in their child’s education is frequently lower
  • Their child’s engagement in school and motivation could be lower
  • The child’s cognitive function might lead them to struggle (of which more in part III)

These are social, rather than income, characteristics of the family.

Pupil premium students do not have homogeneous needs

There are pupil premium students who experience difficulties with attendance and behaviour; there are pupil premium students who do not. There are non-pupil premium students who experience difficulties with attendance and behaviour; there are those who do not. Categorising students as a means of allocating resources in schools is very sensible, if done along educationally meaningful lines (e.g. the group who do not read at home with their parents; the group who cannot write fluently; the group who are frequently late to school). Categorising students as pupil premium or not is a bizarre way to make decisions about who gets access to scarce resources in schools.

Yes, there are mean average differences by pupil premium status in attendance, behaviour and attainment. However, the group means mask the extent to which pupil premium students are almost as different from each other than they are from the non-pupil premium group of students. The DfE chart below highlights this nicely.

8FSMdistribution

In his book, Factfulness, the great, late Hans Rosling implores us not to overuse this type of analysis of group mean averages to make inferences about the world. He explains that ‘gap stories’ are almost always a gross over-simplification. They encourage us to stereotype groups of people who are not as dissimilar to others as the mean average would have us believe.

Why do we like these ‘gap stories’? We like them because we humans like the pattern forming that group analysis facilitates, and having formed the gap story, we are then naturally drawn to thinking of pupil cases that conform to the stereotypes.

Your school’s gap depends on your non-PP demographic

I’ve explained how the pupil premium group in schools does not have a homogeneous background and set of needs. Students not eligible for the pupil premium are even more diverse.

When we ask schools to monitor and report their pupil premium attainment gap, the size of the gap is largely a function of the demographic make-up of the non-pupil premium students at the school. Non-pupil premium students include the children of bus drivers and bankers; it is harder to ‘close the gap’ if yours are the latter. Many schools that boast a ‘zero’ gap (as did one where I was once a governor) simply recruit all their pupils from one housing estate where all the residents are equally financial stretched and socially struggling, though some are not free school meal eligible.  Schools that serve truly diverse communities are always going to struggle on this kind of accountability metric.

Tracking whether or not ‘the gap’ has closed over time is largely meaningless, even at the national level

There are dozens of published attainment gap charts out there, all vaguely showing the same thing: the national attainment gap isn’t closing, or it isn’t closing that much. None of them are worth dwelling on too much since the difference between average FSM and non-FSM attainment is very sensitive to two things that are entirely unrelated to what students know:

  1. We regularly change the tests and other assessments that we bundle into attainment measures at age 5, 7, 11 and 16. This includes everything from excluding qualifications, changing coursework or teacher assessment mix, to rescaling the mapping of GCSE grades to numerical values. Generally speaking, changes that disproportionately benefit higher attaining students widen the gap.
  2. The group of students labelled as pupil premium at any point in time is affected by the economic cycle, by changes in benefit entitlements and by changes to the list of benefits that attract free school meals. For example, recessions tend to close the gap because they temporarily bring children onto free school meals who have parents more attached to the labour market.

It is also worth noting that FSM eligibility falls continuously from age 4 onwards as parents gradually choose (or are forced) to return to the labour market. This means comparisons of Foundation Stage Profile, Key Stage 1, KS2 and KS4 gaps aren’t interesting.

Don’t mind your own school gap

Your school’s attainment gap, whether compared with other schools, compared with your own school over time, or compared across Key Stages, cannot tell you the things you might think it can, for all the reasons listed above.

Moreover, it isn’t possible for a school to conduct the impact analysis required by DfE and Ofsted to ‘prove’ that their pupil premium strategy is working for all the usual reasons. Sample sizes in schools are usually far too small to make any meaningful inferences about the impact of expenditure, and no school ever gets to see the counterfactual (what would have happened without the money).

What’s in the other two blogs…

Part II explains how reporting requirements drive short-term, interventionist behaviour

Part III asks whether within-classroom inequalities can ever be closed

(Punchline for the nervous… No, I don’t think the pupil premium should be removed. I suggest it should be rolled into general school funding.)

Tagged with: , ,
Posted in accountability and inspection, Evidence-based policy, Leadership and management, Teachers and teaching assistants
2 comments on “The pupil premium is not working (part I): do not measure attainment gaps
  1. “Poverty is a poor proxy for educational and social disadvantage
    Even if free school meal eligibility perfectly captured our poorest children, it would still make little sense to direct resources to these children since poverty is a poor proxy for the thing that teachers and schools care about: the educational and social disadvantage of families. Children who come from households who are time-poor and haven’t themselves experienced success at school often do need far more support to succeed at school, not least because:

    Their household financial and time investment in their child’s education is frequently lower
    Their child’s engagement in school and motivation could be lower
    The child’s cognitive function might lead them to struggle (of which more in part III)”

    Becky, you are so right about this. My colleague John Mountford and I have been researching this subject, with our findings published in a number of articles that you can find at

    https://rogertitcombelearningmatters.wordpress.com/category/blogs/ and

    http://www.localschoolsnetwork.org.uk/long-reads

    We absolutely agree with you that the child’s cognitive function is important. In fact, we argue that it is the key issue, for which much of the poverty and social deprivation arguments are indeed highly misleading proxies. Here are the introductions to a few of the articles.

    North-south schools divide not supported by evidence. Posted on September 15, 2018 by rogertitcombe
    This is the title of a Guardian article which appeared on 11 September 2018. The following (in italics) are excerpts. Major study of 1.8 million pupils also challenges ministers’ claims children do better in academies and grammars.

    Social interaction combined with mixed ability teaching can produce large cognitive gains – an accidental breakthrough. Posted on August 22, 2018 by rogertitcombe
    Shortly to be published research by Professor Michael Shayer reveals an accidental breakthrough in how highly significant cognitive development in school students can be contagious through student and staff social interaction, but only where mixed ability teaching takes place.

    Potential for Success – response to Sutton Trust article by Dr Rebecca Montacute
    Posted on August 15, 2018 by rogertitcombe
    Dr Montacute’s article is linked to a Sutton Trust media release of 31 July 2018, purporting to show that high attaining disadvantaged pupils are under-performing at school. Considerable mainstream media coverage resulted, none of which was in any way critical. …

    It is the attainment gap fallacy that is damaging the life chances of FSM children in the north of England and elsewhere. Posted on June 2, 2018 by rogertitcombe
    The ‘north/south attainment gap’ claimed to exist by The Sutton Trust, The Social Mobility Foundation, the DfE, the National Schools Commissioner and virtually the entire English educational establishment is a fallacy.

    Like any country, talent is spread evenly, it’s education and opportunity that isn’t – (or not)
    Posted on May 3, 2018 by rogertitcombe

    Keep up your excellent work, it is a long overdue rebuttal of the ‘attainment gap’ fallacy promoted by the Sutton Trust and so many other organisations that ignore cognitive ability and the enormous scope for raising it in comprehensive schools for pupils of all abilities.

  2. […] a review she has carried out of the impact of pupil premium. You can get the first post on the IoE Blog here, but all three actually can be found on her own blog site […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

UCL Institute of Education

This blog is written by academics at the UCL Institute of Education.

Our blog is for anyone interested in current issues in education and related social sciences.
@IOE_London
Keep up with the latest IOE research

Enter your email address and we'll let you know when a new post is published

Join 31,428 other followers