Unfortunately there is a small problem with the way that attainment by groups is to be reported by the government. It has chosen to report average attainment across quite a large group of pupils (as many as 45% of pupils are in the middle band). Because the group of pupils is very large, reported average attainment across the group tells us (1) partly how well the school is doing, but importantly (2) partly the distribution of prior attainment for the pupils in the school within this group. In this sense it simply replicates the problems of reporting ‘raw’ GCSE attainment: more affluent schools will appear to do better than more deprived schools, at least in part because their prior ability distribution is more favourable.

In our proposed performance table measures we also reported attainment by group, but we deliberately made the size of the groups small: our low attaining group scored in the 20th-30th percentile at KS2; our middle attainment group scored in the 45th to 55th percentile at KS2; and our high attainment group scored in the 70th to 80th percentile at KS2.*

The chart below illustrates how the use of very large groups of pupils in the government measures favours schools with higher prior ability pupils, relative to our measure that uses much smaller groups.** Each of the 3,000 or so tiny blue dots plots capped GCSE attainment for a group of high attaining pupils (government measure of achieving above L4 at KS2) against the average KS2 score (i.e. prior attainment) of pupils at the school. The red dots plot the same relationship for the Allen and Burgess calculation of average performance for high attaining pupils (70th to 80th percentile at KS2 pupils). The two measures are actually highly correlated (over 80%) and it isn’t informative which measure is higher or lower since they make calculations over different groups. The important difference is that on the government measure, schools with higher attaining intakes do proportionately much better than those with lower attaining intakes. On the Allen and Burgess measure this gradient is far less pronounced, suggesting that the gradient is largely due to differences in the ability profile of the high attaining groups across schools.

GCSE attainment of schools for high attaining pupils

The same relationship can be seen for the middle attaining group of pupils…

GCSE attainment of school for middle attaining pupils

…but the slope disappears for the lower attaining group of pupils, perhaps because just 17 percent of pupils are in the government’s low attaining group (though this is still a larger number than the 10 percent in the Allen and Burgess calculations).

GCSE attainment of school for low attaining pupils

While we are on the topic of reporting averages across groups, problems with confounding the composition and performance of groups is the reason why I believe free school meals (FSM) attainment gaps should not be used as a measure of success at a school. Here the problem is that the background characteristics of pupils who are not FSM will vary considerably across schools. So, calculating the group average attainment of non-FSM students at a school tells us a lot about what these non-FSM students are like on entry, and little about how well the school serves them once they arrive.

The chart below calculates the average attainment for non-FSM pupils in each school minus the average attainment for FSM pupils in each school. This is the so-called FSM attainment gap that is used as a performance metric in the new league tables. Non-FSM pupils tend to do about one grade better in each of their eight best GCSE subjects than FSM pupils in the same school, on average. In the chart the size of this gap in a school is plotted against the school’s overall FSM proportion. There is a clear relationship showing that schools that are more deprived overall tend to have a smaller FSM attainment gap.

FSM-nonFSM attainment gap across schools

I first noticed how problematic attainment gaps were in practice as a governor of a school that was struggling to produce strong academic results but was very proud that its FSM gap was zero. All the students at the school came from low income families living on a very large and universally deprived council estate. Some of the families happened to claim benefits that made them eligible for free school meals (they probably weren’t the poorest), others didn’t or couldn’t. Not surprisingly, the GCSE performance of the FSM and non-FSM pupils in this school were no different, on average, because these pupils were no different in their social or educational background. Nothing the school was doing was contributing to this supposed ‘success’.

* The ideal is to fit a line of GCSE performance against KS2 scores separately for pupils in each school and to read-off the scores at the 25th, 50th and 75th percentiles. We originally used this approach, but it is hard to communicate the method to parents and we estimated that the error caused by calculating across groups was not serious.

** The 2011 attainment data is not yet subject to public release so this analysis is carried out on an earlier year of data and is intended to be illustrative of the problem rather than a definitive relative measurement of metrics.

Reblogged from CMPO Viewpoint:

Rebecca Allen and Simon Burgess Two articles in the Times Education Supplement (TES) last Friday nicely illustrate the debate on school choice and school competition. The first reports results from the British Social Attitudes Survey (BSAS), citing research by Sonia Exley, at the LSE, showing that most respondents thought that school choice was not a priority. A familiar refrain in the school choice debate is that “all we want is a good local school”. There should be little doubt that this is indeed …

Reblogged from CMPO Viewpoint:

Rebecca Allen and Simon Burgess As children start their lessons in the 24 Free Schools opening this week, a new experiment begins in English education. The founders and staff will have been working hard for this day over many months and no doubt all will wish the pupils and staff well. There has been a lot of political passion on both sides of the debate, but what is the significance of the Free Schools experiment likely to be? It is not an experiment because of the “free” part – Free schools will …

Reblogged from CMPO Viewpoint:

Rebecca Allen and Simon Burgess From next week, officials in the Department for Education are going to be busy sifting through responses to the consultation exercise around the new School Admissions Code. Two important issues in the proposed code relate to the priority given to school staff, and to random allocation. We believe that as they currently stand, these provisions will set back the goals that the Government has set for its education policy. 1. Prioritising the children of staff Paragraph  1.33 …

Rebecca Allen and Simon Burgess   This Sunday sees the culmination of the National Teachers Awards weekend, with a televised presentation of prizes. This seems very appropriate – in terms of the impact on learning outcomes, hardly anything matters as much as having a good teacher. This is not an empty platitude – research shows that the effect size of having effective versus ineffective teachers is very large relative to most educational int … Read More

via CMPO Viewpoint

Simon Burgess and Rebecca Allen Much of the media comment on today’s National Audit Office’s (NAO) report on academies has rightly focussed on issues of governance and financial management. In this post, we dig a little deeper into some of the other claims in the report.  We are less optimistic than the NAO – less optimistic that the academy programme has had a direct impact on the improvement of deprived and very poorly performing schools; and l … Read More

via CMPO Viewpoint

Rebecca Allen and Simon Burgess This week about half a million students are starting their first term in secondary school. For many of their families, the process of choosing that school will have been very stressful. Is that process fair? The system of school admissions is a major topic of policy controversy, with a lot of debate highlighting the differences in access to high-performing schools. One leading policy proposal is to use lotteries to … Read More

via CMPO Viewpoint

Why? Free school meals eligibility is currently available to a family if they are in receipt of income support, income-based job seekers allowance, child tax credits (with an income ceiling), and a few other benefits. But analysis by Hobbs and Vignoles (2009) shows that this group of children are not the same set of children with the lowest household incomes. Only somewhere between a quarter and a half of the 16 percent of children who are eligible for FSM are in the bottom 16 percent of the distribution of household income, as illustrated in this Figure from their paper.

How can it be that so many of our very poorest families find themselves unable to claim FSM status?  Well, the very act of giving a particular set of families means-tested benefits pushes our FSM children up the income distribution, ahead of other families who are unable to claim additional income but are still very poor indeed.  This is why our FSM children are only a small section of a much larger group of socio-economically disadvantaged pupils and their families.

Suppose policy-makers wanted to target free school meals at poor families, regardless of whether they are in receipt of benefits or not.  Can we do this effectively without giving free school meals to everyone?  Well, survey data tells us that our ‘missing poor’ – the low income families who do not claim FSM – are more likely to be found in schools with FSM children and with higher levels of deprivation.  So, here I try (and largely fail) to target meals at poor families by using the recorded deprivation in their school.

I define being ‘poor’ as having less than 60% of median wage of other households with children (just under 18% in my dataset).  I am using the Longitudinal Survey of Young People in England to illustrate this, which isn’t perfect because the children are in secondary school rather than primary school, but I don’t have quick access to alternatives such as the Millennium Cohort Study.  The data crunching is quick and dirty, but I think it is enough to illustrate the point.

Our aim is to devise a rule for giving meals to children that successfully targets as high a proportion of our poor families as possible, while minimising the size of the overall transfer (this is our burden on taxpayers).  So, for example, the current rule in use only gives meals to FSM eligible pupils.  The overall size of the transfer is relatively small – 16% of all pupils – but only 47% of our poor children are successfully targeted.  Indeed, almost half the transfer goes to children who are not actually in poverty according to our measure.

The first alternative rule I try is to give free meals to all the FSM pupils, plus all pupils in schools where over 30% of the cohort are FSM eligible, regardless of whether they themselves claim benefits.  This clearly becomes a more expensive policy because 23% of pupils now get a free meal, but we do successfully reach more poor children (55%).  However, the ‘leakage’ out to the non-poor that we do not want to target rises to 57%.  If we lower the school-based criteria so that all schools with over 20% FSM rate get free meals then we can successfully reach 64% of poor pupils, but two-thirds of our subsidy now goes to families who are not poor.  This targeting of poor families is not very successful and I cannot do better using alternative school-based deprivation measures (such as the mean IDACI of pupils at the school).

It appears that devising a policy that uses published school deprivation statistics to try to locate poor families is pretty impossible.  If you want to give free meals to families in poverty without collecting further household income data, it seems impossible to avoid making large transfers to families who are not in poverty.  You could ask parents to apply and prove their income, but this might discourage uptake and is administratively costly.  The alternative route to achieving blanket coverage of families in poverty is simply to give free meals to all schools, as the pilot schemes in Durham and Newham do.  I’m not arguing that this is a good or bad use of tax receipts, but the above analysis does explain why the former Labour government decided that there are few obvious alternatives to universality as a route to giving food to children in poverty.

References

Hobbs, G. and Vignoles, A. (2009) Is children’s free school meal ‘eligibility’ a good proxy for family income?  British Educational Research Journal, (forthcoming – the journal, like so many in the social sciences, has a chronically long lag between submission and publication, but you can freely read an early draft of the paper here: http://tiny.cc/42e5d).

Background to reforms

In 1992, Sweden introduced a voucher scheme whereby privately-run (including for-profit) schools could receive public funding for each pupil they educated on the same terms as municipality schools. Like all market reforms of public services, the exact nature of the institutional structures, financing and regulation are critical to ensuring success.

However, the important context for the changes in Sweden was a backdrop of radical supply-side reforms, intended to facilitate innovation and more efficient resourcing decisions, including deregulating teacher pay and conditions, decentralising school financing and increasing school discretion of curriculum, goal setting and test regimes (see Björklund et al., 2005, for an overview of the reforms). Each reform was almost the exact opposite of the New Labour education reforms that were taking place at the same time in the UK.

Today about ten percent of lower secondary aged pupils in Sweden choose to attend free schools, with places strictly allocated on a first-come, first-served basis. There has been large regional variation in free school expansions: it has been greatest in urban, affluent and gentrifying areas and in those places with 2nd generation immigrant communities. Within these municipalities, more educated parents and 2nd generation immigrants are most likely to use the free schools, so the overall system is stratifying a little (Böhlmark and Lindahl, 2007). The largest group of free schools are for-profit providers of a general education, but special pedagogy, religious and special language/ethnic group schools are also prevalent.

Evidence of the impact on academic achievement

The ideal situation for identifying the causal impact of free school competition on pupil achievement is random assignment of the policy experiment across the 290 Swedish municipalities with no families moving across municipalities in response to the reforms. The actual Swedish situation does not reflect this ideal because free schools were not set-up at random across municipalities: they are more prevalent where the municipality is politically supportive and offers high per-pupil funding. Movement of pupils across municipality boundaries is also permissible, although not particularly common. Evaluation of the reforms is especially difficult because Sweden does not routinely collect administrative test score and demographic data on all pupils in the country, as England does. Externally marked test score data in maths, English and Swedish is available for around 30 municipalities, but for the remainder researchers are restricted to using grade levels that are not consistently standardised across the country.

The empirical papers described below adopt a variety of strategies to deal with the observation that areas with many free schools have demographic characteristics that make them systematically different. Two studies rely on cross-sectional data, hoping that the good quality of control variables (and the use of an instrument to predict free school supply) avoid confounding influences. The more recent two studies use a municipality panel to measure the extent to which changes in the share of free schools in the municipality are associated with changes in pupil test scores. This latter approach requires less arduous identification assumptions since it controls for time-invariant demographic factors in the municipality. There is still a problem establishing causality because trends in social demographics (such as an influx of immigrants) are associated with trends in free school growth, but this can be dealt with by collecting data on time-varying demographic characteristics and/or by accounting for pre-reform trends in test scores.

Böhlmark and Lindahl (2007, 2008)

The most recent paper is described first here since it uses the most robust data and methods, requiring relatively few identification assumptions. By describing this first, the relative drawbacks of the other papers’ approaches can be understood. The greatest advantages of their research come from the construction of a long-panel of data from 1988/89 to 2002/3. This gives three years of pre-intervention trend and over a decade of post-reform data to apply a difference-in-differences approach with municipality fixed effects to compare changes in areas with a large growth in free schools to changes in areas with smaller growth in free schools. The sample of data they are able to draw is also very large: a 20 percent sample from the population of pupils across all municipalities. The quality of the background control variables is good, including parental education, income, age and immigrant status, although it does not include any measures of the child’s prior attainment. They do rely on non-standardised attainment data, but have externally-marked test scores for a sample of pupils that is sufficient to confirm that biases in teacher-assessment are not correlated with the policy reform of interest.

They show a moderately positive impact of free school growth on municipality academic performance at the end of 9th grade (end of lower secondary school). This finding is convincing because it is consistently estimated across almost all subjects and model specifications; the greatest beneficiaries are found to be children from highly educated families (the impact on low educated families and immigrants is close to zero). By tracking siblings within family who differed in whether they attended municipality or free schools, they show that this superior performance of areas with private schools is due to both the greater effectiveness of private schools and also municipality schools making improvements in response to school competition, with the latter likely to be more important than the former.

However, they find that the advantages that children who were educated in areas with free schools have by age 16 do not translate into greater educational success in later life. Although there is some (weak) evidence that students in areas with many free schools are more likely to take an academic track in high school, they score no better in high school exit tests at the age of 18/19 and are no more likely to participate in higher education than those who were schooled in areas without free schools. They explore a variety of explanations for this, but conclude that the educational advantages of school competition are simply too small to persist into any long-term gains for young people.

Björklund et al (2005)

Björklund et al.’s short panel of data for municipalities between 1998 and 2001 is only able to analyse the relationship between growth in private school share in a municipality and changes in test scores over a short period of time, with correspondingly less variation in the parameter of interest (just a one percentage point change in free schooling share between these dates) and no pre-reform data to account for pre-existing social trends. They compare estimates between a sample of around 30 municipalities for which they have good quality data and all 290 municipalities where data quality is poor. Overall, they do not find a consistently positive impact of free school share on educational attainment: they identify a small positive impact on English and Swedish attainment, but a zero or even negative impact in maths. Their findings are not consistent across the sample and the population of municipalities, suggesting there may be selection problems in the municipality sample. This is a significant observation regarding data quality since the following two studies both rely on this sample of 30 municipalities.

Ahlin (2003)

Ahlin estimates the impact of the share of private schooling on grade nine test scores in a cross-section of 34 municipalities from 1997/8, hoping that the quality of her control variables are sufficient to avoid any confounding influences. This is the only study that includes the prior attainment of the pupil in grade six and further background controls, thus accounting for systematic differences in the levels of attainment across municipalities but not dealing with differences in expected rates of progress from grade six to grade nine that are due to home background factors. Her findings reverse those of Björklund et al. with quite large positive effects of private schools on overall municipality achievement in maths, but not in Swedish or English.

Sandström and Bergström (2005)

Sandström and Bergström were the first researchers to explore the impact of the free school reforms on overall academic standards in Sweden. Their finding of large positive gains to the reforms have been widely reported and is surprising given their data comes from quite early in the reform period (1997/8) before growth in the free school section became substantial. Their study relies on the largest number of identifying assumptions since they use a cross-section of only municipality schools in just 30 municipalities, using a parametric sample-selection correction to address composition changes caused by lack of data on pupils in free schools. They use a two-stage approach, with an instrument of political control predicting the municipality share of free schools. Critics argue the instrument may not meet the excludability criterion of predicting the growth of free schools but not directly determining education attainment because they are not able to control for most of the social factors in the municipality that explain household educational practices. Given this, it is hard to argue that their large positive finding should contribute to our current knowledge of the impact of the reforms.

Concluding remarks

The experience of Sweden is helpful, but necessarily limited, in the extent to which it can help us predict the impact of school reforms in England. One reason for this is that they also underwent a radical decentralisation of the education system, which would seem to be critical to promoting diversity and productivity gains through experimentation in free schools. They also have fewer reasons to be concerned that a free school system will produce greater school stratification since their lower levels of income and skill inequalities mean there is far less need for parents to choose schools based on social composition. It is also possible that their greater tradition of non-standard schooling (such as Steiner and Montessori) is leading to a greater diversity of provision than English parents would ever demand.

The econometric evidence on the impact of the reforms suggests that, so far, Swedish students do not appear to be harmed by the competition from private schools, but they have not yet transformed educational attainment in Sweden. Bunar (2009) argues that growth in free schools in the first decade was too slow to bring a great transformation due to unclear regulations and uncertainty as to whether the ruling Social Democratic party would further deteriorate the financial conditions for free schools. Also, the rising pupil population in Sweden during the 1990s meant that existing public schools did not lose students in great numbers as free schools opened and poorly performing schools did not need to close. In the past few years, overall demand for school places has fallen as the pupil population shrinks and the supply of free schools places has rapidly grown, so the prospect of a true competitive threat is now real and efficiency gains over the next decade could be larger.

References

Ahlin, Å. (2003) Does school competition matter? Effects of a large-scale school choice reform on student performance, Deparment of Economics, Uppsala University Working Paper, 2.

Björklund, A., Clark, M., Edin, P.-A., Fredriksson, P., and Krueger, A. (2005) The market comes to education in Sweden: an evaluation of Sweden’s surprising school reforms, New York: Russell Sage Foundation.

Björklund, A., Edin, P.-A., Fredriksson, P., and Krueger, A. (2004) Education, equality and efficiency – an analysis of Swedish school reforms during the 1990s. IFAU report, 1.

Böhlmark, A. and Lindahl, M. (2007) The impact of school choice on pupil achievement, segregation and costs: Swedish evidence, IZA discussion paper 2786.

Böhlmark, A. and Lindahl, M. (2008) Does school privatization improve educational achievement? Evidence from Sweden’s voucher reform, IZA discussion paper 3691.

Bunar, N. (2009) Can Multicultural Urban Schools in Sweden Survive the Freedom of Choice Policy? The Stockholm University Linnaeus Center for Integration Studies Working Paper 3.

Sandström, F. M. and Bergström, F. (2005) School vouchers in practice: Competition will not hurt you, Journal of Public Economics, 89:351–380.

Schools with voluntary-aided and foundation status appear to do well in schools league tables including contextual value added calculations, but is it possible to causally attribute this to autonomous status? This causal relationship is difficult to establish because household factors such as parenting styles and household educational backgrounds affect both the likelihood of attending an autonomous school and the chances of achieving good GCSE exam results.

There are two research papers that are suggestive that the advantages of attending faith school are overstated. Gibbons and Silva (2006) argue that children who spend their entire schooling career at voluntary-aided schools are likely to be incomparable to the average child at a secular school. However, we can learn about the effectiveness of voluntary-aided schools by comparing children who transition from faith primary to secular secondary schools with pupils who transition from secular primary to faith secondary schools. Using this approach they show that these children in faith primary schools make no more progress between Key Stages one and two than children in secular primary schools.

Allen and Vignoles look at the impact of the presence of faith secondary schools on area-wide achievement, recognising the difficulties of identifying the impact of actually attending a faith school (Allen and Vignoles, 2009). They show that areas with many faith schools are no more successful than those without faith schools. This finding implies one of two conclusions. It may be that faith secondary schools are no more effective than secular secondary schools and that the greater progress of children in these schools is attributable to unobservable pupil background characteristics. Alternatively, faith schools may be more effective, but their presence causes neighbourhood secular schools to struggle, perhaps due to pupil sorting.

It is more straightforward to make causal claims about the effectiveness of foundation schools because they mostly arose through the grant-maintained policy legislation that created a policy experiment where schools who just won their vote of parents to become grant-maintained can be compared to those schools who just lost their vote and so remain as community schools today. Damon Clark showed that the vote winning schools achieved greater improvements in the proportion of pupils gaining 5+A-C at GCSE in the 1990s (Clark, 2009). This is suggestive that autonomy could be a route to school improvement. However, these short-run impacts may not be informative of the long-run impacts of the policy, especially given the possible negative effects of losing a parental vote, euphoria effects of winning a vote, greater funding associated with the early years of the policy and pupil sorting effects.

I have replicated the approach of Clark in comparing grant-maintained vote winners and losers to look at the long-run impact of foundation status (Allen, 2010). This analysis shows that in 2007 the former grant-maintained vote winners do no better in GCSE attainment to those schools who just lost their grant-maintained vote and so are community schools today. This analysis cannot tell us why this is so, but may help us predict the likely short and long-run impacts of the Academies programme.

References

Allen, R. (2010) Does school autonomy improve educational outcomes? Judging the performance of foundation secondary schools in England, DoQSS Working Paper No. 10/02.

Allen, R. and Vignoles, A. (2009) Can school competition improve standards? The case of faith schools in England, DoQSS Working Paper No. 09/04.

Clark, D. (2009) The performance and competitive effects of school autonomy, Journal of Political Economy, 117(4)745-783.

Gibbons, S. and Silva, O. (2006) Faith schools: better schools or better pupils? CEE discussion paper, CEEDP0072.