On educational performance measures

I have a new paper:

IZA DP No. 5897

Alastair Muriel, Jeffrey A. Smith:

On Educational Performance Measures

(published in: Fiscal Studies, 2011, 32(2), 187-206)

Abstract:

Quantitative school performance measures (QPMs) are playing an ever larger role in education systems on both sides of the Atlantic. In this paper we outline the rationale for the use of such measures in education, review the literature relating to several important problems associated with their use, and argue that they nonetheless have a positive role to play in improving the educational quality. We delineate several institutional reforms which would help schools to respond "positively" to QPMs, emphasizing the importance of agents' flexibility to change the way they work, and the importance of a sound knowledge base regarding "what works" in raising attainment. We suggest that the present institutional setups in both England and the US too often hold schools accountable for outcomes over which they have little control – but that such problems are far from insurmountable.

The working paper version (not gated) is here, and the published version (gated) is here. They are the same other than minor changes at the proof stage.

The paper tries to steer a middle course on the question of quantitative educational performance measures, arguing that they have some value, and so should not be simply abolished, but that they should not be the whole, or even the majority, of the institutional system that seeks to improve and maintain school quality. Moreover, there is much to complain about in regard to the current design of such systems, particularly the system put in place by the No Child Left Behind (NCLB) Act in the US. The performance measures in the US job training system have problems at well, as I (again with co-authors) have written about elsewhere.

This paper is based on a presentation I gave at a conference-within-a-conference at Oxford in the summer of 2010. It is unusual for me in that the nice folks at the Institute for Fiscal Studies provided me with a co-author to help transform the talk into the paper. My co-author Alistair Muriel, whom I first met at the conference, turned out to be the perfect partner, which is to say that he writes well, works hard, and put up with the oddities of my work style. The paper is also notable for the speed of publication; due to a few days lag by me in submitting the final version to IZA, and some backlog at IZA in announcing their new discussion papers, the "working" paper actually came out after the published journal version in Fiscal Studies.

I went with IZA rather than NBER (the NBER prohibits circulation in multiple working paper series, though that rule is widely ignored by people other than me) because of the European content to the paper.

Another "no effect" NYC conditional cash transfer

David McKenzie summarizes the results of the Rand Corporation evaluation and justly celebrates one small victory for evidence over enthusiasm.

Back to school in Charlotte-Mecklenberg

Can there be any greater bummer than finding out that you're not really done with high school, even after you've been through the ceremony and gotten your diploma?

And, really, how hard is this to get right? At least the principal had the good manners to resign.

Why I love the What Works Clearinghouse

From my inbox this morning:

The What Works Clearinghouse (WWC) has released an intervention report this week that reviews the research on Great Books.

Great Books is a program that aims to improve the reading, writing, and critical thinking skills of students in kindergarten through high school. The program is implemented as a core or complementary curriculum and is based on the Shared Inquiry™ method of learning. The program includes both oral and written activities designed to help students think and talk about the multiple meanings of texts. Great Books reading selections are collections of traditional and modern literature. This report focuses on Great Books programs for reading in grade 4 and higher. The WWC identified 36 studies of Great Books for adolescent learners that were published or released between 1989 and 2010. Five studies are within the scope of the Adolescent Literacy review protocol but do not meet WWC evidence standards. Thirty-one studies are outside the scope of the Adolescent Literacy review protocol. No studies of Great Books that fall within the scope of the Adolescent Literacy review protocol meet WWC evidence standards; meaning that, at this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of Great Books on adolescent learners. Read the full report now at [link].

I like this on several levels. First, and most important, it seriously engages and summarizes the available evidence on a particular educational intervention. The summary is based on precise substantive criteria. Second, it highlights how much time, energy and money is wasted in the education literature on studies that do not meet even very basic standards of evidence. Third, it helps to illustrate how educational interventions can spread despite the lack of any serious empirical demonstration of their effectiveness.

The internet, education and bubbles

This piece on education and the internet has something to it, despite being a bit overly triumphal, even after taking account of the caveats at the end.

First, the overstatement. The internet is, in relation to this topic, not all that different from a library. Before, you could get books in the library, now you can get them on the internet. I include lectures here, because a lecture without interaction is really not very different from a book. Some people prefer to listen rather than read, so having both is good for them, but I do not see the great big leap forward here. Both are unidirectional means for transferring information.

Second, the good bits. What I think is worth thinking about is what aspects of the university experience (and I am going to focus on that here) are and are not easily duplicated over the internet. Lecturing is easily duplicated and I think that we will come to see some faculty lectures, particularly for introductory courses that are heavily textbook based and have fairly standardized material - think of introductory economics - replaced with recorded lectures given by the textbook author. This would mean that students would hear from Greg Mankiw himself rather than a live faculty member. Given the wide variation in the quality of lecturers, I think it makes good sense to have more specialization than we do, so that everyone benefits from the best lecturers. This would not mean the end of local faculty, but they would then devote all of their time to the aspects of the course other than unidirectional information transfer. This means, for example, interactive sessions in which students can ask questions and discuss issues raised in the text and the recorded lecture. In some contexts, it may also mean grading, though for many introductory courses multiple choice questions with automated grading are already used.

Many aspects of a university education besides just problem sessions cannot easily be put on the internet. First, universities select peers for the students. Student learn a lot from their peers, not only in group projects but in informal study sessions and in the broader social project of "finding oneself" that takes place during the university years. Universities also, essentially, act as dating services via their selection procedures, and having a highly selected pool of potential partners to choose from is a non-trivial part of the value of attending a top school.

Second, universities can afford to purchase the physical capital required to effectively learn many disciplines such as chemistry or medicine, in part because they can spread the cost over many students and via the joint production of teaching, research and health care. There is no substitute for actually participating in research if one wants to learn how to do research.

Third, universities can provide direct interaction with faculty. Students, particularly undergraduates, make too little use of this aspect of university. For gradual students it is absolutely critical. In learning to give seminars, you need to give seminars to audiences that understand the data and methods being used and the background literature. Similarly, to learn how to write papers you need not just readers, but expert readers who, ideally, have produced their own papers and can give informed feedback.

So, no revolution yet, but I think still some internet-based change is coming down the pike in the teaching part of academia.

Addendum: Peter Thiel simply does not know what he is talking about in terms of the numbers. Perhaps he should read the literature. It's on the internet. The issue with the numbers is more the thoughtless extrapolation of positive (and broadly compelling) estimates of the impact of "treatment on the treated" to the presently untreated. Also, the numbers are about averages, and there are surely individuals for whom higher education does not have a big financial payoff (and maybe even not a big utility payoff, which is the relevant metric for decision-making). There is likely progress to be made in figuring out more effective ways to identify such individuals ex ante.

Grading grades

Graded children – evidence of longrun consequences of school grades from a nationwide reform

By Anna Sjogren, IFAU

Abstract of Working paper 2010:7

Swedish elementary school children stopped receiving written end of year report cards following a grading reform in 1982. Gradual implementation of the reform creates an opportunity to investigate the effects of being graded on adult educational attainments and earnings for children in the cohorts born 1954–1974, using a differ­ence-in-differences strategy. Accounting for municipal time trends and tracing out reform dynamics, there is some evidence that being graded increases girls’ years of schooling, but has no significant average effect on boys. Analysis of effects by fam­ily background suggests that getting grades increases the probability of high school graduation for boys and girls with compulsory school educated parents. Sons of uni­versity graduates, however, earn less and are less likely to get a university degree if they were graded in elementary school.

Keywords: school policy, grades, educational attainment, adult earnings, family background, difference-in-differences.

JEL-codes: I21, I28, J13, J24

I have been meaning to highlight this paper on the blog for some time. I read it last fall in preparation for a trip to Sweden. It presents some surprising and compelling evidence on the effects of having grades in primary school on later outcomes.

The paper reminded me of a talk that Dan Hamermesh gave over lunch at the Society of Labor Economists meetings a few years ago when they were in Austin, Texas. The theme of the talk was (I paraphrase here) it is good that other countries exist because they will try crazy policies that we [i.e. Americans] would never try, and those crazy policies provide very useful information about policy effects. I think Dan is largely correct about this, but as I recall my friends from Canada that I was sitting next to at the lunch were less enthused about this line of reasoning.

Mandatory school lunches

Well this is pretty horrifying. Some Chicago schools are prohibiting kids from bringing lunch from home. I was very fortunate in that I ate only 10 school lunches in my entire primary and secondary career, all on days when my parents were off on vacation and I was staying with friends. I still remember the awfulness of the food. I also still remember how many kids threw out all or a good part of their school lunch.

On the plus side, I can think of few more promising ways to create a cohort of little libertarians.

Miss America does ED

U.S. Secretary of Education Arne Duncan gets valuable insights from noted education researcher Miss America.

Hat tip: Diane Whitmore Schanzenbach on Facebook

Cheating in the DC schools ...

... and it is not by the students. This is not good for Michele Rhee and not good for DC.

I wonder if there are any other institutions in DC that might have inspired the teachers with the idea that in the face of a major problem, the best solution is to make up some performance measures, hand out bonuses, and move on?

Oddly absent from the article is the local teacher union, which is presumably eager to see those teachers who acted unprofessionally sacked both "for the children" and for its own reputation. Cough, cough.

Oh, and it is too bad that the Obama administration decided to shut down the evidence-based DC voucher program. That must have been "for the children" too. Cough, cough, cough, cough.

Hat tip: Diane Whitmore Schanzenbach on Facebook

Laundering studies

A nice piece from EduWonk on how studies get "laundered" when some credulous / differentially not competent person at NYT or WaPo is unable to sort out, and ignore (or even, perish the thought, call out) bad studies, which then become reputable because they have been discussed in a major newspaper.

The discussion of achieving "mixed evidence" by vote counting in which weak studies receive the same weight as strong ones is also useful.

EduWonk presents the issues in terms of the literature and policy discussion surrounding Teach for America, but the same issues apply in nearly every area of policy-related empirical work. They result from a combination of advocacy groups producing output that looks like scholarly research but is not with newspaper reporters, columnists, bloggers and others who lack the quantitative skills and context knowledge required to correctly weigh the evidence.

Summer seminars for undergrads

The seminars run by the Institute for Humane Studies are a lot of fun - at least they were 20 years ago when I did one. I suspect they're even better now.

The seminars provide an introduction to classical liberal thought across a range of disciplines. In my experience, the students were at least as much fun as the faculty. When I did it there were people from all different perspectives within, and in the neighborhood, of the classical liberal perspective, ranging from anarcho-capitalists to Randroids to conservatives to sensitive, wonderful, clear-thinking and nearly always correct people like yours truly. :)

Well worth doing, if an experience like this appeals to you.

I even met someone I dated a few times.

Not sure what to say about this ...

... other than maybe "oh dear" ...

NB: College of DuPage is a public junior college in Illinois.

Hat tip: Dan Black

Remodeling teacher education

Apparently, the big new idea in teacher education is teaching teachers how to teach. As with evidence-based medicine, the current big new idea in medicine, one is led to ask "and just what is it that you were doing before?"

Shanghai and PISA

Several people sent me links to articles about Shanghai's performance on the PISA tests, such as this one from the Christian Science Monitor.

Some thoughts:

1. My Michigan colleague Kevin Miller, who is cited extensively in the article, is a very cool guy. We once ended up stuck at Reagan airport in DC for several hours. I don't know about Kevin, but it was by far the most interesting and enjoyable flight delay I ever had.

2. I am surprised that there was no mention of cheating in the article, not in the sense of the students who took the test cheating on it but rather via strategic manipulation of who showed up on test days. People raised under communism understand how to game performance measures even better than do US educators, whose manipulation of performance measures such as those in the No Child Left Behind Act is well documented.

3. It continues to amaze me how little real knowledge we have about how to educate. That has started to change in recent years with improved research designs but only over the screaming and flailing opposition of many educators and professors of education.

The dull life of a test grader

I found myself more sympathetic to this lonely rant from someone who makes money grading the written bits of standardized tests than I expected to be.

But, and there are a lot of buts in this case, I have some issues:

1. First, it is more important that people learn to read, write functional sentences and paragraphs and do basic math than that they learn to be creative or to experience art and music. It would be nice if everyone could learn all these things, but basic literacy and numeracy have to come first. I agree with Rich Hanushek that the upside of NCLB and the broader fixation on standardized measures of basic skills of which it is both cause and effect is to actually improve literacy and numeracy at the worst public schools. It would be great to find a way to do this while wasting less time and energy at better schools.

2. If the main point of the grades is to be used in class or school level aggregates, then it is optimal not to spend too much time on each one. The measurement error will cancel out. One hopes, though this is not described, that the exams from a given class or school are distributed among many graders so that grader-specific effects average out as well.

3. Given that students receive no feedback other than a number, it is again not worth spending a lot of grader time on each exam. Detailed feedback is important for learning to write, but it should be coming from the classroom teachers.

4. If the market clears, then wages are not too low.

5. The author's obsession with the role of private firms is completely beside the point. The demand for standardized testing did not originate with the private firms, and the grading procedure would be little different, other than taking longer and costing more, if it were undertaken by government employees.

6. I was most concerned about the fact that firms are moving the average up or down at the behest of politicians obsessed with averages over time. I will now trust those average even less than I already did, which was not very much.

Via: portside.org email list

TIF

Yesterday was the first meeting of the Technical Working Group (TWG) for Mathematica's evaluation of the Teacher Incentive Fund (TIF) on behalf of the Institute for Education Sciences of the US Department of Education. I have the good fortune to be on the TWG.

This evaluation will complement the one I blogged on a few weeks ago. I can't, of course, discuss design details, but the interaction at the meeting was lively and interesting. I like doing consulting that I learn from and that helps me think about research ideas, even though this sort of consulting pays less than, say, consulting about people's earnings in court cases related to death or injury on the job.

Naughty teacher?

New York is all aflutter at the fact that former sex worker Melissa Petro revealed her shocking past on-line just after receiving tenure.

This little bit of excitement suggests a panoply of possible responses, here are some:

1) I thought New York was a rough-and-tumble bastion of secularism, not rural Alabama. Though I suppose this kerfuffle is of a piece with the resistance to the "mosque" near the World Trade Center site. Both are narrow-minded and illiberal.

2) New York has many thousands of teachers. Based on published estimates, probably two percent or so of the female teachers have had sex for money at some point. One could round this up on the grounds that New York is not rural Alabama, or round it down on the grounds that college graduates are less likely to have sex for money (do we know this?) but in any case Melissa Petro has dozens, if not hundreds of compatriots in the NY public school teaching corps. Has NYC gone to hell as a result? Are students setting up brothels during recess? Maybe this is not really a very big deal after all.

3) Sex work is legal in most developed countries. Maybe it should be here too?

4) Maybe the real issue is Melissa's highlighting of the lack of any rational justification for tenure for primary and secondary teachers. More broadly, if you have a system with tenure, Melissa's timing is clearly the optimal one. In academia, in my experience people who have gamed the tenure system by successfully concealing their type prior to tenure stop doing research immediately after getting tenure. Why wouldn't they?

5) In what sense is it worse for children to have a former sex worker as a teacher compared to, say, a teacher union organizer? I had a math teacher in junior high who broke her arm [sic] banging it on the car of a substitute teacher hired during a strike. Did she provide a good example to her students? Certainly she did not provide a good example of the life of the mind and the value of resolving disputes non-violently. Melissa, in contrast, provides a fine example of not paying attention to stupid laws and also of honesty.

Teacher performance pay

Here is the WaPo piece on the teacher performance pay experiment that I described in an earlier post and that is being presented at the Ford School this afternoon.

The experimental design here is solid but there are some issues of interpretation that stem directly from the nature of the study design.

1. Only volunteer teachers were randomized. Whether one should expect the effects of performance incentives to be higher or lower for volunteers than non-volunteers is not clear a priori but using volunteers makes the politics of doing a random assignment study much easier. The importance of this issue also depends on the take-up rate, which as I recall was higher than you might think but not 100 percent by any means.

2. This is a temporary program. That means that incentives to invest in being a better teacher are limited, as they pay off only in the short term. The numbers at play in the study are not trivial but also not large relative to, say, the cost of going back to college to get a subject area degree.

3. As Rick Hanushek notes in the article, by design the study looks only at current teachers. One important effect of an ongoing regime of performance pay might be to change the mix of people who become teachers in good ways. This study cannot pick up that effect.

4. The treatment is just performance pay. There is no mentoring or other treatment. If teachers are going to figure out how to do better, they are on their own with the literature. Knowing something about the literature, I know that is problematic. But it is not clear that combining performance pay with the sort of additional treatments mentioned in the article, such as mentoring, would work. For example, the randomized trial that the Dept. of Education funded on additional mentoring for new teachers did not produce clear positive results.

I should note in relation to the last point that things are very different in many developing countries. There the effort margin is often quite important. In some countries, for example, teacher absence rates are shockingly high. Performance pay might well get them to show up, and showing up would likely improve test scores.

A side point: why is Rick Hanushek the only person quoted who is identified as having political beliefs? The Hooover Institution where he works is labeled "conservative leaning" while the fellow from the teacher labor cartel is not given any politics, nor are the people from the federal government, nor is anyone else.

Hat tip: Dann Millimet

School reform, Matt Welch, and democrat posers

Few people do pissed off better than Matt Welch at Reason. In this case he is pissed at democrats cheering this weeks (indirect) defeat for school reform in DC.

I would add that the democrats are the party of the poor in the same sense that the republicans are the party of free markets, which is to say there is lots of talk but, when the choice comes down to walking the walk or selling out to an interest group, they follow the money.

Talk of interest to some readers

There is a talk that some local readers may be interested in; I've pasted the announcement below. I can't go because it conflicts with the practice job talk of a student whose committee I am on. Missing the talk is not as costly as it would otherwise be because I already know the findings because I consulted on the project. The design done well and the findings are quite interesting and on an important policy issue.

------------------------------------------------------------------------------------------

The Center for Local, State, and Urban Policy (CLOSUP)presents
Matthew G. SpringerAssistant Professor of Public Policy and Education, and
Director of the National Center on Performance IncentivesVanderbilt University

“Teacher Pay for Performance: Experimental Evidence from Nashville’s Project on Incentives in Teaching”

Wednesday, September 22, 2010
4:00 – 5:30pm
Free and open to the public.

Gerald R. Ford School of Public Policy
Betty Ford Classroom
1110 Joan and Sanford Weill Hall
735 South State Street, Ann Arbor

Sponsored by the Education Policy Initiative (EPI) at the Center for Local, State, and Urban Policy (CLOSUP). EPI is a program of coordinated activities designed to bring the latest academic knowledge to issues of education policy.

For more information visit http://www.closup.umich.edu/ or call (734) 647-4091

--------------------------------------------------------------------------

Oh, and as an aside, there is actually less, rather than more, information at the website ...

Addendum: There is a bit more at the NCPI website on the nature of the study, but no results. Perhaps the results are being unveiled next Wednesday at the Ford School. Good thing I kept quiet about them in my original post!