Hat tip: Charlie Brown
Whew.
9 years ago
A long pondered but only lately realized blog about economics, politics, evaluation, econometrics, academia, college football and whatever else comes to mind.
A very closely related problem turns on the merits of using performance measures to proxy for rigorous impact estimates. Since these measures were first conceived during the CETA program, attempts to refine them so they actually “work” have amounted to the workforce development field's equivalent of the quest for the Holy Grail. Like its predecessor quest, so far this effort has been futile. There is no convincing evidence that using performance measures as a proxy is a good idea and lots of evidence against it. As explained by Burt Barnow in his chapter here, “Lessons from the WIA Performance Measures,” workforce performance measures do not correlate well with program impacts. That really should not be a surprise, because coming up with reliable performance measures requires that we be able to confidently and consistently solve the evaluation problem.Though nominally aimed at the Europeans, there is much that US policymakers could learn from the book as well. They could also learn from the Europeans (at least some of them) about how to increase the quality of non-experimental program evaluations via better administrative data.
The company has even begun research into its efficacy, and the early results are striking. After one of Marturano’s seven-week courses, 83 per cent of participants said they were “taking time each day to optimise my personal productivity” – up from 23 per cent before the course. Eighty-two per cent said they now make time to eliminate tasks with limited productivity value – up from 32 per cent before the course. And among senior executives who took the course, 80 per cent reported a positive change in their ability to make better decisions, while 89 per cent said they became better listeners.Smith, Whalley and Wilcox mock these sorts of questions and provide evidence from an active labor market program that they do not correlate with impacts estimated in more compelling econometric ways.
Other companies have found that such programmes can generate both health benefits and cost savings. Aetna, partnering with Duke University School of Medicine, found that one hour of yoga a week decreased stress levels in employees by a third, reducing healthcare costs by an average of $2,000 a year.The main problem with this, of course, is that we do not learn the methods the companies used to find this amazing reductions in health care costs. Did they do random assignment? Did they compare participants to non-participants without controls? Did they compare participant health costs before and after the program? Two of those methods typically yield rubbish, one does not. The second problem with this is that the estimate does not really pass the smell test. Employees are large corporations do not have high average health care costs. If they did, in most cases they would not be working. For this group, a $2000 impact would be really large, so large, I suspect, as to be implausible.