WIA is wonderful, or not

Actually, WIA - the Workforce Investment Act - appears to be wonderful only if you do not understand that the performance measures that the government uses for quantitative monitoring of the program have nothing to say about whether or not the program adds value.

Some people called FutureWork Systems, who I think make their money providing services to local WIA programs, have produced a web site, called the WIA dashboard, the neatly summarizes the performance measures. For example, if you click through, you will learn that WIA provided services to about 2.75 million people in the last four quarters for which data are available. At the bottom of the table, the site claims that this is the number of people who "benefit from" the program, but in fact this is wrong on two counts. First, some of the people who benefit from the program are not counted, such as the people who run businesses like FutureWork Systems.

More importantly, providing services is not the same as producing a benefit. The statistics in the table are all about inputs and outcomes, and not at all about impacts, which must be measured relative to the counterfactual of what would have happened to the service recipient had he or she not participated in the program. The data from earlier experimental (i.e. wannabe participants were randomly assigned to the offer of participation or not) evaluations of similar programs make clear that outcome levels and impacts are very different indeed. At the extreme, the predecessor to WIA, the Job Training Partnership Act, was estimated to reduce the average earnings of its male out-of-school youth participants relative to the counterfactual.

The site will also tell you that about 900,000 adult and dislocated worker participants were "placed" in employment. Of course, if you click on the definition at the bottom, you will learn that "placed" here does not really mean placed, it just means that they somehow found a job. In many cases, that job will have nothing to do with whatever services they received from WIA.

Sadly, one must conclude that the people at FutureWorks are either clueless about the difference between outcome levels and impacts relative to a counterfactual, or they know the difference, but are deliberately trying to mislead the innocents who happen to find their website. In either case, shame on them.

And you know, just as an aside, this topic is so important that someone ought to write a book about it.

Hat tip: Carolyn Heinrich, who forwarded an email about the site from someone misleadingly called "Geoff Smith". That email begins with "WIA Matters! Let's show 'em our impact".  Too bad they have no impacts to show.