Archive for August 30, 2014
EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 4
August 30, 2014Finally, for adult women we consider how well the selfassessment data match up with the analyses considered in earlier sections. The selfassessment data are not consistent with the assumption of perfect positive dependence in outcomes across the two states. As shown in Figure 1, for adult women the JTPA data indicate that perfect positive dependence in outcomes between the treated and untreated states implies a strictly positive impact of the program for about 85 percent of participants – all except those with zero earnings in both states.
This value far exceeds the overall selfreported effectiveness rate of 44 percent reported in row 3 of Table 11. The 44 percent rate lies below that found even for the case of perfect negative dependence. Overall, the selfreported impact data appear to be too negative when compared to our analyses of the experimental earnings data. This evidence is consistent with participants reporting a net measure while the experimental “treatment effect” measures gross outcomes.
The lower positive rating of the program from self assessment data than from gross outcome data is all the more striking when it is realized that the selfassessments are only recorded for people who report receiving training while the gross outcome data for participants include those who leave the program and the attriters have lower earnings than the nonattriters.
EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 3
August 28, 2014
The top panel of Table 11 reports JTPA participant responses to a question about whether or not the program made them better off.31 Assuming people answer honestly, and are reporting a gross impact, the selfassessment data clearly contradict the hypothesis of impact homogeneity. For all four demographic groups, 65 to 70 percent of selfreported participants give a positive selfassessment, not the 100% or 0% predicted if impacts were homogeneous add comment.
However, if respondents are reporting a perceived net impact, the evidence reported in Table 11 does not necessarily contradict an assumption of gross impact homogeneity if there is heterogeneity in costs across participants. The entries in the third row of Table 11 reveal that the fractions reporting a positive impact are far lower than those obtained from all of the analyses using outcome data. This evidence is consistent with one of two hypotheses: (a) that respondents are reporting net outcomes and that costs borne by participants are a substantial fraction of gross outcomes or (b) that selfassessments are inaccurate.
National JTPA Study 18 Month Impact Sample
Adult
Males 
Adult
Females 
Male
Youth 
Female
Youth 

Full Sample Percentages  
Percent who selfreport participating:  61.63  68.10  62.62  66.29 
Percent of selfreported participants with a  (0.81)  (0.68)  (1.29)  (1.09) 
positive selfassessment:  62.46  65.21  67.16  71.73 
(1.04)  (0.85)  (159)  (L29)  
Overall percent with positive selfassessments:  38.49  44.41  42.06  47.55 
(0.81)  (0.73)  (1.32)  (1.16)  
Percent of SelfReported Participants with a Positive SelfAssessment by Primary Treatment Received  
None (dropouts)  48.89  51.44  58.90  61.56 
(2.07)  (1.85)  (3.33)  (2.79)  
Classroom training in occupational skills  74.10  73.47  72.73  75.28 
(2.15)  (136)  (3.60)  (2.30)  
Onthejob training at private firm  75.13  78.90  71.00  75.00 
(2.18)  (2.14)  (4.56)  (4.04)  
Job search assistance  59.57  59.80  68.09  68.94 
(2.27)  (2.18)  (3.94)  (4.04)  
Basic education  62.96  56.55  70.97  78.44 
(4.67)  (3.84)  (4.09)  (3.19)  
Work experience  66.67  68.75  82.76  73.17 
(9.83)  (5.84)  (7.14)  (7.01)  
Other  58.47  66.40  62.50  77.98 
(3.65)  (2.98)  (4.77)  (3.99) 
EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 2
August 26, 2014Evidence from SelfAssessments of Program Participants
Selfassessments of program participants represent an alternative to comparisons of observed outcomes as a measure of program impact. Unlike the ex ante measures based on secondorder stochastic dominance, these measures are statements about ex post expectations. There is no reason why the two measures should agree if people revise their assessments based on what they learn about a program by participating in it. In this section, we consider the strengths and limitations of selfreported assessments of satisfaction with the program as an evaluation criterion, and report on selfevaluations by participants in the JTPA experimental treatment group. We also consider what can be learned from selfassessment data regarding the heterogeneity of individual treatment effects and the rationality of program participants.
Using participant assessments to evaluate a program has two main advantages relative to the approaches already discussed. First, participants have information not available to external program evaluators. They typically know more about certain components of the cost of program participation than do evaluators. Most evaluations, including the National JTPA Study, do not even attempt to value participant time, transportation, child care or other costs in evaluating program effectiveness, unless they are paid by the program through subsidies.
Participants are likely to include such information in arriving at their selfassessments of the program. Second, participant evaluations provide information about the values placed on outcomes by participants relative to their perceived cost. They have the potential of providing a more inclusive measure of the program’s effects than would be obtained from looking only at gross outcomesone that includes “client satisfaction”. To some parties in the welfare state, “customer satisfaction” is an important aspect of a program.
(more…)
EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 1
August 24, 2014
Testing For Ex Ante Stochastic Rationality of Participants
If individuals choose whether or not to participate in the program based on the gross gains from the program, if they possess a common, but unknown, concave utility function, and if they know the marginal distribution of outcomes in the participation and nonparticipation states, then secondorder stochastic dominance should order the distributions of outcomes for persons who sought to go into the program. For nonnegative yl,y° this form of rationality implies
(more…)
EVALUATING THE WELFARE STATE: Frechet Bounds 3
August 22, 2014Assuming the Gain Is Independent of the Base
Another source of identifying information for the joint distribution of outcomes and the distribution of impacts postulates that the gain, A, is independent of the base У0, so that Y° HAZ) = 1. Letting R = 1 if a person who applies and is provisionally accepted into the program is randomized into the program, and R = 0 if a provisionally accepted applicant is randomized out, Y = Y° + ЯД, and RA _LL Y°. Throughout we condition on D = 1. This identifying condition would be satisfied if Y° is known but the gain, Д, cannot be forecast at the time decisions are made about program participation. This case is extensively discussed in Heckman and Robb (1985, p. 181), and produces a model that is intermediate between the commoneffect model and the variableimpact model when the impact is anticipated by agents.
(more…)
EVALUATING THE WELFARE STATE: Frechet Bounds 2
August 20, 2014Sensitivity to Alternative Assumptions About Dependence Across the Distributions
Using the sample data, we can pair percentiles of the Y1 and F° distributions for any choice of rank correlation r between 1.0 and 1.0. The case of r = 1.0 corresponds to the case of perfect positive dependence
CHARACTERISTICS OF THE DISTRIBUTION OF IMPACTS ON EARNINGS IN THE 18 MONTHS AFTER RANDOM ASSIGNMENT AT THE FRECHET BOUNDS
Statistic  Lower Bound  Upper Bound 
Impact Standard Deviation  14968.76  674.50 
(211.08)  (137.53)  
Outcome Correlation  0.760  0.998 
(0.013)  (0.001)  
Spearman’s p  0.9776  0.9867 
(0.0016)  (0.0013) 
Heckman, Smith and Clements (1997) show how to obtain random samples of permutations conditional on values of r between 1.0 and 1.0. We display two sets of estimates from their work. The first set assumes positive but not perfect dependence between the percentiles of У1 and У”0, with т = 0.95. Estimates based on a random sample of 50 percentile permutations with this value of r appear in the second column of Table 8. These results show that even a modest departure from perfect positive dependence substantially widens the distribution of impacts. More striking still are the results in the third column of Table 8, which correspond to the case where r = 0.0.
This value of r is implied by independence between the percentiles of Yl and Y°. Here (as in the case with r = 1.0) the distribution of estimated impacts is implausibly wide with large positive values in each distribution often matched with zero or small positive values in the other. However, the conclusion that a majority of adult female participants benefit from the program is robust to the choice of r.
ESTIMATED PARAMETERS OF THE IMPACT DISTRIBUTION PERFECT POSITIVE DEPENDENCE, POSITIVE DEPENDENCE WITH т = 0.95, INDEPENDENCE AND PERFECT NEGATIVE DEPENDENCE CASES
National JTPA Study 18 Month Impact Sample Adult Females  
Perfect
Positive 
Positive  Independence  Perfect
Negative 

Dependence  Dependence  of У^{1} and Y°  Dependence  
Statistic  (r = 1.0)  with г = 0.95  4
II © о ‘W 
( т =1.0) 
5th Percentile  0.00  0.00  18098.50  22350.00 
(47.50)  (360.18)  (630.73)  (547.17)  
25 th Percentile  572.00  125.50  6043.00  11755.00 
(232.90)  (124.60)  (300.47)  (411.83)  
50th Percentile  864.00  616.00  0.00  580.00 
(269.26)  (280.19)  (163.17)  (389.51)  
75th Percentile  966.00  867.00  7388.50  12791.00 
(305.74)  (272.60)  (263.25)  (253.18)  
95th Percentile  2003.00  1415.50  19413.25  23351.00 
(543.03)  (391.51)  (423.63)  (341.41)  
Percent Positive  100.00  96.00  54.00  52.00 
(1.60)  (3.88)  (1.11)  (0.81)  
Impact Std Dev  1857.75  6005.96  12879.21  16432.43 
(480.17)  (776.14)  (259.24)  (265.88)  
Outcome Correlation  0.9903  0.7885  0.0147  0.6592 
(0.0048)  (0.0402)  (0.0106)  (0.0184) 
EVALUATING THE WELFARE STATE: Frechet Bounds
August 18, 2014
The Frechet bounds of expression (17) can also be applied to conditional (on D = 1) distributions. Both the lower and the upper Frechet bounds are proper probability distributions. At the upper bound bound, У1 is a nondecreasing function of У0. At the lower bound, Y° is a nonincreasing function of У1. These bounds are not helpful in bounding the distribution of gains Д = У1 — У0, although they bound certain features of it. From a theorem of Cambanis, et al. (1976), if k(Yl,Y°) is superadditive (or subadditive),27 then extreme values of E(k(Y1, Y°)\D = 1) are obtained from the upper and lower bounding distributions obtained from the experimental data Source.
(more…)
EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 4
August 16, 2014In place of ranks, we work with the percentiles of the У1 and Y° distributions, which have much better statistical properties. (See Heckman and Smith, 1993, Heckman, Smith and Clements, 1997). Equating percentiles across the two distributions, we form the pairs given in expression (13) and obtain the deterministic gain function given in (14). For the case of absolutely continuous distributions with positive density at y°, the gain function (14) can be written as A(y°) = ^11_1(JF0(y°JD = 1)) — y°.
We can test nonparametrically for the classical common effect model by determining if percentiles are uniformly shifted at all points of the distribution. We can form other pairings across percentiles by mapping percentiles from the У1 distribution into percentiles from the Y° distribution using the map T : qo. The data are consistent with all admissible transformations including (more…)
EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 3
August 14, 2014Evidence on Impact Heterogeneity
This subsection presents evidence on variability in the response to training. We find strong evidence against homogeneity. However unless the dependence across outcomes in the treated and untreated states is very high, the estimated variability in program gains is implausibly large.
Suppose that the JTPA experiment satisfies (110). Suppose that there are N treated persons and N nontreated persons. Suppose that the outcomes are continuously distributed. Rank the individuals in each treatment category in the order of their outcome values from the highest to the lowest. Define as as the ith highestranked person in the j distribution. Source Ignoring ties, we obtain two data distributions:
(more…)
EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 2
August 12, 2014We examine the implicit value placed on the program by addressing the following questions: (3a) “Are persons wTho applied to the program and were accepted into it but then randomized out of it placed in an inferior position relative to those accepted applicants who were not randomized out?” We measure ex ante rational regret using secondorder stochastic dominance, which is an appropriate measure under the assumption that individuals are completely uncertain of both V”1 and Y° before going into the program. We also consider ex post evaluations of participants by asking: (3b) “How “satisfied” are participants with their experience in the program?” Selfassessments of programs are widely used in evaluation research (see e.g., Katz, et al., 1975), but the meaning to be placed on them is not clear. Do they reflect an evaluation of the experience of the program (its process) or an evaluation of the benefits of the program?
Our evidence suggests that respondents report a net benefit inclusive of their costs of participating in the program. Groups for whom the program has a negative average impact as estimated by the “objective” experimental data express as much (or more) enthusiasm for the program as groups with positive average impacts. A third source of revealed preference evaluations uses the revealed choices of attriters from the program. Econometric models of selfselection since Heckman (1974a,b) have used revealed choice behavior to infer the evaluations people place on programs either by selecting into them or dropping out of them. The third part of the third question is thus (3c): “What implicit valuation of the program do attriters place on it?”
(more…)