Archive for August 30, 2014

EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 4

August 30, 2014

Finally, for adult women we consider how well the self-assessment data match up with the analyses considered in earlier sections. The self-assessment data are not consistent with the assumption of perfect positive dependence in outcomes across the two states. As shown in Figure 1, for adult women the JTPA data indicate that perfect positive dependence in outcomes between the treated and untreated states implies a strictly positive impact of the program for about 85 percent of participants – all except those with zero earnings in both states.

This value far exceeds the overall self-reported effectiveness rate of 44 percent reported in row 3 of Table 11. The 44 percent rate lies below that found even for the case of perfect negative dependence. Overall, the self-reported impact data appear to be too negative when compared to our analyses of the experimental earnings data. This evidence is consistent with participants reporting a net measure while the experimental “treatment effect” measures gross outcomes.

The lower positive rating of the program from self assessment data than from gross outcome data is all the more striking when it is realized that the self-assessments are only recorded for people who report receiving training while the gross outcome data for participants include those who leave the program and the attriters have lower earnings than the non-attriters.


EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 3

August 28, 2014

i (8)
The top panel of Table 11 reports JTPA participant responses to a question about whether or not the program made them better off.31 Assuming people answer honestly, and are reporting a gross impact, the self-assessment data clearly contradict the hypothesis of impact homogeneity. For all four demographic groups, 65 to 70 percent of self-reported participants give a positive self-assessment, not the 100% or 0% predicted if impacts were homogeneous add comment.

However, if respondents are reporting a perceived net impact, the evidence reported in Table 11 does not necessarily contradict an assumption of gross impact homogeneity if there is heterogeneity in costs across participants. The entries in the third row of Table 11 reveal that the fractions reporting a positive impact are far lower than those obtained from all of the analyses using outcome data. This evidence is consistent with one of two hypotheses: (a) that respondents are reporting net outcomes and that costs borne by participants are a substantial fraction of gross outcomes or (b) that self-assessments are inaccurate.
National JTPA Study 18 Month Impact Sample









Full Sample Percentages
Percent who self-report participating: 61.63 68.10 62.62 66.29
Percent of self-reported participants with a (0.81) (0.68) (1.29) (1.09)
positive self-assessment: 62.46 65.21 67.16 71.73
(1.04) (0.85) (1-59) (L29)
Overall percent with positive self-assessments: 38.49 44.41 42.06 47.55
(0.81) (0.73) (1.32) (1.16)
Percent of Self-Reported Participants with a Positive Self-Assessment by Primary Treatment Received
None (dropouts) 48.89 51.44 58.90 61.56
(2.07) (1.85) (3.33) (2.79)
Classroom training in occupational skills 74.10 73.47 72.73 75.28
(2.15) (1-36) (3.60) (2.30)
On-the-job training at private firm 75.13 78.90 71.00 75.00
(2.18) (2.14) (4.56) (4.04)
Job search assistance 59.57 59.80 68.09 68.94
(2.27) (2.18) (3.94) (4.04)
Basic education 62.96 56.55 70.97 78.44
(4.67) (3.84) (4.09) (3.19)
Work experience 66.67 68.75 82.76 73.17
(9.83) (5.84) (7.14) (7.01)
Other 58.47 66.40 62.50 77.98
(3.65) (2.98) (4.77) (3.99)


EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 2

August 26, 2014

Evidence from Self-Assessments of Program Participants

Self-assessments of program participants represent an alternative to comparisons of observed outcomes as a measure of program impact. Unlike the ex ante measures based on second-order stochastic dominance, these measures are statements about ex post expectations. There is no reason why the two measures should agree if people revise their assessments based on what they learn about a program by participating in it. In this section, we consider the strengths and limitations of self-reported assessments of satisfaction with the program as an evaluation criterion, and report on self-evaluations by participants in the JTPA experimental treatment group. We also consider what can be learned from self-assessment data regarding the heterogeneity of individual treatment effects and the rationality of program participants.

Using participant assessments to evaluate a program has two main advantages relative to the approaches already discussed. First, participants have information not available to external program evaluators. They typically know more about certain components of the cost of program participation than do evaluators. Most evaluations, including the National JTPA Study, do not even attempt to value participant time, transportation, child care or other costs in evaluating program effectiveness, unless they are paid by the program through subsidies.

Participants are likely to include such information in arriving at their self-assessments of the program. Second, participant evaluations provide information about the values placed on outcomes by participants relative to their perceived cost. They have the potential of providing a more inclusive measure of the program’s effects than would be obtained from looking only at gross outcomes-one that includes “client satisfaction”. To some parties in the welfare state, “customer satisfaction” is an important aspect of a program.

EVALUATING THE WELFARE STATE: Evidence from Participant Behavior 1

August 24, 2014

Testing For Ex Ante Stochastic Rationality of Participants

If individuals choose whether or not to participate in the program based on the gross gains from the program, if they possess a common, but unknown, concave utility function, and if they know the marginal distribution of outcomes in the participation and non-participation states, then second-order stochastic dominance should order the distributions of outcomes for persons who sought to go into the program. For non-negative yl,y° this form of rationality implies


August 22, 2014

Assuming the Gain Is Independent of the Base

Another source of identifying information for the joint distribution of outcomes and the distribution of impacts postulates that the gain, A, is independent of the base У0, so that Y° HA|Z) = 1. Letting R = 1 if a person who applies and is provisionally accepted into the program is randomized into the program, and R = 0 if a provisionally accepted applicant is randomized out, Y = Y° + ЯД, and RA _LL Y°. Throughout we condition on D = 1. This identifying condition would be satisfied if Y° is known but the gain, Д, cannot be forecast at the time decisions are made about program participation. This case is extensively discussed in Heckman and Robb (1985, p. 181), and produces a model that is intermediate between the common-effect model and the variable-impact model when the impact is anticipated by agents.


August 20, 2014

Sensitivity to Alternative Assumptions About Dependence Across the Distributions

Using the sample data, we can pair percentiles of the Y1 and F° distributions for any choice of rank correlation r between -1.0 and 1.0. The case of r = 1.0 corresponds to the case of perfect positive dependence


Statistic Lower Bound Upper Bound
Impact Standard Deviation 14968.76 674.50
(211.08) (137.53)
Outcome Correlation -0.760 0.998
(0.013) (0.001)
Spearman’s p -0.9776 0.9867
(0.0016) (0.0013)

Heckman, Smith and Clements (1997) show how to obtain random samples of permutations conditional on values of r between 1.0 and -1.0. We display two sets of estimates from their work. The first set assumes positive but not perfect dependence between the percentiles of У1 and У”0, with т = 0.95. Estimates based on a random sample of 50 percentile permutations with this value of r appear in the second column of Table 8. These results show that even a modest departure from perfect positive dependence substantially widens the distribution of impacts. More striking still are the results in the third column of Table 8, which correspond to the case where r = 0.0.

This value of r is implied by independence between the percentiles of Yl and Y°. Here (as in the case with r = -1.0) the distribution of estimated impacts is implausibly wide with large positive values in each distribution often matched with zero or small positive values in the other. However, the conclusion that a majority of adult female participants benefit from the program is robust to the choice of r.


National JTPA Study 18 Month Impact Sample Adult Females


Positive Independence Perfect


Dependence Dependence of У1 and Dependence
Statistic (r = 1.0) with г = 0.95 4





( т =-1.0)
5th Percentile 0.00 0.00 -18098.50 -22350.00
(47.50) (360.18) (630.73) (547.17)
25 th Percentile 572.00 125.50 -6043.00 -11755.00
(232.90) (124.60) (300.47) (411.83)
50th Percentile 864.00 616.00 0.00 580.00
(269.26) (280.19) (163.17) (389.51)
75th Percentile 966.00 867.00 7388.50 12791.00
(305.74) (272.60) (263.25) (253.18)
95th Percentile 2003.00 1415.50 19413.25 23351.00
(543.03) (391.51) (423.63) (341.41)
Percent Positive 100.00 96.00 54.00 52.00
(1.60) (3.88) (1.11) (0.81)
Impact Std Dev 1857.75 6005.96 12879.21 16432.43
(480.17) (776.14) (259.24) (265.88)
Outcome Correlation 0.9903 0.7885 -0.0147 -0.6592
(0.0048) (0.0402) (0.0106) (0.0184)


August 18, 2014

i (7)
The Frechet bounds of expression (17) can also be applied to conditional (on D = 1) distributions. Both the lower and the upper Frechet bounds are proper probability distributions. At the upper bound bound, У1 is a non-decreasing function of У0. At the lower bound, Y° is a non-increasing function of У1. These bounds are not helpful in bounding the distribution of gains Д = У1 — У0, although they bound certain features of it. From a theorem of Cambanis, et al. (1976), if k(Yl,Y°) is superadditive (or subadditive),27 then extreme values of E(k(Y1, Y°)\D = 1) are obtained from the upper and lower bounding distributions obtained from the experimental data Source.

EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 4

August 16, 2014

In place of ranks, we work with the percentiles of the У1 and Y° distributions, which have much better statistical properties. (See Heckman and Smith, 1993, Heckman, Smith and Clements, 1997). Equating percentiles across the two distributions, we form the pairs given in expression (13) and obtain the deterministic gain function given in (14). For the case of absolutely continuous distributions with positive density at y°, the gain function (14) can be written as A(y°) = ^11_1(JF0(y°|JD = 1)) — y°.

We can test non-parametrically for the classical common effect model by determining if percentiles are uniformly shifted at all points of the distribution. We can form other pairings across percentiles by mapping percentiles from the У1 distribution into percentiles from the Y° distribution using the map T : qo. The data are consistent with all admissible transformations including (more…)

EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 3

August 14, 2014

Evidence on Impact Heterogeneity

This subsection presents evidence on variability in the response to training. We find strong evidence against homogeneity. However unless the dependence across outcomes in the treated and untreated states is very high, the estimated variability in program gains is implausibly large.
Suppose that the JTPA experiment satisfies (1-10). Suppose that there are N treated persons and N nontreated persons. Suppose that the outcomes are continuously distributed. Rank the individuals in each treatment category in the order of their outcome values from the highest to the lowest. Define as as the ith highest-ranked person in the j distribution. Source Ignoring ties, we obtain two data distributions:

EVALUATING THE WELFARE STATE: Evidence on Impact Heterogeneity 2

August 12, 2014

We examine the implicit value placed on the program by addressing the following questions: (3a) “Are persons wTho applied to the program and were accepted into it but then randomized out of it placed in an inferior position relative to those accepted applicants who were not randomized out?” We measure ex ante rational regret using second-order stochastic dominance, which is an appropriate measure under the assumption that individuals are completely uncertain of both V”1 and Y° before going into the program. We also consider ex post evaluations of participants by asking: (3b) “How “satisfied” are participants with their experience in the program?” Self-assessments of programs are widely used in evaluation research (see e.g., Katz, et al., 1975), but the meaning to be placed on them is not clear. Do they reflect an evaluation of the experience of the program (its process) or an evaluation of the benefits of the program?

Our evidence suggests that respondents report a net benefit inclusive of their costs of participating in the program. Groups for whom the program has a negative average impact as estimated by the “objective” experimental data express as much (or more) enthusiasm for the program as groups with positive average impacts. A third source of revealed preference evaluations uses the revealed choices of attriters from the program. Econometric models of self-selection since Heckman (1974a,b) have used revealed choice behavior to infer the evaluations people place on programs either by selecting into them or dropping out of them. The third part of the third question is thus (3c): “What implicit valuation of the program do attriters place on it?”

Pages: 1 2 Next