Sensitivity of Treatment on Treated Effects in the Housing Vouchers Welfare Experiment to Alternative Measures of Compliance

Daniel Gubits
Abt Associates Inc.

Mark Shroder
U.S. Department of Housing and Urban Development

Evaluation Tradecraft
Evaluation Tradecraft presents short articles about the art of evaluation in housing and urban research. Through this department of Cityscape, the Office of Policy Development and Research presents developments in the art of evaluation that might not be described in detail in published evaluations. Researchers often describe what they did and what their results were, but they might not give readers a step-by-step guide for implementing their methods. This department pulls back the curtain and shows readers exactly how program evaluation is done. If you have an idea for an article of about 3,000 words on a particular evaluation method or an interesting development in the art of evaluation, please send a one-paragraph abstract to marina.l.myhre@hud.gov.

The views expressed in this article are those of the authors and do not necessarily represent the official positions or policies of the U.S. Department of Housing and Urban Development or the U.S. government.

In the social experimentation literature, the treatment-control outcome difference is the “intention to treat” and the adjustment of that difference to reflect actual participation in treatment as the “treatment on treated” (TOT) effect of the intervention. Previous contributions to this literature have been silent on the sensitivity of TOT to alternative definitions of treatment.

In this article, we apply alternative methods of estimating treatment-on-the-treated to data from the Effects of Housing Vouchers on Welfare Families experiment. The final report on that experiment employs an original method of calculation of TOT and finds that early negative impacts on earnings fade out after 1.5 to 2 years. We test for sensitivity of these results to alternative concepts of participation: participation at time of measurement, exposure to treatment over time, and definition of the intervention as housing assistance per se rather than as vouchers.

We find that the published TOT results are qualitatively robust to the definition of treatment. We believe this finding is likely to apply more generally in large, well-controlled experiments.