Participation and Labor

Market Impacts for

the First 24 Sites to

Replicate HUD’s Jobs

Plus Program

Final Report

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

DISCLAIMER The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S. Government.

Participation and Labor Market Impacts

for the First 24 Sites to Replicate HUD’s Jobs Plus Program Final Report

Prepared for:

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Submitted by:

Nina Castells

Nandita Verma

Cynthia Miller

Joshua Vermette

Henry Kanengiser

MDRC

September 27, 2021

Acknowledgments

This report would not have been possible without the contributions of many people. We are especially grateful to the families and individuals in the 31 public housing developments covered by this evaluation, whose experiences have informed this report. We are also immensely grateful to the staff members at the 24 Public Housing Authorities who operated the program on the ground: Cohort 1: Boston Housing Authority, Charlotte Housing Authority, Chicago Housing Authority, Cuyahoga Metropolitan Housing Authority, Houston Housing Authority, Memphis Housing Authority, Roanoke Housing Authority, St. Louis Housing Authority, Syracuse Housing Authority; Cohort 2: Austin Housing Authority, Denver Housing Authority, Goldsboro Housing Authority, Metropolitan Development and Housing Authority (Nashville), Norfolk Redevelopment and Housing Authority, Oakland Housing Authority, Philadelphia Housing Authority, Sacramento Housing and Redevelopment Agency, San Antonio Housing Authority; and Cohort 3: the Housing Authority of Baltimore City, Dayton Housing Authority, New York City Housing Authority, City of Phoenix Housing Department, Providence Housing Authority, and Tampa Housing Authority.

This evaluation was sponsored by the Office of Policy Development and Research in the U.S. Department of Housing and Urban Development (PD&R). Nathan Bossie served as our project officer at HUD, who has provided valuable and thoughtful guidance on all aspects of the evaluation, including this report. We are also grateful to other staff at HUD who reviewed drafts of this report or participated in report briefings or webinars. In the Office of Public and Indian Housing (PIH), we thank Jayme Brown, Leigh van Rij, and Isaac Dozier for their insights on Jobs Plus implementation and their valuable feedback on the results of this evaluation. Staff at PD&R, including Anne Fletcher, Carol Star, and Mark Shroder, also provided helpful input on this report.

This report draws on a variety of data, and we often turned to Colleen Moore at Abt to help us understand and interpret data included in the Jobs Plus Data Visualization Tool. Lynn Rodgers at HUD served as our main contact for acquiring IMS/PIC and NDNH data.

At MDRC, James Riccio and Howard Bloom served as senior advisors and reviewers and provided critical guidance on the methods used in the impact analysis and the interpretation of the analysis and findings. Riccio and Bloom played lead roles on the original Jobs Plus demonstration and brought a wealth of historical context to the current evaluation. Bloom also provided valuable expert guidance on the statistical methodology of the study and closely advised the study team on analytical issues that arose during the course of the study. Our Expert Panel advisors, Larry Katz, Harvard University, and Kathy O’Regan, New York University, joined the final briefing on this report and provided important external review. We are grateful for their time and contributions. We also benefited from Andrew Rock’s support on a reconnaissance effort that we conducted early on in the evaluation to understand program operations at the Jobs Plus programs in the evaluation, and from Sam Xia’s analysis of the baseline HUD PIC/IMS data used for the match validation analysis. We also recognize Shawna Anderson’s diligent support on managing the project budget.

Foreword

Between 1998 and 2003, the U.S. Department of Housing and Urban Development (HUD) administered the original Jobs Plus Initiative. This groundbreaking demonstration aimed to increase employment and earnings among public housing residents through a combination of services and rent incentives. Jobs Plus used a place-based approach that engaged residents where they lived. This original demonstration showed remarkable positive results with earnings for participating residents increasing by 16 percent per year in three sites where all components of the Jobs Plus program were fully implemented.

Starting in 2014, Congress passed new funding for a new and expanded Jobs Plus Pilot Program. Over the next few years, HUD was provided with over $136 million, which it distributed through 56 grants to Public Housing Agencies (PHAs).

This report, “Participation and Labor Market Impacts for the First 24 Sites to Replicate HUD’s Jobs Plus Program,” examines the impact of the replication program in the first 24 PHAs to receive Jobs Plus Pilot Program funding that then implemented the program across 31 separate housing developments. These developments represent a wide diversity in terms of size, site demographics, location, and local contexts, offering an opportunity to understand the program's implementation experiences against different backdrops. The report includes some important lessons for program implementation and provides direction for future research. However, the report finds that there were no measurable additional increases in employment or earnings for residents, as compared to other sites that did not receive program funding. It should be noted that there were generally positive economic conditions during the study period, and that employment and earnings were increasing on average across all the study sites as the country continued its long recovery from the Great Recession. Additionally, it should be noted that this study occurred at the outset of the program when the program implementation strategies and documents were still in development.

Given the strength of long-term findings from the three sites in the original Jobs Plus that were fully implemented and sustained, the lack of findings showing employment earning benefits from this replication study are discouraging. How was implementation different for these replication sites from the original Jobs Plus? As discussed in the study, a unique feature of Jobs Plus is that it is intended to operate at saturation levels, where services, incentives, and community support for work to everyone living in the development. The replication study sites did not accomplish the same level of saturation as the original demonstration’s successful sites.

The authors hypothesize “so it is possible that the lack of positive effects on employment and earnings outcomes is at least partially due to participation levels overall not reaching an adequate threshold that would lead to meaningful impacts.”

This study is one of a series of key reports documenting the experiences and lessons of the expanded HUD’s Jobs Plus Program. This new report is a companion study to accompany the

“HUD Jobs Plus Outcomes Evaluation - Long-term Effects from the Original Jobs Plus Demonstration: Employment and Earnings for Public Housing Residents after 20 Years.” This overall body of new research includes a process study on implementation (“HUD’s Jobs Plus iii

Pilot Program for Public Housing Residents: Ongoing Implementation Experiences”) and an interim baseline report (“Scaling Up a Place-Based Employment Program: Highlights From the Jobs Plus Pilot Program Evaluation”) that studied the first nine sites in the expanded pilot. While study findings suggest that sites faced implementation challenges, the findings have informed HUD’s continued efforts to improve the program and further advance families’ ability to achieve quality employment and income growth.

Solomon Greene

Principal Deputy Assistant Secretary for Policy Development and Research U.S. Department of Housing and Urban Development

iv

TABLE OF CONTENTS

Executive Summary .................................................................................................................. ES-1

I.

Introduction ...................................................................................................................................... 1

II.

Brief Policy Context and Prior Research ......................................................................................... 3

III.

The Jobs Plus Outcomes Evaluation .............................................................................................. 13

IV.

Characteristics of Jobs Plus Developments ................................................................................... 21

V.

Characteristics of the Study Sample .............................................................................................. 30

VI.

Resident Participation in Jobs Plus Activities................................................................................ 36

VII.

Impacts ........................................................................................................................................... 51

VIII. Discussion ...................................................................................................................................... 66

IX.

Conclusion ..................................................................................................................................... 70

Appendix A: Match Validation Analysis ....................................................................................... 71

Appendix B: Analytic Model & Specifications ............................................................................. 91

Appendix C: Jobs Plus And Comparison Group Baseline Tables ................................................. 95

Appendix D: Method For Creating The Count Of Eligible Residents ......................................... 100

Appendix E: Additional Participation Tables .............................................................................. 102

Appendix F: Sensitivity Analyses ................................................................................................ 113

Appendix G: Additional Impact Analyses .................................................................................. 118

Appendix H: JPEID Capture Rate And Discontinuities Analysis ............................................... 124

X.

References .................................................................................................................................... 144

v

Executive Summary

Residents of public housing communities face significant challenges to economic mobility.

In the mid-1990s, the U.S. Department of Housing and Urban Development (HUD), the Rockefeller Foundation, and MDRC developed Jobs Plus, a program designed to promote economic mobility through employment services, rent-based work incentives, and a community environment that supports work. This model, which was the subject of a rigorous evaluation, found that developments that fully implemented the program model saw the program boost annual earnings by a substantial margin through the end of 7 years of followup. 1 Since the model was first designed and implemented 20 years ago, the Jobs Plus program has expanded across the country through local and federal replication efforts, most recently a large-scale replication through HUD, the focus of this report.

In 2014, Jobs Plus became part of federal housing policy when Congress authorized $24

million to support the scale-up of this program. 2 HUD awarded the first round of grants to nine public housing agencies (PHAs) in April 2015, with funding ranging from $1.9 million to $3 million, and with each PHA contributing at least 25 percent of their grant through matched contributions and leverage. To date, HUD has awarded about $136 million through 56 grants to 50 PHAs to implement Jobs Plus. Each of these grants is awarded for a 48-month period and is nonrenewable. 3 In 2017, HUD sponsored an outcomes evaluation focused on the 24 developments within the first three cohorts to understand whether Jobs Plus continues to be effective, when the model is widely replicated in varying contexts.

This report presents the findings of this outcomes evaluation and examines the impacts of Jobs Plus on employment and earnings on these first three cohorts of public housing agencies to receive HUD grants to implement this program. 4 Although MDRC’s original evaluation provided credible evidence of the effectiveness of the Jobs Plus model at the time it was implemented, those findings pertained to impacts on residents from a small number of sites almost 20 years ago, when labor market and policy environments were different from those faced by public housing residents today and therefore cannot be taken for granted. Replication through HUD’s current Jobs Plus expansion is in the hands of a broader set of actors, in many more locations, with a wider variety of local adaptations. 5

1 See Riccio (2010) and Bloom, Riccio, and Verma (2005).

2 Funding is authorized by the Consolidated Appropriations Act, 2014, Public Law 113-76, 128 Stat. 5, enacted January 17, 2014.

3 Later cohorts of Jobs Plus grantees were awarded 54-month grants.

4 The cohorts in this study were awarded grants in April 2015, December 2015, and September 2016, respectively.

5 Verma et. al. (2019) capture these adaptations for the Cohort 1 sites.

ES-1

This report describes the Jobs Plus framework and briefly summarizes prior evidence of its implementation and effectiveness. It details the characteristics of the public housing developments and describes how residents engaged in the services offered to them by this program. Finally, the report presents the findings of the impact analysis, including the overall effects of Jobs Plus on employment rates and average earnings and the variation of these effects across sites.

Core Features of the Jobs Plus Framework

The original Jobs Plus program was designed as a place-based response to the many challenges public housing residents face in improving their employment situations: many developments are located in areas of concentrated poverty, and residents often struggle with poor work histories, limited education, lack of adequate childcare, health or medical problems (including substance abuse), and worry about crime and safety in their neighborhoods. The program’s theory of change includes a set of components shown to be promising by prior research in welfare reform and other fields. It was hoped that combining them in a single model would be mutually reinforcing and make Jobs Plus a more powerful intervention. 6 These components are (1) onsite employment-related services and activities, (2) rent-based financial incentives, and (3) community support for work.

Onsite employment services include job search assistance, referrals to education and training programs, and support services coordinated by staff located within the development to make it easier to engage residents and to make Jobs Plus staff more a part of the community they serve. Rent-based financial incentives are meant to encourage families to increase their earnings without affecting their rent contributions when they enter work or increase their earnings, allowing them to see a bigger financial return from work. The third component, Community Support for Work, emphasizes resident-to-resident outreach, information sharing, and mutual support, as well as connections to social networks outside the public housing development. The design of this component was largely influenced by the recognition of the importance of the role social networks and social capital can play to promote employment outcomes.

Another unique feature of Jobs Plus is that it is intended to operate at saturation levels—

that is, to offer services, incentives, and community support for work to everyone living in the development. A place-based program has the potential to benefit residents in a development beyond the personal benefit of receiving employment and other support services through the program and through the rent incentives. Residents who formally enroll in Jobs Plus and receive services or enroll in the rent incentives can influence other residents in the development without formal involvement in the program through strengthening and leveraging social networks among residents, sharing information about employment opportunities and availability of services in the 6 See Bloom, Riccio, and Verma, 2005; Blank and Wharton-Fields, 2008; and Greenberg et al., 2015.

ES-2

community, and encouraging residents to attend Jobs Plus events that may benefit residents.

Engagement and participation cannot be measured directly as they are with non-place-based programs serving individuals, which is important to keep in mind when interpreting measures of formal enrollment and participation in Jobs Plus and related activities.

Since its launch in the 1990s, each replication of Jobs Plus, including HUD’s scale-up of the program, described in this report, has sought to preserve the original framework, making modifications to incorporate lessons and adapt the framework to new operating environments.

The HUD Jobs Plus Replication Evaluation

This evaluation examines residents’ participation in each of the three components of the program and assesses its effects on residents’ employment and earnings. The sample for the impact analysis includes residents in Jobs Plus and comparison developments between 18 and 57

years of age at program launch (i.e., when Jobs Plus grants were awarded to PHAs) and who are not identified as having a disability in the housing agencies’ data. 7 In households with more than one eligible household member, a “focal adult” is identified for the main impact analysis. 8

Four data sources are used in this study. HUD’s Picture of Subsidized Households offers annual snapshots of the characteristics of the Jobs Plus and comparison developments. This report uses the snapshot year prior to the grantee award date. HUD’s Inventory Management System (IMS)/PIH Information Center (PIC) data are used to describe the individuals in the study sample, drawing on the household’s most recent certification (e.g., annual recertification, interim recertification) prior to the launch of Jobs Plus. Program enrollment and engagement patterns are examined using the quarterly reports submitted to HUD by Jobs Plus grantees.

Finally, the impact analysis relies on the National Directory of New Hires (NDNH), which

includes quarterly wage data for workers in employment covered by the unemployment insurance (UI) system. NDNH data from Q3, 2014 to Q3, 2020—covering three to nine quarters of preprogram data, depending on the cohort, and 16 to 22 quarters of followup data—are used in this report.

The two confirmatory outcomes for this evaluation (representing the key hypotheses to be evaluated for the study) are cumulative earnings and average quarterly employment over the 4-year followup period. Program effects are examined separately for each year in the followup period as well. The followup period mostly predates the onset of the COVID-19 pandemic, which triggered unprecedented economic shocks around the country starting around March 2020.

7 Using this definition, the evaluation excludes residents who will become elderly—defined by HUD as 62 years old or older—during the 4-year followup period.

8 The “focal adult” designation is assigned to the head of household or the spouse or co-head if the head of household did not meet the age and disability status criteria. If neither the head of household nor the spouse met these criteria, the focal adult was selected at random from other eligible adults in the household.

ES-3

The last two quarters of the fourth year of followup for the third cohort, however, coincide with the initial months of the pandemic.

The study relies on a matched-comparison design to assess the effects of the Jobs Plus program. Program impacts were estimated using a hybrid random effects and fixed effects statistical model that leverages the data from the 24 replication sites to provide estimates of the overall average effectiveness of Jobs Plus, as well as an understanding of how these impacts vary across sites. The model, a hierarchical (two-level, the individual level and the PHA level) linear model, is designed for evaluations interested in estimating both the average effect of the program across all sites and the variation in effects across sites. Although nonexperimental regression-based approaches are vulnerable to selection bias for samples with few sites, the large number of sites in this present study (24) allow for many of these site-specific biases to average out. 9

Impact estimates on all outcomes are measured for each year after the launch of the program for 4 years of followup, whether sample members stayed or moved out of the development during the followup period. This followup period covers the full 4-year grant period (though some grantees received grant extensions beyond 4 years). To account for the fact that the main analysis estimates effects on more than one outcome, and to avoid the potential for false positives, the p-values for the two confirmatory outcomes are adjusted using the Benjamini-Hochberg multiple hypothesis testing method.

Characteristics of the Jobs Plus Developments and the Study Sample The 24 PHAs in this study comprise 31 public housing developments. These developments represent a wide diversity in terms of size, site demographics, location, and local contexts, offering an opportunity to understand the program’s implementation experiences against different backdrops. All sites were expected to saturate their developments in terms of awareness of and participation in services, meaning that larger developments had to reach more residents with the same funding amount as some smaller developments.

Most of the developments were located in highly economically distressed areas. County-level unemployment rates at the start of program implementation were relatively low across most sites, reflecting the continued fall in unemployment after the Great Recession. Residents living in areas with high poverty rates and low employment rates may face substantial barriers to employment, for example, which may affect their ability to engage in program services or their ability to benefit from these services.

The HUD administrative data show that there were 11,521 nondisabled residents ages 18 to 57 living in 9,220 households at the start of Jobs Plus across the 31 developments in the 24

9 Lipsey and Wilson, 1993; Bloom, Michalopoulos, and Hill, 2005.

ES-4

locations. Overall, the study sample is mostly female (86 percent), on average 36 years old, and mostly African American (78 percent). Nineteen percent of the population is Hispanic. Forty-four percent of the study sample was employed at the start of Jobs Plus, and of those who were employed, their average annual earnings was $15,184. The proportion who are employed ranges from less than one-third in Charlotte, Memphis, and Cuyahoga County to 61 percent in New York City. Average earnings levels generally mirror the variation in local economies, with larger and more expensive cities having the highest average earnings. The majority of households have children under the age of 18 (73 percent), and just over one-half of households have at least one child in the household who is 5 years old or younger.

Program Participation

The Jobs Plus model is premised on the idea that a multi-component approach is more effective at helping residents make progress toward economic mobility than would be the case for single components in isolation. By design, the program does not target any subset of residents but intends for everyone living in a program housing development to be influenced by the program in some way. The aggregate participation data analyzed for this report show the extent to which residents engaged in Jobs Plus or its activities and services (selected participation outcomes are shown in exhibit ES.1). These data, submitted quarterly by the grantees, cover the first 3 years of the program and include all nonelderly (18–61), nondisabled residents, including those who moved into the Jobs Plus development anytime in the 3-year followup period after the program had launched (the impact analysis, described next, focuses on those between 18 and 57).

• By the end of Year 3 of the program, about one-half (or 52 percent) of all eligible residents had completed an initial Jobs Plus assessment, ranging from 26 percent to 79 percent across sites.

At a minimum, completing an initial assessment serves as a useful indicator of whether residents were exposed to (or made aware of) the services offered by the Jobs Plus program and the opportunity to benefit from the Jobs Plus Earned Income Disregard (JPEID), which required separate enrollment. On average, about one-fourth (or 26 percent) of all eligible residents in the development at the time the program launched had enrolled in Jobs Plus and completed an initial assessment by the end of the first year of followup. By the end of the third year, this percentage had increased to about one-half (or 52 percent). Reaching higher levels of enrollment in later stages of the 4-year program also means that there is less overall exposure to program services and incentives (due to the program’s fixed grant end-date and residents exiting the developments).

Post-enrollment followup with case managers remained relatively low over the followup period. During the second year of the program, on average across the sites, 19 percent of work-ES-5

able residents met with their case managers during a given quarter, and in the third year, this percentage increased slightly, to 22 percent (not shown). This quarterly average ranged from less than 10 percent in seven of the sites in the second year and six of the sites in the third year to 30

percent or more in five of the sites in both the second and third years.

The sites with the lowest Jobs Plus enrollment at the end of the followup period unsurprisingly had generally low levels of resident participation in employment services; however, the sites with the highest Jobs Plus enrollment rates did not necessarily have the highest levels of resident participation in employment services, though overall participation rates were relatively higher.

• On average, about 46 percent of the eligible residents in a development received some post-assessment employment services by the end of the third year of the program. Job search assistance was the most frequently provided type of employment service, followed by employment readiness.

All the study sites offered services aimed at helping residents obtain jobs, including job search assistance, job readiness programs, resume writing assistance, interview preparation, and job placement services. Nine reported providing employment services to over half of work-able residents by the end of the third year of implementation. On average, the increase in employment services receipt slowed down after the end of the second year of implementation. In addition to job search assistance, employment readiness assistance was the next most commonly used Jobs Plus employment service. Employment readiness programs provide training on work-related skills considered necessary to be successful in entry-level jobs in any sector, such as work habits and conduct, communication skills, and executive skills. Other employment support services—

including criminal records assistance, physical and behavioral health care assistance, childcare assistance, and transportation assistance—had lower receipt rates across all the developments.

• By the end of the first year of implementation, on average, 19 percent of work-able residents were in a household that was enrolled in the JPEID, and by the end of the third year, the enrollment rate had increased to 40 percent.

All eligible households in Jobs Plus developments have the opportunity to benefit from the JPEID for the duration of the program. After enrolling in the JPEID, any increases in earned income (by any household member) do not result in a higher tenant rent as long as the Jobs Plus program is in place. Note that a household’s enrollment in the JPEID does not indicate that the household received the earnings disregard. Nevertheless, the process of enrolling in the JPEID, at a minimum, reflects awareness of the JPEID and possibly signals an intention to increase earned income and benefit from the earnings disregard. JPEID enrollment rates varied widely across the developments, ranging from 76 percent to 10 percent by the end of Year 3 (see exhibit ES.1).

ES-6

Exhibit ES.1

Selected Jobs Plus Participation Outcomes

Cohorts 1 to 3

Participation

Average

Range

Program Participation

Completed assessment (%)

By the end of

Year 1

26

5–70

By the end of

Year 2

44

16–77

By the end of

Year 3a

52

26–79

Employment Services

Received post-assessment services (%)

By the end of

Year 1

19

2–38

By the end of

Year 2

38

16–55

By the end of

Year 3a

46

26–69

JPEID

Enrolled in JPEID (%)

By the end of

Year 1

19

0–46

By the end of

Year 2

36

7–78

By the end of

Year 3ab

43

10–76

Sample size (Grantees)

23

a Data from Baltimore are missing for the final quarter in Year 3. The total and rate over 2.75 years are included instead.

b JPEID data from New York C ity ar

e missing for the final quarter in Year 3. The rat

e over 2.75 years is included in

this average instead. JPEID data from Sacramento are missing for Year 3 and therefore are excluded from this average.

Notes: The table include s 23 of 24 gr

antees. Memphis data are excluded for reasons described in the report.

Data from Norfolk in Ye ar 3 ar

e exclu

ded bec ause of data issues.

Source: MDRC calculati ons usi ng HU D Jobs Plus Pilot Data Visualization Tool.

ES-7

This study also examined a set of indicators designed to capture resident participation in efforts related to building community support for work. Community support for work occurs both through formal activities and informal interactions between residents and Jobs Plus staff and community coaches, making formal metrics of this concept challenging. The grantees, however, reported to HUD the number of Jobs Plus events (any activities that expose residents to Jobs Plus and foster relationships among residents) held in the Jobs Plus developments, participation levels in those events, and the percentage of work-able residents meeting with community coaches every quarter. On average, 19 percent of work-able residents attended at least one Jobs Plus event in a given quarter in the second year, and 26 percent attended at least one of these events in a given quarter in the third year, though there was a range of attendance rates across PHAs.

Impacts on Employment and Earnings

The impact analysis estimated the effects of Jobs Plus on nonelderly, nondisabled residents—employment and earnings outcomes—during the first 4 years the program was operational at each site. These impacts are assessed for the pooled sample of all 24 sites, along with variation in impacts across sites. A majority of sample members (86 percent) in the Jobs Plus developments worked at some point during the 4 years of followup, and employment levels remained consistently high among those who were working at the time of program launch. On average, sample members in these developments saw their average earning increase by about $1,000 over the followup period.

• Average earnings in each of the 4 years of followup were very similar between the Jobs Plus and comparison groups: the difference is less than 2 percent in all 4 years, and no estimated differences are statistically significant.

On average, residents in comparison developments earned $10,451 in the first year after program start (see exhibit ES.2). This average includes zeros for adults who did not work during the year. About 72 percent of adults worked at some point during the first year, and the average earnings of these workers was $14,515. Average earnings for the comparison group of adults increased steadily by about $1,000 per year over the 4-year followup period; employment rates increased modestly from Year 1 to Year 2 and then remained relatively flat for the remainder of the 4-year followup period. Earnings followed a similar pattern for residents in the Jobs Plus developments, suggesting that the program through Year 4 had no impact on these outcomes.

Across the 16 quarters of followup, average quarterly earnings for Jobs Plus sites and comparison sites were very similar (not shown). Although average earnings of the Jobs Plus sites are slightly lower relative to the comparison group in the last three quarters of the 4-year followup, none of these differences are statistically significant.

ES-8

Exhibit ES.2

Impacts on Earnings and Employment in the 4 Years of Followup Focal Adults: Cohorts 1 through 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,374

10,451

– 78

0.441

Year 2

11,755

11,700

56

0.680

Year 3

12,909

12,939

– 29

0.865

Year 4

13,829

14,070

– 241

0.272

Years 1–4

48,236

48,535

– 299

0.604

Average Quarterly Employment (%)

Year 1

59.7

60.1

– 0.4

0.409

Year 2

62.1

62.0

0.1

0.877

Year 3

63.2

62.8

0.4

0.551

Year 4

63.0

62.9

0.1

0.937

Years 1–4

61.9

61.8

0.0

0.979

Sample Size (total = 19,267)

9,220

10,047

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were ages 18 to 57 and not identified as having a disability by the housing agency at the time Jobs Plus implementation started in their development.

Estimates are adjusted by site indicators.

The impact estimates for the two confirmatory outcomes (4-year total earnings and 4-year quarterly employment) were not statistically significant; therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

Source: National Directory of New Hires

• The average quarterly employment rate across the 4-year followup period is 62

percent for both the program group and the comparison group.

The quarterly employment rates for program and comparison group members during each year of followup are also very similar—they do not differ by more than one-half a percentage point in any followup year, and none of the differences are statistically significant. The patterns of estimated effects look very similar for annual employment rates (defined as employed during at least one quarter in a given followup year). Quarterly employment rates for Jobs Plus sites and comparison sites were very similar across the 16 quarters of followup.

There was some variation in impacts across sites for the study’s two confirmatory outcomes (cumulative earnings and average quarterly earnings) over the 4-year followup period.

Although the individual site-level estimates were largely not statistically significant, the variation in estimated effects across the 24 sites was statistically significant (p=0.047 for cumulative ES-9

earnings and p=0.018 for average quarterly employment). An exploration of the potential sources of this cross-site variation did not find evidence that participation levels (defined as JPEID

enrollment and participation in post-assessment employment services) were correlated with estimated program effects, nor was “completeness” of implementation an important factor for this set of grantees (as examined with an analysis that excluded the three sites that experienced major disruptions to their Jobs Plus implementation due to redevelopment or relocation of residents).

Additional exploratory analyses also do not provide evidence of differential impacts based on local unemployment rates, residents’ public housing tenure, baseline employment status, or grantee cohort. 10 An analysis exploring the relationship between site-level resident mobility rates and program effects did suggest that public housing developments with higher turnover rates had smaller program effects; however, a further analysis examining effects separately for only those residents who remained in a Jobs Plus development for at least 2 years of followup did not find that this group experienced larger effects on key outcomes than the full sample.

Finally, because the study sample was limited to the focal adult for whom data were available for the full followup period, sensitivity tests were conducted to assess whether the estimated program effects on average earnings and employment rates differed for all eligible adults and for all eligible adults for whom data were available in that year (regardless of whether data were available for them in other followup years). The findings were very similar to those for the main study sample: there was no evidence of effects on earnings or employment rates across the 4 years of followup for either alternative sample definition.

Conclusion

This report documents initial participation outcomes and labor market effects for the first three cohorts of grantees that implemented Jobs Plus as part of HUD’s scale-up of this program.

These early Jobs Plus grantees confronted an array of implementation challenges and had to learn and operate the program just as HUD itself was formalizing operational guidelines and administrative requirements for its grantees. 11 Nonetheless, the experiences and outcomes for the early cohorts to operate Jobs Plus provide important insights for the continued implementation and assessment of this program’s effectiveness.

The present evaluation for these first three cohorts did not find evidence of positive effects of Jobs Plus on earnings or employment rates during the 4 years of followup while the program 10 Sensitivity tests were also conducted to assess whether the estimated effects of Jobs Plus for the sample of all adults differed from the sample of focal adults, and whether, in each followup year, they differed for the sample of all eligible adults for whom data were available in that year (regardless of whether data were available for them in other followup years). The findings were very similar to those reported here.

11 Tessler et al., 2017; Verma et al., 2019.

ES-10

was operational. These findings stand in contrast to those from the original Jobs Plus demonstration, in which the three sites (of the six total) that implemented and sustained all three components realized positive and sustained effects on residents’ earnings levels. In the original demonstration, participation rates were also higher in the sites that were deemed to have implemented a stronger program. The present replication study did not find a meaningful correlation between program participation and program impacts, so it is possible that the lack of positive effects on employment and earnings outcomes is at least partially due to participation levels overall not reaching an adequate threshold that would lead to meaningful impacts.

From a continuous improvement perspective and drawing insights from the participation outcomes examined in this report, understanding how higher engagement can be attained and sustained in a place-based intervention is important both for strengthening the program and for providing residents with services that will best help them advance, and it could be beneficial for HUD to further assess how various components of the Jobs Plus framework are being implemented and where there is room for refinement.

The structure of the current Jobs Plus grants may also warrant some attention. The HUD

grantees receive 4-year, nonrenewable grants. Lower participation rates early in the grant period, which increased by the end of the second and then third year, suggests that grantees may need a longer startup period to put into place the infrastructure to operate the program at a steady state.

The sites in the present evaluation received funding for a total of 4 years, which covered startup, implementation, and wind-down, leaving a short timeframe for the sites to achieve a strong, sustained period of steady-state operations. Further, most of these grantees received standard funding levels, leaving the small and large sites with relatively comparable funding. This funding structure may also have resulted in sites making some tradeoffs between services and incentives, unlike the original demonstration.

Finally, it seems worthwhile for HUD to continue tracking the longer-term outcomes of the sample included in this present study and estimate the effects of the program on more recent cohorts of Jobs Plus grantees. It is possible new patterns of results could emerge as HUD

continues to make additional investments in program technical assistance and support grantees to implement stronger programs. The COVID-19 pandemic and its devastating economic shocks for families could also serve as another test for this program because Jobs Plus residents (unlike their counterparts in developments without Jobs Plus) have the support of their case managers to help them navigate personal and employment-related crises throughout the pandemic and access onsite services and supports that might help them take advantage of employment, education, and training opportunities as the economic and public health situation improves.

ES-11

Introduction

In the mid-1990s, the U.S. Department of Housing and Urban Development (HUD), the Rockefeller Foundation, and MDRC conceived the Jobs Plus demonstration to address the significant challenges to self-sufficiency that residents of public housing developments face. The Jobs Plus model encourages economic mobility by providing employment-related services, rent-based work incentives (so that initial earnings increases do not trigger immediate rent increases), and building a community environment that supports work (called “Community Support for Work”). The original Jobs Plus demonstration, which was the subject of a rigorous evaluation, found that developments that fully implemented the program model saw the program boost annual earnings by a substantial margin. 12 Subsequently, because the program was launched 20

years ago, it demonstrated sustained positive effects (where fully implemented) on residents’

earnings through the end of 7 years of followup, 13 and the Jobs Plus program has expanded across the country through local and federal replication efforts, most recently a large-scale replication through HUD.

In 2014, Jobs Plus became part of federal housing policy when Congress authorized $24

million for a Jobs Plus Pilot Program. 14 That program targeted public housing developments with a minimum of 200 nonelderly households in Cohort 1 and 250 nonelderly households in Cohorts 2 and 3 that demonstrated high levels of unemployment (at least 50 percent of the eligible households did not have wage earnings); and, because of the place-based nature of the program and to ease access for residents and staff, the distances between the units was also a consideration, disqualifying scattered sites that did not meet the required radius. 15 16

HUD began funding replication efforts in April 2015. The first nine public housing agencies (PHAs) in this cohort of replication sites received funding that ranged from $1.9 million to $3 million, with each PHA leveraging at least 25 percent of its grant through matched contributions and leverage. 17 To date, HUD has awarded about $136 million through 56 grants to 12 Bloom, Riccio, and Verma, 2005.

13 Riccio, 2010.

14 Funding for this program is authorized by the Consolidated Appropriations Act, 2014, Public Law 113-76, 128

Stat. 5, enacted January 17, 2014.

15 HUD’s definition of an elderly household is one where the head of household, spouse, or co-head is age 62 or older.

16 The minimum number of households requirement dropped from 250 for Cohort 1, to 200 for Cohorts 2 and 3.

Furthermore, the initial requirement for units to be within a ¼ mile radius was removed for Cohort 2, and Cohort 3

applicants had to demonstrate that the program could be successfully operated in non-contiguous developments. See HUD 2014–16 NOFAs.

17 The PHAs in Cohorts 2 and 3 also received funding within this range. Grantees are required to have a match contribution equivalent to at least 25 percent of their total grant amount. Commitments beyond 25 percent are considered leverage, and the match/leverage may be provided as a cash or in-kind donation.

1

50 public housing agencies (PHAs) to implement Jobs Plus. Each of these grants was awarded for a 4-year period and is nonrenewable. 18

To understand whether Jobs Plus is effective in the long run—when the model is widely replicated—HUD sponsored an outcomes evaluation in 2017 for the first three cohorts of PHAs (including 24 grantees) that received funding and selected MDRC to lead the evaluation (see exhibit 1 for evaluation components). This report presents the estimated impacts for these Jobs Plus programs on residents’ employment and earnings for the 4 years after program start.

Estimates of program impact are based on a matched comparison group design. Prior to the evaluation start, HUD selected one or more local area public housing developments similar to the Jobs Plus development to serve as comparison sites. This report assesses the match between the Jobs Plus and comparison sites and then uses that design to estimate program impacts.

Exhibit 1. Jobs Plus Outcomes Evaluation

The evaluation is structured around the following analyses:

 A baseline description of the housing developments in the Jobs Plus Outcomes evaluation, the characteristics of the residents living in targeted housing developments at program launch, and their early program participation patterns.

 The early effects of Jobs Plus on residents’ employment and earnings for the first three cohorts of PHAs awarded 4-year grants to operate Jobs Plus. This analysis relies on quasi-experimental methods to assess Jobs Plus impacts.

 The relationship between the Jobs Plus Earned Income Disregard (JPEID) and resident employment and earnings patterns. This analysis is featured in an appendix in this report.

 The long-term earnings gains of the residents in the original Jobs Plus demonstration and whether the gains in residents’ earnings from the original Jobs Plus demonstration translate into long-term improvements in their children’s employment and earnings. These results are included in a standalone report.

Although MDRC’s original evaluation provided credible evidence of the effectiveness of Jobs Plus, those findings pertain to impacts on a cohort of residents from a small number of sites almost 20 years ago, when labor market and policy environments were different from those faced by public housing residents today. The pressures of automation, stagnant wages, and the need for postsecondary credentials to attain decent-paying jobs have all intensified. Workforce and welfare policies have also changed. The Workforce Investment Act of 1998 (WIA) has evolved to the Workforce Innovation and Opportunity Act (WIOA), Temporary Assistance for Needy Families (TANF) has transformed welfare and now provides cash assistance to a much lower proportion of public housing residents (and low-income families in general), and community colleges have become a more prominent part of the workforce system. Employment interventions for recipients of Supplemental Nutrition Assistance Program (SNAP) (received by 18 See HUD’s website (https://www.hud.gov/program_offices/public_indian_housing/jpi) for award announcements.

2

a majority of subsidized tenants) have grown. Affordable housing is in shorter supply.

Furthermore, the replication of the program model through HUD’s current Jobs Plus expansion is in the hands of a broader set of actors, in many more locations, with a wider variety of local adaptations. 19 All of these changes may have a bearing on the model’s effectiveness, and its success therefore cannot be taken for granted. Furthermore, replication of social programs at a larger scale in varying contexts is notoriously difficult, with many programs that showed promising effects in pilot studies failing to reproduce positive impact findings when replicated at a larger scale. 20

This report, which focuses on the HUD Jobs Plus replication sites, begins by providing a brief backdrop on the policy context and origins of Jobs Plus and the existing evidence about program implementation and impacts on labor market outcomes. The report then describes characteristics of the Jobs Plus developments in the present analysis and the local context at the time Jobs Plus implementation started, including characteristics of the residents living in the housing developments studied. The report then describes baseline characteristics of the study sample in those developments. Next, it uses data reported by the Jobs Plus grantees to HUD to examine resident participation in Jobs Plus activities. The report then presents findings on the impacts of Jobs Plus on residents’ average earnings and employment rates, overall and for selected subgroups of residents, and describes how the effects vary across the 24 PHAs. The report closes with a summary of findings and a discussion of the implications of the present study’s results for the Jobs Plus expansion. Relevant appendixes with supporting analyses are referenced throughout the report.

Brief Policy Context and Prior Research

Families who receive federal housing subsidies are among the poorest and most disadvantaged households in the United States. The federal government helps more than 2

million of these families meet their monthly rental needs, primarily through public housing and housing vouchers. In addition, the public housing system, through employment-focused programs such as the Family Self-Sufficiency Program21 and Jobs Plus, for example, is used to encourage work, increase earnings, and provide families with pathways out of poverty. Some 19 Verma et al. (2019) capture these adaptations for the Cohort 1 sites.

20 The case of the San Jose-based Center for Employment and Training (CET), originally part of two separate small-sample Randomized Controlled Trials (RCTs) in the 1980s (one by MDRC and one by Mathematica), is illustrative.

Although both studies found positive impacts on employment, a five-city replication test sponsored by the U.S.

Department of Labor (DOL) and conducted by MDRC did not.

21 Nationally, close to 80,000 individuals participate in HUD’s Family Self-Sufficiency (FSS) program, though most participants are receiving housing assistance through the Housing Choice Voucher program (and public housing residents are a smaller share of this program). A national evaluation of this program is underway but focuses on FSS

participants in the Housing Choice Voucher program. See Verma et al. (2021).

3

critics, though, have also argued that subsidized rent policies themselves discourage work. They say that the current income-based rent subsidy system, which protects residents from excessively burdensome rents, may also discourage recipients from trying to increase their earnings. Such concerns have prompted housing agencies to experiment with traditional rent rules—such as relaxing the penalties on higher incomes—to see if that can increase work and reduce the need for housing subsidies. 22 Jobs Plus, the subject of this report, combines both employment-focused supports and services with work-based rent incentives, with the aim of promoting participants’

economic self-sufficiency.

The Jobs Plus Model

Jobs Plus was designed as a place-based response to the many challenges public housing residents face in improving their employment situations: many developments are located in areas of concentrated poverty, and residents often struggle with poor work histories, limited education, lack of adequate childcare, health or medical problems (including substance abuse), and worry about crime and safety in their neighborhoods. The program’s theory of change includes a set of components shown to be promising by prior research in welfare reform and other fields. It was hoped that combining them in a single model would be mutually reinforcing and make Jobs Plus a more powerful intervention. 23 These components are (1) onsite employment-related services and activities, (2) rent-based financial incentives, and (3) community support for work.

Onsite employment-related services include job search assistance, referrals to education and training programs, and support services coordinated by staff located within the development to make it easier to engage residents and to make Jobs Plus staff more a part of the community they serve. Rent-based financial incentives are designed to allow families to keep more of their earnings (that is, “make work pay”) when they enter work or increase their earnings, allowing them to see a bigger financial return from work. Influenced by a growing recognition of the importance of social networks and social capital, the designers of the Jobs Plus model envisioned a community support for work (CSW) component that emphasized resident-to-resident outreach, information sharing, and mutual support, as well as connections to potentially instrumental individual and institutional networks outside the development.

Another unique feature of Jobs Plus is that it is intended to operate at saturation levels—

that is, to offer services, incentives, and community support for work to everyone living in the development. A place-based program has the potential to benefit residents in a development beyond the personal benefit of receiving employment and other support services through the program and through the rent incentives. Thus, the model assumes that residents who formally 22 HUD has launched the Rent Reform Demonstration to test the effects of such a strategy (see Riccio, Verma, and Deitch, 2019).

23 See Bloom, Riccio, and Verma, 2005; Blank and Wharton-Fields, 2008; and Greenberg et al., 2015.

4

enroll in Jobs Plus and receive services or enroll in the rent incentives can influence other residents in the development without formal involvement in the program through strengthening and leveraging social networks among residents, sharing information about employment opportunities and availability of services in the community, and encouraging residents to attend Jobs Plus events that may benefit residents. Engagement and participation cannot be measured directly as they are with non-place-based programs serving individuals, which is important to keep in mind when interpreting measures of formal enrollment and participation in Jobs Plus and related activities.

Since Jobs Plus was first launched in the 1990s, each replication has sought to preserve the original framework while making modifications to incorporate lessons and adapt the framework to new local contexts—thus keeping intact the inspiration, theory, and rationale that shape this program. 24 See exhibit 2 for a timeline of replications. 25

24 One departure from the original demonstration was the role of the mandatory collaborative (including the PHA, resident representatives, and local human services and workforce development agencies) and its governance function. Consistent with the spirit of collaboration, however, HUD required PHAs to establish formal service delivery partnership agreements with the local Workforce Development Boards (WDBs) and American Job Centers (or One-Stop Career Centers) and encouraged partnerships with other social service agencies within the community.

25 MDRC’s past publications on the Jobs Plus program can be found at this publication page on the MDRC website.

5

Exhibit 2

1990s

JOBS PLUS PROGRAM DEVELOPED

The Evolution of Jobs Plus

Founded by the U.S. Department of Housing and Urban

Development (HUD), The Rockefeller Foundation, other

funders and foundations, and MDRC to address the

growing concentration of joblessness, underemployment,

and poverty in public housing communities.

1998–2006

TESTED IN SIX CITIES

An MDRC study of six public housing authorities, located in

different housing and labor markets, found that nondisabled

and working-age residents in the three developments that

fully adopted Jobs Plus earned 16 percent more than

residents in comparison developments in the same cities.

2005–2009

REPLICATED IN QUEENS AND EAST HARLEM (NYC)

Urban Upbound implements the model in Queens. Its success

(along with that of the original demonstration) provided

impetus to New York City to replicate the program in in East

Harlem.

2011

REPLICATED IN TWO MORE CITIES

The Mayor’s Fund to Advance New York City and NYC’s

Center for Economic Opportunity used Social Innovation

Fund money to replicate Jobs Plus in the Bronx and San

Antonio. MDRC conducted an implementation study that

offered analyses of program experiences and costs.

2013

EXPANDED IN NEW YORK CITY

The city of New York announced a $24 million investment in

Jobs Plus, increasing the program’s reach into 23 of the city’s public housing communities.

2015–Present

HUD LAUNCHES NATIONWIDE REPLICATION

HUD has awarded Jobs Plus grants to 50 housing authorities

across the country. MDRC is evaluating the effects of Jobs

Plus at the first 24 sites.

6

Prior Evidence of the Jobs Plus Program’s Effectiveness

The MDRC evaluation of the original Jobs Plus demonstration showed that Jobs Plus increased earnings for the six-site pooled sample by a small but statistically significant amount over the 4 years following the end of the rollout period. The earnings effects were large for three of the six sites (Dayton, Los Angeles, and St. Paul) that had fully implemented the essential elements of the model; a fourth site, Seattle, also had positive effects until it started implementing its HOPE VI revitalization grant. These sites also had positive effects across a range of subgroups. A longer-term analysis found that the positive effects were sustained even 3

years after the Jobs Plus program ended. 26 Across the full 7-year period of the analysis (4 years during program and implementation and 3 years after the program ended), Jobs Plus households in these three sites experienced a gain in average annual earnings of 16 percent, which translates into an average gain of $1,300 per year (in 2003 dollars, equivalent to about $1,800 in 2019

dollars).

Part of the original Jobs Plus evaluation also included a development-level analysis, which examined whether the program’s positive effects on residents’ earnings, for example, are reflected in corresponding changes in their public housing developments. This focus on development-level effects considers the role of residential mobility and the ways in which program effects on individual residents might translate into broader neighborhood effects within public housing developments. This analysis showed that when no effects are evident for public housing residents, no effects are produced for the housing developments in which they live. The same three sites with positive effects for residents produced positive effects at the development level, but the magnitude of the development-level effects varied according to the sites’ resident mobility rates—which, in turn, were related to the tightness of the local private rental housing market. In particular, the development-level impacts were lowest in Dayton, where tenant mobility (and access to affordable private rental housing) was highest.

Urban Upbound, a community organization in Queens, New York, was the first organization to replicate the Jobs Plus model after the demonstration ended. 27 It was adapted for the Queensbridge Houses beginning in 2005, though without the rent incentives. Building on the work by Urban Upbound, the City of New York replicated Jobs Plus, initially at a public housing development in East Harlem and later in the Bronx as part of a Social Innovation Fund (SIF) initiative that also included two public housing developments in San Antonio, Texas.

New York City has since expanded the program to eight communities involving 23 New York City Housing Authority (NYCHA) developments. 28 Unlike the sites in the original demonstration, the sites in the Jobs Plus expansion in NYC as part of the SIF did not have the 26 Riccio, 2010.

27 Urban Upbound was named the East River Development Alliance at the time they began implementing Jobs Plus.

28 Not including the development covered by the recently awarded HUD grant.

7

authority to change their rent rules (as was the case in the original Jobs Plus) and had to rely on the existing Earned Income Disregard (EID), which was both restrictive and administratively challenging to implement. 29 This resulted in low takeup of the rent incentive. In the national Jobs Plus expansion, HUD directly addressed this limitation with the Jobs Plus Earned Income Disregard (JPEID, discussed below.) Also, unlike the original Jobs Plus demonstration, in which the Jobs Plus program office was on site and the program was run by the housing agencies, the Jobs Plus programs in the NYC expansion effort were operated by nonprofit agencies that were under contract with the NYC Department of Human Resources Administration (the city’s social services agency). Each nonprofit agency served multiple housing developments, and most of the time the Jobs Plus office was at a nearby location and not on site at the development. Further, residents had to officially enroll in the program and sign a membership agreement, which covered program requirements and expectations from participants. Those who did not sign up for Jobs Plus were not offered program services.

The Urban Institute conducted an evaluation of the New York City Jobs Plus replication, focusing on seven sites that began implementing the initiative between 2013 and 2014. 30 It found that Jobs Plus participants (i.e., those who formally enrolled in the program) had significantly higher employment rates and average earnings in the year and a half following program entry compared with other eligible residents in the same developments who had not (yet) enrolled. The analysis controlled for several demographic and employment characteristics, but it is uncertain how much of the estimated difference is attributable to selection bias (for example, at least hypothetically, residents who enrolled in Jobs Plus early or at all may have been more motivated to improve their employment situation than residents who enrolled much later or didn’t enroll at all) rather than program effects.

In 2017, as part of the present Jobs Plus Outcome evaluation, HUD also commissioned a two-part long-term impact analysis, focusing on the adults and children residing in the developments when the original Jobs Plus program was launched in 1998. The long-term impact analysis, which follows sample members even after they may have left the public housing development and HUD assistance, considers two questions: (1) whether the significant impacts observed on the original adult residents’ earnings due to the implementation of Jobs Plus—and which did not show signs of fading during the full 4 years of implementation or the 3 years following completion of the program—were sustained 15 years after the end of the intervention; and (2) whether the program resulted in long-term improvements in children’s employment and earnings when they became adults, testing the possibility that an effective place-based 29 The six housing agencies in the original Jobs Plus demonstration were granted Moving to Work status, giving those housing agencies the flexibility to change their rent rules for the Jobs Plus program, allowing them to implement flat rent and lower percentage-of-income tenant rent shares as rent incentives.

30 Leopold et al., 2019.

8

employment intervention can also spur intergenerational effects and improve the economic well-being of children growing up in these developments. 31

The results from these analyses, to be published shortly, suggest that Jobs Plus continued to positively affect the earnings of the adult residents in the stronger implementation sites 15 years after the program ended. 32 The estimated effect on average annual earnings was $1,670 in 2018

dollars, or an 11-percent increase. These long-term earnings gains can be at least partly attributed to increased employment rates: the Jobs Plus group had an average quarterly employment rate of 54.3 percent, which is 4.2 percentage points higher than the comparison group during the same time period. The children living in the Jobs Plus developments at the time that the program was implemented also experienced higher earnings and employment in adulthood compared with their comparison group counterparts. The findings for children are exploratory, meaning that they were not the key questions for the study but were examined to generate hypotheses for future research.

Nonetheless, the implications—if the findings are replicated—are important. Overall, the long-term findings emphasize the importance of the robust implementation of the Jobs Plus program because they provide evidence that when well implemented, it can serve as a platform for producing lasting economic gains for adults and children.

Implementation Experiences

The early implementation experiences of New York City and San Antonio, along with lessons from the original Jobs Plus demonstration, helped to inform HUD’s approach to replicating Jobs Plus around the country. HUD also commissioned a process study, led by MDRC, to understand the early operational experiences and participation outcomes for the first nine PHAs (Cohort 1) to receive grants as part of its Jobs Plus replication effort. 33

Early Jobs Plus Replication Initiatives

MDRC’s evaluation of the Jobs Plus implementation as part of the SIF initiative in three New York City developments (all in the Bronx) and two developments in San Antonio revealed both challenges and accomplishments in efforts to replicate the model. In both cities, the program succeeded in engaging a high proportion of residents. At the same time, residents’

31 Miller et al. (forthcoming)

32 The study uses ordinary least squares to examine the long-term effects of Jobs Plus on labor market outcomes. This methodology leverages the close match in baseline earnings and employment levels and trends in the 6 years leading up to Jobs Plus implementation to compare the labor market outcomes for the Jobs Plus program group with the outcomes of the comparison group about 15 years after the program ended in 2003 and includes covariates to further increase the precision of the estimates and control for differences between the Jobs Plus and comparison groups at baseline. The methodology differs from that of the early impact analysis of the Jobs Plus demonstration, which used comparative interrupted time series, but was not feasible for the long-term analysis because the 6 years of pre-Jobs Plus employment and earnings data could not be used to reasonably predict employment and earnings levels and trends 20 to 21 years later.

33 Verma et al., 2019; Tessler et al., 2017.

9

formal engagement with the program tended to trail off, at least during the study’s 3-year observation period. Moreover, as in the original demonstration, it took time for the SIF sites to ramp up program implementation, especially EID enrollment, and it was not until Year 3 that the program was considered to have reached steady-state operations. Implementing rent incentives through the EID structure presented challenges because both housing agency staff and residents found the incentive confusing: it was administratively difficult for PHA staff to implement, and residents often questioned the value of the incentive (it is limited to 2 years, with 100 percent of increased earnings “disregarded” for rent calculations in the first year and 50 percent in the second year). 34 Over time, the sites improved communication around the EID and administratively integrated the EID into housing agency processes, which slightly increased enrollment in the EID. The San Antonio Housing Authority (SAHA) is an MTW agency, so it had the authority to change its rent rules. They modified the EID to simplify and extend it to create a longer-term earned income disregard. 35

In the Urban Institute evaluation of the New York City expansion, although sites initially met their enrollment targets, they achieved only about a 32-percent “saturation level” (as defined by the study, the percentage of households who had at least one member who enrolled in Jobs Plus) on average across the seven sites. Participants who completed interviews expressed that they had very positive experiences with the Jobs Plus program. They appreciated the employment services offered and had positive interactions with program staff. Similar to the experiences of the SIF sites described above, the program in these sites experienced some challenges implementing the rent incentive within the structure of the existing EID and did not have the flexibility that MTW agencies had to make the EID more accommodating to Jobs Plus. The study concluded that participants’ concerns about their rent increasing when their earned income increases may have discouraged some from pursuing opportunities to increase their earnings. 36

HUD Jobs Plus Replication

The HUD-funded process study focused on the first nine public housing agencies to receive funding for Jobs Plus and documented their implementation experiences for the first 3 years of the grant. Similar implementation information is not available for subsequent Jobs Plus grantees in the current evaluation, so whether some of the implementation experiences documented for the Cohort 1

grantees can also be generalized to later cohorts of Jobs Plus grantees is unclear. Regardless, the 34 The “stop the clock” feature of this benefit, which allowed residents to stop and resume earning the EID, was also difficult for PHAs whose systems were not programmed to capture such changes.

35 See Greenberg et al., 2015. At the end of the study’s followup period, about 1 percent of residents in the Bronx and 3 percent in San Antonio have received the EID. SAHA developed a Simplified EID, which was meant to be less complicated for program staff to understand and apply. It extended the EID’s 2-year benefit to an uninterrupted 5 years, implementing a 25-percent earned-income disregard during Year 3, a 20-percent disregard in Year 4, and a 10-percent disregard in Year 5. The incentive’s designers reasoned that this simplified extension would encourage residents to stay employed during this period of time.

36 Leopold et al., 2019.

10

Cohort 1 process study yielded important insights and observations about the sites’ early implementation experiences and provides useful context for thinking about program participation outcomes. 37 Some observations from the process study include the following:

• Employment services: Sites were required to partner with the local Workforce Development Boards and American Job Center(s) to offer employment-related services to residents with a range of employment needs. Most sites offered an array of employment services, including job preparation and job search services, along with some occupational skills training. In the first year of the program, most services were relatively generic, but by the second year of the program, they were being tailored to meet the specific needs and skills of individual participants. The study found that residents’ formal engagement with Jobs Plus case managers or program staff was up to that point largely resident-driven, and that most sites were not setting expectations about the frequency with which case managers should meet with Jobs Plus participants. During the first couple years of the program, only a few sites had developed strong relationships with their workforce development agencies, a mandated partner, and those that did found that the relationship often added little value to the Jobs Plus program. 38 Sites generally began working with job developers—either hired by the program or through partnerships—in the second year of the program, which sites found helped residents find jobs better matched to their skills and interests compared with the initial period of the grant, before the job developers had been hired. Overall, the Cohort 1 sites experienced moderate levels of participation in employment services, though there was a lot of variation in the proportion of eligible residents who participated in employment services across the nine sites.

• Jobs Plus Earned Income Disregard implementation (JPEID): A generous financial incentive, the JPEID disregards 100 percent of any additional earned income throughout the entirety of the Jobs Plus grant period. Thus, once residents formally sign up for the JPEID, their rent contributions do not rise following an increase in household earned income, thereby removing the “tax” on increased work effort. For the Cohort 1 grantees, participation data showed that the JPEID may have generated significant resident interest in Jobs Plus and connected them to Jobs Plus services. These sites, however, reported facing various challenges implementing the JPEID, especially with enrollment procedures, calculating the JPEID, and collecting and reporting the required data.

Several Cohort 1 sites struggled to develop data systems to track and report JPEID

outcomes. Staff across all sites also expressed confusion about implementing this component, partly stemming from mixed guidance from HUD on the JPEID and how to 37 Tessler et al., 2017; Verma et al., 2019.

38 Many Jobs Plus participants could not pass the screening requirements to be eligible for Workforce Development Board-funded training, for example, which usually involved tests of academic proficiency at certain grade levels.

The workforce agencies’ concerns about meeting performance standards could have contributed to their more limited involvement with Jobs Plus.

11

put it into practice (for example, the need for a separate JPEID enrollment process, whether automatic enrollment for the JPEID was permitted [a decision HUD reversed], or which income sources were subject to exemptions). HUD guidance and communication around the JPEID improved in the second year of implementation.

Enrollment in the JPEID varied widely across the Cohort 1 sites, though, with 6 to 40

percent of households having enrolled after about 2 years of implementation, reflecting some of the early startup challenges.

• Community Support for Work (CSW): The CSW component aims to strengthen social networks within public housing communities so that residents can support each other in engaging in meaningful work activities. Although the sites understood the importance and the goals of CSW, as with the original demonstration, operationalizing the component and putting it into practice was harder. Most sites had implemented a range of CSW activities and strategies, generally falling under five categories: community coaches; Jobs Plus enrollment, recruitment, and information-sharing activities; community events; social support and network building; and bridging to the broader neighborhood and metropolitan area. One challenge that many sites had with implementing their strategy of using community coaches (residents who were given the responsibility of promoting the program within the development and encouraging their neighbors to use Jobs Plus services to improve their employment situations) was that those positions remained task-oriented, with the coaches focusing on discrete assignments rather than organically fostering relationships among residents and connections to the Jobs Plus program. Overall, CSW emerged as an effective strategy to promote program recruitment and share information about Jobs Plus services. Its power for influencing residents’ connections to work activity and to each other remained relatively untapped at the end of the process study’s followup period. Staff across study sites were also looking for ways to engage a broader group of stakeholders—to supplement coaches—to plan and implement CSW. HUD served as a resource and provided examples of ways in which the sites could strengthen their CSW strategies and also encouraged them to procure technical assistance, if needed.

The 4-year, time-limited nature of the grant also did not leave sites much time for a steady-state period of operations and continuous improvement. The general startup and rollout period took at least a year, lasting longer for some sites. More structured technical assistance around the building blocks of the program would have enabled sites to learn and more quickly launch their programs, with a clearer understanding of the model and the types of implementation challenges to anticipate before beginning to wind down the program.

Later sections of this report will compare all three cohorts on selected engagement and participation outcomes across these three core components of Jobs Plus.

12

The Jobs Plus Outcomes Evaluation

The evaluation of the Jobs Plus Replication includes an analysis of residents’ participation in each of the three components of the program and an impact analysis that assesses the effects of Jobs Plus on residents’ employment and earnings. The two confirmatory outcomes for this evaluation (representing the key hypotheses to be evaluated for the study) are cumulative earnings and average quarterly employment over the 4-year followup period. 39 Program effects are examined separately for each year in the followup period as well. The evaluation focuses on the first three cohorts of grantees, including nine PHAs in Cohort 1 (grants awarded in April 2015), nine PHAs in Cohort 2 (grants awarded in December 2015) and six PHAs in Cohort 3

(grants awarded in September 2016). Exhibit 3 illustrates the timeline of each cohort’s award date and the 4-year followup period. As described later, the followup period mostly predates the onset of the COVID-19 pandemic.

This section describes the study sample definition, core data sources for the present evaluation, the matched-comparison design used for the impact analysis, and the statistical methods used to estimate program impacts.

39 The quarterly employment rate is the percentage of individuals employed in a given quarter. The average quarterly employment rate for the 4-year period is an average of employment rates over the 16 quarters in the followup period.

13

Exhibit 3. Grantee Timelines by Cohort

Grant

2014

2015

2016

2017

2018

2019

2020

Timeline Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

Cohort 1a

Baseline

Year 1

Year 2

Year 3

Year 4

Cohort 2b

Baseline

Year 1

Year 2

Year 3

Year 4

Cohort 3c

Baseline

Year 1

Year 2

Year 3

Year 4

a Cohort 1 funding award announced April 2, 2015

b Cohort 2 funding award announced December 17, 2015

c Cohort 3 funding award announced September 27, 2016

Source: HUD Program Offices (hud.gov/program_offices/public_indian_housing/jpi) 14

Sample Definition

The impact analysis focuses on residents in the Jobs Plus and comparison developments who are between the ages of 18 and 57 at program launch (when the grant was awarded to the PHA) and who are not identified as having a disability in the housing agencies’ Form 50058

data. 40 In households in which there is more than one household member who meets these age and disability status criteria, one member—called the “focal adult” in the remainder of this report—is identified so that the analysis is limited to one adult per household. 41 The “focal adult”

designation is assigned to the head of household, or the spouse or co-head if the head of household did not meet the age and disability status criteria. If neither the head of household nor the spouse met these criteria, the focal adult was selected at random from other eligible adults in the household. The analysis sample is also restricted to adults ages 18 to 57 without a disability for whom data were available for all 4 years of followup. 42 43

Data Sources

This report relies on five data sources to describe the Jobs Plus developments and their residents, assess participation in Jobs Plus activities, and estimate program impacts.

• HUD Picture of Subsidized Households. HUD Picture of Subsidized Households data are publicly accessible on HUD’s website and include aggregate information on public housing developments, the characteristics of residents in the developments, and some local labor market characteristics for the area in which the development is located. This data source is used to describe the Jobs Plus developments in the study as well as the developments selected for the comparison group. These data are reported annually and present a snapshot of the housing developments on the last calendar day of that year. The HUD Picture of Subsidized Households data also include data from two external sources that were used in this report:

40 Using this definition, the evaluation excludes residents who will become elderly—defined by HUD as 62 years old or older—during the 4-year followup period.

41 One focal adult per household was selected for the analysis because, for confidentiality reasons, the Office of Child Support Enforcement (OCSE) does not allow the National Directory of New Hires data—which the study relies on—to be matched with data identifying specific individuals or which individuals live in the same household.

This follows the precedent of HUD’s Rent Reform Demonstration, which conducted analyses for one person (the household head) per household. As a sensitivity test, impacts on study outcomes were also estimated for the full sample of all adults, and findings were very similar.

42 MDRC did not receive NDNH wage data from HUD for approximately 11 percent of the full eligible sample for the first 2 years of followup. Due to limitations on the amount of historical NDNH data that are maintained by OCSE, these data could not be recovered.

43 Sensitivity tests were conducted to assess whether the estimated effects of Jobs Plus for the sample of all adults differed from the sample of focal adults, and whether, in each followup year, they differed for the sample of all eligible adults for whom data were available in that year (regardless of whether data were available for them in other followup years). The findings were very similar to those reported here. The results of these sensitivity tests are discussed later in the report.

15

1. Census measures for the census tract in which the housing development is located. The census measures are based on the 5-year estimates from the American Community Survey from 2 years prior to the snapshot date. This report uses the snapshot year closest to the grantee award date to describe the developments: 2014 data are used to describe Cohort 1, whose Jobs Plus award was granted in April of 2015; 2015 data are used for Cohort 2, which has a December 2015 grant award date; and 2016 data are used for Cohort 3 (with a September 2016 grant award date).

2. Bureau of Labor Statistics (BLS) data, including Local Area Unemployment Statistics, which include information on county and metropolitan area unemployment rates that are used in this study to provide context for the economic environment when Jobs Plus was launched.

• HUD IMS/PIC. Individual-level baseline demographic data come from the HUD

Inventory Management System (IMS)/PIH Information Center (PIC). This data source is used to describe the study sample (i.e., residents who are between 18 and 57 years of age and do not have disability status) living in the Jobs Plus developments at the time that Jobs Plus was launched and their households. For each of these “eligible” sample members, household-level data and individual-level data on household members were used from the household’s most recent certification (e.g., annual recertification, interim recertification) prior to Jobs Plus launch.

• Grantee reports to HUD. Measures of participation in Jobs Plus activities were created using aggregate data that Jobs Plus grantees report to HUD on a quarterly basis. These data are posted on the online Jobs Plus Data Visualization tool designed and monitored by Abt Associates, the technical assistance provider for the demonstration. Grantees report measures such as current counts of work-able residents (defined as ages 18 to 61

and not having a disability) in the development, Jobs Plus enrollment, JPEID enrollment, and participation in post-assessment services, and the quarterly counts include those that moved into the Jobs Plus development after the program had already launched. For this report, data were available for the first 3 years of Jobs Plus implementation for all three cohorts in the study. Data for Memphis were excluded from the participation analysis due to its unique circumstance of implementing a Choice Neighborhoods grant during the implementation period of Jobs Plus. 44 As part of the Choice Neighborhoods implementation, all the residents of the Jobs Plus development were relocated, and subsequent reporting of resident participation in Jobs Plus activities was compromised.

44 Other isolated data quality issues were assessed on a site-by-site basis.

16

• National Directory of New Hires (NDNH). This data source is used to measure key labor market outcomes for the evaluation. NDNH data provide quarterly employment and earnings information for sample members, both preprogram and over the followup period. These data include quarterly wage data for workers in employment covered by the unemployment insurance (UI) system. These data miss employment that is not covered by the UI system, including informal work and self-employment. This report uses NDNH

data from Q3 2014 to Q3 2020, covering three to nine quarters of preprogram data (the maximum preprogram data available), depending on the cohort, and 16 to 22 quarters of followup data. Program impacts are estimated for 4 years of followup for all three cohorts. Preprogram data are used to assess the quality of the match between the replication and comparison sites and as covariates in the impact model.

Exhibit 4 presents the followup period covered by each of these data sources. As shown, the followup period mostly predates the onset of the COVID-19 health pandemic, which triggered unprecedented economic shocks around the country starting around March 2020. The last two quarters of the fourth year of followup for the third cohort, however, coincide with the initial months of the pandemic.

17

Exhibit 4. Data Sources by Cohort

2014

2015

2016

2017

2018

2019

2020

Data

Sour

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

ces

Picture

of

Cohort 1

Cohort 2

Cohort 3

Subsidi

zed

Househ

olds

HUD

IMS/PIC

All Cohorts

NDNH

Data

All Cohorts

Grantee

Reports

All Cohorts

Source: MDRC summarization based on data from U.S. Department of Housing and Urban Development (HUD) Picture of Subsidized Housing data, HUD

Inventory Management System (IMS)/PIH Information Center (PIC) data, National Directory of New Hires, and participation data reported by grantees in the HUD

Jobs Plus Pilot Data Visualization Tool

18

Matched-Comparison Design

The study relies on a matched-comparison design to assess the effects of the Jobs Plus program. In the design, earnings and employment outcomes for Jobs Plus and comparison developments are compared to assess the impacts of Jobs Plus. The selection of comparison developments (developments that are not implementing Jobs Plus but share similarities with Jobs Plus developments) to represent counterfactual outcomes for Job Plus developments is of vital importance. This is because the validity of Jobs Plus impact estimates rests heavily on the extent to which outcomes of residents in the comparison developments represent what outcomes for Jobs Plus residents would have been without Jobs Plus.

Prior to the present evaluation, HUD implemented a rigorous process to identify comparison developments, limiting their search to non-Jobs Plus developments within Jobs Plus PHAs that met the Jobs Plus eligibility requirements. Among this group, HUD used statistical models to match comparison developments to each Jobs Plus replication site on the basis of employment and earnings for adults ages 18 to 64 without a disability for the 2 years prior to the start of Jobs Plus. Given that the key outcomes of interest are employment and earnings after the launch of Jobs Plus, matching on preprogram versions of these outcomes is critical for ensuring a good match. Among the developments that most closely matched the Jobs Plus site on pre-Jobs Plus employment and earnings, one or more comparison sites were selected on the basis of how well they matched the Jobs Plus sites in terms of selected characteristics of the developments (e.g., size) and their residents (e.g., percent working and percent with children).

An important first step for the present evaluation was to assess the success of the match process completed by HUD and to refine the group of comparisons sites, if necessary. 45 The analyses conducted to assess the match, or the “match validation analysis,” is detailed in appendix A. Unlike the HUD analysis, which was based on employment and earnings captured in housing data (PIC), the present match validation analysis uses NDNH employment and earnings data to better align with the employment and earnings outcomes that are used in the impact analysis, which are created using NDNH wage data. Several statistical tests were used to assess how well preprogram employment and earnings matched for residents in the Jobs Plus replication sites and those in the comparison sites. This analysis was conducted for the pooled sample of all 24 PHAs as well as within each PHA.

Results from these tests indicated that the selection process that HUD used to identify comparison sites for the present study yielded a comparison group sample that was overall a good match for the Jobs Plus replication sites. MDRC further refined this sample by dropping eight developments that did not match their respective Jobs Plus developments as well as the 45 This step was built into the current evaluation.

19

other developments in the same public housing agency. 46 As discussed in appendix A, dropping these developments improved the overall match quality.

Impact Analysis Approach

Program impacts are estimated using a hybrid random effects and fixed effects statistical model that leverages the data from the 24 replication sites to provide estimates of the overall average effectiveness of Jobs Plus, as well as an understanding of how these impacts vary across sites. The model, described in more detail in appendix B, is a hierarchical (2-level, the individual level and the PHA level) linear model with fixed site-specific intercepts and a program impact that can vary across PHAs. This model is designed for multisite evaluations, in which there is interest in estimating the average effect of the program across all sites but also in accurately estimating the variation in effects across sites. It provides a built-in way of weighting the sites on the basis of sample size, the ratio of the sample in the treatment versus comparison developments, and the extent of cross-site variation, among other factors. 47 Although nonexperimental regression-based approaches are vulnerable to selection bias for samples with few sites, the large number of sites in this present study (24) allow for many of these site-specific biases to average out; as described in the meta-analysis literature, as the number of sites increases, the mean impact estimates between the randomized designs and the nonexperimental designs become more similar. 48 In fact, the match validation analysis described in the above section (and detailed in appendix A) showed that although each site-specific quasi-experimental impact estimate is subject to potential bias, much of which can vary randomly across sites, it averages out to almost zero in the estimate of the average impact across sites (see exhibit A.1 in appendix A for the findings of the analysis estimating the level of bias in the pooled sample).

Although the comparison sites were selected to match the Jobs Plus sites as closely as possible, it is important to include independent variables in the model that capture any remaining differences between the two groups of sites. The models therefore include several individual- and household-level characteristics, including adults’ employment and earnings in the three to six quarters before program start, age, gender, and race, as well as the number and ages of children in the household. Indicator variables for each site are also included in the model to account for differences in context across areas.

Finally, as a sensitivity analysis, an additional site-level variable was included in the model. Given that the selection of developments for Jobs Plus was not random within each PHA, this variable is designed to capture unobserved factors that may have led a given development to be selected as a Jobs Plus site. The research team collected information from PHAs about why 46 The developments that were removed from the study sample are listed in appendix A.

47 See Bloom et al. (2017) for more information on the model.

48 Lipsey and Wilson, 1993; Bloom, Michalopoulos, and Hill, 2005.

20

they selected the specific development or developments for Jobs Plus. Potential reasons might suggest that residents’ employment and earnings would be higher or lower than those in comparison developments. For example, if a development was selected in part because it was located close to public transportation or because it had connections to community-based organizations, then it might be expected that residents in that development would have higher employment and earnings than those in other developments, even in the absence of Jobs Plus.

Alternatively, if a development was selected because its residents were viewed as more disadvantaged or if the local area was viewed as more distressed, then the opposite pattern might occur. Finally, some PHAs might say that the development was selected because it was the only one that met the eligibility criteria for the grant.

Overall, the 24 PHAs reported a mix of reasons for selecting developments for Jobs Plus. A slightly higher percentage (45 percent) reported reasons that would suggest that employment and earnings might be lower for Jobs Plus residents than for residents in other developments.

Twenty-five percent of the PHAs reported reasons that might suggest higher employment and earnings for Jobs Plus residents, and the remaining 30 percent of PHAs reported reasons suggesting no difference in employment and earnings for residents in Jobs Plus versus other developments. As a sensitivity test to the main analysis, a PHA-level variable was included in the impact model to indicate each of these three categories. The addition of this variable did not change the findings (see appendix exhibit F.4).

Impact estimates on all outcomes are measured for each year after the launch of the program for 4 years of followup, whether sample members stayed or moved out of the development during the followup period (not just while they were living in public housing). This followup period generally covers the full 4-year grant period (though several grantees received grant extensions beyond 4 years). 49 The study uses two summary measures as confirmatory outcomes: cumulative earnings and average quarterly employment over the 4-year followup period. To account for the fact that the main analysis estimates effects on more than one outcome, and to avoid the potential for false positives, the p-values for the two confirmatory outcomes are adjusted using the Benjamini-Hochberg multiple hypothesis testing method.

Characteristics of Jobs Plus Developments

This section presents the characteristics of the 24 study sites at the start of Jobs Plus, focusing on the years 2014, 2015, and 2016 for the first, second, and third PHA cohorts, respectively. This sample of 24 PHAs comprises 31 public housing developments receiving HUD

funding to implement the Jobs Plus program. As shown below, the Jobs Plus grantees involved 49 On the basis of information provided by HUD, 22 grantees received extensions to their 4-year grants. Seventeen grantees received 3- to 12-month grant extensions, and five received extensions of more than 1 year.

21

represent a wide diversity in terms of size, site demographics, location, and other program features, offering an opportunity to understand the program’s implementation experiences against different backdrops. For example, 6 of the 24 grantees have Moving to Work (MTW) designation, which gives them additional flexibility to implement and test strategies around rent incentives and services to help residents increase their employment and earnings; two, Charlotte and Chicago, used their MTW flexibility to institute work requirements, by which nonelderly, nondisabled residents are required to work or participate in work-related activities for at least 20

hours per week. 50

Exhibit 5 presents selected characteristics of the Jobs Plus developments, such as their size (number of units), mobility rates, and local economic context. The exhibit presents averages for all grantees. 51 To illustrate the variation across developments, the exhibit also presents the minimum and maximum values for each characteristic. Exhibits 6 through 8 also present grantee-level information for selected characteristics. 52

Exhibit 6 presents data on the size of the Jobs Plus developments, which is defined in this report as the total number of subsidized units available in the developments implementing Jobs Plus in that PHA. The figure illustrates substantial variation in grantee size, ranging from large developments with nearly 1,500 units in Chicago and Cuyahoga County to smaller ones with 281

units in Phoenix. The figure also shows the percentage of subsidized units occupied by a nonelderly, nondisabled head of household in gray. All grantees had more than 50 percent of their units occupied by nonelderly, nondisabled heads of household. Five grantees—Cuyahoga County, St. Louis, Nashville, San Antonio, and Tampa—were predominantly occupied by nonelderly, nondisabled households (at least 75 percent). Similarly, the grantees and developments vary in the number of total residents. The top panel of exhibit 5 presents data on the total number of residents per grantee and the number of nonelderly, nondisabled residents.

The total number of residents ranges from a high of more than 3,500 residents in Altgeld Gardens in Chicago to a low of 643 residents in Marcos de Niza in Phoenix. Across most sites, about 35 to 40 percent of all residents (counting children and adults) are nonelderly, nondisabled adults.

50 In Charlotte, the housing agency expanded work requirements to the Jobs Plus development in 2017. In Chicago, work requirements were in place prior to the launch of Jobs Plus.

51 For grantees that have more than one development in the study, a weighted average is calculated across developments, with the weights based on the total number of units in the development.

52 For the grantee-level data presented in exhibits 5, 7 and 9, data for the New York City Jobs Plus development also include Vandalia Houses, a senior-only development, under the same asset management project number as Penn-Wortman Homes, the Jobs Plus development. It is not possible to disaggregate the two developments in the HUD

Picture of Subsidized Housing development-level data.

22

Exhibit 5

Jobs Plus Development Characteristics and Local Contexts

Cohorts 1 to 3

Characteristic

Average

Range

Resident Characteristics

Total residents

1,490

643–3,512

Work-able residents

571

244–1,396

Development Characteristicsa

Subsidized units available (N)

629

281–1,569

Mobility rate (%)

12.7

2.0–22.0

Length of residency (years)

8

3–14

Local Economic Context

County unemployment rate (%)

5.4

3.3–7.6

Local fair market rent for 2-bedroom

unit ($)

985

691–1,578

Local poverty rate (%)

53

27–78

Sample size (Grantees)

24

a If a Jobs Plus grantee has more than one development, the table shows the total number of subsidized units across all developments, the weighted mobility rate, and the weighted length of residency.

Sources: MDRC calculations using HUD Picture of Subsidized Housing data; U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

23

Exhibit 6

Total Subsidized Units and Percentage of Units Occupied by Work-able Households Cohorts 1 to 3

Range across 23 sites:

1,569

299 to 1,569 units

1,473

1,094

77%

72%

818

837

744

728

670

622

553

542

551

480

495

428

424

80%

434

59%

374

380

380

299

300

71%

72%

281

58%

60%

86%

57%

66%

68%

60%

63%

66%

82%

55%

73%

53%

50%

55%

63%

25%

o

s

e

s

o

k

a

o

o

x

e

ton

te

er

pa

lot

ton

ile

ore

ton

cag

phi

stin

bor

fol

hv

land

phi

ent

oni

Bos

env

or

ay

har

hi

ahoga

ous

em

oanok

D

ds

as

N

ak

nt

ltim

D

idenc

Tam

C

C

uy

H

ol

O

Phoeni

C

M

R

St. Loui

Syracuse

Au

G

N

Ba

Philadel

Sacram

San A

Prov

Cohort 1

Cohort 2

Cohort 3

Subsidized units available (N)

Occupied by work-able households (%)

Notes: These counts represent the total number of subsidized units in Jobs Plus developments within each grantee.

This table shows the percentage of occupied units that are occupied by work-able households.

For each item, the range provides the minimum and maximum mean values for the 24 Jobs Plus developments in the evaluation.

Occupancy rates for the Jobs Plus developments range from 81 to 100 percent. The total number of subsidized units (the black bar) includes both occupied and unoccupied units, and the percentage of households that are work-able (the gray bar) represents the percentage of all households living in the public housing development (therefore, all occupied units) that are work-able.

Data for the New York City Jobs Plus development also include Vandalia Houses, a senior-only development, under the same asset management project number as Penn-Wortman Homes, the Jobs Plus development, because the two developments are not disaggregated in the HUD Picture of Subsidized Housing development-level data.

Because of this, New York City is not shown in this figure.

Sources: MDRC calculations using aggregate data from the HUD Picture of Subsidized Households: data for Cohort 1 come from the 2014 Picture report; data for Cohort 2 come from the 2015 Picture report; data for Cohort 3 come from the 2016 Picture report

24

Implementing Jobs Plus might be more challenging in larger sites, particularly because the HUD grants for these cohorts were not based on the size of the development. All sites were expected to saturate their developments in terms of awareness of and participation in services, meaning that larger developments had to reach more residents with the same funding amount as some smaller developments. 53

Exhibits 5 and 7 also present mobility rates, defined as the percentage of residents in the Jobs Plus development(s) who had entered public housing during the year prior to the date that the site data were collected. 54 Mobility rates are 13 percent on average and range from a low of 2

percent in Philadelphia to a high of 22 percent in Dayton. Exhibit 7 illustrates this variation: there are several sites with quite low mobility and several grantees with very high rates. High rates of mobility are generally associated with lower average lengths of residency.

53 To set more reasonable outcomes expectations, the number of work-able residents used in HUD’s performance monitoring measures was capped at 600.

54 The key measure of interest is the rate of mobility out of the Jobs Plus developments, but these data are not available in the period prior to implementation. To assess whether the in-mobility rate presented in the table is a reasonable proxy for out-mobility, the study used IMS/PIC data to examine the percentage of residents who left the Jobs Plus developments in the first year of Jobs Plus implementation. This rate, although somewhat higher on average (because it captured both exits from public housing and moves from the Jobs Plus developments to non-Jobs Plus developments), was very highly correlated with the in-mobility rate.

25

Exhibit 7

Percent of Residents Moving into the Jobs Plus Development in Prior Year Cohorts 1 to 3

Range across 24 sites:

2 to 22 percentage

22

20

20

19 19

17

16

16

14 15

14 15

12

11

11

11

9

9

8

5

5

6

5

2

te

o

s

e

s

o

k

a

o

o

ty

x

e

ton

er

pa

lot

ton

phi

stin

ile

bor

fol

land

phi

ent

oni

ore

ton

cag

env

hv

or

ay

Bos

har

hi

ahoga

ak

nt

idenc

C

ous

ds

Tam

uy

H

em

oanok

N

D

St. Loui

D

ol

as

O

ork Ci

Phoeni

C

M

R

Syracuse

Au

N

Baltim

C

G

Philadel

Sacram

San A

w Y

Prov

Ne

Cohort 1

Cohort 2

Cohort 3

Mobility rate (%)

Average (13%)

Notes: For each item, the range provides the minimum and maximum mean values for the 24 Jobs Plus developments in the evaluation.

The mobility rate represents the percentage of households living in Jobs Plus developments that entered public housing within the year prior to Jobs Plus implementation.

Sources: MDRC calculations using aggregate data from the HUD Picture of Subsidized Households; data for Coh ort

1

come from the 2014 Picture report; data for Cohort 2 come from the 2015 Picture report; data for Cohort 3 come from the 2016 Picture report

Given the place-based nature of the program, mobility and tenure can have an important influence on program implementation and impacts. If high numbers of residents move out after short periods, they may benefit less individually from Jobs Plus and make it more difficult for the program to foster community supports for work. Sites experiencing higher residential turnover require staff to continually introduce new residents to the program, and they have less time to follow up with residents who tend to move out quickly. Not surprisingly, mobility rates are also somewhat negatively associated with fair market rents in the local area. Boston, New 26

York, and Oakland, for example, have relatively tight housing markets, high fair market rents (about $1,500), and low rates of mobility.

Finally, poverty rates for surrounding census tracts, at the start of implementation, indicate that most of the developments are located in highly economically distressed areas. Poverty rates exceed 30 percent for all but three developments, and nearly half of the developments have poverty rates of over 50 percent (exhibit 5). County-level unemployment rates (shown in exhibit 8) at the start of program implementation are relatively low across most sites, reflecting the continued fall in unemployment after the Great Recession. But there is modest variation across sites, with some areas facing rates of 6 percent or higher and others with rates under 4 percent.

However, the county unemployment rate is also measured at a large geographic area and may not reflect high rates of unemployment near the developments.

In fact, the baseline rates of employment for residents living in these developments are low, as shown in exhibit 9. That table presents selected characteristics of the residents across all developments, reflecting the time period around the start of program implementation. As mentioned above, eligibility for grants was restricted to developments with high levels of unemployment (i.e., at least 50 percent of the eligible households did not include wage earnings).

On average, only 30 percent of households had wage income as their main source of income, and this rate was less than 50 percent in all the Jobs Plus developments.

Variation in economic opportunity is likely to affect program implementation and impact.

Residents living in areas with high poverty rates and low employment rates may face substantial barriers to employment, for example, which may affect their ability to engage in program services or their ability to benefit from these services.

27

Exhibit 8

County Unemployment Rate in the Year of Program Launch, Cohorts 1 to 3

Range across 24 sites:

3.3 to 7.6 percent

7.5

7.6

7.1

6.6

6.0

6.2

6.2

6.1

6.0

5.3

5.6

5.6

5.5

5.0

5.2

4.9 5.3

4.4

4.7

4.6

4.4

3.8

3.3 3.7

o

s

e

s

o

k

a

o

o

x

e

ton

te

er

ty

pa

lot

ton

ile

ore

ton

cag

phi

stin

bor

fol

hv

land

phi

ent

oni

Bos

env

or

ay

har

hi

ahoga

ous

em

oanok

D

ds

as

N

ak

nt

D

idenc

Tam

C

C

uy

H

ol

O

ork Ci

Phoeni

C

M

R

St. Loui

Syracuse

Au

G

N

Baltim

Philadel

Sacram

San A

w Y

Prov

Ne

Cohort 1

Cohort 2

Cohort 3

County Unemployment Rate (%)

Average (5.4%)

Note: For each item, the range provides the minimum and maximum mean values for the 24 Jobs Plus developments in the evaluation.

Sources: Bureau of Labor Statistics: Local Area Unemployment Statistics: data for Cohort 1 come from the 2014

Local Area Statistics; data for Cohort 2 come from the 2015 Local Area Statistics; data for Cohort 3 come from the 2016 Local Area Statistics; for independent cities not in a county (Roanoke and St. Louis), the percent unemployment in the metropolitan area

28

Exhibit 9

Baseline Characteristics of All Residents at Jobs Plus Developments Cohorts 1 to 3

a Total number of people reported as living in the development at the date of the Picture of Subsidized Households snapshot.

b Percentage of households where the majority of household income is derived from wages and/or business.

Notes: The table describes the characteristics of all residents living in the development in the year prior to the launch of Jobs Plus, regardless of their eligibility for the program.

Sources: MDRC calculations using HUD Picture of Housing data, U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data: data for Cohort 1 come from the 2014 Picture report; data for Cohort 2 come from the 2015 Picture report; data for Cohort 3 come from the 2016 Picture report

Annual household income is also quite low for residents, at just under $10,000. Income tends to be lowest in developments with the lowest rates of wages as a primary income source.

Average income is lowest (at under $5,000), for example, in Dayton, Phoenix, and Baltimore developments, where the lowest percentage of households derive most of their income from wages.

Finally, on average, two-thirds of the residents in the developments are non-Hispanic Black. Considering the developments individually, in most developments, the large majority of residents are non-Hispanic Black. A few developments, such as those in Austin, are more evenly mixed in terms of race and ethnicity, while the remaining few sites (such as Denver, Phoenix, and Providence) are a majority Hispanic. The original Jobs-Plus program generated large earnings gains for public housing residents of different racial/ethnic groups, demonstrating that it 29

can be effective for many different types of public housing residents. The next section will look more closely at the characteristics of the working-age residents living in these developments.

Characteristics of the Study Sample

Whereas the above section described the place and communities where Jobs Plus was implemented (including demographic characteristics for all residents in the developments), this section describes the residents included in the sample for the present impact analysis. As described above in the Sample Definition section, the study sample includes adults between the ages of 18 and 57 and not having disabled status based on HUD’s definition, and when there is more than one adult in a household that fits these criteria, a focal adult is selected. 55 All residents, however, regardless of age and disability status, are eligible to receive Jobs Plus employment services such as job search assistance, employment readiness assistance, and other job supports such as childcare, transportation, and criminal records assistance. These residents are also eligible to enroll in the Jobs Plus Earned Income Disregard. The impact analysis, discussed in later sections, focuses on the subset of residents who are 18 to 57 years old at the time that Jobs Plus launched to exclude residents who will become elderly during the 4-year followup period.

Characteristics of Residents

Exhibit 10 presents demographic characteristics from HUD administrative data for the study sample living in Jobs Plus developments at the time that the Jobs Plus program was launched in that development. 56 Data on demographics, household composition, income, and other household and household member characteristics are collected from households by PHA staff when households join the housing subsidy program and are updated at regular recertifications, interim certifications when there are changes to household income or family composition, and relocation. 57

55 The HUD definition of a disabled individual is an individual who has a physical or mental impairment that substantially limits one or more of the major life activities.

56 HUD Inventory Management System/PIH Information Center data, described in Data Sources section above.

57 Appendix C presents a comparison of baseline characteristics between the Jobs Plus and comparison groups.

30

Exhibit 10

Baseline Characteristics of Residents in the Jobs Plus Developments

Focal Adults: Cohorts 1 to 3

Characteristic

Average

Range

Female (%)

86.8

74.0 - 95.5

Age (%)

18 - 24

13.6

6.0 - 22.3

25 - 34

40.6

22.6 - 50.2

35 - 44

24.4

15.7 - 34.8

45 or older

21.5

8.8 - 35.1

Average age (years)

35.2

31.6 - 38.7

Race (%)

White

20.7

0.4 - 94.0

Black/African American

78.0

5.3 - 99.3

Other (w ith note)

4.4

0.0 - 42.7

Hispanic or Latino (%)

18.5

0.0 - 91.2

Income sources (%)

Wages

44.0

27.2 - 60.1

TANF

17.8

2.0 - 67.9

Social Security/SSI/Pensions

2.8

0.0 - 16.2

Other

35.1

10.1 - 71.5

Annual income from w ages($)

$0

56.0

39.9 - 72.8

$1 - $4,999

6.0

1.2 - 19.4

$5,000 - $9,999

9.7

2.3 - 13.5

$10,000 - $19,999

17.3

10.5 - 25.0

$20,000 - $29,999

7.6

1.5 - 15.6

$30,000 or more

3.4

0.0 - 22.0

Average annual income from w ages

for individuals w ith any w age income ($)

15,151

9,529 - 27,464

Sample size (total = 9,220)

9,220

Not

es:

The study s

ample

consi

sts of focal adults (one adult per household who were age 18–57 and not ident ified as having a disability by the housing agency at the time that Jobs Plus implementation started in their deve l opm

ent. T

his tabl

e pres

ents

bas

eline c

harac

teristi

cs for study sa

mpl

e m

embers in t

he J

obs

Plus res

ear

ch

gr

oup only. L

ow

and

High site

-level

m

eans

refle

ct th

e minimu

m and maxim

um m

ean v

alue

s for Jobs Pl

us

deve

lopments at the 24 PHAs participating in the study. Sample sizes for specific outcomes may vary because of m

issing valu

es. R

oundi

ng may cause slight discrepancies in calculating sums and differences.

Det ail m

ay s

um t

o more t

han 100.

0 per

cent

for ques

tions that

allow m

ore than one response.

Calculati

ons for

basel

ine c

haracteristics

were deriv

ed from

eac

h household’s last certification before the

base

line date for their cohort.

Source: MDRC calculations usi

ng U.S. Department of Housing and Urban Development Inventory Mana 31

The HUD administrative data show that there were 11,521 nondisabled residents ages 18 to 57 living in 9,220 households at the start of Jobs Plus across the 31 developments in the 24

locations. The study sample includes one focal adult from each household that has at least one nonelderly, nondisabled household member. Overall, the study sample is mostly female (87

percent), on average 35 years old, and mostly African-American (78 percent). Nineteen percent of the population is Hispanic. These overall demographic characteristics represent considerable variation across grantees.

Forty-four percent of the study sample were employed at the start of Jobs Plus, and of those who were employed, their average annual earnings was $15,184. Exhibit 11 shows the variation in baseline employment rates across sites. The proportion who are employed ranges from less than one-third in Charlotte, Memphis, and Cuyahoga County to 61 percent in New York City.

Exhibit 11

Employment R

ate A mong Focal

A

dults at

B

aseline, C

ohorts 1 t

o 3

Range across 24 sites:

28 to 61 percent

61

58

60

54

56

50

48

49 51 50

45

47

46

41

42

42

36 36

35

34 32

28

29

28

e

s

o

k

d

o

e

ix

e

ton

te

ia

ty

ago

ga

his

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

mp

acu

en

orfol

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

N

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

Me

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Individuals employed (%)

Average (44%)

SO

C

C

S

f

S

N

ote: For eac

h item,

the r

ange provides the minimum and maximum mean values for the 24

Jobs Plus developments in the evaluation.

Source: MDRC calculations using U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information (PIC) data

As noted earlier, variation in employment levels at the start of the Jobs Plus program has implications for the implementation of the program and its potential impacts. Developments with 32

lower employment rates among nonelderly, nondisabled residents may be contending with weaker labor markets or geographically less access to job opportunities, making it less likely that Jobs Plus can help residents improve their employment situation, resulting in smaller impacts compared with developments in stronger labor markets or with better access to jobs. Conversely, these developments might have lower employment rates because residents have lower education levels, less work history, or a higher need for childcare, for example, making those residents more likely to benefit from some types of employment services offered by Jobs Plus, such as job search assistance or job readiness assistance, resulting in larger impacts compared with developments with higher employment rates among nonelderly, nondisabled residents who have fewer of these types of barriers to employment. Developments with more nonelderly, nondisabled residents already working may have different needs relating more to job retention and career advancement, and for these developments, the effects of Jobs Plus might be influenced by the extent to which those sites offer and focus on these types of services.

Exhibit 12 shows the average earnings among working residents across all the sites.

Average earnings levels in this figure generally mirror the variation in local economies, with larger and more expensive cities such as New York City, Oakland, and Boston having the highest average earnings.

33

Exhibit 1 2

Average Ear

nings Am

ong Em

ployed Foca

l A

dults at

B

aseline, C

ohorts 1 t

o 3

Range across 24 sites:

$9,554 to $26,661

661

$26,

031

322

$20,

432

$19,

137

507

$18,

276

$17,

$17,

830

674

256

546

$15,

$16,

336

655

360

$15,

579

856

$14,

$14,

891

493

$13,

$13,

$13,

729

$12,

320

585

$12,

849

$12,

717

$12,

$11,

$11,

$10,

$10,

554$9,

$10,

e

s

o

k

d

o

e

ix

e

ton

te

ia

ty

ago

ga

his

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

mp

acu

en

orfol

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

N

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

Me

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Average Earnings from Earners (%)

Average ($14,650)

Note: For each item, the range provides the minimum and maximum mean values for the 24 Jobs

Plus developments in the evaluation.

Source:

MDR

C c

alculati

ons usi

ng U

.S. De

partm

ent of Ho

usin

g and Ur

ban De

vel

opm

ent Inventory

Management System (IMS)/PIH Information Center (PIC) data

Characteristics of Households

Exhibit 13 presents characteristics of the study sample households for all the grantees.

These households have an average of three family members. The majority of households have children under the age of 18 (73 percent), and of those households with children, just over half of households have at least one child in the household that is 5 years old or younger.

The large proportion of households with children suggests that childcare may be an important factor for Jobs Plus to help residents gain and maintain employment, whether through helping residents access childcare or by building support networks within developments to share childcare responsibilities when parents are working. Over half of the nonelderly, nondisabled residents are single parents (they are the only adult in their household with at least one child.) The fact that many of these households are single-parent households points to potential further challenges in achieving employment-related goals, as does the large number of households with very young children.

34

Exhibit 13

Baseline Characteristics of Focal Adult Households

At Jobs Plus Developments, Cohorts 1 to 3

Characteristic

Average

Range

Average number of family members

3.0

2.2 -

4.0

Families w ith more than one adult (%)

28.4

10.2 -

57.0

Number of children in family (%)

None

25.7

8.3 -

47.6

1 child

26.9

18.6 -

34.3

2 or more

47.4

26.2 -

73.1

Families w ith one adult and children (%)

56.9

29.2 -

73.6

For families w ith children, age of youngest child (%)

0 - 5 years

52.0

33.7 -

63.3

6 - 12 years

34.5

23.5 -

50.3

13 - 17 years

13.6

6.5 -

19.3

Current/anticipated annual family income (%)

$0

15.4

0.0 -

41.4

$1 - $4,999

29.7

4.9 -

59.4

$5,000 - $9,999

21.0

13.3 -

42.5

$10,000 - $19,999

20.7

11.8 -

28.6

$20,000 or more

13.2

1.9 -

46.4

Average current/anticipated annual family income ($)

9,281

4,790 - 24,076

Income sources (%)

Wages

47.4

29.6 -

66.7

TANF

19.5

2.0 -

75.1

Social Security/SSI/Pensions

17.0

5.4 -

30.7

Other income sources

36.9

10.5 -

81.3

Average annual w age income for families w ith w age income ($) 16,400

10,173 - 28,616

Average total family contribution ($)a

234

120 -

570

Percent paying flat rents (%)

4.0

0.0 -

22.6

Average family contribution as a percent of gross monthly income (%) 38.6

21.8 -

54.9

Sample size (total = 9,220)

9,220

(continued)

35

Exhibit 13 (continued)

a

For non-MTW households, total family contribution is equal to the sum of tenant rent and utility allowance or to the flat rent amongst for households that pay flat rent. For M TW households total family contribution is equal to the greater value of 10% of gross monthly income or 30% of adjust ed monthly income.

Notes: The study sample consists of focal adults (one a dult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development. Low and high site-level means reflect the minimum and maximum mean values for Jobs Plus develops at the 24 PHAs parti cipating in the study. Sample sizes for specific outcomes may vary because of missing values. Round may cause slig ht discrepancies in calculating sums and differences. Detail may sum to more than 100.0 percent for questions that allow more than one response. Calculations for baseline characteristics were derived from each household’s last certif ication before the baseline date for their cohort.

Sour ce: MDRC calculations using U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

The average household income for the study sample was $9,281. This measure of income includes all sources of income, including earnings, TANF, and Social Security. Although 44

percent of the sample members were working at the start of Jobs Plus, 48 percent were living in households where at least one household member was working. Nineteen percent of households were receiving Temporary Assistance for Needy Families, or TANF; 17 percent were receiving Social Security, Supplemental Security Income, or pensions; and 37 percent had income from other sources, such as child support and unemployment insurance.

Resident Participation in Jobs Plus Activities

The Jobs Plus model is based on the premise that a multi-component approach is more effective than individual components at helping residents to make progress toward economic mobility. Jobs Plus is also intended to operate at “saturation” levels so that everyone living in a given public housing development has access to or an opportunity to benefit from the program in some way. In the original Jobs Plus demonstration that was implemented from 1998 to 2003, only the three sites (of the six total) that implemented and sustained all three components realized positive and sustained impacts on residents’ earnings. These earlier findings suggest that full implementation of the three components (employment services, the rent incentive, and Community Support for Work) and saturation, whereby a large proportion of nonelderly, nondisabled residents are engaged in Jobs Plus activities, may be essential to the program’s effectiveness in improving residents’ earnings.

This section reports on 3 years of program participation data for the Jobs Plus developments in the present study. It relies exclusively on grantee-reported aggregate participation data to examine resident engagement in Jobs Plus activities. Unlike Cohort 1, which 36

had the benefit of a comprehensive implementation study, similar data, which are useful for interpreting program implementation, are not available for the remaining cohorts in this present evaluation. Further, unlike the present study’s impact analysis that is described later in the report, which follows a cohort of residents that were living in the Jobs Plus development at program launch, these aggregate data reported by the sites include all nonelderly, nondisabled residents, including those that moved into the Jobs Plus development anytime in the 3-year followup period after the program had already launched.

A few points to note about these data. First, they capture residents’ participation in specific Jobs Plus activities, but they do not provide measures of the intensity or quality of those activities. Second, the Jobs Plus grantee reports from which we obtain our data provide aggregate and cumulative counts of nonelderly, nondisabled residents who participate in or receive certain services or activities but do not report a cumulative count of all the work-able residents who lived in the development during the reporting period—the denominator required to calculate participation rates.

MDRC therefore imputed those counts to calculate these rates. 58 Third, as described earlier, it is important to keep in mind that the enrollment and participation data presented in this section only capture “formal” participation and do not measure the proportion of residents who may have benefited from Jobs Plus indirectly—for example, through social networks, information sharing about job and education or training opportunities, or community events.

Jobs Plus Enrollment and Participation in Case Management

To take advantage of Jobs Plus employment services and the JPEID, residents are required to complete an initial assessment to enroll in Jobs Plus. For at least the first cohort, the HUD

Jobs Plus Process Study found that sites had a wide range of approaches for this initial assessment, which ranged from basic enrollment in Jobs Plus to a more comprehensive assessment of participants’ employment goals, barriers to employment, and service strategies through an Individual Training and Services Plan, as specified in the Notice of Funding Availability (NOFA). At a minimum, completing an initial assessment serves as a useful indicator of whether a resident was exposed to (or made aware of) the services offered by the Jobs Plus program and the opportunity to benefit from the JPEID, which required separate enrollment.

Exhibit 14 reports the percentage of nonelderly, nondisabled residents who completed an initial assessment (however it was defined by each site) and the percentage of nonelderly, 58 Appendix D describes the imputation approach.

37

nondisabled residents who met with their case manager for all sites combined. 59 The exhibit shows these percentages for the first 3 years of followup, the period for which these data were available. 60

Exhibit 14

Initial Jobs Plus Assessment Completion and Case Management Participation Cohorts 1 to 3

a Data from Baltimore are missing for the final quarter in Year 3. The total and rate over 2.75 years are included

instead.

Notes: The table includes 23 of 24 grantees. Memphis data are excluded for reasons described in the report.

Quarterly information is not available for Year 1.

Data from Norfolk in Year 3 are excluded because of data issues.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool The top panel of the table lists the average number of work-able residents across the 23

Jobs Plus developments for which participation data are available. 61 Based on quarterly data in the second and third years of implementation, the Jobs Plus developments had an average of 540

work-able residents in any given quarter across the 2 years with quarterly participation data. 62

Because the participation counts are cumulative, to calculate participation rates in Jobs Plus 59 The online reporting tool shows data for all residents ages 18 to 61 who do not have a disability. This population is slightly broader than the analysis sample for the present impact study, which includes residents who were ages 18

to 57 and did not have a disability at the start of Jobs Plus.

60 Year 4 data reported by the sites were not available in time for this report.

61 The Data Sources section details the reasons participation data for Memphis are excluded from this report.

62 Average quarterly data are not available for the first year of implementation because data were not collected on a quarterly basis during the first year.

38

activities, average quarterly counts, along with cumulative counts in HUD’s PHA-reported administrative data, were used to approximate counts of all work-able residents who lived in the development during the 1-year or 2-year time period.

On average, 26 percent of residents had completed an initial assessment (an indicator of program enrollment) across the Jobs Plus developments by the end of the first year of implementation, 44 percent by the end of the second year, and 52 percent by the end of the third year. 63,64 This percentage ranged from 26 percent in Boston to 79 percent in Chicago in the third year of the program. 65 66 Exhibit 15 shows the percentage of work-able residents who completed the initial assessment across the 22 sites for which there were 3 years of data available. Eleven of the 22 sites for which data were available had at least half of work-able residents enrolled by the end of the third year of implementation. Chicago, Austin, Houston, and Charlotte had particularly high assessment rates, at 79 percent, 77 percent, 76 percent, and 74 percent, respectively. Five sites had one-third or less of work-able residents enrolled by the end of the third implementation year: Boston, Cuyahoga, Oakland, Dayton, and Providence.

63 For purposes of calculating performance metrics, HUD capped the total number of work-able residents that larger grantees were responsible for targeting at 600. Seven of the 24 Jobs Plus developments had more than 600 work-able residents and therefore were affected by the cap; however, by the end of the second year of implementation, none of these seven developments had reached the 600 cap in Jobs Plus enrollment. Participation rates are calculated among all work-able residents in the developments, regardless of whether a development was affected by the cap.

64 The original Jobs Plus demonstration defined “attachment” to the Jobs Plus program as an eligible resident who either (1) personally enrolled in Jobs Plus, or (2) belonged to a household who was receiving a Jobs Plus rent incentive. The average “attachment rate” across the six developments was 62 percent for an early cohort and 76

percent for a later cohort (Bloom, Riccio, and Verma, 2005)

65 Grantee-level rates of assessment completion are shown in appendix exhibit E.1.

66 The Cohort 1 process study found that there were significant delays in launching the Jobs Plus program in Boston, in part because the site was waiting on Jobs Plus Earned Income Disregard guidance from HUD (Tessler et al., 2017). This site also received a smaller Jobs Plus grant. It is important to note, however, that this detailed process analysis is not available for the second and third cohorts of Jobs Plus sites, so the evaluation is at somewhat of a disadvantage for interpreting their participation data.

39

Exhibit 15

Percent of Res

idents who C

om

pleted the

Ini

tial

Assessment by

the

en

d of Year

3, C

ohor

ts 1

to 3

Range across 22 sites:

26 to 79 percent

79

77

74

76

71

67

63

56

59

55

50

47

43

44

39

41

42

32

31

33

26

28

e

s

o

d

o

e

ix

e

ton

te

ia

ty

ago

ga

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

acu

en

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Completed assessment (%)

Average (52%)

Notes: This sample includes all residents in Jobs Plus developments that were ages 18–57 and did not identify as disabled at the start of their baseline quarter.

For

each it

em, the r

an

ge provi

des

the

minimum and maxi

mum me

an v

alues f

or t

he 22 J

obs Pl

us

develop

ments

i

n th

e evaluatio

n.

Data from Memphis are excluded from this figure due to data being compromised by the Choice Neighborhoods impl

ementati

on

overla

ppin

g with the Jo

bs Pl

us im

plem

entation peri

od.

Data f

rom Norfolk

in Y

ear

3 ar

e exc

luded becaus

e of data issues.

Dat

a from

Bal

timor

e are

missi

ng for t

he final

quart

er in Year 3. The value shown above for this site covers 2.75

years since Jobs Plus began.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool Overall, the increase in the initial assessment rate from the end of the first year to the end of the second year (26 percent to 44 percent) is steeper than the increase from the end of the second year to the end of the third year (44 to 51 percent). This pattern holds up across individual sites as well (with the exception of Nashville, which had a steeper increase between the end of the second year and the end of the third year compared with the increase between the end of the first year and the end of the second year).

As shown in exhibit 14, during the second year of the program, on average across the sites, 19 percent of work-able residents met with their case manager during a given quarter, and in the 40

third year, this percentage increased slightly to 22 percent. 67 Case managers are Jobs Plus staff who both complete the initial assessment with participants and then continue to work with them to connect them to employment services and other community services. A case manager meeting includes initial assessments as well as ongoing meetings to discuss progress and other topics.

This quarterly average ranged from less than 10 percent in seven of the sites in the second year and six of the sites in the third year, and 30 percent or more in five of the sites in both the second and third years.

Participation in Employment Services

All 24 replication sites offered general services aimed at helping residents obtain jobs, including job search assistance, job readiness programs, resume writing assistance, interview preparation, and job placement services. Among the Cohort 1 sites, the implementation study documented that staff at most sites also recognized the need for services that would help residents advance in their careers and began focusing on those types of services as well. Boston and Chicago, however, emphasized the importance of training and advancement from the outset.

However, 2 years into program implementation, pre-employment services such as job search assistance and job readiness programs remained the focus of Jobs Plus employment services. 68

On average, as shown in exhibit 16, 19 percent of the eligible residents in a development received some post-assessment employment services by the end of the first year of implementation, 38 percent by the end of the second year, and 45 percent by the end of the third year. Nine of the 23 sites reported providing employment services to more than half of work-able residents by the end of the third year of implementation. 69 At the lower end, Boston, Cuyahoga, Oakland, and Dayton data suggest that these sites only provided employment services to less than a third of their work-able residents by the end of the third year. For some developments the increase in the cumulative percentage of eligible residents who participated in employment services from the end of the first year to the end of the second year of implementation is quite steep; for other developments, it is smaller. This difference across sites could possibly reflect differences in enrollment practices and the pace of Jobs Plus rollout at each site—i.e., that the establishment and availability of employment services occurred more quickly in some sites than others. On average, the increase in employment services receipt slowed down after the end of the second year of implementation.

67 As mentioned earlier, it was not possible to calculate average quarterly variables—including meetings with case managers—for the first year of implementation because the data were not collected quarterly for each quarter in the first year in the online reporting tool.

68 Verma et al., 2019.

69 Grantee-level participation rates are shown in appendix exhibit E.2.

41

Exhibit 16

Participation in Employment Services, Cohorts 1 to 3

a Dat

a from Baltimore are missing for the final quarter in Year 3. The service receipt rates cover 2.75 years since Jobs Plus began instead of 3 full years.

Notes: The table includes 23 of 24 grantees. Memphis data are excluded for reasons described in the report.

Data f rom Norfolk in Year 3 are excluded because of data issues.

Sourc e: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool Exhibit 17 shows the percentage of work-able residents who participated in employment services by the end of the third year of implementation by site (the gray sections of the bars in the graph), alongside site-specific levels of Jobs Plus enrollment (the full length of the bars in the graph, summing the gray and the black sections of the bars). The sites with the lowest Jobs Plus 42

enrollment at the end of the followup period—including Boston, Cuyahoga County, Oakland, Dayton, and Providence—unsurprisingly had generally low levels of resident participation in employment services; however, the sites with the highest Jobs Plus enrollment rates did not necessarily have the highest levels of resident participation in employment services, though overall participation rates were relatively higher. The most apparent discrepancy is Chicago, with a high Jobs Plus enrollment rate of 79 percent of work-able residents but only 51-percent participation in employment services. With the exception of Chicago, Austin, and Houston, who all had relatively high Jobs Plus enrollment rates, most residents who enrolled in Jobs Plus also participated in employment services. Across all sites, 46 percent of residents who enrolled received some form of post-assessment service from the Jobs Plus program.

Exhibit 17

Initial Assessment Completion and E

mploym

ent Services Rece

ipt

by the

E

nd of

Y

ear 3,

C

ohort 1 t

o 3

79

76

77

74

71

67

63

69

59

64

56

64

62

55

61

50

47

56

55

51

52

43

44

42

39

41

47

45

42

43

42

42

32

38

31

33

40

28

26

31

31

30

26

27

e

s

o

d

o

e

ix

e

ton

te

ia

ty

ago

ga

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

acu

en

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Completed assessment (%)

Received post-assessment services (%)

Notes: Data from Memphis are excluded from this figure due to data being compromised by the Choice Nei

ghborhoods i

mplem

entat

ion overl

appi

ng w

ith t

he Jo

bs

Plus

implementation

peri

od.

Data from N

orfolk in Ye

ar 3 ar

e exclu

ded

due to data q

uality issues.

When the bar indicating Completed Assessment is not visible (e.g., Boston), the same percentage of participants complet

ed the init

ial assessm

ent

and recei

ved post-asses sment services.

Data fr

om B

altimore

are

missi

ng for

t

he fi

nal quart

er in Ye

ar 3.

The values s

hown above f

or this site

cover 2.

75 year

s

since Jobs Plus began.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool 43

Across almost all the sites, job search assistance was the most frequently provided type of employment service, followed by employment readiness. Job search assistance includes a coach or case manager helping a participant with their job search, helping to create or strengthen a resume, and referrals to specific job opportunities. On average, 27 percent of work-able residents had received some form of job search assistance by the end of the third year of implementation, and the participation rate ranged from 7 percent in Philadelphia to 48 percent in Houston and Norfolk. Employment readiness assistance was the next most commonly used Jobs Plus employment service. Employment readiness programs provide training on work-related skills considered necessary to be successful in entry-level jobs in any sector, such as work habits and conduct, communication skills, and executive skills. By the end of the third year of implementation, on average, 18 percent of work-able residents had participated in an employment readiness program. This ranged from 2 percent in Oakland to 50 percent in Nashville. Other employment support services—including criminal records assistance, physical and behavioral healthcare assistance, childcare assistance, and transportation assistance—had much lower rates of participation across all the developments.

Enrollment in JPEID

All eligible households in Jobs Plus developments have the opportunity to benefit from the Jobs Plus Earned Income Disregard for the duration of the program. After enrolling in the JPEID, any increases in earned income (by any household member) do not result in a higher tenant rent as long as the Jobs Plus program is in place. 70 The HUD Jobs Plus process study found that the Cohort 1 sites adopted one of two enrollment procedures for the JPEID: requiring residents to enroll in the JPEID and Jobs Plus separately or enrolling residents in Jobs Plus and the JPEID simultaneously. Although a process study was not conducted for Cohorts 2 and 3 and similar JPEID implementation details are not available for the current evaluation, exhibit 18

displays JPEID enrollment rates alongside Jobs Plus program enrollment rates. It is important to note that in most sites, residents had to already be enrolled in the JPEID to benefit from the disregard once their earnings increased; however, based on information reported by the sites to MDRC, it appears that four of the Jobs Plus sites—Sacramento, San Antonio, Norfolk, and Baltimore—did not have residents enroll in the JPEID until they experienced an earnings increase. Thus, for these four sites, the JPEID participation data in the Jobs Plus Data Visualization tool reflect JPEID receipt or takeup, compared with JPEID enrollment in the case of the remaining sites. Exhibit 18 illustrates that JPEID enrollment does not always closely mirror Jobs Plus enrollment, even excluding the four sites where the JPEID enrollment rates in the exhibit represent JPEID receipt rates. For example, Chicago, New York, Nashville, and, to 70 One of the exploratory analyses included in the Jobs Plus Outcomes Evaluation is to examine how the structure of the JPEID affects residents’ work behavior and their reporting of earnings—specifically, whether the sudden onset and end of the JPEID affects residents’ work and earnings reporting behaviors or has other unintended consequences. The methods used for this analysis and the findings are detailed in appendix H.

44

some extent, Phoenix all have substantially lower rates of JPEID enrollment than Jobs Plus program enrollment, possibly due to the separate enrollment process for the JPEID. 71

Exhibit 18

Jobs Plus Enrollm

ent

and JPEID E

nro

llment

by the

End of Yea

r 3, C

ohor

ts 1 t

o 3

79

76

77

74

71

76

74

75

67

63

59

67

56

63

55

50

57

47

56

55 43

44

42

39

41

47

39

32

41

31

33

28

37

26

32

33

33

29

26

21

15

15

10

e

s

o

d

o

e

ix

e

ton

te

ia

ty

ago

ga

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

acu

en

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Participants enrolled in JobsPlus (%)

Participants enrolled in JPEID (%)

Notes: Data from Memphis are excluded from this figure due to data being compromised by the Choice Nei

ghbor

hoods i

mplem

entati

on overl

appin

g wi

th t

he Jo

bs Plus

impl

ementation

peri

od.

Data from N

orfolk in Ye

ar 3 ar

e exclu

ded

because of dat

a issues.

Data from Baltimore and New York City are missing for the final quarter in Year 3. The values shown above for these two

sites

cover 2.

75 year

s since J

obs Pl

us began.

JPEID

data f

rom Sacrament

o ar

e missin

g for

Year

3

and therefor

e ar

e excl

uded from t

his aver

age. J

PEID

data from

Norfol

k are exclu

ded

in Ye

ar 3

due t

o data q

uality issues.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool As exhibit 19 shows, by the end of the first year of implementation, on average, 19

percent of work-able residents were in a household that was enrolled in the JPEID, and by the end of the third year, the enrollment rate had increased to 43 percent. Note that a household’s enrollment in the JPEID does not indicate that the household received the earnings disregard.

Nevertheless, the process of enrolling in the JPEID, at a minimum, reflects awareness of the 71 See Tessler et al., 2017, for a discussion of JPEID enrollment issues among the Cohort 1 sites.

45

JPEID and possibly signals an intention to increase earned income and benefit from the earnings disregard.

Exhibit 19

Participation in JPEID and Financial Education, Cohorts 1 to 3

a Data from Baltimore are missing for the final quarter in Year 3. The rate over 2.75 years is included in this average instead.

b JPEID data from New York City are missing for the final quarter in Year 3. The rate over 2.75 years is included in this average instead. JPEID data from Sacramento are missing for Year 3 and therefore are excluded from this average.

Notes: The table includes 23 of 24 grantees. Memphis data are excluded for reasons described in the report.

Data from Norfolk in Year 3 are excluded because of data issues.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool; HUD data collected directly from developments

JPEID enrollment rates varied widely across the developments, as can be seen in exhibit 18

(grantee-level JPEID enrollment rates are shown in appendix exhibit E.3). The highest enrollment rate by the end of year 3 is 76 percent in Houston. This high rate likely reflects the fact that Houston initially adopted an automatic enrollment process for the JPEID (and was the only site to do so, at least among Cohort 1 sites). 72 Besides Houston, six sites reported JPEID

enrollment rates over 50 percent: Roanoke (63 percent), Austin (75 percent), Denver (57

percent), Goldsboro (67 percent), Philadelphia (56 percent), and Charlotte (74 percent). Boston and New York both had JPEID enrollment rates of less than 25 percent. 73 It is notable that in the original Jobs Plus demonstration, which tracked individual-level financial incentive receipt rates, developments with large impacts on earnings (Los Angeles, St. Paul, and Dayton) had higher rent incentives takeup rates that range from 60 to 77, whereas rates were much lower in the two 72 HUD later determined that sites could not automatically enroll eligible residents into the JPEID and would need them to complete a separate enrollment process for benefiting from the JPEID.

73Baltimore and San Antonio also had JPEID enrollment rates under 25 percent, but this is likely due to the data representing JPEID receipt. JPEID enrollment rates for Norfolk and Sacramento, who also had residents enroll in the JPEID only after they experienced an earnings increase, are not included in exhibit 15 due to data quality issues.

46

sites that had no impacts (Baltimore and Chattanooga, at 19 and 38 percent, respectively).

However, the sites with high rent incentive takeup rates also implemented the other two Jobs Plus components fully, so the impacts cannot necessarily be attributed to the rent incentive alone (or at all).

The implementation study of the nine sites in Cohort 1 also found that these sites had integrated some form of financial empowerment services to function alongside with the JPEID.

These included financial literacy programs and budget and credit counseling to help residents use the money they save with the rent incentive for savings and other financial goals. Overall, as reported, participation in these services was relatively low, with an average of 6 percent of work-able residents participating across the developments by the end of the first year, increasing to 15

percent by the end of the third year. Six sites had participation rates under 10 percent, and New York City had the highest rate at 34 percent (grantee-level financial education services enrollment rates are presented in appendix exhibit E.3).

Participation in CSW-Related Activities

The goal of the Jobs Plus Community Support for Work (CSW) component is to foster relationships among residents so that they can support each other in their efforts to improve their work situations and help create an environment of shared expectations and aspirations around work. The Cohort 1 process study found that there was a lot of variation in how sites defined CSW and that by the end of the second year of the program, many were still in the early stages of developing a vision for CSW and operationalizing this component.

By definition, CSW is a particularly difficult construct to quantify. Using available data, this report examines two indicators, which are used as rough proxies: levels of participation in Jobs Plus events and formal connections with community coaches. As discussed earlier, informal interactions between residents and Jobs Plus staff and community coaches, and among residents themselves, are the primary pathway through which CSW is intended to operate. The measures available—and examined here—do not capture this informal engagement. For example, because community coaches are residents in the Jobs Plus developments, they may have communicated about Jobs Plus with their neighbors informally without recording these communications as formal meetings. The two indicators also do not align with observations from the Jobs Plus Process Study. For example, the process study found that two sites, Charlotte and Chicago, demonstrated stronger implementation of the CSW component by the end of the second year of implementation relative to other sites in the first cohort; however, these two sites had low levels of participation in Jobs Plus events and formal connections with community coaches in the participation data for the present study. This discrepancy further underscores the incompleteness of these two measures for assessing the strength of CSW implementation.

47

Exhibit 20 includes measures of CSW-related activities: the number of Jobs Plus events held in the Jobs Plus developments, participation levels in those events, and the percentage of work-able residents meeting with community coaches. 74 Jobs Plus events are any activities that expose residents to Jobs Plus and foster relationships among residents. These include events such as workshops and social activities. On average, developments held 21 Jobs Plus events during the first year of implementation, 81 events during the second year, and 96 events during the third year. 75 About 19 percent of work-able residents attended one of these events each quarter in the second year, and about 26 percent attended one of these events in each quarter in the third year, though there was a range of attendance rates across PHAs.

Exhibit 20

Participation in CSW-Related Activities, Cohorts 1 to 3 Grantees a Data for Denver are excluded from this measure’s summary statistics.

b Data for Nashville are excluded from this measure’s summary statistics.

Notes: The table includes 23 of 24 grantees. Memphis data are excluded for reasons described in the report.

Data from Norfolk in Year 3 are excluded because of data issues.

Some sites reported very high values for some measures, and these data were not validated for this study, so

findings in this table should be interpreted with caution.

Quarterly information is not available for Year 1.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool

74 Exhibits that show the quarterly attendance rates for Jobs Plus events currently exclude data from Denver, and exhibits that show the average quarterly percentage of work-able residents who connected with community coaches currently exclude data from Nashville, because of data quality concerns. In addition, some of the counts that PHAs provided on the total Jobs Plus events per quarter and participation rates in those events were very high and could not be validated for this study, so findings based on these measures should be interpreted with caution.

75 Grantee-level participation in CSW is shown in appendix exhibit E.4.

48

Overall Formal Resident Participation

A key finding of the evaluation of the original Jobs Plus demonstration launched in 1998 is that the three of the six sites that fully implemented all three components of the Jobs Plus program had positive and sustained impacts on average earnings, whereas the program did not show any positive effects on labor market outcomes for residents in sites that did not fully implement the program. For the sites in this replication study, later sections explore whether sites with higher levels of implementation across the three components have larger positive effects on residents’ labor market outcomes, and to the extent that the implementation strength of each component can be measured, what the relative importance of each program component is for program effects.

The participation data presented in this section provide a preliminary indication of implementation levels and patterns across the three components. Exhibit 21 shows the post-assessment employment services participation rates alongside the JPEID enrollment rates by site.

(This exhibit focuses just on these two components because there is no direct participation measure for the CSW.) The graph shows that there is a relatively strong correlation (ρ=0.73) between the percentage of work-able residents who receive Jobs Plus employment services and the percentage of work-able residents who enrolled in the JPEID by the end of the second year of implementation. Excluding the three sites where the JPEID enrollment rate reflects JPEID

receipt (Sacramento, San Antonio, and Baltimore), the correlation is even higher (ρ=0.80). Thus, in most sites, enrollment in the overall Jobs Plus program and in the JPEID component tended to go hand in hand.

49

Exhibit 21

Percent Rece

ived Post

-Assessment Servi

ces and JPE

ID Enr

ollment

by the

End of

Year

3, C

ohor

ts 1 t

o 3

6974

76

75

64

64

61

6267

56

63

55

51

52

56

57

45

47

55

42

43

42

42

47

38

40

39

31

31

41

37 30

26

27

32

33

33

29

26

21

15

15

0

10

e

s

o

d

o

e

ix

e

ton

te

ia

ty

ago

ga

ok

ui

se

tin

ver

or

nc

pa

os

lot

ic

ston

us

bor

lan

ph

ento

oni

en

B

acu

en

el

ayton

har

aho

ou

oan

t. Lo

A

D

ds

shville

ak

nt

tim

D

ho

Tam

C

Ch

uy

H

yr

ol

O

ram

al

ork Ci

P

C

R

S

S

G

Na

ladhi

ac

an A

B

rovide

P

S

S

w Y

P

Ne

Cohort 1

Cohort 2

Cohort 3

Received post-assessment services (%)

Participants enrolled in JPEID (%)

Notes: Data from Memphis are excluded from this figure due to data being compromised by the Choice Neighborhoods implementation overlapping with the Jobs Plus implementation period.

Data fr

om N

orfol

k in Year

3

are exclu

ded

bec

aus

e of

data iss

ues.

Data from B

altimore

and New

Y

ork Cit

y ar

e missing for t

he final quarter in Year 3. The values shown above for these

two sites cover 2.75 years since Jobs Plus began.

S

ource:

MDRC calc

ulati

ons us

ing HUD J

obs Plus Pilot Data Visualization Tool

The high correlation between participation rates in different Jobs Plus components makes it difficult to assess whether a particular component is driving the program’s overall effects.

Although the study was not designed to measure the effect of individual Jobs Plus components, the study attempts to do so non-experimentally using variation in implementation of these components across sites. However, if the sites with higher receipt rates of employment services are the same sites that have high levels of JPEID enrollment, disentangling the effects of each component is more challenging. A later section examines whether the variation in program participation across sites and program components is substantial enough to support this type of analysis.

50

Impacts

The previous section reported that half of residents ages 18 to 61 without a disability in the Jobs Plus developments completed an initial assessment, and 45 percent participated in Jobs Plus employment services. These rates varied substantially across the sites, suggesting that program impacts might also vary. This section presents overall impacts on employment and earnings for the pooled sample of all 24 sites. It also assesses variation in impacts across sites and whether a comprehensive measure of program participation rates can help to explain that variation.

Overall Impacts

Exhibit 22 presents impact estimates for employment and earnings for the full sample of 24

PHAs over the 4 years of followup. Overall, the results suggest that Jobs Plus had no impacts on nonelderly, nondisabled residents’ average earnings or employment rates for the 4 years after program launch. Exhibit 22 shows that the difference between the Jobs Plus group and the comparison group in cumulative earnings between Year 1 and 4 (one of the two confirmatory outcomes for this study) is -$299, a difference of less than 1 percent that is not at all statistically significant. 76

76 The p-value is used to measure statistical significance and indicates the probability of obtaining the given impact estimate if the true effect of the program were zero. Smaller p-values provide stronger evidence that the program had an impact. Differences with p-values less than .10 are generally deemed statistically significantly different from zero.

51

Exhibit 22

Impacts on Earnings and Employment in the 4 Years of Followup Focal Adults: Cohorts 1 to 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,374

10,451

– 78

0.441

Year 2

11,755

11,700

56

0.680

Year 3

12,909

12,939

– 29

0.865

Year 4

13,829

14,070

– 241

0.272

Years 1–4

48,236

48,535

– 299

0.604

Average Quarterly Employment (%)

Year 1

59.7

60.1

– 0.4

0.409

Year 2

62.1

62.0

0.1

0.877

Year 3

63.2

62.8

0.4

0.551

Year 4

63.0

62.9

0.1

0.937

Years 1–4

61.9

61.8

0.0

0.979

Employed at Least One Quarter (%)

Year 1

72.3

72.4

– 0.1

0.866

Year 2

73.8

73.7

0.2

0.775

Year 3

74.8

74.1

0.7

0.316

Year 4

74.0

73.9

0.1

0.825

Years 1–4

85.7

85.1

0.5

0.266

Sample size (total = 19,267)

9,220

10,047

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values. Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance. The study sample consists of focal adults (one adult per household) who were ages 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators .

The impact estimates for the two confirmatory outcomes (4-year total earnings and 4-year quarterly employment) were not statistically significant; therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

Source: National Directory of New Hires

Average earnings in each of the 4 years of followup are also very similar between the Jobs Plus and comparison groups: the difference is less than 2 percent in all 4 years, and no estimated differences are statistically significant. On average, residents in comparison developments earned $10,451 in the first year after program start. This average includes zeros for adults who did not work during the year. About 72 percent of adults worked at some point during the year, and earnings for these workers was on average $14,515. Average earnings for the comparison group of adults increased steadily by about $1,000 per year over the 4-year followup period; employment rates increased modestly from Year 1 to Year 2 and then remained relatively flat for the remainder of the 4-year followup period. Employment and earnings followed a similar pattern for residents in the Jobs Plus developments, suggesting that the program through Year 4

had no impact on these outcomes.

52

In addition, there is no evidence of an impact of Jobs Plus on employment rates. The average quarterly employment rate across the 4-year followup period is 62 percent for both the program group and the comparison group. 77 Furthermore, the quarterly employment rates for program and comparison group members during each year of followup are also very similar; they do not differ by more than half a percentage point in any followup year, and none of the differences are statistically significant. The patterns of estimated effects look very similar for annual employment rates (defined as employed during at least one quarter in a given followup year). 78 Quarterly data are shown in exhibit 23. Across the 16 quarters of followup, employment rates and average earnings for Jobs Plus sites and comparison sites were very similar. Panel A of exhibit 23 shows that in the third quarter, the comparison sites had an average employment rate that was slightly higher than that of the Jobs Plus sites, and in Panel B, the graph shows that the average earnings of the Jobs Plus sites are slightly lower relative to the comparison group in the last three quarters of the 4-year followup. However, none of these differences are statistically significant.

77 The impact estimates for the two confirmatory outcomes (cumulative earnings and average quarterly employment over the 4 years of followup) were not statistically significant; therefore, based on the Benjamini-Hochberg multiple hypothesis testing approach, no further adjustments to the p-values were needed.

78 Recall that the study sample is limited to one adult (the focal adult) per household for whom there is data available for the full followup period. Two sensitivity tests were conducted to assess whether the estimated effects of Jobs Plus on average earnings and employment rates for this study sample differ from the estimated effects for (1) the sample of all eligible adults (rather than selecting one focal adult per household, and (2) the sample of all eligible adults for whom data were available in that year (regardless of whether data were available for them in other followup years). For both sensitivity tests, the findings were very similar to those for the main study sample: there was no evidence of effects on earnings or employment rates across the 4 years of followup for either alternative sample definition.

53

Exhibit 23

Quarterly Impacts on Employment Earnings Within First 4 Years of Followup 80

Program Group

Comparison Group

75

70

(%) 65

entm 60

oypl 55

ly Em 50

ter 45

uar

Year 1

Year 2

Year 3

Year 4

Q 40

35

30

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Followup quarter relative to Jobs Plus grant announcement

3,700

Program Group

Comparison Group

3,500

3,300

($) 3,100

ngsni 2,900

ly Ear 2,700

ter

uar 2,500

Q

2,300

2,100

Year 1

Year 2

Year 3

Year 4

1,900

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Followup quarter relative to Jobs Plus grant announcement

(continued)

54

Exhibit 23 (continued)

Notes: Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Quarter 1 (Q1) is the quarter following Jobs Plus launch.

A two-tailed t-test was applied to differences between research groups.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

Variation in Program Impacts Across Sites

As mentioned earlier, the two-level random-effects model used to estimate program impacts allows impacts to vary across sites, which makes it possible to estimate the cross-site mean and variance of program impacts and assess the statistical significance of those estimates.

The results from the model are also used to produce visuals (described below) that provide a clear and accurate picture of variation in Jobs Plus impacts across sites.

Exhibits 24 and 25 are caterpillar plots that summarize site-specific impact estimates using statistical methods that “shrink” or reduce the amount of variation in effects across sites due to site-level sampling and estimation error, providing a more accurate picture of existing cross-site variation in true programs effects. 79 One benefit of this approach is that, even in cases when the overall average impact is modest, there may be significant variation across sites, with the program in some sites leading to large positive effects and in other sites leading to no effects, or in theory, even negative effects.

These two exhibits present plots of site-level impacts for the two confirmatory outcomes of the present analysis: quarterly employment rates during Years 1 through 4 and average cumulative earnings over the same period. Sites are ordered in the figures from the largest positive to the largest negative estimated impact. 80 Each impact estimate is represented by a square, and its confidence interval is represented by a vertical line through the corresponding square. The 95-percent confidence interval represents a range in which there is high probability (95 percent) that the true program effect falls within that range. Impact estimates for which the confidence interval includes zero are not statistically significant at the 5-percent level.

The site-level estimated effects of Jobs Plus on 4-year cumulative earnings range from $2,474 to -$3,445, and the estimated site-level effects of Jobs Plus on average quarterly employment range from a positive 5 percentage points to a negative 2 percentage points.

However, almost all site-level estimates are not statistically different from zero. For cumulative earnings (exhibit 24), no site-level impact estimates are statistically significant, and for average 79 This method is technically referred to as Bayesian shrinkage estimates. Recent analyses suggest that conventional Bayesian shrinkage estimates can reduce the variation in impacts too much, providing an underestimate. For that reason, this report uses a modified form of these estimates (discussed in Bloom et al., 2016) that adjusts for this overcorrection.

80 Sites are not named to preserve anonymity and to avoid giving too much import to effect estimates based on very small samples.

55

quarterly employment (exhibit 25), only one site-level impact is statistically significant.

However, the variation in impacts across the sites is statistically significant (p=0.047 for cumulative earnings and p=0.018 for average quarterly employment). 81

Exhibit 24

Cater

pi

llar Plot of C

umulat

ive Earnings,

Years 1

to 4

$8,000

$6,000

$4,000

$2,000

arnings

$0

ulative E

um -$2,000

n C

-$4,000

pact oIm -$6,000

-$8,000

-$10,000

Site

Notes: The impact estimate for each site is shown with a solid dot. The vertical line running through each dot is the 95 percent confidence interval around that estimate.

T

he wid

er the c

onfidenc

e i

nterv

al, t

he

broader

th

e margi

n of

err

or. Impa

ct es

timates

with co

nfide

nce

int

erv

als

that

do

not include z

ero ar

e statistic

ally

signific

ant at

the 5

per

cent level.

Source: MDRC calculations using National Directory of New Hires (NDNH) data

81 Using a 0.10 significance level leads to the same conclusions, with no individual site-level estimates of impacts on earnings being statistically significant at the 0.10 level and only one site-level employment impact estimate being statistically significant at the 0.10 level.

56

Exhibit 25

Cater

pil

lar Plot of Q

uarter

ly Employm

ent Rate, Y

ears 1

to 4

10

8

)

6

ate (%

ent R 4

ploym 2

m

0

uarterly E -2

n Q

-4

pact oIm -6

-8

Site

Notes: The impact estimate for each site is shown with a solid dot. The vertical line running through each dot is the 95

perc

ent confidenc

e interv

al aroun

d that esti

mate.

The w

ider the confidence interval, the broader the margin

of er

perc ror. Impact estimates with confidence intervals that do not include zero are statistically significant at the 5

ent level.

Source: MDRC calculations using National Direct of New Hires (NDNH) data Assessing whether this variation is statistically significant then sets the stage for trying to

“explain” the variation with program or site-level characteristics. An important question in this regard is whether program participation can help to explain variation in program impacts. To examine this question, a broad measure of program participation (interacted with Jobs Plus or comparison group status) was added to the analysis model. This measure was the average for each site of the percentage of residents who received any post-assessment services through Year 3 and the percentage of residents who enrolled in the JPEID at any time during Years 1 through 3.

The results, shown in exhibit 26, indicate that higher rates of participation in Jobs Plus in the first 3 years of implementation were not correlated with program impacts on earnings or employment rates during the 4 years of followup. In other words, there was no evidence that sites with higher levels of participation in Jobs Plus had more positive effects on residents’

57

employment and earnings during the 4 years of Jobs Plus implementation. 82 A separate analysis, described later in this report, estimates impacts for the sample of replication sites that did not experience any major disruption that would make a site non-representative of Jobs Plus implementation. This separate analysis more closely aligns with the analytic approach of the original demonstration, which estimated impacts separately for sites that fully implemented Jobs Plus and sites with major disruptions or challenges.

Section IV of this report, which described the characteristics of Jobs Plus developments, documented variability across the Jobs Plus sites in local context, particularly local area unemployment rates (exhibit 8) and mobility rates (exhibit 7) within Jobs Plus grantees. To examine whether either of these site-level factors explained some of the variation in impacts on earnings and employment rates, analyses were conducted that used the same analytic method as that used to examine the relationship between participation levels and impact estimates: for each of the two site-level characteristics, the site-level variable (local unemployment rates or development mobility rates) were interacted with research group status (Jobs Plus vs.

comparison group) and included in the model.

The findings do not show any evidence that local unemployment rate was a significant factor in predicting program effects. However, the results do suggest a negative relationship between grantee-level mobility rates and program effects, showing that higher mobility rates are associated with less positive (or more negative) impacts on 4-year cumulative earnings. As shown in exhibit 26, a 1-percentage point increase in a site’s mobility rate (the percentage of households who newly entered public housing in the year prior to Jobs Plus launch) is associated with a reduction of program effects of $215. This estimate is statistically significant at the 5-percent level (p=0.020). The association with average quarterly employment over those 4 years is also negative but not statistically significant.

To further explore this relationship, impacts were estimated separately for only those residents who remained in a Jobs Plus development for at least 2 years after program implementation began and would have had greater exposure to the program83 (results are shown in appendix exhibit G.2). However, there was no evidence that Jobs Plus had any positive effects on the 4-year employment and earnings outcomes of these less mobile residents. This finding suggests that the correlation between mobility rates and program impacts found in the previous analysis may not be robust.

82 Due to measurement issues with the JPEID enrollment data mentioned earlier, where the enrollment rate represents JPEID enrollment in 20 sites and JPEID receipt in 4 sites, MDRC conducted a sensitivity analysis that estimated impacts on employment and earnings excluding the sites for which the JPEID enrollment data represented JPEID receipt to test whether this measurement issue was influencing the overall findings. Findings from this analysis (not shown) are similar to the main findings and show that even when excluding these sites, there is still no evidence that sites with high levels of JPEID enrollment had more positive effects on employment or earnings.

83 Sixty percent of the program group remained in a Jobs Plus development and 68 percent of the comparison group remained in a comparison development for at least 2 years after Jobs Plus implementation started.

58

Exhibit 26

Association Between Program Participation and Program Impacts and Between Local Contextual Factors and Program Impacts in the 4 Years of Followup Focal Adults: Cohorts 1 to 3

Average Quarterly

Earnings in Years 1–4 ($)

Employment Years 1–4 (%)

Parameter

Parameter

Explanatory Measure

Estimate P-Value

Estimate P-Value

Received post-assessment services and JPEID

enrollment through year 3

– 56

0.121

– 0.020

0.588

County-level unemployment rate at baseline

82

0.881

– 0.226

0.670

Mobility rate at baseline

– 215

0.020 **

– 0.132

0.168

Sample size

19,267

19,267

Notes: Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance. Statistical significance levels are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent.

Memphis and Norfolk are excluded from the first model (“Received post-assessment services and JPEID

enrollment through year 3”) due to data issues. The sample size for that analysis is 17,655.

Sources: National Directory of New Hires; U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data; HUD Jobs Plus Pilot Data Visualization Tool Impacts for Subgroups

Average impacts for all residents can sometimes mask differences in impacts for different types of residents or different types of PHAs. Residents facing certain disadvantages, such as low education levels or lack of childcare, for example, may have more difficulty taking up and benefiting from program services. Others, in contrast, may benefit little from services if they would have moved into jobs and increased their employment over time on their own, in the absence of the program. The present study examined two confirmatory subgroups, one defined at the site level and one defined at the resident level: a site’s “completeness” of Jobs Plus implementation and residents’ employment status at the time of the start of Jobs Plus implementation. It also examines the effects of Jobs Plus for two exploratory subgroups: resident tenure in public housing and Jobs Plus grantee cohort.

Completeness of Jobs Plus implementation was a particularly important factor in explaining the differential effectiveness of the original Jobs Plus demonstration and therefore was prioritized for the present replication study. To measure this program feature, MDRC

worked with HUD to identify sites that experienced major disruptions to the Jobs Plus program—for example, a major relocation of residents or redevelopment initiative, which 59

affected a site’s ability to implement all three Jobs Plus components. Three grantees were identified as meeting this definition:

• Memphis, which was awarded a Choice Neighborhoods grant soon after it was awarded its Jobs Plus grant and, as a result, relocated all residents of the Jobs Plus development at the beginning of the Jobs Plus grant period.

• Baltimore, where a part of the Jobs Plus development containing about one-fifth of the development’s units was demolished (and all residents in those units were relocated) during the grant period.

• Tampa, where a Rental Assistance Demonstration (RAD) conversion occurred a few days after the Jobs Plus grant award, and although residents were not relocated due to the RAD conversion, it led to the JPEID not being implemented in that site.

We then reestimated pooled impacts excluding those three sites. 84 Exhibit 27 presents the results of this analysis, which indicate that our overall findings do not qualitatively change.

84 A formal subgroup analysis—which would statistically compare the effects for the 21 sites without major disruptions with the three sites with major disruptions—was not feasible due to limited statistical power for the subgroup of three sites with major disruptions.

60

Exhibit 27

Impacts on Earnings and Employment in the 4 Years of Followup for Sites that Fully Implemented the Jobs Plus Model

Focal Adults: Cohorts 1 to 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,522

10,617

– 95

0.355

Year 2

11,912

11,863

49

0.726

Year 3

13,071

13,082

– 11

0.951

Year 4

14,060

14,248

– 188

0.409

Years 1–4

48,906

49,132

– 226

0.711

Average Quarterly Employment (%)

Year 1

60.0

60.4

– 0.5

0.425

Year 2

62.6

62.3

0.2

0.737

Year 3

63.7

63.1

0.6

0.438

Year 4

63.7

63.4

0.3

0.629

Years 1–4

62.4

62.2

0.2

0.762

Employed at least One Quarter (%)

Year 1

72.5

72.5

0.0

0.954

Year 2

74.2

73.8

0.3

0.631

Year 3

75.1

74.4

0.7

0.300

Year 4

74.7

74.2

0.4

0.524

Years 1–4

85.9

85.2

0.7

0.185

Sample size (total = 17,779)

8,439

9,340

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were ages 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators .

Based on HUD input, Baltimore, Memphis, and Tampa were excluded from this analysis. Full program implementation was significantly hampered at these sites due to Rental Assistance Demonstration or Choice Neighborhoods Initiative redevelopment, relocation, or demolition activities.

Source: National Directory of New Hires

For the baseline employment subgroup, effects are assessed by two measures of employment: whether a resident was employed at program start and whether a resident was employed during all three quarters prior to program start or not (the best possible measure of employment history that could be constructed with available NDNH data). It is reasonable to expect that program effects vary based on residents’ prior employment history because a number of studies have found that impacts on future employment and earnings are greater for individuals with less prior employment. This is because programs often find that helping unemployed 61

individuals get jobs is easier than helping working individuals increase their earnings or advance to higher wage jobs. 85

Exhibit 28 presents program impacts by employment status at program start. It shows that the effects of Jobs Plus on average earnings and employment rates over the 4-year followup period did not differ based on employment status at baseline: there was no evidence of effects on average earnings or employment rates for either the group of focal adults who were working at program launch or the group of focal adults who were not working at program launch.

Employment at baseline, not surprisingly, is highly predictive of post-program employment.

Among residents who lived in comparison developments and who were employed at baseline, 99

percent were employed in at least one quarter during the followup period, with an average quarterly employment rate of 83 percent across the 4 years. Cumulative earnings were $70,544, or average annual earnings of $17,636. For residents living in comparison developments who were not employed at baseline, 32.8 percent did not have any formal earnings for the full 4-year followup period. On average, their quarterly employment rate was only 34 percent over the 4

years. Their cumulative earnings over the 4-year period was $18,712, or an average of $4,678

annually. In terms of impacts, however, the program did not lead to observable differences by baseline employment status.

This null result was also observed for a different measure of baseline employment: employment stability. For this analysis, residents were considered stably employed if they worked for all three quarters prior to Jobs Plus launch and not stably employed if this were not the case. 86 As shown in appendix exhibit G.3, there is no evidence of effects on average earnings or employment rates during the followup period for either the subgroup of residents who were stably employed or those who were not stably employed.

85 See, for example, Riccio, Verma, and Deitch (2019) and Verma et al. (2017).

86 This timeframe was used because three quarters pre-launch were available for all three cohorts.

62

Exhibit 28

Impacts on Earnings and Employment in the 4 Years of Followup by Employment Status at Baseline

Focal Adults: Cohorts 1 to 3

Employed at Baseline

Not Employed at Baseline

Program

Control

Program

Control

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

16,339

16,437

98

0.494

2,188

2,279

– 91

0.477

Year 2

17,306

17,207

– 100

0.617

4,106

4,252

– 147

0.541

Year 3

18,307

18,376

69

0.763

5,490

5,601

– 111

0.690

Year 4

19,140

19,451

311

0.260

6,594

6,747

– 153

0.586

Years 1–4

70,363

70,554

191

0.781

18,123

18,712

– 589

0.467

Average Quarterly Employment (%)

Year 1

86.9

87.1

0.1

0.780

22.6

23.3

– 0.6

0.461

Year 2

83.5

82.9

– 0.6

0.340

32.9

33.6

– 0.7

0.550

Year 3

81.7

81.6

– 0.1

0.888

38.1

37.2

0.9

0.405

Year 4

79.8

79.8

– 0.1

0.918

40.1

40.2

– 0.1

0.956

Years 1–4

82.9

82.8

– 0.2

0.749

33.4

33.5

– 0.1

0.874

Employed at Least One Quarter (%)

Year 1

96.9

96.9

– 0.1

0.853

38.7

39.1

– 0.4

0.740

Year 2

93.0

92.8

– 0.2

0.736

47.8

47.7

0.1

0.933

Year 3

91.6

91.3

– 0.3

0.625

51.8

50.8

1.0

0.354

Year 4

89.7

89.9

0.1

0.875

52.6

52.3

0.3

0.763

Years 1–4

99.2

99.2

0.0

0.933

67.2

66.2

1.0

0.296

Sample size (total = 19,247)

5,283

5,818

3,931

4,215

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were ages 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent;

† = 10 percent.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Source: National Directory of New Hires

63

The next subgroup definition that was explored is tenure in public housing, defined as having lived in the PHA for 4 or more years versus less than 4 years at program start. Longer term residents of public housing may be less likely to respond to the program if, for example, they are more disadvantaged or less willing to increase earnings and risk losing eligibility. On other hand, the increased stability offered by housing assistance may put them in a better position to respond to an employment program by moving into work or increasing their earnings. As shown in exhibit 29, program impacts did not differ between these two groups.

The final subgroup is grantee cohort. Program impacts might vary by cohort if the experiences of implementation in the first cohort helped to improve implementation in the second and third cohorts. Differences in the timing of PHA entry to Jobs Plus might also suggest differences in context for the three cohorts. For example, national unemployment rates continued to fall from 2014 through 2016, and the first cohort sites had slightly higher local unemployment rates than the other two cohorts. However, findings in appendix exhibit G.4 do not indicate impact differences across PHA cohorts. 87

87 The second half of the fourth year of followup for Cohort 3 (Q2 2020 to Q3 2020) overlapped with the COVID-19

pandemic. To assess whether the effects of the pandemic on residents’ employment and earnings outcomes influenced the main impact findings, a subgroup analysis was conducted as a sensitivity test that compared the estimates of program impacts for the first two cohorts with those of the third cohort. As shown in appendix exhibit F.3, there were no differences in effects across these two subgroups.

64

Exhibit 29

Impacts on Earnings and Employment in the 4 Years of Followup, by Housing Tenure at Baseline

Focal Adults: Cohorts 1 to 3

At Least 4 Years in Public Housing

Less than 4 Years in Public Housing

Program

Control

Program

Control

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

9,083

9,105

22

0.888

11,490

11,584

– 94

0.477

Year 2

10,409

10,556

147

0.450

12,909

12,671

238

0.217

Year 3

11,663

11,661

– 2

0.992

13,991

14,007

– 16

0.945

Year 4

12,664

12,899

235

0.380

14,839

15,058

– 219

0.448

Years 1–4

43,475

43,838

363

0.611

52,383

52,477

-95

0.898

Average Quarterly Employment (%)

Year 1

60.2

60.0

– 0.2

0.780

59.3

60.0

– 0.7

0.253

Year 2

62.0

62.4

0.3

0.643

62.1

61.7

0.4

0.622

Year 3

63.8

62.9

– 0.9

0.239

62.7

62.8

– 0.1

0.913

Year 4

63.7

63.8

0.1

0.944

62.3

62.4

– 0.1

0.919

Years 1–4

62.3

62.2

– 0.2

0.786

61.4

61.6

– 0.1

0.838

Employed at Least One Quarter (%)

Year 1

74.6

74.2

– 0.4

0.574

70.2

70.9

– 0.7

0.336

Year 2

75.8

76.1

0.3

0.776

72.0

71.7

0.3

0.707

Year 3

76.6

76.5

– 0.1

0.888

73.1

72.2

0.9

0.342

Year 4

76.0

76.3

0.4

0.680

72.2

71.9

0.3

0.747

Years 1–4

88.4

88.4

0.0

0.954

83.1

82.5

0.6

0.411

Sample size (total = 19,263)

4,260

4,628

4,960

5,415

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were ages 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent;

† = 10 percent.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Source: National Directory of New Hires

65

Discussion

The present paper documents the characteristics of residents and communities in the first three cohorts of public housing agencies (out of seven cohorts to date) to receive Jobs Plus replication grants and examines the program’s effects on key labor market outcomes during the 4

years following the start of their Jobs Plus implementation. 88 These early cohorts of Jobs Plus grantees confronted an array of implementation challenges and had to learn and operate the program just as HUD itself was formalizing operational guidelines and administrative requirements for its grantees. 89 Nonetheless, the experiences and outcomes for the early cohorts to operate Jobs Plus provide important insights for the continued implementation and assessment of this program’s effectiveness.

As shown in this report, the first three cohorts of Jobs Plus grantees vary on a number of dimensions, ranging greatly in development size, local unemployment rates, housing markets, neighborhood poverty, and tenure in public housing, for example. Given the eligibility criteria for selecting Jobs Plus developments, employment levels were low at the start of Jobs Plus in these developments, ranging from 23 percent in Memphis to 53 percent in Boston, and among those residents who did work, average earnings were quite low. In that sense, the program was being targeted to public housing developments and residents who might benefit from a place-based employment program such as Jobs Plus.

The 4-year followup window used for this evaluation covered, for the most part, the full funding period of the one-time, nonrenewable Jobs Plus grant; all but 2 of the 24 evaluation sites, though, received short grant extensions to spend down their program funding. This 4-year timeframe allowed the present evaluation to assess program enrollment, participation patterns, and program impacts over time and determine whether increasing participation levels—as evidenced by higher Jobs Plus enrollment or services receipt over time, for example—shaped program impacts in the later years of the 4-year followup period. Further, by focusing on the three cohorts funded within a 2-year period (April and December 2015 and September 2016), the evaluation is also able to assess whether participation outcomes and impacts varied across the three cohorts, with the later cohorts (especially Cohort 3) benefiting from more developed operational and administrative guidance from HUD and its technical assistance provider.

The participation indicators analyzed for this report reveal the extent to which residents engaged with the program or its activities and services. The Jobs Plus model is premised on the idea that a multi-component approach is more effective at helping residents make progress toward economic mobility than would be the case for single components in isolation.

88 To replicate and scale-up Jobs Plus, across seven cohorts, HUD has awarded $136 million through 56 grants to PHAs.

89 Tessler et al., 2017; Verma et al., 2019.

66

Furthermore, the program does not target any particular subset of residents but intends for everyone living in a program housing development to be influenced by the program in some way. On average, about one-fourth (or 26 percent) of all eligible residents in the development at the time the program launched had enrolled in Jobs Plus and completed an assessment by the end of the first year of followup. By the end of the third year, this percentage had increased to about half (or 51 percent) and ranged from 26 percent in Boston to 79 percent in Chicago; reaching higher levels of enrollment in later stages of the 4-year program also means that residents receive less exposure to program services and incentives (the program’s fixed grant end-date and residents exiting the developments). Post-enrollment followup remained relatively low over the followup period. The sites with the lowest Jobs Plus enrollment at the end of the followup period unsurprisingly had generally low levels of resident participation in employment services; however, the sites with the highest Jobs Plus enrollment rates did not necessarily have the highest levels of resident participation in employment services, though overall participation rates were relatively higher.

The present evaluation does not have the benefit of a corresponding implementation study for all three cohorts—as noted earlier, the nine sites in Cohort 1 were the only sites to have their implementation experiences documented. As a result, this present evaluation has limited implementation information to unpack and interpret the participation patterns documented in the aggregate reports the Jobs Plus grantees submitted to HUD. The Jobs Plus process study for the first nine sites documented that those grantees needed more than the allotted 6-month period for launching the program and experienced a slower startup and service delivery relative to HUD’s expectation. From the Jobs Plus enrollment and participation data examined in this current evaluation, many of the Cohort 2 and 3 sites also appear to have experienced similarly slower rollout periods, and Jobs Plus or JPEID enrollment or service penetration rates increased gradually over time, potentially leaving much of a housing development’s community less engaged in the program’s direct services or activities.

This report also provides an assessment of residents’ employment and earnings outcomes and the impacts of Jobs Plus on those outcomes during the first 4 years the program was operational at each site. In this regard, it was found that a majority of sample members (86

percent) in the Jobs Plus developments worked at some point during the 4 years of followup, and employment levels remained consistently high among those who were working at the time of program launch. On average, sample members in these developments saw their average earnings increase by about $1,000 over the followup period. Employment and earnings followed a similar pattern for the residents in the comparison developments as well.

67

The analysis revealed that, on average, there was no program impact on eligible residents’

average earnings or employment rates throughout the 4 years of followup examined. There was some variation in impacts across sites for the study’s two confirmatory outcomes—cumulative earnings and average quarterly earnings over the 4-year followup period. Although the individual site-level estimates were largely not statistically significant, the variation in estimated effects across the 24 sites was statistically significant.

An exploration of the potential sources of this cross-site variation did not find evidence that participation levels (defined as JPEID enrollment and participation in post-assessment employment services) were correlated with estimated program effects, nor was “completeness”

of implementation an important factor for this set of grantees (as examined with an analysis that excluded the three sites that experienced major disruptions to their Jobs Plus implementation due to redevelopment or relocation of residents). There were also no differences in effects for residents not employed at enrollment compared with those who were working.

Additional exploratory analyses also do not provide evidence of differential impacts based on local unemployment rates, residents’ public housing tenure, or grantee cohort. An analysis exploring the relationship between site-level resident mobility rates and program effects did suggest that public housing developments with higher turnover rates had smaller program effects; however, a further analysis examining effects separately for only those residents who remained in a Jobs Plus development for at least 2 years of followup did not find that this group experienced larger effects on average earnings or employment rates than the full sample.

These findings stand in contrast to the those from the original Jobs Plus demonstration, where there was a clear pattern of positive impacts for the three sites with stronger implementation and a lack of impacts for the three sites with weaker implementation. However, the differences between the stronger and weaker implementation sites in the original demonstration were stark: only the three sites (of the six total) that implemented and sustained all three components realized positive and sustained effects on residents’ earnings levels.

In the original demonstration, participation rates were also higher in those sites that were deemed to have implemented a stronger program. The participation measures used in that study are not exactly comparable to those available for the present replication study. One key finding from that study, though, is that a relatively high proportion of residents in the Jobs Plus developments were “touched” by the program. For that study, an attachment rate—a broad measure that reflects either enrollment in Jobs Plus, the use of its rent incentives, or both—was constructed to gauge residents’ formal connection to the program. According to this indicator, across all developments, a majority of residents (62 percent of the 1998 cohort and 76 percent of the 2000 cohort) had a connection to Jobs-Plus—a significant accomplishment. These site-level attachment rates ranged from 50 to 85 percent for the 1998 cohorts and 61 to 95 percent for the 68

2000 cohort, with the higher attachment rates for the later cohorts attributed to a steadier state of program operations for the later cohort. It is possible that the weak correlation between program participation and impacts in the present replication study of 24 sites is due to participation levels overall not reaching a meaningful threshold that would lead to positive effects on residents’

employment and earnings outcomes.

The findings for the stronger implementation sites in the original demonstration, along with the recently completed 20-year followup impact evaluation of participants in the original JobsPlus evaluation—which found evidence of sustained gains in earnings for residents in the stronger implementation sites 20 years after program launch—together provide evidence that a comprehensive, well-implemented place-based employment program for public housing residents has the potential to meaningfully improve residents’ economic mobility in both the short and long term for the nonelderly, nondisabled adults living in the development and for the children living in their households.

The lack of evidence of economic gains for eligible residents in the Jobs Plus developments in the present evaluation of the first 24 sites of HUD’s Jobs Plus expansion might also reflect the implementation challenges that sites experienced during this early phase of HUD’s scale-up effort, and it is possible that positive effects on residents’ economic mobility outcomes could emerge in later cohorts that benefit from more developed program implementation guidance and technical assistance and from the experiences of the early cohorts. The original six Jobs Plus sites were part of a demonstration that offered considerable technical assistance from MDRC and its subcontractors. The availability of high-touch and tailored technical assistance could also make a marked difference in program implementation and outcomes of the replication sites.

From a continuous improvement perspective, and relying on insights based on the participation metrics examined in this report, HUD may also want to assess how various aspects of the Jobs Plus framework are being implemented and where there is room for refinement. The participation data, for instance, show higher rates of program enrollment and much lower rates of engagement in post-assessment employment-related supports and services (which slowed down after the end of the second year of implementation). Understanding how higher engagement can be attained in a place-based intervention is important both for strengthening the program and for providing residents with the types of services that will best help them advance.

The structure of the current Jobs Plus grants may also warrant some attention. The HUD

Jobs Plus replication grantees receive 4-year, nonrenewable grants. Lower participation rates early in the grant period for the 24 grantees, which increased by the end of the second and then third year, suggest that each grantee that implements the program may need a longer rollout period to put into place the infrastructure to operate the program at a steady state in that site. The original demonstration allowed a 2-year rollout period to help gear the programs up, recognizing the challenges of setting up a place-based program, with substantial resident involvement in the planning process. The 4-year followup included in the original evaluation period followed this 2-69

year rollout period. The sites in the present evaluation received funding for a total of 4 years, which covered startup, implementation, and wind-down, leaving a short timeframe for the sites to achieve a strong, sustained period of steady-state operations. Further, most of the sites in the current evaluation received standard funding levels, leaving the small and large sites with relatively comparable funding rather than having higher funding for sites with larger resident populations. This funding structure may also have resulted in sites making some tradeoffs between services and incentives, unlike the original demonstration.

Finally, the findings from the present study suggest the importance of further examining variation in Jobs Plus implementation on the ground to identify promising practices that lead to robust implementation and high levels of participant engagement to help future grantees realize the potential of the Jobs Plus program.

Conclusion

This report examined Jobs Plus participation rates and impacts on employment and earnings for nonelderly, nondisabled residents of the first three cohorts of sites to implement Jobs Plus under HUD’s current expansion initiative. The results reported suggest few differences in labor market outcomes for eligible residents in Jobs Plus developments and comparison sites.

As HUD continues to scale up Jobs Plus and fund more housing agencies to implement the program and build on operational and implementation insights from the early cohorts, it is possible that these initial patterns of program participation and impacts will improve.

In addition, the COVID-19 pandemic and its devastating economic shocks for families could also serve as another test for this program because Jobs Plus residents have the support of their case managers to help them navigate employment and other issues, unlike their counterparts in developments without the program. In the post-COVID-19 phase, Jobs Plus residents where the program continues to operate have access to onsite services and supports that might help them bounce back and enter the labor force more quickly. Building on the methods developed for this evaluation, HUD expects to continue tracking and estimating effects on employment and earnings outcomes for residents of Jobs Plus and comparison developments. The results in this report provide important context for ongoing monitoring, management, and assessment of this program.

70

APPENDIX A

MATCH VALIDATION ANALYSIS

71

This appendix presents the full results of the match validation analyses. It includes the results from three statistical tests used to assess the closeness of the match between the Jobs Plus developments and the set of developments that HUD identified for the comparison group: 1. Cohen’s D effect size differences between the program and comparison group.

2. Overlays of kernel densities.

3. Estimates of bias due to differences between the program and comparison group at baseline.

The match validation analysis focused on two baseline measures for the full work-able population, created using National Directory of New Hires (NDNH) wage data: (1) average preprogram quarterly employment rates and (2) average pre-program quarterly earnings. For Cohort 1, three quarters of pre-program NDNH data are available: for Cohort 2, six quarters, and for Cohort 3, five quarters.

After assessing the overall match for the pooled sample, the same assessment is made for each individual site. Based on that analysis, eight comparison developments were dropped from the analysis sample, given that they (1) were relatively less well-matched to the Jobs Plus development(s) in that public housing agency (PHA), and (2) were in a PHA with at least one other candidate comparison development (so that dropping that development would not result in dropping the entire PHA from the analysis).

The statistical tests were then rerun using the refined sample. Using the refined sample, an additional analysis was conducted to estimate the amount of bias one could expect in the impact estimates after accounting for any adjustments made to the estimates using the covariates—both historical employment and earnings measures as well as demographics and household characteristics.

In sum, the match validation found that the set of developments that HUD identified for the comparison was overall a good match for the set of Jobs Plus sites. Refining the comparison sample by dropping the eight developments, however, improved the overall match. An additional analysis found that the covariates used in the impact model, particularly historical measures of employment and earnings, are highly predictive of employment and earnings in the followup quarters and reduce the estimated bias of the impact estimates to nearly zero.

Assessing the Match of the Initial Comparison Sample

Prior to the start of the Jobs Plus Outcomes study, HUD identified candidate developments for the comparison group using historical Inventory Management System (IMS)/PIH Information Center (PIC) records of work-able residents’ employment rates and average earnings for earners for the 2 calendar years prior to grant award dates for each cohort, as well as demographic 72

information. 1 The results presented here assess the closeness of the match for these comparison and Jobs Plus developments. These statistical assessments rely on pre-program employment rates and average earnings using NDNH data.

Cohen’s D Effect Sizes

Cohen’s D effect size is one measure used to assess the match between groups and is calculated as the difference between two groups in the average of a given outcome divided by the full sample standard deviation of that outcome. Dividing the difference by the standard deviation standardizes the measure and allows effect sizes estimates to be compared for outcomes that are in different units, particularly useful for outcomes such as earnings. Differences between the program and comparison groups of less than 0.25 are typically viewed as satisfactory, and differences of less than 0.1 as ideal (as discussed in Hollenbeck and Huang, 2016).

For the annual earnings measures, the Cohen’s D effect sizes are calculated for each of the 24 sites as follows:

𝑚𝑚𝑒𝑒𝑒𝑒𝑛𝑛 𝑒𝑒𝑒𝑒𝑓𝑓𝑛𝑛𝑒𝑒𝑛𝑛𝑒𝑒𝑠𝑠 𝐶𝐶𝑓𝑓 𝑝𝑝𝑓𝑓𝐶𝐶𝑒𝑒𝑓𝑓𝑒𝑒𝑚𝑚 𝑒𝑒𝑓𝑓𝐶𝐶𝑔𝑔𝑝𝑝 − 𝑚𝑚𝑒𝑒𝑒𝑒𝑛𝑛 𝑒𝑒𝑒𝑒𝑓𝑓𝑛𝑛𝑒𝑒𝑛𝑛𝑒𝑒𝑠𝑠 𝐶𝐶𝑓𝑓 𝑐𝑐𝐶𝐶𝑚𝑚𝑝𝑝𝑒𝑒𝑓𝑓𝑒𝑒𝑠𝑠𝐶𝐶𝑛𝑛 𝑒𝑒𝑓𝑓𝐶𝐶𝑔𝑔𝑝𝑝

𝐶𝐶𝐶𝐶ℎ𝑒𝑒𝑛𝑛′𝑠𝑠 𝐷𝐷 𝑓𝑓𝐶𝐶𝑓𝑓 𝑒𝑒𝑒𝑒𝑓𝑓𝑛𝑛𝑒𝑒𝑛𝑛𝑒𝑒𝑠𝑠 =

𝑓𝑓𝑔𝑔𝑓𝑓𝑓𝑓 𝑠𝑠𝑒𝑒𝑚𝑚𝑝𝑝𝑓𝑓𝑒𝑒 𝑠𝑠𝑠𝑠𝑒𝑒𝑛𝑛𝑠𝑠𝑒𝑒𝑓𝑓𝑠𝑠 𝑠𝑠𝑒𝑒𝑑𝑑𝑒𝑒𝑒𝑒𝑠𝑠𝑒𝑒𝐶𝐶𝑛𝑛

For the employment rates, the match is assessed using the simple difference in employment rates (such as percent employed in the comparison group versus percent employed in the program group) for each of the 24 sites.

The results are presented as histograms in exhibits A.1 and A.2, illustrating the distribution of differences across PHAs. Exhibit A.1, shows the distribution of the difference in employment rates. Employment rates between the Jobs Plus developments and the comparison developments within each PHA are generally within 5 percentage points of each other, indicating a relatively close match, and the distribution is centered just below zero. There are, however, two sites with much larger differences. In Goldsboro, North Carolina, the employment rates for the Jobs Plus development is 11 percentage points lower than for the comparison development, and in New York City, the employment rate for the Jobs Plus development is 7 percentage points higher than for the average of the three comparison developments.

1 See Request for Proposal (RFP) for a detailed description of how HUD created index scores to identify candidate comparison developments.

73

Exhibit A.1

. Dif

ference in Ba

seline E

mployment R

ate b

etween Jo

bs P

lus an

d

Comparison Develo

pments Using

HUD-S

elected Comp

arison Samp

le

Note: There are 24 PHAs included in this analysis.

Source: MDRC calculations using National Directory of New Hires (NDNH) data Exhibit A.2 shows the distribution of Cohen’s D effect sizes for average earnings of work-able residents in the development(s). Most of the PHAs are within the 0.25 effect size threshold (indicated with the vertical lines), with more PHAs having negative effect sizes—such as, the comparison group’s average earnings are higher than the Jobs Plus group’s average earnings within that PHA— than positive effect sizes. The Memphis site is a clear outlier, with the Jobs Plus development’s average earnings 0.36 standard deviations lower than average earnings at its three comparison developments. The St. Louis site has a similarly negative effect size of about 0.25.

74

Exhibit A .2. C ohen’s D Eff

ect Si

zes fo

r Aver

age E

arnings A

mong A

ll W

ork-Able R

esidents

Using HUD-Selecte

d Comparison Sample

Notes: There are 24 PHAs included in this analysis.

Red vertical lines indicate a 0.25 effect size threshold.

Sourc e: MDRC

cal

culati

ons usi

ng Nati

onal Directory of New Hires (NDNH) data

Kernel Density Plots

A kernel density plot overlays the distribution in the measure of interest (employment rates or average earnings) for the program group with the distribution for comparison group. The benefit of this measure is that is goes beyond comparing average outcomes to compare the entire distribution of outcomes and assesses the overlap between the two groups.

For each baseline measure, this section displays a PHA-level plot followed by a development-level plot. Development-level plots were created in addition to the PHA-level plots to account for the fact that there are typically more comparison group developments than Jobs Plus developments within a given PHA (there are 51 comparison developments in total, compared with 31 Jobs Plus developments). This imbalance means that the PHA-level distribution of outcomes will likely be less spread out for comparison developments than for Jobs Plus developments, given that they are averaged over more developments. 2

2 The impact analysis is estimated with the PHA as the unit of analysis, rather than the development given that there is often more than one comparison development per PHA.

75

The kernel density plots for employment rates, exhibits A.3 and A.4, both show a relatively normal distribution, and the distributions for the Jobs Plus program group and the comparison group are relatively similar.

Exhibit A.3

. Ker

nel De

nsity Plo

t of PH

A-Level E

mployment R

ates

Using HUD-Selecte

d Comparison Sample

Notes: There are 24 PHAs included in this analysis.

Density curves show a proportional distribution smoothed to show trends in a continuous variable.

Source:

MDR

C calc

ulations usi

ng National

Directory

of

New

Hires (

N

DNH) data

76

Exhibit A.4.

Ker

nel Den

sity Plo

t of De

velopment-Level E

mployment R

ates

Using HUD-Selecte

d Comparison Sample

Notes: There are 31 program group developments and 51 comparison group developments.

Density

curves

show a pr

oporti

onal dist

ribution s

moot

hed t

o show tr

ends in a continuous variable.

Source:

MDR

C calc

ulations usi

ng National

Directory

of

New

Hires (

N

DNH) data

Exhibits A.5 and A.6 present the PHA-level and development-level plots for average earnings among work-able residents. The distributions in average earnings are relatively similar between the Jobs Plus program group and the comparison group. As expected, the comparison group in the PHA-level kernel density plot (exhibit A.5) has a narrower distribution than the program group, given that it is created using a larger number of developments. But both graphs are centered around a very similar mean. Exhibit A.6, created using the development as the unit of analysis (rather than the PHA) shows distributions that with a more similar spread.

77

Exhibit A.5

. Ke

rnel De

nsity Plo

t of PH

A-Level A

verage E

arnings

Among All Wor

k-Able Res

idents Us

ing HUD

-Selected Co

mparison S

ample

Notes: There are 24 PHAs included in this analysis.

Density

curv

es show a pr

oporti

onal dist

ribution s

moot

hed to

show tr

ends in a continuous variable.

Source:

MDR

C calc

ulations usi

ng National

Directory

of

New

Hires (

N

DNH) data

78

Exhibit A.6

. Ker

nel De

nsity Plo

t of De

velopment-Level A

verage E

arnings

Among All Wor

k-Able Resi

dents Usi

ng HUD

-Selected Co

mparison S

ample

Notes: There are 31 program group developments and 51 comparison group developments. Density curves show

a prop

ortional

distributio

n smo

othed to

show tr

ends in

a conti

nuous v

ariable.

Source: MDRC calculations using National Directory of New Hires (NDNH) data Refining the Comparison Sample

While the statistical assessments described above show that, overall, the HUD-selected comparison developments are a good match to the group of Jobs Plus developments, the Cohen’s D distributions do highlight some “outlier” comparison developments that are less well-matched.

MDRC dropped developments from the comparison group sample if they met the following three criteria:

1. The difference in employment rate between the comparison development and pooled Jobs Plus developments was greater than or equal to 10 percentage points.

2. The effect size for average quarterly earnings among the full work-able population was greater than or equal to 0.25.

3. There were other comparison developments in that PHA that did not meet the above criteria (for example, dropping the development would not result in dropping the entire PHA from the study).

Using these criteria, MDRC dropped the following eight developments from the comparison group sample:

79

Number

Employment

Work-Able

Rate

Avg Earnings

PHA Name Development Name

Residents Difference (%)

Effect Size (%)

Charlotte

Sunridge/Robinsdale/Claremont/Victoria

94

– 0.11

– 0.26

Chicago

Lake Parc Place

211

– 0.10

– 0.53

Memphis

Uptown Square

36

– 0.34

– 1.31

Memphis

University Place Phase III

29

– 0.24

– 1.03

St. Louis

St. Louis City

104

– 0.11

– 0.27

St. Louis

Renaissance Place at Grand II

37

0.00

– 0.60

St. Louis

Renaissance Place at Grand III

58

– 0.11

– 0.56

NYC

Williams Plaza

378

0.14

0.34

There were three developments that met the second criteria of having a difference in earnings of at least 0.25 standard deviations, but they did not have a difference in employment rates that exceeded 10 percentage points, so these developments were kept in the sample. These developments were located in Auston, Dayton, and Providence. There were also two developments that met both criteria of having large average quarterly earnings and employment rate differences, but for one of them (in Goldsboro), it was the only development in the comparison group, and for another (in Memphis), it was the closest match of the three HUD-selected comparison developments.

The resulting refined study sample includes the 31 Jobs Plus developments plus 43

comparison developments.

Assessing the Match of the Refined Comparison Sample

After excluding the eight “outlier” developments from the comparison group sample, the statistical assessments were rerun. Exhibits A.7 through A.12 present the results. The plot showing the difference in employment rates is slightly improved, although there is one PHA now with a difference of more than 10 percentage points. This PHA includes the one development in Memphis that was not dropped because it was the only site left in that PHA after the other two were dropped because they had even larger differences in baseline employment. These differences were in the opposite direction remaining site, so that on average the PHA-level differences masked the very large differences for individual developments.

Finally, for the remined sample, the Cohen’s D effect size for earnings shows fewer outliers. There are not major changes to the kernel density distributions.

80

Exhibit A.7. Difference in B

aseline Employm

ent

Rate Bet

ween Jobs P

lus and

Comparison Developments Using Refined Comparison Sample

Note: There are 24 PHAs included in this analysis.

Source: MDRC calculations using National Directory of New Hires (NDNH) data 81

Exhibit A.8. Cohen’s D Effect Sizes for Average Earnings Among All Work-Able Residents

Using Refined Comparison Sample

Notes: There are 24 PHAs included in this analysis.

Red ver

tical li

nes indicat

e 0.25

effect si

ze thres

hold.

Sour

ce: MD

RC c

alculati

ons u

si

ng National Direct

ory of New Hires (NDNH) data

82

Exhibit A.9

. Ker

nel Den

sity Plo

t of PH

A-Level E

mployment R

ates

Using Refined Compar

ison Sample

Notes:

There

are 24 PHA

s incl

uded in t

his analy

sis.

Density curves show a proportional distribution smoothed to show trends in a continuous variable.

Source: MDRC calculations using National Directory of New Hires (NDNH) data 83

Exhibit A.1

0. Kern

el Den

sity Plo

t of Devel

opment-Level E

mployment R

ates

Using Refined Compar

ison Sample

Note: T

here ar

e 31 progr

am gr

oup dev

elopment

s and 4

3 co

mpariso

n group developments.

Sour ce: MDR

C calculati

ons usi

ng National D

irec

tory of New Hi

res (N

DNH) data

84

Exhibit A.11. Kernel Density Plot of PHA-Level Average Earnings

Among All Work-Able Residents Using Refined Comparison Sample Notes: There are 24 PHAs included in this analysis.

Density

curves

show a pr

oporti

onal dist

ribution s

moot

hed t

o show tr

ends in a continuous variable.

Source:

MDR

C calc

ulations usi

ng National

Directory

of

New

Hires (

N

DNH) data

85

Exhibit A.12. Kernel Density Plot of Development-Level Average Earnings

Among All Work-Able Residents Using Refined Comparison Sample Note: There are 31 program group developments and 43 comparison group developments.

Source: MDRC calculations using National Directory of New Hires (NDNH) data

Estimate of Bias

Pre-program differences in baseline employment and earnings between the program and comparison groups may lead to bias in the impact estimates. If the comparison group has higher average earnings than the program group before Jobs Plus started, for example, those differences would be expected to persist in the program’s absence. Therefore, any positive effect of the program would be understated by comparing post-program earnings. A more direct method for assessing the extent to which dropping the eight “outlier” developments improved the match of the comparison group to the Jobs Plus program group is to compare the expected level of bias for both samples.

The average difference in employment rates and average earnings across the 24 sites represents the upper bound of this expected bias. (It represents the upper bound because the covariates included in the statistical models estimating impacts, including pre-program employment and earnings, are highly predictive of post-program outcomes and can potentially reduce much of the bias.)

The average differences are presented in exhibit A.13. For example, the mean difference in employment rates between the program and comparison in the original HUD-selected sample (which included 51 comparison group developments) is 1.2 percentage points. This difference is 86

negative, indicating that the comparison group had an employment rate that was 1.2 percentage points higher than the Jobs Plus group. Therefore, without any further adjustments, it would be expected that, at most, the effects of Jobs Plus on annual employment rates would be underestimated by 1.2 percentage points.

Exhibit A.13.

Aver

age E

ffect S

izes an

d Min

imu

m Detecti

ble Estimat

es

Using the

HUD

-Selected vs R

ef

ined Co

mparison Gr

oup Sam

ples

HUD-Selected

Refined Comparison

Comparison Sample

Sample

Standard

Standard

Effect Size/Difference

Average

Deviation

Average

Deviation

Employment Rate (Difference) (%)

-1.2

3.9

-0.4

4.1

Avg Earnings among all Work-Able

-0.049

0.128

-0.021

0.100

Residents (Effect Size)

Sample size

24

24

Note: There are 24 PHAs included in this analysis.

Source: MDRC calculations using National Directory of New Hires (NDNH) data Refining the sample leads to a significant improvement in this bias estimate, which decreases from the 1.2 percentage points to 0.4 percentage points. It also improves the bias estimate for average earnings, reducing the estimate from 0.049 standard deviations to 0.021

standard deviations, with negative values again indicating that the comparison group had higher average earnings than the Jobs Plus group.

As mentioned, these bias estimates are upper bound estimates. The statistical model used to estimate impacts for this report includes several covariates that are predictive of employment and earnings outcomes, including pre-program employment and earnings as well as demographic characteristics. These covariates will help control for any differences at baseline and therefore reduce any bias caused by differences in pre-program employment and earnings. Historical measures of employment and earnings, in particular, are especially predictive of subsequent employment and earnings. While it is not possible to calculate the actual amount of bias in the study’s impact estimates, it is possible to assess to what extent the covariates included in the impact models (including pre-program earnings and employment and demographic and other baseline characteristics) further reduce this bias.

An analysis was conducted using the same statistical model that was used to estimate impacts. In that model, the “outcomes” included the two quarters of employment and earnings just prior to program start and were regressed on (1) treatment status, (2) employment and earnings in the three to six pre-program quarters, and (3) all the other covariates included in the impact models, including demographic characteristics such as gender, race and ethnicity, and 87

household composition. After accounting for the covariates in the model, the coefficients on the treatment estimate for both the employment rate and the average earnings measures were nearly zero (-0.016 percentage points, p= 0.969 for employment rate, and $8, p=0.709 for average earnings). 3 These results suggest that the covariates used in the impact models account for most of the bias from pre-program differences. The final list of developments included in this analysis is shown in exhibit A.14.

3 Running these models with only site indicators and no covariates produce results that closely match the bias estimates in appendix exhibit A.1.

88

Exhibit A.14.

Jobs Plus and Comparison Developments

Jobs Plus

Comparison

Cohort PHA Name Development

Development

1

Boston

Charlestown

Mary Ellen McCormack

Old Colony

1

Charlotte

Southside Homes

Dillehay Courts

Tall Oaks/Tarlton/Savanna

Meadow Oaks/Glaedale/Wallace

1

Chicago

Altgeld Gardens

Wentworth Gardens

Dearborn Homes

Trumbull Park Homes

1

Cuyahoga

Outhwaite

King Kennedy

Carver Park

Cedar Central

1

Houston

Cuney Homes

Clayton Homes

Kelly Village

1

Memphis

Foote Homes

Montgomery Plaza

1

Roanoke

Lansdowne Park

Hunt Manor

Jamestown Place

1

St. Louis

Clinton-Peabody

Cambridge Heights

1

Syracuse

James Geddes

Central Village

Pioneer Homes

2

Austin

Chalmers Courts

Meadowbrook Courts

Booker T. Washington

Santa Rita Courts

2

Denver

Quigg Newton Homes

Sun Valley Homes

2

Goldsboro

West Haven

Woodcrest-Elmwood-Little Washington

2

Nashville

J. C. Napier

Cayce Place

2

Norfolk

Young Terrace

Tidewater Park

Diggs Park

(continued)

89

Exhibit A.14 (continued)

Jobs Plus

Comparison

Cohort PHA Name

Development

Development

2

Oakland

Campbell Village

Lockwood Gardens

Peralta Village

Mandela Gateway

Chestnut Court

2

Philadelphia

Raymond Rosen

Harrison Plaza

Parkview-Fairhill Apartments

2

Sacramento

Alder Grove

Meadow Commons

Marina Vista

Oak Park

The Mill

2

San Antonio

Cassiano Homes

Alazan/Guadalupe

3

Baltimore

Gilmor Homes

Latrobe Homes

Brooklyn Homes

3

Dayton

DeSoto Bass Courts

Westdale

Mount Crest

3

New York City

Penn-Wortman Homes

Hope Gardens

Marcus Garvey (Group A)

3

Phoenix

Marcos de Niza

Frank Luke Jr.

3

Providence

Hartford Park

Chad Brown

Manton Heights

Codding Court

3

Tampa

Robles Park Village

North Boulevard Homes

90

APPENDIX B

ANALYTIC MODEL & SPECIFICATIONS

91

Program impacts are estimated using a regression-based approach in which the outcome of interest is regressed on an indicator for whether the individual is in a Jobs Plus development plus a range of other variables capturing residents’ demographic characteristics and pre-program earnings. The specific model used is a hierarchical (two-level) linear model with site-specific intercepts and a program effect that can vary across PHAs. This model is designed for multi-site evaluations in which there is interest in estimating the average effect of the program across all sites, but also in accurately estimating the variation in effects across sites. 1

First, for clarity, the model is written for a single site. In this model, in which the outcome of interest (Y) is regressed on Treatment status (T) and Prior earnings (PY). This familiar specification shown in equation [1] is the centerpiece of many evaluations, including Randomized Controlled Trials (RCTs).

[1]

In this model: Yi = annual earnings (or employment) for individual I α = the mean outcome for the comparison development(s) Ti = 1 if individual i lives in the Jobs Plus development and 0 if they live in the comparison development

β = the mean program effect at that site.

= a series of quarterly variables indicating the pre-test (P) value of Y for individual i for each pre-treatment quarter q, γ q = a series of coefficients for the series of quarterly (q) pre-test variables. 2

ei = A random error that varies independently across individuals with a mean of zero and a variance that can differ between treatment and comparison group members.

The hierarchical linear model builds on this basic model and has two levels: the individual level and the site (PHA) level. In the model that follows, subscripts are added for site (j), and additional baseline characteristics (X) are added.

1 See Bloom et al., (2017) for more information on the model.

2 Each of the three cohorts has a different number of quarters of pre-program data (between 3 and 5 quarters). We will include all available quarters of data in the model, and for any cohort that does not have data for a particular quarter, we will use mean imputation and include missing value indicators for those quarters.

92

The individual-level model is as follows:

𝑌𝑌

∑γ PY

δ X

𝑖𝑖𝑖𝑖 = α + 𝛽𝛽

+ ∑

+ 𝑒𝑒

j

𝑖𝑖T𝑖𝑖𝑖𝑖 +

q

qij

k

kij

𝑖𝑖𝑖𝑖

[2]

q

k

Where:

Yij = annual earnings for individual i from each site j, αj = the mean outcome for the comparison development(s) at site j, Tij = 1 if individual i from site j lives in a Jobs Plus development and 0 if they live in a comparison development,

βj = the mean program effect for site j,

= a series of quarterly variables indicating the pre-test (P) value of Y for individual i for each pre-treatment quarter q at each site j where NDNH data are available3,

γq= a series of coefficients for the series of quarterly (q) pre-test variables,

= resident characteristic k for participant i from site j, 4

δk = a set of coefficients for the background characteristics, and

eij = A random error that varies independently across individuals in sites with a mean of zero and a variance that can differ between treatment and comparison group members and across sites.

Level two of the hierarchical model (the PHA level) is represented by equations 3 and 4

below. Equation 3 represents the site-specific fixed intercepts, included in the model as a series of 23 site indicators (for the 24 sites). Equation 4 represents the site-specific random treatment effects.

𝑒𝑒𝑖𝑖 = 𝐴𝐴𝑒𝑒𝑖𝑖 [3]

And

3 Note that in order to simplify the notation, this model is written as if each site has the same number of prior quarters. In reality, the number of prior quarters varies by cohort.

4 Covariates include baseline employment and earnings from the NDNH and selected demographics available in the HUD PIC data (such as age, race/ethnicity, and household composition).

93

𝛽𝛽𝑖𝑖 = 𝛽𝛽 + 𝑏𝑏𝑖𝑖 [4]

The properties of this model are fully explained in Bloom et al. (2017), but a couple of useful properties are worth emphasizing. First, the random effects strategy fully utilizes the 24 replicates and provides a built-in means of weighting the sites based on sample size, sample balance, and the extent of cross-site variation, among other factors. This approach uses empirical Bayesian analysis to produce shrinkage estimates of site-level program effects and a distribution of the effects across sites that are closer to the distribution of true program effects. Regular regression with site-fixed effects would dramatically overestimate the variation in site impacts, creating the false impression that true site impacts vary quite more than they likely do, because it would not distinguish between cross-site variation in true impacts and cross-site variation in site-specific impact estimation error.

Another important benefit of this hybrid random effects model is the potential for it to reduce selection bias by incorporating individual-level and site-level information on the source of the bias.

All estimation models have an error term representing uncertainty that is comprised of two components: (1) true variance in program impacts, and (2) variance due to random estimation error. In an RCT, the variance due to random estimation can be subtracted from the total variance to isolate the true variance in program impacts. In a non-randomized design, such as this one, after the variance due to random estimation is subtracted from the total variance, the variance leftover is comprised of the true variance in program impacts and the non-experimental error variance. The non-experimental error variance is due to selection bias at the site level and is the main threat to the validity of estimates using a non-experimental design. The model for this analysis can minimize selection bias by incorporating information on the selection mechanisms—for example, reasons for the PHA selecting a particular development for Jobs Plus, and how households are assigned to particular developments—into the model. This will reduce the unexplained nonexperimental error variance in the model and bring the resulting estimates even closer to the true effects. As noted in the main report, as a sensitivity test, the model was estimated including a site-level variable that attempts to capture selection bias by measuring the reasons the PHA selected the site for Jobs Plus.

94

APPENDIX C

JOBS PLUS AND COMPARISON GROUP

BASELINE TABLES

95

Baseline Characteristics of the Refined Comparison Sample

Overall, the pre-Jobs Plus employment rates and average earnings are very similar between the Jobs Plus work-able residents and the work-able residents in the comparison group, though they differ in some demographic characteristics. Appendix C presents a comparison of these baseline characteristics between the two research groups.

Appendix exhibit C.1 compares the pre-program earnings and employment levels for the Jobs Plus program group with those for the comparison group. 5 The two research groups are very similar across all four baseline measures of employment and earnings. Average total earnings in the year before Jobs Plus was launched was $9,003 in the Jobs Plus group and $9,031

in the comparison group. This difference is statistically significant, which is unsurprising given the large sample size. More than 68 percent of both the Jobs Plus group and the comparison group were employed sometime during that year. The quarterly employment rate during this baseline year was 56.7 percent for both research groups, and work-able residents in both research groups worked an average of about two quarters during that year.

Appendix exhibit C.2 compares the baseline characteristics of the work-able adults in the two research groups, and appendix exhibit C.3 compares the baseline characteristics of their households. The first columns in each exhibit that present the summary statistics for the Jobs Plus research group are identical to the summary statistics presented in exhibits 10 and 13, but these exhibits add a second column for the refined comparison group sample and test whether the differences between the two groups are statistically significant. Overall, the differences between the two research groups across characteristics, including gender, race, average age and age distribution, household composition, income sources, and housing subsidies, are very small (within 1 to 2 percentage points), but many are statistically significant, which is unsurprising given the very large sample size. One more notable difference is that while the percent of work-able residents and households with any earnings in the past year was very similar (45 percent at the individual level and 49 percent at the household level), the average earnings for earners is higher in the comparison group than the Jobs Plus group (about $16,100 compared with $15,700

at the individual level and about $17,800 compared with $17,100 at the household level).

5 Cohorts 2 and 3 have a full year of pre-program NDNH data, but Cohort 1 only has 3 quarters of baseline NDNH

data. Baseline measures for Cohort 1 were annualized using the following calculations: (1) for total annual earnings, the sum of the three quarters were multiplied by 4/3; (2) for annual employment rate, quarterly employment rate, and average number of quarters worked, the measure for the three quarters for Cohort 1 was multiplied by the ratio of the measure for the full baseline year to the measure for the same three relative quarters for Cohorts 2 and 3.

96

Exhibit C.1. Baseline Employment Characteristics of Focal Adults at Jobs Plus and Comparison Developments

Jobs Plus Comparison

Outcome

Adults

Adults

Total earnings in year before Jobs Plus launch ($)

9,003

9,031 *

Employment rate in year before Jobs Plus launch (%)

68.7

68.4

Quarterly employment rate in year before Jobs Plus launch (%)

56.7

56.7

Average number of quarters worked in year before Jobs Plus launch 2.3

2.3

Sample size (total = 19,234)

9,210

10,024

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates are adjusted by site indicators.

Source: The National Directory of New Hires

97

Exhibit C.2. Baseline Characteristics of Focal Adults

at Jobs Plus and Comparison Developments

Jobs Plus Comparison

Outcome

Adults

Adults

Female (%)

86.0

86.6

Age (%)

18–24

13.5

13.9

25–34

40.2

39.7

35–44

24.2

24.3

45 or older

22.0

22.1

Average age (years)

35.4

35.4

Race (%)

White

23.1

24.7 ***

Black/African American

76.1

74.3 ***

Other

5.1

5.4

Hispanic or Latino (%)

21.0

20.6

Income sources (%)

Wages

45.2

44.1

TANF

17.3

17.2

Social Security/SSI/Pensions

2.8

3.1

Other

34.5

31.6 ***

Annual income from wages (%)

***

$0

54.8

55.9

$1–$4,999

6.1

5.5

$5,000–$9,999

9.6

9.3

$10,000–$19,999

17.5

17.3

$20,000–$29,9999

8.0

7.5

$30,000 or more

4.0

4.6

Average annual income from wages for individuals with any wage income ($) 15,655

16,143 **

Sample size (total = 19,267)

9,220

10,047

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because categories are not mutually exclusive.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The study sample consists of focal adults (one adult per household) who were age 18-57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates are adjusted by site indicators.

Calculations for baseline characteristics were derived from each household's last certification before the baseline date for their cohort.

Source: MDRC calculations using U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

98

Exhibit C.3. Baseline Characteristics of Households with Any Work-Able Adult at Jobs Plus and Comparison Developments

Jobs Plus Comparison

Outcome

Households Households

Average number of family members

3.0

3.1 ***

Families with more than one adult (%)

29.4

32.0 ***

Number of children in family (%)

**

None

26.7

26.4

1 child

26.9

25.6

2 or more

46.4

47.9

Families with one adult and children (%)

55.7

54.2 **

For families with children, age of youngest child (%)

0–5 years

51.7

50.9

6–12 years

34.6

35.7

13–17 years

13.7

13.4

Current/anticipated annual family income (%)

***

$0

14.1

16.6

$1–$4,999

29.0

25.6

$5,000–$9,999

21.2

21.6

$10,000–$19,999

21.1

20.9

$20,000 or more

14.4

15.3

Average current/anticipated annual family income ($)

9,941

10,409 ***

Income sources (%)

Wages

48.8

48.4

TANF

19.0

18.9

Social Security/SSI/Pensions

17.7

20.3

Other income sources

36.4

33.4

Average annual income from wages, for families with any wage income ($) 17,055

17,830 ***

Average total family contributiona ($)

249

258 **

Percent paying flat rents (%)

4.2

3.8 *

Average family contribution as a percent of gross monthly income (%) 0.377

0.387 ***

Sample size (total = 19,267)

9,220

10,047

a For non-MTW households total family contribution is equal to the sum of tenant rent and utility allowance or to the flat rent amount for households that pay flat rent. For MTW households total family contribution is equal to the greater value of 10 of gross monthly income or 30 of adjusted monthly income.

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because categories are not mutually exclusive.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The study sample consists of focal adults (one adult per household) who were age 18-57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

The study sample consists of focal adults (one adult per household) who were age 18-57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates are adjusted by site indicators.

Calculations for baseline characteristics were derived from each household's last certification before the baseline date for their cohort.

Source: MDRC calculations using U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

99

APPENDIX D

METHOD FOR CREATING THE COUNT

OF ELIGIBLE RESIDENTS

100

To calculate cumulative participation rates in Jobs Plus activities, the quarterly counts of the number of eligible residents in each development reported by the grantees were adjusted to account for tenant turnover (for example, residents moving out of the developments and new residents moving in). MDRC used HUD Information Management System (IMS)/PIH

Information Center (PIC) data to approximate counts of all eligible residents who lived in the development during the 1-year or 2-year time period. Without these adjustments, the participation rates would be inflated, especially in developments with high tenant turnover rates: a resident who leaves the development partway through the implementation year would be counted in the cumulative participation rate (the numerator of the participation rate calculation) but not in the count of eligible residents in that development (the denominator).

For example, in the Boston Jobs Plus development, during the second year of implementation, on average there were 1,026 eligible residents living in the development at any given point in time based on the site’s reports on how many eligible residents were living in the development at the end of each quarter during that year. MDRC used HUD IMS/PIC data to adjust that count to a cumulative 2-year count.

To estimate the cumulative one-year counts, MDRC first calculated the ratio of the average number of eligible residents (reported monthly in the HUD IMS/PIC data) for the second implementation year to the total number of eligible residents who lived in the development at any point during that year. That ratio was then used to adjust the average quarterly counts to the cumulative 1-year count, adjusting for tenant turnover. The estimated cumulative count of the total number of eligible residents who lived in the development at some point in the first year of implementation is 1,130. The same calculation was done for the 2-year cumulative counts, except that we used the ratio of average quarterly counts from the participation data to the cumulative 2-year counts in the HUD IMS/PIC data for the first 2 years of implementation. 1 In Boston, the estimated cumulative 2-year count is 1,237.

1 Note that the HUD IMS/PIC-eligible resident counts were used to adjust the eligible counts in the participation data to create cumulative counts for the participation measures (using the method described in this section) instead of the counts being used directly in the denominators to calculate participation rates because there were some inconsistencies between the HUD IMS/PIC point-in-time counts of eligible residents and the participation data point-in-time counts of eligible residents, and the size of the discrepancy varied by site. Using these HUD PIC

quarterly point-in-time counts in conjunction with the HUD IMS/PIC cumulative counts to adjust the participation data counts in place of using them directly as denominators in the participation rate measures preserves the internal consistency of the participation rate measures and avoids the potential of introducing bias into these participation measures, which would have been especially problematic for this report that compares participation rates across sites.

101

APPENDIX E

ADDITIONAL PARTICIPATION TABLES

102

Exhibit E.1

Initial Jobs Plus A

sse

ssmen

t Completio

n and Case Manag

emen

t Participation

,

Cohort 1

Participation

Boston Charlotte

Chicago Cuyahoga

Houston

Roanoke St. Louis

Syracuse

Total number of work-able residents

Year 2 Quarterly Average

1026

302

1372

1229

365

284

354

434

Year 3 Quarterly Average

977

364

1646

1136

356

283

348

434

1-Year Cumulative Total

1130

359

1481

1474

465

356

397

534

2-Year Cumulative Total

1237

373

1705

1685

536

422

457

596

3-Year Cumulative Total

1345

388

1929

1897

607

488

518

658

Program Participation

Completed assessment (%)

By the end of Year 1

12

41

29

5

70

35

31

16

By the end of Year 2

16

58

67

22

77

41

50

37

By the end of Year 3

26

74

79

28

76

47

56

39

Met w ith case manager (%)

Year 2 Quarterly Average

5

50

6

8

15

33

7

23

Year 3 Quarterly Average

4

61

12

7

18

61

43

8

(continued)

103

Exhibit E.1 (co

ntinued)

Cohort 2

Participation

Denver

Austin Goldsboro

Oakland

Sacramento

Nashville

Norfolk

Philadelphia San Antonio

Total number of work-able residents

Year 2 Quarterly Average

308

313

212

619

742

717

697

517

541

Year 3 Quarterly Average

296

322

217

598

769

726

.

625

554

1-Year Cumulative Total

358

352

260

595

715

998

724

740

646

2-Year Cumulative Total

406

395

303

821

867

852

866

683

821

3-Year Cumulative Total

455

438

346

1046

1019

707

.

626

996

Program Participation

Completed assessment (%)

By the end of Year 1

30

42

40

14

16

21

45

24

20

By the end of Year 2

56

70

53

28

37

35

54

43

37

By the end of Year 3

59

77

67

32

43

71

.

55

41

Met w ith case manager (%)

Year 2 Quarterly Average

21

27

30

12

7

25

20

22

8

Year 3 Quarterly Average

18

21

26

32

12

7

.

19

18

(continued)

104

Exhibit E.1 (co

ntinued)

Cohort 3

Participation

Phoenix

Dayton

Baltimore

Tampa New York City Providence

Total number of work-able residents

Year 2 Quarterly Average

208

283

314

390

320

821

Year 3 Quarterly Average

191

282

313

404

266

824

1-Year Cumulative Total

275

455

342

404

341

1166

2-Year Cumulative Total

309

500

361

524

394

1141

3-Year Cumulative Totala

342

544

380

644

447

1115

Program Participation

Completed assessment (%)

By the end of Year 1

24

14

25

24

12

16

By the end of Year 2

42

26

48

48

31

26

By the end of Year 3a

50

31

63

44

42

33

Met w ith case manager (%)

Year 2 Quarterly Average

23

6

30

39

22

5

Year 3 Quarterly Averagea

7

22

5

24

35

22

a

Data from Baltimore are missing f or the final quarter in Year 3. The total and rate over 2.75 years is included instead.

Notes:

T

he table in

cl

udes

23 of 24 grante

es.

Memph

is data

is excluded for reasons described in the report.

Data from

Norfolk in Y

ear 3

are excl

u

ded bec

ause of data issues.

Quarter

ly inf

ormatio

n is

not availa

bl

e for

Year

1.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool 105

Exhibit E.2

Participation in

Em

ployment Ser

vices

Cohort 1

Participation

Boston Charlotte

Chicago Cuyahoga

Houston

Roanoke St. Louis

Syracuse

Employment Services

Received post-assessment services (%)

By the end of Year 1

11

38

17

2

8

26

30

15

Job search assistance

12

21

6

2

19

21

6

10

Employment readiness assistance

4

18

2

2

21

1

5

1

Criminal records assistance

0

1

0

0

0

0

0

2

Physical health care access

1

1

0

0

1

0

0

0

Behavioral health care access

2

1

0

0

0

0

1

0

Childcare assistance

2

4

0

0

0

1

1

0

Transportation assistance

1

20

4

0

5

5

1

1

By the end of Year 2

16

55

31

19

53

40

49

35

Job search assistance

16

33

19

7

42

29

12

27

Employment readiness assistance

11

28

8

9

39

12

12

4

Criminal records assistance

3

2

0

0

0

0

4

6

Physical health care access

3

18

0

0

3

1

3

0

Behavioral health care access

5

2

1

1

3

0

2

0

Childcare assistance

6

13

0

0

4

2

3

0

Transportation assistance

1

37

7

3

8

11

9

6

By the end of Year 3

26

69

51

27

64

45

56

42

Job search assistance

22

36

36

14

48

32

11

31

Employment readiness assistance

15

32

24

10

40

18

13

4

Criminal records assistance

5

2

1

0

0

0

4

6

Physical health care access

3

17

1

0

3

1

5

2

Behavioral health care access

6

4

1

1

4

0

3

0

Childcare assistance

11

15

1

1

4

2

6

0

Transportation assistance

2

47

15

6

11

15

12

8

(continued)

106

Exhibit E.2 (co

ntinued)

Cohort 2

Participation

Denver

Austin Goldsboro

Oakland

Sacramento

Nashville

Norfolk

Philadelphia San Antonio

Employment Services

Received post-assessment services (%)

By the end of Year 1

14

34

23

14

14

17

29

24

16

Job search assistance

13

11

12

9

10

14

30

0

8

Employment readiness assistance

10

7

11

1

3

12

7

0

10

Criminal records assistance

0

1

0

0

0

0

0

1

0

Physical health care access

0

0

1

0

0

0

2

0

1

Behavioral health care access

6

0

3

0

0

1

1

0

0

Childcare assistance

2

3

1

1

3

2

1

0

2

Transportation assistance

6

4

5

7

1

5

17

1

2

By the end of Year 2

45

49

53

28

34

31

54

40

35

Job search assistance

44

20

23

10

22

20

50

3

13

Employment readiness assistance

17

16

16

2

6

22

14

4

21

Criminal records assistance

0

0

0

1

2

0

1

1

0

Physical health care access

10

0

2

1

16

0

2

0

5

Behavioral health care access

11

0

4

0

1

2

3

0

1

Childcare assistance

9

7

3

2

9

3

5

0

4

Transportation assistance

22

6

9

10

6

9

32

2

3

By the end of Year 3

47

61

62

31

43

64

.

52

38

Job search assistance

47

27

30

9

43

34

.

7

17

Employment readiness assistance

21

21

18

2

12

50

.

6

26

Criminal records assistance

2

1

0

0

2

0

.

1

2

Physical health care access

12

2

2

1

32

0

.

0

7

Behavioral health care access

11

1

4

0

1

3

.

1

1

Childcare assistance

19

8

5

2

7

4

.

0

9

Transportation assistance

26

12

14

20

10

19

.

4

4

(continued)

107

Exhibit E.2 (

continued)

Cohort 3

Participation

Phoenix

Dayton

Baltimore

Tampa New York City Providence

Employment Services

Received post-assessment services (%)

By the end of Year 1

18

14

18

24

12

12

Job search assistance

11

15

5

13

11

9

Employment readiness assistance

1

14

4

13

2

1

Criminal records assistance

0

1

2

0

0

0

Physical health care access

0

6

0

0

0

0

Behavioral health care access

0

3

2

0

0

0

Childcare assistance

1

2

2

2

0

0

Transportation assistance

1

4

1

6

2

0

By the end of Year 2

36

26

41

44

31

21

Job search assistance

22

24

15

22

32

13

Employment readiness assistance

12

25

13

22

3

2

Criminal records assistance

0

4

4

0

0

0

Physical health care access

1

11

2

1

2

0

Behavioral health care access

0

5

3

0

1

0

Childcare assistance

2

4

3

4

0

0

Transportation assistance

4

9

4

16

4

0

By the end of Year 3a

42

31

55

40

42

30

Job search assistance

27

31

24

20

33

15

Employment readiness assistance

22

24

18

19

6

3

Criminal records assistance

0

7

5

0

0

0

Physical health care access

2

14

2

1

4

0

Behavioral health care access

3

10

3

0

6

1

Childcare assistance

10

4

4

6

0

0

Transportation assistance

10

13

5

16

3

0

a

Data from Baltimore are missing for the final quarter in Year 3. The service receipt rates cover 2.75 years since Jobs Plus began instead of 3 full years.

Note:

The t

able i

ncl

udes

23 of 24 grant

ees.

Mem

phis dat

a is excluded for reasons described in the report.

Data fr

om N

orfolk i

n Year

3 ar

e excl

u

ded

becaus

e of dat

a issue

s.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool.

108

Exhibit E

.3. P

articipation in

JP

EID an

d Fin

ancial Ed

ucation

Cohort 1

Participation

Boston Charlotte

Chicago Cuyahoga

Houston

Roanoke St. Louis

Syracuse

JPEID

Enrolled in JPEID (%)

By the end of Year 1

11

39

14

4

14

36

32

16

By the end of Year 2

14

54

27

19

78

42

49

37

By the end of Year 3

21

74

29

26

76

63

56

39

Received financial education services (%)

By the end of Year 1

7

5

2

3

0

10

12

0

By the end of Year 2

14

18

4

11

1

16

12

2

By the end of Year 3

21

18

5

15

10

17

15

2

Cohort 2

Participation

Denver

Austin Goldsboro

Oakland

Sacramento

Nashville

Norfolk

Philadelphia San Antonio

Enrolled in JPEID (%)

By the end of Year 1

28

26

40

14

15

21

46

23

4

By the end of Year 2

55

57

54

28

35

35

51

40

11

By the end of Year 3a

57

75

67

32 .

47

.

55

15

Received financial education services (%)

By the end of Year 1

2

6

13

2

5

8

24

5

6

By the end of Year 2

6

12

17

14

10

14

45

13

13

By the end of Year 3

11

23

19

20

13

27

.

22

19

(continued)

109

Exhibit E.3 (co

ntinued)

Cohort 3

Participation

Phoenix

Dayton

Baltimore

Tampa New York City Providence

Enrolled in JPEID (%)

By the end of Year 1

10

11

2

23

0

0

By the end of Year 2

26

27

7

47

9

25

By the end of Year 3ab

37

33

10

41

15

33

Received financial education services (%)

By the end of Year 1

2

14

2

0

6

0

By the end of Year 2

5

25

3

3

26

2

By the end of Year 3b

7

29

3

2

34

4

a

JPEID data from New York City ar e missing for the final quarter in Year 3. The rate over 2.75 years is included in this average instead. JPEID data from Sacra

ment

o are

missi

ng for

Year 3 and ar

e theref

ore ex

cl

uded from

thi

s aver

age. JPEID

d

ata f

rom Norfolk are excluded in Year 3 due to data quality issues.

b Data f rom

Baltim

ore

are

missing for the

final

quarte

r in Y

ear 3. The rate over 2.75 years is included in this average instead.

N

otes: T

he table in

cl

udes

23 of

24 gr

antees. M

emph

is data ar

e exclu

ded for

reas

ons des

cribed in the

re

port.

Data fro

m N

orfolk in Y

ear

3 are ex

clu

ded

bec

ause of

data iss

ues.

Sources: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool, HUD data collected directly from developments 110

Exhibit E

.4. P

articipation in

C

SW-Related Ac

tivities

Cohort 1

Participation

Boston Charlotte

Chicago Cuyahoga

Houston

Roanoke St. Louis

Syracuse

Number of Jobs Plus Events in Year 1

14

18

30

47

20

11

17

17

Number of Jobs Plus Events in Year 2

31

23

58

178

64

15

98

78

Number of Jobs Plus Events in Year 3

124

25

142

135

191

13

103

25

Attended Jobs Plus Event (%)

Quarterly average in Year 2

6

6

3

16

36

33

6

16

Quarterly average in Year 3

7

5

11

12

75

32

18

6

Connected w ith Community Coach (%)

Quarterly average in Year 2

3

14

10

8

31

48

6

3

Quarterly average in Year 3

3

7

2

15

11

31

22

12

Cohort 2

Participation

Denver

Austin Goldsboro

Oakland

Sacramento

Nashville

Norfolk

Philadelphia San Antonio

Number of Jobs Plus Events in Year 1

38

9

10

20

9

19

8

16

8

Number of Jobs Plus Events in Year 2

120

17

18

79

49

63

18

183

186

Number of Jobs Plus Events in Year 3

105

20

30

46

54

164

.

350

175

Attended Jobs Plus Event (%)

Quarterly average in Year 2a

.

36

18

10

19

13

7

7

28

Quarterly average in Year 3a

.

34

17

8

37

43

.

6

20

Connected w ith Community Coach (%)

Quarterly average in Year 2b

10

21

20

9

11

.

15

6

12

Quarterly average in Year 3b

4

32

7

27

23

.

.

6

7

(continued)

111

Exhibit E.4 (co

ntinued)

Cohort 3

Participation

Phoenix

Dayton

Baltimore

Tampa New York City Providence

Number of Jobs Plus Events in Year 1

13

44

21

48

12

44

Number of Jobs Plus Events in Year 2

65

74

122

178

43

112

Number of Jobs Plus Events in Year 3

64

113

28

161

43

68

Attended Jobs Plus Event (%)

Quarterly average in Year 2

18

54

23

51

16

5

Quarterly average in Year 3

29

130

12

35

49

4

Connected w ith Community Coach (%)

Quarterly average in Year 2

23

9

24

28

15

4

Quarterly average in Year 3

34

12

6

50

36

10

a Data for Den

ver are excl

uded fr

om

this

measure’

s su

mmary

statistics

.

b Data for Nashville are excluded from this measure’s summary statistics.

Notes: The table includes 23 of 24 grantees. Memphis data are excluded for reasons described in the report.

Data from Norfolk in Year 3 are excluded because of data issues.

Some sites reported very high values for some measures and these data were not validated for this study, so findings in this exhibit should be interpreted with caution.

Quarterly information is not available for Year 1.

Source: MDRC calculations using HUD Jobs Plus Pilot Data Visualization Tool 112

APPENDIX F

SENSITIVITY ANALYSES

113

Exhibit F.1

Impacts on Earnings and Employment in the Four Years of Followup All Adults, Cohorts 1 to 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,137

10,267

– 130

0.158

Year 2

11,796

11,865

– 69

0.598

Year 3

13,207

13,338

– 131

0.429

Year 4

14,290

14,550

– 261

0.220

Years 1–4

48,779

49,377

– 598

0.278

Average Quarterly Employment (%)

Year 1

59.6

60.2

– 0.6

0.210

Year 2

62.7

62.9

– 0.2

0.710

Year 3

64.3

64.1

0.2

0.701

Year 4

64.3

64.2

0.1

0.822

Years 1–4

62.6

62.7

– 0.1

0.793

Employed at least One Quarter (%)

Year 1

72.8

72.9

0.0

0.925

Year 2

74.9

74.8

0.1

0.899

Year 3

76.1

75.5

0.6

0.237

Year 4

75.5

75.3

0.1

0.817

Years 1–4

86.9

86.3

0.6

0.141

Sample size (total = 24,678)

11,521

13,157

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

The study sample consists of focal adults (one adult per household) who were age 18-57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Source: The National Directory of New Hires

114

Exhibit F.2

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults, Cohorts 1 to 3

Full Sample

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,374

10,451

– 78

0.441

Year 2

11,706

11,724

– 18

0.897

Year 3

12,662

12,890

– 229

0.252

Year 4

13,521

14,004

– 483 *

0.050

Years 1–4

46,026

47,051

– 1,026

0.115

Average Quarterly Employment (%)

Year 1

59.7

60.1

– 0.4

0.409

Year 2

61.8

62.1

– 0.3

0.699

Year 3

62.9

63.1

– 0.2

0.775

Year 4

62.5

63.1

– 0.6

0.360

Years 1–4

59.5

60.0

– 0.5

0.374

Employed at least One Quarter (%)

Year 1

72.3

72.4

– 0.1

0.866

Year 2

73.6

73.8

– 0.2

0.765

Year 3

74.7

74.7

0.0

0.999

Year 4

74.0

74.4

– 0.5

0.420

Years 1–4

85.4

85.2

0.2

0.711

Sample size (total = 21,719)

10,392

11,327

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

Residents who are between the ages of 18 and 57 and non-disabled are included in these averages.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

The full sample includes a small proportion of residents (approximately 11 percent) who were excluded from the main impact analysis because they were not included in the initial sample member file constructed by HUD.

Source: The National Directory of New Hires

115

Exhibit F.3

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults, Cohorts 1 and 2

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

9,965

10,067

– 101

0.365

Year 2

11,333

11,282

51

0.740

Year 3

12,424

12,488

– 64

0.760

Year 4

13,723

13,871

– 148

0.569

Years 1–4

46,849

47,087

– 239

0.732

Average Quarterly Employment (%)

Year 1

59.6

59.8

– 0.2

0.709

Year 2

62.3

62.0

0.3

0.688

Year 3

63.4

62.8

0.6

0.514

Year 4

64.4

63.7

0.7

0.315

Years 1–4

62.3

61.9

0.4

0.576

Employed at least One Quarter (%)

Year 1

72.2

72.1

0.1

0.918

Year 2

74.0

73.6

0.5

0.564

Year 3

75.0

74.3

0.7

0.387

Year 4

75.0

74.5

0.5

0.509

Years 1–4

86.1

85.4

0.7

0.206

Sample size (total = 15,082)

7,588

7,494

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Source: The National Directory of New Hires

116

Exhibit F.4

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults, Cohorts 1 to 3, with Selection Bias Indicator

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,374

10,451

– 78

0.441

Year 2

11,755

11,700

56

0.680

Year 3

12,909

12,939

– 29

0.865

Year 4

13,829

14,070

– 241

0.272

Years 1–4

48,236

48,535

– 299

0.604

Average Quarterly Employment (%)

Year 1

59.7

60.1

– 0.4

0.409

Year 2

62.1

62.0

0.1

0.877

Year 3

63.2

62.8

0.4

0.551

Year 4

63.0

62.9

0.1

0.937

Years 1–4

61.9

61.8

0.0

0.979

Employed at least One Quarter (%)

Year 1

72.3

72.4

– 0.1

0.866

Year 2

73.8

73.7

0.2

0.775

Year 3

74.8

74.1

0.7

0.316

Year 4

74.0

73.9

0.1

0.825

Years 1–4

85.7

85.1

0.5

0.266

Sample size (total = 19,267)

9,220

10,047

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Source: The National Directory of New Hires

117

APPENDIX G

ADDITIONAL IMPACT ANALYSES

118

Exhibit G.1

Impacts on Employment Stability and Earnings History in the Four Years of Followup

Focal Adults, Cohorts 1 to 3

Program

Control

Outcome

Group

Group Difference

P-Value

Number of quarters employed

9.9

9.9

0.0

0.979

Length of longest employment spell

8.5

8.5

0.0

0.899

Ever worked 4 consecutive quarters (%)

71.5

71.5

0.0

0.989

Highest earnings (%)

Earned above $7,500/year

68.1

67.9

0.2

0.780

Earned above $10,000/year

62.7

62.7

0.1

0.931

Earned above $15,000/year

50.6

51.0

-0.4

0.552

Difference between highest quarterly earnings in Year 4

and Year 1 ($)

1,189.5

1,232.7

– 43.2

0.414

Sample size (total = 19,267)

9,220

10,047

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Source: The National Directory of New Hires

119

Exhibit G.2

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults Remaining in Study Developments the First-Two Years After Program Launch: Cohorts 1 to 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

10,372

10,450

– 78

0.505

Year 2

11,802

11,701

100

0.598

Year 3

13,045

12,989

56

0.825

Year 4

14,009

14,136

– 28

0.578

Years 1–4

48,695

48,624

72

0.911

Average Quarterly Employment (%)

Year 1

58.9

59.3

– 0.5

0.475

Year 2

61.5

61.3

0.2

0.753

Year 3

62.9

62.2

0.7

0.435

Year 4

62.8

62.7

0.1

0.836

Years 1–4

61.4

61.3

0.1

0.851

Employed at least One Quarter (%)

Year 1

70.8

71.1

– 0.3

0.694

Year 2

72.6

72.3

0.4

0.661

Year 3

74.0

73.1

0.9

0.299

Year 4

73.4

73.1

0.3

0.677

Years 1–4

84.7

83.7

1.1

0.109

Sample size (total = 12,407)

5,584

6,823

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Sources: The National Directory of New Hires and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data

120

Exhibit G.3

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults, Cohorts 1 to 3

by Baseline Employment Stability

Stably Employed

Not Stably Employed

Program

Control

Program

Control

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

17,948

18,012

– 64

0.687

3,832

3,913

-81

0.467

Year 2

18,891

18,670

221

0.332

5,593

5,740

– 147

0.470

Year 3

19,855

19,822

33

0.903

6,893

7,064

– 170

0.493

Year 4

20,721

20,980

– 259

0.460

7,912

8,178

– 266

0.302

Years 1–4

76,635

76,600

35

0.968

23,960

24,664

-703

0.340

Average Quarterly Employment (%)

Year 1

90.3

90.1

0.2

0.606

33.5

34.4

– 0.8

0.300

Year 2

86.6

85.8

0.7

0.391

41.1

41.7

– 0.6

0.574

Year 3

84.5

84.6

0.0

0.971

45.0

44.3

0.8

0.415

Year 4

82.9

82.4

0.5

0.536

46.1

46.4

– 0.4

0.641

Years 1–4

86.0

85.7

0.3

0.603

41.3

41.6

– 0.3

0.718

Employed at least One Quarter (%)

Year 1

97.9

97.8

0.1

0.755

50.3

50.6

– 0.3

0.771

Year 2

94.4

94.3

0.0

0.968

56.2

56.0

0.2

0.868

Year 3

92.8

93.0

– 0.2

0.818

59.3

58.1

1.2

0.191

Year 4

91.8

91.3

0.4

0.629

58.9

59.1

– 0.3

0.789

Years 1–4

99.4

99.5

0.0

0.853

73.9

73.0

0.9

0.288

Sample size (total = 19,236)

4,178

4,684

5,032

5,342

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent;

† = 10 percent.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Source: The National Directory of New Hires

121

Exhibit G.4

Impacts on Earnings and Employment in the Four Years of Followup Focal Adults

by Cohort

Cohort 1

Cohort 2

Program

Control

Program

Control

Outcome

Group

Group Difference

P-Value

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

9,858

9,809

49

0.727

10,148

10,362

214

0.208

Year 2

11,229

11,035

195

0.340

11,503

11,570

67

0.771

Year 3

12,292

12,154

138

0.561

12,653

12,860

207

0.555

Year 4

13,511

13,664

– 153

0.717

14,021

14,112

91

0.786

Years 1-4

46,313

46,365

– 52

0.966

47,662

47,979

317

0.714

Average Quarterly Employment (%)

Year 1

58.9

58.1

0.9

0.350

60.7

61.9

1.3 *

0.090

Year 2

61.7

60.7

1.0

0.419

63.2

63.6

0.4

0.716

Year 3

62.6

61.8

0.8

0.552

64.4

64.0

– 0.4

0.773

Year 4

63.9

63.4

0.5

0.591

65.0

64.1

– 0.9

0.413

Years 1–4

61.7

60.9

0.8

0.439

63.2

63.3

0.1

0.943

Employed at least One Quarter (%)

Year 1

71.1

70.0

1.1

0.254

73.8

74.8

1.1

0.218

Year 2

72.9

72.2

0.7

0.608

75.5

75.4

– 0.1

0.887

Year 3

74.0

72.9

1.1

0.301

76.4

76.1

– 0.3

0.819

Year 4

74.0

73.8

0.3

0.768

76.2

75.4

– 0.8

0.552

Years 1–4

85.3

84.1

1.2

0.191

87.1

87.0

– 0.1

0.870

Sample size (total = 19,267)

4,219

4,298

3,369

3,196

(continued)

122

Exhibit G.4 (continued)

Cohort 3

Program

Control

Outcome

Group

Group Difference

P-Value

Total Earnings ($)

Year 1

11,880

11,848

-31

0.920

Year 2

13,322

13,189

– 133

0.694

Year 3

14,726

14,570

– 156

0.697

Year 4

14,229

14,760

532

0.261

Years 1–4

53,405

53,698

294

0.800

Average Quarterly Employment (%)

Year 1

60.1

60.8

0.7

0.463

Year 2

61.5

61.8

0.3

0.786

Year 3

63.1

62.7

– 0.4

0.758

Year 4

58.0

59.9

1.9

0.188

Years 1–4

60.5

61.2

0.7

0.455

Employed at least One Quarter (%)

Year 1

72.8

73.1

0.3

0.778

Year 2

73.3

73.8

0.5

0.699

Year 3

74.2

73.2

– 1.0

0.463

Year 4

70.8

71.3

0.5

0.755

Years 1–4

84.2

84.2

0.0

0.986

Sample size (total = 19,267)

1,632

2,553

Notes: Rounding may cause slight discrepancies in sums and differences.

Sample sizes may vary because of missing values.

Distributions may not add to 100 percent because of rounding.

The p-value indicates the likelihood that the difference between the program group and the control group arose by chance.

The study sample consists of focal adults (one adult per household) who were age 18–57 and not identified as having a disability by the housing agency at the time that Jobs Plus implementation started in their development.

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 percent; † =

10 percent.

Estimates were regression-adjusted for baseline characteristics of sample members and site indicators.

Source: The National Directory of New Hires

123

APPENDIX H

JPEID CAPTURE RATE AND

DISCONTINUITIES ANALYSIS

124

One of the exploratory analyses included in the Jobs Plus Outcomes Evaluation is to examine how the structure of the Jobs Plus Earned Income Disregard (JPEID) affects residents’

work behavior and their reporting of earnings—specifically, whether the sudden onset and end of the JPEID affects residents’ work and earnings reporting behaviors or has other unintended consequences. As described in the body of the main report, the JPEID encourages residents in Jobs Plus developments to increase their income by removing the disincentive of an income-based rent increase. It is possible, however, that the sudden onset of the JPEID may incentivize residents to decrease their earnings—or reduce the amount of earnings they report—in the period leading up to JPEID enrollment to lock in a lower rent for the remainder of the program period.

When the JPEID ends, the sudden and potentially large jump in tenant rent that would occur if the household did increase their earnings during the program might lead residents to decrease their earnings to return to a lower and more manageable rent amount.

To explore these research questions, the analysis described in this appendix examines trends in average quarterly earnings and quarterly employment rates in the periods immediately preceding and following JPEID enrollment (for all 24 sites) as well as before and after the expiration of the benefit (for the two sites for which data are available for this time period). In sum, the exploratory analysis did not find evidence that nonelderly, nondisabled residents in Jobs Plus developments decreased their actual earnings in anticipation of the JPEID enrollment, or that the JPEID led to an increase in underreporting of earned income.

Data Sources and Analytic Approach

This analysis examines earnings and reporting trends for the same sample of 19,267 focal adults (9,220 in the Jobs Plus group and 10,047 in the comparison group) used for impact study: one adult per household aged 18 to 57 and who did not have disabled status based on HUD’s definition. The analysis exploring the potential effects of the sudden onset of the JPEID includes all 24 sites that are included in the main impact study.

Due, however, to almost all Jobs Plus grantees receiving grant extensions past the initially planned 4-year grant period, a minimum of two quarters of post-JPEID earnings data were only available for two sites in the study sample: Boston and Syracuse. As described in the main report, the combination of the sample sizes in individual sites being too small and the nonexperimental design of the study means that impact estimates in each individual site by themselves are likely not reliable; instead, the analysis focuses on the overall estimate pooling across the large number of sites as well as the characterization of variation across sites. 1

Similarly, for this analysis of the JPEID structure, less importance can be placed on trends in any one individual site. Nevertheless, given that sufficient post-program data are only available for 1 The Boston site includes 649 focal adults in the Jobs Plus group and 843 in the comparison group. The Syracuse site includes 239 focal adults in the Jobs Plus group and 668 focal adults in the comparison group.

125

two sites, the two sites are treated as case studies and their site-level trends are examined to assess whether there are any clear discontinuities in trends at the JPEID end date that might warrant further exploration, in a larger number of sites, of whether the structure of the JPEID, with the sudden end potentially leading to a sudden increase in rent (if a household increased earnings during the study period), leads to the unintended consequence of residents lowering their earnings after the program ends.

126

Exhibit H.1. Job

s Plus a

nd JPE

ID Star

t Date

s

Jobs Plus

JPEID

NDNH Quarter

Grantee

Start Date

Enrollment Date

Corresponding w ith JPEID Onset

Cohort One

Cohort One

Boston

April 2015

September 2015

Q3 2015

Charlotte

April 2015

October 2015

Q4 2015

Chicago

April 2015

September 2015

Q3 2015

Cuyahoga

April 2015

December 2015

Q4 2015

Houston

April 2015

April 2015

Q2 2015

Memphis

April 2015

March 2016

Q1 2016

Roanoke

April 2015

July 2015

Q3 2015

St. Louis

April 2015

August 2015

Q3 2015

Syracuse

April 2015

September 2015

Q3 2015

Cohort Tw o

Cohort Two

Oakland

December 2015

May 2016

Q2 2016

Sacramento

December 2015

June 2016

Q2 2016

Denver

December 2015

July 2016

Q3 2016

Goldsboro

December 2015

August 2016

Q3 2016

Philadelphia

December 2015

August 2016

Q3 2016

Nashville

December 2015

May 2016

Q2 2016

Austin

December 2015

July 2016

Q3 2016

San Antonio

December 2015

October 2016

Q4 2016

Norfolk

December 2015

January 2016

Q1 2016

Cohort Three

Cohort Three

Tampa

September 2016

April 2017

Q2 2017

Phoenix

September 2016

March 2017

Q1 2017

Baltimore

September 2016

April 2017

Q2 2017

Dayton

September 2016

April 2017

Q2 2017

New York City

September 2016

April 2017

Q2 2017

Providence

September 2016

April 2017

Q2 2017

The analysis uses two data sources for examining these trends: National Directory of New Hires (NDNH) wage data and the HUD Inventory Management System (IMS/PIH Information Center (PIC) data:

127

• This JPEID analysis uses the same individual-level NDNH wage data as the impact analysis described in the main body of the report except that, due to varying program implementation approaches and challenges (discussed in the main report) leading to different timelines in when the JPEID rolled out in each site, the data are aligned to the JPEID start date instead of the program start date.

Appendix exhibit H.1 shows each site’s JPEID start date (reported to MDRC by each PHA) relative to its grant award date. Depending on the grantee, JPEID was implemented between 0 (the same month as the grant award month) to 11 months after the grant was awarded. 2

• Individual-level earnings reported in the HUD PIC data were aggregated to the quarterly level to align with the NDNH quarterly wage data. While the NDNH

data include data on “formal” employment (such as employment covered by the Unemployment Insurance system), the PIC data also include any informal employment (not covered by the UI system) that the resident reports. Another notable difference is that while the NDNH data reflect actual earnings within a given quarter, the PIC data reflect reported earnings only for the month when a certification is made effective. For example, if residents increase their hours and therefore their earnings from $1,000 per month to $2,000 per month in January 2021, they may report that to the housing agency in February 2021 (within the agency’s reporting guidelines); then the housing agency may verify their earnings and make their new, higher rent effective in April 2021 to give the household the required one calendar month’s notice before increasing the tenant rent. Therefore, the residents’ higher earnings that began in January 2021 and were reported appropriately by the residents would not appear in the PIC data until April 2021.

To assess whether the sudden onset or end of the JPEID potentially had any effect on Jobs Plus residents’ actual earnings behaviors, average quarterly earnings and average quarterly employment rates in the period leading up to and following the JPEID enrollment start and end dates are examined for discontinuities in the trends relative to the comparison group’s trends. To assess whether this JPEID structure potentially affected Jobs Plus residents’ earnings reporting behavior, a “capture rate” measure was constructed that represents the proportion of NDNH

earnings that is “captured” by the earnings amount reported in the PIC data. This quarterly capture rate is examined over the study period to assess whether there are any discontinuities in the trends at the onset or end of the JPEID that might suggest that the JPEID incentivizes underreporting of earnings.

2 As a result, the quarter of NDNH data immediately following JPEID enrollment will sometimes include months that are both before and after the JPEID onset date. This may make earnings data from the first followup quarter more difficult to interpret since it may include earnings accrued both before and after JPEID enrollment. (Confirm site-reported start dates).

128

A major caveat that affects all the JPEID analyses conducted in this study is that the JPEID enrollment start date indicates when enrollment opened to households living in the Jobs Plus development. However, residents enrolled in an ongoing basis. The participation data presented in the main body of the report showed that JPEID enrollment levels were uneven across sites, and that JPEID enrollment tended to be ongoing, rather than all or most residents enrolling in the JPEID as soon as enrollment opened. By the end of the first year, only 19 percent of work-able residents were in households that were enrolled in the JPEID. By the end of the third year (the last year of participation data that was available), the enrollment rate had only increased to 40 percent. The likelihood of detecting a noticeable discontinuity in a trend throughout the study period decreases significantly with a slower enrollment and with a lower proportion of households enrolling for the benefit. The JPEID end date, on the other hand, ended at the same time for everybody when the grant ended, however, data for the post-program period are only available for two sites.

JPEID and Residents’ Work Behavior

This section describes the findings of the analysis of the trends in average quarterly NDNH

earnings leading up to (and following) the JPEID onset for the pooled sample of 24 sites, and it seeks to address the research question of whether the JPEID structure incentivized nonelderly, nondisabled residents to reduce their earnings in the period leading up to JPEID enrollment (and if so, if earnings increased again following JPEID enrollment) to lock in a lower rent for the remainder of the Jobs Plus program. It also reports the findings of the case study analysis of Boston and Syracuse that seeks to assess whether there is any indication that the sudden end of the JPEID leads to residents decreasing their earnings in response to the jump in their tenant rent, which would occur if they increased their earnings during the program period, because their earnings would have been fully disregarded until the program ended.

JPEID Onset and Residents’ Work Behavior

Earnings data from the period leading up to (and following) JPEID enrollment do not suggest that the onset of JPEID enrollment influenced earnings behavior for Jobs Plus residents.

As shown in exhibit H.2, average quarterly earnings for Jobs Plus residents were very similar to the earnings of residents in comparison developments during all four quarters of the pre-enrollment period, and through the 4 years following JPEID enrollment (and any small differences were not statistically significant during any quarter of the analysis period). The absence of any discontinuity in earnings trends of Jobs Plus residents relative to comparison group residents indicates that there is no evidence that nonelderly, nondisabled residents altered their earnings behavior in anticipation of enrolling the JPEID. It is likely that residents who are already working at the time that Jobs Plus began would be more likely to reduce their earnings in anticipation of being able to enroll in the JPEID, because they already had earnings to reduce.

Similar to the full sample findings, however, data for the subgroup with any earnings at program start show no compelling evidence that JPEID enrollment altered earnings behavior leading up to JPEID enrollment. As seen in exhibit H.2, earnings for Jobs Plus residents track very closely 129

with the control group during this period, and the difference in earnings between Jobs Plus residents and comparison group residents is not statistically significant for the employed at baseline subgroup at any point during the analysis period.

Exhibit H.2. Quart

erly Earn

ings Befo

re, Duri

ng, and Afte

r JPE

ID Imp

lementation Be

gins

Cohorts 1 to 3

A. Focal Adults

$5,000

$4,500

$4,000

$3,500

$3,000

$2,500

$2,000

$1,500

$1,000

JPEID Begins

$500

$0pre-Q4

pre-Q3

pre-Q2

pre-Q1

Q1

Q2

Q3

Q4

Q5

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

B. Focal Adults Employed at Baseline

$5,000

$4,500

$4,000

$3,500

$3,000

$2,500

$2,000

$1,500

$1,000

JPEID Begins

$500

$0pre-Q4

pre-Q3

pre-Q2

pre-Q1

Q1

Q2

Q3

Q4

Q5

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

(continued)

130

Exhibit H.2 (continued)

Notes: Estimates were regression-adjusted using ordi

nary least squares, controlling pre-program characteristics of sample members.

JPEID implementation start dates were reported by the site to MDRC. Quarter 1 (Q1) is treated as the quarter in which the sites began implementing the JPEID. Pre-Q 1 to Pre-Q4 are the quarters leading up to JPEID

implementation.

Rounding may cause slight discrepancies in calculati

ng sums and differences.

A two-tailed t-test was applied to differences between research groups.

The p-value indicates the likelihood that the differenc

e between the new rent rules group and the existing rent rules group arose by chance. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10

percent.

Sample sizes for specific outcomes may vary because of missing values.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

JPEID End and Residents’ Work Behavior

If a household increases its earnings during the program period, and 100 percent of those earnings are disregarded during the program period, the expiration of the JPEID could result in a steep increase in tenant rent. This section describes the exploratory case study analysis of Boston and Syracuse, which seeks assess whether there is any indication that residents might change their work behavior in response to the JPEID ending. It is also possible that Jobs Plus households who increased their earnings during the program period might begin decreasing them in the months leading up to the JPEID ending in anticipation of their rent increasing. In sum, NDNH

earnings data from Boston and Syracuse in the quarters leading up to and immediately following the expiration of the JPEID benefit do not provide any suggestion that the end of the earnings disregard influences earnings behavior.

As shown in exhibit H.3A, in Boston, the average quarterly earnings level of nonelderly, nondisabled residents remained relatively stable in the period leading up to the end of the program, as well in the two quarters following the end of the program. Average earnings are somewhat higher for nonelderly, nondisabled Jobs Plus residents during the program period compared with the comparison group, but the gap does not begin to close after the program ends, suggesting that Jobs Plus residents are not reducing their earnings in response to any rent increase they experienced at the end of the program. (In fact, the difference became more statistically significant in later quarters. 3) Exhibit H.3B, which shows these trends in Boston for the subgroup of sample members who were employed at the start of the Jobs Plus program, also does not suggest any work behavior change among this subgroup after the JPEID expired.

Exhibit H.4 shows these trends for nonelderly, nondisabled Jobs Plus residents in Syracuse.

Like in Boston, average quarterly earnings for Jobs Plus residents in Syracuse also remain 3 Note that data were only available for Boston for two quarters after the end of the program period, so if there was any lag in residents reducing earnings in response to rent increases beyond two quarters, it would not be captured in this analysis.

131

similar to those for residents in the comparison group during all four post-program quarters, as well as the quarters leading up to the JPEID expiration. This pattern is true for both the full analysis sample and for the subgroup of sample members who were already employed at baseline. There is a trend beginning two quarters after the end of the grant period that shows that the earnings of Jobs Plus residents are lower than that of the comparison group, which might be consistent with the hypothesis that Jobs Plus residents might decrease their earnings in response to a rent increase; however, the difference is not statistically significant, and more importantly, the earnings trends for Jobs Plus residents leading up to the JPEID expiration relative to the comparison group do not indicate that Jobs Plus residents increased their earnings in response to the earnings disregard, making it unlikely that there was a steep rent increase at the end of the grant period to respond to with changes in work behavior.

132

Exhibit H.3. Quart

erly Earni

ngs Befor

e, Duri

ng, and Afte

r JPEI

D Impl

ementation Be

gins

Boston

A. Focal Adults

$8,000

$7,000

$6,000

$5,000

$4,000

$3,000

$2,000

$1,000

JPEID Begins

JPEID Ends

(September 2015)

(September 2019)

$0

*

*

*

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

B. Focal Adults Employed at Baseline

$8,000

$7,000

$6,000

$5,000

$4,000

$3,000

$2,000

$1,000

JPEID Begins

JPEID Ends

(September 2015)

(September 2019)

$0

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

(continued)

133

Exhibit H.3 (continued)

Notes: Estimates were regression-adjusted using ordi

nary least squares, controlling pre-program characteristics of sample members.

JPEID implementation start dates were reported by t

he site to MDRC. Quarter 1 (Q1) is treated as the quarter in

which the sites began implementing the JPEID. Pre-Q1 to Pre-Q4 are the quarters leading up to JPEID

implementation.

Rounding may cause slight discrepancies in calculating sums and differences.

A two-tailed t-test was applied to differences betw

een research groups.

The p-value indicates the likelihood that the difference between the new rent rules group and the existing rent rules group arose by chance. Statistical significance levels

are indicated as: *** = 1 percent; ** = 5 percent; * = 10

percent.

Sample sizes for specific outcomes may vary becaus

e of missing values.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

134

Exhibit H.4. Qua

rterly Ear

nings Befor

e, Duri

ng, and After

JPEID

Imple

mentation Begins

Syracuse

A. Focal Adults

$6,000

$5,000

$4,000

$3,000

$2,000

$1,000

JPEID Begins

JPEID Ends

(September 2015)

(March 2019)

$0

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

B. Focal Adults Employed at Baseline

$6,000

$5,000

$4,000

$3,000

$2,000

JPEID Begins

JPEID Ends

$1,000

(September 2015)

(March 2019)

$0

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

(continued)

135

Exhibit H.4 (continued)

Notes: Estimates were regression-adjusted using ordi

nary least squares, controlling pre-program characteristics

of sample members.

JPEID implementation start dates were reported by the site to MDRC. Quarter 1 (Q1) is treated as the quarter in

which the sites began implementing the JPEID. Pre-Q1 to Pre-Q4 are the quarters leading up to JPEID

implementation.

Rounding may cause slight discrepancies in calculating sums and differences.

A two-tailed t-test was applied to differences between research groups.

The p-value indicates the likelihood that the difference between the new rent rules group and the existing rent rules group arose by chance. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10

percent.

Sample sizes for specific outcomes may vary because of missing values.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires

JPEID and Residents’ Earnings Reporting Behavior

This section describes the findings of the analysis that explores whether there is any indication that Jobs Plus residents change their earnings reporting behavior in response to the sudden onset and the sudden expiration of the JPEID. For example, subsidized housing residents might respond to the earnings disincentive of the traditional percent-of-income rent policy (where households pay approximately 30 percent of their income toward their housing costs) by underreporting their income. If this were the case, JPEID would provide an even stronger incentive to underreport income in the period leading up to JPEID enrollment, because residents would lock in a lower rent for the entirety of the program period. The JPEID expiration may also incentivize underreporting for the same reason.

The analysis described in this section examines trends in reporting behavior in the period leading up to the JPEID onset date, and for the two sites with post-program NDNH earnings data—Boston and Syracuse—the period following the JPEID end date. The measure used to examine these reporting trends is the “capture rate,” which is calculated as a proportion of total NDNH earnings in a quarter that is captured by PIC earnings in that same quarter. 4 The capture 4 The capture rate was calculated at the aggregate level, as the total earnings reported in PIC in a given quarter across the full sample divided by the total NDNH earnings for the same quarter. Calculating a capture rate at the individual-level (and then taking an average of the capture rates) was not feasible because this method was highly susceptible to individual capture rate outliers. Since PIC earnings should approach, but not exceed, NDNH

earnings, the expected maximum capture rate value is 1 (or 100 percent). When PIC earnings are substantially larger than NDNH earnings, however, the resulting capture rate is many times greater than 100 percent. These individual outliers can distort the overall mean and inflate the capture rate for a quarter. The influence that extreme high values exert on the mean can be addressed by omitting the values from the calculation or top-coding them to a more appropriate value. Removing or top-coding high-end outliers, however, can deflate the value of the aggregate capture rate by excluding cases where PIC earnings are high and NDNH earnings are low, but retaining observations where PIC earnings are low but NDNH earnings are high. Since a quarterly capture rate that uses individual ratios cannot be calculated in a manner that is not susceptible to outliers, the analysis instead calculates the capture rate as the ratio of the sample mean PIC earnings to the sample mean of NDNH earnings. To retain a measure that is created from individual values rather than at the sample level a difference in earnings was also calculated. Both measures are used examine potential changes in reporting behavior.

136

rate should remain relatively stable throughout the study period. A disruption in this trend where the proportion of NDNH earnings that is reported in the PIC data drops in the period leading up to JPEID enrollment (or in the period following JPEID expiration) might suggest that the JPEID structure incentivized earnings underreporting.

JPEID Onset and Residents’ Earnings Reporting Behavior

The overall trend in capture rates for the Jobs Plus group relative to the comparison group in the period leading up to and immediately following JPEID enrollment is presented in Appendix exhibit H.5. This figure shows that the capture rate for nonelderly, nondisabled Jobs Plus residents was constant over the analysis period, both by itself and relative to the comparison group. The figure shows no indication that Jobs Plus residents reduced the amount of earnings they reported to the housing agency in anticipation of benefiting from the JPEID.

Appendix exhibit H.6 shows the average dollar differences in PIC earnings and NDNH

earnings, and the patterns in this exhibit also do not suggest increased underreporting. For example, in the fourth quarter prior to JPEID enrollment, the average quarterly earnings based on the NDNH data was $2,131, and the average quarterly earnings based on the PIC earnings was $1,524. The difference between these two averages is $608; therefore, in the fourth quarter prior to JPEID enrollment, residents reported earnings that were, on average $608 lower than what employers reported to the Unemployment Insurance system for that quarter. The $1,524 average PIC earnings is 71.5 percent of the $2,131 average NDNH earnings. The dollar differences increase over time, however, they increase in proportion to the quarterly average earnings, which are also increasing over time.

During the eight-quarter analysis period, the capture rate for Jobs Plus residents ranges from 69.6 percent to 74.2 percent, meaning that the mean PIC earnings for the sample was approximately 70 to 74 percent of the mean NDNH earnings. The focus of this analysis is on the stability of the capture rate over time and if there are any discontinuities in the trends, especially in the time leading up to JPEID enrollment. It is important to note that the less-than-100-percent capture rate does not necessarily indicate that residents are consistently underreporting earnings.

There are several reasons that would lead to average PIC earnings being lower than average NDNH earnings that are not related to underreporting. First, even when a resident reports a new source of earnings or an increase in earnings, there is often a lag between when it is recorded in NDNH for that quarter (in real time) and when it is recorded in the PIC data, since the household has a period of time when they must report the income increase. Then, the housing agency must give them at least a calendar month’s notice for an increase in tenant rent. Since there will always be households who are in this lag period, total PIC earnings will always look lower than NDNH earnings for any UI-reported earnings. Second, it is possible that in some housing agencies, a household is required to report an increase in income immediately (or within a short period of time), but their tenant rent is not increased until their next recertification. This policy would lead to an even longer lag and would widen this gap between PIC and NDNH earnings further. Because some households will always be in this lag period in any given quarter, it is not 137

unexpected that PIC-reported earnings would consistently be lower than NDNH-reported earnings.

JPEID End and Residents’ Earnings Reporting Behavior

The trends in the capture rate for Boston and Syracuse can also provide some insight into whether there is any evidence that the JPEID ending (and any accompanying rent increases) led to underreporting of earnings in the period immediately following it. Appendix exhibit H.7

shows the capture rate trends for Boston starting four quarters before JPEID started through two quarters after JPEID ended, and appendix exhibit H.8 shows the capture rate trends for Syracuse starting four quarters before JPEID started through four quarters after JPEID ended. Neither of these graphs show any clear indication that nonelderly, nondisabled residents reduced how much earnings they reported in response to the JPEID ending.

In Boston, the capture rate for both the Jobs Plus group and the comparison group remained relatively stable throughout the followup period and did not deviate significantly from the comparison group. There were no differences in capture rates between the Jobs Plus and comparison groups in any quarters in the followup period.

The trends in Syracuse were less stable, and the Jobs Plus and comparison group rates did not track as closely as they did in Boston. The less stable capture-rate trend for the Jobs Plus group is likely due to the smaller sample sizes in this group (there were 239 focal adults in the Jobs Plus group in Syracuse and 668 focal adults in the comparison group in Syracuse).

Conclusion

Findings from this analysis that examined trends in actual earnings from NDNH wage data, as well as trends in earnings reporting behavior using a measure that compares reported PIC data with NDNH data, show that, overall, there is no indication that the JPEID, which is structured with a sudden onset at enrollment and a sudden end at expiration, changes work or income reporting behavior among nonelderly, nondisabled residents. The trends in earnings and reporting do not suggest that residents decrease their employment to reduce their earnings or reduce how much of their actual earnings they report in anticipation of the JPEID starting. There is also no evidence that nonelderly, nondisabled residents reduce their actual earnings or their reported earnings in response to the JPEID ending. There was, however, also no evidence from the main analysis that Jobs Plus had an impact on earnings, so it is possible that residents did not face a steep increase in rent due to increased earnings during the study period, so the Jobs Plus sample members would not have faced any differential incentive than the comparison group sample members.

There are several important caveats to keep in mind when interpreting these findings. First and foremost, while it was expected that most Jobs Plus households would enroll in the JPEID at the start of the Jobs Plus program, participation data indicated that residents gradually enrolled over an extended period of time, and that many sites did not achieve high enrollment rates, even 138

by the end of the third year. These two factors make it highly unlikely that an analysis of earnings and reporting trends over time would detect any discontinuity in those trends. Second, as described in the main body of the report, in 4 of the 24 sites, residents were not enrolled in Jobs Plus until their earnings increased. The fact that a change in earnings behavior is a prerequisite for enrollment makes it more difficult to determine how the JPEID may influence earnings behavior in those sites. Third, the lags in the PIC data in reflecting increased earnings (due to reporting lags and lags in higher rents becoming effective) mean that the two data sources are not closely aligned, making it difficult to detect changes in reporting behaviors based on a comparison between the two data sources within quarters. Finally, the fact that NDNH data are only available at the quarterly level (while PIC data are available at the monthly level) make these comparisons even less precise. Future analyses to examine the relationship between earnings, rent incentives, and income-reporting behaviors will need to take into consideration these limitations and challenges.

139

Exhibit H

.5. Capture R

ate Befor

e, Dur

ing, and

After

JPEID Implementation Begins

Focal A

dults, C

ohorts 1 to 3

90

80

70

) 60

(%e 50

Rate 40

ur

Capt 30

20

10

0

JPEID Begins

pre-Q4

pre-Q3

pre-Q2

pre-Q1

Q1

Q2

Q3

Q4

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

Notes: The capture rate is the ration between earnings reported to the housing authority and earnings doc

umented in the ND

NH records. T

his rat

e is calcul

ate

d in t

he aggregat

e f

or the

full s

ample at each quart

er.

JPEI

D implem

entation start

dates wer

e reported

by the sit

e to MD

RC. Quarter

1 (Q1)

is tre

ated as the

quarter in which the sites began implementing the JPEID. Pre-Q1 to Pre-Q4 are the quarters leading up to the

JPEID i

mple

ment

ati

on.

S

ource:

MDRC

calculations

usi

ng q

uarterly w

age data fr

om

t

he National

Direct

ory

of New

Hires and U.S.

Department of Housing and Urban Development Inventory Management System (IMS/PIH) Information Center (PIC) data

140

Exhibit H.6. Captu

re R

ate of Ea

rning

s, J

obs Plus Foc

al Adults

PIC Mean

NDNH Mean

Capture

Quarter Relative to JPEID Enrollment

Earnings

Earnings

Difference

Rate (%)

Pre-baseline Q4

$1,524

$2,131

-$608

71.5

Pre-baseline Q3

$1,577

$2,265

-$688

69.6

Pre-baseline Q2

$1,636

$2,302

-$665

71.1

Pre-baseline Q1

$1,683

$2,354

-$672

71.5

Follow -up Q1

$1,730

$2,447

-$717

70.7

Follow -up Q2

$1,798

$2,610

-$812

68.9

Follow -up Q3

$1,881

$2,619

-$738

71.8

Follow -up Q4

$1,978

$2,665

-$687

74.2

Notes:

The sample f

or this t

able sam

ple only incl

udes

people

who are in subsidized housing in the quarter of

interest. JPEID implementation start dates were reported by the site MDRC. Quarter 1 (Q1) is treated as the quarter

in which the sit

es began implementing t

he JP

EID.

P

re-Q1 t

o Pre-Q

4

are t

he quarter

s l

eadi

ng up t

o

JPEID

impl

em

entati

on.

Source: The National Directory of New Hires (NDNH) and U.S. Department of Housing and Urban Development Inventory Management System (IMS)/PIH Information Center (PIC) data 141

Exhibit H

.7. Capture R

ate Befo

re, D

uring, an

d After JPEID

Imp

lement

ation

Focal Adults: B

oston

100

90

80

) 70

% 60

e (

Rat 50

e ur 40

Capt 30

20

JPEID Begins

JPEID Ends

10

(September 2015)

(September 2019)

0

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

Notes: The capture rate is the ration between earnings reported to the housing authority and earnings documented

in the NDNH

records.

This rate is c

alculate

d in t

he aggregate f

or the f

ull s

ample at each

quarter.

JPEI

D imple

men

t

ati

on star

t dates w

ere rep

orted by the

sit

e to MD

RC. Quarter

1 (Q1) is treated as t

he

quarter in which the sites began implementing the JPEID. Pre-Q1 to Pre-Q4 are the quarters leading up to JPEID implementation.

Sources: MDRC calculations using quarterly wage data from the National Directory of New Hires and 142

Exhibit H

.8. Capture R

ate Befo

re, D

uring, an

d After JPEID

Imp

lement

ation

Focal Adults: S

yracuse

100

90

80

) 70

(% 60

e

Rat 50

e ur 40

Capt 30

20

JPEID Begins

JPEID Ends

(September 2015)

(March 2019)

10

0

Follow up quarter relative to quarter of JPEID Start

Jobs Plus Developments

Comparison Developments

Notes: The capture rate is the ration between earnings reported to the housing authority and earnings document

ed in the ND

NH recor

ds. This rate i

s calcul

ated in t

he aggregat

e for t

he full

sample at each

quarter.

JP

EID im

plem

en

tati

on st

art dates

were r

eported by t

he s

ite t

o he MDRC. Q

uarter 1 (Q1) is treated

as the

quarter in which the sites began implementing the JPEID. Pre-Q1 to Pre-Q4 are the quarters leading up to JPE

ID im

plementation.

Source: MDRC calculations using quarterly wage data from the National Directory of New Hires and U.S.

Department of Housing and Urban Development Inventory Management System (IMS/PIH Information Center (PIC) data

143

References

Blank, Susan, and Donna Wharton-Fields. 2008. Helping Public Housing Residents Find and Keep Jobs: A Guide for Practitioners Based on the Jobs-Plus Demonstration. New York: MDRC.

Bloom, Howard S., Charles Michalopoulos, Carolyn J. Hill. 2005. “Using Experiments to Assess Nonexperimental Comparison-Group Methods for Measuring Program Effects.” In Learning More from Social Experiments, edited by Howard S. Bloom. New York: Russell Sage Foundation: 173–235.

Bloom, Howard S., James Riccio, and Nandita Verma. 2005. Promoting Work in Public Housing: The Effectiveness of Jobs-Plus Final Report. New York: MDRC.

Bloom, Howard S., Stephen W. Raudenbush, Michael J. Weiss, and Kristin Porter. 2017. “Using Multisite Experiments to Study Cross-Site Variation in Treatment Effects: A Hybrid Approach with Fixed Intercepts and a Random Treatment Coefficient,” Journal of Research on Educational Effectiveness, 10:4, 817-842, DOI: 10.1080/19345747.2016.1264518

Greenberg, David M., Aurelia De La Rosa Aceves, David H. Greenberg, Ari Oppenheim, and M.

Victoria Quiroz-Becerra. 2015. The Second Generation of Jobs-Plus Programs: Implementation Lessons from San Antonio and the Bronx. New York: MDRC.

Hollenbeck, Kevin M. and Wei-Jang Huang. 2014. “Net Impact and Benefit-Cost Estimates of the Workforce Development System in Washington State.” Upjohn Institute Technical Report No. 13-029. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.

https://research.upjohn.org/up_technicalreports/29/

Leopold, Josh, Theresa Anderson, Marla McDaniel, Christopher R. Hayes, Sade Adeeyo, Rob Pitingolo. 2019. Helping Public Housing Residents Find Jobs and Build Careers Evaluation Findings from New York City’s Jobs-Plus Expansion. Washington, DC: Urban Institute.

Lipsey, Mark W., and David B. Wilson. 1993. “The Efficacy of Psychological, Educational, and Behavioral Treatment: Confirmation from Meta-Analysis,” American Psychologist 48 (12): 1181.

Miller, Cynthia, Nina Castells, Nandita Verma, Joshua Vermette, and Henry Kanengiser. In press. Long-Term Findings from the Original Jobs Plus Demonstration: Employment and Earnings for Public Housing Residents After 20 Years. New York: MDRC.

144

Riccio, James A. 2010. “Sustained Earnings Gains for Residents in a Public Housing Jobs Program: Seven-Year Findings from the Jobs-Plus Demonstration.” Policy brief. New York: MDRC.

Riccio, James, Nandita Verma, and Victoria Deitch. 2019. The Rent Reform Demonstration: Interim Findings on Implementation, Work, and Other Outcomes. Washington, DC: HUD

PD&R. https://www.huduser.gov/portal/publications/RentReform-InterimFindings.html.

Tessler, Betsy L., Nandita Verma, Jonathan Bigelow, Victoria Quiroz-Becerra, Kirstin P.

Frescoln, William M. Rohe, Michael D. Webb, Amy T. Khare, Mark L. Joseph, and Emily K. Miller. 2017. Scaling Up a Place-Based Employment Program: Highlights From the Jobs Plus Pilot Program Evaluation. New York: MDRC.

U.S. Department of Housing and Urban Development (HUD). 2014. “Jobs Plus Pilot Program NOFA.” FR-5800-N-24. Washington, DC: U.S. Department of Housing and Urban Development.

https://www.hud.gov/program_offices/administration/grants/fundsavail/nofa14/jppp.

———. 2015. “Jobs Plus Pilot Program NOFA.” FR-5900-N-10. Washington, DC: U.S.

Department of Housing and Urban Development.

https://www.hud.gov/sites/documents/2015JPPNOFA.PDF.

———. 2016. “Jobs Plus Pilot Program NOFA.” FR-6000-N-14A. Washington, DC: U.S.

Department of Housing and Urban Development.

https://www.hud.gov/sites/documents/2016JPI-NOFA.PDF.

Verma, Nandita, Edith Yang, Stephen Nunez, and David Long. 2017. Learning from the Work Rewards Demonstration: Final Results from the Family Self-Sufficiency Study in New York City. New York: MDRC. Available at SSRN: https://ssrn.com/abstract=3062110

Verma, Nandita, Stephen Freedman, Betsy L. Tessler, and Barbara Fink. 2021. Work, Engagement, and Well-Being at the Midpoint: Findings From the Family Self-Sufficiency Evaluation. Report prepared for the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. Washington, DC: HUD PD&R.

https://www.huduser.gov/portal/publications/FSS-Midpoint-2021.html

Verma, Nandita, Betsy L. Tessler, David H. Greenberg, Edith Yang, Sophia Sutcliffe, Michael D. Webb, William M. Rohe, Atticus Jaramillo, Amy T. Khare, Emily K. Miller, and Mark L.

Joseph. 2019. HUD’s Jobs Plus Pilot Program for Public Housing Residents: Ongoing Implementation Experiences. New York: MDRC.

145

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

April 2023

Document Outline

- Participation and Labor Market Impacts for the First 24 Sites to Replicate HUD’s Jobs Plus Program Final Report

- DISCLAIMER

- Acknowledgments

- Foreword

- TABLE OF CONTENTS

- Executive Summary

- Core Features of the Jobs Plus Framework

- The HUD Jobs Plus Replication Evaluation

- Characteristics of the Jobs Plus Developments and the Study Sample

- Program Participation

- By the end of Year 3 of the program, about one-half (or 52 percent) of all eligible residents had completed an initial Jobs Plus assessment, ranging from 26 percent to 79 percent across sites.

- On average, about 46 percent of the eligible residents in a development received some post-assessment employment services by the end of the third year of the program. Job search assistance was the most frequently provided type of employment service, followed by employment readiness.

- By the end of the first year of implementation, on average, 19 percent of work-able residents were in a household that was enrolled in the JPEID, and by the end of the third year, the enrollment rate had increased to 40 percent.

- Impacts on Employment and Earnings

- Average earnings in each of the 4 years of followup were very similar between the Jobs Plus and comparison groups: the difference is less than 2 percent in all 4 years, and no estimated differences are statistically significant.

- The average quarterly employment rate across the 4-year followup period is 62 percent for both the program group and the comparison group.

- Conclusion

- Introduction

- Brief Policy Context and Prior Research

- The Jobs Plus Model

- Prior Evidence of the Jobs Plus Program’s Effectiveness

- Implementation Experiences

- Early Jobs Plus Replication Initiatives

- HUD Jobs Plus Replication

- The Jobs Plus Outcomes Evaluation

- Sample Definition

- Data Sources

- Matched-Comparison Design

- Impact Analysis Approach

- Characteristics of Jobs Plus Developments

- Characteristics of the Study Sample

- Characteristics of Residents

- Characteristics of Households

- Resident Participation in Jobs Plus Activities

- Jobs Plus Enrollment and Participation in Case Management

- Participation in Employment Services

- Enrollment in JPEID

- Participation in CSW-Related Activities

- Overall Formal Resident Participation

- Impacts

- Overall Impacts

- Variation in Program Impacts Across Sites

- Impacts for Subgroups

- Discussion

- Conclusion

- APPENDIX A MATCH VALIDATION ANALYSIS

- Assessing the Match of the Initial Comparison Sample

- Cohen’s D Effect Sizes

- Kernel Density Plots

- Refining the Comparison Sample

- Assessing the Match of the Refined Comparison Sample

- Estimate of Bias

- APPENDIX B ANALYTIC MODEL & SPECIFICATIONS

- APPENDIX C JOBS PLUS AND COMPARISON GROUP BASELINE TABLES

- Baseline Characteristics of the Refined Comparison Sample

- APPENDIX D METHOD FOR CREATING THE COUNT OF ELIGIBLE RESIDENTS

- APPENDIX E ADDITIONAL PARTICIPATION TABLES

- APPENDIX F SENSITIVITY ANALYSES

- APPENDIX G ADDITIONAL IMPACT ANALYSES

- APPENDIX H JPEID CAPTURE RATE AND DISCONTINUITIES ANALYSIS

- Data Sources and Analytic Approach

- JPEID and Residents’ Work Behavior

- JPEID Onset and Residents’ Work Behavior

- JPEID End and Residents’ Work Behavior

- JPEID and Residents’ Earnings Reporting Behavior

- JPEID Onset and Residents’ Earnings Reporting Behavior

- JPEID End and Residents’ Earnings Reporting Behavior

- Conclusion

- References

Table of Contents

Participation and Labor Market Impacts for the First 24 Sites to Replicate HUD’s Jobs Plus Program Final Report

DISCLAIMER

Acknowledgments

Foreword

TABLE OF CONTENTS

Executive Summary

Core Features of the Jobs Plus Framework

The HUD Jobs Plus Replication Evaluation

Characteristics of the Jobs Plus Developments and the Study Sample

Program Participation

By the end of Year 3 of the program, about one-half (or 52 percent) of all eligible residents had completed an initial Jobs Plus assessment, ranging from 26 percent to 79 percent across sites.

On average, about 46 percent of the eligible residents in a development received some post-assessment employment services by the end of the third year of the program. Job search assistance was the most frequently provided type of employment service, followed by employment readiness.

By the end of the first year of implementation, on average, 19 percent of work-able residents were in a household that was enrolled in the JPEID, and by the end of the third year, the enrollment rate had increased to 40 percent.

Impacts on Employment and Earnings

Average earnings in each of the 4 years of followup were very similar between the Jobs Plus and comparison groups: the difference is less than 2 percent in all 4 years, and no estimated differences are statistically significant.

The average quarterly employment rate across the 4-year followup period is 62 percent for both the program group and the comparison group.

Conclusion

Introduction

Brief Policy Context and Prior Research

The Jobs Plus Model

Prior Evidence of the Jobs Plus Program’s Effectiveness

Implementation Experiences

Early Jobs Plus Replication Initiatives

HUD Jobs Plus Replication

The Jobs Plus Outcomes Evaluation

Sample Definition

Data Sources

Matched-Comparison Design

Impact Analysis Approach

Characteristics of Jobs Plus Developments

Characteristics of the Study Sample

Characteristics of Residents

Characteristics of Households

Resident Participation in Jobs Plus Activities

Jobs Plus Enrollment and Participation in Case Management

Participation in Employment Services

Enrollment in JPEID

Participation in CSW-Related Activities

Overall Formal Resident Participation

Impacts

Overall Impacts

Variation in Program Impacts Across Sites

Impacts for Subgroups

Discussion

Conclusion

APPENDIX A MATCH VALIDATION ANALYSIS

Assessing the Match of the Initial Comparison Sample

Cohen’s D Effect Sizes

Kernel Density Plots

Refining the Comparison Sample

Assessing the Match of the Refined Comparison Sample

Estimate of Bias

APPENDIX B ANALYTIC MODEL & SPECIFICATIONS

APPENDIX C JOBS PLUS AND COMPARISON GROUP BASELINE TABLES

Baseline Characteristics of the Refined Comparison Sample

APPENDIX D METHOD FOR CREATING THE COUNT OF ELIGIBLE RESIDENTS

APPENDIX E ADDITIONAL PARTICIPATION TABLES

APPENDIX F SENSITIVITY ANALYSES

APPENDIX G ADDITIONAL IMPACT ANALYSES

APPENDIX H JPEID CAPTURE RATE AND DISCONTINUITIES ANALYSIS

Data Sources and Analytic Approach

JPEID and Residents’ Work Behavior

JPEID Onset and Residents’ Work Behavior

JPEID End and Residents’ Work Behavior

JPEID and Residents’ Earnings Reporting Behavior

JPEID Onset and Residents’ Earnings Reporting Behavior

JPEID End and Residents’ Earnings Reporting Behavior

Conclusion

References