FINAL RESEARCH REPORT

Assessing HUD

Technical Assistance

Programs

Evaluating Change

Since Implementation of

Community Compass

U.S. Department of Housing and Urban Development | Office of Policy Development and Research

DISCLAIMER

The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S.

Government.

FINAL RESEARCH REPORT

Assessing HUD Technical

Assistance Programs

Evaluating Change Since Implementation of Community Compass

Corianne Payton Scal y

Diane K. Levy

Camille H. Anoll

Alyse D. Oneto

Clare Salerno

Christina Plerhoples Stacy

Joseph Schilling

Claudia Aranda

Ananya Hariharan

Jorge Morales-Burnett

Peace Gwam

Kimberly Burrowes

Amanda Gold

Urban Institute

November 2020

Foreword

HUD technical assistance (TA) enhances the capacity of HUD’s partners to implement HUD programs and policies. HUD TA comes in many forms, from online trainings to comprehensive one-on-one support. Individual HUD program offices have funded TA activities for decades, but in 2014, HUD

consolidated its TA funding into the Community Compass Technical Assistance and Capacity Building Program (Community Compass). For fiscal years 2020 and 2021, HUD has made available approximately $161 million in funding for TA through Community Compass.

This “Assessing HUD Technical Assistance Programs” report evaluates the first 3 years of the Community Compass program, from 2014 to 2017. Using data obtained through interviews, case studies, administrative data, and document review, the study asks three questions: what TA does HUD

provide? how does HUD provide TA? and how effective is the TA perceived to be in the context of the implementation of the Community Compass structure in 2014? The study shows that HUD TA affects all aspects of HUD program implementation because it reaches so many HUD customers, including local and state governments, nonprofit organizations, public housing authorities, Tribally Designated Housing Entities, and Continuums of Care. The study also finds that although the majority of TA funds continue to be spent on TA customized to the needs of specific customers, over time, HUD has increased the use of universal TA approaches, such as online training, that provide a cost-effective approach to assisting customers facing similar problems. The study further finds that the shift to the Community Compass model centralized TA program administration, which reduced duplication, improved the award processing, and standardized the TA request and review process.

Although the study documents many recent improvements in the TA program, it concludes that the Community Compass goal of increasing cross-program collaboration throughout the Department is still a work in progress and notes mixed results as to the timeliness of the TA process.

Further, the study reveals room for improvement in HUD’s oversight of the TA program and in HUD’s ability to evaluate the effectiveness of TA. Since the period of review, the Technical Assistance Division has made efforts to improve the risk mitigation, monitoring, standardization of award administration protocols, and performance reporting.

One of the more important findings from the study is that data collection and reporting through HUD’s TA Portal and the Disaster Recovery Grant Reporting (DRGR) systems are not well coordinated. Subsequent to this report’s period of review, however, the Technical Assistance Division

has merged the two data collections into one system, thereby fixing the disconnect across the platforms.

The findings from this first assessment of the Community Compass program will help HUD

continue to improve its TA delivery processes, ultimately better supporting HUD’s customers and ensuring more effective use of HUD TA funding.

Todd Richardson

General Deputy Assistant Secretary for Policy Development and Research U.S. Department of Housing and Urban Development

Contents

List of Exhibits

ix

Acknowledgments

xi

Executive Summary

xii

Findings

xiii

Technical Assistance Customers and Providers

xiii

The Technical Assistance Process: Step by Step

xiv

Timelines and Costs

xv

Communication, Coordination, and Collaboration

xvi

Monitoring and Evaluation

xviii

Perceived Impacts

xix

Recommendations for Moving Forward

xx

Improve Communication and Coordination

xx

Develop Efficient and Flexible Work Plans

xxi

Increase Access to Work Products

xxi

Improve Data Systems and Data

xxi

Track Clear Outcomes

xxii

I. Introduction

1

HUD Technical Assistance Objectives

1

HUD Technical Assistance Types

3

Community Compass Overview

5

Existing Research and Study Contribution

7

Organization of the Report

8

Definitions of Key Terms

9

II. Study Design

12

Research Questions

12

Data

14

Administrative Data and Variables

14

Interview Data

18

Document Data

18

Sampling and Case Study Selection

19

Technical Assistance Programs and Interview Respondents

19

Case Studies

20

Analysis

23

Study Limitations

24

i v

C O N T E N T S

Technical Assistance Customers

24

Costs

25

Work Plans

25

III. Technical Assistance Customers and Providers

26

Introduction

26

Key Takeaways

26

Customers

26

Providers

27

Technical Assistance Customers

27

Customer Characteristics

27

Customer Capacity

28

Technical Assistance Providers

29

HUD Offices as Technical Assistance Providers

31

IV. The Technical Assistance Process: Step by Step

32

Introduction

32

Key Takeaways

32

Notice of Funding Availability and Provider Selection Process

33

Technical Assistance Requests and Need Identification

33

Provider Assignment and Work Scope

33

Technical Assistance Engagement

34

Closeout

34

Process Overview

34

Notice of Funding Availability and Provider Selection Process

35

Technical Assistance Requests and Need Identification

38

Targeted Technical Assistance

38

Universal Technical Assistance

40

Required Technical Assistance

40

Technical Assistance Requests Not Approved

41

Provider Assignment and Work Scope

42

Technical Assistance Engagement

44

Targeted Technical Assistance Compared with Universal Technical Assistance 45

Targeted Technical Assistance

45

Universal TA

47

Delivering and Managing Technical Assistance Engagements

48

Closeout

49

V. Timelines and Costs

50

Introduction

50

C O N T E N T S

v

Key Takeaways

50

Timeliness of the Technical Assistance Process

50

Types of Technical Assistance and Costs Over Time

51

Timeliness of the Technical Assistance Process

52

Request Approvals, Provider Assignments, and Work Plan Approvals

52

Period of Performance and Length of Technical Assistance Engagements

53

Types of Technical Assistance and Costs over Time

54

VI. Communication, Coordination, and Collaboration

58

Introduction

58

Key Takeaways

58

Cross-Program Collaboration at HUD

58

Shifting Relationship and Roles: HUD Field Offices

59

Communication Between Technical Assistance Providers and HUD

59

Coordination and Collaboration Across Technical Assistance Providers

59

Cross-Program Collaboration at HUD

60

Shifting Relationship and Roles: HUD Field Offices

61

Communication Between Providers and HUD

63

Coordination and Collaboration Across Technical Assistance Providers

64

VII. Monitoring and Evaluation

68

Introduction

68

Key Takeaways

68

Reporting Processes

68

HUD Oversight

69

Evaluation

69

Reporting Processes

69

HUD Oversight

71

Evaluation

72

Choosing the Right Metrics

73

Evaluating Outcomes: Troubled Public Housing Agency Technical Assistance 75

Collecting Informal Feedback

75

Ongoing Evaluation Efforts

77

VIII. Perceived Impacts

81

Introduction

81

Key Takeaways

81

Customer Perceptions

81

Provider Perceptions

82

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance 82

v i

C O N T E N T S

Customer Perceptions

82

Satisfaction with Technical Assistance Engagement

82

Impact of Technical Assistance

84

Provider Perceptions

86

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance 87

IX. Recommendations for Moving Forward

88

Introduction

88

Improve Communication and Coordination

88

Customers

89

Providers

89

Across HUD

90

Develop Efficient Work Plans

91

Increase Access to Work Products

91

Improve Data Systems and Data

92

Document and Update Systems

92

Improve Data Quality

93

Track Clear Outcomes

95

X. Summary of Findings and Future Research

96

Introduction

96

Summary of Key Findings

96

Future Evaluation and Research

98

Appendix A. Community Compass Offices, Divisions, Grant Programs,

and Appropriation Accounts

100

Appendix B. Additional Information on Study Implementation

103

Identifying Troubled Public Housing Agency Technical Assistance Customers 103

Identifying SNAPS Technical Assistance Customers

104

In-Depth Interview Respondent Characteristics

104

Respondents by Technical Assistance Type and Program Office

105

Respondents by Stakeholder

106

Troubled Public Housing Agency Interview Respondent Characteristics

110

HUD Headquarters Staff

110

Technical Assistance Customers

110

Technical Assistance Providers

112

SNAPS Interview Respondent Characteristics

112

HUD Headquarters Staff

113

HUD Field Office Staff

113

Technical Assistance Customers

113

C O N T E N T S

v i i

Technical Assistance Providers

114

Appendix C. Troubled Public Housing Agency Notification and Technical Assistance Need Identification Process

115

References

117

Additional Reading

119

v i i i

C O N T E N T S

List of Exhibits

Exhibit 1: HUD Technical Assistance Types and Definitions by Universal and Targeted Approaches .... 5

Exhibit 2: HUD Technical Assistance Appropriations, FY 2013–FY 2019 ..................................................... 7

Exhibit 3: Research Questions by Category and Associated Data Sources ................................................. 13

Exhibit 4: Variables by Analytic Category Drawn from TA Portal .................................................................. 16

Exhibit 5: SNAPS Office and Programs ................................................................................................................ 22

Exhibit 6: The Office of Public and Indian Housing and Troubled PHA TA ................................................. 23

Exhibit 7: Number and Percentage of Troubled and NonTroubled Public Housing Agencies that

Received Technical Assistance, FY 2012–FY 2017 .......................................................................................... 28

Exhibit 8: Number of Technical Assistance Providers and Work Plan and Budget Amounts per

Technical Assistance Providers, FY 2013–FY 2017 .......................................................................................... 30

Exhibit 9: General Technical Assistance Process ............................................................................................... 35

Exhibit 10: NOFA Process and Provider Selection ............................................................................................ 36

Exhibit 11: Technical Assistance Need Identification ....................................................................................... 38

Exhibit 12: HUD Technical Assistance Estimated Work Plan Budget by Work Plan Origination and

Technical Assistance Type, FY 2012–FY 2017 .................................................................................................. 39

Exhibit 13: Provider Assignment and Work Scope ............................................................................................ 42

Exhibit 14: SNAPS Provider Assignment Process .............................................................................................. 43

Exhibit 15: Technical Assistance Engagement ................................................................................................... 44

Exhibit 16: Help Desk Support .............................................................................................................................. 47

Exhibit 17: Median Number of Days Since Request Date For Technical Assistance Assignments,

Approvals, and Completions for Technical Assistance Requested through the Technical Assistance

Portal, FY 2012–FY 2017 ....................................................................................................................................... 52

Exhibit 18: Technical Assistance Portal Estimated Budget by Technical Assistance Type, FY 2013–FY

2017 ............................................................................................................................................................................ 55

Exhibit 19: Percentage of HUD Technical Assistance Estimated Budget by Technical Assistance Type

versus Work Plans by Technical Assistance Type, FY 2017 ........................................................................... 56

Exhibit 20: Adding Technical Assistance for priority initiatives ...................................................................... 66

Exhibit 21: HUD Guidance On Standard Outcomes By Category ................................................................. 79

Exhibit 22: Interviewees by Technical Assistance Type ................................................................................ 105

Exhibit 23: Interviewees by HUD Program Office ......................................................................................... 106

Exhibit 24: Providers by Technical Assistance Type and Program Office ................................................. 109

Exhibit 25: Number of Years Troubled ............................................................................................................. 111

C O N T E N T S

i x

Exhibit 26: Number of Public Housing Agencies with Troubled Designation .......................................... 111

Exhibit 27: Public Housing Agency Interviewee Characteristics between Fy 2011 and FY 2018 ....... 112

Exhibit 28: Office of Community Planning and Development Field Offices by HUD Region, by Number

and Type of Technical Assistance work Plans, FY 2012–FY 2017 ............................................................. 113

Exhibit 29: SNAPS Technical Assistance Providers by Number of Work Plans, FY 2012–FY 2017 ... 114

x

C O N T E N T S

Acknowledgments

This report was funded by the U.S. Department of Housing and Urban Development (HUD) under Cooperative Agreement H-21671CA. We are grateful to HUD and to all our funders, who make it possible for Urban to advance its mission.

The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at urban.org/fundingprinciples.

The authors thank the HUD staff who supported, advised, and reviewed this project, especially Lynn Rodgers, who served as the Government Technical Reviewer. HUD staff also provided helpful support and advice: Christine Brown, Felicia Gaither, Peter Han, John Huggins, Jeffrey Hunt, Peter Kahn, Hunter Kurtz, David Larimer, Abbilyn Miller, Christopher Narducci, Ted Payne, Claudia Parrin Krisko, Lindsey Reames, Kenneth Rogers, Stephen Schneller, Trenessa Sidney, Candace Simms, Emily Sokol, and Stephanie Stone. HUD staff from Policy Development and Research provided additional review and feedback: Heidi Joseph, Todd Richardson, Elizabeth Rudd, Mark Shroder, and Carol Star.

The authors also would like to thank external expert advisors for their review and support: Ophelia Basgal, Anna Blasco, Louise Parker, and Cliff Taffet.

The authors are grateful to all the research participants who helped support this research project, including HUD headquarters staff, HUD regional and field staff, technical assistance (TA) providers, and TA customers.

Finally, the authors thank their colleagues at the Urban Institute who contributed to this project’s development and data analysis—Ilana Brody, Lydia Lo, and Justin Morgan—and Brett Theodos for serving as technical reviewer.

A C K N O W L E D G M E N T S

x i

Executive Summary

The U.S. Department of Housing and Urban Development (HUD) supports public housing agencies (PHAs), owners and operators of multifamily housing properties, state and local governments, and other funding recipients through the provision of technical assistance (TA). TA is intended to help recipients acquire the skills and knowledge necessary to implement programs, adjust to policy changes, or improve program performance and compliance. Historically, HUD program offices offered TA independently of each other. However, the structure for TA provision began to change in 2010, when HUD began a shift toward a more coordinated and collaborative approach among program offices through its OneCPD Integrated Practitioner Assistance System and the Transformation Initiative. The Transformation Initiative began increasing coordination across the Community Planning and Development (CPD), Public and Indian Housing, and Fair Housing and Equal Opportunity offices.

In 2014, HUD initiated Community Compass by consolidating TA funding for several programs across five offices: CPD, Public and Indian Housing, Fair Housing and Equal Opportunity, Office of Housing, and Office of Economic Development.

This report presents findings from an assessment by the Urban Institute conducted of Community Compass TA programs. The descriptive study pursued three questions: what TA does HUD provide, how does HUD provide TA, and how effective is the TA perceived to be? This study examines Community Compass broadly and through two case studies: (1) TA provided by the Office of Special Needs Assistance Programs (SNAPS) with a focus on TA for recipients of HUD funding to address homelessness and (2) TA provided to PHAs that HUD identified as troubled, based on assessments of management quality. Findings from this first assessment of Community Compass are intended to inform HUD’s TA delivery by providing detailed descriptions of TA delivery processes, information on what is working well, and suggestions for improvements.

Study findings are based on analysis of multiple sources of data: administrative data from HUD’s TA Portal system generally from FY 2013 through FY 2017, a series of broad interviews with HUD

headquarters and field staff, TA providers and customers, and more targeted interviews across these stakeholders on the two case study programs supplemented by program documentation.1 TA Portal 1 In addition to the TA Portal, HUD collects administrative data through the Disaster Recovery Grant Reporting (DRGR) System, which is used by TA providers to invoice and document funded activities and by HUD to monitor and review the activities. TA Portal data are used for this assessment because they include variables unavailable from DRGR, such as TA funding source, details on types of TA provided, and names of TA providers and customers, and because initial analysis found inconsistent entry of the TA work plan identification variable in DRGR, which resulted in a low match rate of work plans across the two datasets.

x i i

E X E C U T I V E S U MM A R Y

data tracks TA request status and approvals and customer names and work plan information, including work plan ID, estimated budget, dates, funding source, program office, and topic. HUD uses the TA Portal to assist with TA management. Interviews focused on steps in the TA process, how the process and interactions among key stakeholders have changed since the implementation of Community Compass, perceptions of the value and impact of HUD TA, and suggestions for how TA implementation might be improved. Interviews conducted for the case studies focused on these topics relative to the particular TA program.

Throughout the report, TA is discussed in terms of general approach as defined by the literature: universal or targeted. Universal TA—such as self-paced, web-based courses and interactive tools that can be accessed independently—is applicable to all customers who need similar information and typically requires no direct contact with a TA provider. Targeted TA involves direct contact between a TA provider and customer to determine needs and develop and implement a work plan to meet them.

HUD defines eight TA types through its Notices of Funding Availability (NOFAs) to solicit TA providers: three types using a predominantly universal approach (tools and products, help desk support, and self-directed and group learning) and five types that use a more targeted TA approach (on-call; direct TA and capacity building; needs assessments; data reporting, analysis, and management; and knowledge management).

Findings

The report presents findings in six chapters, addressing who receives and provides TA under Community Compass; the TA process itself; TA timelines and costs; communication, coordination, and collaboration integral to TA; monitoring and evaluation; and perceived impact.

Technical Assistance Customers and Providers

The study examined who receives and provides TA to understand key stakeholders to the TA process.

Findings include the following:

◼

TA customers include local and state governments, nonprofit organizations, PHAs, Tribally Designated Housing Entities, and Continuums of Care (CoCs). Based on limited customer identification data available from TA Portal work plans, slightly more than one-half of the unique customers are PHAs and CoCs. For the case study programs, a majority of SNAPS

E X E C U T I V E S U MM A R Y

x i i i

customers are CoCs. For PHAs, 13 percent that are designated as troubled PHAs receive TA, but 13 percent of nontroubled PHAs received TA as well.

◼

Larger customer organizations appear to have more experience with HUD TA and greater capacity to navigate it than smaller ones, based on interviews with a small sample of customers.

◼

Most providers are consulting firms or housing and community development organizations, many of which supplement staff with subcontractors or consultants. Firms may specialize in providing TA for a program or type of TA.

◼

The number of providers has increased over time, as has the average number of work plans each provider executes. Work plans and budget data indicate an increase in the number of providers that receive relatively small contracts.

◼

Finally, HUD staff provide TA when warranted by their role in overseeing a program or their subject matter expertise.

The Technical Assistance Process: Step by Step

The five key steps in the TA process under Community Compass—the NOFA process and provider selection, need identification, provider assignment and work scope, TA engagement, and closeout—

can vary based on the approaches and types of TA and the HUD office and program involved.

◼

NOFA process and provider selection: This step was centralized into one process that reduced duplication and sped up TA providers receiving the funding awards.

◼

Need identification: Community Compass standardized the TA request process for targeted TA to be submitted through the TA Portal, but these requests often originate from HUD

headquarters or field staff. Universal TA projects often result from observations by HUD

headquarters and field staff identifying a shared need across multiple customers; these needs are often identified through HUD’s annual TA planning process and submitted to Congress.

HUD headquarters staff may require HUD funding recipients to participate in some forms of TA, typically to promote knowledge about new guidelines or build capacity to implement new requirements, and sometimes as a condition for receiving HUD funds or participating in a HUD program, or when a troubled PHA includes TA in its recovery agreements. Many requests in the TA Portal were not approved (44 percent); many of them were requests for assistance not available through Community Compass TA programs. These requests include x i v

E X E C U T I V E S U MM A R Y

those for housing assistance from individuals (who are not eligible for TA) and grantees who are referred to other resources and HUD’s Ask-A-Question help desks.

◼

Provider assignment and work scope: TA providers have conversations with HUD

headquarters staff about their availability and capacity for a TA engagement before or upon receiving an assignment by the HUD Government Technical Reviewer (GTR). TA providers develop TA work plans in conjunction with HUD headquarters or field staff before beginning a TA engagement.

◼

TA engagement: HUD has increased the use of universal TA across program offices to maximize its reach to customers facing similar problems while minimizing the cost of providing TA. Targeted TA is still needed to support TA customers with unique challenges by providing TA tailored to individual contexts. Some TA engagements lead to additional TA requests, such as escalation from on-call TA to a direct TA engagement.

◼

Closeout: The TA Portal provides little guidance or information on closing out TA work plans, and some work plans stay open after the TA engagement has ended. TA providers receive guidance and close out work plans in the Disaster Recovery Grant Reporting System (DRGR), the official award management system for TA cooperative agreements.

Timelines and Costs

Findings on the timeliness of TA processes and the types of TA that are provided over time and their associated costs include the following:

TIMELINESS OF THE TA PROCESS

◼

According to administrative data, one-half of all TA requests received through the TA Portal take less than a week to be approved by the relevant program office as an eligible request for TA and for a TA provider to be assigned to respond to the request. The median time it takes from the request date to approval of the work plan developed by the TA provider is 2 months for the Government Technical Monitor within the program office and a little over 3 months by the GTR, typically based in CPD’s Technical Assistance Division.

◼

Some HUD headquarters staff believe approval times improved under Community Compass, whereas other staff thought moving GTRs from most program offices to CPD’s Technical Assistance Division slowed the TA administration process because the offices lost immediate access to staff who approve and oversee the TA.

E X E C U T I V E S U MM A R Y

x v

◼

CPD field staff indicated that changes since the shift to Community Compass have increased the time it takes to approve requests.

◼

At least one-half of all TA work plans are completed in less than a year, including the delivery of direct TA and the development of universal TA tools and products.

◼

Customers can work their way through available universal TA tools and products quickly, but targeted TA engagements often take longer, from developing work plans to completing the TA delivery.

TYPES OF TA AND COSTS OVER TIME

◼

Direct TA accounts for most of HUD’s TA costs documented in the TA Portal but represents a decreasing share of costs as other TA types, including tools and products and self-directed and group learning, have increased since Community Compass. Direct TA and tools and products work plans cost the most of all TA types, whereas self-directed and group learning work plans cost the least.

◼

SNAPS TA represents around 40 percent of HUD’s overall estimated TA budget and has a similar distribution of TA types over time to HUD’s overall TA budget.

◼

PHA customers account for almost one-half of HUD’s direct TA budget, a significant increase since Community Compass launched. Based on a review of work plans, PHAs almost always receive direct TA, although customer access to online universal TA materials is not represented in the TA Portal data.

Communication, Coordination, and Collaboration

The study considered how the TA process works from the perspective of key stakeholders and how processes for communication, coordination, and collaboration have changed under Community Compass.

CROSS-PROGRAM COLLABORATION AT HUD

◼

There is little evidence from interviews of increased cross-program collaboration among HUD

headquarters program offices, although the NOFA process changed and many TA processes were centralized within CPD.

x v i

E X E C U T I V E S U MM A R Y

◼

Many HUD program office staff think the complexity of HUD program-specific TA structures limits the ability of HUD staff to collaborate across offices, although HUD TA is moving slowly toward increased efficiency and collaboration.

SHIFTING RELATIONSHIP AND ROLES: HUD FIELD OFFICES

◼

Since the shift to Community Compass, some field offices have maintained a close connection with TA processes, but others feel more disconnected than before.

» Public and Indian Housing field office staff who oversee PHAs have a closer relationship with HUD headquarters staff and TA providers and a deeper engagement in the TA process than other field office staff.

» Community Compass reduced some field offices’ roles—including SNAPS within CPD—by centralizing decisions and funding in HUD headquarters program offices, leaving field staff feeling disconnected. CPD field staff expressed concern that customer needs may not be met effectively without the field staff providing more input into the TA process.

COMMUNICATION BETWEEN TA PROVIDERS AND HUD AND COORDINATION AND

COLLABORATION ACROSS TA PROVIDERS

◼

Communication between HUD staff and providers occurs regularly, and TA providers find HUD staff to be supportive across program offices and TA programs.

◼

Provider coordination is encouraged under Community Compass, but evidence of improved coordination and collaboration across TA providers was uneven and varied by program office.

» The SNAPS office prioritizes coordination across providers by bringing them together to help design TA around program priorities, share topical knowledge, and assign providers to individual TA engagements based on regional needs. SNAPS TA focuses on priority initiatives—which target specific populations experiencing homelessness—and promotes greater collaboration among providers and sharing of TA materials.

» Troubled PHA TA providers collaborate when working in the same region, with the same client, or on the same product when asked by HUD to do so.

◼

The proprietary nature of the work and information asymmetries are challenges to provider collaboration.

E X E C U T I V E S U MM A R Y

x v i i

Monitoring and Evaluation

The description of HUD’s TA monitoring processes and evaluation efforts includes key stakeholders’

perceptions of the effectiveness of reporting systems for facilitating the TA process.

REPORTING PROCESSES

◼

The data and reporting processes for TA engagements are seen as effective because they provide a single place to track all TA requests and engagements in the TA Portal, set a standard process for customers to request TA and for providers to receive assignments using the TA Portal, and simplify the process for providers to submit monthly activity reports and invoices in the DRGR.

◼

Improvements are needed to the data and DRGR and TA Portal reporting systems to enable linking of data across systems, reduce the time it takes to enter reports, and ensure that entries are saved correctly.

◼

Some providers use additional reporting systems at the request of HUD staff, which can require duplicative information.

HUD OVERSIGHT

◼

Standardized reporting mechanisms assist HUD staff with their oversight activities, but limited access to the reporting systems makes it more difficult to monitor progress on a TA engagement over time, leading some HUD staff to create their own methods for tracking TA.

◼

Field office staff expressed that their ability to monitor TA and customers’ progress was limited by a lack of access to data systems and training on how to use them.

EVALUATION

◼

The standardized data systems created under Community Compass, containing all work plans across program offices, could be useful for evaluating outputs and outcomes associated with HUD TA if challenges with data quality and completeness were addressed.

◼

There are ongoing debates within HUD’s program offices on whether the impact of TA should be measured at the level of the individual TA customer or a broader systems level.

◼

TA providers and HUD staff agree that the current outputs and outcomes tracked are too broad and short term to be useful for evaluating TA engagements.

x v i i i

E X E C U T I V E S U MM A R Y

◼

In the absence of established metrics for measuring outcomes and impacts, HUD staff and TA providers have relied on informal methods for gathering feedback by conducting internal program assessments and hosting calls with customers and providers.

◼

HUD implemented new policies in 2018 to evaluate TA effectiveness, including administering a standard survey to all TA customers who receive direct TA or participate in the in-person training.

Perceived Impacts

The study examined customers’ and providers’ perceptions of the impact of HUD TA. Findings include the following:

◼

Most of the TA customers interviewed indicated that the TA they received satisfied their initial request and appropriately addressed their organization or community’s need. Customers indicated dissatisfaction with the TA process and outcomes when it took a long time to receive TA after requesting it and when the TA they received was not specific enough to their needs.

◼

Customers found universal TA best for answering discrete, simple questions and more accessible to smaller organizations, whereas targeted TA was best for learning new processes and identifying root causes of issues they faced.

◼

Qualitative evidence from a small, diverse sample of TA customers indicates a positive impact of TA on improving organization and program management and developing strategic approaches to their work. TA customers and HUD staff thought that the impact of TA is directly related to a recipient’s preparedness to receive TA.

◼

TA providers agreed that TA engagements satisfied the original TA request. The TA had a positive impact on TA customers, particularly when a needs assessment was conducted first.

◼

Absent a rigorous evaluation, many TA providers were wary of overstating the long-term impact of TA or attributing too much of a TA customer’s success or progress to the TA they received.

QUANTIFYING IMPACTS: TROUBLED PHA TA

◼

The case study on troubled PHAs highlighted that the impacts of HUD TA are more easily quantified when metrics are defined at the outset of an engagement.

E X E C U T I V E S U MM A R Y

x i x

◼

Many PHAs demonstrated improvements in metrics during and after TA, from increased occupancy rates and financial security to more assisted households. Management processes and financial health were also reported as improving, including faster work order response times, exiting receivership, and accruing financial reserves.

Recommendations for Moving Forward

HUD staff, TA providers, and customers identified ways in which TA processes could be improved.

The following recommendations are based on their suggestions and the research team’s analysis of administrative and qualitative data. Recommendations concern improvements to communication and coordination practices, the development of efficient and flexible work plans, wider access to work products, improved data systems and data, and better tracking of outcomes associated with TA.

Because HUD is continually improving its TA processes, changes have been made to Community Compass since data were collected for this study. Some of the recommendations or underlying concerns may have been addressed.

Improve Communication and Coordination

HUD could improve communication and coordination for TA customers by communicating information on TA through existing channels used to distribute information to funding recipients on the programs they are administering, developing best practices for field office communications and guidance on TA that could be replicated across the country, and clarifying the TA application process, particularly for smaller or first-time customer organizations, while facilitating avenues for building relationships with HUD staff early in the process.

Improvements for TA providers could include clarifying processes to avoid limiting the range of organizations able to provide TA by using NOFA webinars to set expectations around the unpredictability of TA assignments and encourage providers to engage subcontractors earlier in the process, given the challenges of anticipating the expertise needed to address TA requests. HUD could expand regularly scheduled communications with TA providers through webinars or calls with HUD

headquarters staff to share expectations, changes in requirements, reporting practices, and other pertinent information.

Improvements for HUD staff could include standardizing communication and notifications between HUD headquarters and field staff at key steps in the TA process to support TA monitoring, x x

E X E C U T I V E S U MM A R Y

including communicating TA survey results to appropriate field staff and training them on data interpretation. HUD also could assess cross-program communications and explore options for improvement, from standardizing communications at specified points in the TA approval process to larger changes, such as embedding GTRs in all program offices or making the Technical Assistance Division an independent office outside of any program office.

Develop Efficient and Flexible Work Plans

HUD could allow providers greater flexibility in developing and amending work plans efficiently by expanding the use of needs assessments to ensure customers’ needs are understood before determining the TA type they need or developing a work plan and budget. HUD could establish and fund a more iterative process for work plan development to save time and resources and to allow quicker work plan approvals to speed up customer engagement.

Increase Access to Work Products

HUD could encourage the sharing of resources across providers and work plans while allowing for customization and could incentivize multiple uses of TA materials. Steps could include creating a repository of TA work products to make available to providers to tailor to their customers’ needs while recycling standardized information and best practices to extend the value of HUD TA investments.

HUD could encourage and incentivize providers’ use of open-source platforms for widely used products that allow for shared ownership of TA materials or allow materials to be retained and updated directly by HUD or with the support of another provider.

Improve Data Systems and Data

To improve the data available to HUD staff, TA providers, and researchers, HUD could enhance documentation of data and variable definitions and streamline duplicate data entry within the TA Portal and DRGR. Enabling data linking across systems through a customer identification code would enhance TA monitoring and support assessments of TA outcomes. Improving data consistency across TA types to the extent possible would improve uniformity and comparability, and expanding the range of date fields could help monitor the timeliness of approvals and performance periods. HUD could address functionality issues in HUD IT systems that affect data and information entry, such as system speed and save functions.

E X E C U T I V E S U MM A R Y

x x i

HUD could improve data quality by reducing text-based entries so data can be extracted and summarized easily, which would reduce data entry errors and increase the ability to match work plans across systems. Improving how HUD staff and providers enter and report on TA work plans in the TA Portal by including naming conventions and data entry standards can increase data usability. Key variables could be required so that important fields could not be skipped. Easing the data amendment process for providers could ensure that the most up-to-date information is available in work plans while minimizing duplicate entries and tracking updates. Finally, collecting information on subcontractors would help HUD know who provides the TA and how best to monitor TA provision.

Track Clear Outcomes

HUD could improve monitoring and support evaluations of HUD TA performance and outcomes by standardizing and expanding feedback mechanisms to gather consistent input from customers. HUD

collects some feedback from all TA customers but in an inconsistent manner. Providers also would benefit from receiving customer feedback, including results from customer surveys already administered. Expanding the tracking of TA outputs and outcomes, including establishing metrics for each work plan, assessing baseline values, and measuring progress at the end of a TA engagement, would lay the groundwork for future evaluations of TA program effectiveness.

x x i i

E X E C U T I V E S U MM A R Y

I. Introduction

The U.S. Department of Housing and Urban Development (HUD) provides technical assistance (TA) to recipients of an award from HUD to carry out an activity under a HUD program (HUD, 2020). TA is intended to provide “guidance which enables HUD’s customers to overcome a lack of specific skills or knowledge of the associated HUD programs and, by doing so, results in the successful performance and compliance of those programs” (HUD, 2014). TA is available to all of HUD’s diverse funding recipients, including public housing agencies (PHAs), owners and operators of multifamily housing properties, Continuums of Care (CoCs), state and local governments, Tribes and Tribally Designated Housing Entities, and others.

Historically, various program offices within HUD have provided TA independently of each other.

However, in fiscal year (FY) 2014, HUD rolled out Community Compass, making significant changes to its TA programs’ structure, delivery process, and data and reporting systems. Building upon changes that had started in FY 2010 under the OneCPD process, Community Compass encourages coordination across programs and TA delivery.

The Urban Institute was awarded a cooperative agreement by HUD to assess the effectiveness of the TA process and perceived satisfaction with the TA process and outcomes under the Community Compass structure to ensure that public dollars appropriated for TA are being spent to obtain the desired outcomes. HUD’s objectives for the assessment were to provide a detailed examination of the overall operations of TA programs, including data systems, and to provide case studies of select TA programs. This study addresses what TA is provided, how TA is provided, and perceived outcomes associated with receiving TA. It is based on analysis of HUD administrative data; interviews with HUD

staff, TA providers, and TA customers; and two case studies: (1) TA provided through the Office of Special Needs Assistance Programs (SNAPS) to HUD funding recipients addressing homelessness and (2) TA provided to troubled PHAs.

HUD Technical Assistance Objectives

Recipients of HUD funding are engaged in significant work at the national, state, regional, and local levels to address some of the most urgent housing needs in the United States, including reducing homelessness and improving housing affordability for low-income households. To help achieve the goals of the work and comply with federal regulations, HUD funding recipients may need TA to

improve their understanding of regulations and policies, internal operations, and housing program strategies.

The TA that HUD provides should help “customers navigate complex housing and community development chal enges by equipping them with the knowledge, skills, tools, capacity, and systems to implement HUD programs and policies successfully and sustainably and provide effective administrative and managerial oversight of HUD funding.”

-FY 2018/2019 Community Compass Notice of Funding Availability

To meet these needs, HUD provides a variety of TA opportunities for its funding recipients focused on improving capacity, management, and compliance. The first Community Compass Notice of Funding Availability (NOFA) released in FY 2014 outlined 13 stated objectives for HUD TA (HUD, 2014):

1. Facilitate local collaboration, strategic planning, and service coordination among HUD

customers and stakeholders, including jurisdictions, Continuums of Care, PHAs, and nonprofit organizations and consultation with Tribes.

2. Develop and implement strategies to repair, preserve, and recapitalize federally assisted rental housing, Tribal housing, public housing, permanent supportive housing, transitional housing, and other multifamily and single-family affordable housing that uses HUD grant funds.

3. Improve financial, management, physical, and governance deficiencies at public and tribal housing authorities and Tribally Designated Housing Entities, state and local units of government, and nonprofit customers and build capacity to sustain these improvements in performance over the long term.

4. Provide housing and supportive services for special needs populations.

5. Facilitate better delivery of housing counseling services through networks of housing counseling agencies, improve the accessibility of services to city and county residents, and increase awareness among nonprofits, city agencies, and the public that housing counseling services are available.

2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6. Increase homebuyer knowledge by providing homebuyer education, individualized counseling, and access to information and resources.

7. Increase understanding of strategies to reduce energy consumption, onsite renewable energy deployment, and related financing to accelerate clean energy solutions.

8. Improve management of HUD funding.

9. Develop and implement policies and procedures that support affirmatively furthering fair housing, per applicable federal regulations, and employment, training, and contracting opportunities to low- and very low-income persons and the businesses that employ them.

10. Ensure that Indian Housing Block Grant funds are used effectively, and ensure effective delivery of housing and Native American Housing Assistance and Self-Determination Act eligible services. Assist with Indian Housing Block Grant formula administration.

11. Develop and implement strategies for improved data collection, validity, analysis, and use by customers for reporting, planning, decision making, performance measuring, and outcomes tracking.

12. Provide effective management, use, and reporting of Homeless Management Information System data.

13. Use technology and the Internet for effective knowledge management to improve access to information and resources, build skills, reduce errors, enhance data collection, and streamline processing requests for assistance via websites, mobile applications, virtual help desks, remote and virtual learning modalities, and automated tools and calculators.

HUD Technical Assistance Types

Through its TA programs, HUD provides TA to recipients of HUD funds through a variety of mechanisms to help achieve TA objectives. To meet these objectives, multiple offices within HUD

administer TA programs that are delivered by providers who are awarded cooperative agreements by HUD to provide TA based on their expertise and experience.

In general, TA is often delivered through one of two approaches—universal or targeted TA.

Universal TA is broadly applicable to all TA customers needing similar information, such as those operating within the same funding program. It includes self-paced, web-based courses, training, videos, and interactive tools that can be accessed by a TA customer independently, usually self-A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3

directed with no direct contact with a TA provider.2 For this approach, HUD requests TA providers to develop and deliver several types of TA, including appropriate tools, products, and training, and to assist with an online help desk.

In contrast, targeted TA—also called tailored TA or intensive TA—is more customized to the needs of a specific customer (Baumgartner, Cohen, and Meckstroth, 2018; Richards et al., 2017).3 Targeted TA can supplement or provide TA that cannot be met through universal TA resources. It involves direct contact between a TA provider and a customer—either in person or remote—to determine needs and develop a work plan to address them. Within HUD, the targeted approach includes types of TA such as on-call TA, direct TA, and needs assessments. Exhibit 1 presents various HUD TA types, as defined by HUD, according to whether they fall within a universal or targeted approach to meeting customer needs.

2 “Technical Assistance Centers and Demonstration Projects,” U.S. Department of Education, accessed November 6, 2020.

3 “Technical Assistance Centers and Demonstration Projects,” U.S. Department of Education, accessed November 6, 2020.

4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 1: HUD TECHNICAL ASSISTANCE TYPES AND DEFINITIONS BY UNIVERSAL AND TARGETED

APPROACHES

TA Approach

TA Type

Description

Universal TA

Tools and products

Materials developed to assist HUD customers with understanding

complex program statutes and regulations and federal requirements.

These products may take the form of web-based courses, trainings,

videos, and interactive tools.

Help desk support

Services such as the Ask-A-Question form submitted online through

HUD Exchange and currently available for many Community Compass

programs and data systems.

Self-directed and

TA intended to increase capacity and close competency gaps among

group learning

customers by equipping the customers with the necessary program

knowledge and skill sets needed to administer and manage HUD

programs.

Targeted TA

On-call

Remote TA engagements generally limited to less than 16 hours of

TA. They can serve the same purposes as direct TA.

Direct technical

TA providers are tasked to work on site and/or remotely with

assistance and

customers to improve PHA operations, help implement new programs,

capacity building

train staff, and provide other technical assistance activities.

Engagements are long term and often comprehensive.

Needs assessments

A process that helps to determine the nature and scope of TA and

capacity building needed. Assessment requests can be generated by

recipients of HUD program funding, the field office, and/or the TA

provider.

Data reporting,

TA that includes analyzing data to better understand the impact of

analysis, and

program, policy, and reporting changes on recipients of HUD funding;

management

trends in performance; technical assistance needs; and assisting

recipients with using and reporting data.

Knowledge

TA that includes the development, operation, maintenance, and/or

management

hosting of websites.

TA = technical assistance.

Note: NAHASDA Allocation Formula Administration and Negotiated Rulemaking Support is listed as a distinct eligible activity within the TA Notices of Funding Availability; however, the TA activities described fall into the listed TA types under targeted TA, so this is not broken out as a separate type here.

Sources: HUD (2014, 2016, 2018)

TA can be delivered via multiple modes regardless of the TA approach (universal or targeted) or TA type (tools and products, needs assessments, etc.). The selected mode can vary based on the intended audience (individual versus group), platform (onsite versus virtual), level of participant engagement (active or passive), purpose, and who delivers learning (peers or consultants) (Barbee, DeSantis, and Richards, 2017; Baumgartner, Cohen, and Meckstroth, 2018; Crosby and Bryson, 2010; Hunter et al., 2009; Norton et al., 2016; Stevenson et al., 2002; West at al., 2012).

Community Compass Overview

HUD began to shift from a siloed, program-specific approach to TA provision to a more coordinated and collaborative process across programs beginning with the FY 2010 Technical Assistance and A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5

Capacity Building NOFA issued by the Office of Community Development and Planning (CPD). This effort launched the OneCPD Integrated Practitioner Assistance System (OneCPD) as part of the Transformation Initiative, which pooled resources across HUD programs to invest in strengthening TA programs, among other agency improvements.4,5 Changes continued with the integration of the Office of Public and Indian Housing TA programs in FY 2013 (OneCPD+) and the addition of two more TA funding streams in FY 2014 to form Community Compass.

Beginning in FY 2014, Community Compass consolidated TA funding for several HUD programs across five HUD offices: CPD, Public and Indian Housing (PIH), Office of Housing, Office of Economic Development, and Fair Housing and Equal Opportunity (FHEO). Funding for the TA comes from five appropriation accounts: (1) departmental funding through the Transformation Initiative, which predated Community Compass but began the process by coordinating across CPD, PIH, and FHEO

offices; (2) McKinney-Vento Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) TA; (3) McKinney-Vento National Data Analysis Project; (4) Public Housing Receivership and Recovery; and (5) Native American Housing Assistance and Self-Determination Act (NAHASDA) TA. A list of the HUD programs covered by Community Compass is included in appendix A, linked to the office and office division that runs them and the appropriation account that funds the TA related to each program.

Since FY 2014, several major changes have taken place in the TA process. The most substantial changes include the following:

◼

Unification of the NOFA process for awarding cooperative agreements to TA providers for all programs included in Community Compass,

◼

Continued transition from program-specific TA to cross-program TA,

◼

Data systems and processes to improve TA management, and

◼

Streamlined online assistance and resource pages available to TA customers through the HUD

Exchange website.

4 “About OneCPD,” Housing Assistance Council. https://ruralhome.org/about-onecpd/.

5 “U.S. Department of Housing and Urban Development Fiscal Year 2012 Program and Budget Initiatives,” HUD,

https://www.hud.gov/sites/documents/PDR.PDF.

6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Changes in TA structure and process have occurred, along with changes in HUD TA appropriations. Between FY 2013 and FY 2019, the amount of funds appropriated by Congress to HUD to fund TA increased from approximately $35.3 million to $36.8 million but peaked in FY 2016

at $60.6 million (exhibit 2).

EXHIBIT 2: HUD TECHNICAL ASSISTANCE APPROPRIATIONS, FY 2013–FY 2019

$70

$60

s)nolli $50

s (mi

$40

ion

riatp $30

rop

$20

D ApUH $10

$0

2013

2014

2015

2016

2017

2018

2019

Fiscal Year

Notes: All years shown are fiscal appropriations years. Because 2013 appropriations preceded Community Compass, the amounts totaled cover technical assistance (TA) funded through OneCPD+, the immediate precursor to Community Compass, plus the Office of Native American Programs and McKinney-Vento Homeless Management Information Systems (HMISs). HUD

released 2018 and 2019 funding in a combined Notice of Funding Availability that allocated a total of $73.5 million for both years, but they have been divided equally in the graph. Dollar amounts have been inflation adjusted to 2018 dollars.

Sources: HUD, 2014, 2015, 2016, 2017, 2018; Sullivan, 2013; White, 2013

Existing Research and Study Contribution

There has been limited research to date on TA provision and effectiveness (Soler, Cocozza, and Henry, 2013; Tyson and McNeil, 2009). Studies focusing broadly on TA and TA provision have found that flexibility and adaptability are critical for meeting customers’ needs. This adaptability often takes the form of tailoring existing TA products or responding directly to address specific challenges a customer faces rather than providing a universal training or tool (Barbee, DeSantis, and Richards, 2017; Baumgartner, Cohen, and Meckstroth, 2018; Spadaro et al., 2011). Direct one-on-one interaction and peer-support networks have been found effective for delivering TA (Barbee, DeSantis, and Richards, 2017), whereas self-directed, virtual modes of delivering TA are not always taken up by those most in A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7

need (Baumgartner, Cohen, and Meckstroth, 2018). These studies focusing on effective practices have not examined the breadth of TA delivery approaches or types of TA. There also has been limited study on processes for providing TA because much of the existing literature evaluates individual TA programs (National Resource Center, 2010). Researchers and practitioners have emphasized that TA programs should take a collaborative approach (Litwin et al., 1994; National Resource Center, 2010) and, to maximize effectiveness, should develop both individual and organizational capacity (Blase, 2009; Foster-Fishman et al., 2001). In general, TA delivery is strengthened when recipients are integrated into the process of diagnosing the problem and TA types and delivery are tailored to the local context (Baumgartner, Cohen, and Meckstroth, 2018; Theodos et al., 2018).

Research on TA to date has focused on best practices for providing assistance but has not examined in detail the types of TA provided, how it is provided, or its perceived effectiveness. This study, the first assessment of HUD TA under the Community Compass structure, contributes to an understanding of TA by providing detailed descriptions of the processes for providing TA, insights into customers’ reception of TA, and the interactions among HUD, providers, and customers. Findings are intended to inform HUD’s ongoing TA delivery through Community Compass by highlighting what is working well—from the perspective of staff, TA providers, and TA customers—and suggesting areas for improvement in the TA process, including soliciting TA providers, developing universal TA products, requesting targeted TA, delivering and completing TA, and monitoring and tracking TA-related outcomes and impacts through data collection efforts. Because HUD is continually improving its TA processes, changes have been made to Community Compass since data were collected for this study.

Some of the recommendations or underlying concerns may have been addressed already.

Organization of the Report

This report is organized into nine additional concise chapters describing the research, answering study questions, and providing recommendations on how HUD TA under the Community Compass structure could be improved in the future. A series of appendixes provide supplemental information on the implementation of the study and study findings.

The remaining chapters cover the following topics:

◼

Chapter II outlines the study’s research questions, design, and implementation.

◼

Chapter III describes TA customers and providers.

8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

Chapter IV provides a step-by-step overview of the TA delivery process under Community Compass from the NOFA through the delivery and completion of TA.

◼

Chapter V discusses how TA timelines and costs have changed since Community Compass.

◼

Chapter VI explores changes to TA collaborations, communications, and coordination across HUD headquarters and field offices and with providers since the inception of Community Compass.

◼

Chapter VII reports on monitoring and evaluation activities across TA programs.

◼

Chapter VIII details the perceived impacts of TA from the perspective of TA customers and TA providers.

◼

Chapter IX presents recommendations for improving HUD TA processes, data, and outcomes through Community Compass.

◼

Chapter X summarizes findings and considers implications stemming from this assessment for the field of TA broadly.

Definitions of Key Terms

Key terms used throughout the report are defined below.

◼

Continuum of Care (CoC): Local coordinating body responsible for the design, implementation, and oversight of a comprehensive strategic plan to end homelessness.

◼

Disaster Recovery Grant Reporting (DRGR) System: HUD’s reporting system used by TA providers to invoice and document funded activities and by HUD staff to monitor and review the activities.

◼

Government Technical Monitor (GTM): HUD TA cooperative agreement management position located in program offices and whose duties include technical (programmatic) advice, monitoring, and assistance throughout the TA process.

◼

Government Technical Representative (GTR): HUD TA cooperative agreement manager centralized in HUD’s Office of Community Planning and Development (CPD) whose responsibilities consist of management and fiscal oversight of the award (e.g., invoicing and work plan approval).

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9

◼

HUD programs: Funding provided by HUD to public and nonprofit agencies for specified purposes to serve vulnerable people and communities in support of HUD’s mission—for example, the Community Development Block Grant program.

◼

Funding recipient: A non-federal entity receiving an award directly from HUD to carry out an activity under a HUD program, including public housing agencies, owners and operators of multifamily housing properties, Continuums of Care, state and local governments, Tribes and Tribally Designated Housing Entities, and others.

◼

Program Office Technical Assistance Coordinator: HUD staff member whose work involves coordinating TA activities specific to the HUD program office through subject matter experts and task support and determining hours and degree of effort needed for a work plan.

◼

Public Housing Agency (PHA): Any state, county, municipality, or other government entity or public body or an agency or instrumentality of these entities that is authorized to engage or assist in the development or operation of low-income housing under the U.S. Housing Act of 1937.

◼

SNAPS TA: TA provided for recipients of HUD funding through the Office of Special Needs Assistance Programs (SNAPS) to address the housing needs of special needs populations, including individuals and families experiencing homelessness.

◼

TA customers: HUD funding recipients that receive TA to help them use HUD program resources efficiently and effectively.

◼

Technical Assistance Division (TAD): Division within HUD’s Office of Community Planning and Development that provides operational and administrative support for TA across program offices at HUD.

◼

TA engagement: The development and delivery of TA by a TA provider.

◼

TA Portal: HUD reporting system developed to track TA requests and work plans and assist with TA management.

◼

TA programs: TA administered by a HUD office and/or office division for one or more of its programs (see appendix A)—for example, SNAPS TA (defined above).

◼

TA providers: Organizations and businesses that receive a cooperative agreement through HUD’s competitive NOFA process to provide one or more types of TA under one or more HUD program offices.

1 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

TA request: A request for HUD TA, which can either be submitted through the TA Portal by any HUD Exchange user (which includes the general public), submitted directly to a field office or HUD headquarters, or assigned to a HUD funding recipient.

◼

TA type: Defined by HUD (see exhibit 1 above), the specific types of TA allowed to be delivered to TA customers, including universal approaches—tools and products, help desk support, and self-directed and group learning—and targeted approaches—on-call; direct TA and capacity building; needs assessments; data reporting, analysis, and management; and knowledge management.

◼

TA work plan: A detailed scope of work submitted to HUD by a TA provider assigned to provide TA to a customer after HUD approves a request.

◼

Targeted TA: TA customized to the needs of a specific customer. Examples include on-call TA, direct TA, and needs assessments.

◼

Troubled PHA TA: TA provided to address the challenges faced by PHAs designated as troubled based on poor performance of their public housing or Housing Choice Voucher programs as determined by HUD’s assessment systems.

◼

Universal TA: TA that is broadly applicable and easily accessible to a large number of TA customers. Examples include self-paced, web-based courses, training, videos, and interactive tools, as well as an online help desk.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 1

II. Study Design

This descriptive study of HUD TA—beginning with the shift to the Community Compass structure—is based on analysis of administrative data from FY 2013 through FY 2017 and qualitative interview data collected from HUD headquarters and field office staff, TA customers, and TA providers. Overall findings on the Community Compass structure, TA provision, and perceived impact are also informed by case studies of two TA programs: SNAPS TA, which is provided to organizations and agencies working to reduce homelessness, and TA provided to low-performing PHAs to improve management deficiencies. This chapter introduces the research questions and data used to address them. It discusses sampling for administrative data and interview respondents and case study selection and data analysis before concluding with a discussion of study limitations.

Research Questions

This study addresses three overarching questions about the TA provided under Community Compass: (1) What TA does HUD provide, (2) How does HUD provide TA, and (3) How effective is HUD TA?

Each of these questions consists of multiple sub-questions, as shown in exhibit 3. The exhibit also shows sources for data analyzed to address the questions.

1 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 3: RESEARCH QUESTIONS BY CATEGORY AND ASSOCIATED DATA SOURCES

Category

Research Questions

Admin Interviews

Case

Data

Studies

1. What

What TA types does HUD fund, and how much do they cost?

X

X

TA does

Who are the TA providers and TA customers by TA type?

X

X

HUD

What types of TA are provided, and in what quantity?

X

X

provide?

What types of TA are requested each year through the TA Portal,

X

X

X

and who enters these requests?

How many requests are rejected each year and why, and to what

X

extent do rejected requests appear to represent valid unmet

needs?

Which types of TA can be required by HUD? How does that affect

X

X

the delivery of TA?

2. How

How does the TA process work from the perspective of HUD

X

X

X

does HUD

Community Compass program staff? Of TA providers? Of TA

provide

customers? How does it compare with how HUD provided TA in

TA?

the past?

How do processes compare among TA programs? Across different

X

X

TA types?

Has the Community Compass process led to the use of

X

X

standardized TA materials and processes across TA providers? If

not, why not? What can TA providers or HUD do to reduce

duplication of effort or lack of standardization?

Has Community Compass facilitated better coordination,

X

X

collaboration, and communication across HUD offices and

programs and TA providers?

What is the average time between steps for TA provision, from TA

X

X

X

request to the completion of the work plan?

How much contact do TA providers have with customers? What

X

X

are the types and nature of those contacts?

How quickly does HUD respond to TA requests?

X

X

X

3. How

Do customers think the TA they received met their expectations

X

X

effective is and helped them further their goals and objectives? Did the TA HUD TA

help them improve their programs? If so, in what ways?

perceived

How effective are HUD data and reporting systems for TA

X

X

to be?

providers, TA customers, and HUD staff in facilitating the TA

process?

How could HUD obtain regular feedback from its providers and

X

X

customers about the delivery and impact of HUD TA? How does

HUD evaluate the impact of TA on customers’ financial,

managerial, physical property, and governance practices?

TA = technical assistance.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 3

Four research questions proposed in the original research design for this study were not addressed because of data limitations (discussed further below) and the case studies selected; therefore, these questions are not included in exhibit 3:

◼

For each TA program, what are the goals, process of awarding cooperative agreements and assigning the TA, TA topics covered, and delivery methods used over time?

◼

How frequently does an individual customer receive TA through any Community Compass program?

◼

What percentage of TA funds are spent on administration and coordination?

◼

Does HUD TA facilitate better program coordination and collaboration by TA customers and within the jurisdictions they serve? If so, to what extent? If not, why?

Data

Administrative Data and Variables

Administrative data for the study came from HUD’s TA Portal and Disaster Recovery Grant Reporting (DRGR) System datasets. HUD uses the TA Portal to receive and input TA requests, review requests, assign work to providers, and approve and monitor TA work plans. Providers use the system to submit and revise work plans and to report on progress. DRGR is used for financial management and oversight of TA, including submitting and receiving HUD approvals on proposed costs, submitting invoices and progress reports, and authorizing payment for completed work. The TA Portal focuses on the process of the TA engagement, from request or initiation to closeout.

The research team planned to analyze data from the TA Portal and DRGR datasets, assuming the data could be merged. The TA Portal was chosen as the best data source for this study because it includes variables needed for analysis that are not available in DRGR, such as funding source, detailed TA type, topic, provider name, and customer name. The work plan ID variable, meant to be the linking variable across datasets, is entered inconsistently in the TA Portal and DRGR, making it difficult to match work plans across systems. After using advanced merging techniques, the team merged only 65

percent of the work plans.

The team also merged data from HUD’s Public Housing Assessment System (PHAS) and Section Eight Management Assistance Program (SEMAP) for the case study on troubled PHAs with TA Portal 1 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

data. Key variables from PHAS and SEMAP included PHA codes, used to identify customers, troubled status, and years troubled. See appendix B for details on the process of merging PHAS and SEMAP

score data with the TA Portal data.

The team compiled data on TA requests, work plans, and workflow from the TA Portal from FY

2012 to FY 2017. The research team initially completed an analysis of TA Portal data for the entire 6-year period. This analysis showed fewer observations for FY 2012 relative to subsequent years, indicating low use and reporting in the TA Portal. An examination of the data suggested that changes between FY 2012 and FY 2013 likely were attributable to an increase in TA Portal use rather than changes in TA provision. Where possible, analyses have been updated across all TA programs to include data from FY 2013 to FY 2017, inclusively; the FY 2012 data are excluded from charts and analyses of changes over time. The case studies were completed by the time the decision was made to remove FY 2012 data. Some case study analyses still include data from FY 2012.

Analytical categories of interest and associated variables are shown in exhibit 4. Data type—

whether a variable is in the form of open text, an item selected from dropdown options, or numeric—is identified in the Data Format column after each variable. The variables below were the original variables provided in the TA Portal.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 5

EXHIBIT 4: VARIABLES BY ANALYTIC CATEGORY DRAWN FROM TA PORTAL

Category

Variables

Data Format

TA requests

Request status

Dropdown text (approved, assigned, closed, denied)

Request status dates

Numeric

Request topics

Open text

Reason not approved

Open text

TA customers Name

Open text

State

Dropdown text

HUD region

Dropdown text

Field office

Dropdown text

Work plans

Work plan ID

Open text

Funding source

Dropdown text (Community Compass) (15 sources),

OneCPD (3 sources), CDBG TA, CHDO TA, HMIS

TA, HOME TA, Hawaii, McKinney-Vento TA,

NAHASDA TA, NSP TA, NSP3 TA)

TA provider

Open text

TA type

Dropdown text (direct TA, needs assessment, on-call

TA, product development, training delivery)*

HUD program office

Open text

Scope

Open text

Approval date

Numeric

Start date

Numeric

End date

Numeric

Point of contact

Open text

TA topic

Open text

Estimated budget

Numeric

TA = technical assistance.

*TA type in the TA Portal also includes regional outreach as a category. Because this category is not defined in the HUD NOFA and represents less than 0.5 percent of all TA work plans and budgets during the study period, we have omitted this category from the analysis.

These variables were critical to the analysis of TA providers, TA customers, TA types and costs, and the general steps and timelines for the TA process. The team analyzed the frequency, either through the number of work plans or the estimated budget of each TA type (e.g., direct TA, needs assessment, on-call TA, tools and products, self-directed and group learning).6 Case studies used 6 This report and its exhibits use the terms from HUD NOFA definitions rather than the TA Portal terms when referencing budget and work plan amounts by TA type from the TA Portal. The report uses tools and products to refer to anything listed as product development and self-directed and group learning to refer to training delivery because the work plan activities detailed in the TA Portal categories most closely reflected those HUD definitions.

1 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

program office (i.e., SNAPs) or customer (e.g., SNAPS customers or troubled PHAs) variables to identify SNAPS and troubled PHA TA. The funding source variable provided information on which specific Community Compass categories (e.g., Departmental TA, McKinney-Vento TA, PHA Receivership) or non-Community Compass sources (e.g., OneCPD) funded TA to illustrate the distribution of TA funds over a diverse customer base. Date variables provided a starting point for understanding the sequencing of steps in TA delivery. The team identified which types of TA originated with a request and which did not. The provider variable was used to illustrate the diversity and budget distribution of TA providers. The provider was also used—along with the point of contact, region, field office, and customer—to sample interviewees. The SNAPS case study used the topic variable to assess SNAPS TA goals.

All reported dollar amounts reflect inflation-adjusted values to 2018 dollars. For work plans that spanned more than 1 year, the estimated budget was spread equally over all years of the work plans’

performance period. For the case study on troubled PHAs, work plan counts and estimated budget amounts also were weighted equally across all PHA customers who received TA through that work plan. See appendix B for additional information on case study data and analysis.

Many variables in the TA Portal are open text entries, which, given the number and inconsistency of entries, presented challenges for summarizing the data. Open text entries limited the research team’s ability to describe who received TA, where, and for what purpose. The team implemented a targeted text search of the open text variables (most notably “scope” and “customer”) to better understand their content and determine if the data were usable for the study. For example, the team visualized the text of denied requests using a word cloud to better understand the reasons for TA request denials.

The TA Portal contained several variables that might have been useful for analysis but for their limitations. The customer type variable (e.g., PHA, CoC Grantee, Homeless Management Information System [HMIS]) and PHA status variables (e.g., PHA risk designation) could have enabled precise identification of customers for the SNAPS and troubled PHA case studies, but they were not complete enough to use, nor were they descriptive enough to differentiate among types of grantees (for instance, the most common entry was “Grantee,” which could have applied to PHAs, CoCs, or other groups). The team identified customers using other variables in the TA Portal (e.g., program office, funding source), text searches of work plans from the TA Portal, and HUD PHAS and SEMAP scores.

HUD region, field office, and state variables were used for interview sampling, but analysis of work plan amounts, budgets, or other variables by geography proved misleading because regions vary in size and have unique housing needs and histories. Variables related to whether a recipient had received TA A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 7

previously also were incomplete, so the team relied on customer and work plan ID information to approximate TA repetition.

The TA Portal contains various task-level variables that provide details about the specific components of the TA delivery, including dates, budgeting, and staffing. The open-endedness, overall number, and granularity of these variables made it difficult to use them. In addition, tasks were created inconsistently across TA work plans; some work plans had one or two tasks, whereas others had up to 72 tasks. Some plans included a distinct TA administration task, and others spread administration costs across all tasks. Also, the tasks and their associated budgets and timelines were entered before TA implementation began and were not consistently updated, which, according to HUD staff, meant that many of the tasks did not reflect how the work was carried out. After consultation with HUD, the team opted to use the overall work plan dates because HUD staff suggested these dates were more likely to be updated than those associated with specific tasks.

Interview Data

Data from in-depth and case study interviews with HUD staff, TA providers, and TA customers include the following: details on the steps of the TA process, including TA requests, assignments, and delivery; information on how the process, communication, coordination, and collaboration have changed since Community Compass; perceptions of the value and impact of the TA; and suggestions for improving TA implementation. Interviews from the troubled PHA TA case study also collected clarifying details on the process for targeting TA to troubled PHAs and providers’ perceptions of the types of TA best suited to address troubled PHAs’ needs. Interviews for the SNAPS TA case study provided information on the program office’s implementation of several unique processes, including regional teams, initiative-based TA, on-call TA, and the evolution of the SNAPS TA structure. Interviews with field office staff provided perspectives on the impact of Community Compass on their role in the SNAPS

TA process and their ability to connect jurisdictions with HUD resources. See the separate technical appendix for discussion guides.

Document Data

The research team obtained documentation related to the TA administration strategy from HUD and TA providers for the SNAPS and the troubled PHAs case studies. For the SNAPS case study, the team reviewed TA administration protocols and an internal memo on proposed TA goals for CoCs that SNAPS headquarters staff provided. Troubled PHA case study documents included samples of 1 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

improvement plans, troubled designation notices sent from HUD’s Real Estate Assessment Center office to PHAs, Take Action Letters that HUD field offices sent to PHAs after agencies appeared on the quarterly Operational Troubled List, recovery agreements, and action plans. The documents provided valuable contextual information to supplement interviews, including TA content and administration, HUD’s improvement strategy for troubled PHAs, and details about the TA administered.

Sampling and Case Study Selection

Technical Assistance Programs and Interview Respondents

The research team conducted 44 group and individual interviews with a total of 65 respondents. In-depth interviews were conducted with HUD staff from headquarters and field offices, TA providers, and TA customers (case study interviews are discussed below). In aggregate, the interviews provided data on every type of TA and most of the program offices under Community Compass. HUD

headquarters staff, field staff, and TA providers were selected purposively, whereas customers were selected through a hybrid stratified-random and purposive sampling process using limited customer data. Appendix B provides a summary of respondent characteristics by TA type and HUD program office.

HUD headquarters staff were selected for programmatic diversity. The team interviewed 14 staff members at headquarters from the CPD, PIH, FHEO, Housing, and Office of Policy Development and Research (PD&R) and spoke with several of them multiple times; no staff from the Office of Economic Development were interviewed. HUD field staff were selected to cover the range of TA types. The team conducted seven interviews with 17 staff from eight field offices that oversee numerous TA programs and engagements that fall under CPD and PIH, including staff within the Office of Native American Programs (ONAP).

To select TA providers, the research team extracted data from the TA Portal from FY 2012 to FY

2017 on TA engagements before and after Community Compass funding. Data included providers’

total budget under Community Compass, the type of TA administered, the TA topic or the office under which TA was administered, locations served and whether providers served rural customers, and the years providers received funding from HUD. The team prioritized diversity across those factors, conducting interviews with 12 providers that included 21 staff.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 9

To select TA customers, the research team extracted TA Portal data from FY 2012 to FY 2017.

The data extract included the type of TA that customers received, the program office that provided the TA, the period of performance for TA engagement, and contact information for the customer or the customer’s HUD contact. The team drew a stratified random sample of customers who had received targeted TA (i.e., direct TA, needs assessment, or on-call TA) rather than both targeted and universal TA because of the limited information about universal TA customers available from the TA Portal and concern for universal TA customers’ recall ability. Once customer data were sorted by TA type and program office, the team selected customers by most recent TA receipt as of FY 2017 to increase the likelihood of recall. To compensate for the imbalance of targeted and universal TA customers, interviews included questions on customers’ experiences with universal TA. The team spoke with 13 staff during interviews with 11 customer entities, of which 4 were nonprofit organizations, 4 were local governments, 2 were PHAs, and 1 was a Continuum of Care.

Case Studies

After considering several potential priority TA areas that HUD staff identified as important for learning about Community Compass, the research team proceeded with two case studies: SNAPS TA and troubled PHA TA. The team selected SNAPS because it administers a significant portion of HUD TA—

more than 25 percent of all work plans and more than 20 percent of the entire HUD TA budget—and its unique approach to offering TA, including the on-call TA model. The case study covered how one office (i.e., SNAPS) administers their TA program to funding recipients and organizations that work on ending homelessness, assessing SNAPS’ process for offering TA, the TA program structure, and TA assignment and administration. See exhibits 5 and 6 for overviews of the SNAPS program and troubled PHAs.

The research team chose troubled PHAs because administrative data indicate PHAs are among the most common types of HUD TA customers. The case study explores how TA has been tailored to meet the needs of troubled PHAs, highlights perceptions of TA providers and customers, and reports on the perceived impact of TA on troubled PHAs from the interviewees’ perspective. It does not focus on the subset of PHAs in receivership, although they were included in the troubled PHA TA customer population analysis.

To identify troubled PHAs for interview selection, the team examined PHAS and SEMAP scores—

ratings of the public housing and Housing Choice Voucher (HCV) program performance, respectively—

from 2011 to 2018. PHAS scores track the performance of PHAs’ public housing programs in four key 2 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

areas: the physical condition of public housing units (PASS sub-score), the management capacity of the PHA (MASS sub-score), the financial condition of each property (FASS sub-score), and capital fund obligation (CFP sub-score).7 SEMAP scores assess PHA performance in 14 areas, including administrative processes for waitlists, housing quality inspections, payment standards, recertification, and rent reasonableness.8 PHAs can be exempt from both scoring systems if they participate in the Rental Assistance or the Moving to Work Demonstrations or if they are Tribally Designated Housing Entities (TDHEs). Scoring data were used to categorize PHAs as troubled (scoring below 60 out of 100), exempt from reporting, or non-troubled and to identify the number of years troubled. Using scoring and TA Portal data, the research team selected PHA interviewees to reflect diversity in geography, PHA size (number of units), years troubled based on their PHAS or SEMAP score, and the number of TA work plans received. See appendix B for more details on identifying PHA customers for this case study.

Additional interviews were conducted for the case studies. For the study of SNAPS TA, the research team conducted 16 interviews: 3 staff from the SNAPS program office in headquarters, 6

field staff that administer SNAPS TA, 3 TA providers, and 4 customers. To select providers, the team prioritized diversity in location service and the number of years providers received funding from HUD

among those that worked with the SNAPS office. Customers included a statewide CoC (i.e., a Balance of State), a CoC from a smaller metropolitan area with relatively low population density, and a CoC

from a larger, urban metropolitan area. For the troubled PHAs case study, the research team conducted 12 interviews: 3 staff from the PIH office in headquarters, 5 providers associated with PHA TA, and 4 customers. Customers included PHA staff and board members, local- and state-level government departments, housing-related nonprofits, and non-housing-related nonprofits. The analysis also drew on in-depth interviews with HUD field office staff in which PIH TA processes were discussed.

7 For more information, see https://www.hudexchange.info/onecpd/assets/File/PHA-Lead-the-Way-

Understanding-PHAS.pdf. Scores are reported annually for all PHAs except small PHAs (agencies with fewer than 250 units). For those agencies, high performers are assessed every 3 years, standard and substandard performers are assessed every other year, and troubled and capital fund-troubled PHAs are assessed every year.

8 For more information, see https://www.hudexchange.info/resources/documents/PHA-Lead-the-Way-

Understanding-SEMAP.pdf. All agencies may not report or be required to report data for all of the indicators due to differences in jurisdictions’ and PHAs’ voucher programs, which results in differences in the raw SEMAP scores.

To control for this variation, the research team conducted analysis using the overal SEMAP designation (i.e., high, standard, troubled), worked with HUD to identify four individual indicators that are consistent across agencies, and gained an understanding of how HUD assigns scores to agencies that submit partial data.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

2 1

EXHIBIT 5: SNAPS OFFICE AND PROGRAMS

The SNAPS office resides within the CPD. A SNAPS headquarters staff member stated the program’s mission as ending homelessness and ensuring compliance with regulations. The SNAPS office offers several grants—most notably, Emergency Solutions Grants and the Continuum of Care Program—to help nonprofit providers, community-based organizations, and local and state governments work to end homelessness.

The following key programs and data systems are administered by the SNAPS office.

◼ The Continuum of Care Program (CoC Program) supports planning bodies that coordinate housing and services funding for families and individuals experiencing homelessness and was created through the Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act, which amended and reauthorized the McKinney-Vento Homeless Assistance (McKinney-Vento) Act in 2009. Through HEARTH, the CoC Program consolidated three federal homelessness HUD programs: the Shelter Plus Care program, the Supportive Housing program, and the Single Room Occupancy (SRO) program.

HEARTH also facilitated creating a strategic plan to end homelessness nationally, first in 2010 with the release of “Opening Doors: Federal Strategic Plan to Prevent and End Homelessness” and in the 2015

update to that document.

◼ Emergency Solutions Grants Program (ESGs) assists people experiencing homelessness in emergency or transitional shelters in obtaining permanent housing and typically are awarded to cities, counties, territories, and states.

◼ Homeless Management Information System (HMIS) is a local information technology system used by each CoC to collect client-level data for individuals and families experiencing or at risk of experiencing homelessness.

◼ Rapid Re-Housing connects families and individuals experiencing homelessness to permanent housing, recognizing that financial or other shocks lead to homelessness; most homeless families or individuals do not substantively differ from housing-secure families or individuals; homelessness adversely affects individuals and families, and those adverse impacts intensify with time; and resources to end homelessness should be used as efficiently as possible. Core components of rapid re-housing programs include locating available housing, rental and move-in assistance, and case management.

Sources: “What is a Continuum of Care?, ” National Alliance to End Homelessness, January 14, 2010;

“Homeless Emergency Assistance and Rapid Transition to Housing Act,” HUD Exchange; “Emergency

Solutions Grants Program, ” HUD Exchange; “Homeless Management Information System, ” HUD Exchange;

“Rapid Re-Housing,” HUD Exchange 2 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 6: THE OFFICE OF PUBLIC AND INDIAN HOUSING AND TROUBLED PHA TA The PIH office targets TA to PHA staff to improve their management and financial practices. PHAs manage several programs that provide affordable housing or rental assistance to eligible low-income renter households. The two largest programs on which PHAs are rated for their performance, affecting their designation as troubled PHAs, are public housing and the Housing Choice Voucher programs.

◼ Public housing is the oldest federal housing assistance program. Properties are funded directly by HUD through capital and operating funds but are owned and operated by local PHAs. More than 3,000 local PHAs manage more than 1 million public housing units that cap rent at 30 percent of an eligible household’s income to keep units affordable. Recent programmatic innovations have explored allowing flexible use of federal funds for public housing through the Moving to Work Demonstration program and the recapitalization and preservation of the physical buildings via the Rental Assistance Demonstration program to convert units to other funding streams (Scally et al., 2018). PHAS scores rate PHAs on their management of public housing.

◼ The Housing Choice Voucher program is the largest federal housing assistance program, helping more than 2.3 million households afford rent in the private market. This program is funded by HUD and administered primarily by local PHAs. Income-eligible households receive a voucher that helps pay rent for a private-sector unit of their choice that passes a housing quality inspection (Scally et al., 2018). SEMAP scores rate PHAs on their management of the HCV program.

Sources: Scally, Corianne Payton, Samantha Batko, Susan J. Popkin, and Nicole DuBois. 2018. The Case for More, Not Less:

Shortfalls in Federal Housing Assistance and Gaps in Evidence for Proposed Policy Changes. Washington, DC: Urban Institute Analysis

The team analyzed administrative data from the TA Portal from FY 2012 to FY 2017 to observe how TA changed after the advent of Community Compass in FY 2014. The team prepared descriptive statistics on the frequency and total dollar amount of HUD TA over time by type, topic, and funding source to better understand the TA HUD provides. The team then summarized who provides HUD TA, including a number of work plans and total budgets by year. The team also summarized the budget and work plan shares by providers, customers, program offices, and TA types. Using TA Portal fields on dates of submissions and approvals, annual TA requests were summarized, including denied requests. In addition, the team calculated the average time between steps in the TA process.

For the SNAPS case study, the research team tracked SNAPS’ estimated TA budget by the program office’s homelessness goals, funding source (e.g., HEARTH TA, McKinney-Vento TA), TA Portal topics (e.g., CoC TA, HMIS TA), and TA type; estimated average budget per work plan; the frequency of TA by the seven most common HUD-defined TA topics that providers would link TA engagements to; and estimated budget by regional or field office. For the troubled PHA case study, A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

2 3

the research team described TA use for PHAs that HUD flagged as troubled at least once during the FY 2012-to-FY 2017 period—including the number of TA work plans, their budgets, and the types of TA used—and compared TA uptake among troubled PHAs with PHAs that had other designations that received TA during the same timeframe. See appendix B for more detailed information on how SNAPS

and troubled PHA customers were identified using the data.

Data from document reviews and transcribed in-depth and case study interviews were analyzed for insights into each step in the TA process, perceptions of effectiveness and how to improve TA provision, and the two case study topics. The research team developed a codebook with key themes related to the research questions and issues uncovered during data collection. Using NVivo software, the team organized and summarized data thematically to develop overall findings and findings from the case studies. Triangulation across administrative data, interview data, and document reviews supported analyses of the TA offered and the TA process—its formal structure and divergences in practice.

Study Limitations

Several limitations of the study may have influenced the research findings. Major limitations and how they may affect the analysis of customers, costs, and work plans are detailed below.

Technical Assistance Customers

◼

Customer names were not standardized in TA Portal data, and sometimes names were duplicated across multiple work plans covering the same TA engagement. In addition, the majority of information about which customers received universal TA is not included in the TA Portal. Because of the lack of standardization and information on TA customers, the research team could not determine the extent to which customer data from the TA Portal reflect the actual population. The inconsistent data on customer names and limited data on customers by TA type affected the development of the sampling frame for customer interviews. It is possible that were full and accurate customer information accessible, the sampling frame and respondents interviewed would have changed, which might have influenced findings.

◼

Responsiveness to interview requests varied, which led to fewer interviews with TA customers than planned (11 completed out of 20 planned), decreasing the sample’s diversity.

It is possible that other experiences or perspectives would have emerged had all interviews been conducted.

2 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

Interview respondents’ ability to recall specific TA engagements varied, and several customers had difficulty differentiating between TA provided before and after the implementation of Community Compass. This circumstance limited the research team’s ability to compare experiences of TA recipients who requested or received TA before Community Compass with those after the shift. Customer interviewees for the SNAPS case study also had difficulty distinguishing between TA supported by the SNAPS office and TA provided by other HUD

offices. Better customer recall may have produced further analytical insights.

Costs

TA Portal budget data are based on the estimated rather than the actual budget. The team could not use DRGR budget data, which may be more accurate, because other data inconsistencies prevented us from merging DRGR with the TA Portal data, and the TA Portal overall had more useful variables. The cost analyses may be affected by using estimated rather than actual budgets by TA type and work plans.

Work Plans

The research team could not link work plans that transitioned from on-call TA to direct TA because there was no linking identifier indicating this escalation and no ability to align dates between the end of an on-call work plan and the beginning of a separate direct TA work plan for an individual TA customer. This shortcoming was particularly relevant for the SNAPS case study because most SNAPS

TA usually starts with on-call TA. An escalation analysis would have provided a more detailed picture of TA engagement and how it changes over time with individual customers.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

2 5

III. Technical Assistance Customers

and Providers

Introduction

Understanding what is included in HUD TA and how HUD offers TA requires consideration of who receives and who provides it. These key participants in the TA process—the customers and providers—

are described across all TA programs and in association with the SNAPS TA program and TA for troubled PHAs. The information presented here—drawn from analysis of administrative data, key stakeholder interviews, and the case studies—addresses the following question:

◼

Who are the TA customers and TA providers by TA program?

Key Takeaways

Customers

◼

TA customers include local and state governments, nonprofit organizations, PHAs, and CoCs.

Based on limited customer identification data available from TA Portal work plans, slightly more than one-half of the unique customers are PHAs and CoCs.

◼

A majority of SNAPS customers are CoCs.

◼

For PHAs, 13 percent that are designated as troubled PHAs receive TA, but 13 percent of nontroubled PHAs received TA as well.

◼

Larger customer organizations appear to have more experience with HUD TA and greater capacity to navigate it than smaller ones, based on interviews with a small sample of customers.

2 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Providers

◼

Most providers are consulting firms or housing and community development organizations, many of which supplement staff with subcontractors or consultants. Firms may specialize in providing TA for a particular program or type of TA.

◼

The number of providers has increased over time, as has the average number of work plans each provider executes. Data on work plans and budgets by providers indicates an increase in the number of providers that receive relatively small contracts.

◼

HUD staff also provide TA when warranted by their role in overseeing a program or their subject matter expertise.

Technical Assistance Customers

Customer Characteristics

HUD funding recipients who receive TA through Community Compass represent various entities, including local governments, state and regional agencies, PHAs, and CoCs. Based on limited customer identification data available from TA Portal work plans, there are 1,707 unique customers listed from FY 2012 to FY 2017 in the TA Portal, although most work plans fail to list the type of customer (87

percent of funds were for work plans that are missing customer type).9

The SNAPS case study found that 413 customers received SNAPS TA from FY 2012 to FY 2017.

Of these customers, 62 percent were CoCs; 14 percent were cities; 6 percent were states or territories; 3 percent were counties, parishes, or boroughs; and 15 percent were other types of organizations, including nonprofits.10

The troubled PHAs case study analysis found that 13 percent of all PHAs, or 517 PHAs, received TA during the study period. Among the PHAs that received TA, 11 percent were identified as troubled on the basis of low PHAS and SEMAP scores (exhibit 7).11 The same percentage of troubled agencies 9 This analysis does not account for customers that are listed in the TA Portal under multiple names.

10 See appendix B for additional information on how SNAPS customers were identified in the TA Portal.

11 Slightly more than one-half of PHAs that received TA from FY 2012 to FY 2017, 55 percent, were exempt from PHAS or SEMAP scoring because of RAD or MTW status or because the agency was a TDHE, so only 45 percent of the PHAs that received TA could potentially be troubled.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

2 7

received TA as nontroubled PHAs (13 percent).12 It is important to note that given the time it takes to receive a designation of “troubled” and execute a recovery agreement and action plan, PHAs may no longer be troubled by the time TA can be initiated.

EXHIBIT 7: NUMBER AND PERCENTAGE OF TROUBLED AND NONTROUBLED PUBLIC HOUSING AGENCIES

THAT RECEIVED TECHNICAL ASSISTANCE, FY 2012–FY 2017

Troubled

Nontroubled

Ever Received TA

Never Received

Total

Ever Received TA

Never Received

Total

TA

TA

Count

Percent

Count

Percent

Count

Count

Percent

Count

Percent

Count

-age

-age

-age

-age

57

13%

375

87%

432

460

13%

3,102

87%

3,562

TA = technical assistance.

Notes: Troubled and nontroubled counts may include Rental Assistance Demonstration or Moving to Work public housing agencies that were troubled before entering the demonstration. TA receipt is based on whether they ever received TA, not whether they received TA in the year they were designated as troubled.

Sources: Urban Institute analysis of TA Portal “Work Plan File” received from HUD on August 9, 2018, and PHAS/SEMAP data provided by HUD

Customer Capacity

Larger customer organizations appear to have more experience with HUD TA and greater capacity to navigate the system’s processes than smaller ones, based on interviews with a small sample of customers. Larger customer organizations interviewed often had more experience with HUD TA processes, including existing relationships with TA providers or HUD field or regional offices and previous use of HUD Exchange to request TA. Larger customers, regardless of their history with TA, also tend to have the staff expertise and the capacity to navigate the TA process without the assistance of a field or regional office. For example, state housing agencies are more experienced with TA and have adequate staff resources to explore TA.

12 See appendix B for additional information on how troubled PHAs were identified using TA Portal and PHAS and SEMAP scoring data.

2 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Technical Assistance Providers

Provider Characteristics and Selection

TA providers are firms that receive a cooperative agreement through HUD’s competitive NOFA process to provide one or more types of TA under one or more HUD program offices. Providers are selected on the basis of their subject matter expertise, experience working with particular types of funding recipients or regions, specialty with TA types, and experience working with HUD. The selection process through the NOFA is described in greater detail in the next chapter.

Most providers are consulting firms or housing or community development intermediaries, although some are nonprofit organizations. Most providers use subcontractors or consultants to supplement their TA staff, particularly when they lack the diversity of skills in house for specialized TA needs. Some firms specialize in delivering TA exclusively for one program office or program, especially if they have built a strong relationship with that office and expertise developing and delivering TA associated with its HUD programs. Specialization is especially common among providers working with the Office of Native American Programs and with PHAs.

Provider Work Plan and Budget Amounts

The number of TA providers increased from 19 in FY 2013 to 32 in FY 2017, as shown in exhibit 8.13

The median number of work plans each provider executed each year decreased from 17 to 13 (column B), whereas the median annual estimated budget across all work plans per provider increased from $75,010 in FY 2013 to $240,660 in FY 2017 (column E). The growth in median budgets (column E), along with the decreases in the minimum (column D) and maximum (column F) budgets per provider, point toward an increase in the number of providers that received smaller cooperative agreements.

13 New providers who received funding through NOFAs after FY 2017 are not represented in the data.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

2 9

EXHIBIT 8: NUMBER OF TECHNICAL ASSISTANCE PROVIDERS AND WORK PLAN AND BUDGET AMOUNTS

PER TECHNICAL ASSISTANCE PROVIDERS, FY 2013–FY 2017

Number of

Annual Estimated Budget per

Fiscal Year

Number of Work Plans Per Provider

Providers

Provider

Minimum

Median

Maximum Minimum

Median

Maximum

(A)

(B)

(C)

(D)

(E)

(F)

2013

19

2

17

113

$75,010

$75,010

$6,100,358

2014

21

1

14

114

$11,576

$158,208

$5,087,102

2015

28

1

10

112

$ 3,273

$ 219,499

$4,388,278

2016

30

1

12

135

$1,775

$260,290

$5,717,902

2017

32

1

12.5

190

$3,664

$240,660

$5,590,591

Notes: The universe for this figure is all work plans in the technical assistance (TA) Portal. Dollar amounts have been inflation adjusted to 2018 dollars. For work plans that span more than 1 year, the estimated budget is spread equally over all fiscal years of their period of performance. All years shown are fiscal years. These summary statistics exclude providers that received $0 in funding and zero work plans in the given year.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

For SNAPS, the number of TA providers stayed about the same from FY 2013 to FY 2017, whereas the average number of work plans per provider more than doubled, from 17 in FY 2013 to 39

in FY 2017 (not shown). The increase in the number of work plans per provider partially reflects an increase in reporting activity in the TA Portal; however, some changes reflect increases in the number of work plans. The increased average budget—from around $540,000 to $1.38 million—and the number of work plans per provider—from a maximum of 44 to a maximum of 81—between FY 2013

and FY 2017 suggest an increase in providers’ SNAPS TA workload (not shown). A few program staff said that SNAPS prioritized working with new providers during the shift to Community Compass, so although the number of providers did not change substantially—from 9 to 11—the distribution of the proportion of the SNAPS budget that each provider received may have changed between FY 2013 and FY 2017.

Budget Share of Top Five Providers

The TA budget received by the top five providers based on all work plans in the TA Portal declined from $21.4 million in FY 2013 to $18.5 million in FY 2017 (in 2018 inflation-adjusted dollars; not shown). The overall budget share held by the top five providers also decreased, from 74 percent in FY

2013 to 49 percent in FY 2017 (not shown). This shift in the budget share reflects the greater diversity in TA providers, which increased from 19 in FY 2013 to 32 in FY 2017.

Compared with the distribution of the TA budget overall, the SNAPS budget share held by the top five providers in each year did not decrease to the same extent—dropping from 84 percent in FY 2013

3 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

to 79 percent in FY 2017. The overall SNAPS TA budget increased during this time, however, and the budget received by the top five providers increased.

HUD Offices as Technical Assistance Providers

HUD headquarters and field staff provide TA outside of the Community Compass NOFA process. For example, regional CPD staff deliver workshops on financial management for HUD funding recipients assessed at higher risk of financial distress. Such TA is provided as part of field staff’s day-to-day grants management role, placing them in a role similar to firms that provide TA. Subject matter experts within HUD program offices also provide TA when the program or policy is highly technical or requires HUD oversight. For example, provider interviewees said that HUD program staff provide training and webinars on HOME program-related policy issues. When customers require tailored solutions to address their needs, HUD typically engages providers for TA delivery. If regional offices identify customers with compliance findings that require remedial assistance, regional offices may provide their own TA (e.g., CPD training or a webinar) to correct the issue.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3 1

IV. The Technical Assistance

Process: Step by Step

Introduction

This chapter describes how the Community Compass program delivers TA from provider selection through TA engagement. It highlights process innovations, divergence in processes among programs, and areas where the process does not run smoothly. The research team analyzed the administrative data analysis, in-depth interviews, and case studies to better understand how TA is administered at HUD. This chapter addresses the following questions:

◼

What types of TA are requested each year through the TA Portal, and who enters these requests?

◼

How many requests are rejected each year and why, and to what extent do rejected requests appear to represent valid unmet needs?

◼

Which types of TA can be required by HUD? How does that affect the delivery of TA?

◼

How do processes compare among TA programs? Across different TA types?

◼

Has Community Compass facilitated better coordination, collaboration, and communication across HUD offices and programs and TA providers?

◼

How much contact do TA providers have with customers? What are the types and nature of those contacts?

Key Takeaways

◼

There are five key steps in the TA process: the NOFA and provider selection process, need identification, provider assignment and work scope, TA engagement, and closeout. This process varies on the basis of the approaches and types of TA and the HUD office and program involved.

3 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Notice of Funding Availability and Provider Selection Process

◼

The NOFA and TA provider selection process was centralized into one process that reduced duplication and the time it takes for TA providers to receive the funding awards.

Technical Assistance Requests and Need Identification

◼

Community Compass standardized the TA request process for targeted TA to be submitted through the TA Portal; however, only a few TA requests are made through the TA Portal.

Often, targeted TA requests originate from HUD headquarters and field staff or are requested by TA customers through informal channels.

◼

Universal TA typically stems from HUD headquarters and field staff identifying a need shared by multiple customers.

◼

HUD headquarters staff may require HUD funding recipients to participate in some forms of TA, typically to promote knowledge about new guidelines or capacity to implement new requirements. As an alternative, TA participation may be required as a condition for receiving HUD funds or participating in a HUD program. HUD does not require troubled PHAs to participate in any specific TA by HUD, although the agencies must take the actions mutually agreed upon in their recovery agreements. Actions may include specific TA.

◼

Forty-four percent of the requests for TA in the TA Portal were rejected, many of them requests for assistance not available through the Community Compass TA program. Most of these were requests for housing assistance or simple questions better addressed through a referral to HUD’s Ask A Question page.

Provider Assignment and Work Scope

◼

TA providers typically have informal conversations with HUD headquarters staff about their availability and capacity to take on a TA engagement before being officially assigned the request in the TA Portal by the HUD Government Technical Representative (GTR). TA providers scope out the TA work plans in conjunction with HUD headquarters, field staff, or both before beginning a TA engagement.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3 3

Technical Assistance Engagement

◼

HUD has increased the use of universal TA across all program offices over time to maximize its reach to customers who face similar problems while minimizing the cost of providing TA.

◼

Targeted TA is still widely needed to support TA customers with unique challenges by providing TA tailored to their individual needs and contexts.

◼

Some TA engagements lead to additional TA requests, such as a needs assessment leading to delivering TA for identified issues or on-call TA escalating to a direct TA engagement.

Closeout

◼

The TA Portal provides little guidance or information on closing out TA work plans. Work plans often stay open in the TA Portal after a TA engagement has ended. Work plans are generally closed out in DRGR, the official award management system, upon completing the work plan.

Process Overview

Policy Development & Research provides agencywide leadership for HUD TA, and the Technical Assistance Division (TAD) within CPD provides operational, technical, and administrative support for TA across program offices at HUD. The TA process varies greatly by the type and approach of TA, HUD program office, and funding source, although there are five core phases of the process that remain consistent across variations: (1) the NOFA process and provider selection, (2) identifying a customer’s need and examining TA requests, (3) assigning a provider and scoping the work, (4) developing and delivering the TA (TA engagement), and (5) closing out the process. Exhibit 9 shows the general TA process.

Variations in the TA process typically stem from the different approaches and types of TA. For example, the process for developing tools and products accessible to numerous customers differs from the process for providing direct assistance to a customer to solve organizational challenges. TA processes also vary across HUD offices because of differences in their organizational structure, office-specific theories of change, or the types of TA customers with whom they work. For example, the ONAP division operates differently from the rest of PIH and receives its funding from a separate source, the Native American Housing Assistance and Self-Determination Act. Differences in processes 3 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

among HUD offices also stem from their TA processes before Community Compass. Some of the legacy processes have carried over—reducing standardization but maintaining the fit of the processes to the program or TA involved.

EXHIBIT 9: GENERAL TECHNICAL ASSISTANCE PROCESS

NOFA = Notice of Funding Availability. TA = technical assistance.

Notice of Funding Availability and Provider Selection

Process

The NOFA and provider selection process is one of the key steps in the TA process that was consolidated across programs and streamlined with the Community Compass program. Before Community Compass and OneCPD, each HUD office conducted its own procurement and provider selection process. This decentralization created duplicative work for the HUD headquarters staff and meant that providers had to complete NOFA applications to provide HUD TA for each program office they wished to work. Under this model, the TA funding also was siloed by office, which caused challenges in addressing multiple needs of customers who interacted with more than one HUD office.

Community Compass’s shift to a cross-program funding approach and consolidated NOFA process reduced the burden for TA providers and allowed TA “to address the needs of recipients and subrecipients across multiple HUD programs, often within the same engagement” (HUD, 2018).

PD&R leads the drafting of the Community Compass NOFA language and chairs the Technical Assistance Executive Committee. After the NOFA language is finalized, the rest of the process for provider selection unfolds, as shown in exhibit 10. HUD posts the NOFA to solicit applications and award TA provider cooperative agreements for the next fiscal year. Starting in FY 2018, the NOFA awards cover 2 fiscal years rather than 1, another change designed to increase efficiency. The TA provider applicants have between 6 weeks and 2 months to submit an application in response to the NOFA.14 After the submission period closes, HUD evaluates applications and selects providers through a multistep process.

14 The 2018/2019 NOFA deadline was originally for 2 months, but it was extended due to the federal government shut down.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3 5

EXHIBIT 10: NOFA PROCESS AND PROVIDER SELECTION

CPD = Community Planning and Development. NOFA = Notice of Funding Availability. TA = technical assistance.

A panel of HUD headquarters reviewers scores applications from different HUD offices with diverse subject matter expertise and administrative knowledge. At least two reviewers from the panel score each application on the basis of scoring criteria and reviewer guidance developed by TAD

through the Announcement Review Module and discuss discrepancies across their scoring. The panel then provides the applications that met a minimum threshold of 75 points (on a 100-point scale) to HUD’s program offices, along with providers’ program interests and capabilities.

The next step in the approval process diverges on the basis of the source of funding. For TA funding earmarked for specific programs, the TA provider selection decisions are made by the divisions that administer those programs; for example, Public Housing Receivership and Recovery funding is determined by PIH, Native American Housing Assistance and Self-Determination Act TA is determined by ONAP, and McKinney-Vento funding is determined by SNAPS. Some departmental TA funding is earmarked for specific offices—PIH, Housing, and CPD. The divisions select providers with whom they prefer to work in the upcoming fiscal year(s) out of the scored applications that met the minimum threshold for these departmental funds. The divisions are not required to select the applications with the highest score, but they must justify their selections, including why they skipped a higher scoring application for a lower scoring one. The Assistant Secretary for the division gives the final sign-off on the selections.

For departmental TA that is cross-cutting, the TA Executive Committee takes the lead. The TA Executive Committee comprises the Assistant Secretaries of CPD, PIH, Housing, FHEO, and PD&R.

Cross-cutting departmental TA funding is negotiated among the TA Executive Committee to determine how much funding should go to which providers. The TA Executive Committee formally votes to approve all award decisions, and the Deputy Secretary signs off on the award selections.

3 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

The TA provider cooperative agreements award an allocation of funding that the TA provider can earn through particular TA engagements. The TA engagements assigned to that provider must be fulfilled within that allocation of funding, and the provider is paid only for the TA delivered. Funding allocations can roll over year-to-year for up to 3 years. Some providers might not be awarded funding for the year through the NOFA process because they have a large amount of allocated funding rolling over from previous fiscal years.

HUD staff said the provider selection process improved in a couple of ways after the NOFA process was centralized. HUD headquarters staff noted that switching to the new NOFA process “was a huge relief because the classic contracting model was so cumbersome, and our turnaround times on requests for TA shortened….That was a significant improvement.” The new process creates fewer cooperative agreements across HUD offices and decreased the administrative burden for HUD staff, who had previously managed multiple grant competitions and programs. TA providers reported that the burden had been greatly reduced by the consolidated NOFA, although responding to the NOFA is still challenging. Some TA providers—particularly those who regularly work with HUD—have staff who specialize in responding to the NOFAs each year. These staff members note that the NOFA requirements change year-to-year, which requires them to scramble to provide new information.

Over the course of Community Compass, HUD has continued making adjustments to the NOFA process to further reduce the burden on applicants and make it accessible to new and smaller TA providers. As noted in the TA Provider section in Chapter III, only a small number of TA providers were awarded cooperative agreements when Community Compass began. Five providers were awarded most of the funding. Providers reported that the concentration could be due to the NOFA application burden and preferential treatment of previous TA providers. To make the NOFA application and process more accessible, HUD altered the NOFA application and the way providers are scored. In the past, providers who previously had received a HUD TA award would be awarded extra points on their applications. HUD stopped that preferential scoring, opening up funding opportunities to new applicants. As discussed in the previous chapter, the number of providers gradually increased from 19

in FY 2013 to 32 in FY 2017, and the budget concentration with the top five providers reduced by 25

percent. In the FY 2018/2019 NOFA, HUD switched to a 2-year NOFA period so providers would not need to apply each year.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3 7

Technical Assistance Requests and Need Identification

The second step of the TA process is to identify a customer’s need for TA, which begins either with submission of a TA request or when HUD staff recognize a need. The starting point of need identification depends on the program office, division, TA approach, or goals of the TA. A customer or HUD field staff may submit requests. TA also may be initiated by HUD program office or field staff.

Once the relevant HUD program office receives a request, the office reviews the TA to approve or deny the request. This process is shown in the top row of exhibit 11. The process differs considerably by type of TA and by HUD office.

EXHIBIT 11: TECHNICAL ASSISTANCE NEED IDENTIFICATION

TA = technical assistance.

Targeted Technical Assistance

Community Compass expanded the use of the TA Portal, developed under precursor TA structures, as a mechanism to standardize the process for submitting TA requests, particularly for targeted TA.

Targeted TA typically is offered in response to a customer’s request, sometimes made in conjunction with a customer’s field office or HUD headquarters. When there is a critical mass of requests, some program offices assemble a review committee to evaluate requests and vote on whether to approve them.

The standard processes for requesting TA and reviewing requests vary across HUD program offices. Some program offices—such as the SNAPS office—discourage field offices from submitting requests and prefer customers to do so instead, others allow only the field offices to submit requests, and some TA requests come directly from HUD headquarters program offices. PIH’s approach to TA requests is distinct among the program offices. According to interviews, all TA requests to PIH—other than those related to the troubled PHA and the receivership notification process or initiated by HUD

3 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

headquarters staff—are submitted by PHAs to the field offices, where staff determine if the requests should be submitted through the TA Portal. 15 The field office staff then draft the system requests in the TA Portal and forward them to the regional office. Regional office directors assess requests and determine that a resource other than TA can help address an issue and decide whether one agency’s TA request should be prioritized. Once the regional director requests are forwarded to HUD

headquarters, regional and headquarters staff consult further, if necessary, to define TA needs and scope.

Analysis of administrative data found that about 81 percent of targeted TA funding—direct, on-call, and needs assessment—did not begin with a request submitted through the TA Portal. Exhibit 12

shows the portion of TA work plans that originated from a TA Portal request by TA type, with direct TA and on-call TA most likely to originate from a TA Portal request.

EXHIBIT 12: HUD TECHNICAL ASSISTANCE ESTIMATED WORK PLAN BUDGET BY WORK PLAN

ORIGINATION AND TECHNICAL ASSISTANCE TYPE, FY 2012–FY 2017

TA = technical assistance.

15 The troubled PHA notification process in PIH has yet a different process for TA requests and reviews. After either the field office or a TA provider conducts a needs assessment, a troubled PHA submits a plan to PIH for rectifying the problems that led to unsatisfactory performance assessment scores. The plan can include the TA needed to improve the agency. Additional detail on the troubled PHA notification process and how TA is requested and agreed upon is provided in appendix C.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

3 9

Notes: The universe for this figure is all work plans in the TA Portal. Dollar amounts have been inflation adjusted to 2018

dollars. Outside of TA Portal requests, work plans could originate from HUD headquarters initiatives, requests informally submitted, or mechanisms such as help desk support.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

Interviews suggest several reasons why not all TA work plans start with a TA Portal request. The TA Portal request process can be bypassed for direct TA when TA need identification originates in HUD headquarters. If a PHA is designated as troubled or is placed in receivership, for example, PIH

creates the necessary statement of work or scope of services for the development of a work plan in the TA Portal and moves to the provider assignment phase, bypassing the request process.

Alternatively, when a TA engagement identifies the need for additional TA, such as an on-call TA engagement that must continue as a direct TA engagement, a new work plan assignment will be created without a request in the TA Portal.

Universal Technical Assistance

The need for universal TA development typically is determined by HUD headquarters or field staff and rarely originates from the TA Portal request process. Less than 1 percent of universal TA (i.e., tools and products and self-directed and group learning) stems from a TA Portal request (exhibit 12). Instead, when HUD receives numerous requests for the same type of information or support for which no products are developed, staff consider developing new universal TA offerings. Providing universal TA allows HUD to maximize its reach to customers while minimizing the cost of TA provision. For example, HUD headquarters staff will develop a regional, in-person training for multiple TA customers or create online resources available through HUD Exchange. PIH has used a universal approach, most notably, with the self-guided, online Lead the Way training—a self-paced training targeted primarily at PHA board members or commissioners and executive staff to provide them with a baseline knowledge to better govern and manage their PHA. The majority of universal TA offerings—outside of offerings such as regional training—are available to all HUD funding recipients through HUD Exchange and are accessible without submission of a TA request. Other examples of offerings available on HUD

Exchange include a Family Self-Sufficiency Guidebook and sexual harassment prevention training recently created for PHA leaders.

Required Technical Assistance

HUD headquarters staff may direct providers to deliver required TA to HUD funding recipients to provide information about regulatory guidelines or new HUD processes and protocols; for example, a 4 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

provider held a webinar about the implementation of Small Area Fair Market Rents. Sometimes a provider will work with a HUD field office to design and develop TA that the field office will deliver (e.g., webinar or training curriculum). As providers pointed out, one challenge with required TA is that TA providers are less likely to know if the content is specific enough to adequately address the individual challenges that customers will face. Regional offices cannot require TA, but if they identify customers with compliance findings that require remedial assistance, regional offices may provide their own TA (e.g., CPD training or a webinar) to correct the issue.

HUD headquarters or field offices may require a customer to receive TA as a condition for receiving HUD funds or participating in a HUD program. HUD requires PHAs in receivership to participate in TA as a condition to return to local control. HUD does not require TA for troubled PHAs, but activities agreed upon by HUD and the PHA in the recovery agreements are required to be completed. According to interviewees, in the past, HUD could require troubled PHAs to receive TA as a condition for curing their troubled status. Under Community Compass, however, HUD allows troubled PHAs to determine the action steps for recovery to recognize agencies’ independence. In response to a PHA’s proposed recovery agreement and action plan, however, HUD may recommend—

sometimes strongly—specific TA offerings. The PHA is not required to accept the recommendations but must demonstrate how it will remedy the identified issues; otherwise, HUD may reject an agency’s plan, and the PHA must revise and resubmit the plan for approval. If the final approved and executed recovery agreement between the PHA and HUD includes TA, the specific TA activities outlined in the agreement then become required. The HUD field office monitors the PHA’s progress toward achieving the targets identified in the plan through monthly reports.

Technical Assistance Requests Not Approved

Of the TA engagements requested through the TA Portal for which data were provided, approximately 44 percent, or 1,872, were approved and 2,200 were denied. Data on reasons for the denials are limited, although examination of TA request descriptions suggests that many of the requests denied were not appropriate TA requests. A number of these requests were from individuals seeking housing assistance rather than recipients of HUD funding seeking TA. In addition, the TA Portal records referrals to HUD’s Ask A Question help desks as “Not Approved,” although requestors receive answers to their submitted questions.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

4 1

Provider Assignment and Work Scope

After HUD headquarters staff in TAD and the relevant program office decide to fund a TA engagement, they outline a scope of work and assign a provider to develop a work plan and provide the TA. The provider develops an understanding of the purpose of the TA through discussions with HUD staff. From there, the provider finalizes the work plan and submits it to HUD for approval.

Exhibit 13 shows this basic workflow.

EXHIBIT 13: PROVIDER ASSIGNMENT AND WORK SCOPE

NOFA = Notice of Funding Availability. TA = technical assistance.

The process for assigning TA providers and developing scopes of work is fairly consistent across TA approaches; however, the criteria used for provider assignments and scope development and the stakeholders involved vary across HUD offices.

During interviews, TA providers said that, based on the standard process, their engagement with an assignment begins when they receive an assignment notice via the TA Portal that they have been given an approved TA engagement. Nevertheless, nearly all of these providers noted that typically, they have informal conversations with HUD in advance of receiving an assignment to discuss the TA request and the providers’ capacity and expertise to take on a project. According to HUD

headquarters staff, HUD makes assignments on the basis of prior experiences with providers and the providers’ past work with certain types of clients. One provider explained that compared with OneCPD, which provided program-specific awards, Community Compass’s TA could cover multiple topics or require support from multiple program offices. Conflicts can arise when, for example, one office prefers working with one provider, but another office is not convinced the provider is the right choice. This conflict can lead to delays in executing work plans and launching TA. Exhibit 14 details the provider assignment process for SNAPS.

4 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 14: SNAPS PROVIDER ASSIGNMENT PROCESS

Before Community Compass implementation in FY 2014, SNAPS approved TA requests via their field offices, maintaining their own cooperative agreements with providers. This approach ensured that the providers regularly interacted with field office staff to apply for TA funds and report on TA engagements. After Community Compass implementation, the SNAPS office shifted to a more centralized approach and began assigning TA engagements to providers through the SNAPS

headquarters office. Most types of TA—including direct TA, products, and training—are assigned to TA providers on the basis of their expertise. When appropriate, the SNAPS office gathers input from its regional teams to inform provider assignments. Once headquarters approves a TA request, the request is forwarded to the relevant regional team. The regional team will then recommend a TA provider on that team to SNAPS headquarters staff for the engagement on the basis of the provider’s expertise and the regional team member’s relevant experience working in similar communities. The SNAPS

office then assigns a provider to the TA engagement, considering the regional team’s recommendation.

Customers receive notifications about the provider assigned to a particular TA engagement via emails originating from the TA Portal. The same notification also may be sent to a customer’s desk officer, the SNAPS employee at HUD headquarters whose responsibility is to oversee TA in that jurisdiction and act as a primary point of contact for HUD funding recipients. The CPD Field Office Director also is notified when a TA provider is assigned to a request.

Once formally assigned a TA request, providers work with HUD headquarters and field staff to determine the specific tasks and deliverables. Developing TA work plans can be a complicated process of negotiating the interests of multiple stakeholders, although the TA customer often is not involved in these conversations to set the work plan.

The TA provider submits the work plan through the TA Portal for review and approval, first by the HUD program office and then the GTR in TAD, before beginning work with the customer. (Details on the TA process timeline are discussed in Chapter V.) HUD headquarters staff can request that a provider make additional revisions to a work plan and resubmit it through the TA Portal. According to providers, their informal discussions with HUD staff before submitting a work plan may help limit revisions and resubmissions. Without the discussions, the process could be drawn out.

Community Compass allows some flexibility in the delivery of TA to ensure that customers’ core challenges are tackled and to foster good relationships among HUD, providers, and customers. One provider reported having had several projects that began with a TA work plan focused on conducting a needs assessment to ensure that they adequately understood the customers’ priorities. This initial work would lead to scoping a direct or on-call TA engagement for the customer with HUD

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

4 3

headquarters. Such iteration is not available in most TA engagements, which do not include a needs assessment task, because once the TA work plan is set, there is little room for adjustments to scope and timeline. TA work plans are written in vague terms, however, to allow providers some flexibility to address customers’ needs that are identified after work plan approval. Another provider spoke about how they developed the scope of TA plans that covered several PHAs. To ensure that work could be tailored to the customers’ needs, the provider would first submit a general TA work plan applicable to all PHAs for approval. Then, as part of the TA engagement, they would conduct a “remote review” and a site visit with each PHA to develop project plans based on an individual PHA’s needs.

Technical Assistance Engagement

The TA engagement process looks different for targeted TA than universal TA, although engagement for both approaches begins with developing material, such as creating tailored content, training, or implementation plans. Providers’ development of targeted TA materials and support often coincides with coordinating with the customer and learning more about their needs. By contrast, universal TA typically is developed in coordination with HUD staff before engagement with customers. During TA delivery, providers must report monthly to HUD on the progress and outputs of an engagement.

(Reporting requirements and processes are covered in Chapter VII). Exhibit 15 provides an overview of the standard TA engagement process. Within the TA approaches, the processes also vary across program offices and programs.

EXHIBIT 15: TECHNICAL ASSISTANCE ENGAGEMENT

NOFA = Notice of Funding Availability. TA = technical assistance.

4 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Targeted Technical Assistance Compared with Universal Technical Assistance HUD has increased the use of universal TA across all program offices over time. This shift allows HUD

to maximize its reach to customers while minimizing the cost of providing TA. Universal TA provides resources to multiple customers who face similar problems that can be solved with similar solutions.

Standardized materials and training are not appropriate for all TA engagements or for all customers.

Provider and customer interviewees explained that even when a TA customer faces organizational issues comparable to those experienced by other HUD funding recipients, their challenges are often related to their local context and circumstances. In these instances, a TA customer requires customized, direct TA. For example, as one TA provider explained, all troubled PHAs face “ineffective, weak, or poor oversight; staffing issues; …loss of staff; [or] the hiring of people that weren’t qualified….

I think the global issues are fairly common”; however, “it’s not the same at any one of the places.”

Even when providers administer direct, customized TA, they may incorporate standardized processes and products. One provider begins engagement with a troubled PHA by reviewing the PHA’s PHAS and SEMAP reports and each deficiency to determine if any of the findings can be appealed and to understand the organizational areas needing improvement. Providers draw from effective practices and tools they have used with similar customers in the past. Starting with manuals, templates, and other material used elsewhere, they alter content to meet a customer’s specific needs and update the provider’s own policies and procedures.

Targeted Technical Assistance

Targeted TA addresses the individualized needs of TA customers. Among the types of targeted TA—

needs assessment, on-call, and direct—direct TA is used most widely, as indicated by analysis of budgets and number of work plans in TA Portal. Direct TA is used for longer term engagements with one customer or a small number of customers with similar challenges. Most direct TA is provided remotely, through phone calls and emails, although it often includes onsite visits with the customer to support collaboration and develop a better understanding of the local context in which the customer operates. For example, a collaborative of five communities based in the Kansas City region received direct TA from HUD to support the communities’ regional assessment of fair housing and the development of fair housing goals. The TA provider helped the jurisdictions work together and provided data and mapping support. TA was delivered through site visits, telephone calls, and email exchanges.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

4 5

On-call TA addresses similar challenges to direct TA, but engagements are capped at 16 hours, and they are conducted remotely. The SNAPS office, in particular, starts engagements with on-call TA whenever possible. Rather than channeling a customer request into a lengthier direct TA process, starting with up to 16 hours of assistance through on-call TA allows TA providers to begin work immediately to try and provide a timely solution to a customer’s needs. The quick start possible with on-call TA means providers can begin engagement without going through the process to develop a customized work plan and submit it for review and approval. Instead, SNAPS provides standard guidance to providers for on-call TA on how to use the time with customers. On-call TA also can reduce the need for a separate needs assessment. Program staff and providers shared that providers often used the first few hours working with a customer to conduct a needs assessment. Limiting the length of time of a TA engagement also may reduce overall TA costs. One HUD headquarters staff shared an internal cost analysis showing that the average direct TA engagement costs around $50,000, compared with an average cost of $1,600 for an on-call TA engagement.16

Both on-call and needs assessment TA may be used as precursors to direct TA. As previously discussed, some offices use these types of TA to increase staff understanding of a customer’s need and the scope of the assistance required before approving a direct TA work plan. Interviews suggest that, in practice, needs assessments might be provided as part of on-call TA or rolled into a direct TA work plan rather than appearing in the TA Portal as a separate TA engagement. PIH’s troubled PHA designation process provides an example of early use of needs assessments. As explained in detail in appendix C, field offices are required to conduct or assign a TA provider to complete a needs assessment of a troubled PHA. The needs assessment helps the provider fully scope the PHA’s problems and formulate a recovery plan, including requests for TA that were suggested during the assessment.

Although many TA requests can be resolved with the initial 16 hours of TA, circumstances may lead providers to submit an escalation request to transition from on-call to direct TA engagement, which is more resource and time intensive. The SNAPS office approves these requests relatively infrequently—only when one of two conditions is met: an emergency exists that may significantly affect residents’ housing status (e.g., public health crisis, natural disaster), or there is a notable increase in the overall number of people or a specific subgroup of people experiencing homelessness within the CoC service area. SNAPS headquarters staff and TA providers explained that SNAPS maintains stringent standards and protocols to ensure they do not escalate on-call TA without sufficient justification because direct TA costs more. As one TA provider said, “There are protocols that the 16 The research team was not able to validate this statement with the data available for analysis.

4 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

SNAPS office has developed to really ensure that we are, as TA firms, trying to do as much as we can in the 16-hour situations and leverage other resources...to do more intensive engagements in just those limited circumstances that have been identified.”

When escalation is necessary, HUD notifies a TA provider that it expects the engagement to extend beyond the initial 16 hours of remote work and requests providers to “work with the community as quickly as possible to validate that understanding, and then put it in a broader and more comprehensive work plan to include a higher level of effort,” according to one provider. Alternatively, a TA provider can prepare a TA upgrade request if they think an on-call work plan will not be sufficient to resolve the problem at hand. The request details the problem, the work a provider has done already, and the provider’s suggestion for the next steps to solve the problem.

Universal TA

Universal TA, intended as a cost-effective approach to meet the needs of many TA customers at once, includes tools and products, help desk support, and self-directed and group learning. Offerings are standardized, providing information applicable to all similar customers. Universal TA materials are available through HUD Exchange, the website created for Community Compass to house information and support for HUD funding recipients. Examples of universal tools and products include web-based courses, such as a recent COVID-19 Planning and Response for Homeless Assistance Providers webinar, and program manuals, such as the CPD Income Eligibility Calculator User Manual. Help desk support, also called Ask A Question, is hosted on HUD Exchange and is available for most programs under Community Compass. Insight into customers’ experiences with the help desk is provided in exhibit 16.

EXHIBIT 16: HELP DESK SUPPORT

HUD funding recipients with discrete questions about policies and programs may submit a question through HUD Exchange using the help desk’s Ask A Question (AAQ) form. The TA provider for the help desk receives questions and answers them directly or refers requestors to existing materials also available on HUD Exchange, which is helpful in many cases. The AAQ function of the help desk offers a centralized, light-touch form of TA available to all funding recipients that is relatively easy to access.

Customers reported that the AAQ request form is easy to complete, and simple questions received a quick response. They also noted some challenges with more complicated questions, including miscommunications caused by differences in language used to describe the issue motivating customers’ requests and longer wait times for responses to questions that require input from multiple staff or departments. Based on these reflections, customers shared that AAQ may be effective in A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

4 7

answering simple questions promptly, but customers might need to address more complicated questions to their HUD program officer or other HUD staff.

Self-directed and group training were the types of universal TA most discussed during interviews with HUD staff and customers. Several HUD headquarters and field staff said that HUD headquarters program staff initiate the development of self-guided and group training once they have received requests from many customers for information on a content area. Onsite trainings are held in a central location to encourage HUD funding recipients to attend. Self-guided webinars, hosted on HUD

Exchange, also have been a popular TA approach. PIH has used it, most notably for the online Lead the Way training. The online training covers topics such as asset management, budgeting, ethics, and, most recently, preventing sexual harassment, among others. PHA staff said the Lead the Way training has been very helpful to their organizations. Several PHAs reported that they require their board members to complete the training as a condition for serving on their board.

Delivering and Managing Technical Assistance Engagements

HUD offices and programs vary in how they organize and structure TA engagements and work plans.

Working across multiple HUD TA programs and work plans can complicate managing TA engagements for a single provider. Some programs, such as ONAP, manage their TA by approving many small TA work plans for engaging a customer, each of which covers different deliverables or customer needs.

Other programs create larger work plans encompassing all the TA and support that a particular customer needs.

Providers reported having anywhere from one to almost 100 open work plans in the TA Portal.

These firms have more work plans open when they deliver TA under multiple programs or have more complex work plans. Some HUD program offices have their own TA process that includes additional steps in delivering direct TA, resulting in numerous work plans or amendments. For example, SNAPS

offers a more comprehensive approach to TA than other program offices. Its approach involves strategic planning for providers on formulating the work plans and task orders before they are entered into the system.

Larger direct TA engagements can include more than one type of support within the work plan, and these types of supports can mirror other types of TA engagements. For example, interviews and work plan reviews indicate that some direct TA work plans set the first task to be completing a needs assessment. Other direct TA work plans include developing agency-specific training to improve staff 4 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

capacity or working with the customer to develop customized tools and products, such as new processes or required documents. The inclusion of several types of TA in the direct TA work plans makes it difficult to identify in TA data the TA customers actually receive. For example, a review of TA Portal data indicates that troubled PHAs predominantly receive direct TA; however, TA providers and customers said several approaches are taken to build PHA capacity during direct TA engagements.

Closeout

The TA Portal provided little information about closing out a TA engagement, and in general, interviewees reported that the process was disjointed and not standardized. Generally, a TA engagement is complete when all promised deliverables have been submitted to both the TA customer and HUD. TA work plans often stay open within the TA Portal long after the TA engagement is completed, however, because HUD must close out billing and administrative tasks on their side. While a work plan remains open, providers are supposed to continue submitting regular reports on work that is no longer in progress. These open work plans make assessing and analyzing TA challenging. It is difficult to understand the actual performance period of an engagement or measure the timeliness of a provider’s TA delivery when work plans remain open in the TA Portal beyond the completion of work.

Providers would like more and better reporting and analysis on outcomes associated with the TA they have completed. Currently, each work plan assignment has its own report; however, there are no followup reports on the impact of the TA. TA providers are not analyzed on their overall effectiveness for their TA work with HUD across TA engagements. More information about the reporting and evaluation of TA is provided in chapter VII.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

4 9

V. Timelines and Costs

Introduction

Part of assessing the impacts of Community Compass on TA is to examine data and perceptions on the timeliness of TA processes, the types of TA that are provided over time, and their associated costs.

Using HUD administrative data and qualitative data from interviews with HUD staff, TA customers, and providers, this chapter addresses the following questions:

◼

What TA types does HUD fund, and how much do they cost?

◼

What types of TA are provided, and in what quantity?

◼

How does the TA process work from the perspective of HUD Community Compass program staff? Of TA providers? Of TA customers? How does it compare with how HUD provided TA in the past?

◼

What is the average time between steps for TA provision, from TA request to the completion of the work plan?

◼

How quickly does HUD respond to TA requests?

Key Takeaways

Timeliness of the Technical Assistance Process

◼

According to administrative data, one-half of all TA requests received through the TA Portal take less than a week to be approved by the relevant program office as an eligible request for TA and for a TA provider assigned to respond to the request. The median time it takes from the request date to approval of the work plan developed by the TA provider is 2 months for the Government Technical Monitor (GTM) and a little more than 3 months by the GTR. GTMs are located in program offices, and their duties are programmatic through monitoring and assisting throughout the TA process, whereas GTRs are centralized in CPD’s TAD and focus on overall award management and oversight administrative work (e.g., voucher and work plan approval).

5 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

Some HUD headquarters staff believe approval times improved under Community Compass, but other staff thought moving GTRs from most program offices to TAD slowed the TA administration process because the offices lost immediate access to staff who approve and oversee the TA.

◼

CPD field staff indicated that changes since the shift to Community Compass had increased the time it takes to approve requests.

◼

At least one-half of all TA work plans are completed in less than a year, including the delivery of direct TA and the development of universal TA tools and products.

◼

Customers can work their way through available universal TA tools and products quickly, whereas targeted TA engagements often take longer, from developing work plans to completing the TA delivery.

Types of Technical Assistance and Costs Over Time

◼

Direct TA accounts for most of HUD’s TA costs documented in the TA Portal but represents a decreasing share of costs as other TA types, including tools and products and self-directed and group learning, have increased since the beginning of Community Compass.17

◼

Direct TA and tools and products work plans cost the most of all TA types, whereas self-directed and group learning work plans cost the least.

◼

According to the data, SNAPS TA represents around 40 percent of HUD’s overall estimated TA budget and has a similar distribution of TA types over time to HUD’s overall TA budget.

◼

PHA customers account for almost one-half of HUD’s direct TA budget, a significant increase since Community Compass launched. Based on a review of work plans, PHAs almost always receive direct TA, although customer access to online universal TA materials is not represented in the TA Portal data.

17 This report and its exhibits use the terms from HUD NOFA definitions rather than the TA Portal terms when referencing budget and work plan amounts by TA type from the TA Portal. The report uses tools and products to refer to anything listed as product development, and self-directed and group learning to refer to training delivery because the work plan activities detailed in the TA Portal categories most closely reflected those HUD definitions.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5 1

Timeliness of the Technical Assistance Process

Request Approvals, Provider Assignments, and Work Plan Approvals

According to administrative data, TA requests received through the TA Portal are quickly approved and assigned to providers. It typically takes less than a week for a request to be approved by the program office as an eligible request for TA and for a TA provider to be assigned to respond to the request (exhibit 17). The median time for GTM approval by the program office is 2 months and for GTR approval is a little more than 3 months, which means one-half of the approvals go more quickly, and one-half take longer.

Some HUD headquarters staff believe these approval times improved with Community Compass implementation. One staff member argued that Community Compass had shortened the approval process significantly, noting that it changed “from as much as 18 months down to just 3 or 4 months.”

EXHIBIT 17: MEDIAN NUMBER OF DAYS SINCE REQUEST DATE FOR TECHNICAL ASSISTANCE

ASSIGNMENTS, APPROVALS, AND COMPLETIONS FOR TECHNICAL ASSISTANCE REQUESTED THROUGH

THE TECHNICAL ASSISTANCE PORTAL, FY 2012–FY 2017

GTM = Government Technical Monitor. GTR = Government Technical Representative.

Notes: The universe for this figure is only work plans with available request data in the TA Portal (37 percent) from FY 2012–FY

2017. On the scale, 0 indicates the date the request was submitted. The research team could not explain why some dates appear out of order (e.g., period of performance start date occurring before the request date or request approval date).

Source: TA Portal “Request File,” “Assignment File,” and “Work Plan File” received from HUD on August 9, 2018

More HUD headquarters staff indicated, however, that Community Compass had complicated and lengthened the approval process by shifting internal processes in a way that slowed staff approval rates. Some program offices could turn around work plans in a day, whereas now GTRs have a 30-day 5 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

window for approval. HUD staff and TA providers suggested it can now take up to several months for a work plan or amendment to be approved.

As GTRs were shifted to TAD within CPD as part of Community Compass, TA requests and administration were pulled out of other program offices. Multiple interviewees shared the opinion that moving GTRs from most program offices to TAD slowed the TA administration process, as other program offices no longer have immediate access to people who approve and oversee the TA.18

Interviewees also mentioned that having GTRs placed outside the program office administering the TA can prolong the TA process because the GTRs may lack technical expertise in the program for which they are approving work plans.

CPD field staff believe changes since Community Compass have increased the time it takes to approve requests. A field staff member explained that before Community Compass, TA was managed by field staff. Requests came directly to their office, they would assign the provider, and they would issue work plan approvals more quickly.

“We actually managed the money that was awarded to each TA provider from our office. The process was a lot faster. Once it was moved to headquarters, the process slowed down a lot. We were able to provide TA a lot faster. Then we were able to be more involved in it because we would be the ones that assigned the TA provider to work with the recipient.”

—SNAPS HUD field staff

Period of Performance and Length of Technical Assistance Engagements

At least one-half of all TA work plans are completed in less than a year, including the delivery of direct TA and the development of universal TA tools and products. The median number of days from the beginning to the end of a period of performance for all work plans is 313 days (not shown).19 For those 18 The Office of Procurement Contract Services oversees all PHA Receivership and ONAP TA awards.

19 According to administrative data, the period of performance began as early as 43 days before the request date for TA that originates from a request. The research team was not able to resolve this data entry issue with HUD.

The performance period start date is used here.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5 3

TA engagements specifically initiated by requests through the TA Portal (37 percent of all work plans), which are mostly for direct and on-call TA, the performance period is slightly shorter (median of 273

days; exhibit 17). This difference could be due to multiyear work plans for some universal TA engagements not initiated via request, such as the Lead the Way curriculum or the HUD Exchange online platform, although work plan content was not analyzed to confirm this theory.

Customers can work their way through available universal TA tools and products quickly, but TA engagements involving targeted TA often take longer, from developing work plans to completing TA delivery. Based on interviews, we found that targeted TA engagements tend to take the longest, lasting from a few weeks to a few months, depending on the severity of a customer’s problem. The difference in the length of time needed to complete targeted TA compared with universal TA may be that universal TA serves a wider audience and provides less customized TA to reach more people.

According to some providers, because work plans for universal TA have already been developed, they can be delivered more quickly and require less communication with HUD.

Types of Technical Assistance and Costs over Time

Direct TA accounts for most of HUD’s TA costs documented in the TA Portal but represents a decreasing share because other types of TA, including tools and products and self-directed and group learning, have increased since the beginning of Community Compass. Adjusted to 2018 dollars, direct TA decreased from $20.3 million out of HUD’s total TA budget of $29.1 million in FY 2013 to $17.2

million in FY 2014, then increased gradually to account for $21.1 million out of HUD’s total TA budget of $37.6 million in FY 2017. Self-directed and group learning increased from $3.6 million in FY 2013 to $5.5 million in FY 2017. Needs assessments declined dramatically from $1.2 million to just over $290,000 from FY 2013 to FY 2017. On-call TA fluctuated from around $800,000 in FY 2013 to peak at $2.5 million in FY 2015 before dropping to $2.1 million in FY 2017. Exhibit 18 shows the budget share of TA types from 2013 to 2017.

5 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 18: TECHNICAL ASSISTANCE PORTAL ESTIMATED BUDGET BY TECHNICAL ASSISTANCE TYPE, FY

2013–FY 2017

TA = technical assistance.

Notes: The universe for this figure is all work plans in the TA Portal. Dollar amounts have been inflation adjusted to 2018

dollars. For work plans that span more than 1 year, the estimated budget is spread equally over all fiscal years of their period of performance. All years shown are fiscal years. Labels in parentheses are the alternative TA type labels used within the TA Portal system.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

Direct TA and tools and products work plans cost the most of all types of TA, whereas self-directed and group learning work plans are the least expensive. A breakdown of the work plan and budget amounts by TA type for 2017 indicates a higher proportion of the TA budget (56 percent) went to direct TA compared with the numbers of direct TA work plans in the TA Portal (52 percent) (exhibit 19). Tools and products also took a larger share of the 2017 TA budget (24 percent) compared with the percentage of work plans in the system (19 percent). Self-directed and group learning is the only TA type with significantly more work plans (21 percent) than its share of the TA budget in 2017

(13 percent). The apparent decline of needs assessment work plans and costs within the TA Portal may not accurately represent the level of actual needs assessment TA being administered due to qualitative study findings that needs assessments are often being performed during on-call or direct TA engagement as opposed to separately. This practice may also inflate the costs associated with direct TA work plans. Because it is unclear from the data how many customers each work plan ultimately delivers TA to, particularly how many customers eventually access the products and training that may A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5 5

have been developed under a single work plan, a cost per customer for each TA type cannot be established.

EXHIBIT 19: PERCENTAGE OF HUD TECHNICAL ASSISTANCE ESTIMATED BUDGET BY TECHNICAL

ASSISTANCE TYPE VERSUS WORK PLANS BY TECHNICAL ASSISTANCE TYPE, FY 2017

TA = technical assistance.

Notes: The universe for this figure is all work plans in the TA Portal in FY 2017. Dollar amounts have been inflation adjusted to 2018 dollars. For work plans that span more than 1 year, the estimated budget is spread equally over all fiscal years of their period of performance. Work plan counts by TA type are approximate. The research team could not account for duplicated work plans or link multiple work plans that were part of the same TA engagement. Labels in parentheses are the TA type labels used within the TA Portal system.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

SNAPS TA represents around 40 percent of HUD’s overall estimated TA budget and has a similar distribution of TA types over time to HUD’s overall TA budget (not shown). In FY 2017, the SNAPS TA budget ($15.1 million) represented about 40 percent of HUD’s total TA budget ($37.6 million). Tools and products increased gradually between FY 2013 and FY 2017, whereas direct TA increased markedly, from $3.0 million in FY 2013 to $8.7 million in FY 2017. Self-directed and group learning and on-call TA also increased steadily between FY 2013 and FY 2017 but at a slower rate. Needs assessments, as a distinct TA type, declined from $179,000 to $30,000, although, as noted, the 5 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

assessments often may be included in on-call or direct TA. SNAPS staff, in particular, pointed out that the shift to funding more tools and products and on-call TA was partially due to the expense of administering direct TA and the increasing reliance on universal TA.

PHAs account for almost one-half of the direct TA budget, a significant increase since Community Compass was launched. According to TA Portal data on costs by TA type, nearly all TA provided to PHAs—including troubled, nontroubled, and those exempt from scoring—was direct TA. Total direct TA to all PHAs represented about 43 percent ($9 million; not shown) of the $21 million spent on all direct TA by HUD in FY 2017 (exhibit 19). This amount reflects an increase in both absolute and relative terms, as only about $2 million expended on direct TA in FY 2014 went to PHAs (not shown), or about 12 percent of the $17 million spent that year on direct TA (exhibit 19). Given the needs of PHAs designated as troubled to specifically improve their PHAS and SEMAP scores, the use of targeted TA, which can be customized to their organizational needs, may be most appropriate; however, data do not reflect when PHAs access universal types of TA, such as online training—for example, Lead the Way.20

20 Lead the Way, an online training on financial management available to PHA board members, is represented as a work plan under product development in the TA Portal. Data on customer access is stored separately.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5 7

VI. Communication, Coordination,

and Collaboration

Introduction

In addition to highlighting both the standardized and more nuanced steps within the implementation process of HUD TA, this assessment looked across programs at how the TA process works from the perspective of multiple stakeholders—HUD program staff, providers, and customers—and how it compares with how HUD provided TA in the past, as applicable. Findings draw upon interviews and case studies to address the following questions:

•

How does the TA process work from the perspective of HUD Community Compass program staff? Of TA providers? Of TA customers? How does it compare with how HUD provided TA in the past?

•

Has the Community Compass process led to the use of standardized TA materials and processes across TA providers? If not, why not? What can TA providers or HUD do to reduce duplication of effort or lack of standardization?

•

Has Community Compass facilitated better coordination, collaboration, and communication across HUD offices and programs and TA providers?

Key Takeaways

Cross-Program Col aboration at HUD

◼

There is little evidence from interviews of increased cross-program collaboration among HUD

headquarters’ program offices, although the NOFA process changed and many TA processes were centralized within CPD.

◼

Many HUD program office staff think the complexity of HUD program-specific TA structures limits the ability of HUD staff to collaborate across offices, although HUD TA is moving slowly toward increased efficiency and collaboration.

5 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Shifting Relationship and Roles: HUD Field Offices

◼

Since the shift to Community Compass, some field offices have maintained a close connection with TA processes, whereas others feel more disconnected than before.

◼

PIH field office staff who oversee PHAs seem to have a closer relationship with HUD

headquarters staff and TA providers and a deeper engagement in the TA process than other field office staff.

◼

Community Compass reduced some field offices’ roles—including SNAPS within CPD—by centralizing decision making and funding in HUD headquarters program offices, leaving field staff feeling disconnected. CPD field staff expressed concern that customer needs may be ineffectively met without the field staff providing more direct input into the TA process.

Communication Between Technical Assistance Providers and HUD

◼

Communication between HUD staff and providers occurs regularly.

◼

TA providers find HUD staff to be supportive across program offices and TA programs.

Coordination and Collaboration Across Technical Assistance Providers

◼

Provider coordination is encouraged under Community Compass, but evidence of improved coordination and collaboration across TA providers was uneven and varied by program office.

◼

The SNAPS office prioritizes coordination across providers by bringing them together to help design TA around program priorities, share topical knowledge, and assign providers to individual TA engagements based on regional needs. SNAPS TA focuses on priority initiatives, which target specific populations experiencing homelessness, and promotes greater collaboration among providers and sharing of TA materials.

◼

Troubled PHA TA providers collaborate when working in the same region, with the same client, or on the same product when asked by HUD to do so.

◼

The proprietary nature of the work and information asymmetries are challenges to provider collaboration.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

5 9

Cross-Program Collaboration at HUD

Several shifts in internal processes at HUD were initiated to encourage greater communication, coordination, and collaboration across HUD TA offices and divisions. Although the NOFA process changed and many TA processes were centralized within CPD, there is not much evidence of increased cross-program collaboration with HUD headquarters’ program offices. Among program staff, those whose offices collaborated before Community Compass kept doing so, but the rest noted no increase in collaboration.

The NOFA process was centralized under Community Compass so that program offices would work together to allocate funds; however, once funds are allocated, HUD programs often work independently and administer their TA programs differently. For the NOFA process, program offices pool funds and jointly select TA providers for cooperative agreements spanning multiple HUD offices and programs. Several HUD program staff viewed the new NOFA process positively, but others had a more neutral opinion. Regardless, after program offices divide the funds and award the cooperative agreements, the TA process reverts to being segmented along program lines.

Many HUD staff interviewed noticed no significant change in cross-program collaboration under Community Compass. Some TA providers also noted a lack of increased cross-program collaboration, despite the goals of the unified NOFA process.

“[HUD’s] goal is to be holistic...to braid together all these funding sources to create a comprehensive approach to technical assistance in communities. I have not seen a single engagement where that has actual y happened in practice.”

—TA Provider

There are multiple reasons why more substantial and sustained collaboration has not occurred between program offices throughout the TA process. First, although some HUD program office staff believe that HUD TA is moving slowly toward greater efficiency and collaboration, many discussed how the complexity of HUD program-specific TA structures limits the ability of HUD staff to collaborate across offices. Each office has its own unique procedures for its TA, so engaging with other offices means adding more steps to an already multifaceted process.

6 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Second, other program offices’ centralization of some administrative tasks within CPD is viewed as weakening cross-program collaboration. As one HUD headquarters staff said, although all programs now use the same NOFA to solicit TA providers, program offices still know “how much [funding] is available for CPD, so we’re still kind of in our [silo]… even though we’re all in this big NOFA together.”

One reason for the perceived siloes between programs is the centralization of GTRs within TAD in CPD. Several headquarters staff claimed that this made it more difficult to work closely with other program offices when so much administrative responsibility—including approving work plans—is concentrated in CPD.

Shifting Relationship and Roles: HUD Field Offices

Before Community Compass, a significant portion of TA management was done by HUD field offices.

Since Community Compass began, field offices have experienced some changes. Field offices are more closely connected to the HUD funding recipients in their geographic region, and some of them directly contracted with TA providers to support these funding recipients and managed all funding and reporting. Some field offices have maintained a close connection with TA processes, but others feel more disconnected than before.

PIH field office staff who oversee PHAs seem to have a closer relationship with HUD

headquarters staff and TA providers and a deeper engagement in the TA process than other field office staff. According to one field office, about 90 percent of the PIH TA is initiated or submitted by the field office, and about 10 percent is initiated directly by HUD headquarters. Although HUD

headquarters has to accept the request and assign the provider, the PIH field office generally prepares the scope of work, helps providers adjust their budget and expectations regarding the TA engagement, and coordinates conversations between the provider and PHA. There are regular calls between the field office and GTMs at HUD headquarters for progress updates on the TA throughout the TA process. Some field offices have monthly or quarterly calls with the TA provider, but this communication process can be less formal and vary depending on the particular needs of the TA engagement.

Field offices can serve as important advocates for the TA needs of their HUD funding recipients.

As one PHA stated, “The field office was our advocate because they’re on the ground, and they know what’s needed….The HUD field office is a tremendous asset to HUD and the housing authority.” On the other hand, one PHA that did not feel fully supported by their field office noted that plans to work A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6 1

through their challenges were all developed “in house,” and they were not made aware of other TA opportunities available through HUD headquarters.

Community Compass reduced some field offices’ roles—including SNAPS within CPD—by centralizing decisions and funding within HUD headquarters program offices, leaving field staff feeling disconnected. According to HUD headquarters staff familiar with SNAPS TA, Community Compass implementation targeted improving the efficiency of SNAPS TA delivery. Headquarters staff now manage all TA requests, select and assign providers to work, and approve and monitor work plans.

Field office staff were to remain involved in the TA process through routine updates on the status of TA engagements. Quarterly TA reports are distributed from the headquarters office to the CPD Field Office Directors, including updates on all TA engagements in that region; however, the dissemination of the quarterly reports to staff within each field office varies. This variation leads to inconsistent field office involvement and has caused field staff to feel disconnected from the TA process.

“Just overall, I wish there was more involvement or at least more information provided to the field about what TA is happening, when requests come in, when they’re approved, what kind of TA is being provided. If possible, it would be great if we could be somewhat involved every once in a while in each TA request with one of our funding recipients, even if it’s just a cal at the start of the TA and at the end of the TA to fil us in on what the goals are and what was achieved and get our feedback.”

—CPD field staff

Unclear communications between HUD headquarters, field staff, and TA providers create some inefficiencies, including inadequate HUD funding recipient monitoring and poorly informed or duplicate TA efforts. Several CPD field staff said they are not notified when their CoCs or other funding recipients apply for or receive TA, hindering their ability to fulfill their funding recipient oversight responsibilities. CPD field staff also expressed concern that funding recipient needs may be ineffectively met without the field staff providing more direct input into the TA process. As one field staff member described, “I think [how] the system is set up right now—where recipients can entirely bypass the field by directly requesting TA and the field just has no idea that it’s even going on, what 6 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

the topic is, what assistance is provided, [that] what they might really need that is just being overlooked.” TA providers have given CoCs incorrect information about their grant requirements and management in at least one instance. Field staff, who were unaware of the guidance from TA providers, could not correct them in a timely manner. In another instance, several CoCs within a single field office’s jurisdiction were receiving TA for similar reasons. CPD field staff believed these were separate engagements rather than coordinated ones and lamented that HUD could have used its resources more efficiently by administering TA to several CoCs at once.

Communication Between Providers and HUD

Clear communication between TA providers and HUD is critical for ensuring that TA is delivered smoothly and promptly. In general, TA providers were satisfied with their level of communication with HUD, noting clear lines of communication and helpful input from HUD staff on the TA content. For example, some providers mentioned that when a lag in HUD approval of a work plan or plan amendment threatens to hinder the workflow of a TA engagement, they have been able to communicate with HUD staff through emails or biweekly phone conversations to keep engagements on track.

Communication between HUD headquarters staff and providers occurs regularly. Communication with HUD headquarters staff generally occurs with one of three job roles and depends mostly on the responsibilities of each: the assigned GTRs, who are centralized in TAD within CPD and whose responsibilities consist of administrative work (e.g., voucher and work plan approval); the GTMs, located in program offices and whose duties are more programmatic through monitoring and assisting throughout the TA process; and program office TA coordinators, whose work involves coordinating TA activities specific to the HUD program office through subject matter experts and task support and determining hours and degree of effort needed for a work plan. Depending on the project, communication among providers, GTRs, and GTMs involve biweekly or monthly calls. Program office technical assistance coordinators participate in these conversations depending on the level of engagement they desire and the need for subject matter expert input. The frequency, type of communication (i.e., phone, email, in-person), and degree of communication can vary for each TA assignment and across TA programs. One provider, for example, indicated that in SNAPS, there is significant communication with the program office TA coordinator and less with the GTR. In ONAP, coordination requires more engagement with the regional office than with HUD headquarters staff.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6 3

TA providers for troubled PHAs noted that communication with HUD headquarters and field staff has been great. TA providers find HUD staff to be supportive across program offices and TA programs.

They have many opportunities to coordinate with HUD, scope out the TA work plan, and get questions answered, and they find HUD staff supportive and responsive when providers raise issues and questions. Communication between SNAPS TA providers and HUD headquarters staff on initiative-based TA is much more frequent than on other TA engagements. Several SNAPS TA providers described biweekly calls with their primary point of contact at HUD headquarters. One provider explained that between calls and emails about various SNAPS initiatives and working groups she is a part of, she communicates with HUD almost every day.

“I think SNAPS does a fairly good job of bringing together the [TA provider] firm leads and initiative leads…. We have a half-hour standing meeting every other week.

We have a running agenda on all of the big efforts that are happening, and that cal is an opportunity to hear and ask questions about specific things that are happening.”

—SNAPS TA Provider

Coordination and Collaboration Across Technical

Assistance Providers

A possible outcome of the reorganization of TA under Community Compass implementation is that TA providers could be brought together to provide coordinated input into TA program design, share materials across providers and TA engagements for greater consistency, and collaborate more on actual TA implementation. This assessment found some evidence of strengthened coordination and collaboration across TA providers, but it varies by program office. TA providers who participate in HUD groups to promote collaboration have developed relationships and additional mechanisms for sharing information; however, challenges remain in creating a culture of collaboration among TA providers that compete with each other for HUD funding and develop proprietary materials.

The SNAPS office prioritizes coordination across providers by bringing them together to help design TA around program priorities, share topical knowledge, and assign providers to individual TA 6 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

engagements on the basis of regional needs. SNAPS uses several different structures to engage providers in these activities: “brain trusts,” working groups, and regional teams.

“Brain trusts.” SNAPS convenes “brain trusts,” or a group of TA providers with relevant subject matter expertise, after the program office sets its priorities but before it designs its program initiatives (exhibit 20). These convenings may take the form of a single meeting or several conversations.

According to SNAPS headquarters staff, the interactions have led to concrete ideas on implementing a plan or achieving a particular policy objective and building respect among providers. Providers can interact in a closed setting that allows them to share subject matter expertise and ideas on implementing SNAPS directives. The group setting differs from most TA engagements, which involve a single provider.

Working groups. SNAPS’ topically focused working groups also offer TA providers space to learn best practices from each other and build relationships. These groups bring together invited TA providers across different firms to advise on data and specific initiatives and discuss challenges through quarterly meetings. As one particularly effective example, during interviews, the research team learned about the data working group. A provider said that the group is an example of successful collaboration and coordination across firms because the group members had been working together for several years before the group’s official convening. They came together already knowing each other’s work styles, strengths, and weaknesses. Members felt comfortable relying on each other to tackle issues together.

Regional teams. In 2019, SNAPS headquarters staff focused on strengthening their region-based teams of TA providers by working with these teams to facilitate the administration of on-call TA. TA requests are directed to a regional team by a SNAPS desk officer. The team lead, a provider, recommends that HUD assign a TA request to a provider on the team based on relevant subject matter expertise and contextual knowledge about a community or organization. This structure for recommendations for assigning TA engagements is dependent on communication and coordination among providers in each team.

In addition, the focus of SNAPS TA on priority initiatives promotes greater collaboration among providers and sharing of TA materials. One example is TA provided for the Encampments and Unsheltered Homelessness Initiative, which aims to decrease the number of people living in encampments or experiencing unsheltered homelessness. SNAPS offered TA to 10 communities across the United States that experienced spikes in unsheltered homelessness and encampment habitation rates. Participating providers conduct monthly calls that include providers and consultants A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6 5

who do not work on this initiative but have experience with the subject matter. One of the initiative providers talked about providers’ use of a centralized repository of TA resources that other providers can access using Google Drive for use across customers. The repository helps standardize TA administration across communities and allows providers to influence future TA. Providers may edit resources that are found to be ineffective. This open access to resources helps providers and HUD

improve TA offerings for current and future funding recipients.

EXHIBIT 20: ADDING TECHNICAL ASSISTANCE FOR PRIORITY INITIATIVES

The SNAPS office develops priority initiatives and expanded their TA offerings to support the implementation of these initiatives by funding recipients to supplement the demand-response TA still available. Initiatives typically target specific populations and subpopulations of people experiencing homelessness—such as unsheltered homelessness, youth homelessness, or veteran homelessness—and may accompany grant funds that SNAPS releases to help communities address these specific problems. Two of the most frequently discussed initiatives during interviews with SNAPS staff were the Vets@Home and the Youth Homelessness Demonstration Program (described below). Initiative-based TA is developed to accompany these predefined strategies and is intended to be proactive, although organizations and communities might apply for it only if they recognize the topic of the initiative as a problem they face or a priority of theirs.

Vets@Home. In 2010, Opening Doors, “the nation’s first comprehensive federal strategy to prevent and end homelessness,” elevated ending veteran homelessness to a U.S. Interagency Council on Homelessness goal. To accomplish this goal, SNAPS launched the HUD/VA Priority Communities TA initiative and Zero: 2016, two initiative-based TA programs to ameliorate veteran homelessness.

Continuing this effort, SNAPS offers Vets@Home TA, associated with the Vets@Home initiative, to every community that applies for it.

SNAPS Youth Homelessness Demonstration Program. This initiative aims to reduce and prevent youth homelessness. Communities can apply for funding under YHDP and, regardless of whether they received it, are eligible for on-call TA under the initiative. As evidenced by the review of eight work plans extracted from the TA Portal, which are similar in scope, TA content is largely consistent across plans, although customer contexts are unique. YHDP has a well-defined feedback loop that communities and providers can use to report on their experiences receiving or administering TA. Each month, a program specialist speaks to customers about their experiences receiving TA and debriefs providers separately about their progress administering TA in customer communities.

TA providers for troubled PHAs collaborate when working in the same region, with the same client, or on the same product when asked by HUD to do so. Providers collaborate when there are multiple provider firms operating in the same geographic area or with the same client. For example, while working with Puerto Rico after Hurricane Maria, a provider was charged with holding a regular call with all TA providers working there to ensure that information was communicated effectively and 6 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

providers were not duplicating work and to identify places where competing efforts might impede each other’s work. This practice has also been important for TA providers working with a PHA attempting to exit receivership.

When developing the Lead the Way training, HUD compelled collaboration among TA providers by assigning multiple providers to the project, representing what one provider described as HUD’s

“preference to spread the work.” One provider was awarded the project initially to develop the training, although HUD headquarters staff required the provider to subcontract with another provider that owned a proprietary Learning Management System (LMS). Later, to develop the Sexual Harassment training module, one provider was assigned to lead tools and products. Another provider was tasked with developing the curriculum and modules for the training.

There are some challenges and inefficiencies to provider collaboration, including the proprietary nature of the work and information asymmetries. Some TA providers explained that they are hesitant to collaborate because they want to preserve their intellectual property, which gives them a competitive edge in funding bids. This practice means that if a cooperative agreement stipulates that only one provider is in charge of a particular task, that same provider also would be responsible for updating the proprietary system or content over time. Were another provider assigned to update materials, they would have to start from scratch and recreate all existing systems and content that they did not originally produce.

Provider interviewees said that although a division of labor across multiple providers can create efficiencies resulting from specialization, it also can create asymmetric information issues across providers. One provider explained that HUD communicates with the prime on a project, who then communicates information to subcontractors. The subcontractors then have few opportunities to ask questions or otherwise communicate directly with HUD, so they may not receive clear feedback on their work or have the opportunity to formulate questions for the end users or TA customers to answer for measuring the outcomes of the TA provided. Providers acknowledged, however, that the degree of communication depends heavily on the HUD point of contact; some HUD staff members were said to be more inclusive than others.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6 7

VII. Monitoring and Evaluation

Introduction

With the implementation of Community Compass, HUD developed mechanisms to facilitate the oversight and evaluation of the TA process and outcomes. This chapter summarizes how HUD collects data to monitor the TA process across TA programs. It then discusses HUD’s evaluation efforts, including developing outcomes measures and standard processes for collecting feedback on TA engagements.

The findings from this chapter are drawn from the in-depth interviews with key stakeholders and the two case studies. Core research questions addressed in this chapter are—

◼

How effective are HUD data and reporting systems for TA providers, TA customers, and HUD

staff in facilitating the TA process?

◼

How does HUD obtain regular feedback from its providers and customers about their delivery and the impact of HUD TA?

◼

How does HUD evaluate the impact of TA on customers’ financial, managerial, physical property, and governance practices?

Key Takeaways

Reporting Processes

◼

The data and reporting processes for TA engagements are effective because they provide a single place to track all TA requests and engagements in the TA Portal, create a standard process for customers to request TA and for providers to get assigned requests using the TA Portal, and simplify the process for providers to submit monthly activity reports and invoices in the Disaster Recovery Grant Reporting System.

◼

Some providers use additional reporting systems at the request of HUD staff, sometimes requiring duplicative information.

6 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

Improvements are needed to the data and DRGR and TA Portal reporting systems to enable linking of data across systems, reduce the time it takes to enter reports, and ensure that entries are saved correctly.

HUD Oversight

◼

Standardized reporting mechanisms assist HUD staff with their oversight activities, but limited access to the reporting systems makes it more difficult to monitor progress on a TA engagement over time, leading some HUD staff to create their own methods for tracking TA.

◼

Field office staff expressed that their ability to monitor TA and customers’ progress was limited by a lack of access to both data systems and training on using them.

Evaluation

◼

The standardized data systems created under Community Compass, containing all work plans across program offices, could be useful for evaluating outputs and outcomes associated with HUD TA if challenges with data quality and completeness were addressed.

◼

There are ongoing debates with HUD’s program offices around whether the impact of TA should be measured at the level of the individual TA customer or at a broader systems level.

◼

TA providers and HUD staff agree that the outputs and outcomes tracked are too broad and short term to be useful for evaluating TA engagements.

◼

In the absence of established metrics for measuring outcomes and impacts, HUD staff and TA providers have relied on informal methods for gathering feedback by conducting internal program assessments and hosting calls with customers and providers.

◼

HUD implemented new policies in 2018 for evaluating TA effectiveness, including administration of a standard survey to all TA customers who receive direct TA or participate in in-person training.

Reporting Processes

HUD collects data on TA via two data systems—DRGR and the TA Portal—to track five components of the TA process: TA requests, TA customers, TA providers, TA work plans, and tasks within those work A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

6 9

plans. TA providers and HUD staff use both systems; the DRGR system is used to report and track financial and quarterly activities, and the TA Portal is used for management tasks.

DRGR is used for financial management and oversight of the work, including submitting and receiving HUD approvals on proposed costs, followed by submitting invoices and authorizing payment for completed work. TA providers use the DRGR system to obtain HUD approval on the following: staff rates, invoices, and individual work plans, including budget line items, staff hours, and tasks. HUD

staff individually approve line-item vouchers in DRGR, which authorizes payment to the TA provider for the technical assistance delivered. For monthly reporting, TA providers are required to submit a narrative by TA activity—detailing the number of hours spent, personnel involved, and tasks completed—and an invoice to DRGR as a monthly voucher. Once a quarter, providers are required to submit progress reports that include financial summaries of their quarterly spending.

Management task reporting in the TA Portal focuses on the process of the TA engagement, from request or initiation to closeout. The TA Portal was developed to create a single online platform for assigning and monitoring TA work. Through the TA Portal, HUD receives and reviews or initiates requests for TA, assigns TA requests to TA providers, reviews and approves work plans, and oversees TA engagements. TA providers access the system to receive assignments, submit and revise work plans, and report on work progress. Work plans in the TA Portal include goals, outputs, outcomes, and processes for the TA engagement, and TA providers are required to report their monthly progress on work plan activities throughout the engagement. When a work plan is completed, TA providers submit a final narrative of the overall work plan as a part of their closeout process.

Based on interview findings, most HUD staff and TA providers find value in the standardized reporting processes created by Community Compass on TA engagements via the TA Portal. The TA Portal provides a single place to track all TA requests and engagements. In particular, the creation of the TA Portal allowed HUD to institute a standard process for customers to request TA and for providers to get assigned requests. HUD also made improvements to the DRGR system, which streamlined the process for providers to submit monthly activity reports and invoices.

Some TA engagements—in particular, those with multiple customers—require documentation to supplement the standard monthly reports, sometimes requiring duplicative information.21 Providers mentioned that this documentation could be related to outcomes or to qualitative information about a 21 These qualitative findings are based on interviews with TA providers. The research team did not quantitatively compare estimated time and cost burdens submitted under the Paperwork Reduction Act to real time burdens, as it was beyond the scope of this evaluation.

7 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

TA customer’s progress. For example, one provider spoke about their experience with one HUD TA initiative. The HUD program office created an additional spreadsheet that had to be submitted to HUD regularly, which included qualitative data from their DRGR monthly reports and information about the TA customer’s progress toward the specific goals of the initiative.

HUD staff and TA providers also reported that significant improvements are needed across the reporting systems to reduce duplication in reporting and enhance system functionality. First, because there is no way to effectively link data between the two systems, some information for a single work plan must be entered in both the DRGR and the TA Portal. This duplication for both HUD staff and TA providers requires additional time and staff capacity to properly complete their reporting duties, which can cause delays in the TA process and missing or conflicting information between the two systems.

Smaller TA providers, in particular, reported that the reporting process is challenging because they have fewer staff members and more limited capacity; larger TA providers can dedicate staff solely to reporting. The lack of an effective link across data systems also makes it difficult for HUD staff and TA providers to report on a single engagement or summarize TA work plan trends. Second, interviews discovered that the functionality of the two systems could make them difficult to use. In particular, providers noted that the TA Portal tended to run slowly and that reporting in both the TA Portal and DRGR must be completed in one sitting because draft work plans and reporting did not always save correctly.

HUD Oversight

HUD is responsible for overseeing the provision of TA, both through the Technical Assistance Division within CPD and individual program offices. TAD GTRs monitor the progress of all TA programs through financial and quarterly activities reported through the DRGR, including monthly spending on TA activities and invoice approvals. Each program office also has individual processes for monitoring a TA engagement, including approving requests for TA, approving work plans, and reviewing monthly reporting.

Standardized reporting mechanisms created by Community Compass assist HUD staff with their oversight activities. Having the ability through the TA Portal to track all TA engagements allows HUD

staff to have a better understanding of the universe of TA their program office is offering and to track the progress of individual engagements more easily. Some of the previously discussed challenges with the reporting systems, however, limit HUD’s ability to fully capitalize on all data entered in both DRGR

and TA Portal systems.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7 1

Across HUD staff, the research team learned that access to DRGR and the TA Portal is not universal, which makes it difficult for staff to track TA engagements or report on specific work plans.

Limited access to the reporting systems makes it more difficult to monitor progress on a TA engagement over time, leading some HUD staff to create their own methods of tracking TA. Some HUD staff reported not having access to one or both reporting systems. Others mentioned having access only to view a TA engagement but not to modify or add notes to the record, which meant, for some work plans, only a single HUD staff member had access to update the engagement. Differential access often compounds the challenges staff experience with the data systems. The team heard from some staff that they could not fill in missing data because they did not have access to edit the work plan; other staff mentioned that they could update only missing or incorrect data in one system, usually the TA Portal. Some HUD staff have created their own tracking methods outside the standard systems to deal with these challenges, such as keeping an updated Excel spreadsheet of all TA engagements they need to monitor.

In interviews, field staff often expressed that their ability to monitor TA and customers’ progress was compromised by a lack of access to both data systems and training on using them. Often only one HUD staff member in a given field office had access to the TA Portal, and none of the staff interviewed had access to DRGR. For both the PIH and CPD field staff, limited access to the TA Portal hindered their ability to monitor funding recipients’ regular duties. For example, field staff are actively involved in the TA request and work plan development process for troubled PHAs, but they may be unable to view the work plan or make any notes about progress throughout the engagement. In addition, field staff who did have access to the TA Portal often felt as though they did not use the system as often as they should for monitoring purposes because they had never received training and did not know how to pull reports or find specific information. It should be noted that the issues with limited access to data systems and training could be due to a reduced role of field offices in TA provision since the implementation of Community Compass.

Evaluation

Our interviews with HUD headquarters staff in TAD and various program offices revealed a desire to better evaluate the effectiveness of HUD TA. The information desired ranged from whether the TA provided met the need of those who requested it—including examining the intent of a specific engagement, whether the final deliverable met that intent and the needs of the community, and how 7 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

can it be improved—to the overall performance of TA providers. Staff also expressed an interest in learning more about how HUD TA affects specific customers and communities more broadly.

Until recently, HUD did not require program offices or TA providers to measure the impact of TA and did not guide how TA performance should be measured. HUD did not use the implementation of Community Compass to establish standard evaluation methods for program offices or a formal method for gathering feedback across all HUD TA programs. As a result, until 2018, most of the evaluation efforts for HUD TA were determined by each program office. Some program offices encourage TA providers to create metrics to track TA customers’ progress, whereas others rely on informal feedback from TA customers and TA providers to assess their TA. The GAO Report on HUD Management (2002) recommended that HUD ask program offices to establish performance measures and report on objectives and intended outputs and outcomes. Efforts to standardize the evaluation of HUD TA across program offices are currently underway.

The standardized data systems created under Community Compass, despite some existing challenges, could be useful for evaluation purposes in the future because they contain all work plans across program offices. For example, providers pointed to the output and outcomes fields in the TA Portal as a potentially useful data point to evaluate the impact of TA. Discussions with HUD staff and TA providers, however, as well as analysis of the TA Portal and DRGR data, highlighted challenges in data and reporting systems that could hinder HUD’s ability to properly analyze the data and evaluate TA. Specifically, as previously discussed, reports can be difficult to pull and do not pull together data across the two systems. Reports would also be missing any additional information being tracked outside the data systems by TA providers or HUD staff. In addition, the current fields in the TA Portal are not customizable by program office, which makes tracking metrics difficult, and does not allow TA providers or TA customers to submit feedback on the TA or upload the feedback they have collected.

Choosing the Right Metrics

Interviews with both TAD and individual program offices revealed ongoing debates around whether the impact of TA should be measured at the level of the individual TA customer or a broader systems level. Interviews with SNAPS headquarters staff highlight the office’s challenges in determining the appropriate evaluation measures for SNAPS TA. The research team learned that HUD headquarters staff aligned the overarching goals of SNAPS TA with the office’s theory of change, which targets reducing or eliminating homelessness through initiative-based TA focused on changing the system of homeless housing and service delivery. Some staff expressed that, in addition to evaluating SNAPS TA A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7 3

using customer-specific outcomes such as satisfaction with the TA or whether there was an increase in customers’ knowledge of reporting requirements, evaluation of TA should include measures that are aligned with the SNAPS mission, such as whether the TA had a positive impact on reducing homelessness in a specific community. Staff expressed concerns that metrics aligned with the SNAPS

mission would be harder to quantify than whether a customer was satisfied or if their use of HUD data systems had improved. They also expressed hesitation in implementing community-level measures because of the complications of measuring impact across communities receiving TA, especially for communities with more acute challenges.

Since April 2018, after a 2-year pilot period, TA providers have been required to include outputs and outcomes in all work plans for the TA they deliver, but providers and HUD staff agree that these outputs and outcomes are too short term and too broad to be used to evaluate the impacts of specific TA engagements. Outcomes tend to focus on customer progress through the end of a TA engagement, as opposed to long-term progress or the effect of the TA. For example, evaluating a TA engagement focused on developing an organization’s hiring strategy at the end of the engagement would not cover the implementation of the strategy itself. In addition, the 13 standardized HUD outcomes currently selectable in work plans from a drop-down menu—an improvement from the previous open text field allowing nonstandardized entries—tend to be vague and do not provide specific metrics as evidence of meeting their goal. For example, an outcome might read, “Improved capacity to gather and use resident and community input” or “Improved coordination and alignment with other community or regional plans.” Without specific metrics on key concepts—such as capacity, coordination, and alignment—along with baseline data and clear definitions of improvement from that baseline, it is difficult to quantify what it means for an individual customer or a TA work plan to meet that outcome.

Universal TA types may be able to track metrics more easily through automated systems, but choosing the right metrics can still be difficult. For example, the Lead the Way training uses customer access through HUD Exchange to generate quarterly usage reports to HUD. The metrics track enrollment, types of curriculum accessed, training completions, and customer feedback provided and can also be broken down by PHA, state, and region. This reporting system, however, has some limitations, including that data are not available on all training modules, on those who started but did not complete the training, or on results from a series of “knowledge checks” surveys given to customers throughout the training to measure retention of the learned information.22

22 There were discussions between HUD and the TA provider developing Lead the Way on whether responses from the “knowledge checks” should be tracked; however, they decided that these checks should be for the participant to gauge their progress and should not be used as performance metrics for the training.

7 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Evaluating Outcomes: Troubled Public Housing Agency Technical Assistance Across a sample of TA work plans identified and reviewed for troubled PHAs, work plan deliverables were largely based on completing a TA task—such as completing an implementation plan with a PHA, administering a training, or completing an assessment. When a TA engagement focused on updating a PHA’s policies and procedures, for example, the provider was required to submit the final updated policies and procedures to HUD. Targeted outcomes were usually general and unmeasurable statements about improving the staff’s capacity in a certain domain, such as financial management, operations management, or voucher administration. Only four sample work plans of the more than 50

reviewed had a deliverable focused more broadly on measurable outcomes: three work plans focused on the relocation of a set number of public housing residents into new housing, and one work plan focused on improving management by increasing unit occupancy to 96 percent or higher.

PHAs who have a troubled status must participate in the TA and activities they agreed upon in their recovery agreement and action plan and show improvements in their PHAS and SEMAP scores.

The providers and customers interviewed who are tracking TA impact for their own information outside HUD TA systems predominately measure factors that affect the PHAS and SEMAP scores.

Metrics mentioned by providers and customers include occupancy rates of public housing units, number of units newly made available after a long-term modification, number of days to close out a maintenance work order, and whether rent was collected at the right time. One provider said that progress on these metrics is critical in deciding whether to close out a TA engagement, in the case of measured improvements, or submitting updated recommendations on how to address the PHAs’

needs to the field office.

Collecting Informal Feedback

In the absence of established metrics for measuring outcomes and impacts, HUD staff and TA providers have relied on informal methods for gathering feedback by conducting internal program assessments and hosting calls with customers and providers. A few HUD programs—such as RAD and HOME—implemented internal processes for conducting assessments of the delivery of their program-specific TA. Other program offices, such as SNAPS, create feedback mechanisms for specific initiatives. For example, for initiative-based TA, the SNAPS headquarters staff for the Youth Homelessness Demonstration Program holds group calls with TA customers to collect feedback on how the TA is progressing, what could be improved in the ongoing engagements, and how the TA is helping them move their work forward. These calls happen every 6 months and do not include the TA A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7 5

providers. Program offices also use funding recipient affinity groups to solicit feedback. As one provider described, HUD staff will organize calls for groups of TA customers, TA providers, and HUD

staff who work on or have experience with similar topics (e.g., rural communities, HMIS) to give feedback on TA delivery and discuss general support needs. HUD field staff reported that they either reached out to solicit feedback or that TA customers would use their compliance calls to report to field staff how the engagement was progressing.

Most customers rarely—if at all—reported providing feedback to HUD staff on the delivery of TA they received; however, those who did had preexisting relationships with HUD staff or were experiencing an issue with the TA or TA provider. Interviews revealed that receiving feedback may depend on interpersonal relationships between TA customers and HUD headquarters and field staff.

Interviewees across these stakeholder groups cited their relationships with each other as part of why they felt comfortable sharing feedback. For example, some customer interviewees felt comfortable giving unsolicited feedback on TA and training because of their longstanding relationship with their field office staff; others stated that they would never initiate a feedback conversation with HUD

headquarters staff. One customer mentioned that he did so continuously and often provided feedback to the TA provider and the HUD field staff, but it was more so due to him being “outspoken” than his input being solicited.

“As a TA provider, I’m soliciting that [feedback] all the time. When I’m working with the community, and that just to me is a regular part of TA, is having that conversation about is this meeting your needs, what else do we need to do. I don’t know that that’s anything formalized that we do with communities.”

—TA provider

When a TA customer is either having problems with the TA they’re receiving or feels as if the TA is not meeting their needs, they will sometimes take their feedback directly to their program officer rather than discussing it with the TA provider. For instance, one provider shared an instance in which a customer who received SNAPS TA under an initiative felt that they were being provided a different TA than others in the initiative and wanted to understand why there was a difference across TA providers.

7 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

In that situation, the customer felt more comfortable addressing the issue with HUD TA program office staff rather than the TA provider.

TA providers regularly solicit feedback from their customers using their own methods and systems. TA providers highlighted that they informally solicited feedback during TA delivery, after an engagement had closed, and if they had previously worked with the TA customer before an engagement started, to understand their past performance. This feedback included debriefing on past engagements, determining what was going well or could be improved in the current engagement, and what the customer was hoping to receive going forward. For example, one TA customer reported that they used their biweekly meetings with their TA provider to give feedback on whether the TA was helpful and if anything needed to change. TA providers used this feedback to improve their TA delivery and maintain and develop their relationships with TA customers.

The systems that providers use to independently collect feedback range from requesting verbal feedback to administering surveys, and the methods vary by type of TA. For direct TA, providers asked either informally or through a customer survey about satisfaction and if the TA met the customers’

needs. This type of feedback solicitation was useful for measuring short-term success, such as whether the customer adhered to a regulation, received required training, or believed the TA led to improved results. Concerning universal TA, some providers interviewed routinely administered surveys to customers who participated in training or group learning. Providers gathered less structured feedback over email or in conversations during the TA engagement.

Ongoing Evaluation Efforts

In 2018, HUD implemented new policies to evaluate the effectiveness of TA engagements, including standardized outcomes, learning objectives, and feedback surveys. These activities launched after data collection for this assessment began, and their implementation and early results were outside the scope of work. Through the interviews and document review, however, the study team could glean some early findings of how these new policies are being implemented.

TA providers are now required to tie all work plans to standard HUD outcomes; all tools and products and in-person training must also be assigned learning objectives related to the outcomes for the work plan. TA providers were required to create target outcomes in the TA Portal for each work plan—and track progress toward these targets monthly and at the close of the engagement—before these policy changes; however, the assigned target outcomes were not standardized and were included as a text field that TA providers could tailor for each work plan. There was no standard A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7 7

process for collecting feedback at the end of a TA engagement; some in-person training administered pre- and post-surveys, and direct TA engagements generally relied on collecting informal feedback (detailed above).

Now providers must use one of the standard HUD outcomes for each work plan. There are 13

HUD outcomes, which fall into three categories of capacity building: using community, policy, and program needs to inform plans and priorities; designing strategies that align with plans and priorities; and addressing community needs through projects, programs, or systems (exhibit 21). For certain types of TA—tools and products and in-person training—providers are also required to create learning objectives, which identify the goals of a specific product or training and the knowledge or skills that the customers are expected to gain through the TA.

7 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 21: HUD GUIDANCE ON STANDARD OUTCOMES BY CATEGORY

HUD Outcome Category #1: Improved capacity to develop plans and priorities informed by community needs and program/policy goals

◼ 1A: Improved capacity to collect, analyze, and share data

◼ 1B: Improved capacity to gather and use resident and community input

◼ 1C: Improved capacity to develop high-quality, compliant plans that identify clear priorities and accurately reflect community needs and input

◼ 1D: Improved coordination and alignment with other community or regional plans HUD Outcome Category #2: Improved capacity to design strategies that align with plans and priorities

◼ 2A: Improved capacity to select programmatic strategies that address program or policy goals and community

◼ 2B: Improved capacity to design systemwide strategies that address community needs

◼ 2C: Improved capacity to design effective partnerships or cross-jurisdictional relationships that address community needs

◼ 2D: Improved alignment of resources with program-, system- or communitywide goals HUD Outcome Category #3: Improved capacity to deliver projects, programs, or systems that address community needs

◼ 3A: Improved staffing or organizational structure

◼ 3B: Improved administrative processes or infrastructure

◼ 3C: Improved financial management systems, controls, oversight to conform with 2 CFR Part 200 (OMB Omnicircular), and generally accepted accounting principles

◼ 3D: Improved program, grant, and regulatory compliance

◼ 3E: Improved capacity to substantively change the ways partners interact or conduct their work to address community needs

Sources: HUD (2018). “TA Outcomes Guidance.” Washington, DC: U.S. Department of Housing and Urban Development

[Internal document]

Feedback Surveys

HUD now administers standard surveys to all HUD TA customers who either receive direct TA or participate in in-person training. HUD is developing a survey for all online training. This process, implemented in 2018, was the first time that HUD implemented a cross-program TA survey process.

The surveys were implemented to standardize the collection of customer feedback across TA engagements and HUD program offices and to build a feedback mechanism to better understand the A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

7 9

strengths and weaknesses of the TA program as well as customers’ perspectives about the TA they received and the performance of the TA provider. The surveys focus on the TA process and the satisfaction of TA customers, TA providers, and HUD staff with the TA engagement.

◼

Direct TA surveys collect information on more intensive direct TA engagements and how they align with the outcomes identified in the work plan. HUD administers these surveys at the end of a TA engagement to the customers, providers, and HUD staff associated with the direct TA engagement. All three groups are asked to give an update on the status of the engagement and provide the following information so HUD can better understand the TA from multiple perspectives: an evaluation of their satisfaction with the TA engagement; an assessment of progress the customer has made toward meeting the outcomes laid out in the work plan and factors that facilitated or hindered their success; and suggestions on improving the HUD TA process.

◼

In-person training surveys, administered by TA providers, collect information on learning objectives, trainer’s knowledge, and satisfaction from anyone attending a HUD TA training.

Surveys are administered to training participants before the training begins and after it ends to gauge any increase in attendees’ knowledge on the training topic. Attendees are also asked to describe their satisfaction with the TA, including whether they found the training useful and whether they would recommend it to others.

Findings from these surveys go directly to the TAD office and are distributed quarterly to some members of different program offices; however, they are not widely distributed to all of the participants in the TA engagement (e.g., TA providers, HUD field staff, or HUD headquarters staff). As of January 2020, no TA providers interviewed had seen results from these surveys; HUD is working on a process to share aggregated results with them. Providers also expressed an interest in additional training on applying outcomes and learning objectives to work plans and administering surveys. It is important to note that the HUD TA customers that the research team interviewed received TA before implementing the formal surveys that HUD asked providers to begin administering in 2018. All of the customers interviewed expressed that they would like additional opportunities to provide feedback on the TA they received, and many supported implementing a survey.

8 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

VIII. Perceived Impacts

Introduction

Neither analysis of administrative data nor HUD’s ongoing evaluation efforts can quantitatively determine TA’s impact on customers; therefore, this chapter focuses on the perceived effects of HUD

TA from the perspective of TA customers and TA providers drawn from the in-depth interviews with key stakeholders and the two case studies. The chapter also highlights an example of the benefits of creating clearly defined metrics in quantifying the impact of TA. Finally, this chapter addresses the following research questions:

◼

Do customers think the TA they received met their expectations and helped them further their goals and objectives?

◼

Did the TA help customers improve their programs? If so, in what ways?

Key Takeaways

Customer Perceptions

◼

Most of the TA customers interviewed indicated that the TA they received satisfied their initial request and appropriately addressed their organization or community’s need. Customers indicated dissatisfaction with the TA process and outcomes, however, when it took a long time to receive TA after requesting it and when the TA they received was not specific enough to their needs.

◼

Customers found universal TA best for answering simple discrete questions and being more accessible to smaller organizations, whereas targeted TA was best for learning new processes and identifying root causes of issues they faced.

◼

Qualitative evidence from a small, diverse sample of TA customers showed a positive impact of TA on organization and program management and strategic approach development to their work. TA customers and HUD staff expressed that the impact of TA was directly related to the recipients’ preparedness to receive TA.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

8 1

Provider Perceptions

◼

TA providers agreed that TA engagements satisfied the original TA request. The TA had a positive impact on TA customers, particularly when an assessment of need was conducted first.

◼

Absent a rigorous evaluation, many TA providers were wary of overstating the long-term impact of TA or attributing too much of a TA customer’s success or progress to the TA they received.

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance

◼

The case study on troubled PHAs highlighted how the impacts of HUD TA are more easily quantified when metrics are defined at the outset of an engagement.

◼

Many PHAs demonstrate improvements in their metrics during and after TA, from increased occupancy rates and financial security to more assisted households.

◼

Management processes and financial health were also reported as improving, including faster work order response times, exiting receivership, and accruing financial reserves.

Customer Perceptions

Satisfaction with Technical Assistance Engagement

Most of the TA customers interviewed indicated that the TA they received satisfied their initial request and appropriately addressed their organization or community’s need. Unsurprisingly, TA customers who received targeted TA that was highly tailored to their specific needs and context were particularly satisfied with the TA they received. Customers were also satisfied with specific universal TA options, such as the SNAPS office’s help desk support or PIH’s Lead the Way training. For the help desk support, CoCs, other HUD funding recipients, and interested stakeholders with discrete questions about policies and programs may submit a question through HUD Exchange, and the HUD

headquarters program office can effectively address simple questions promptly.

Lead the Way was widely noted by troubled PHA TA customers as a good tool for PHAs and their boards. Although PHAs expressed a need for TA customized to their needs and local context, they 8 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

shared mainly positive experiences with Lead the Way. Around 2010, HUD identified a lack of training on PHA operations and governance as contributing factors to a PHA’s eventual troubled designation and began developing the online Lead the Way training. According to one HUD staff member, this training represents a more “proactive,” as opposed to reactive, way to meet the fundamental needs of PHAs. A provider highlighted that whereas Lead the Way reflects HUD’s shifting approach toward more universal forms of delivering TA, each module is tailored to the experiences of PHAs and allows for individual learning through its self-paced framework. The training fills a common need across most PHAs and provides a basic understanding of PHA governance. Most PHA customers interviewed required their board to complete the training. They agreed on the need for board members and executive staff to be updated on the best practices related to operations and governance.

“I think that HUD’s probably about 10 years late with that one because that should’ve been done a long time ago—a Lead the Way training for board or commissioners. That’s a good—that’s an excel ent [training] document as wel .”

—TA customer

Customers found universal TA best for solving discrete problems and more accessible to smaller organizations, whereas targeted TA was best for learning new processes and identifying root causes.

Customer organizations of all sizes said the universal TA, such as training and webinars, was the most useful when they had discrete questions or were familiar with HUD’s systems and processes but needed to update their knowledge to comply with a policy change or expand their capacity. Customers found targeted TA most useful when they were unfamiliar with HUD’s systems and processes and when they could not identify the root cause of an issue they were experiencing. Smaller customer organizations that could not leave work to attend a training or a webinar also found targeted TA most useful. Although smaller customer organizations could benefit from targeted TA, they found it more difficult to access if they did not have a relationship with their local field or regional office to assist with the application process.

Customers indicated dissatisfaction with the TA process and outcomes when it took a long time to receive TA after requesting it and when the TA they received was not specific enough to their needs.

First, as discussed in Chapter V, the period between when a TA customer requests and receives TA varies widely and can be a lengthy process. For example, one organization submitted a direct TA A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

8 3

request for help building their HMIS system to comply with their grant requirements; they did not receive an offer for TA for about 18 months. This extended timeframe can also exist for universal TA requests. In one interview, a TA customer explained that their organization submitted a question to help desk support about a grant application several months before the application deadline and did not receive a response until the deadline had passed.

Second, some customers interviewed said that TA they received did not meet their needs because it was not specific enough. For example, the standard protocol for help desk support is to direct customers to existing materials. Some customers expressed frustration with this protocol, stating that they were already familiar with HUD Exchange resources and needed a more specific response. For direct TA engagements, some TA customers expressed that they received general assistance and referrals to form materials to help fulfill their requests when they needed TA tailored to their specific organizational concerns and needs. For example, one customer was given a template for creating a departmental policy for escalation to adjust to their organizational processes, but they required more hands-on assistance in overhauling their organizational practices in their individualized context.

Despite these experiences, customers interviewed who were somewhat dissatisfied with their previous experience with TA expressed that they still see a lot of value in HUD TA and would continue to request TA as needed.

Impact of Technical Assistance

In addition to revealing whether the TA engagement satisfied their request for assistance, qualitative evidence from a representative—but not generalizable—sample of TA customers shows a positive impact of TA on improving organization and program management and developing strategic approaches to their work.

Organization management. In interviews with TA customers, respondents detailed how HUD TA gave their organization an increased ability to comply with program funding and management requirements and helped improve internal processes, such as hiring and performance management.

Some of these engagements also included a policy handbook that allowed the TA customer to train their partners and staff more efficiently on processes developed during the TA engagement. One customer cited their HUD TA as helping them identify and develop a system to address the deficiencies in their monitoring processes. Another customer credited the processes they created during their HUD TA engagements because they were able to come out of receivership. One organization used its TA engagement to implement strategies to be nimbler as an organization and to 8 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

be proactive in identifying and addressing potential problems rather than reacting to issues as they arise.

Customers of TA for troubled PHAs discussed a variety of particularly beneficial outcomes, including increasing the capacity of the PHA staff and board and assisting with amending and updating the PHA’s policies to be clear and compliant with regulations. All three of the SNAPS customers that the research team interviewed expressed that the TA they had received improved their operations, including following reporting and program guidelines, increasing their capacity to apply for grants, and assisting them in creating internal policies that aligned with HUD’s program requirements.

Program management. TA customers also asserted that HUD TA gave them additional tools and strategies to better manage their HUD programs. Multiple TA customers worked with their provider to create internal program targets and outcome measures or develop program designs and implementation plans for their HUD grants. For example, troubled PHA TA customers reported creating a system for assessing the physical needs of the housing stock or a system for planning for necessary construction; SNAPS TA customers who were CoCs used the assistance to make significant progress in developing their coordinated entry system. In addition, one customer was able to learn how to use their existing tracking systems more effectively. SNAPS TA customers said that TA allowed them to develop systems to improve their homelessness programs, serve their clients better, decrease veteran homelessness, and increase community involvement in the Point-in-Time Count, an annual count of people experiencing homelessness on a single night in January in a community.

“I think after TA… you’re better at administering the programs, making sure that you are following the guidelines, and serving clients better.”

—TA customer

Strategic approaches. In addition to reporting more standard impacts, such as improving program management or organizational capacity, TA customers expressed that the TA they received allowed their organization to develop strategic approaches to their work and focus on the broader impacts beyond program-level outcomes. One TA customer increased their capacity for ongoing financial sustainability efforts, including pursuing funding opportunities and grant applications. One TA customer leveraged a series of TA convenings for organizations focused on preventing and ending A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

8 5

veteran homelessness to establish relationships with their peers. Their relationship building led to creating a veterans’ collaborative that continued to meet and conduct peer learning after the TA engagement ended. Finally, a few customers pointed to their TA engagements as the impetus for thinking beyond program compliance and assessing the intended and unintended impacts of their programs in the community.

Both TA customers and HUD staff expressed that the impact of TA is directly related to the recipients’ preparedness to receive TA. For some organizations, their preparedness for TA is affected by high staff turnover or limited resources, such as staff capacity or financial assets. For example, a TA customer might not have the leadership or staff capacity to implement changes and processes created during a TA engagement. A TA engagement could also help set up a system or ongoing events that the organization cannot sustain without continued funding. For other organizations, it can be an issue of prioritization for leadership and frontline staff. As one customer said, “You can get all the TA in the world, but if you don’t do the legwork behind the scenes or in between the TA meetings, then you’re not going to get anything out of it.”

Provider Perceptions

TA providers generally agreed that TA engagements satisfied the original TA request and that the TA they provided had a positive impact on TA customers, particularly when an assessment of need was conducted first. Providers shared that this is often why the first task for a direct TA work plan in the TA Portal is often a needs assessment. Providers in the SNAPS office used the first few hours of an on-call TA engagement to conduct a needs assessment. Providers could use that time to determine the customer’s needs and tailor the remaining time for the TA engagement to best meet their needs.

Absent a rigorous evaluation, however, many TA providers were wary of overstating the long-term impact of TA or attributing too much of a TA customer’s success or progress specifically to the TA they received. Often, a TA provider does not have extended contact with the TA customer after completing the work plan. Providers that worked with customers over an extended period—such as engagements lasting longer than a single year—were able to point to specific changes that they witnessed but, in most cases, could not speak to the TA customer’s status after the engagement had ended. Providers were quick to point out that other factors could have contributed to TA customers’

successes. For example, one provider spent 2 years with one TA customer to address limitations in staff capacity. They provided resources and made recommendations but knew that the scale of change required was beyond the level of TA they could give. The provider remained in contact with the field 8 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

office and learned that the community made significant progress. They attribute some of that success to the TA and recommendations that they offered but could not say how much of a difference their TA made.

Quantifying Impacts: Troubled Public Housing Agency

Technical Assistance

The case study on troubled PHAs highlighted how the impacts of HUD TA are more easily quantified when metrics are clearly defined at the outset of an engagement. All troubled PHAs, once identified by HUD, must work with HUD headquarters and field staff to create a recovery agreement and action plan. For each performance management issue to be addressed, the recovery agreement and action plan must include a strategy for improvement, a clear set of performance targets (including statutory measurements), and a date by which the goals will be achieved.

According to TA provider and customer interviewees, many PHAs demonstrate improvements in their metrics during and after TA. Both TA providers and customers reported direct impacts of the TA on improving PHA performance and recovering PHAs from troubled and receivership status. Selected metrics generally measured TA impacts such as improving occupancy rates and the corresponding PHAS or SEMAP scores, assessing and updating deteriorating housing stock, increasing financial security, and improving organizational management. Occupancy rates were reported as increasing from percentages in the low 70s to 95 percent in one example and from below 80 percent up to 98

percent in another. One jurisdiction increased their assisted households by more than 1,000 families through TA targeted at helping boost their voucher use.

Management processes and financial health were reported as improving as well. Several TA customers discussed updating their standard processes around rent collection, responding to work orders, and updating units. One TA provider shared that they worked with a PHA who started with a backlog of 3,000 pending work orders. During the TA provision, the PHA got their response time down to 60

days to complete a work order and finally down to 10 days or less to respond to a resident’s request.

TA was noted as helping several PHAs come out of receivership, with one PHA managing to accrue a significant amount in their financial reserves.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

8 7

IX. Recommendations for Moving

Forward

Introduction

Community Compass has moved HUD toward more transparent and efficient TA, but there is room for improvement. Numerous recommendations surfaced during data collection with HUD staff, TA providers, and TA customers on how processes could improve, and analysis of the administrative and qualitative data revealed additional suggestions. This chapter highlights recommendations that are relevant across programs and TA types, grounded in goals of ensuring that TA customers receive the assistance that will be most helpful and effective while using TA dollars efficiently. These recommendations seek to address the following questions:

◼

How could HUD obtain regular feedback from its providers and customers about the delivery and impact of HUD TA?

◼

How does HUD evaluate the impact of TA on customers’ financial, managerial, physical property, and governance practices?

Recommendations concern improvements to communication and coordination practices, the development of efficient and flexible work plans, wider access to work products, improved data systems and data, and better tracking of outcomes associated with TA. Because HUD is continually improving its TA processes, changes have been made to Community Compass since data were collected for this study.

Improve Communication and Coordination

Communication among key TA stakeholders is fundamental to achieving TA purposes. Customers need to know what TA is available to seek the assistance they need, and providers need full information on applying for NOFA awards and regular communication to support the successful implementation of their work. Effective communication among HUD staff involved with TA provision and oversight—and between them and the other stakeholders—is essential for the flow of information, TA effectiveness, and efficient use of federal monies.

8 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Customers

This assessment revealed that not all HUD funding recipients—even if they have been TA customers in the past—are aware of all TA that may be available to them, along with various options for accessing it.

Some differences in awareness occur, in part, on the basis of whom funding recipients interact with among HUD staff. For example, if a funding recipient’s primary point of contact is a field office, they may be at a disadvantage because not all field offices are engaged in TA directly and may have inadequate information about the range of available TA and how funding recipients can access it. In addition, funding recipients who want to access targeted TA may not know how to develop an appropriate TA request and submit it. Evidence from this study suggests several steps HUD could take to increase recipients’ awareness of TA options and the process for making requests.

◼

Communicate information on TA through existing channels. HUD regularly distributes information to funding recipients on the programs they are administering. For example, when a policy or regulatory change is made, funding recipients receive memos providing updated guidance for complying with the change. Standardized communication about TA options could take place through a similar mechanism.

◼

Develop best practices for field office communications and guidance on TA. HUD could survey field offices’ practices for communicating TA availability and access to customers and develop clear guidance and materials for field offices to replicate across the country. Although not all HUD program offices leverage field offices to the same extent on TA, all field staff could be kept updated on regulations and programmatic goals and equipped to communicate with their funding recipients on the availability of TA options and how to access them.

◼

Clarify the TA request process, particularly for smaller or first-time customer organizations.

Smaller or first-time customers can find the TA request process challenging, especially concerning articulating the issue they would like TA to address. A clearer application process, with avenues for building relationships with HUD staff early in the process, would help customers craft requests that align strategically with the TA available.

Providers

Most providers affirmed that they benefited from strong, clear communications with HUD TA program officers once they have been contacted for a potential TA engagement and throughout the work plan development and approval process. Providers indicated, however, that they want more communication from HUD on the logistics of applying to be a provider and on program guidelines, requirements, and changes.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

8 9

◼

Clarify processes to avoid limiting the range of organizations able to provide TA. Many of the providers interviewed—especially smaller organizations—want a more predictable approach in receiving TA assignments so providers can ensure staff coverage and availability, particularly if they need to secure the assistance of a subcontractor to provide TA. HUD headquarters staff could use the NOFA webinars to set expectations, communicate the process timeline, and encourage providers to engage subcontractors earlier in the process to the extent possible, given the challenges of anticipating the expertise needed to address TA requests and of having the resources available for earlier engagement.

◼

Strengthen communication outside specific TA engagements. Expand regularly scheduled communications, such as webinars or calls, for HUD headquarters staff and providers.

Providers suggested that HUD use the meetings to communicate guidance to providers on general expectations, changes in requirements, reporting practices, and other pertinent information. Such sessions would help ensure that information applicable across providers would be communicated consistently.

Across HUD

Myriad HUD staff are engaged in HUD TA programs within headquarters and across field offices.

Some field staff feel disconnected from the TA process under Community Compass. These field staff could be better informed about the TA administered across program offices, which could improve the efficiency and effectiveness of TA overall. In addition, some program office staff outside the Office of Community Planning and Development would like better access to the Technical Assistance Division.

◼

Standardize communication between HUD headquarters and field staff at key steps in the TA process. Alert field offices when a TA request is received, when a request is approved and a provider is assigned, when a work plan is approved, and when a TA engagement ends. This communication will enable field offices to track funding recipients’ requests and monitor the TA they receive. It also will improve their communications with providers to increase providers’ awareness of field office responsibilities related to various TA programs. Granting field staff access to the TA Portal data or implementing a notification system when data are entered could help.

◼

Communicate TA survey results to appropriate field staff to aid in monitoring. Provide field staff the data from the recently implemented customer surveys and support staff training on data interpretation. Access to survey customer survey data would support field office TA monitoring efforts.

◼

Assess cross-program communications, and explore options for improvement. Identify key communications barriers across all HUD headquarters program offices engaged in TA and then develop processes and systems to overcome these barriers. These could range from more standardized communications at specified points in the TA approval process to larger 9 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

changes, such as embedding Government Technical Representatives in all program offices or considering whether TAD could be an independent office outside any program office.

Develop Efficient Work Plans

The work plan for an individual TA engagement is the primary agreement between the HUD TA GTR, the Government Technical Monitor and program office, and the provider on the TA type and delivery process. The work plan is an important document for laying out the scope of work for which the provider is accountable, although providers reported often discovering customer TA needs that extend beyond an existing plan’s scope. In addition, the process for expanding or amending an existing work plan can be onerous. Consequently, providers sometimes will enter separate work plans rather than amend a current plan because of the time it can take to receive approvals for amendments. There are several actions HUD could take to allow providers greater flexibility in developing work plans to best meet customers’ needs efficiently.

◼

Expand use of needs assessments. Ensure that customers’ needs are understood before determining the TA type they need or developing a work plan and budget. Some providers often started customer engagements with a brief needs assessment as part of on-call SNAPS

TA engagements or before initiating work plans for longer direct TA engagements with troubled PHAs but said that they and customers would benefit from time spent on a formal needs assessment work plan. A more comprehensive needs assessment would increase the likelihood that the best TA type is matched to the customer to maximize the impact. Providers suggested that this is especially the case for new customers.

◼

Develop and fund a more iterative process for work plan development. Consider allowing providers and relevant HUD staff to engage in a more iterative process for plan development before providers enter a work plan into the TA Portal. Some providers take this approach informally, which can result in uncompensated time.

◼

Allow quicker work plan approvals. HUD headquarters staff and providers agreed that the 30-day period is too long to wait for approvals of work plans and amendments when customers require immediate assistance. Coupled with a change to a more iterative process for work plan development, a shorter timeframe would enable providers to engage customers quickly.

Increase Access to Work Products

There was little evidence of the standardization and sharing of final TA resources and products across providers. As a result, different providers develop similar products to serve customers with similar TA A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9 1

needs rather than building upon existing materials. HUD could encourage sharing resources across providers and work plans while allowing for customization and incentivizing multiple uses of TA materials produced under Community Compass cooperative agreements.

◼

Consider creating a repository of TA work products. HUD headquarters or regional offices could collect providers’ reports, sample processes and procedures, and relevant guidance documents to make available to other providers. Providers could tailor materials to meet their customers’ needs while recycling standardized information and best practices to extend the value of HUD TA investments.

◼

Encourage providers’ use of open-source platforms for widely used products. Using open-source platforms for training and other universal TA materials would allow HUD to update TA tools to ensure relevancy over time. HUD could incentivize providers to use such platforms by offering additional points in the NOFA scoring process. HUD also could write award terms that provide shared ownership of TA materials or allow materials to be retained and updated directly by HUD or with the support of another provider. Such actions would maximize the use of TA resources and extend the life of provider-developed training and tools.

Improve Data Systems and Data

Analyses conducted for this assessment were reliant on data available through the TA Portal, which were discovered to be inadequate for fully addressing a number of research questions. To improve data available to HUD staff, TA providers, and researchers, HUD could consider several actions to improve the data systems and data quality and expand the data variables collected. Implementation of the recommended improvements would require additional funds.

Document and Update Systems

Analysis of HUD administrative data revealed the need to update both the TA Portal (used throughout this assessment) and DRGR system (not used) and their documentation to streamline data entry, support data linking, and improve data interpretation. Based on this analysis, the following updates may be useful:

◼

Maintain documentation of data and variable definitions. At present, no complete data dictionary with variable definitions exists for internal use within HUD, which increases the likelihood that variables will be misinterpreted and data misused. HUD could maintain an up-to-date data dictionary for all data collected through the TA Portal.

9 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

◼

Streamline entry of duplicate data within the TA Portal and DRGR. Because the TA Portal contains additional data on the TA process and engagements and DRGR is the record system, providers often enter the same information in both systems. This duplication is inefficient and leads to data entry errors and delays in the TA process. Some providers suggested a temporary workaround by enabling a copy-paste function to allow their staff to enter information across systems. Ideally, the two systems would share information to reduce redundancy and avoid duplicative entries by providers.

◼

Enable data linking across systems. Data cannot be matched across systems because of inconsistent data entry or lack of an identification code used across systems. For example, data cannot be matched between the TA Portal and DRGR because the work plan ID variable is not entered consistently by providers. Adding a PHA identification code field could clearly identify PHA customers and allow linking TA Portal data with PHAS and SEMAP data. Taking steps to improve data linking would enhance TA monitoring and support assessments of TA outcomes. HUD could establish and require the use of a customer identification code for use across data systems and consider systems tools that block the submission of customer codes entered incorrectly.

◼

Work toward data consistency to the extent possible across TA types. Differences across programs and TA types necessitate a degree of variation in the data collected. HUD could review what and how data are collected within a program or TA type and make changes to improve consistency and uniformity. HUD also could work toward as much data consistency across programs and TA types as possible, such as establishing the same format for documenting work plan escalation across programs and TA types.

◼

Expand the range of date fields collected. For example, include the actual start and closeout dates of TA delivery, and create additional fields to accommodate updates. These data would allow HUD to track accurately how long approvals take and how long TA engagement performance periods last to monitor timeliness. It also would help better align the timing of TA receipt with changes in funding recipient performance measures, such as PHAS/SEMAP

scores, to enable more robust outcomes evaluations.

◼

Address IT functionality issues that affect data and information entry. Whether due to HUD’s IT systems generally or the TA Portal and DRGR data systems specifically, the TA Portal system runs slowly, and providers experience challenges entering and saving work plans and reports in both the TA Portal and DRGR. HUD could assess the IT issues to determine whether adjustments to the existing systems are possible.

Improve Data Quality

Examination of the administrative data, coupled with feedback from providers, showed that data quality is compromised by a reliance on text entries, inconsistencies in data entry, and missing values.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9 3

The following changes would improve data entry procedures and enhance data accuracy and completeness:

◼

Reduce text-based entries. Require less text entry and offer more drop-down response options so data can be extracted and summarized easily. This change will reduce data entry errors, increase the ability to match work plans across the systems, and support analyses to describe who receives TA, where, and for what purpose.

◼

Improve how HUD staff and providers enter and report on TA work plans in the TA Portal.

Analyses of administrative data showed that work plans are entered into the TA Portal inconsistently. Also, interviews revealed that providers often have their own processes for standardizing work plan entry that might conflict with another entity’s procedures. As a result, providers and HUD program staff have entered duplicative and incorrect work plans. Data standards could include clear guidance for identifying customers, naming work plans, and entering these data consistently across data systems and work plans. Instructions to providers and program staff could emphasize the importance of opening work plans early in the process to avoid backdating the period of performance start date. Those who enter data could also be instructed on deleting duplicative work plans when identified within the system. Data systems could prompt users to update work plans or close plans that have been inactive for a certain amount of time to improve the accuracy of period-of-performance end dates or anticipated end dates.

◼

Require the entry of key variables. There were high rates of missing values in the TA Portal for several variables that identify who receives TA and which HUD offices are associated with the TA. These missing values include indicators for the type of customer (e.g., CoC, PHA, TDHE), program office (e.g., SNAPS, PIH, ONAP), and regional and field offices. For example, in the TA Portal, 87 percent of entries in 2017 for customer type were missing, and those that were filled in often read “grantee,” with no additional information on customer type. Requiring entry of key variables (e.g., forcing a valid entry before saving) could improve internal monitoring and future external evaluation.

◼

Ease the data amendment process. Make changes to the TA Portal to allow providers to create, amend, and close work plans. This modification can address key issues already noted above, such as ensuring that providers can update work plans easily when new customers’

needs come to light and reducing the occurrence of duplicate work plans. Data systems also could include fields for amendment dates and the number of times amended.

◼

Collect information on subcontractors. Add fields to the TA Portal for the names of any subcontracting organizations on TA engagements, at a minimum. The TA Portal does not include data fields to collect information on whether a TA provider has subcontracted some or all of the work on a work plan or the names of subcontractor organizations. HUD and evaluators need subcontractor information to know who provides the TA and how best to monitor TA provision.

9 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Track Clear Outcomes

This assessment uncovered gaps in collected data and feedback where more information would lead to closer monitoring and better evaluations of HUD TA performance and outcomes.

◼

Standardize and expand feedback mechanisms to gather and track input from customers for monitoring and evaluation purposes. HUD collects some feedback from all TA customers but not in a consistent manner. Providers also have said they would benefit from receiving feedback. The information would help them make any necessary adjustments to their TA.

Next steps could include reviewing metrics currently collected against information needed to assess customer satisfaction and TA effectiveness; sharing results from the existing TA customer surveys with HUD staff and TA providers who do not already receive them; and gathering information from surveys and other formats, such as scheduled calls organized by TA type, for all types of TA engagements across programs.

◼

Expand tracking of TA outputs and outcomes. Collect systematic data on work plan outputs and customer outcomes associated with all TA types. Currently, there is no systematic collection of outcome data using a standardized set of metrics across different types of TA. To collect output and outcome metrics data, HUD could develop or require providers to establish metrics for each work plan, assess baseline values, and measure progress at the end of a TA engagement.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9 5

X. Summary of Findings and Future

Research

Introduction

TA offered under Community Compass covers an array of topics, from homelessness to public housing to disaster recovery. It is used to support efforts to build inclusive communities; meet the need for quality, affordable rental homes; strengthen the nation’s housing market; and use housing as a platform for improving quality of life, and it does so across a variety of program offices, geographies, and populations. Studying Community Compass is challenging because of the variations in the basic process across programs and TA types and the data limitations documented in this report.

This final chapter summarizes findings from this assessment of HUD’s Community Compass TA structures, processes, and content. It also includes suggestions for future research on TA outcomes and impacts. The findings are intended to be of value to HUD to continue improving TA provision and identifying future research efforts.

Summary of Key Findings

TA serves diverse customers, including local and state governments, agencies, and nonprofit organizations, many of whom receive TA more than once. Slightly more than one-half of the customers represented in available data are PHAs and CoCs. The number of providers has increased over time, and more providers receive relatively smaller cooperative agreement awards than in the past. Many providers, most of whom are consulting firms or housing and community development organizations, supplement their staff with subcontractors and consultants to address staffing needs and expertise for carrying out TA work assignments.

Five core phases of the TA process—the NOFA process and provider selection, identifying customer needs, assigning a provider and scoping the work, developing and delivering TA (TA engagement), and closing out the process—are common across TA programs. The way each step is carried out, however, varies considerably across types of TA, HUD office or program, and the extent to which legacy processes have been retained since the shift to Community Compass. Overall, 9 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

administrative and qualitative data indicate that TA assignments and engagements occur promptly, for the most part, with significant outliers; for example, 17 percent of work plans took more than 2 years to complete.

Centralization of the TA process under Community Compass was intended to create efficiencies and flexibilities. It has achieved this goal in some areas; for example, because NOFAs now cover 2

fiscal years, providers do not have to reapply annually. The shift from assignment-focused contracts to a blanket cooperative agreement with each provider that can span multiple programs also increases both the efficiency of the contracting process and the flexibility of HUD and providers in work assignments. There were losses, however, in the transition to a more centralized structure; for example, the consolidation of GTRs within CPD has left some program offices feeling disconnected from the TA process and has slowed TA approval and monitoring processes when substantive knowledge is needed that the CPD GTR lacks. Field office roles also have changed within some programs, leaving some program staff in field offices feeling cut off from the TA process altogether.

HUD shifted to the Community Compass structure to increase collaboration among TA programs and TA delivery. This report has documented some gains in collaboration; for example, there has been increased collaboration at the NOFA stage of the TA process, and the SNAPS program has made advances in collaboration among its TA providers; however, gains were not found among HUD

program offices, across providers broadly, or in direct TA delivery to an individual customer.

Universal TA has grown. New curricula, such as Lead the Way for PHA management, received high marks from customers for providing practical training. Expansion of HUD Exchange as a one-stop shop for resources and a gateway for accessing universal TA has helped put information at HUD

funding recipients’ fingertips, but help desk support can be frustrating for funding recipients looking for customized answers. Some customers thought answers were too generic or referred customers to online resources that they had already reviewed.

Direct TA remains important and is the largest part of HUD’s overall TA budget, but there is still little standardization in its delivery across HUD TA programs. Providers report that needs assessments often are completed under the guise of other types of TA because they need to understand a customer’s specific circumstances and needs before launching a work plan to address those needs.

The TA Portal system was designed for use by several HUD TA offices to manage TA and has expanded over time to include additional offices; however, it is not well aligned with the DRGR system used for processing vouchers—one cannot link all customers or TA types across the systems, and it is A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9 7

difficult to track TA process timelines in the data. Also, not all parties who need access to TA-related data, such as field staff who monitor TA requests and work plan progress, have access.

Community Compass has standardized reporting processes, a change welcomed by HUD staff and TA providers. The data systems, however, still require duplicative reporting. Limits on HUD staff’s access to reporting systems, especially at field offices, continue to hinder their ability to monitor TA engagements. Evaluation of TA is made difficult by a lack of consensus on the level—individual or system—at which impact should be measured, lack of established metrics for measuring outcomes and impact, and a short timeframe during which data are gathered, which may not align with the timeline for expected results.

Customers reported satisfaction with the TA they received, indicating that it addressed their organization’s or community’s needs. They said TA had a positive impact on their organizational, program, and grants management; however, customers reported dissatisfaction with the length of time it can take to receive TA after requesting it and the quality of some TA that they saw as generic and insufficient for addressing a specific need. HUD does not have a systematic approach for measuring impact or tracking effects over time across a TA program, a provider, or a customer. Collected information is not available to field staff or providers who do not have access to customer reports or, in the case of the PHA customer survey, survey results. Certain TA programs, such as TA for troubled PHAs, provide examples for establishing clearly defined metrics to evaluate TA effectiveness.

Future Evaluation and Research

Key questions to address in future research concern outcomes and impact of HUD TA for customers’

financial, administrative, and managerial practices and the ultimate beneficiaries of HUD TA—the people and communities customers serve and the agencies or organizations and properties they manage. Issues with data quality and completeness will continue to pose challenges for research, although there are approaches for minimizing those challenges. Here are four suggestions for research:

◼

Collect pre- and post-TA data from customers. Systematically collecting pre- and post-TA data from customers on outcomes of greatest interest to HUD would increase the information available to staff for monitoring and evaluation purposes and to providers for considering improvements to TA products if results were shared with them. In addition, expanding the timeframe for data collection would improve the value of the data. Post-TA data should be 9 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

collected three times—at the end of TA, 6 months after TA, and 12 months after TA to capture data while the TA experience is still fresh in customers’ minds. Data collection could be carried out by external researchers or by field staff with data collection and analysis skills.

◼

Track changes in PHAS and SEMAP scores. The usefulness of scores for assessment improvements related to TA would improve if scores could be linked to TA data systems. In addition, linking the data would support analysis of correlations between score changes and details of TA received, such as TA type and duration.

◼

Conduct detailed examinations of work plan budgets. Analysis of work plan budgets would improve understanding of TA costs and budget expenditure rates, helping identify where savings might be possible. For example, examinations could focus on budgets associated with the same type of TA (e.g., budget comparisons across needs assessments); differences in budgets across TA types (e.g., direct TA compared with in-person training); and budgets for the same type of TA offered through various program offices (e.g., direct TA through SNAPS

and ONAP).

◼

Conduct randomized controlled trials (RCTs) of TA effectiveness. RCTs are expensive to undertake but would be the best approach for studying TA impact and effectiveness. A trial could focus narrowly on a type of TA to understand its effects against the absence of the TA.

A trial could also examine differences in effects on an outcome of interest by comparing two or more TA types. RCTs would be limited in breadth, although findings could shed light on aspects of TA that could be valuable across TA programs and types.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

9 9

Appendix A. Community Compass

Offices, Divisions, Grant Programs,

and Appropriation Accounts

Program Office

Division

Grant Programs

Appropriation

Account

Community

Office of Block

◼

Community Development Block

Departmental TA;

Planning and

Grant Assistance

Grant (CDBG) Entitlement and Non-

Development

Entitlement Programs

CDBG—Disaster

Recovery TA

◼

CDBG—Disaster Recovery

◼

Section 108 Loan Guarantee

Office of Special

◼

Continuum of Care (CoC) Program

McKinney-Vento

Needs Assistance

TA

◼

Programs

Emergency Solutions Grant Program

(ESG)

Departmental TA

◼

Housing Opportunities for Persons

National Homeless

with AIDS (HOPWA)

Data Analysis

Project

◼

Emergency Solutions Grants

Youth

◼

McKinney-Vento Homeless

Homelessness TA

Assistance

◼

National Homeless Data Analysis

Project

◼

Youth Homelessness

Office of Rural

◼

Rural Housing Assistance Programs

Departmental TA

Housing and

Economic

Development

Office of

◼

HOME Investment Partnerships

Departmental TA

Affordable

(HOME)

Housing Programs

◼

Housing Trust Fund (HTF)

Office of Technical ◼

Runs TA programs, not HUD

Departmental TA

Assistance &

programs

Management

Public and Indian

Office of Native

◼

Indian Housing Block Grant

Native American

Housing

American

Housing and

◼

Programs

Title VI Loan Guarantee

Community

◼

Section 184 Home Loan Guarantee

Development TA

◼

Tribal HUD-Veterans Affairs

Supportive Housing (HUD-VASH)

1 0 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Program Office

Division

Grant Programs

Appropriation

Account

◼

Indian Community Development

Block Grant

◼

Native Hawaiian Housing Block

Grant

Office of Public

◼

Public Housing Capital Fund

Departmental TA;

Housing Programs

PHA

◼

Public Housing Operating Fund

Administrative

◼

PHA Receivership and Recovery

Receivership and

Recovery TA

Office of Public

◼

Resident Opportunities and Self-

Departmental TA

Housing

Sufficiency (ROSS)

Investment

◼

Moving to Work Demonstration

Program (MTW)

◼

Jobs Plus

◼

Family Self-Sufficiency

◼

Rental Assistance Demonstration

◼

Choice Neighborhoods

◼

ConnectHome

Office of Housing

◼

Housing Choice Voucher Program

Departmental TA

Choice Vouchers

◼

Project-Based Vouchers

Fair Housing and

N/A

◼

Affirmatively Furthering Fair

Departmental TA;

Equal

Housing (AFFH)

National Fair

Opportunity

Housing Training

◼

Fair Housing Act (Title VIII of the

Academy

Civil Rights Act of 1968)

Housing

Office of

◼

Multifamily Housing Programs,

Departmental TA

Multifamily

including Project-Based Section 8,

Housing

Supportive Housing for the Elderly

(Section 202), and Supportive

Housing for Persons with

Disabilities (Section 811)

◼

Multifamily Housing Preservation,

including Rental Assistance

Demonstration

◼

Multifamily Housing Production

Initiatives

Office of Single

◼

Housing Counseling

Departmental TA

Family Housing

Office of

◼

Rental Assistance Demonstration

Departmental TA

Recapitalization

(Recap)

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 0 1

Program Office

Division

Grant Programs

Appropriation

Account

Office of Policy

N/A

◼

Low-Income Housing Tax Credit

Departmental TA

Development &

(LIHTC) data collection and quality

Research

for LIHTC tenants and projects

Cross-Office

N/A

◼

Lead Hazard Control and Healthy

Departmental TA

Homes

◼

Smoke-Free Housing

◼

Knowledge management, including

the HUD Exchange

◼

Energy efficiency and high-

performance building retrofits for

assisted housing properties

◼

Community engagement and

economic opportunity programs,

including Opportunity Zones and

Promise Zones

◼

EnVision Center Demonstration

◼

Benchmarking and utility data

analysis

◼

Environmental Reviews for the

National Environmental Policy Act

(NEPA) and related federal

environmental laws and authorities

◼

Uniform Relocation Act (URA) and

Section 104(d) requirements

◼

Violence Against Women Act

(VAWA)

◼

Economic Opportunities for Low

and Very Low-Income Persons

(Section 3)

◼

Davis Bacon

Source: HUD Technical Assistance Notice of Funding Availability, 2018, 2019

1 0 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Appendix B. Additional Information

on Study Implementation

This appendix provides supplemental information on study implementation, including the process for identifying TA customers for the troubled PHA and SNAPS case studies, the interview sampling strategy, and respondent characteristics for in-depth and case study interviews.

Identifying Troubled Public Housing Agency Technical

Assistance Customers

Working with a dataset of all PHAs—organized by year, linked to PHAS and SEMAP scores, and indicating any TA received from 2011 through 2018—the research team created variables to designate whether a PHA was ever troubled, if they ever received TA, and in what years each of those things occurred. To merge agency scores and TA Portal data to create this dataset, the team began by cleaning the SEMAP data so the variable names would match with the PHAS data. SEMAP scores from before 2011 were dropped because that was the earliest year of the PHAS data. Two datasets were joined to clean the PHAS data, one from 2011–2018 and one with a complete set of scores from 2017

and 2018. Duplicates were dropped. In cases in which there were two observations for a PHA in a given year but differing scores, the research team kept the most recently released scores. When PHAS

and SEMAP scores were merged, a small number of agencies (742 observations) were listed in SEMAP

data as providing both Section 8 and public housing units but were not present in the PHAS data.

Seventy percent of the 742 scores that did not merge were from 2018 onward, so the PHAS data may not have been updated.

To prepare the TA Portal data for merging, the team split the recipient variable data to separate each customer when multiple entities received TA under one work plan. The team then reshaped the data by both customer and fiscal year of the performance period so that each customer and year had its own entry and the data would match the PHAS/SEMAP data organized on customer and year.

To maintain consistency with other data analysis in this assessment, work plan and budget amounts were weighted over all years of a work plan’s performance period (see chapter V for a discussion of dates and approval timelines using TA Portal data). Once merged, the data showed PHAS

and SEMAP scores and the work plans for a customer in a given year.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 0 3

To identify PHA customers, the research team used the work-plan recipient field. Exploratory analysis of this field revealed that the acronyms “PHA” and “HA” were never used, and “housing” was always spelled out; therefore, customers without the word “housing” in this field were dropped. The remaining customers were then vetted manually, and the team dropped those that were not PHAs. To ensure that no PHAs were accidentally excluded, the research team also searched the descriptions of work plans that did not contain the word “housing” and found 14 customers that were cities, counties, or municipal housing departments that included TA for the local PHA. These customers were matched to their PHA using their unique PHA code assigned by HUD.

For the final list of PHA customers, the research team entered their PHA codes in Excel by hand to merge them with the TA Portal data using those codes. Organizations on the list identified as not being a PHA were dropped. The hand-coded TA Portal data were then merged with the PHAS/SEMAP

data.

Tribally Designated Housing Entities and Rental Assistance Demonstration and Moving to Work agencies were kept, although they do not receive PHAS or SEMAP scores. The research team also identified five PHAs in the TA Portal data for which TA could not be assigned to a specific PHA because of duplicate names across multiple agencies. Two additional PHAs were missing the HUD

PHA code in the PHAS and SEMAP data. Although Tribally Designated Housing Entities (185 of them) and these seven PHAs were not merged with the PHAS/SEMAP data, the team kept them in the dataset, and they were included in the analysis of TA received by PHAs.

Identifying SNAPS Technical Assistance Customers

About 55 percent of the observations in the TA Portal were missing the key variable (“program office”) needed to identify TA associated with the SNAPS program office. For this study, only work plans that listed SNAPS in the program office variable were used; therefore, SNAPS work plans in the TA Portal may not have been identified as such because of the missing program office information.

In-Depth Interview Respondent Characteristics

Interviews were conducted by teams of two researchers and lasted approximately 60 minutes. All interviews were conducted by telephone except one, which was conducted in person. Respondents 1 0 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

were assured confidentiality following Urban’s IRB approval. With the permission of the respondents, interviews were audio-recorded and then transcribed.

Respondents by Technical Assistance Type and Program Office

In aggregate, HUD staff, TA provider, and TA customer respondents spoke about every TA type and five of the six program offices under Community Compass. Direct TA and training were the most prevalent types of TA with which interviewees engaged, as shown in exhibit 22. CPD and PIH were the most frequently appearing program offices in the interview sample, as shown in exhibit 23.

Although exhibit 23 reflects the major HUD program offices providing TA through Community Compass, many of the TA engagements discussed during the interviews involved the divisions and programs within the program offices; for example, the HOME Investment Partnerships Program (HOME), the Office of Affordable Housing Preservation, SNAPS, and the Community Development Block Grant all operate within CPD. In addition, customers and providers had experience working with ONAP, which falls under PIH. PD&R was not included in the research design, although several providers mentioned TA-related work they did with that office.

EXHIBIT 22: INTERVIEWEES BY TECHNICAL ASSISTANCE TYPE

Number of Interviews

30

25

20

15

10

5

0

Direct TA +

Help Desk

Needs

On-call

Tools and

Training

Capacity Building

Support

Assessment

Products

TA Type

TA = technical assistance.

Note: Because interviewees had experience with multiple TA types, the number of instances a TA type was discussed is larger than the number of individuals interviewed.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 0 5

Source: Interviews and TA Portal “Work Plan File” received from HUD on August 9, 2018

EXHIBIT 23: INTERVIEWEES BY HUD PROGRAM OFFICE

Number of Interviews

Providers

Customers

Field Offices

HUD HQ

35

30

25

20

15

10

5

0

CPD

FHEO

Housing

OED

PD&R

PIH

HUD Program Office

CPD = Office of Community Planning and Development. FHEO = Office of Fair Housing and Equal Opportunity. Housing =

Office of Housing. HQ = headquarters. OED = Office of Economic Development. PD&R = Office of Policy Development and Research. PIH = Office of Public and Indian Housing.

Notes: Because interviewees had experience with multiple technical assistance (TA) types, the number of instances a TA type was discussed is larger than the number of individuals interviewed. Program offices represent the office that housed the TA engaged by respondents.

Sources: TA Portal “Work Plan File” received from HUD on August 9, 2018, and interviews Respondents by Stakeholder

HUD HEADQUARTERS STAFF

One-half of the 14 interviews conducted with HUD headquarters staff were with CPD staff, and the team interviewed at least one staff member from each program office except for the Office of Economic Development. Because of each respondent’s program expertise, the focus of the interviews varied, and not all interviewees mentioned the types of TA with which they had an experience. Among respondents who did, the most commonly discussed TA types during interviews with HUD

headquarters staff were on-call, direct TA, and tools and products. In addition, one interviewee 1 0 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

mentioned they had experience with needs assessments, and another interviewee had experience with help desk support TA.

HUD FIELD OFFICE STAFF

The seven field office interviews with 17 staff from eight field offices covered all TA types except help desk, which is administered from HUD headquarters with limited field office involvement. Field office staff spoke about direct TA and training more than other types of TA. Only two respondents indicated involvement with needs assessments and two with on-call TA, and one respondent mentioned discussing TA tools and products. As learned from the in-depth interviews, field offices tend to be more involved with administering training and direct TA, whereas HUD headquarters tends to be more involved with help desk TA, on-call TA, needs assessments, and the development of tools and products.

TECHNICAL ASSISTANCE CUSTOMERS

Interviews with 11 customer entities involved 13 customer staff members. The respondents mostly had received direct TA (six respondents) and training TA (seven respondents). This finding is in line with the two largest TA work plan types in the TA Portal—52 percent of work plans are for direct TA, and 21 percent involve self-directed and group learning TA. Four of the customers received help desk support, four received on-call TA, and three received tools and products TA. The sample included customers who received needs assessments, but the customer staff did not discuss their needs assessment experience. The interviewees might not have been able to distinguish between the delivery of needs assessment and direct TA if a needs assessment progressed into direct TA delivery.

Most of the customers interviewed received TA mainly through CPD—a by-product of the sampling approach (see chapter II for the discussion of sampling). The team partially stratified the customers by the program or program office under which they received TA, emphasizing interviewing customers from program offices where the team was also interviewing HUD staff. The remaining customers were randomly sampled for program offices within each TA type. Those offices (CPD, PIH, and Housing) also have the largest number of work plans in the available data.

TECHNICAL ASSISTANCE PROVIDERS

Interviews with 21 staff members from 12 provider entities covered the administration of all TA types for five of the six program offices under Community Compass. Although the sample included multiple providers for most types of TA, only one interviewee administered help desk support TA (also called A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 0 7

Ask A Question) because only one provider is responsible for this service. Exhibit 24 shows the TA types and program office affiliation for each provider.

1 0 8

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 24: PROVIDERS BY TECHNICAL ASSISTANCE TYPE AND PROGRAM OFFICE

TA Type

Self-Directed

Tools and

Direct TA,

Help

and Group

Program

Providers

Needs

On-

Products

Capacity

Desk

Learning

Assessments

Call

(Product

Building

Support

(Training

Office

Development)

Delivery)

CPD,

FHEO,

Provider #1

X

X

X

X

X

Housing,

PD&R, PIH

Provider #2

X

X

CPD

Provider #3

X

X

X

X

X

CPD, PIH

Provider #4

X

X

X

X

X

CPD, PIH

CPD,

Provider #5

X

X

X

X

X

X

Housing

Provider #6

X

X

X

X

X

CPD, PIH

CPD,

FHEO,

Provider #7

X

X

X

X

Housing,

PD&R, PIH

CPD,

Provider #8

X

X

X

X

Housing,

PD&R, PIH

CPD,

Provider #9

X

X

X

X

FHEO,

Housing

Provider

CPD,

X

X

X

X

#10

PD&R, PIH

Provider

X

X

X

X

FHEO, PIH

#11

Provider

X

X

X

CPD

#12

CPD = Office of Community Planning and Development. FHEO = Office of Fair Housing and Equal Opportunity. Housing =

Office of Housing. PD&R = Office of Policy Development and Research. PIH = Office of Public and Indian Housing. TA =

technical assistance.

Sources: Technical Assistance Portal “Work Plan File” received from HUD on August 9, 2018, and interviews A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 0 9

Troubled Public Housing Agency Interview Respondent

Characteristics

Twelve interviews were conducted for the case study on troubled PHAs: three with HUD

headquarters staff, five with TA providers, and four with PHA staff or board members who participated in the TA. The case study also drew on information from interviews conducted with two providers, two customers, and two HUD staff (HUD headquarters and field office) during an earlier study phase.

HUD Headquarters Staff

The selection of HUD headquarters staff in PIH for interviews was based on recommendations from PIH staff interviewed during the reconnaissance and case study design phases. Special consideration was given to staff respondents’ program expertise and experience working with troubled PHAs throughout the troubled designation process.

Technical Assistance Customers

PHAs were considered for inclusion if they received TA and had a troubled designation in either PHAS

or SEMAP anytime between 2011 and 2017, inclusive. In any given year, the vast majority of PHAs were nontroubled, between 2 and 5 percent were troubled, and between 1 and 7 percent were exempt from scoring.23 The number of troubled PHAs remained relatively constant during the time considered. Most troubled PHAs were troubled for 1 year (8.1 percent), as shown in exhibit 25, and agencies were more often designated as troubled on the basis of their SEMAP score rather than their PHAS score, as shown in exhibit 26. A small subset of troubled PHAs were troubled according to both PHAS and SEMAP scores.

23 Because scoring designations change each year, data are presented by year rather than summarized over the entire timeframe.

1 1 0

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

EXHIBIT 235: NUMBER OF YEARS TROUBLED

Years Troubled

Number of PHAs

Percent

0

3,466

88.9%

1

314

8.1%

2

67

1.7%

3

22

0.6%

4

17

0.4%

5

11

0.3%

6

1

0.0%

PHAs = public housing agencies.

Source: Urban Institute analysis of PHAS/SEMAP scores

EXHIBIT 246: NUMBER OF PUBLIC HOUSING AGENCIES WITH TROUBLED DESIGNATION

Fiscal Year

PHAs Troubled

SEMAP Troubled

Both Troubled

Total

2012

34

33

6

73

2013

38

144

8

190

2014

27

73

2

102

2015

36

44

2

82

2016

39

52

3

94

2017

51

45

6

102

PHAs = public housing agencies. SEMAP = Section Eight Management Assistance Program.

Source: Urban Institute analysis of PHAS/SEMAP scores

To the extent possible, PHAs were selected to reflect diversity in geography, PHA size (number of units), years troubled based on their PHAS or SEMAP scores, and number of TA work plans received (see exhibit 27). The initial sample of agencies included 12 potential case participants. After identifying the PHAs and confirming the list with HUD, the research team contacted the executive directors to request agency participation and ask for assistance identifying staff and board members who participated in the TA activities. Ultimately, four agencies agreed to participate in the study. All four were designated troubled on the basis of PHAS scores, and two of the four also were designated troubled on the basis of SEMAP scores.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 1 1

EXHIBIT 257: PUBLIC HOUSING AGENCY INTERVIEWEE CHARACTERISTICS BETWEEN FY 2011 AND FY

2018

Years PHAS

Years SEMAP

Number of TA Work Plans

Interviewee

Troubled

Troubled

Size

Received

PHA 1

1

NA

Large

3

PHA 2

2

1

Small

1

PHA 3

6

0

High Medium

1

PHA 4

7

3

High Medium

2

NA = Not applicable. PHA = public housing agency. SEMAP = Section Eight Management Assistance Program. TA = technical assistance.

Notes: HUD categorizes PHA size on the basis of the number of units as Very Small (0–49), Small (50–249), Low Medium (250–

499), High Medium (500–1,249), Large (1,250–9,999), and Very Large (10,000+). See HUD, PHA by Size Category document available at https://www.hud.gov/sites/documents/DOC_26196.PDF. PHAS/SEMAP score data spans FY 2011–FY 2018; TA Portal data spans FY 2012–FY 2017, so PHA interviewees may have received additional TA before FY 2012 or after FY 2017.

Source: Urban Institute analysis of PHAS/SEMAP scores

Technical Assistance Providers

TA providers were selected for interviews on the basis of findings from the administrative data analysis, findings from the in-depth interview phase, and recommendations from HUD staff.

Administrative data analysis was used to identify who provides the most TA to troubled PHAs in the number of work plans and large budget work plans. Providers were then selected on the basis of expertise, specialty, and recommendations from HUD. Two providers were selected for diversity in the characteristics of their TA customers (e.g., PHA size or geography). The first two providers selected had seven and two TA work plans in the TA Portal. Two additional providers were selected to better understand the Lead the Way training development. Following a couple of customer interviews, an additional provider was selected that had delivered TA to them to get as complete a picture of TA engagement as possible.

SNAPS Interview Respondent Characteristics

Sixteen interviews were conducted for the SNAPS case study: three HUD program staff in the SNAPS

office, six field staff who work in three CPD field offices, three providers who administer SNAPS TA, and four customers who have received SNAPS TA. The case study also drew on information from interviews with a HUD headquarters office staff member and a provider conducted during an earlier phase of the study.

1 1 2

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

The selection of TA providers, customers, and HUD field staff was made using TA Portal data.

Where contact information was missing or incomplete or the point of contact was no longer engaged with HUD TA, researchers used the available information to contact new staff or, in the case of the field offices, worked directly with SNAPS headquarters staff to identify the appropriate contact.

HUD Headquarters Staff

SNAPS headquarters staff were selected on the basis of recommendations from the director of the SNAPS TA program for staff who could speak to SNAPS’ strategic goals and decisionmaking. Sampled staff include the director of the SNAPS office, the SNAPS TA coordinators, and program officers.

HUD Field Office Staff

HUD field offices were selected on the basis of the amount and types of TA assigned to their field office (see exhibit 28), geographic diversity (Midwest, South, and West), and community type (two urban and one rural).

EXHIBIT 268: OFFICE OF COMMUNITY PLANNING AND DEVELOPMENT FIELD OFFICES BY HUD REGION, BY NUMBER AND TYPE OF TECHNICAL ASSISTANCE WORK PLANS, FY 2012–FY 2017

Field Office

Overall

On-Call TA

Direct TA

HUD Region

Field Office #1

123

34

89

9

Field Office #2

42

12

30

4

Field Office #3

19

10

9

7

TA = technical assistance.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

Technical Assistance Customers

SNAPS TA customers were chosen from a convenience sample based on the sampled field offices.

This approach allowed comparison of customers’ perspectives on specific TA engagements with the perspectives from field staff. Customers included one statewide CoC (i.e., a Balance of State), one CoC

for a smaller metropolitan area with relatively low population density, and one CoC for a larger, urban metropolitan area. This diversity allowed exploration of how the type of CoC (Balance of State or communitywide) may affect a CoC’s TA experience.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 1 3

Technical Assistance Providers

TA Portal data were used to identify the top 10 TA providers by dollar amount and number of work plans. After receiving input from SNAPS headquarters staff on the initial sample, the final sampling decision was made. The sample includes organizations that provide a variety of types of TA, including needs assessments and tools and products (see exhibit 29). One provider was selected because the organization specializes in administering homelessness TA rather than multiple types of HUD TA.

EXHIBIT 29: SNAPS TECHNICAL ASSISTANCE PROVIDERS BY NUMBER OF WORK PLANS, FY 2012–FY 2017

Self-Directed

Work

Tools and

and Group

Work

Plans, FY

On-

Products

Learning

TA

Plans, All

2016/ FY

Call

Direct

Needs

(Product

(Training

Providers

Years

2017

TA

TA

Assessment Development)

Delivery)

TA provider

196

99

9

99

8

46

34

#1

TA provider

154

86

9

77

3

51

14

#2

TA provider

152

63

9

97

0

10

36

#3

TA = technical assistance.

Source: TA Portal “Work Plan File” received from HUD on August 9, 2018

1 1 4

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

Appendix C. Troubled Public

Housing Agency Notification and

Technical Assistance Need

Identification Process

When PIH determines from at least one of the two assessment systems (PHAS, SEMAP, or both) that a PHA is troubled, the PHA must identify the actions it will take to remedy the identified deficiencies.

Troubled PHAs outline these actions in the recovery agreement and action plan mandated by HUD, including whether it will use any TA. Key steps of the notification and recovery process for troubled PHAs include the following:

◼

Real Estate Assessment Center assesses PHAs’ performance and provides a score and designation report to the PHAs on a rolling basis throughout the year.

◼

PIH reviews these new reports quarterly and sends notice to the field offices of newly designated troubled PHAs in an operational troubled list.

◼

Within 30 days after the field office has been notified, the field office sends a letter to the PHA, their board, and their local official with jurisdictional authority, providing them official notice of their troubled designation and guidance on next steps. Although this take-action letter requests a draft recovery plan within 30 days, HUD staff noted they often do not hear a response from the PHA.

◼

Approximately 30 to 60 days after the letter is sent, the field office must facilitate an assessment for the PHA to determine the type of actions a PHA needs to take to improve their score(s). In some circumstances, the field office conducts these assessments; however, they are often done by an independent assessment team.

◼

The troubled PHA then develops a recovery agreement and action plan that it executes with HUD. Plan development often is an iterative process, during which HUD headquarters may provide suggestions for the approaches and resources PHAs might use, including recommendations for TA. The PHA is not required to participate in any TA; however, they must remedy the issue causing the low performance score(s) and their troubled designation.

For each performance management issue to be addressed, the recovery agreement and action plan must include a strategy for improvement, a clear set of performance targets (including statutory measurements), and a date by which the goals will be achieved.

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

1 1 5

◼

If the PHA does not complete an activity in their recovery agreement and action plan, the field office will issue a notice of noncompliance, which can result in HUD taking action by forcing the PHA to coordinate with other entities to operate their units.

◼

In a few places in the troubled notification process, TA has been leveraged. TA providers have been brought in to conduct the required PHA needs assessments when an independent team is required. Also, TA providers have supported PHAs as they developed their recovery agreements and action plans, helping to identify rectifying actions and areas for which to request TA assistance and negotiating its execution with HUD. TA is often agreed upon in the plan, committing both HUD to provide the TA and the PHA to participate.

1 1 6

A S S E S S I N G H U D T E C H N I C A L A S S I S T A N C E P R O G R A M S

References

Barbee, Anita P., James DeSantis, and Tammy Richards. 2017. “APHSA Introduction to the Special Issue: Building Capacity in Child Welfare Systems,” Training and Development in Human Services: The Journal of the National Staff Development and Training Association 9 (1): 5–19.

Baumgartner, Scott, Adam Cohen, and Alicia Meckstroth. 2018. Providing TA to Local Programs and Communities: Lessons from a Scan of Initiatives Offering TA to Human Services Programs. Washington, DC: Mathematica Policy Research.

Blase, Karen A. 2009. Technical Assistance to Promote Service and System Change. Roadmap to Effective Intervention Practices #4. Tampa, FL: University of South Florida, Technical Assistance Center on Social Emotional Intervention for Young Children.

Crosby, Barbara C., and John M. Bryson. 2010. “Integrative Leadership and the Creation and Maintenance of Cross-Sector Collaborations,” The Leadership Quarterly 21 (2): 211–230.

Foster-Fishman, P.G., S.L. Berkowitz, D.W. Lounsbury, S. Jacobson, and N.A. Al en. 2001. “Building Collaborative Capacity in Community Coalitions: A Review and Integrative Framework,” American Journal of Community Psychology 29, 241–261.

General Accounting Office (GAO). 2002. HUD Management: Impact Measurement Needed for Technical Assistance. GAO-03-12. Washington, DC: GAO. https://www.gao.gov/assets/240/236157.pdf.

Hunter, Sarah B., Matthew Chinman, Patricia Ebener, Pam Imm, Abraham Wandersman, and Gery W. Ryan. 2009.

“Technical Assistance as a Prevention Capacity-Building Tool: A Demonstration Using the Getting To Outcomes Framework,” Health Education & Behavior 36 (5): 810–828.

Litwin, Janice, Janet Lansberry, William Slotnik, and John Vaughn. 1994. “Getting the Most From Technical Assistance.” Shelterforce (July 1). https://shelterforce.org/1994/07/01/getting-the-most-from-technical-

assistance/.

National Resource Center. 2010. Delivering Training and Technical Assistance. Washington, DC: National Resource Center.

https://www.google.com/url?esrc=s&q=&rct=j&sa=U&url=https://pdf4pro.com/cdn/delivering-training-and-technical-assistance-564d16.pdf&ved=2ahUKEwjmwbfF1-

fxAhXGqp4KHcSXA2YQFjAAegQIChAB&usg=AOvVaw3L1nOnQd2FJ7y2R7B-M0J7.

Richards, Tammy, Michelle Graef, Kathy Deserly, Peter Watson, and Mark Ells. 2017. “Structure and Service Delivery Approach of the Children’s Bureau’s Resource Centers and Implementation Centers,” Training and Development in Human Services 9 (1): 8–14.

Scally, Corianne Payton, Samantha Batko, Susan J. Popkin, and Nicole DuBois. 2018. The Case for More, Not Less: Shortfal s in Federal Housing Assistance and Gaps in Evidence for Proposed Policy Changes. Washington, DC: Urban Institute. https://www.urban.org/research/publication/case-more-not-less-shortfalls-federal-

housing-assistance-and-gaps-evidence-proposed-policy-changes.

Soler, Mark, Joseph J. Cocozza, and Alan Henry. 2013. Providing and Receiving Technical Assistance: Lessons from Models for Change. Washington, DC: Center for Children’s Law and Policy and National Center for Mental Health and Juvenile Justice. http://www.modelsforchange.net/publications/591.

Stevenson, John F., Paul Florin, Dana Scott Mills, and Marco Andrade. 2002. “Building Evaluation Capacity in Human Service Organizations: A Case Study,” Evaluation and Program Planning 25 (3): 233–243.

Sullivan, Brian. 2013. “HUD Awards $25 Million to Help Local Communities Tackle Homelessness and Carry Out

‘Place-Based’ Development.” HUD Archives: News Releases. Washington, DC: U.S. Department of Housing and Urban Development. https://archives.hud.gov/news/2013/pr13-137.cfm.

Theodos, Brett, G. Thomas Kingsley, Eric Burnstein, Sara McTarnaghan, Christina Plerhoples Stacy, and Ruth Gourevitch. 2018. Strong Cities, Strong Communities National Resource Network Program Evaluation.

Washington, DC: U.S. Department of Housing and Urban Development.

Tyson, S., and M. McNeil. 2009. “How to… Provide Effective Technical Assistance.” BJOG: An International Journal of Obstetrics & Gynaecology

116 (Suppl. 1): 93–95.

U.S. Department of Housing and Urban Development. Community Planning and Development (HUD). 2014.

HUD Community Compass Technical Assistance and Capacity Building FR-5800-N-12 (FY2014 NOFA).

https://www.hud.gov/sites/documents/2014HCCTANOFA.PDF.

———. 2015. Community Planning and Development.

HUD Community Compass Technical Assistance and Capacity Building Program FR-5900-N-06 (FY2015

NOFA). https://www.hud.gov/sites/documents/2015CCTACBNOFA.PDF.

———. 2016. Community Planning and Development.

HUD Community Compass Technical Assistance and Capacity Building Program FR-6000-N-06 (FY2016

NOFA). https://www.hud.gov/sites/documents/2016CCTACBNOFA.PDF.

———. 2017. Community Planning and Development.

HUD Community Compass Technical Assistance and Capacity Building Program FR-6100-N-06 (FY2017

NOFA). https://www.hud.gov/sites/documents/2017CCTANOFA.PDF.

———. 2018. Community Planning and Development. HUD's FY 2018 and FY

2019 Community Compass Technical Assistance and Capacity Building Program Notice of Funding Availability (NOFA) FR-6200-N-06 (FY2018/2019 NOFA). https://www.hud.gov/sites/dfiles/SPM/documents/FR-6200-

N-06-CCTA_CBP.pdf.

———. 2020. Community Planning and Development. HUD’s FY 2020 and FY 2021 Community Compass Technical Assistance and Capacity Building Program Notice of Funding Availability (NOFA) FR-6400-N-06.

https://www.hud.gov/sites/dfiles/SPM/documents/HUD_FY2020andFY2021_ccta.pdf.

White, Donna. 2013. “HUD Awards $7.3 Million to Provide Training, Affordable Housing Consulting for Native Americans, Alaskans and Hawaiians.” HUD Archives: News Releases. Washington, DC: U.S. Department of Housing and Urban Development. https://archives.hud.gov/news/2013/pr13-025.cfm.

Additional Reading

Norton, Sophie, Andrew Milat, Barry Edwards, and Michael Giffin. 2016. “Narrative Review of Strategies by Organizations for Building Evaluation Capacity,” Evaluation and Program Planning 58: 1–19.

Spadaro, Antonia J., Jo Anne Grunbaum, Nicola U. Dawkins, Demia S. Wright, Stephanie K. Rubel, Diane C. Green, and Eduardo J. Simoes. 2011. “Training and Technical Assistance to Enhance Capacity Building Between Prevention Research Centers and Their Partners,” Preventing Chronic Disease 8 (3): A65.

http://www.cdc.gov/pcd/issues/2011/may/10_0093.htm.

West, Gary R., Sheila P. Clapp, E. Megan Davidson Averill, and Willard Cates Jr. 2012. “Defining and Assessing Evidence for the Effectiveness of Technical Assistance in Furthering Global Health,” Global Public Health 7 (9): 915–930.

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Washington, DC 20410-6000

September 2021

Document Outline

Assessing HUD Technical Assistance Programs

Foreword

Contents

List of Exhibits

Acknowledgments

Executive Summary

Findings

Technical Assistance Customers and Providers

The Technical Assistance Process: Step by Step

Timelines and Costs

Communication, Coordination, and Collaboration

Monitoring and Evaluation

Perceived Impacts

Recommendations for Moving Forward

Improve Communication and Coordination

Develop Efficient and Flexible Work Plans

Increase Access to Work Products

Improve Data Systems and Data

Track Clear Outcomes

I. Introduction

HUD Technical Assistance Objectives

HUD Technical Assistance Types

Community Compass Overview

Existing Research and Study Contribution

Organization of the Report

Definitions of Key Terms

II. Study Design

Research Questions

Data

Administrative Data and Variables

Interview Data

Document Data

Sampling and Case Study Selection

Technical Assistance Programs and Interview Respondents

Case Studies

Analysis

Study Limitations

Technical Assistance Customers

Costs

Work Plans

III. Technical Assistance Customers and Providers

Introduction

Key Takeaways

Customers

Providers

Technical Assistance Customers

Customer Characteristics

Customer Capacity

Technical Assistance Providers

Provider Characteristics and Selection

Provider Work Plan and Budget Amounts

Budget Share of Top Five Providers

HUD Offices as Technical Assistance Providers

IV. The Technical Assistance Process: Step by Step

Introduction

Key Takeaways

Notice of Funding Availability and Provider Selection Process

Technical Assistance Requests and Need Identification

Provider Assignment and Work Scope

Technical Assistance Engagement

Closeout

Process Overview

Notice of Funding Availability and Provider Selection Process

Technical Assistance Requests and Need Identification

Targeted Technical Assistance

Universal Technical Assistance

Required Technical Assistance

Technical Assistance Requests Not Approved

Provider Assignment and Work Scope

Technical Assistance Engagement

Targeted Technical Assistance Compared with Universal Technical Assistance

Targeted Technical Assistance

Universal TA

Delivering and Managing Technical Assistance Engagements

Closeout

V. Timelines and Costs

Introduction

Key Takeaways

Timeliness of the Technical Assistance Process

Types of Technical Assistance and Costs Over Time

Timeliness of the Technical Assistance Process

Request Approvals, Provider Assignments, and Work Plan Approvals

Period of Performance and Length of Technical Assistance Engagements

Types of Technical Assistance and Costs over Time

VI. Communication, Coordination, and Collaboration

Introduction

Key Takeaways

Cross-Program Collaboration at HUD

Shifting Relationship and Roles: HUD Field Offices

Communication Between Technical Assistance Providers and HUD

Coordination and Collaboration Across Technical Assistance Providers

Cross-Program Collaboration at HUD

Shifting Relationship and Roles: HUD Field Offices

Communication Between Providers and HUD

Coordination and Collaboration Across Technical Assistance Providers

VII. Monitoring and Evaluation

Introduction

Key Takeaways

Reporting Processes

HUD Oversight

Evaluation

Reporting Processes

HUD Oversight

Evaluation

Choosing the Right Metrics

Evaluating Outcomes: Troubled Public Housing Agency Technical Assistance

Collecting Informal Feedback

Ongoing Evaluation Efforts

Feedback Surveys

VIII. Perceived Impacts

Introduction

Key Takeaways

Customer Perceptions

Provider Perceptions

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance

Customer Perceptions

Satisfaction with Technical Assistance Engagement

Impact of Technical Assistance

Provider Perceptions

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance

IX. Recommendations for Moving Forward

Introduction

Improve Communication and Coordination

Customers

Providers

Across HUD

Develop Efficient Work Plans

Increase Access to Work Products

Improve Data Systems and Data

Document and Update Systems

Improve Data Quality

Track Clear Outcomes

X. Summary of Findings and Future Research

Introduction

Summary of Key Findings

Future Evaluation and Research

Appendix A. Community Compass Offices, Divisions, Grant Programs, and Appropriation Accounts

Appendix B. Additional Information on Study Implementation

Identifying Troubled Public Housing Agency Technical Assistance Customers

Identifying SNAPS Technical Assistance Customers

In-Depth Interview Respondent Characteristics

Respondents by Technical Assistance Type and Program Office

Respondents by Stakeholder

Troubled Public Housing Agency Interview Respondent Characteristics

HUD Headquarters Staff

Technical Assistance Customers

Technical Assistance Providers

SNAPS Interview Respondent Characteristics

HUD Headquarters Staff

HUD Field Office Staff

Technical Assistance Customers

Technical Assistance Providers

Appendix C. Troubled Public Housing Agency Notification and Technical Assistance Need Identification Process

References

Additional Reading

Table of Contents

Assessing HUD Technical Assistance Programs

Foreword

Contents

List of Exhibits

Acknowledgments

Executive Summary

Findings

Technical Assistance Customers and Providers

The Technical Assistance Process: Step by Step

Timelines and Costs

Communication, Coordination, and Collaboration

Monitoring and Evaluation

Perceived Impacts

Recommendations for Moving Forward

Improve Communication and Coordination

Develop Efficient and Flexible Work Plans

Increase Access to Work Products

Improve Data Systems and Data

Track Clear Outcomes

I. Introduction

HUD Technical Assistance Objectives

HUD Technical Assistance Types

Community Compass Overview

Existing Research and Study Contribution

Organization of the Report

Definitions of Key Terms

II. Study Design

Research Questions

Data

Administrative Data and Variables

Interview Data

Document Data

Sampling and Case Study Selection

Technical Assistance Programs and Interview Respondents

Case Studies

Analysis

Study Limitations

Technical Assistance Customers

Costs

Work Plans

III. Technical Assistance Customers and Providers

Introduction

Key Takeaways

Customers

Providers

Technical Assistance Customers

Customer Characteristics

Customer Capacity

Technical Assistance Providers

Provider Characteristics and Selection

Provider Work Plan and Budget Amounts

Budget Share of Top Five Providers

HUD Offices as Technical Assistance Providers

IV. The Technical Assistance Process: Step by Step

Introduction

Key Takeaways

Notice of Funding Availability and Provider Selection Process

Technical Assistance Requests and Need Identification

Provider Assignment and Work Scope

Technical Assistance Engagement

Closeout

Process Overview

Notice of Funding Availability and Provider Selection Process

Technical Assistance Requests and Need Identification

Targeted Technical Assistance

Universal Technical Assistance

Required Technical Assistance

Technical Assistance Requests Not Approved

Provider Assignment and Work Scope

Technical Assistance Engagement

Targeted Technical Assistance Compared with Universal Technical Assistance

Targeted Technical Assistance

Universal TA

Delivering and Managing Technical Assistance Engagements

Closeout

V. Timelines and Costs

Introduction

Key Takeaways

Timeliness of the Technical Assistance Process

Types of Technical Assistance and Costs Over Time

Timeliness of the Technical Assistance Process

Request Approvals, Provider Assignments, and Work Plan Approvals

Period of Performance and Length of Technical Assistance Engagements

Types of Technical Assistance and Costs over Time

VI. Communication, Coordination, and Collaboration

Introduction

Key Takeaways

Cross-Program Collaboration at HUD

Shifting Relationship and Roles: HUD Field Offices

Communication Between Technical Assistance Providers and HUD

Coordination and Collaboration Across Technical Assistance Providers

Cross-Program Collaboration at HUD

Shifting Relationship and Roles: HUD Field Offices

Communication Between Providers and HUD

Coordination and Collaboration Across Technical Assistance Providers

VII. Monitoring and Evaluation

Introduction

Key Takeaways

Reporting Processes

HUD Oversight

Evaluation

Reporting Processes

HUD Oversight

Evaluation

Choosing the Right Metrics

Evaluating Outcomes: Troubled Public Housing Agency Technical Assistance

Collecting Informal Feedback

Ongoing Evaluation Efforts

Feedback Surveys

VIII. Perceived Impacts

Introduction

Key Takeaways

Customer Perceptions

Provider Perceptions

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance

Customer Perceptions

Satisfaction with Technical Assistance Engagement

Impact of Technical Assistance

Provider Perceptions

Quantifying Impacts: Troubled Public Housing Agency Technical Assistance

IX. Recommendations for Moving Forward

Introduction

Improve Communication and Coordination

Customers

Providers

Across HUD

Develop Efficient Work Plans

Increase Access to Work Products

Improve Data Systems and Data

Document and Update Systems

Improve Data Quality

Track Clear Outcomes

X. Summary of Findings and Future Research

Introduction

Summary of Key Findings

Future Evaluation and Research

Appendix A. Community Compass Offices, Divisions, Grant Programs, and Appropriation Accounts

Appendix B. Additional Information on Study Implementation

Identifying Troubled Public Housing Agency Technical Assistance Customers

Identifying SNAPS Technical Assistance Customers

In-Depth Interview Respondent Characteristics

Respondents by Technical Assistance Type and Program Office

Respondents by Stakeholder

Troubled Public Housing Agency Interview Respondent Characteristics

HUD Headquarters Staff

Technical Assistance Customers

Technical Assistance Providers

SNAPS Interview Respondent Characteristics

HUD Headquarters Staff

HUD Field Office Staff

Technical Assistance Customers

Technical Assistance Providers

Appendix C. Troubled Public Housing Agency Notification and Technical Assistance Need Identification Process

References

Additional Reading