APPAM Conference Highlights Data Tools

APPAM Conference Highlights Data Tools

Mark Shroder, Associate Deputy Assistant Secretary for Research, Evaluation, and Monitoring.

At the 2019 Association for Public Policy Analysis and Management (APPAM) Fall Research Conference, I attended presentations that highlighted four different tools to collect, organize, and analyze data. As both a consumer and a researcher, I find that some questions come quickly to mind when considering the use of a data tool: For what purpose would this tool be used? Who would use it? For how long are the data valid, and could the tool be reused in other contexts? These presentations took me out of my intellectual comfort zone, which is always a good thing, and I will try to focus my discussion of this event by taking a consumer’s viewpoint throughout.

Policy Mapping

The first tool that I will examine is policy mapping, a technique to visualize relationships and interdependencies across different governments and government agencies. Athena Tang, a lead associate at Booz Allen Hamilton, discussed how policy mapping could be used to analyze the federal statutes authorizing and delimiting telehealth policy — the provision of health services through electronic and telecommunication technologies, often over long distances.

Policy mapping visualizes both the present and missing connections among federal agencies that work in telehealth. In this context, this tool is useful for both the private and public sectors. A vendor of telehealth devices might use the tool to develop an overview of prospective buyers and regulators to sell their products more effectively and serve more customers. Legislators who are considering telehealth issues could use a policy map to understand other laws that already govern the field to inform their bill-writing process. If government leaders, such as the Secretary of the U.S. Department of Health and Human Services (HHS), were to propose a rule or take an action governing telehealth, they could use this tool to decide what stakeholders should be at the first meeting. Finally, policy mapping could be useful should HHS decide to reorder the agencies governing telehealth to shorten the learning curve of the staff undertaking that task.

This tool could be useful to analyze policies in other areas, although in affordable housing or community development, the agency overlaps may often be too limited to be worth mapping. When the overlaps are worth mapping, the data on the agencies involved and their priorities, are likely to change somewhat over time, meaning that the data would need to be updated regularly for the tool to remain effective. To help the data remain updated, the Policy Mapping tool uses algorithms to catalogue, digitize, and compare policy documents.

National Equity Research Database

A second tool is the National Equity Research Database (NERD), which translates American Community Survey (ACS) data from census tracts to defined neighborhood geographies and allows users to examine the data for each ethnic group in a given neighborhood. Clemens Noelke, a research scientist at Brandeis University, presented an update on the current state of NERD, which is currently publicly available only in Boston.

NERD is the descendant of a series of exercises conducted by the Kirwan Institute at Ohio State University to support local or federal desegregation or upward mobility efforts. A team at Brandeis expanded and standardized this approach to create NERD. The creators designed NERD to empower policymakers to find socioeconomic indicators of equity and upward mobility at the neighborhood level. NERD presents various measures of neighborhood inequality in America based on the understanding that neighborhood inequality is multidimensional, and a single variable, such as poverty or racial concentration, doesn’t capture it. Because NERD can also sort by racial or ethnic group, it allows policymakers to see how the tracked indicators affect different groups that live in the same community.

ACS data are updated annually, providing a steady stream of data for the NERD tool to use. Because other sources offer data at the census tract level rather than at the neighborhood level, NERD could be useful in other contexts to provide a more neighborhood-focused view of inequality.

Although NERD offers multiple cross-sectional measures of inequality, other tools developed by Raj Chetty and his colleagues at Harvard University rate neighborhood opportunity using outcomes of young people who grew up in those neighborhoods over two decades. These tools might be used alongside NERD to create a more comprehensive understanding of neighborhood inequality.

Randomized Messaging Experiments

Randomized messaging experiments are the third tool that I will consider. Shruthi Arvind of the Philadelphia Department of Revenue discussed how her city used a series of randomized messaging experiments to evaluate three interventions to identify people with delinquent property taxes, notify them about programs for reducing their tax liability, and enroll them in affordable repayment plans. According to Arvind, the randomized messaging experiments revealed that the combination of mailing invitations and hosting community events was a more effective way to enroll people in targeted, opt-in benefits programs such as the city’s Owner-Occupied Payment Agreement (OOPA) program than traditional canvassing. The results of the experiments also indicated that canvassing in high-need neighborhoods resulted in a slight increase in enrollment in programs such as the Homestead Exemption, and redesigned notices using plain language reduced the rate of agreement breaches for lower-income homeowners participating in the city’s OOPA program.

These findings should encourage policymakers to conduct more experiments to understand the best ways to serve their constituents. The report’s authors argue that the experiments not only found a better way to serve community members but also, because it improved participation in a program that increased tax revenue, saved the city money in the long term and demonstrated the possibility of a novel form of community outreach that might not have been considered without an experiment to prove its viability.

Randomized messaging experiments could be useful in measuring the impact of many other types of outreach and engagement efforts as well as determining whether a public policy problem could be addressed by a simple change in communications methods. HUD’s Office of Policy Development and Research has conducted several randomized messaging experiments to determine how changes in communications strategy affect college enrollment among young people in HUD-assisted households.

Although this tool does not rely on any updates of outside data sources, the tool requires the collection and analysis of data related to the effectiveness of the messaging experiments.

Infrastructure Visualization

The final tool that I will report is a visualization tool. Riley Sandel, a graduate student at Purdue University-Indianapolis, demonstrated a visualization tool that allows people to see images of proposed physical improvements and choose among several infrastructure options. Sandel used the tool in his research to display two potential street views to participants, asking respondents to select between the two. The tool helped Sandel understand how residents’ fear of gentrification might affect their views of potential improvements.

A visualization tool could be useful for governments or other planning bodies that are gathering community input on proposed projects. By providing a vision and offering different options from which to choose, the tool may help community members see change in a positive light rather than reject change altogether. In addition, policymakers who are considering several approaches to address a particular issue, such as improved safety for cyclists, could use this tool to find out which improvements will garner the least public opposition while still meeting policy goals.

Policymakers, planners, or others using the visualization tool would need to have information on each possible improvement to use the visualization tool. In addition, for the visualization tool to help leaders gauge public support for improvements, the tool would need to be coupled with efforts to collect and analyze data on public reactions. Some local officials may find this tool helpful; it is hard for me to see how Federal or state officials could benefit.

As with all data tools, the responsibility to make something of the information lies with the user. APPAM panelists and their teams demonstrated how innovative tools can help researchers examine data and support policymakers, planners, governments, and the private market in their various efforts. As a consumer, one must ask of each tool not just if it is useful for the task at hand but also whether it is likely to improve decision-making.