Evidence Matters: Measuring Sustainability

Measuring Sustainability

Highlights

• Measuring sustainability is difficult because there is no universal definition and it encompasses many aspects of our society, economy, and governing institutions in addition to our interaction with the natural environment.
• Indicators can help measure key facets of sustainability to allow organizations to make decisions about how best to become more sustainable.
• Indicators can be bundled into indicator systems for a more comprehensive picture; they can also be aggregated into indices, but this raises concerns about the loss of precision.
• Location efficiency is a thriving area of research on metrics because it ties together the travel and economic impacts of the built environment.

Concerns about sustainability have become increasingly influential in shaping both government policy and the behavior of individuals and businesses. In unveiling PlaNYC, a long-term sustainability plan for New York City, Mayor Michael Bloomberg pronounced, “The science is there. It’s time to stop debating it and to start dealing with it.…Let’s recognize that many of the gains we have made in the quality of our air, water, and land will be lost — if we don’t act.”1 However, debate continues about which actions we should take, what our specific goals should be, and how we should measure progress. The 1983 World Commission on Environment and Development is generally thought to have inaugurated the international effort to answer these questions. The report that resulted from the conference — entitled Our Common Future but better known as the Brundtland report — originated the now-classic definition of sustainability: “[D]evelopment that meets the needs of the present without compromising the ability of future generations to meet their own needs.”2 This definition is high-level and open to interpretation, which probably has helped popularize sustainability but has also made it difficult for groups with different priorities to agree on a common course of action. The lack of consensus has also complicated attempts to measure the effectiveness of sustainability efforts and make comparisons across programs and geographies. A growing body of work on measurement, however, is providing the means to better document current conditions and demonstrate progress. This article will review the research on metrics — indicators, indicator systems, and indices — that attempts to resolve the challenges of measurement. Sustainability is commonly described as comprising three interdependent pillars or dimensions: social, economic, and environmental. (A fourth, governance or institutions, is often included as well). These dimensions, however, only outline the general scope of a measurement framework. Sustainability is a complex and subjective concept, so in order to measure it, organizations must first define what it means to them and then choose indicators that give the most relevant information about their system and time scale of interest.3

Sustainability Indicators

Sustainability indicators are derived from existing data sources that most closely approximate the qualities people are interested in measuring. Environmental quality, for instance, is typically described in terms of air and water quality, levels of greenhouse gas emissions, and biodiversity. Economic health is usually described using macro-level indicators such as GDP per capita, along with factors such as access to employment, education, and credit, which emphasize the role of individual opportunity. Social equity is measured in literacy rates, infant mortality, health and life expectancy, and crime rates. Indicators that apply to more than one dimension of sustainability can be more holistic; for instance, measuring the quality of a community’s wastewater treatment and its land-use patterns can help illustrate how environmental quality varies among different areas, providing information about socioeconomic conditions as well as the environment.

The Partnership for Sustainable Communities (PSC), a collaboration among HUD, the Department of Transportation, and the Environmental Protection Agency (EPA), recently undertook a study with the University of Pennsylvania School of Design to review and evaluate existing indicators, with the goal of developing a core set of indicators for widespread use by U.S. cities and regions. They compiled a list of 336 indicators contained in a collection of 22 indicator systems and then narrowed the list down to the 155 most useful ones. To better understand how this optimized set of indicators might be applied, they categorized them in two ways: by dimension (environmental, economic, or social) and according to the PSC’s six livability principles, which the participating agencies are using to guide investments in sustainability. Table 1 illustrates this rubric, including an example of an indicator for each theme.4

One universal difficulty is that the indicators that are available are often only a rough approximation of the physical quality being measured. Although the characteristic feature of an indicator is less-than-ideal accuracy, this imprecision has fed skepticism about the possibilities for measuring sustainability.5 This concern, however, arises from a misconception about the purpose of indicators. The use of sustainability indicators is patterned after ecologists’ use of indicator species to understand and track ecosystem health and gross trends, rather than precisely documenting natural systems. Likewise, the goal of sustainability indicators is to give organizations enough information to set objective, attainable goals for sustainability and then make evidence-based policy decisions that bring them closer to those goals rather than precisely documenting natural or human systems.6

Indicator Systems

Sustainability indicators can be more useful for planners and policymakers when they are bundled into indicator systems or aggregated into indices. Much has been done on this front; the Compendium of Sustainable Development Indicator Initiatives, which operates as “a worldwide directory of who is doing what in the field of sustainability indicators,” currently lists 894 systems and indices.7 When selecting a set of indicators to describe a complex system of any scale, one rule of thumb is parsimony: represent the scope and complexity of the system using the smallest possible number of indicators. In addition, several available frameworks can guide index assembly in terms of both process and substance (see “Criteria for Building Measurement Systems”).

As well as comprehensive, national-level indicator systems such as the Indicators of Sustainable Development published by the United Nations Department of Economic and Social Affairs, some systems have been developed recently that focus on more granular data, making them better suited for measuring urban or regional conditions and the effects of jurisdiction-level policies.8 The Global Cities Indicator Program, which has attracted significant and growing participation, offers a comprehensive menu of indicators that enables meaningful comparisons among 142 cities worldwide; the ICLEI USA STAR Community Index has similar aspirations but focuses on U.S. cities.9,10

Although these systems are designed to maximize usage and comparability, a different strategy is to look at sustainability thematically, creating indicator systems designed for specific disciplines or perspectives. For example, the Smart Mobility Framework from the California Department of Transportation (Caltrans) focuses on sustainability from the perspective of transportation planning, allowing it to concentrate on this sector while emphasizing its crosscutting implications. The framework encourages policymakers to consider six Smart Mobility Principles when making transportation decisions: location efficiency, reliable mobility, health and safety, environmental stewardship, social equity, and robust economy. It cites 17 goals associated with one or more principles and suggests several indicators for measuring each one.11 The framework was developed with funding from the EPA and has contributed to the agency’s just-released “Guide to Sustainable Transportation Performance Measures.”12

The Center for Clean Air Policy’s recently released report, Growing Wealthier, is another example of a thematic indicator system; it summarizes recent empirical research that connects the economic and emissions-reducing benefits of sustainable development with Smart Growth policies and then uses this as a framework for measuring sustainability. The report outlines 37 potential benefits from Smart Growth and categorizes them by type (return on investment, savings on expenditures, or improved quality of life) and primary beneficiary (businesses, households, municipalities/regions, or the entire country).13 While the Caltrans Smart Mobility Framework starts with a set of normative policy principles and provides a system for measuring their outcomes, Growing Wealthier aims to use demonstrable outcomes — the range of economic benefits measured in addition to reduced vehicle miles traveled (VMT) — as an argument for adopting sustainability principles in policymaking.

Sustainability Indices

A more concise alternative to indicator systems are indices, which combine two or more metrics or indicators mathematically to arrive at a single summary indicator. Table 2 summarizes five commonly referenced indices and illustrates the unavoidable tradeoffs of representing sustainability using any one value.14

An obvious difficulty is covering all three dimensions of sustainability in one index. Even systems that cover the same dimensions can differ widely in approach. Ecological Footprint (EF) and the Environmental Performance Index (EPI) both focus on the environmental pillar but consider it from different perspectives: EF through the lens of carrying capacity and EPI through that of policy. The Human Development Index (HDI) considers economic and social sustainability but overlooks the environment, whereas the Happy Planet Index (HPI) approximates the environmental costs of human quality of life but ignores the economic costs. The only one of the five that attempts to cover all three pillars of sustainability, the Genuine Progress Indicator (GPI), lacks a standardized methodology and requires a host of assumptions to impute the values of various noneconomic costs. All of these indices vary in scope, from an elegant but narrow accounting of the environmental cost of happiness (HPI) to a system that aspires to capture and monetize all (both internalized and externalized) costs and benefits of development (GPI). One possible way to make the most of the available tools is to use them in combination; for example, the Footprint Network suggests using EF in combination with HDI.15

A more fundamental problem is that few if any sustainability indices employ standard mathematical rules of indexing. In 2007 Böhringer and Jochem reviewed 11 sustainability indices (including the 5 described here) for mathematical validity and found several methodological flaws. They note “a high degree of arbitrariness without mentioning or systematically assessing critical assumptions” in the normalization and weighting of component metrics. Even more concerning, the indices largely neglect the mathematical rules that govern the aggregation of metrics into a single index.16 Even though these flaws cast serious doubt on the indices’ ability to inform policy, they are still widely used. Organizations should therefore avoid weighing their rankings in these indices too heavily in their decisionmaking.

Empirical Innovation: Metrics for Location Efficiency

As the field of sustainability measurement evolves, a number of efforts are underway to develop metrics that respond to specific ideas or theories. A prime example is location efficiency — the idea that locating housing near employment, schools, and commercial amenities in transit-rich and pedestrian-friendly environments shifts people’s travel behaviors toward less energy-intensive (and more healthful) transportation options such as public transit, walking, or biking. A recent meta-analysis by Reid Ewing and Robert Cervero looked at the findings from more than 200 studies on the effects of various characteristics of the built environment on travel behavior.17 They found that for a given location, the number and proximity of nearby jobs and the distance to downtown affected VMT nearly twice as much as any of the other factors studied (e.g., density, land-use mix, etc.). Meanwhile, measures of pedestrian connectivity — intersection density, percentage of four-way intersections, and distance to the nearest transit stop — had the greatest effect on transit usage.18

Although these findings support the salience of location efficiency, they provide neither a formula for measuring it nor a larger socioeconomic context. The Center for Neighborhood Technology’s Housing + Transportation Affordability Index (H+T), developed by compiling transit system or tract-level transportation data from across the United States, complements Ewing and Cervero’s work by showing the economic consequences of changes in travel behavior due to residential relocation. Because consumers undervalue transportation costs when choosing a place to live, they tend to move farther from job centers, pursuing lower housing costs at the expense of higher transportation costs. H+T shows that for the 337 metropolitan areas studied, any housing cost savings are negated by increases in transportation costs at commute distances of 10 miles or more.19

Although H+T may eventually make it possible to measure the true cost of location decisions nationwide, questions remain about its validity. A recent report commissioned by the National Association of Home Builders raised a number of concerns about H+T’s data. The report noted that the index uses some outdated or insufficiently detailed data, fails to incorporate transit service data, and uses housing data that apply to current, but not future, residents.20 Moreover, despite widespread adoption of the index, the logarithms used to calculate it are proprietary and cannot be reviewed by users. Nevertheless, H+T has enriched the conversation by demonstrating the connection between sustainability and affordability, and HUD is taking an active role in enhancing H+T’s applicability, announcing an effort to expand the database beyond the initial study areas and determine how it can be incorporated in HUD programs.21

The Walkable Urban Places (WUP) study currently being conducted by the Brookings Institution Metropolitan Policy Program is seeking to quantify location efficiency’s broader economic benefits. Using data from 66 regionally significant WUPs in the Washington, DC area, the research team is looking to determine which characteristics of urban places (such as land-use mix, built environment features, and transportation and pedestrian infrastructure) correlate with economic performance. In addition, they plan to integrate their economic performance metrics with social equity metrics and LEED for Neighborhood Development to produce a comprehensive, place-based sustainability metric. The goal is to allow communities to evaluate the contribution of their built environment in terms of the three dimensions of sustainability and use the results to make evidence-based decisions about how to grow and develop.

Next Steps in Metric Development

In September 2010 the Metropolitan Institute at Virginia Tech convened a research roundtable to identify top research priorities for the PSC. Three of the 12 items on the participants’ list of recommendations called for more work on sustainability metrics: research into how households and firms make location decisions given preferences and financial constraints, evaluation methods for identifying policies that have successfully fostered affordable housing in location-efficient neighborhoods; and research into the performance and cost-effectiveness of residential green retrofits by building characteristics (see the previous article, “Quantifying Energy Efficiency in Multifamily Rental Housing”).22 Because sustainability is contextual and lacks a universal definition, there may never be a single common index or indicator system to measure it. Nevertheless, as sustainability increasingly permeates the public and private sectors, rapidly advancing progress in the field will continue to provide policymakers with better tools for making data-driven decisions about how best to advance sustainability.

Related Information:

Developing a New Approach for Measuring Regional Sustainability
Criteria for Building Measurement Systems

1. Michael Bloomberg. “Michael Bloomberg Delivers PlaNYC: A Greater, Greener New York,” press release, 22 April 2007.
2. World Commission on Environment and Development. 1987. Our Common Future. Oxford: Oxford University Press, 43, as cited in International Institute for Sustainable Development, “What is Sustainable Development?” Accessed 19 August 2011.
3. Simon Bell and Stephen Morse. 2008. Sustainability Indicators: Measuring the Immeasurable? London, UK: Earthscan, 12.
4. Unpublished paper by Amy Lynch, Stuart Andreason, Theodore Eisenman, John Robinson, Kenneth Steif, and Eugenie L. Birch. 2011. “Sustainable Urban Development Indicators: State of the Art and its Potential Congruence with U.S. Department of Housing and Urban Development Sustainability Initiatives,” 21–2.
5. Helen Briassoulis. 2001. “Sustainable Development and its Indicators: Through a (Planner’s) Glass Darkly,” Journal of Environmental Planning and Management 44:3, 409–27.
6. Bell and Morse, 43.
7. “Compendium: A Global Directory to Indicator Initiatives.” International Institute for Sustainable Development website (www.iisd.org/measure/compendium). Accessed 16 May 2011.
8. United Nations Department of Economic and Social Affairs. 2007. Indicators of Sustainable Development: Guidelines and Methodologies. New York: United Nations.
9. “Participating Cities,” Global City Indicator Facility website (www.cityindicators.org/Participants.aspx). Accessed 15 June 2011.
10. ICLEI — Local Governments for Sustainability USA. 2010. “STAR Community Index Sustainability Goals & Guiding Principles.”
11. Ellen Greenberg, Jerry Walters, Richard Lee, and Jeffrey Ang-Olson. 2010. Smart Mobility 2010: A Call to Action for the New Decade, Sacramento, CA: California Department of Transportation, 8, 55.
12. See Environmental Protection Agency. 2011. “Guide to Sustainable Transportation Performance Measures.” Environmental Protection Agency website (www.epa.gov/smartgrowth/pdf/Sustainable_Transpo_Performance.pdf). Accessed 2 September 2011.
13. Chuck Kooshian and Steve Winkelman. 2011. Growing Wealthier: Smart Growth, Climate Change and Prosperity. Washington, DC: Center for Clean Air Policy.
14. For a more comprehensive (if slightly less current) listing, see Christoph Böhringer and Patrick E. P. Jochem. 2007. “Measuring the Immeasurable: A Survey of Sustainability Indices,” Ecological Economics 63:1, 1–8.
15. “Our Human Development Initiative.” Global Footprint Network website (www.footprintnetwork.org/en/index.php/GFN/page/fighting_poverty_our_human_development_initiative). Accessed 18 May 2011.
16. Böhringer and Jochem, 1–8.
17. Reid Ewing and Robert Cervero. 2010. “Travel and the Built Environment: A Meta-Analysis.” Journal of the American Planning Association 76:3, 265–94.
18. Ibid.
19. María Choca Urban. 2010. “Penny Wise, Pound Fuelish: New Measures of Housing + Transportation Affordability,” 10.
20. Abt Associates Inc. 2010. “A Methodological Review of the Center for Neighborhood Technology’s ‘Housing + Transportation Affordability Index.’”
21. HUD Launches Development of a National Housing and Transportation Affordability Index, https://archives.hud.gov/news/2011/pr11-180.cfm press release, 30 August 2011.
22. Casey Dawkins, Joseph Schilling, and Mariela Alfonzo. 2011. “Policy Research Priorities for Sustainable Communities: Insights and Ideas for the U.S. Department of Housing and Urban Development and the Federal Interagency Partnership for Sustainable Communities.”