PRB

(August 2011) On June 28, 2011, the Population Reference Bureau and the Annie E. Casey Foundation co-hosted a workshop to discuss opportunities and challenges presented by the U.S. Census Bureau’s American Community Survey (ACS), with a particular focus on the five-year ACS estimates.

The ACS is a nationwide, continuous survey designed to provide communities with reliable and timely demographic, housing, social, and economic data every year. The annual ACS sample is much smaller than that of the Census 2000 long-form sample, which included about 18 million housing units. As a result, the ACS needs to combine population or housing data from multiple years to produce reliable numbers for small counties, neighborhoods, and other local areas.

The five-year ACS estimates, first released in December 2010, are the heart of the ACS program and were designed to replace the decennial census long form. The purpose of this workshop was to bring together Census Bureau staff and data users to discuss the benefits and challenges of this new resource and strategies to make the estimates more useful to data users. Workshop participants represented end users involved in advocacy, government, business, and research at the national, state, and local levels, as well as several key staff members from the U.S. Census Bureau. Funding for the meeting was provided by the Annie E. Casey Foundation.

• View meeting agenda (PDF: 158KB)  View list of participants (PDF: 140KB)

Here are five key points made by participants at the meeting:

• The ACS is an extremely valuable resource that creates new opportunities for tracking U.S. social and economic trends by providing updated information for small geographic areas each year. However, there are many concerns about the quality of the ACS estimates for small geographic areas and small population subgroups, and using the ACS to measure change over time.
• The Census Bureau is currently producing a vast array of ACS data products and tables on an annual basis, putting a strain on staff resources.There is a possibility that some products could be modified or eliminated in the future, but the Bureau recognizes that there needs to be a formal process, including input from data users, to makes these changes.
• There are concerns about the large standard errors associated with the ACS estimates for small geographic areas and small population groups. Data users are dealing with these errors in different ways, depending on their particular applications and audiences. Some present margins of error along with every ACS estimate, while others ignore sampling error under certain circumstances. Strategies for ensuring the accuracy of the data included combining geographic areas, suppressing unreliable data, and validating the ACS data against other sources.
• Data user groups can assist the Census Bureau in their work. Data users can play an important role in helping the Census Bureau answer questions about which data products or tables are most useful, and which could potentially be trimmed back or eliminated. They can also point the Census Bureau to new tools, technical documentation, trainings, and other resources that would help people make better use of ACS data. For example, there was widespread interest in a statistical calculator that would simplify the calculation of standard errors.
• There are concerns about the new American FactFinder system. ACS data are scheduled to be transferred to the new American FactFinder system this fall. However, some data users find the new system to be confusing and difficult to navigate. There was a suggestion at the meeting that the Census Bureau should delay the transition to American FactFinder II until data users are comfortable with system’s new design/functionality.

Panelists Describe Their Experiences With the Five-Year Data

A panel of experts described some of the key issues around the five-year ACS data. The panel included staff from the Census Bureau as well as experts representing a diverse set of data user communities. Panelists discussed not only their concerns about the five-year ACS estimates but also the opportunities and untapped potential of the data.

Featured panelists included:

• Debbie Griffin, U.S. Census Bureau (PDF: 505KB)
• Ken Hodges, Nielsen Claritas (PDF: 302KB)
• Joe Salvo, New York City Department of City Planning (PDF: 3.3MB)
• Richard Rathge, North Dakota State University (PDF: 1.8MB)
• Leah Hendey, Urban Institute (PDF: 1.2KB)

Debbie Griffin, from the Census Bureau, kicked off the discussion by providing an overview of the five-year ACS data products that have been released thus far. The 2005-2009 ACS estimates covered essentially all of the topics available from the 2000 Census long form. For the five-year data, the Census Bureau produced 930 detailed tables for 670,000 geographic areas. The amount of data being released each year is putting a strain on Census Bureau staff and may not be sustainable in the long term. Therefore, one of the major questions going forward is how to pare down the scope of the one-, three-, and five-year data products, e.g., by putting limits on the number of tables, geographic areas, or products that are available. The challenge for the Census Bureau is that they serve diverse user communities with different needs. Debbie concluded with several steps the Census Bureau is taking to improve ACS operations as well as the quality of the data, including a systematic review of the entire ACS program, an expansion of the ACS sample from 2.9 million addresses to 3.5 million addresses per year, a new initiative to follow up with all nonrespondents in selected areas, and a reallocation of the ACS sample to improve estimates for small jurisdictions.

Ken Hodges, from Nielsen Claritas, discussed how the ACS estimates figure into their annual demographic updates. He addressed several challenges in working with the five-year ACS data, including the difficulty in building a time series of “point-in-time” estimates from ACS period data, choosing a base year for comparison when the “base” is updated each year, managing the large standard errors associated with the block group data, and the broader challenge of making the transition from long form data to the ACS. He also discussed the relative benefits and tradeoffs of using the one-, three-, and five-year ACS estimates in terms of the reliability and currency of the data.

Joe Salvo, from the New York City Department of City Planning, discussed a practical solution for overcoming the large standard errors associated with the five-year ACS tract-level data. He demonstrated the large margins of error associated with census tract-level data in New York City. By combining tracts into “neighborhood tabulation areas” (NTAs), he was able to dramatically reduce the sampling error associated with the neighborhood data. These data have been useful to policymakers trying to pinpoint the location of “fire traps,” characterized by lower-income immigrants living in crowded housing conditions.

Richard Rathge, from North Dakota State University, addressed the importance of data user communities in exploring limitations of the ACS data and solutions to common problems. He is concerned about the sufficiency of the five-year ACS data for measuring change in dynamic, small-area populations. For example, parts of the state are experiencing a population boom due to a rapid increase in crude oil production. Using county- and regional-level data for North Dakota as an example, he looked at different ways to triangulate ACS data with information from other sources to measure change in areas of rapid population growth. He stressed the importance of producing reliable data for small jurisdictions, which may have relatively small populations but are extremely important to local policymakers.

Leah Hendey, from the Urban Institute, described the National Neighborhood Indicators Partnership (NNIP) network and their experiences in working with the five-year ACS neighborhood data. The Urban Institute works with NNIP partners around the country to help build integrated neighborhood data systems, facilitate the use of ACS data by community and city leaders, and build the capacities of organizations and residents in distressed neighborhoods. To ensure the accuracy of the ACS data, she and her colleagues have stressed the importance of aggregating geographies when possible, being creative in selecting indicators, and verifying the data against other sources.

Survey of Data Users Highlights Specific Areas of Concern

PRB presented the results of a (nonscientific) survey that participants completed prior to the meeting (PDF: 137KB). The purpose was to collect input about how ACS data are being used, and key opportunities and challenges:

• The availability of annually updated data was listed as the top benefit of the ACS data.
• Issues related to large standard errors in the data for small geographic areas were the key areas of concern among ACS data users. Others mentioned the difficulties in assessing change over time when using multiyear estimates. Starting this fall, the ACS estimates will be controlled to data from the 2010 Census, which could create additional challenges for data users, especially those tracking population counts. Percentages should be less affected by this change in population controls.
• Data users deal with sampling error in the ACS in different ways, depending on their purpose and their particular audiences. Some present margins of error associated with every estimate, while at the other end of the scale, some ignore sampling error altogether. Other strategies for ensuring the accuracy of the data included combining geographic areas, suppressing unreliable data, and validating the data against other sources.
• Several of the participants mentioned that the new American FactFinder is confusing and that they prefer the interface of the legacy FactFinder system. Some data users also find the sheer volume of ACS data available to be overwhelming.
• The majority of the participants had used the Census Bureau’s ACS Compass Handbooks, mostly to look up formulas or as a reference guide. Many thought it would be useful for the Census Bureau to update these handbooks (e.g., with new five-year data or screenshots from the new American FactFinder).
• Several participants would like the Census Bureau to develop a statistical calculator that could simplify the process of calculating margins of error. Others would like to be able to combine geographic areas within the FactFinder system.

Strategies for Moving Forward

After lunch, we broke into small groups to discuss key ACS data issues in more detail, and group leaders summarized their discussions for the larger group.

One of the biggest concerns with the ACS is the large sampling errors associated with data for small geographic areas. What tools or resources could the Census Bureau or others provide to help those working with the data for small geographic areas or small population groups?

• Develop a statistical calculator to simplify calculation of standard errors for aggregated/derived estimates.
• Provide guidance to help data users answer the question: What is an acceptable margin of error?
• Enable data users to calculate special tabulations from the full ACS microdata file (e.g., through a full-sample Public Use Microdata Sample).
• Provide guidance on how to handle tables with “zeros” in many cells.
• Provide guidance on working with confidence intervals in mapping applications.

How could the Census Bureau’s current ACS data products be improved?

• Work with data users to help determine which products are most important.
• Consider retaining the old American FactFinder system, or key elements from that system.
• Continue to produce collapsed tables in FactFinder, in addition to detailed tables.
• Evaluate filters that are currently used in American FactFinder to limit results (e.g., based on population size).
• Develop a simplified front-end to American FactFinder to give casual users quicker access to higher-level information. Provide a separate interface for veteran data users who do not need as much help.
• For some data products, consider updates every other year instead of annual updates.
• Collapse some tables and expand others with input from the data user community.
• Make more data available to third parties that could create customized data products.

Are there particular ACS resource guides, technical documents, or training materials that would be useful in your work?

• Provide more examples of best practices or success stories in using ACS estimates.
• Although staffing may be an issue, consider writing more reports summarizing key findings in the ACS data.
• Write an ACS blog that highlights key findings from the data. This blog could potentially be maintained outside of the Census Bureau, and could be a source for data highlights as well as technical issues and information about the ACS.
• Continue to provide guidance to data users who want to compare the ACS with other data resources (e.g., how ACS tables compare with those from the 2000 Census SF3 data tables).
• Streamline the website to make ACS materials easier to find.
• Provide more online materials around the ACS poverty estimates.
• Provide additional webinars/training opportunities about ACS data.

Would it be useful to form an ongoing ACS data user group, and how could such a group be organized/structured?

• First, we need to define what is meant by “user group.” In establishing an ACS data user group, consider the responsibilities of each party and establish clearly defined goals. The Census Bureau may not be able to take a lead role in establishing this group but may be able to provide some monetary support to help cover administrative costs.
• Use the ACS data user group to facilitate communication with the Census Bureau—both channeling information to the Census Bureau and helping the Bureau communicate more effectively with the ACS user community
• Establish the data user group quickly in order to help the ACS staff in their program review effort. In the short term, the Census Bureau needs input from data users to help provide a detailed review of the data product line. This is an ambitious effort because everyone has different data needs.
• Establish a group that can provide practical, user-oriented advice on specific data products, table shells, and operations issues (not methods). A point person could summarize comments from the broader group in order to reduce duplication of efforts. The data user group should be able to meet and provide guidance in a short timeframe.
• Decide how group members should be selected, what is their tenure, and what is the right size for the group? Members should represent different data user communities with a broad range of skills.
• Use state data centers or other intermediaries to communicate results of the meetings to the broader ACS data user community.
• Convene meetings/conferences so that data users can learn from each other.
• Establish an e-mail list/website where ACS data users can share information and ideas.
• Consider the Association of Public Data Users (APDU) as a starting point for identifying members of the data user group. APDU has a broad and diverse membership and a history of working with the Census Bureau.
• Look toward other data user communities, such as Friends of NCHS, for ideas about how to increase public support for ACS data.
• Consider organizing a two-to-three day ACS user conference. Such a conference could provide research based on the ACS, discuss needs and desires, and provides a mechanism to bring data users and providers together to learn. It would also provide an opportunity for Census Bureau staff to talk among themselves about these issues. Data users would decide on the topics to be addressed at the conference, rather than Census Bureau staff.