Partnering for the People: Improving the U.S. Census Bureau s MAF/TIGER Database

Transcription

1 Partnering for the People: Improving the U.S. Census Bureau s MAF/TIGER Database Frederick R. Broome and Leslie S. Godwin Abstract During the past few years, the term partnership has been elevated in the Federal government to the state of an integral business practice, and rightly so. However, some Federal agencies have been partnering successfully for many decades. Two such agencies, the Department of Commerce s U.S. Census Bureau and the Department of the Interior s U.S. Geological Survey (USGS), have had partnerships that resulted in fundamental changes to the field of geographic data processing. While these agencies had cooperated successfully on projects earlier, from 1983 to 1989, they conducted a major partnership that resulted in the creation of the USGS 1:100,000-scale Digital Line Graph (DLG) and the Census Bureau s Topologically Integrated Geographic Encoding and Referencing (TIGER) database. These datasets changed not only Federal geographic data processing practices but also set the course for future geographic data processing in state, local, and tribal governments, as well as in academia and private industry. Now, both agencies have embarked on ambitious programs to improve and evolve their datasets to meet more demanding Federal and other needs for the coming decades. The Census Bureau is improving its combined Master Address File (MAF) and TIGER database through an effort known as the MAF/TIGER Accuracy Improvement Project (MTAIP). Simultaneously, the USGS is undertaking the creation of The National Map, the initial foundation of which is existing datasets such as the National Hydrography Dataset (NHD). Although each agency s program objectives are unique to their respective missions, the products of their labors are, once again, complementary. Given their history of partnership, it was logical for the Census Bureau and the USGS to again share risks and results. Commonalities between their respective databases and missions have been identified and an effort launched to reduce redundancy in data collection and preparation, while still assuring the objectives of the resulting databases are achieved and meet the overarching objectives of the National Spatial Data Infrastructure (NSDI). Introduction Partnership is defined as the sharing of the risks and benefits from an activity between two or more participants. During the past few years, the word partnership has been elevated in the consciousness of Federal agencies and is now an integral business practice and rightly so. For decades, many Federal agencies have confronted two serious problems: shrinking F.R. Broome is with the National Geographic Partnerships Team, Geography Division, and L.S. Godwin is with the Decennial Systems and Contracts Management Office, both at the U.S. Census Bureau, GEO-7400, 4700 Silver Hill Road, Washington, DC (frederick.r.broome@census.gov; leslie.s.godwin@census.gov). production staffs and relatively shrinking budgets. Further, the technology and skill required to perform many of the unique tasks formerly performed largely by Federal agencies have become commonplace at state and local levels and within private firms. Therefore, individual agencies have turned towards partnerships with each other and with other sectors as an efficient and practical way to perform their missions. Nowhere has this shift from lone undertakings to partnerships been more apparent than in geographic data collection, processing, and dissemination. Two agencies, the Department of Commerce s U.S. Census Bureau and the Department of the Interior s U.S. Geological Survey (USGS), have had partnerships that have resulted in fundamental changes in the entire field of geographic data processing. While they had cooperated successfully on projects for many years, the major partnership they implemented from 1983 through 1989 resulted in the creation of the USGS 1:100,000-scale Digital Line Graph (DLG) and the Census Bureau s Topologically Integrated Geographic Encoding and Referencing (TIGER) database two datasets that changed not only Federal geographic data processing practices but also set the course for future geographic data processing in state, local, and tribal governments, as well as academia and private industry. In addition, the Census Bureau established thousands of geographic partnerships to enhance TIGER with state, local, and tribal government agencies in conjunction with both the 1990 census and Census Partnerships continue to be a cornerstone of the MAF/TIGER system. Now, the Census Bureau and the USGS are committed to improving and evolving their datasets to meet more demanding Federal and other needs for the coming decades. The Census Bureau is improving its combined Master Address File (MAF) and TIGER database through an effort known as the MAF/TIGER Accuracy Improvement Project (MTAIP). Simultaneously, the USGS is undertaking the creation of The National Map, the foundation of which is existing datasets such as digital orthophotographs and the National Hydrography Dataset (NHD). Although each agency s program objectives are unique to their respective missions, the products of their labors are, once again, complementary. TIGER A Partnership of National Scope After the 1980 Decennial Census of Population and Housing, as indeed after each census, the Census Bureau reviewed the problems associated with the production and delivery methods Photogrammetric Engineering & Remote Sensing Vol. 69, No. 10, October 2003, pp /03/ $3.00/ American Society for Photogrammetry and Remote Sensing PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING October

2 used to provide three basic geographic products: maps, address reference files, and geographic reference files, and their quality, accuracy, consistency, and currentness. It was clear that the Census Bureau could not operate in the future as it had in the past. It could no longer treat these products as separate entities to be produced through a series of independent processes. Out of this analysis grew the concept of a single, integrated, digital database that would be the sole source for all maps, address information, and geographic reference data. The file would contain all the physical features commonly found on Census Bureau field maps (e.g., roads, railroads, hydrography, and selected landmarks); all city delivery addresses associated with the roads; and all the geographic boundaries, codes, and names for the governmental units and statistical areas. This database, along with all the software, hardware, operations, and administrative activities, was named the TIGER System. The Census Bureau quickly realized that it could not complete a task this large without partners. Partnerships were not new to the Census Bureau. Past geographic partnerships included working with state, local, and tribal governments to gather boundary and annexation data and Federal and local level partnerships that built the Address Coding Guides of the late 1960s and the GBF/DIME files 1 of the 1970s. The Census Bureau had even entered into geographic research partnerships with academia, industry, and local governments, such as the New Haven Census Use Study ( ) through which many of the principles and practices later incorporated into the GBF/DIME files and the follow-on TIGER database were developed. However, this time (about 1983) the Census Bureau also needed a partner with experience in building geographic databases of national scope. The USGS had just such an effort underway to produce a new map series at 1:100,000-scale, with scannable symbology to create a digital database of features, the DLG. However, the Census Bureau needed to produce maps at a wide variety of scales, from 1:500,000 or smaller to 1:2,500 or larger. The two agencies tested and proved the feasibility of using data from a 1:100,000-scale source to generate maps at scales as large as 1:2,500 for census applications. Based on this and other information, Census Bureau management made the commitment to use the USGS DLG to achieve a nationwide standard of quality and content that would serve as the base for TIGER in the more than 95 percent of the Nation that had no GBF/DIME file coverage, while continuing to use the GBF/DIME files for the larger urban areas. For their part in the partnership, the USGS changed their production procedures to accommodate the Census Bureau s schedule, which required completion in time for the 1990 census. At the start of the effort, the USGS had produced only about 600 1:100,000-scale map sheets of approximately 1,800 needed to cover the conterminous United States. The USGS put more resources on the project and changed their normal production procedure so that they completed the map layers the Census Bureau needed, while delaying production of the map layers not needed, such as contours. For its part, the Census Bureau added names and classification codes to all the road features. The result was a successful partnership. The TIGER System has served the Census Bureau s geographic needs successfully for two decades. However, due to the original source scale, the relatively low quality of the coordinates in the GBF/DIME file areas, and the highly variable quality of subsequent updates, the TIGER database has become 1 GBF/DIME stands for Geographic Base File/Dual Independent Map Encoding. These files were the first Census Bureau product to be topologically structured (U.S. Census Bureau, 1970). increasingly difficult to maintain. It also has become more difficult to import data from (and match TIGER data to) more advanced sources and technologies, such as the Global Positioning System (GPS), high resolution orthoimagery, and state/ local/tribal GIS files. Hence, the current need to improve TIGER. The MAF/TIGER Enhancements Program The majority of the geographic community, no doubt, is aware of the Census Bureau s MAF/TIGER Enhancements Program. Introduced in 2000, its broadly stated objective is to further develop the capabilities of the Census Bureau s MAF and TIGER database. The five strategic objectives of the program are: Objective 1 Improve Address/Street Location Accuracy; Implement Automated Change Detection, Objective 2 Implement a New Processing Environment, Objective 3 Expand and Encourage Geographic Partnership Options, Objective 4 Launch the Community Address Updating System (CAUS), and Objective 5 Implement Periodic Evaluation Activities/ Expand Quality Metrics. The Census Bureau selected Objective 1, known also as the MAF/TIGER Accuracy Improvement Project (MTAIP), as its immediate initiative. Research and development began in early 2000 with an overarching goal of making TIGER information suitable for incorporation in the National Spatial Data Infrastructure (NSDI). At approximately the same time, the USGS began defining The National Map. As the two initiatives moved from broad concepts to definitive implementation projects, it became apparent that, although aspects of each initiative makes them unique, the underlying foundation of both projects promotes partnership opportunities that could contribute to the NSDI. Specific Requirements of the MTAIP The MAF and TIGER database are used to support the mapping, geographic analysis, and GIS activities of the Census Bureau in meeting the statistical data needs of its many and varied customers. Early on, it was realized that a minimum level of positional accuracy would be necessary to support the performance objective, Realign TIGER coordinates to correctly place an enumerator, relying on a GPS receiver, at the desired structure 100 percent of the time, on the correct side of the street (i.e., in the correct census block) 100 percent of the time, and the structure be in the correct relationship to corporate boundaries, other boundaries, and neighboring structures 100 percent of the time (U.S. Census Bureau, 2001). The Census Bureau s analysis revealed that there was no alternative possible to having a uniform minimum accuracy given the expectations by all levels of government officials and the citizenry that all areas, along with their associated streets/other map features/buildings, be of uniformly high quality. 2 The Census Bureau is driven by prescheduled, immovable events. The 2010 decennial census dictates an MTAIP completion date of Thus, the work must be completed within 6 years (2002 was the planned start date for Objective 1). The MAF and TIGER database extend over all parts of the Nation, including all counties and statistically equivalent entities in the United States, Puerto Rico, and the associated Island Areas. This strict time constraint, coupled with the magnitude of work to be performed, among other considerations, led the 2 During the MTAIP Phase 1 (Specifications Development) deliberations, the possibility of varying positional accuracy to accommodate differing morphology of rural, suburban, and urban areas was considered and rejected in favor of one minimum positional accuracy standard (or better) for the entire Nation October 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

3 Census Bureau to the decision to accomplish the majority of the MTAIP work using contract resources and best available data, whenever possible, to reduce the need for new collection efforts. The latter decision could be implemented by expanding and encouraging geographic partnerships both with other Federal agencies and with state/local/tribal governments. This is Objective 3 of the overall MAF/TIGER Enhancements Program. The intent of the MTAIP was to use the locational accuracy of features in the existing source data for the accuracy improvement work, not to create new sources. This included the need to identify and add new features to the database, as well as to identify and remove features that do not exist on the ground, thereby achieving positional and content accuracy, completeness, and currentness in the files. A more serious constraint is that the MTAIP design has to accommodate the existing TIGER data import methodology, yet remain flexible enough to flow into a new database environment scheduled for development well after TIGER data improvement work has begun (Objective 2). Complementary Products Promote a Partnership From the beginnings of the MTAIP and The National Map program, similarities have been apparent between the projects. Each effort relies, in essence, on a multi-source data capture process to be following by an ongoing, and equally demanding, maintenance process. Each agency is seeking to reduce redundancies in their data gathering activities and build on partnerships at all levels. Both agencies share common data partners, including state, local and tribal governments and other Federal agencies, that are the acknowledged authorities of their neighborhood datasets. A more accurate TIGER database aligned with these GIS files is bound to facilitate future digital data exchange, reducing duplication of effort and expense between the organizations and furthering the goals of the NSDI. Once corrected TIGER initial alignments are achieved, using state/local/tribal GIS files, the ability to continue to exchange data and continued utilization of such files as a change detection/maintenance methodology becomes possible. The Census Bureau and the USGS agreed, in principle, that it was important for their projects to use coordinated, continuing data gathering techniques and have an ongoing exchange of data, both between themselves and with partners at other levels of government. Early Partnership Activities Exploratory work in areas important to MAF/TIGER enhancements began in Research initiatives explored two key areas to determine whether the state of the industry would support an intensive TIGER realignment activity. First, different data source types were used to identify the level of positional accuracy that could be achieved by each and their associated costs. The focus was on relatively new technologies, such as lidar (Light Detection And Ranging) and IFSAR (Interferometric Synthetic Aperture Radar), as well as on industry-proposed techniques for improving the positional accuracy from existing products (Figure 1), such as Digital Orthophoto Quarter Quadrangles (DOQQ). Second, methodologies for realigning existing TIGER features were tested. The issue was whether commercial vendors could successfully match and conflate vector data extracted from source data to existing TIGER vectors, adjust non-matched features, and add new features to the database without destroying its topological integrity? Was it possible to work outside Live TIGER using interim data structures in support of the effort without compromising existing and ongoing programs, and how much additional effort on the part of the Census Bureau would be required to then incorporate such changes into Live TIGER? The USGS participated in the research initiatives in several ways. USGS subject matter experts provided technical Figure 1. TIGER road centerline data (dashed lines) compared to centerline data extracted from an ortho-rectified image (solid lines). Note the different amounts and direction of adjustment required (A and B) and change needed to feature length (C). expertise in developing areas of remote sensing, along with their years of experience in producing DOQQs, to assist the Census Bureau in evaluating proposed data sources. A Memorandum of Understanding (MOU) between the agencies allowed for contracted data collection using IFSAR technology and for testing the use of the collected data for differing purposes. The Census Bureau s interest was in specific, urban feature extraction, matching, and realignment work while the USGS wished to quantify possible improvements to their vertical and horizontal data models. Further Clarification The Census Bureau s review of the USGS draft proposal for The National Map in 2001 identified the following high priority objectives for both agencies: Having seamless national digital data that are current and positionally accurate, Better data integration, Increased reliance on partnerships with state/local/tribal governments and commercially available data, More current information, Geographic names for physical and cultural features, and Making maps for any required areas. The Census Bureau USGS Partnership On the basis of the common objectives, the Census Bureau and the USGS identified clear goals for a partnership and pledged a renewed alliance. As a cornerstone to the alliance, the USGS endorsed the Census Bureau s plan to enhance the positional accuracy of its TIGER database at levels that would meet or exceed the proposed standards for The National Map (positional accuracy commensurate with the primary series USGS topographic quadrangle map of an area). The USGS designated this realigned information as its primary source of road and governmental unit boundary data for The National Map. Although a formal MOU does not exist between the agencies PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING October

4 in this regard, published documentation identifies the agreement. 3 Perhaps more important than any formal agreement or written documentation, are the subsequent actions undertaken by both agencies to ensure the agreed upon principles can be, and are, implemented. A series of information exchange meetings was held between the Census Bureau and the USGS to ensure each was proceeding in a direction that allowed and encouraged the exchange of data. Topics included operational models, levels of data consistency, definition of data completeness, approaches to normalization of data, seamless datasets, and the ability and mechanisms to share data. A major topic of discussion was the need to identify a minimum set of features and attributes essential to each agency within the transportation layer (a core database). The USGS list of desired features was more extensive than that needed by the Census Bureau. The Census Bureau agreed that, although it would not actively solicit or extract an expanded list of features, it would provide a location within its database structure to store any features that state/local/tribal GIS files contain that could be of possible interest to The National Map and other groups, such as the Federal Geographic Data Committee s Transportation Working Group. 4 The Census Bureau took this opportunity to review its current Census Feature Classification Code (CFCC) schema so that new data will be collected in a much broader manner than currently allowed. This includes collecting and storing basic attributes and extensive feature-level metadata. The stored information will allow categorization of features into groupings desired by particular users and will limit the loss of information associated with identifying simply a single CFCC for each feature. When included with the commitment to collect and maintain feature level metadata, this will, in part, resolve the difficulty of arriving at common feature definitions. The Census Bureau emphasized that the MTAIP was about improving the quality of TIGER. Although TIGER is a seamless database, it is not necessarily a consistent database because of the multitude of sources used in its compilation, update, and now, enhancement. In order to successfully incorporate data from a multitude of state/local/tribal GIS files, metadata are necessary to document differences in content, account for sources and actions taken, and improve exchange of information. The MTAIP Contract The MTAIP contract was awarded to the Harris Corporation in June The project is structured in two phases: Phase 1 (Specifications Development), which began in July 2002 and was completed in December 2002, was devoted to translating the performance objectives into specifications and developing the project baseline against which performance would be measured. Phase 2 (Processing) began in January 2003 and will continue through 2008, at which time the universe of all 3,232 counties and statistically equivalent entities will have been initially enhanced (realigned) and have entered change management, or change detection, to ensure their continued currentness. 3 The National Map vision document (U.S. Geological Survey, 2001) names the Census Bureau as a Federal partner for roads (the primary feature that is the focus of TIGER realignment) and government unit boundaries (the Census Bureau is the Federal agency responsible for governmental unit boundaries and is identified as such in the Federal Geographic Data Committee s Geospatial One-Stop initiative.) 4 Although the Census Bureau has made provisions to store features/attributes desired by other groups, this does not mean that the Census Bureau is responsible for guaranteeing the quality or content of the information. More than 80 individuals from the Census Bureau and the USGS were invited to participate on five Harris-sponsored Phase 1 working groups whose sole purpose was to define specifications to serve as the MTAIP technical baseline. After this initial development, modifications to the technical baseline require Census Bureau Change Control Board (with the USGS as a participating member) approval. The following important baseline decisions directly contribute to the Census Bureau-USGS partnership leading to a successful MTAIP and success for The National Map partnership: A 7.6-meter (25-feet) or better positional accuracy requirement has been specified for matched, realigned road centerlines. To be accepted, an improved file must pass a Census Bureauconducted acceptance test consisting of 110 quality assurance points. 5 The basis for the positional accuracy requirement is the need for the combined error of linear features and structures to be small enough to assure a geocoding and location accuracy for 99.6 percent of the structures. This requirement assures The National Map will receive a road layer well within its accuracy standard. Harris identified unique threads or processing pathways through its realignment process. County thread selection depends in large part on the type and quality of available source data and cost considerations. Four major source data types are identified: state/county/tribal GIS files, commercial GIS files, aerial imagery and remotely sensed data (which may include lidar or IFSAR technologies), and field data collection (by Harris and/or its subcontractors) using GPS-based technology. The source evaluation process still needs a methodology for determining the quality of GIS file feature attributes. However, all agreed-upon available feature characteristics will be delivered by Harris as part of a realigned file along with extensive feature- and attribute-level metadata. The technical baseline requirements for feature attribution differ depending upon source data type. The amount of attribution obtained from commercial GIS files will depend upon availability and cost. Very limited attribution can be derived from aerial imagery and remotely sensed data. A predetermined minimal set of road feature attributes will result from field collection activities. A major MTAIP concern is the potential effect of the realignment process upon boundaries, in particular, the possible impact of conflation on nonvisible boundaries when the data source does not include the boundaries. Boundary realignment transcends simple conflation. Harris and the Census Bureau are developing a set of boundary realignment scenarios to ensure boundary characteristics are maintained (such as particular angles). When boundaries are part of the source data, the Census Bureau s subject-matter experts have an extended role in the process. The Census Bureau provides explicit and unique boundary instructions to Harris for each county, and extensively reviews the product to ensure that the integrity of governmental unit boundaries is maintained, to the extent possible. It is understood that the process will not end with initial MTAIP county enhancement. The Census Bureau s 5 Census Bureau statisticians identified 110 quality assurance points for each data source material as required to assure the positional accuracy of a realigned area. In the majority of cases, the data source covers a county (e.g., a large number of available state/county/tribal GIS files are county-based.) However, there may be multiple sources used to realign a county (e.g., the use of a city s GIS file for a major city within a county, supplemented by a county s GIS file for the remaining area) or a source may cover multiple counties (e.g., aerial imagery flown for a multiple-county region.) The quality assurance points are randomly selected from a predetermined grid pattern applied to each source area. The points are identified using areal and feature density criteria. All quality assurance points are intersections of selected categories of roads existing in TIGER that meet collection criteria for being well-defined GPS collection points October 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

5 ongoing Boundary and Annexation Survey (BAS) program will be used to review and validate the accomplished work. The NHD, which is the surface water feature data component of The National Map, was identified as the premier source of hydrographic data and will be used as the improvement data source for a majority of threads. State/county/tribal GIS files will serve as the source data for hydrography only in those instances in which the GIS file is identified as having higher quality data. To ensure that the NHD, and ultimately The National Map, can utilize any higher quality hydrographic data obtained through state/county/tribal GIS files, and to further ensure the continued exchange of hydrographic data once initial enhancement is complete, the Census Bureau will, when possible, assign the NHD reach codes, other appropriate identifier, and capture and include the associated feature names. The county processing sequence is based on a combination of Census Bureau and other agency needs. The Census Bureau worked closely with the USGS to create the TIGER Enhancement Database (TED) 6 an information repository about existing and known source data that serves as the major input into the county sequencing activity. Other partnership considerations also have been included, such as the high-resolution data requirements for cities and border counties that support homeland security priorities. This consideration provides relatively early processing of counties identified as priority counties and ensures the timely availability of improved data for incorporation into The National Map. Next Steps Continued partnership opportunities between the Census Bureau and the USGS are likely to be identified, developed, and refined as both the MTAIP and The National Map move forward from planning and specifications building to full-scale implementation. Harris will complete 60 enhanced counties during Phase 2 of the MTAIP in Government Fiscal Year Lessons learned by both agencies during this start-up phase will be incorporated through the change control process. The approach to attribution will evolve as quality assurance measures are identified. Harris is conducting a pilot project for the Census Bureau on improving the MAF through the collection of building coordinates and address information; the Census 6 The USGS receives periodic extracts of TED in return for its support of this resource. Bureau will determine the scope of this aspect of the MTAIP late in Information ages, so maintenance remains an area for further consideration. The Census Bureau and Harris (with participation by the USGS) must identify a change-detection methodology that is both efficient and timely. Conclusion Without a doubt, neither the U.S. Census Bureau nor the USGS will accomplish the goals of their respective geographic data projects, the MTAIP and The National Map, respectively, without their mutually beneficial partnership and without other effective partnerships at the state, local, and tribal levels, as well with academia and private industry. If the history of development and cooperation related to the TIGER database is any indication, the current partnerships will be successful. The MAF/TIGER database will be improved and be ready to support the geographic needs of the Census Bureau s statistical programs. The National Map will become the integrated source for the NSDI framework data, to which the Census Bureau will contribute enhanced road and geographic boundary information. This truly will be a partnership for the people. References Marx, Robert, A Vision for a 21 st Century MAF/TIGER, presented to the National Academy of Sciences Division of Behavioral and Social Sciences and Education, Panel on the Research on Future Census Methods, December, Washington, D.C. (Project ID CNST-I A, National Academy of Sciences), unpublished. U.S. Census Bureau, Census Use Study: The DIME Geocoding System, Report No. 4, U.S. Census Bureau, Washington, D.C., 46 p., MAP/TIGER Accuracy Improvement Project Request for Proposals, U.S. Census Bureau, Washington, D.C., last accessed 09 June U.S. Geological Survey, The National Map: Topographic Mapping for the 21 st Century, U.S. Geological Survey, Reston, Virginia, fs01802.html, last accessed 09 June PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING October