The Role of Data Quality in Biometric Systems

Transcription

1 The Role of Data Quality in Biometric Systems Austin Hicklin, Rajiv Khanna 9 February 2006 Abstract Data quality limits the accuracy of biometrics. Poor data quality is responsible for many or even most matching errors in biometric systems and may be the greatest weakness of some implementations. This paper uses an expansive view of data quality focused on its implications for operational databases and systems. Data quality is defined here not just in regard to samples (images), but also in regard to metadata, which is the information associated with the samples, such as biographic or cross-referencing information. The impact of poor data quality can be reduced in various ways, many of which depend on effective methods of automated data quality measurement. This paper analyzes the causes and implications of poor-quality biometric data, methods of measurement, prevention, and potential remedies. The paper draws on the experience of the authors in engineering and analysis work on a variety of large-scale operational biometric systems and biometric evaluations.

2 Acknowledgements This work was performed for the Department of Homeland Security (US-VISIT Program). Mitretek Systems was funded by the Department of Homeland Security, under contract through the U.S. Department of Interior, Contract NBCH-D , Delivery Order D Early drafts of this paper were performed for the National Institute of Standards and Technology, under contract through the U.S. Department of Interior, Contract NBCH- D , Delivery Order D We are grateful to Elham Tabassi (NIST), Tom Hopper (FBI), Ramakrishnan Krishnan (Raytheon), George Kiebuzinski, Larry Nadel, Shahram Orandi, and Brad Ulery (Mitretek) for their reviews and comments on earlier drafts of this document; to John Daugman for his insight into medical conditions that affect iris recognition and for examples of problem iris images; and to Delia McGarry and Don D Amato (Mitretek) for examples of problem face images. 1 February /77

3 Executive Summary The variations in biometric system accuracy and failures to match can be attributed to a combination of matcher accuracy and data quality. This paper uses an expansive view of data quality focused on its implications for operational databases and systems, in which the primary components of biometric data quality are: The quality of the samples (images), for both of the samples used in matching. The quality of the metadata, which is the information associated with the samples. This includes biographic, cross-reference, or indexing errors such as invalid or misattributed samples, as well as characteristics such as the age of the samples. Note that this definition of data quality goes beyond most discussions of biometric quality, which focus on the concept of sample quality. Sample quality deals with the capture fidelity of the subject s physical characteristics and the intrinsic data content of those characteristics. However, an equally important issue for any operational system is metadata quality: databases need to be concerned with erroneous relationships between data elements, which generally come from administrative rather than biometric-specific causes. Although metadata quality problems (also known as database errors) are not unique to biometric systems, the consequences of such errors can be particularly serious in many biometric systems, and therefore the methods of preventing and remedying such errors demand special attention for those systems. The purpose of this paper is to describe a framework that can be used in the description and analysis of data quality problems in a wide range of systems, not to provide a handbook of detailed solutions, which vary by specific applications. The paper defines the following framework: The causes of poor-quality data include physical characteristics or behavior of the subject, problems due to data collection, data processing, matcher errors, or administrative errors. Implications can vary substantially based on the application, but often include acquisition and enrollment decisions, decreased matching accuracy, increased computational complexity, or decreased processing speed. The implications are not just immediate and related to a current search; poor- 1 February /77

4 quality data enrolled in a database can have significant long-term consequences. The detection and measurement of data quality are necessary for dealing with poor-quality data: accurately quantifying quality is required for prevention and remediation. Measurement quantifies the characteristics, fidelity, and properties of a sample, particularly in terms of the sample s utility. Prevention of poor data quality depends on the effective implementation of collection device quality standards, sample quality metrics, sound human factors engineering, quality-focused operational procedures, and ongoing monitoring of operational quality including proper operator training. The remedies for unavoidable poor-quality data are not limited to rejecting data. Poor-quality data can be processed using specialized techniques. Matching accuracy can be improved by the use of optimization techniques based on sample quality, and mitigating the effect of a single poor-quality sample through collection of additional data and use of fusion techniques. One of the key objectives of this paper is to describe a number of lessons learned, operational issues observed, and advancements over the years that the authors encountered during their experiences in the development of systems such as IAFIS, IDENT, NCIC 2000, and US-VISIT. The development of these systems provided a wealth of knowledge that has previously never been published. This paper also draws heavily on biometric analyses conducted on behalf of DOJ, NIST, and DHS. Biometric data quality is gaining visibility as a key issue for biometric systems, and becoming a more prevalent topic for research and development. This is appropriate, because data quality is one of the most important factors in biometric systems effectiveness. It is the intent of this paper to provide a broad discussion of the nature and impact of biometric data quality. The authors hope that this will be a useful tool that can be used by designers, implementers, operators, and stakeholders of biometric systems to assist in understanding and anticipating data quality problems, so that they can either be prevented or minimized. 1 February /77

5 Contents 1 Introduction Summary of Previous Work Methods of Describing Quality Sample quality Character and fidelity Utility Metadata quality Causes of Poor-Quality Biometric Data Causes of quality problems related to subjects Intrinsically limited biometric data content Degraded or obscured characteristics Subject behavior Fraud Data collection problems Collection device problems Collection processes Compression and Sample Processing Problems Feature extraction errors Matching errors Administrative and database problems Implications of Poor-Quality Data Decreased matching accuracy The implications of sample quality on matching The implications of metadata quality and extremely poor sample quality on matching Long-term consequences Performance degradation System attack Detecting and Measuring Biometric Data Quality Approaches to measuring sample quality Generic image quality metrics Feature-specific metrics and localized quality metrics Representation-based aggregate metrics Matcher prediction aggregate metrics Matcher score analysis Stage-specific metrics Population quality measurement Collection device fidelity Automated quality measurement Biometric quality metric errors and implications Uses for sample quality metrics Metadata quality detection Prevention of Poor-Quality Data February /77

6 7.1 Controlling quality at the source Intrinsically limited data content Degraded or obscured characteristics Subject behavior Fraud Collection devices Collection processes Compression and sample processing Feature extraction Matching accuracy Administrative and database issues Quality policy Recapture of poor-quality data Rejection policy Quality monitoring Remedies for Poor-Quality Data Optimizing matching based on sample quality Specialized processing for poor-quality samples Multi-biometric capture and fusion Conclusions...67 References...68 Glossary February /77

7 1 Introduction The accuracy of biometric matching varies, both between subjects, and between samples collected from the same subject. Why is this? The variations in matching accuracy and failures to match can be attributed to a combination of matcher accuracy and data quality. This paper uses an expansive view of data quality focused on its implications for operational databases and systems, in which biometric data quality is composed of the data-specific factors that affect matching accuracy: Sample* 1 quality, for both the probe* and gallery* samples used in matching. Metadata quality, or the quality of the information associated with the samples, which includes biographic, cross-reference, or indexing errors such as invalid or misattributed samples, or quality issues associated with the age of the samples in the database. Variability between samples, due to factors such as the variability of the characteristics being measured or the collection process, or variations in correspondence between the probe and gallery samples. Note that this definition of data quality goes beyond most discussions of biometric quality, which focus on the concept of sample quality. Sample quality deals with the capture fidelity of the subject s physical characteristics and the intrinsic data content of those characteristics. However, an equally important issue for any operational system is metadata quality: databases need to be concerned with erroneous relationships between data elements, which generally come from administrative rather than biometric-specific causes. Although metadata quality problems (also known as database errors) are not unique to biometric systems, the consequences of such errors are particularly serious in some biometric systems, and therefore the methods of preventing and remedying such errors demand special attention for those systems. A biometric system can be seen as a combination of pattern recognition, database, and systems engineering challenges [Crawford]: sample quality is primarily a pattern recognition concern, while metadata quality is primarily a database and systems engineering concern. 1 A biometric sample is a representation of a physical or behavioral human feature. For biometrics such as face, fingerprint, or iris, sample and image are equivalent terms. For the remainder of the paper, words defined in the glossary are marked with an asterisk (*) on first use. 1 February /77

8 Although variability between samples is responsible for much variability in scores, it is not as important a limitation as sample and metadata quality. As matchers improve and methods of using data quality metrics to optimize the matcher performance (normalizing scores) are more broadly used, sample variability may be reduced to a minor role in determining the biometric system s performance. It can be argued that variability between samples actually is composed of trivial or immeasurable variations in factors that would be considered sample quality problems if greater in magnitude. This paper seeks to develop a broad view of quality that provides a basis for describing the role quality plays in biometric systems. Biometric data quality encompasses all data associated with the system: human characteristics, biometric samples, intermediate products (feature sets, templates, etc.), and administrative data (e.g. the links to other information) as it relates to system performance. The purpose of this paper is to describe a framework that can be used in the description and analysis of data quality problems in a wide range of systems, not to provide a handbook of detailed solutions, which vary by specific applications: The paper defines the following framework: The causes of poor-quality data include physical characteristics or behavior of the subject, problems due to data collection, data processing, matcher errors, or administrative errors. (Section 4) Implications can vary substantially based on the application, but often include acquisition and enrollment decisions, decreased matching accuracy, increased computational complexity, or decreased processing speed. The implications are not just immediate and related to a current search; poorquality data enrolled in a database can have significant long-term consequences. (Section 5) The detection and measurement of data quality are necessary for most methods of dealing with poor-quality data: accurately quantifying quality is required for prevention and remediation. Measurement quantifies the characteristics, fidelity, and properties of a sample, particularly in terms of the sample s utility. (Section 6) Prevention of poor data quality depends on the effective implementation of collection device quality standards, sample quality metrics, sound human factors engineering, quality-focused operational procedures, and ongoing monitoring of operational quality including operator training. (Section 7) 1 February /77

9 The remedies for unavoidable poor-quality data are not limited to rejecting data. Poor-quality data can be processed using specialized techniques. Matching accuracy can be improved by the use of optimization techniques based on sample quality, and mitigating the effect of a single poor-quality sample through collection of additional data and use of fusion techniques. (Section 8) Biometric data quality is gaining visibility as a key issue for biometric systems, and becoming a more prevalent topic for research and development. This is appropriate, because data quality is one of the most important factors in biometric systems effectiveness. 2 Summary of Previous Work Neither sample quality nor database quality is a new concept. The sample quality of fingerprints and facial images was a concern long before automated systems, when manual systems of identification were used. Database quality (especially invalid, inconsistent, or contradictory data) has been an ongoing issue since large-scale databases were first implemented as manual, paper-based systems. In the 1980s and 1990s, the importance of both biometric sample quality and database quality were recognized during the development of large-scale Automated Fingerprint Identification Systems (AFIS*), leading to the development of automated image quality metrics (IQMs*) to detect poor sample quality, and procedures to detect and limit the impact of database errors such as duplicate records. Unfortunately, most of this research was never made publicly available. One of the key objectives for this paper is to describe a number of lessons learned, issues, and advancements over the years that the authors encountered during their experiences in the development of systems such as IAFIS*, IDENT*, NCIC-2000*, and US-VISIT*. The development of these systems provided a wealth of knowledge that has previously never been published or placed in the context of the general issues relating to data quality. In 2000, the Fingerprint Image Quality Study [IQS] was conducted in association with the Department of Justice s IDENT/IAFIS* integration activity. This study evaluated a variety of image quality metrics in terms of their ability to predict matching scores for the IAFIS matcher, showing the effects of sample quality on matcher performance, as well as the expected improvements in accuracy when additional fingers were used. This paper draws heavily on analyses conducted as part of IQS*. 1 February /77

10 In the past few years, a variety of studies have confirmed that biometric matching accuracy is dependent on sample quality [SDK, VTB, FpVTE, ITIRT]: poor sample quality decreases the likelihood of a successful match, while extremely poor quality samples may be impossible to match. [FpVTE] and [SlapSeg] discussed both sample quality and metadata quality, the prevalence of key types of data quality problems, and detailed results of their impact on matching accuracy. Many of the findings in this paper are results of analyses conducted in the FpVTE and SlapSeg evaluations. In 2004, NIST s* Fingerprint Image Quality [NFIQ] introduced a method of using neural nets to optimize the combination of various image quality metrics to predict the scores of various matchers. Notably, NFIQ is freely available, unlike most IQMs. Also in 2004, [Wein04] underscored how poor data quality is a vulnerability in US-VISIT that could be exploited. He assumed that terrorists could choose individuals with intrinsically poor fingerprints or deliberately degraded fingerprints, so that those individuals could pass through US-VISIT undetected. Wein s report to Congress dramatically increased the visibility of biometric data quality issues. In 2005, US-VISIT s [Goats] study showed that most of the poor quality fingerprints encountered from frequent US-VISIT travelers were not from individuals with intrinsically poor fingerprints (nicknamed goats ), but were instead due to sample-specific causes, presumably collection problems. There is an increasing focus on biometric quality. The M1 standards body has a draft Biometric Sample Quality Standard in progress [M1/ ]. NIST and DHS* are co-sponsoring a workshop on biometric data quality in early 2006 [BQW]. We hope that the results of all of this activity will be a greater understanding of the role of biometric data quality, with development of improved methods of prevention and remediation. 3 Methods of Describing Quality 3.1 Sample quality [M1/ ] defines three prevalent connotations of biometric sample quality: Character refers to the quality of the inherent physical features of the subject. 1 February /77

11 Fidelity is the degree to which a sample is an accurate representation of the original features. Utility refers to the value of a sample in terms of biometric system performance Character and fidelity The relationship between character and fidelity is illustrated in Figure 1, which is derived from the excellent Reference Model in [M1/ ]. Source compression acquisition Sample Compressed or processing fidelity Sample fidelity extraction fidelity Template Character Fidelity Figure 1: Character and fidelity in sample quality Character is driven by the intrinsic data content of the physical entity being measured, as described in section 4.1. Note that every transformation of the data can have an impact on fidelity, so there are three sub-categories of fidelity: Acquisition fidelity refers to the accuracy (or possible loss of data) of the capture/acquisition process. Poor acquisition fidelity is caused by data collection and processing problems, discussed in section 4.2. Compression or sample processing fidelity refers to the accuracy of any image/sample transformations used, such as compression, formatting, color mapping, cropping, or segmentation. Poor sample processing fidelity is caused by compression and processing problems, discussed in section 4.3. Extraction fidelity refers to the accuracy with which a template, feature set, or model is generated from the sample. Extraction fidelity is directly related to the accuracy of the feature extraction algorithm(s) used, discussed in section February /77

12 Note that the possible problems within each of these categories vary markedly both between biometric modalities and between systems, so we are not attempting to delineate all possible examples of problems that may be encountered, but to define a framework that can be used in the description and analysis of such problems in a wide range of systems Utility The ultimate value of a sample in terms of biometric system performance is purely its value in matcher accuracy, as measured by matcher score. Therefore, determining the true utility of a sample requires the use of one or more matchers. In practice this is complex because matcher scores are not just dependent on the quality of a single sample, but on the quality of both probe and gallery samples: a high-quality probe can still get a very low matcher score if the gallery sample is poor quality. For this reason there has been a long-standing use for image quality metrics in estimating the utility of a given sample. The measurement of sample utility, using matchers or quality metrics, is discussed in detail in Section 6. 1 February /77

13 3.2 Metadata quality Metadata quality is here defined as those aspects of quality that cannot be determined through analysis of a sample. A biometric database does not just contain large numbers of samples (or templates): the biometric samples are of value because of metadata associated with them. A biometric database may contain a variety of descriptive fields, but the key fields for our purposes are those associating the sample(s) with a unique identity (usually a database-specific identification number, since other values such as names or social security numbers are not truly unique), and the fields used to define the type of biometric instance* used (such as right or left iris, or the finger position). The accuracy of the system is limited by the accuracy of these associations. Virtually every large-scale operational database including but not limited to biometric databases has to deal with issues of metadata quality. According to a study by The Data Warehouse Institute, Current data quality problems cost U.S. businesses more than 600 billion dollars a year. [Dasu1] The MIT Total Data Quality Management Program states Anecdotal evidence and a growing literature point to data being defective at levels of 10% or more in a variety of applications and industrial contexts. [TDQM] Of the variety of concepts that can be used to define data quality [Dasu1], several are applicable to biometric data: Accuracy refers to how closely a reported value reflects the true value which unfortunately cannot always be determined. In terms of biometric data, accuracy can refer to the accuracy (or fidelity) of the sample or of the metadata. Uniqueness is particularly relevant to identification systems, which strive to create unique records that correspond to an individual subject. Consistency refers to conflicts within or between records in the database. Timeliness refers to the currency (and relevance) of the data. A database with many biometric samples that are decades old can be seen to have a certain type of metadata quality issue. Completeness refers to how fully the domain is represented in the database. For many negative ID systems watchlists in particular completeness is an issue in that the database can only hope to include a portion of the target population. 1 February /77

14 For biometric databases, these can be further narrowed down to a few important types of problems: Subject association errors are inaccurate links between a sample and a unique identity: Unconsolidated records (often known as consolidations ) are cases in which an individual has two or more supposedly unique identities in the database. The proportion of unconsolidated records in the database is driven by the false reject rate (FRR)* of the system s matcher (explained in section 4.4). Unconsolidated records are particularly prevalent in databases created by combining multiple pre-existing databases (explained in section 4.6). Misidentified records (often known as mis-idents ) are cases in which an identity in the database is associated with samples from a different individual. More frequent than a misidentification within a database is a misidentification between databases, in which each database is selfconsistent, but the inter-database links are in error; such inter-database references are more difficult to validate. Sample association errors are inaccurate links between a sample and its type. These vary by biometric modality*, but particularly prevalent are sequence errors, in which a sample is associated with the incorrect instance, such as swapped left and right irises, or an index finger included as a middle finger. These problems can be insidious if only some of the instances in a set are switched (i.e. some of the fingerprints in a 10-print set), because matches including those samples will fail, but other matches will succeed. Invalid data could be considered either a sample or metadata quality issue. Blank or non-biometric samples may be due to operator error or software errors. Aging* is a more subtle type of quality problem. If the samples being compared differ substantially in age, matching accuracy may be affected. This varies depending on biometric modality: for example, facial recognition is reputed to be more sensitive to data aging than iris or fingerprint. However, any biometric system will be sensitive to age if relevant physical changes occurred in the intervening period: disease and trauma are obvious changes, but more subtle physical degradation may become an issue as the difference in age increases. Aging can be considered a metadata issue because the quality of the samples themselves is not the issue: the age difference is the cause of the degradation of accuracy. 1 February /77

15 An additional metadata issue that is generally an annoyance rather than a serious problem in an operational system is the duplication of samples (also known as self-idents ). Duplicate samples are two images intended to be distinct samples, but are the same. These may be bitwise identical, or may differ due to compression, cropping, or through different scans of the same material. Duplicate samples are a serious concern in evaluations, where they can distort results due to the unrealistic matcher scores they generate. 4 Causes of Poor-Quality Biometric Data Any attempt to analyze biometric data quality problems should address the root causes of those problems. In general terms, the causes of problem data are due to the subjects, data collection, data processing, or administrative issues, as shown in Table 1. Poor character Poor acquisition fidelity Poor compression / processing fidelity Poor encoding fidelity Unconsolidated records Problems Related to Subjects 4.1 Intrinsically Limited Data Content x Degraded or Obscured Characteristics x x Subject Behavior x x x Fraud x x x x x x Data Collection Problems 4.2 Collection Device Problems x x Collection Processes x x Compression and Sample Processing Problems 4.3 x x Feature Extraction Problems 4.4 x x Matching Errors 4.5 x x Administrative and Database Problems 4.6 x x x Misidentified records Sequence errors Invalid data Table 1: Relationship between quality problems and causes Causes of quality problems related to subjects may be categorized as physical characteristics or behavior. A subject s physical characteristics must be captured accurately in a sample for effective use of biometrics: poor quality 2 For behavioral biometrics (such as speaker recognition or dynamic signature) subject behavior is character. 1 February /77

16 samples may be due to intrinsic (natural) problems, or may be degraded, obscured, or fabricated; overall quality is closely coupled to the behavior of subjects and their degree of cooperation. (Section 4.1) Causes of quality problems related to data collection include problems such as collection device problems, inadequate training of operators, and other operator errors. (Section 4.2) Compression and sample processing problems include problems such as overcompression of images, improperly formatted images, and corrupted files caused by software errors. (Section 4.3) Feature extraction errors are based on the accuracy with which a template, feature set, or model is generated from the sample by a given feature extraction algorithm. (Section 4.4) Matching errors, false reject rates and false accept rates, can be directly responsible for unconsolidated records and (to a lesser extent) misidentified records. (Section 4.5) Administrative and database problems are based on the administrative tasks of entering, maintaining, and updating biometric data in a database: all can result in metadata errors, specifically unconsolidated records and misidentified records. (Section 4.6) All of these factors are not equally likely to cause quality problems. Although some individuals do have intrinsically poor-quality biometric characteristics, in practice, the data collection process is the weakest link in quality: even a subject with ideal biometric characteristics can leave a poor sample. Compression, sample processing, and feature extraction vary by system, but are most likely to be problems if those issues are ignored. Matching errors (specifically false reject rates) are a known source of unconsolidated records, almost by definition. Administrative problems vary substantially by system. Any attempt to make blanket statements about the frequency of different causes of data quality problems should be met with skepticism, because this frequency varies dramatically by source. This is well illustrated by [SlapSeg], which provides a detailed breakdown of quality issues for datasets obtained from various operational sources. 4.1 Causes of quality problems related to subjects Biometric systems are based on mathematical representations of a subject s biological characteristics. Quality problems occur when: 1 February /77

17 the intrinsic data content of these biological characteristics is unusually limited (Section 4.1.1), the characteristics are degraded or obscured (Section 4.1.2), the subject is uncooperative (Section 4.1.3), or the subject resorts to fraud (Section 4.1.4) Intrinsically limited biometric data content The characteristics used by biometric systems are distributed unevenly from person to person: this should go without saying, since these characteristics are used to differentiate between people. However, this also means that to some extent, the degree of distinctiveness varies from person to person. For example, an arch fingerprint typically has fewer minutiae than other types of fingerprints, making these fingerprints harder to match in minutiae-based systems (see Figure 2); the changes to facial and skin structure that occur with age often make children harder to identify (and older people easier to identify) for some facial recognition systems. Limited intrinsic characteristics are problematic from the perspective of quality: a perfectly clear, highly detailed, large fingerprint would ordinarily be described as high quality, but if such a fingerprint is one of the unusual cases that only contains a half dozen minutiae, the limited intrinsic character outweighs the high quality (or fidelity) of the image. 1 February /77

18 Figure 2: Arch fingerprint with few minutiae [Example Data] 3 The most obvious limitations are those posed by missing biometrics: examples include birth defects, amputations (fingers, hands, and arms), missing eyes, and muteness. The absence of a feature cannot necessarily be taken as an identifiable characteristic, since the loss of characteristics may occur between collection encounters, and the absence of a characteristic (albeit unusual) is not unique. Missing biometrics can only be remedied through alternative processes, such as through the collection of additional biometric instances (e.g. multiple fingers) or modalities (e.g. a combination of face, fingerprint, and iris; see Section 8.3). The extent to which people with limited intrinsic characteristics occur in any population differs substantially depending on the biometric modality, and the definitions used. The term Goat is now widely used in biometrics to refer to a person who is intrinsically difficult to match. 4 In the case of fingerprints, a persistent factoid that 2% of the general population was hard to fingerprint has been shown to be false or misleading: a recent study of US-VISIT [Goats] showed 3 Operational samples of data quality problems are used in this paper whenever possible. However, when such real-world samples could not be published because of sensitivity or privacy concerns, samples were captured or created in the laboratory to duplicate the observed problems as closely as possible. The created examples are identified by the legend [Example Data]. 4 The concept of a biometric menagerie was introduced in 1997 in [Campbell] and expanded in [Doddington], defining sheep (subjects who are easily matched), goats (subjects who are particularly difficult to match), lambs (subjects exceptionally vulnerable to impersonation), and wolves (subjects exceptionally successful at impersonation). The concept of goats does not differentiate between subjects with intrinsically limited characteristics and those with degraded characteristics. 1 February /77

19 that no more than 0.05% of frequent US-VISIT travelers were consistently hard to match. This type of data quality problem is not spread evenly across the population: the fact that there are vanishingly few goats in an affluent population that was predominantly years old does not imply a similar conclusion for other populations, such as the elderly, manual laborers, and children Degraded or obscured characteristics Trauma, disease, or other medical conditions can permanently or temporarily degrade characteristics that might intrinsically be of high quality. Permanently degraded biometric features are particularly troublesome if the change occurred between samples. They can introduce false features, reduce true features, or both, making the new sample sufficiently different from previous ones or more similar to other samples. Fingerprint matching performance is hindered by scars, disfigurement, cracking of fingerprint ridges, and certain skin diseases that degrade ridges; fingerprint degradation increases with age, and in some occupations due to abrasion or chemical exposure. Burns and other scarring hampers face recognition. Iris recognition may be less accurate or even impossible due to physical damage to the eye or certain diseases, such as surgical error, aniridia, pupil anomaly, or nystagmus [DaugmanB, DPB]. Physical chronic characteristics are not physically degraded, but are conditions that make accurate capture difficult, resulting in similar effects. Examples include shallow ridges and chronically dry skin in fingerprint systems. Temporarily degraded biometrics may be observed in one sample but not in the next. For example, a shallow cut may result in a temporary scar that disappears over time. The temporarily degraded biometric affects the likelihood of matching or, if enrolled into databases, will affect future searches. Temporarily obscured biometrics may be correctable or non-correctable. These obscurities may hinder system performance. Correctable obscurities may include glasses, clothing, hair, and dirty fingers. Non-correctable obscurities include bandages and clothing that cannot be removed for social/cultural reasons (e.g. veils). Purposely obscured biometric samples are a special case, considered under Fraud (Section 4.1.4). 1 February /77

20 Figure 3: Iris partly obscured by contact lens and eyelashes [DaugmanA] Subject behavior Subject behavior can be described by levels of cooperation and degree of familiarity. Cooperative, familiar subjects can be expected to result in higher quality data, on average. Actively uncooperative subjects can have a serious impact on data quality; in extreme cases, uncooperative behavior can be considered fraud (see Section 4.1.4). [Wayman] discusses partitioning biometric applications into various binary categories, two of which are directly related to subject behavior: Levels of cooperation are defined in terms of cooperative versus noncooperative subject behavior. In a positive ID system, it is to the subject s benefit to be recognized, and therefore the system rewards the subject for cooperation (such as in an access control system). Generally, subjects help such systems perform well. In a negative ID system, it is to the subject s disadvantage to be recognized, and therefore the subjects of greatest interest can generally be assumed to be uncooperative (such as when checking subjects for prior criminal arrests). Some systems are a combination of both types, such as the US-VISIT system, which performs a background check (negative ID) and verification of visa information (positive ID). Degrees of familiarity are defined in terms of habituated versus nonhabituated subject behavior. Habituation refers to how frequently subjects interact with the system and their comfort and knowledge of the system, 1 February /77

21 generally applying to cooperative subjects. For negative ID systems and uncooperative subjects, familiarity with the system can compromise performance if they learn to exploit system weaknesses. Other system characteristics in Wayman s classification are closely related to subject behavior: Overt versus covert collection refers to whether subjects are aware that biometric identifiers are being measured. Both levels of cooperation and degree of familiarity really only apply to overt collection: subjects cannot actively cooperate with (or be habituated to) covert systems, and are limited in deliberately uncooperative behavior. Attended versus non-attended systems are defined by the supervision of an operator. The degree to which a subject can be actively uncooperative can be limited (to some extent) by supervision. Cooperative subjects can be assisted and made much more effective under supervision. More gradations between the cooperative and uncooperative extremes help develop a better view of subject behavior, particularly if motivations 5 are related to behaviors. It should be stressed that levels of cooperation should be considered a continuum, extending through various levels: Cooperative and habituated - subjects comply with instructions and procedures, and are familiar enough with the system to be effective. Subjects that are familiar with instructions and system processes are well suited to support positive ID systems: the subject s desire and system objective to match are aligned. Cooperative but non-habituated - subjects comply with instructions and procedures, but are ineffective. Lack of familiarity or clarity in procedure limits the ability of the subject to cooperate. Note that a poorly-designed system or unclear procedures can prevent cooperative subjects from being habituated, effective users. Ambivalence subjects don t know or care about instructions and system procedures. Passively uncooperative subjects do not comply with instruction and system procedures. 5 Motivations might include evasion, deception, candor, uncertainty about performance, and trust/distrust of the system. 1 February /77

22 Actively uncooperative subjects take specific measures to avoid recognition, as described under Fraud (see Section 2.1.4). Cultural influences can create a degree of apparent non-cooperation. For example in Amish, Mennonite, and some Islamic cultures, women traditionally wear veils or head-coverings in public; some Jewish and Muslim men wear caps; Sikhs wear turbans; various cultures or sects avoid photographs [CAIR]. Other individuals for personal or political reasons may have a particular dislike or distrust of biometric systems. Collection policy and multi-biometric systems can help reduce cultural limitations (Sections 7.1.6, 8.3) Fraud An actively uncooperative subject who goes beyond simple non-compliance may be actively attempting to foil the system with techniques that include the following: Evasion Actions that prevent/lessen the likelihood of matching; techniques include degrading or obscuring physical characteristics. Spoofing Purposefully attempting to match a different person; techniques include modifying biological characteristics and using fabricated characteristics. Forensic data such as fingerprint or DNA evidence has related categories of fraud: Forged evidence is fraudulently placed on a surface, while fabricated evidence never existed on the surface from which it supposedly came. [Wertheim] Note that the intent in evasion is to avoid matching anyone in the system, and therefore being regarded as an unenrolled individual. The intent in spoofing is to match another individual already enrolled in the system. Generally, evasion is an attack against a negative ID system, and spoofing is an attack against a positive ID system. Biometric data associated with evasion can have the same properties as those listed above for passively uncooperative subjects, as well as making changes to biometrics like wearing down fingerprints, scarring, mutilating characteristics, or fabricating characteristics. Such data is purposely attempting to be poor quality, and so may be detected through image quality metrics. Figure 4 shows an example of evasion in a fingerprint system, in which the subject used the back of the finger, presumably with an inattentive operator. 1 February /77

23 Figure 4: Back of finger used in evasion [NIST-IQS] Systems that permit subjects to submit the samples that are used for identification are as susceptible to fraud as any unsupervised system. Figure 5 shows examples of ways that subjects attempt to evade identification in facial recognition systems where the subject submits photographs: one example shows manipulation by changing the aspect ratio, elongating the image; the other a digitally manipulated hairstyle. Figure 5: The same photograph digitally manipulated in two ways for evasion: (Left) Distorted aspect ratio; (Right) Hairstyle added digitally [Example Data]. 1 February /77

24 Fabricated characteristics have been shown to spoof biometric systems. [Matsumoto] reported that gummy fingers made of gelatin were accepted at extremely high rates by particular fingerprint devices using optical or capacitive sensors. [Lukasik] indicates magazine photographs and high-resolution images of faces have been enrolled into facial recognition systems, so it would be reasonable to assume good disguises can fool face recognition systems. Spoofed data is purposely trying to be valid, so it may not be detectable through the use of image quality metrics; liveness tests are more likely to be effective. For either method of fraud, prevention at the time of capture is more likely to be successful than detection after the fact, particularly through the use of observant operators. 4.2 Data collection problems A large proportion of poor-quality data is caused by faulty data collection, otherwise known as poor acquisition fidelity. Biometrics are never trivial to capture: fingerprints require high-resolution images of a structure that can be distorted in three dimensions; facial images require focused, evenly lit images of a structure even more susceptible to distortion; iris images require focused, evenly lit images of a structure that dilates, moves, and is usually partly occluded by the eyelid. The collection process is dependent on subject characteristics as discussed above, but also on the correct use of appropriate devices (Section 4.2.1), by trained and attentive operators (Section 4.2.2). After collection, transmission, storage, and processing of the biometric samples should not be responsible for serious quality problems if carefully thought-out and well-implemented procedures are in place but otherwise can be a serious source of error (Section 4.3) Collection device problems Collection devices vary dramatically in the quality of images they produce, and frequently vary from the vendor s official specifications. Analysis of image collection device quality can focus on these areas, detailed in [D Amato]: Spatial Sampling Frequency and Geometric Distortion Quantization and Tonal Response Spatial Resolution Spatial and Temporal Uniformity 1 February /77

25 Color Accuracy Illumination Source (for some systems) These issues should be detected and resolved before implementation, through use of collection device standards and procedures. Collection devices should not be a serious source of errors if the specific device being used is compliant at the time of use with well-thought-out quality standards, and it is used according to well-thought-out procedures (as discussed in section 5.4). In practice, unfortunately, this means that collection devices are a serious source of data quality errors. Lack of standards is a serious issue for some systems. Any biometric system needs to define minimum standards for collection devices, or require a preexisting standard. Unfortunately, not every system does this: systems have been implemented without ever clearly defining image format and quality requirements, or defining the procedures for certification of devices. Lack of certification processes is nearly as serious as a total lack of standards. Inappropriate types of devices are models of devices that do not even claim the specifications required by the standard. For example, a camera with the box specification of 1 megapixel obviously could never comply with a standard requiring 3 megapixel images, or a fingerprint scanner with a resolution of 385 pixels per inch and 64 levels of gray could never comply with a standard requiring 500 pixels per inch and 256 levels of gray. Uncertified types of devices are models of devices that have not been tested and certified as compliant with a particular standard. Generally, this level of testing is for a model overall, showing that at least one instance of that brand and model complies with the standard. Certification is required to detect those devices that do not meet their own specifications (such as invalid resolution), produce distorted images, have optical abnormalities, uneven illumination, etc. The authors have observed scanners and imaging devices that do not produce the stated resolution, show internal reflections, cannot be focused precisely, and produce optical distortions or artifacts in images. This type of problem may be the result of poor design or design tradeoffs (e.g. performance vs. cost). Specific device failures are instances where a specific device does not perform properly in operations, even though the model was certified. These problems may be the result of manufacturing problems, abuse, or a failure in operations. In some cases, a particular device may be defective; more frequently, the individual device was dropped or damaged. Device failures 1 February /77

26 may be intermittent or occur regularly. For example, interlacing errors observed in an image may be due to failure of an analog to digital converter chip, and has been seen to occur after a properly-working device has been left on for a long period, but goes away after restarting the device. In one biometric system, poor quality regions of the CCD chip were simply masked so that they appeared as white pixels in images. Certification at acquisition and recertification during operations helps detect and reduce the effects of device failure Collection processes Just because devices are capable of capturing high-quality biometric samples does not mean that they always will. In NIST s FpVTE test, some of the highest and lowest quality results came from systems that were identical in hardware and software, leading to the conclusion Capture devices alone do not determine fingerprint quality. Different operational fingerprint sources can use the same type of collection hardware and software and yet result in substantially different performance. [ ] Therefore, the subject populations, collection environment, staff training, and equipment maintenance are some of the other factors that are believed to have a substantial impact on fingerprint quality. [FpVTE, Conclusion #4] NIST s SlapSeg test showed that higher device quality does not necessarily result in higher system performance: of the operational livescan fingerprint datasets, the worst-performing data came from devices certified to a more rigorous quality standard than those used for the best-performing data. [SlapSeg, Section 4.2] Using an appropriate or certified collection device to collect biometric samples of appropriate fidelity can fail in various ways. Prevention of most of these issues is discussed in Section 5.4. Lack of collection policy and procedures is unfortunately a serious issue. Equipment cannot be used appropriately if there is no definition of appropriate use of equipment. Operators without clear procedures make decisions on a case-by-case basis, resulting in highly variant data that reduces overall quality. If procedures are unclear or impractical, the result is only theater, in which the appearance of procedures disguises their de facto absence. To show one example, Figure 6 shows several problems that should be eliminated through the effective implementation of collection policy and procedures. 1 February /77

27 Figure 6: Example of "fisheye effect" or distortion from use of a short focal length lens. Note also the uncontrolled lighting and pose [Example Data]. Subject issues related to collection differ depending on whether collection is supervised or not. Lack of subject guidance in unsupervised applications can prevent cooperative subjects from being effective and impede habituation. Uncooperative subjects have already been discussed (in Section 4.1.3). For negative ID systems (in which the cooperation of subjects should not be assumed), supervision by an operator is always necessary. Although cooperation of subjects in positive ID systems can generally be assumed, rebellious subjects can be a problem. For supervised systems, lack of effective operator oversight is responsible for a variety of problems. These errors include misusing collection devices (failure to follow procedure), inconsistent sampling, providing errant guidance to subjects, failure to properly oversee subject interaction with the system, or incorrect data entry. Such issues can be the result of lack of operator training, incompetence, apathy, overwork, or unrealistic throughput expectations as well as the fact that any human activity will have some level of error. Errors and quality problems tend to increase with operator workload and stress. If such operator issues are not addressed, other methods of controlling quality may be moot: operators confronted with a substantial workload often ignore policy or override system quality controls, such as feedback to retake poor-quality data. 1 February /77

28 Figure 7: Examples of poor-quality fingerprints, retained due to ineffective operator oversight [NIST-IQS] Equipment and configuration issues can take a variety of forms: Poorly designed physical configuration of equipment is often ignored in the implementation of systems. Equipment cannot simply be made available without considering how its physical arrangement affects usability and ultimately data quality. Poor physical configuration design can be directly responsible for degraded capture quality (due to factors such as inappropriate lighting or backgrounds for facial recognition systems 6 ), or indirectly, through lack of regard for human factors issues. (Figure 8) If human factors (or ergonomic) issues are ignored, both subjects and operators may be impeded in their attempts to use the system, and subjects or operators who are out of the norm (such as very large, very small, or handicapped individuals) may be prevented from using the system. As an example of how minor issues can cause ongoing effects, in many databases fingerprints from subjects left hands are worse than those from the right hands; this is believed in large part to be due to the fact that right hands are captured first in the affected systems, and the subjects and/or devices are not repositioned appropriately when the left hands are captured. [FpVTE, SDK] 6 While all biometrics require well-thought-out physical configuration of equipment, some particularly require focus in this area: facial recognition requires attention to background and lighting, defining a capture volume larger than simply the face itself; speaker recognition requires attention to background noise. 1 February /77

29 Figure 8: (Left) Reflections caused by lighting problems [DaugmanA]; (Right) Uneven illumination [Example Data]. Misuse of equipment is due to the failure of collection procedures, either through the lack of procedures, unclear or impractical procedures, or failure of operators. Such misuse can take a number of forms, such as incorrectly focused cameras or misdirected lighting. Figure 9: Out-of focus iris and face images [Daugman04, Example Data] Poorly maintained or damaged devices can be responsible for many problems that can be particularly serious. Issues such as dirty or damaged lenses on cameras or platens on scanners will degrade all data collected. 1 February /77

30 Devices that are dropped or get out of calibration can result in strange and sometimes insurmountable problems. The authors have seen such problems in operational data as interlacing (every other row or column in the image shifted a few pixels, or set to black), image artifacts such as black stripes across or around images, color shift (inability to capture true color images), dramatic illumination drop-off (in which the corners are much darker than the center of the image), black-for-white reversal of images, left-right flipping of images, and corrupted data files. See Figure 10 for examples. Figure 10: (Left) Detail of fingerprint with interlacing errors. [Example Data]. (Right) Fingerprint with illumination drop-off [Example Data]. Environmental problems such as temperature, humidity, background lighting, and background noise can all directly affect data capture, or can cause temporary or permanent damage to equipment. For example, temperature and humidity can result in condensation on camera lenses or scanner platens; on fingerprint livescan devices where the subject s hand temperature differs substantially from the device temperature, such condensation results in halos around the fingers, as shown in Figure February /77

31 Figure 11: Darkened halo effect on a livescan due to condensation on the platen. Note that latent fingerprints are visible in the halo. The stripes across the background are an artifact of inappropriate image processing [Example Data]. 1 February /77

32 4.3 Compression and Sample Processing Problems Unlike character and acquisition fidelity, compression and sample processing fidelity do not generally change between subjects or encounters. The causes and types of sample processing vary greatly from system to system. Figure 12 shows some of the types of technologies used in various biometric systems. It should be noted that almost any device or algorithm used can result in a known degradation of quality (such as with lossy compression), or can have glitches that result in corrupted data. Figure 12: Relative Frequency of Technology Application to Automated Biometric Systems Some of the key compression and sample processing issues have to do with data compression, image formatting, color mapping, cropping and segmentation, or corrupted data caused by faulty software: Data compression may range from lossless (where the original data can be exactly reproduced from the compressed data), to lossy (where the decompressed data represents the input data but is not exactly the same). Lossy compression usually permits much greater compression than lossless, but with some degradation of quality. For this reason, compression is a special case in which quality is directly affected by policy: when compression standards are set for a system, the result is a loss in quality defined as acceptable given the space savings. Lossy compression methods allow designers, developers, or operators to select a parameter that controls the variance of the decompressed data from the original. There are various terms 1 February /77

33 used for this parameter the JPEG compression standard uses the term quality factor; other systems use the term compression ratio. Setting the parameter for higher compression reduces the quality of data. Overcompression may introduce artifacts and indistinct edges into data, as shown in Figure 13. The artifacts in over-compressed images have a direct impact on encoding and matching accuracy. In IAFIS, extensive analysis was conducted before defining a 15:1 WSQ* compression ratio for non-latent fingerprints; lossy compression was not found to be acceptable at any compression ratio for latent fingerprints. Another problem specific to compression is that of recompression: if lossy compressed images are decompressed and then recompressed, the resulting images will show a dramatic decrease in quality, usually with many details (artifacts) that did not exist in the original image. Figure 13: Details showing over-compression: (Left) JPEG [Example Data]; (Right) WSQ [SlapSeg] Image formatting: some biometric systems use methods of processing other than (or in addition to) compression. Facial recognition systems that collect images from a variety of different sources are particularly susceptible to problems stemming from different types of image processing and formats; two examples are shown in Figure February /77

34 Figure 14: Details of inappropriate processing: limited color space (left) and halftone (right) [Example Data] Image cropping or segmentation is necessary in some systems. For example, in iris recognition, the edge of the iris must be correctly detected for the iris to be correctly processed. As another example, slap fingerprints, in which four fingerprints are captured in a single image, must be segmented into four individual fingerprints. Specialized software used to segment the fingerprints can make mistakes, as shown in Figure 15. A detailed analysis of slap segmentation error was conducted and published in [SlapSeg]. Figure 15: Incorrect segmentation of a slap fingerprint image [NIST-IQS] Software errors or transmission problems can cause corrupted data. Thankfully, these types of problems are becoming less prevalent as software and communication technology mature, but such problems still do occur. Figure 16 shows an example of a corrupted image file. Corrupted data can 1 February /77

35 cause system faults, trigger error conditions, and invoke the wrong processes on data, leading to accuracy and throughput problems. Robust error handling software can help limit the effects of corrupted data. Figure 16: Example of corrupted image: incorrect width shifts each row by 1 pixel [Example Data] 4.4 Feature extraction errors Extraction fidelity refers to the accuracy with which a template, feature set, or model is generated from the sample. Extraction fidelity is directly related to the accuracy of the feature extraction algorithm(s) used. Determining extraction fidelity can be very difficult, primarily because it is often impossible to determine what a correct feature extraction should be for a given sample. For the purpose of being able to evaluate feature extraction, there are three possible alternatives: Relatively few feature extraction models or templates can be created or validated accurately by humans. Notable exceptions are fingerprint minutiae and pattern classifications, where the state of the art in automated feature extraction is still far inferior to an expert fingerprint examiner. During IAFIS design and engineering, it became clear that some AFIS vendors had never evaluated the accuracy of their minutiae detection algorithms. IAFIS conducted extensive evaluation and tuning of feature extraction and pattern classification algorithms based differential comparisons with ground-truth* feature sets defined by teams of human experts. This analysis evaluated the proportion of true, false, missed, and misplaced features. The existence of a large pool of latent fingerprint examiners who are highly trained specifically 1 February /77

36 to detect fingerprint features made such analysis possible. It should be noted that some types of fingerprint feature definitions defined for automated use (such as localized pattern flow) are less appropriate for human groundtruthing*. Feature extraction models that are defined in a standard can be evaluated by comparing the results of multiple implementations of that standard. These tests can use a round-robin method of comparing the match results of the various templates, such as is used in NIST s ongoing Minutiae Exchange Test 2004 [MINEX04]. A round-robin test of feature extraction can evaluate the algorithms in relative terms (indicating the best and worst) but does not evaluate them in absolute terms (indicating how good they could theoretically be). It is possible to estimate the validity of each feature by a voting process, in which the truth of the feature is based on the number of algorithms that detect a given minutiae; unfortunately, the majority is not necessarily correct, so this approach should be used with caution. It may be difficult or impossible to evaluate extraction fidelity for template definitions that are implemented in a single algorithm, and that cannot be validated by a human. Many biometric templates fall into this category. Such feature extraction algorithms have unknown extraction fidelity. 4.5 Matching errors No biometric matcher is perfectly accurate. Matchers have to deal with non-zero false reject rates (FRR) and false accept rates (FAR). Samples that cannot be processed may be rejected as failures to enroll (FTE)* or failures to acquire (FTA)*, or may be passed to the system where they will presumably become false rejections. Multi-stage matchers have specialized types of errors, such as binning errors* which for our purposes can be treated as a special case of false rejections. False rejections, failures to enroll/acquire, and binning errors: False reject rates (also know as false non-match rates (FNMR)) are often 1-2% or higher in identification systems, especially after including the other types of errors that result in failures to match, such as binning errors and failures to enroll (FTE). False reject rates have a direct and obvious effect in that the person being searched was not detected, and therefore the system failed at its primary purpose. For identification systems in which probes that are not successfully searched are added to the database, FRR has a second effect, in that it creates unconsolidated records in the database. For example, if a system has a 2% FRR, that means that 2% of the time an individual who is in the database is not detected, and a second record is created. These unconsolidated records 1 February /77

37 are difficult to detect, because the biometric samples in them are by definition difficult to match. False accept rates: While false reject rates are directly responsible for unconsolidated records, false accept rates are less directly responsible for misidentified records. If a false accept were to occur, there would be an obvious and serious failure in that the subject was incorrectly identified. However, it is not clear that a database error would necessarily follow. A new record would not be created for the probe. In most cases, none of the samples from the probe would be added to the database, so the probe would generally be ignored, and the database would not be in error. Because misidentifications are very serious, FARs are set to extremely low levels in most identification systems, and may be augmented by human verification. For these reasons, misidentified records due to false accepts are unlikely to occur, as opposed to consolidations due to false rejects, which are typically more likely. 4.6 Administrative and database problems The administrative tasks of entering, maintaining, and updating biometric data in a database all can result in metadata errors, specifically unconsolidated records and misidentified records. These problems include data entry errors, reuse of identification numbers, merging of databases, and administrative attack: Data entry errors are a curse of any system that relies on human input. Large systems generally use much less human data entry than they once did, but this still can be a problem. An example seen in one smaller AFIS (long since corrected) was that when a subject was found to have a criminal record on a different system, there was no facility to automatically fill in that information, so the long, alphanumeric identifier was written on a yellow sticky note. Some percentage of the time, digits were transposed, which meant that that the subject was not associated with his own criminal record, but with an invalid/unused record, or even a different person s record (a misidentification). Reuse of identification numbers will always result in misidentified records. As an example, several recent lawsuits have alleged that a livescan fingerprint system in some cases would reuse supposedly unique reference numbers when it printed out fingerprint cards, and that the numbers were used in some state or local AFIS to incorrectly associate subjects with different people s criminal histories. 1 February /77

38 Merging databases is a serious source of error in any database, not just those used in biometric systems [Dasu1]. The problem is that some subjects may be found in both databases, but that there are no foolproof ways of finding all of these duplicates without error. If the databases are combined without performing this consolidation* task, the result will be unconsolidated records. It seems natural to check for consolidations by running biometric checks of each subject as the databases are combined. Unfortunately, this is a solution that is not always practical due to scale: if the databases concerned each have millions of records, the time and computational resources needed for a complete consolidation check may not be practical. In addition, non-zero false reject rates mean that not all unconsolidated records will be detected (as explained in section 4.5). Consolidations can be based on name or identification numbers if such are not automatically assumed to be matches, but are backed up by biometric matching or human review. Consolidation checks can also be performed on a limited subset of subjects, based on similar attributes such as location or age. Administrative attack refers to fraudulent or deceitful activity by people responsible for administering the system. This includes fabrication or modification of data (including databases, links, and linked information) by investigators, examiners, or operators. Documented cases of administrative attack include fingerprint fabrication, mixing of samples, changing links to evidence, and falsifying documentation. 5 Implications of Poor-Quality Data Biometric data quality problems reduce the accuracy of matching, and in extreme cases make matching impossible. Depending on the system, quality problems may also have additional effects, including slower or more computationally intensive processing, and long-term performance consequences. In a study conducted as part of the design of IAFIS, [Kiebuzinski] found that Poor quality images result in decreased reliability [true accept rate] for both ten-print and latent searches, resulting in increased response times, and require substantial computer and personnel resources while making fewer identifications. 1 February /77

39 5.1 Decreased matching accuracy The implications of sample quality on matching Poor sample quality increases the system s false reject rate (FRR), and thereby decreases the true accept rate (TAR)*. This is critical: for identification systems, these errors represent failures to provide linked information (missed criminal records, gang and terrorist group affiliations, historical visa applications, etc); for access control systems, these errors keep authorized people from accessing information or facilities. This degradation in accuracy can be seen in the following charts, which show results from [FpVTE] for two matchers on fingerprints of different quality. Figure 17 shows that there is distinct separation in performance for NIST s VTB reference matcher [NFIS], which was roughly average in performance in the tests. Figure 17: Effect of quality on performance for NIST s VTB reference matcher, on the FpVTE Medium-Scale Test (MST) [FpVTE] Figure 18 shows the same breakdown in data for the NEC matcher, which was one of the more accurate matchers in the evaluation. Note that the scale on the 1 February /77

40 Y-axis (True Accept Rate) has been changed. Accuracy is higher for this matcher than for NIST-VTB, but the relative distinctions between fingerprints of different quality remain: poorer-quality samples are still associated with poorer performance. In absolute terms, a higher accuracy matcher is more robust in terms of quality: the absolute performance of poor-quality data increases, and the differences between samples of different quality decrease. Figure 18: Effect of quality on performance for one of the more accurate commercial matchers. Note the scale of the Y-axis has changed. [FpVTE] The increased likelihood of false rejections for poor-quality samples stems from less meaningful data that can be used in match comparisons. For fingerprints, a decrease in quality results in fewer true features and more false features [Kiebuzinski]. For iris, [Daugman04] states even for poorly focused eye images, the bits of a demodulation phase sequence are still set, primarily by random CCD noise. This prevents poorly focused eye images from being falsely matched, as they may be in amplitude-based representations. The effective result in both cases is that scores for mated comparisons are substantially reduced for poorquality samples, and that scores for non-mated comparisons may be unaffected or slightly increased due to correspondence from random noise or false features. In any case, the result is decreased separation of similarity scores between 1 February /77

41 groups of mated and unmated pairs of biometrics. Because the relationship between mate and non-mate distributions differs based on quality, there is a good argument to adjust decision thresholds based on sample quality. Unfortunately, adjusting thresholds based on quality will improve FRR at the expense of FAR, resulting in more false accepts. This is known as quality-based thresholding, and is discussed further in Section 8.1. For identification systems, undetected false accept errors can have serious consequences such as wrongful arrest, detainment, subject delays, sentencing errors, additional investigation, or denial of visa. In access control applications, false accepts allow people unauthorized access to information or facilities and potential leaks of classified information and materials The implications of metadata quality and extremely poor sample quality on matching The relationship of sample quality on matching is basically continuous: as sample quality decreases, matching accuracy decreases. For metadata quality, the effect is binary: if the database information is incorrect, the subjects cannot be correctly associated. In general, unconsolidated records are incorrectly treated as if they were false accepts, and misidentified records are treated as if they were false rejects. This is shown in Figure 19: note that the performance of each matcher is bounded by a TAR of 98% and (to a lesser degree) by a FAR of This can be used as a good indication (but not a proof) that 2% of the supposed mate comparisons are actually misidentified records, and that (assuming in this case that a total of 10 7 comparisons were performed) about ten (10 7 * 10-6 ) of the supposed non-mate comparisons are actually unconsolidated records. 1 February /77

42 100% 90% Misidentified records are incorrectly treated as false rejects, causing this dropoff. 80% TAR 70% 60% Unconsolidated records are incorrectly treated as false accepts, causing this dropoff. 50% 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00 FAR Figure 19: Effect of unconsolidated and misidentified records on matcher accuracy In practice, the relationships between metadata quality and matching are somewhat more complex. The drop-off in TAR at the right of the chart (where FAR=1) is caused either by misidentified records or by samples that are of unusably poor quality. Unusably poor quality refers to mated samples that are of such poor quality that the matcher cannot determine any similarity between them whatsoever, which is what FAR=1 means. This is a good systemindependent definition of the term failure to enroll (FTE). Often these unusably poor quality samples are nearly or completely blank, or are samples that cannot be processed by a feature extractor. In any system, TAR cannot exceed a limit defined by the proportion of misidentified records and unusably poor-quality samples in the dataset. The drop-off at the left of the chart is likely to have been caused by unconsolidated records, especially since there is general concurrence among the three matchers, but such a result could also include actual false accepts, samples that are coincidentally similar. At the left end of such a chart, a very small number of comparisons can result in such a stairstep. 1 February /77

43 If metadata errors were caused by administrative problems (discussed in section 4.6), then the resulting unconsolidated records will appear to be non-mates with unusually high scores, and misidentified records will appear to be mates with unusually low scores; these cases will be fairly obvious during analysis. Unfortunately, database errors that were caused by matcher errors (section 4.5) are much less obvious, because their scores are by definition not anomalies, and will appear within the ordinary non-mate and mate distributions. Detecting database errors that were caused by matcher errors requires different matchers, samples, instances, biometric modalities, or non-biometric information. This is one use of a variation of biometric fusion: if multiple samples, instances, or modalities are used in a system, database errors are much less likely to occur, and are much easier to detect if they do occur Long-term consequences Poor-quality samples used as probes (searches) have immediate consequences in terms of matching errors, although they may not be recognized immediately. These errors may never be recognized, or may be recognized only after extensive review. In addition, we need to remember that we should not regard the poor-quality sample as a problem that goes away when that first search is conducted. For many identification systems, searches that do not detect mates are automatically enrolled in the database. Poor-quality biometric data that is enrolled in a database can have two long-term consequences: Poor sample quality in the new enrolled record will degrade accuracy for all searches in the future. This problem may occur because of a change in policy (accepting lower image quality data), because of changes in the characteristics of the data collection devices, or other factors. The net result maybe a bimodal distribution of the database quality which may be corrected in part by appropriate optimization techniques related to the age of the data. A poor-quality sample that cannot be matched with its true mate and is then enrolled in the database will result in a new unconsolidated record, which then will be the source of new problems. 7 This will be discussed further in the forthcoming NIST report Selected Topics in Biometric Fusion. 1 February /77

44 5.3 Performance degradation [Kiebuzinski] reported that a poor quality fingerprint search can require substantially more in the way of computer resources and processing time than a normal quality search. This does not necessarily apply to all biometric systems, but is a serious issue for some. There are three cases in which poor-quality samples may be more computationally complex to process than normal samples: For systems that use human verification, this stage is much slower for poorquality samples. [Kiebuzinski] Multi-stage systems use fast screening matchers or classifiers exclude the majority of obvious non-matches so that the computationally demanding detail matchers only need to process a small proportion of all possible comparisons. Poor-quality samples cannot be excluded as readily by screening or classification, and therefore must be processed by the detail matcher. For a typical AFIS and a normal-quality probe sample, only a few percent of the gallery needs to be compared against the probe by the detail matcher(s); for a poor-quality probe, this may require comparison against the majority of the gallery, which results in at least an order of magnitude increase in computational demands. Some matchers and feature extraction algorithms have processing time and computational requirements that do no vary substantially between samples. However, for some, the processing time and computational requirements increase based on factors such as image size and the number of features detected; many minutiae-based fingerprint systems fall in this category. Poor quality images that contain a great deal of noise and that therefore have many false features can take much more time to process that an ordinary sample. 5.4 System attack In a 2004 report to Congress, Stanford Business School Professor Lawrence Wein discussed the possibility that terrorists could take advantage of the accuracy implications of poor-quality samples. As he stated in a later publication, The present study stems from the belief that terrorist organizations can exploit the image quality-dependent performance of the biometric identification system by choosing from their large pool of potential U.S.- bound terrorists, those that have either inherently poor image quality (e.g., worn out fingers) or deliberately reduced image quality (e.g., surgery, chemicals, sandpaper). [Wein05] 1 February /77

45 Wein s argument is lessened somewhat by the assumption made in his model that the subject is the primary determinant of sample quality, but his point is still worthy of consideration. 6 Detecting and Measuring Biometric Data Quality Biometric data quality is complex, and measuring data quality is no less so. Data quality measurements need to address this complexity by quantifying parameters that affect or are related to system performance. Sample quality metrics may be directly related to performance (as a predictor), a measurement of the data content of a sample, or a measurement of the reliability of an individual feature. In addition, different types of quality metrics can be used to measure the fidelity of capture devices, or to measure the frequency of a characteristic across a population. Detecting metadata errors is discussed in section Approaches to measuring sample quality Sample quality measures properties of a sample that affect system performance, and pertains to both probe samples and gallery samples. Sample quality can be measured at various different levels: Lower-level quality metrics may be useful by themselves, but are more likely to be of value when combined into an aggregate quality metric: Generic image quality metrics provide measurements using broadly applicable image quality metrics that are not specifically oriented towards a biometric system. A large number of generic methods are available. Feature-specific metrics and localized quality metrics are based on the reliability of individual features within the sample. Aggregate quality metrics are based on combinations of lower-level quality metrics: Representation-based metrics are aggregate measures of the data content of the sample. Matcher prediction metrics are aggregate measures that use other types of measurement to predict the scores of one or more matching algorithms. Matcher score analysis is the process of using matcher scores from multiple samples to measure quality. Since metrics (especially matcher prediction 1 February /77

46 metrics) are estimates of matcher performance, the actual matcher scores can be seen as the definitive measure of data quality. Stage-specific metrics measure the fidelity of a stage of processing, such as data compression. They differ from the other measures in that they do not measure the overall quality of the sample, but the degradation of quality that can be attributed to a given stage of processing, such as scanning or compression. It should be noted that there is not universal agreement as to what should be considered a quality problem, or what characteristics should be measured as inputs to quality metrics. To use fingerprints as an example, is the overall size a quality-related metric? Is the degree to which the image is centered? Is rotation? For quality metrics that have the end goal of maximizing the correlation with matcher score, any such factors that affect the matcher score need to be considered among the various inputs to the quality metric Generic image quality metrics Since many biometric samples are images, generic image quality approaches that are not specific to biometrics can be used to assess their quality. 8 These approaches grew out of digital image processing techniques and analyses of the human visual system (HVS) and use measurement methods that can be applied to any type of image [Schalkoff, Chalmers]. Examples of generic image quality metrics include the brightness, contrast, entropy (pixel variance), spectral magnitude/phase, and the like (the complete list is quite extensive). Since many biometric systems are designed to extract information from images, measuring and controlling generic image quality can improve system performance. Some generic image quality metrics can be effective when used in assessing biometric sample quality, such as the use of an image's spatial frequency power spectrum as a measure of quality [IQM]. Even those generic image quality metrics that are not of great value in measuring sample quality by themselves can be of use as inputs to an aggregate quality metric. For example, the number of colors (or shades of gray) in an image is not generally an effective quality metric by itself, but is a valuable input to aggregate measures, which can use limited color space as one of many indications of poor quality. 8 One of the reasons to refer to sample quality rather than image quality is that some people argue that the term image quality metric should be used solely to refer to generic image metrics rather than biometric-specific metrics. 1 February /77

47 6.1.2 Feature-specific metrics and localized quality metrics Feature-specific metrics are based on the reliability of individual features within the sample. These are likely to be by-products of feature extraction algorithms. For example, a fingerprint minutiae detector may determine the relative certainty that the minutia is valid, and have an area of uncertainty around the defined location. The same encoder may assign a quality value to regions or even every pixel in the image: these are known as localized quality metrics. These featurespecific metrics and localized quality metrics are used in aggregate or global metrics. In addition, these metrics are used in the fault-tolerant logic of matchers, so that the relative importance of a specific feature in matching is based on the quality of that specific feature Representation-based aggregate metrics Representation-based metrics are aggregate measures of the data content of the sample. Representation-based measurement makes an overall assessment of the quality of the sample, generally based on a combination of individual feature- (or stage-) specific metrics. For example, a low-ridge count whorl fingerprint sample could be expected to contain two deltas; if one or both are missing, the sample may be missing fingerprint pattern area and could be considered poor quality. Examples of representation-based metrics include: Fingerprints pattern location, number of minutiae, number of minutiae of quality X, overall minutiae quality, pattern area quality, ridge flow consistency, overall ridge quality, core quality, and ridge frequency [IQS] [Chen1]. Iris camera quality and imaging conditions, iris boundaries, eye-lid occlusion, imaging angle, image blur, and wavelets [Daugman05] [Chen2] [Dorairaj] [Kalka]. Face head and face location and cropping, imaging angle, identification of eye glasses and other obstructions, and eye-finding [IDX] Matcher prediction aggregate metrics Matcher prediction metrics are aggregate measures that use other types of measurement to predict the scores of one or more matching algorithms. Many matcher algorithm vendors have developed matcher prediction metrics for their own purposes. There are a variety of matcher prediction metrics that have been implemented over the years. For example: 1 February /77

48 When IAFIS was engineered, metrics were designed and tuned to predict the behavior of individual algorithms within the matching process; these matcher prediction metrics have been critical to the efficient operation of the system. The IDENT/IAFIS Image Quality Study [IQS] evaluated fifteen fingerprint quality metrics (stage-specific or representation-based) for the purpose of predicting IAFIS performance on fingerprints of different types. A combination of rule-based and linear regression methods were used to define an optimal combination of four of these metrics, resulting in a Unified Image Quality Metric that uses the same scale as the target matcher scores. The metrics used by UIQM were themselves aggregate metrics [NIST-IQS], based on multiple feature-specific metrics (minutiae quality), localized quality metrics (assessment of localized ridge flow quality and consistency), and general image quality metrics (contrast). [NFIQ] introduced a model and implementation for generic matcher prediction, based on approaches can be applied to any matcher, including an elegant method of using a neural net to associate several lower-level quality metrics with match scores from multiple matchers. NFIQ is based on counts of minutiae of differing qualities, and aggregate measures of ridge flow quality and consistency. It is important to note that there are limits to the effectiveness of using sample quality to predict matching accuracy [IQS]. Matching accuracy depends on the following factors: Sample quality for both the probe sample and the gallery sample. Since matchers compare probe and gallery samples, poor quality in one or both hurts matcher accuracy. Sample correspondence between the probe and gallery. For example, two poor-quality face samples can match with a high score if the pose, lighting, and expressions correspond closely; two distorted fingerprints can match well if distorted in similar ways. Measurement of such correspondence requires use of both probe and gallery samples, and a matcher. It can be argued that the degree of correspondence should be considered a quality issue. For example, face matcher scores between captures will vary based mainly on trivial variations in lighting, face angle/pose, lens type, and expression: extreme problems with any of these factors are clearly quality problems, so these correspondence issues can be seen as minor or immeasurable cases of quality issues. 1 February /77

49 Metadata quality. Metadata errors are insidious because they are difficult to detect, cannot be detected by evaluation of a single sample, and can make a match fail even if all of the other factors are perfect. A matcher prediction metric is only as good as the care spent in its training: if the matcher(s) used do not fully represent all target matchers, or if the samples used in training do not fully represent the range of samples to be used, the metric cannot be expected to be reliable. When training, is may be wise to include nonbiometric samples in the training set: the authors have seen quality metrics that designate completely blank images or images of trees as good quality! The factors that affect the accuracy of one matcher may not affect the matcher score for all matchers, so a single metric will not correlate equally to matcher performance for all matchers. Generally, more accurate matchers gain that accuracy by being tolerant of many characteristics that would cause some less accurate matcher to fail: the differences in accuracy between matchers can mostly be attributed to differences in the processing of poor or marginal quality samples. Therefore, a metric tuned to those characteristics that cause lessaccurate matchers to fail may identify as poor quality many cases that would match without problem in more-accurate matchers; a metric tuned solely to more accurate matchers would only identify a subset of the cases that would be problematic for less-accurate matchers. Human examiner-based measurements are a variation of matcher prediction metrics that predict human examiner matching performance by quantifying certainty associated with their decisions about a sample (ranging from easy to identify to unidentifiable). This approach asks examiners to judge whether a sample can be accurately matched. Low variance in their response indicates agreement among the examiners and more certainty whereas high variance indicates less certainty. Classifying samples by data quality type could allow for extrapolations beyond the survey set of data (Section 6.1.6) Matcher score analysis The gold standard of quality is the actual measurement of matcher scores. In practice, this can be complex. Multiple samples are used to limit sample-specific factors (a good image can match poorly if the gallery image is bad). Having only two samples per subject will not permit isolating quality as a variable (one match score needs two samples), so three or more samples are necessary. Likewise, multiple matchers should be used to avoid matcher-specific characteristics. 1 February /77

50 Matcher score analysis was used to determine ground-truth quality in [NFIQ] and [Goats] Stage-specific metrics A similar category of metrics includes those measuring the fidelity of a stage of processing. For example, a compression algorithm can use a peak signal-to-noise ratio to express a Compression Fidelity Score, a classification algorithm can assign a value to the certainty of its determination of class, or scanner certification procedures [Nill] quantify the fidelity of image capture (discussed specifically in Section 6.3). 6.2 Population quality measurement For some characteristics, measuring samples does not provide an accurate method of measuring some traits, especially if rare or unevenly distributed. Human characteristic measurement or population analysis relies on demographic, medical, and forensic techniques to measure fundamental information about biometric characteristics and their distribution in the population. For example, the implementers of an iris system may need to have an understanding of the percentage of the population with missing eyes. Knowledge of at least some of these measurements is crucial to developing systems and this type of knowledge can be used to support operations and effective system use. Nystagmus is an involuntary rhythmic oscillation of one or both eyes [NYST]. This motion may be accompanied by tilting of the head. Iris identification systems can accommodate people with this condition if they capture a sequence of images and use the one that is corrected for tilt and on-axis [DaugmanB]. Other problems like Aniridia (congenital absence of the iris) that affects 1:50,000 people cannot be overcome by iris identification alone [DaugmanB]. Human characteristics like these form the basis for biometric systems. Qualitative and quantitative information about characteristics exposes natural limitations and enables systems to overcome them. 6.3 Collection device fidelity Collection device fidelity is a specific and extremely important quality metric, used to test collection devices to measure how accurately the output samples represent the inputs. Collection device fidelity measurements use controlled test targets to measure how well a collection device represents input information. The Federal Bureau of Investigation s (FBI) Electronic Fingerprint Transmission 1 February /77

51 Specification (Appendix F) provides standard metrics for geometric accuracy, modulation transfer function, signal-to-noise ratio, and gray scale ranges, linearity, and uniformity [EFTS]. It specifies test targets and acceptable measurement ranges. [Nill] provides test procedures for verifying fingerprint scanners and printers. As implemented, the specification and procedure are a standard for measuring and controlling the design of fingerprint scanners and printers. Fidelity metrics can be designed and developed for many types of collection devices. Imaging device quality can be measured using the standard metrics above although threshold parameter values will differ. 6.4 Automated quality measurement Many measurement techniques can be automated or partially automated. They often rely on a sample that contains information pertaining to human characteristics and collection device fidelity as well as other quality factors such as subject behavior, collection process, and environmental conditions. Automation provides objectivity, consistency, repeatability, and cost reduction, and supports (but won t replace) effective quality management that reviews quality related problems with the objective of improving the system. After test target samples are collected, the calculations and threshold comparisons for collection device fidelity can be automated. This provides a convenient method for assessment since operators need not be familiar with the detailed theoretical and mathematical calculations of the quality assessment. They simply need to capture samples of the test targets. The quality assessment can be conducted at acquisition, integration and installation, and periodically during operations to assure that collection devices are working properly. Generic image quality, representation-based methods, matcher prediction/score approaches, and sample quality classification can be fully automated. If there are sufficient computer resources, they can be executed in real-time, providing instantaneous feedback to the operator. They can also be used to feed quality parameters forward to matching and decision processes to reduce errors. If computer resources are limited at operator sites, fast versions can be created for instantaneous feedback and rigorous versions can create matcher information or trade-offs can be assessed to determine optimal configurations. One advantage of these methods is their ability to operate continuously; minimizing the collection of large amounts of poor quality data due to not detecting a device failure in a 1 February /77

52 timely manner. They can also be retained in a database to provide quality monitoring and analysis capability (Section 7.2.3). Sample quality classification and objective measurements can be used extensively in system analysis. For objective measurements, computations, database queries, threshold comparisons, and reporting can be automated although test conduct and analysis typically requires human oversight and analysis. Both quality classification and objective measurements provide consistent benchmarks for system evaluation and support extrapolations that reach beyond the information generated by testing alone. 6.5 Biometric quality metric errors and implications Quality metrics have an inherent possibility of measurement error, whether automated, partially automated, or manual. The determination of error is not trivial, as it requires comparisons against some method of determining the ground truth of quality. No method of groundtruthing quality is ideal: generally these are determinations or estimates of ground truth based on judgments of experts or matcher scores obtained from multiple samples for each subject. The use of matcher scores to groundtruth quality is effective if multiple samples are available for each subject, but the process of selecting or creating datasets with multiple samples per subject should be reviewed carefully for bias. Also, groundtruthing by using matcher scores cannot readily differentiate between causes of quality issues. Human judgment is subjective but provides an ability to account for factors that may be difficult to identify through the use of matcher scores. Measurement errors may be discrete (e.g. false labeling) or continuous (e.g. spatial feature location). A specialized case is binary labeling (i.e. accept/reject) that is required for re-collection or rejection decisions. Measurement errors for binary quality determinations have binomial distribution if they result from statistically independent trials. Other measurement errors may be multinomial or may have continuous distributions. Most quality metrics ultimately have one or more thresholds that are used to differentiate between acceptable and unacceptable quality. These determinations have type I and type II error rates, which are similar to FRR and FAR: A good sample is falsely labeled poor (Type I similar to false reject) A poor sample is falsely labeled good (Type II similar to false accept) 1 February /77

53 Type I and II error rates can be measured using tests similar to those developed for matcher accuracy although care must be taken to differentiate sample-specific quality from subject-specific quality and account for statistical dependencies when estimating probabilities. Dependency between the error types can be represented by a ROC curve and operating points can be chosen to tradeoff the errors. The consequences and costs of errors vary depending on how the metric is used. The distribution of other measurement errors will vary by the type of measurement. Discrete classification errors will be generally multinomial and can have various combinations of errors. For example, three mutually exclusive quality classes (A, B, and C) have nine possible error types. A may be misclassified as B (A-B error) or C (A-C error). There are also B-A, B-C, C-A, and C-B errors (there are N 2 -N error types for N classes). A confusion matrix can be used to describe these errors. In this representation, each cell in the matrix contains one of the misclassification errors. For errors that have a continuous distribution, functions may be selected from various classes to fit their distributions. In addition, there are continuous distribution approximations for some discrete distributions. 6.6 Uses for sample quality metrics Sample quality metrics have multiple uses. As an example, a large-scale identification system may use quality metrics in various ways in processing: At capture locations, quality metrics are used to flag samples that must be recaptured. Quality metrics (especially over time) can be used to identify problems associated with a given device, operator, or location. For systems that have limited control over capture locations, image quality metrics may be employed when the samples are received, providing a basis for rejection. Quality metrics can be used for quality-specific tuning of match process. If multiple biometric instances or modalities are collected, processing of good-quality data may only require the use of a subset of these, while poorquality data may require the use of a broader range of data. Quality metrics are used to select the data used for matching. 1 February /77

54 If a match is successful, quality metrics can be used to determine if the existing gallery samples should be replaced (or augmented) with the probe samples. Data quality measurements can be used to help assess system effectiveness. For example, measurements of human characteristics can provide insight into performance boundaries, helping define system goals, and sometimes showing that historical views of boundaries can be expanded. System managers and engineers can use data quality information for system evaluation, analysis and planning. For example, a manager presented with a high percentage of excessively skewed images could initiate review of data collection processes and initiate better operator training and guidance. In another example, performance measurements of system components can be used as a basis for acquisition or development, and can be used to help identify errant operational components. Analysts and forensic scientists can estimate the certainty associated with match decisions and change methods accordingly. Measurements are the basis for quality control. At the front end of the system, automated quality metrics can be used as the basis for recapture or rejection decisions (Sections 7.2.1, 7.2.2). These metrics support a binary decision about whether to process a sample or collect a new one. Quality metrics can be used as the basis for more intricate processing. Processing chains for samples with specific properties can be developed and executed conditionally. For example, specialized image enhancement could be used in some cases and it might not be needed in others. In addition, quality-specific match decision thresholds can be implemented (Section 8.1). Quality metrics can help with the identification of problem collection devices, problems with specific locations, and process and procedure problems. These processes can be implemented to provide continuous or periodic monitoring (Section 7.2.3). These metrics can be retained for system analysis and planning. 6.7 Metadata quality detection The detection of metadata quality problems is very different from the detection and measurement of sample quality. Metadata quality problems cannot be detected through analysis of a single sample, but are most readily detected through analysis of outliers in matcher results. For example, many unconsolidated records can be detected by finding supposed non-mates with 1 February /77

55 high matcher scores, while many misidentified records can be detected by finding supposed mates with low scores. See section for more detail. Operational consolidation of unconsolidated records is practical and strongly recommended: any operational search of a database may come back with more than one valid match; these should be investigated as likely candidates for consolidation. Special cases such as errors in slap fingerprint segmentation require specific analysis: [SlapSeg] describes methods based on matching every finger on a hand with a different (rolled) sample of every finger on a hand, so that fingers out of order can be detected by matches with an unexpected finger. Once metadata errors are detected, the administrative process of correcting the problems is known as data cleansing. 7 Prevention of Poor-Quality Data Because the implications of poor-quality data are so serious, it should be obvious that poor-quality data should be prevented or corrected whenever practical. Prevention can and should focus on each of the causes listed previously (discussed in section 7.1), but also requires a holistic, system-wide focus on quality as a matter of policy (discussed in section 7.2). 7.1 Controlling quality at the source Each of the causes of quality problems enumerated in section 4 should be addressed in an attempt to detect, prevent, and/or correct such problems at the source Intrinsically limited data content Subjects with limited intrinsic characteristics may not be a preventable problem. The best solution is likely to be collection of additional data: use of multiple instances (e.g. fingerprints from ten fingers) or multiple biometric modalities (e.g. fingerprint and iris) reduce or eliminate the impact of an individual with a single poor-quality biometric instance. If the system is limited to a single biometric modality and instance, multiple samples should be captured and the best of the set retained; findings from US-VISIT and NIST s MBARK project have indicated that quality improves if multiple images are captured automatically and a quality metric is used to choose the best of the set. Subjects with missing biometrics (such as amputations) should be clearly designated at the time of capture, and that designation should be permanently 1 February /77

56 attached to the biometric record; otherwise, an unexplained blank image will continue to cause problems in the system. (The same is true of temporarily unobtainable biometrics (such as bandaged features). Procedures should be clear to operators so that missing biometrics are consistently treated Degraded or obscured characteristics Degraded biometrics that cannot be corrected at the time of capture should be treated in the same way as subjects with limited intrinsic characteristics. Operators should be aware that they should correct those obscured characteristics that can be corrected (such as glasses, clothing, hair, and dirty fingers) and know which should not be corrected (such as bandages or clothing like veils that cannot be removed for social/cultural reasons) Subject behavior The system and its procedures should be designed to minimize the negative effects of subject behavior. For cooperative or ambivalent subjects, this is primarily a human factors* issue: it should be as intuitive as possible to subjects what is expected of them and how they are to accomplish it. The complexity of the interface visible to the subject should be minimized so that training/habituation is as unnecessary as possible. Usability evaluations of the system interface and physical configuration are important. Some of the issues that may be considered during human factors evaluation may include Collection devices and processes need to be designed for ease of use of both subject and operator. Unnecessary complexity reduces the chance of collecting good quality data. Provide clear and simple feedback to subjects and operators. There may be opportunity to re-collect data if needed. Consider the use of a 2-tiered approach when quality determination may cause queuing problems: fast metric to eliminate obvious problems, a rigorous one for final determination. Devices need to be properly aligned for good quality data collection. Subjects need to be provided sufficient room to adequately position themselves. For example, reach across the body can skew fingerprint images. Design specialized processes for handling subjects who attempt to evade or spoof the system. 1 February /77

57 In environments in which subjects may be actively uncooperative or hostile, consider that the collection device could be used as a weapon, and that close proximity to subjects can be a risk to operators. Design for exceptions as well as the norm. Operators or subjects may have disabilities and can vary in size. Design for consistency of use and consistency of data. Design for anticipated operator fatigue. Data collection is a repetitive process and fatigued operators can let quality lapse. Include operator performance metrics to identify poor or lapsed training and/or data collection device degradation. The appropriate procedures to deal with actively uncooperative subjects are presumably specific to each system. Policies regarding uncooperative subjects should be made clear to operators so that they are uniformly and appropriately managed Fraud The first line of defense against spoofing and evasion is during data collection, through the use of trained operators and supporting technology. Operators should be made aware that evasion or spoofing may be attempted, and be informed of how to identify suspicious behaviors. Subjects acting suspiciously should be confronted. Technology can help prevent fraudulent samples: [Van der Putte] describes using temperature, conductivity, blood pressure, heartbeat, dielectric constant, blood pressure, and detection under the epidermis as methods of verifying livescan fingerprints. Special image processing algorithms can be implemented to detect suspicious data. For example, samples that are too perfect, exactly repeated, are blurred, or contain edges or suspicious marks may be fabricated. Collection of multiple biometric instances or modalities helps reduce risk by collecting more data of different types. Successful evasion or spoofing is not simple for a single biometric instance, but is made much more complex if multiple biometric instances or modalities are collected. For example, an attacker who wants to be identified as someone else would need to forge multiple fingerprints in a multi-fingerprint system and would need a disguise if face images are collected. 1 February /77

58 Unattended collection of biometrics should be suspected as a source of fraudulent samples Collection devices Collection devices should not be a serious source of errors if the specific device being used is compliant at the time of use with well-thought-out quality standards, and it is used according to well-thought-out procedures. This means that Standards for collection devices need to be defined, carefully evaluated, and enforced. Certification processes need to be implemented; use of certified products should be enforced. There should be an ongoing process to monitor the quality of specific devices (see section 7.2.3). Procedures should be in place for rapid replacement of problem devices. A good example of such standards and certification is the FBI s certification process for fingerprint scanners. The most widely-used standard for biometric capture devices is the FBI s IAFIS Image Quality Specifications, Appendix F of [EFTS], which was described in section 6.3. The FBI uses these standards to certify fingerprint collection devices, card printers, integrated products, and identification flats systems. To date, 252 products have been certified, from 41 different companies [CERT]. The FBI certification program, defined in [Nill], requires vendors use their products to collect images of standard test targets. The FBI tests the images and certifies products if they pass the test. Compliance is not intended to endorse any product over any other. This type of testing shows that the product design can be compliant with the standard and requires vendors to produce at least one compliant product. It provides acquirers with an initial reference to identify applicable products. The FBI s certification program does not address validation that individual devices are compliant at a given point in time Collection processes Policies and procedures should carefully define the process of sample collection so that it is performed correctly and consistently. Some of the aspects that should be covered include: 1 February /77

59 Collection policy and procedures should be defined and enforced. Whenever possible, operators should be made aware of quality problems in real time so that samples can be recaptured: recapture of poor-quality data is desirable whenever possible. Operators should be trained and tested as to proficiency. Ongoing monitoring or sampling of operator proficiency should be put in place, through a combination of human and automated processes (see section 7.2.3). Operator workload should be considered so that it does not become a source of quality problems. Operators should be trained in correct and consistent guidance and oversight of subjects, including procedures to handle exceptions (such as missing biometrics or uncooperative subjects). The physical configuration of equipment and the collection environment should be clearly defined in standards or best practices documents. This configuration should be designed so that it can be readily replicated in different locations. Illumination, temperature, and humidity need to be considered. Fingerprint, face, and iris use imaging that vary with lighting. Sunlight can cause harsh shadows and has been observed to cause halos in livescan fingerprint devices. Temperature and humidity can change the characteristics of collected samples, or increase the risk of device failure. One example of a successful collection policy is from the DHS Benefits program, which was notable in terms of quality and consistency for an operational data set in the SlapSeg evaluation. Those collection guidelines are included in Appendix B.11 of [SlapSeg] Compression and sample processing Compression and sample processing problems are determined by the algorithms used. The compression and sample processing algorithms must be evaluated to determine their impact on sample quality. Once acceptable use of the algorithms is determined, correct use of the algorithms and use of the correct parameters must be defined and enforced. When possible, parameters that should be set system-wide should not be available for manipulation by operators. 1 February /77

60 7.1.8 Feature extraction Feature extraction fidelity is determined by the algorithms used. Systems that have developed their own feature extraction methods (or have engineering oversight over development) should evaluate their effectiveness rather than assuming that they are optimal by definition. In many systems this is not possible Matching accuracy Matcher accuracy is a key concern in every system, and presumably improving error rates is a goal for any system. We assume that the fact that matcher errors cause database errors as a secondary effect underscores this concern Administrative and database issues The processes of entering, maintaining, and updating biometric data in the database all need to be made as foolproof as possible: The system should be designed to minimize human data entry, and to provide immediate lookups of any cross-references to minimize typographic errors. Any significant human-initiated changes, such as merging or deleting records, should be flagged and verified. Keys from other systems that purport to be unique should not necessarily be regarded as unique. Preventing administrative fraud is a system-specific issue, but one that should not be overlooked. Biometric systems can follow the model of forensic applications, which require processes designed to prevent fraud and administrative attack. [Wertheim] suggests keeping evidence intact when possible, good logging and tracking of evidence, using witnesses to corroborate key processing steps, using reviews and audits to show process and procedure is followed. 7.2 Quality policy If the prevention of poor quality data is going to be more than an ad hoc solution, it requires a holistic, system-wide focus on quality as a matter of policy. Policy and administration defines goals, objectives, requirements, acceptable quality levels, processes and procedures, and administrative controls. Quality engineering (e.g. Total Quality Management (TQM)*, or Six Sigma*) provide 1 February /77

61 quality-by-design, analyzing and building quality improvements into each process step. Quality assurance provides confidence that requirements will be met, relying heavily on quality measurement, monitoring and control. Policy and administration are two key elements of systems management that can prevent poor data quality and help improve existing data quality. Policy provides guidance to systems engineers and implementers to help resolve major data quality issues. Administration helps ensure that policy is implemented effectively by reviewing outcomes and taking the actions necessary to meet objectives. Administration reviews the effectiveness of current policy, ensuring that processes are effectively supporting objectives, developing and implementing administrative controls, and ensuring adequate resources. For example, in one face image collection implementation, engineers noticed that the camera was mounted on a boom and operators were swinging the camera to different positions for each image. This led to inconsistent samples and subsequent matching errors. These problems could be prevented by better design focused on human factors that oversaw stationary mounting of the camera and standard subject positioning (Section 7.1.6), defining operator procedures and best practices and training operators on them (Section 7.1.6), and monitoring data quality for quicker detection and remediation (Section 7.2.3). Quality assurance reports quality successes and needed improvements to system management, providing confidence that requirements will be met. An effective quality assurance program monitors system outcomes, issues, and quality and works with system management to resolve problems. Quality assurance can utilize the methods, metrics, and guidelines identified in this paper as an initiator for a successful program Recapture of poor-quality data Real-time re-collection provides operator feedback for poor quality data and allow for additional sample collection. A rigid approach rejects the poor quality sample and requires the operator to collect another sample, resulting in an FTA or FTE error if a good quality sample cannot be collected. A less rigid approach provides guidance to the operator to improve the sample and can indicate not what is wrong, but what can be done to correct the problem (e.g. Press harder ). Systems can collect multiple samples, allowing for selection of the best one. Humans are not necessarily accurate at monitoring data quality: studies in US- VISIT or MBARK have found automated detection and acceptance of sample 1 February /77

62 quality is more accurate than human. Some systems provide an operator override capability so that a sample can be entered regardless of the automated quality assessment: while these are necessary in some exceptional circumstances, studies of IDENT and VISIT found that human override was often used inappropriately, usually in reaction to time pressure. Real-time re-collection improves the sample and improves chances of a true match. Re-collection, however, requires more time and use of resources. Estimating the workloads associated with re-collection and having adequate resources can help reduce queuing issues Rejection policy One seemingly easy way of preventing poor-quality data is to refuse to process it. By labeling poor-quality inputs as rejections or failures to enroll (FTE) the system accuracy on the remaining data will increase, and the other implications off poor-quality data will abate. A long-standing quip among skeptical readers of rosy predictions of performance is that the greatest determinant of system accuracy is not the quality of the matcher, but the person who gets to determine how much data is disregarded as FTE. Whether any data can be rejected or how much varies depending on the system. Systems that have direct, real-time control over the people and devices that collect their data have leverage that they can use to guarantee quality, possibly by rejecting and demanding new data, possibly through punitive measures. However, some systems do not have the luxury of disregarding any data. Watchlists in particular have to deal with whatever limited biometric samples can be obtained, regardless of quality. Systems that rely on multi-agency of inter-jurisdictional data have similar issues. One example relates to the change in quality of inked fingerprint impressions over the last few decades. Early law enforcement policy defined ten-print fingerprints to be clear rolled ink nail-tonail impressions on standard fingerprint cards. This policy was administered with a manual review and rejection process that prevented poor quality fingerprints that enabled development of early scanning, digitization, and automation of fingerprint matching technologies. As IAFIS was being implemented, analysis showed a clear deterioration in the quality of fingerprints submitted to the FBI from the mid-1960s through the mid-1990s [Kiebuzinski]. The reason most frequently cited for this degradation is a decrease in training for operators, particularly in respect to civil background checks. Rejection policy in such cases was not straightforward, especially because of the time lag due to paper-based processing. If a fingerprint card was rejected by the FBI, the 1 February /77

63 collecting agency often would refuse to submit another, leaving an unsatisfying choice of poor-quality data or missing the data altogether. Rejection policy depends on the specific requirements of the system in question. Systems that have the ability to reject data because they have the control or leverage to get replacement data should take full advantage of this. But it should be noted that not every system has this luxury Quality monitoring Quality monitoring is a key tool in determining the sources of quality problems and correcting them. By implementing an ongoing quality monitoring process that measures the overall quality based on factors such as location, device, operator, and time, a system can isolate sources of failure. Quality monitoring is based on the analysis of the distribution of values from image quality metrics. Outliers could be indicative of a quality problem. If the quality distribution associated with a specific location, device, or operator differs substantially from the norm, corrective action can be taken. Bad quality data from a specific location can be monitored, controlled, improved, or labeled. Operators making mistakes can be trained or fatigue can be identified early. Failing devices can be replaced or repaired quickly before adding more poor-quality data into the system. Quality monitoring should be continuous if possible, or at least requires sufficient sampling frequency to allow for quick corrective action. High volume / high throughput systems need higher monitoring frequency because a point failure can introduce a lot of bad quality data. A collection device could fail after its monthly test and continue to provide poor quality data until its next check. 8 Remedies for Poor-Quality Data Preventing or retaking poor quality samples before they get into the system is obviously desirable, but may not always be feasible. Most organizations managing watchlists have to use whatever data they can get. Searching a poor quality sample suspected to be from a highly sought fugitive often outweighs negative consequences. A large gallery created from legacy data cannot be changed quickly. In cases like these, biometric systems have to process the data at hand. There are options available, if poor-quality samples must be processed, beyond simply processing the data as normal. Operational methods for handling poor quality data include accepting different tradeoffs between error rates, 1 February /77

64 optimizing the existing algorithms based on sample quality, using specialized algorithms, and using biometric fusion. 8.1 Optimizing matching based on sample quality If sample quality drops, average matching accuracy will decrease. This does not necessarily dictate what will happen to specific error rates, though: the system policy that dictates the tradeoffs between false accept rates and false reject rates can vary based on quality. A hypothetical example is shown in Figure 20: note that if the normal system operating point is a matcher score corresponding with (FAR=10-4, TAR=98%), there are various operating points on the poor-quality curve that could be selected: if FAR=10-4 is required, TAR will decrease to 75%, but if FAR=10-3 is required, TAR will only decrease to 81%. 100% 90% 80% TAR 70% 60% Normal quality Poor quality 50% 1.E-06 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00 FAR Figure 20: Tradeoffs between error rates as quality decreases (hypothetical example) This is known as quality-based match thresholding, which was discussed in [Wein] (but the concept long predates that paper): different decision thresholds can be used for determining a match, depending on the quality of the probe. The rationale for this is that samples of different qualities have different score 1 February /77

65 distributions and therefore different tradeoffs between errors, and so it should be more accurate to have thresholds based on quality rather than one-size-fits-all. This approach can also be used for non-quality characteristics such as the type of collection device or the subject s sex. The tradeoffs between FAR and TAR are some of the most sensitive policy determinations made for a system: it should be clear to the decision makers that they vary and can be set differently based on quality. The settings of these parameters in theory could be related to the perceived risk levels and they could be used as an element in a game theory implementation to balance the system accuracy performance versus response time and system cost, as discussed in [Wein]. More sophisticated than quality-based match thresholding, but more computationally demanding, is a method known as cohort normalization, or probe-specific normalization. The rationale for this is that every probe has a different score distribution and therefore different tradeoffs between errors, and so it should be more accurate to have threshold specific to each individual probe. The difficulty here is in determining the probe-specific score distributions for every probe: this requires knowing the distribution of non-match scores for the probe in question. [Grother] indicates how the non-match score distribution can be determined from a sample of a few hundred scores, rather than requiring the entire gallery. Both quality-dependent matcher thresholds and cohort normalization have been shown to be effective, but are not appropriate for all uses. The proportional increase in the number of matcher comparisons required for cohort normalization is likely to be reasonable for large-scale identification systems, but is likely to be unacceptable for verification systems: a few hundred more comparisons is not a substantial cost when added to hundreds of thousands or millions of comparisons in an identification system, but clearly would have a severe impact on performance if added to a single-comparison verification system. 8.2 Specialized processing for poor-quality samples Another approach for handling poor quality data is to develop specialized processing. Specialized processing for poor-quality data can involve using different encoding algorithms, different matchers (or combinations of matchers), or an increased role for human validation. Whether human validation is practical depends on the biometric modality. Human validation can be effective in the case of face and fingerprint: face results 1 February /77

66 can be verified (to some extent) by untrained people, but with limited degree of accuracy; an infrastructure of highly trained individuals exists to validate fingerprint comparisons at a higher degree of accuracy; no such infrastructure of experts exists for iris comparisons or most other biometrics. However, human validation is problematic: human validation is much slower than any automated system, and determining the accuracy of human validation is difficult. Nevertheless, in problem cases or contested identifications, human beings are likely to have to resolve the problem. 8.3 Multi-biometric capture and fusion If it is not practical to prevent poor-quality samples from being used, probably the most effective way of dealing with them is by collecting more data, lessening any dependence on individual problem samples. This is often described as multibiometric fusion: the combining of multiple sources of data to improve the accuracy, efficiency, and/or robustness of a system. The fusion of multiple sources of data can mean using multiple images of each iris (multi-sample fusion), fingerprints from all ten fingers (multi-instance fusion), or a combination of fingerprint, face, and iris (multi-modal fusion). Fusion increases system robustness and fault tolerance by overcoming the deficiencies inherent in using single sources of data. Many failures to enroll, failures to match, and false matches are due to characteristics of a specific sample: fusion reduces the system s sensitivity to sample-specific noise. Fusion can be used to lessen sensitivity to poor-quality data, erroneous input, or fraud; it can also help enable the use of systems by all subjects. It also should be noted that in cases of indeterminate or contested identifications (which are more frequent when dealing with poor-quality data), a mechanism needs to exist for independent verification and validation, which can be satisfied in part through the use of multi-biometric fusion. Multiple methods of determining identity are invaluable by making it possible to verify results, whether used in all cases or just in problematic cases. Of these methods, the most frequently used in operational systems is multiinstance fusion: one of the reasons that most large-scale fingerprint systems are based on ten fingerprints is to limit the impact of a single poor-quality fingerprint. Some biometric systems have successfully used multiple samples in the gallery to attempt to overcome poor data quality in any one sample. To limit the actions of subjects who may deliberately try to take advantage of poor-quality data (as discussed in Section 5.4), systems can use a game 1 February /77

67 theoretical approach by capturing multiple biometrics and using conditional matching or introducing randomness into biometric matching. These approaches could be fully automated. The subversive subject would not know which biometric is used, making subversion harder and increasing risks of being caught. A combination of factors such as captured biometric data quality, chance of successful match, and workload could be used. The operational cost function (subject queuing, system and operator workloads, etc) will need to be evaluated when considering approaches. 9 Conclusions This paper describes a framework for characterizing biometric data quality in terms of its causes, implications, measurement, prevention, and remedies. Due to the wide variety of biometric systems, and the variation of data quality issues and implications among them, it is impractical to delineate all possible examples of problems that may be encountered. This paper strives to define a framework that can be used in the description and analysis of such problems across most biometric systems and applications. The authors hope that this will be a useful tool that can be used by designers, implementers, operators, and stakeholders of biometric systems to assist in understanding and anticipating data quality problems, so that they can either be prevented or minimized. 1 February /77

68 References [ANSI/NIST] [BQW] [CAIR] [Campbell] [Cert] [Chalmers] [Chen1] [Chen2] [Crawford] [D Amato] Data Format for the Interchange of Fingerprint, Facial, and Scar Mark & Tattoo Information, ANSI/NIST-ITL ; NIST Special Publication , (ftp://sequoyah.nist.gov/pub/nist_internal_reports/sp a16.pdf) NIST Biometric Quality Workshop; ( Religious Accommodation in Driver s License Photographs: A review of codes, policies and practices in the 50 states. ( Campbell; Speaker Recognition: A Tutorial ; Proceedings of the IEEE, Vol. 85, No. 9, September ( Products Certified For Compliance with the FBI's Integrated Automated Fingerprint Identification System Image Quality Specifications, A. Chalmers et al, Image Quality Metrics, Course #44, SIGGRAPH 2000, New Orleans, July Chen, Dass, Jain; Fingerprint Quality Indices for Predicting Authentication Performance ; ( ) Chen, Dass, Jain; Localized Iris Image Quality Using 2-D Wavelets ; To appear in ICB06. ( C. Crawford, E. Mjolsness; Automated Fingerprint Identification: An Independent Study ; D. D Amato; Imaging Systems: the Range of Factors Affecting Image Quality ; Guides to Quality in Visual Resource Imaging; Digital Libraries Group, 1 February /77

69 [Dasu1] [Dasu2] [DaugmanA] T. Dasu, T. Johnson, Exploratory Data Mining and Data Cleaning; Wiley; ISBN: ; T. Dasu, T. Johnson, Problems, Solutions & Research in Data Quality, AT&T Labs Research, 12 April John Daugman; Sample iris images; [DaugmanB] exchanges with John Daugman, Nov [Daugman04] [Daugman05] [Doddington] [Dorairaj] [DPB] [EFTS] [FpVTE] John Daugman; How Iris Recognition Works, IEEE Transactions On Circuits And Systems For Video Technology, Vol. 14, No. 1, January ( J. Daugman, Results from 200 Billion Iris Cross-comparisons, University of Cambridge Computer Laboratory, June ( G. Doddington, W. Liggett, A. Martin, M. Przybocki, D. Reynolds; Sheep, Goats, Lambs And Wolves: A Statistical Analysis of Speaker Performance in the NIST 1998 Speaker Recognition Evaluation; ( V. Dorairaj, Exploring and Developing Processing Techniques for Iris Recognition System. ( Developing%20Processing%20Techniques.pdf) Deutches Bundespolizei; Frequently asked questions (FAQ) about biometrics; ( ercontrols/faq.html) Criminal Justice Information Services (CJIS) Electronic Fingerprint Transmission Specification, ( Wilson, Hicklin, Korves, Ulery, Zoepfl, Bone, Grother, Micheals, Otto, Watson; Fingerprint Vendor Technology Evaluation 2003, NIST Interagency Report 7123; June ( 1 February /77

70 [GAO] [Geller] [Goats] [Grother] [IDENT] [IDX] [IQM] [IQS] Border Security: Challenges in Implementing Border Technology, Statement of Nancy Kingsbury, Managing Director Applied Research and Methods, Testimony Before the Subcommittee on Terrorism, Technology, and Homeland Security and Subcommittee on Border Security, Immigration, and Citizenship, Committee on the Judiciary, United States Senate; March 12, ( B. Geller, et al; A chronological review of fingerprint forgery ; Journal of Forensic Sciences 1999; 44(5):p ( A. Hicklin, C. Watson, B. Ulery; "The Myth of Goats: How many people have fingerprints that are hard to match?"; NIST Interagency Report 7271, P. Grother, Face Recognition Vendor Test Supplemental Report ; NIST Interagency Report 7083, 02 February C. Wilson, et al; Matching Performance for the US-VISIT IDENT System Using Flat Fingerprints, NIST Interagency Report 7110, May (ftp://sequoyah.nist.gov/pub/nist_internal_reports/ir_7110.pdf) Identix Product Website, MITRE; Image Power Spectrum Method Image Quality Measure (IQM) ; Description: ( Software: ( D. D Amato, Hicklin, Khanna, Kiebuzinski, Nadel, Splain; IDENT- IAFIS Image Quality Study Final Report Department of Justice, Justice Management Division, 7 Dec (Never publicly released, but used as a basis for [NIST-IQS]) [ITIRT] Independent Testing of Iris Recognition Technology, Final Report; May ( htm) [Kalka] Nathan D. Kalka; Image Quality Assessment for Iris Biometric, Master s Thesis, West Virginia University; February /77

71 ( Briefing: nday%20september%2019/poster%20session/kalka_bcc05quality_sub mit.pdf) [Khanna] [Kiebuzinski] R. Khanna, N. Ratha (ed.); Systems Engineering for Large-Scale Fingerprint Systems ; Automatic Fingerprint Recognition Systems, Springer Verlag, New York, G. Kiebuzinski, et al, Improving Fingerprint Submission Quality to Improve Reliability, Accelerate Throughput, and Reduce Cost, Draft Report, November (never publicly released) [Lukasik] C. Lukasik, The Physiognomy of Biometrics, Part II, 1 Oct ( [M1/ ] Biometric Sample Quality Standard Draft (Revision 4), M1/ , 6 May ( [M1/ ] [Matsumoto] WG 3 Quality Reporteur Group (WRG) Report #1, M1/ , September 14, (Identical to ISO/IEC JTC 1/SC 37 N 1128). ( T. Matsumoto, et al; Impact of Artificial Gummy Fingers on Fingerprint Systems ; Proceedings of SPIE Vol. #4677, Optical Security and Counterfeit Deterrence Techniques IV, January 2002 ( or ) [MINEX04] Minutiae Exchange Test 2004, [NFIQ] [NFIS] [Nill05] Tabassi, E. et al., Fingerprint Image Quality NIST Interagency Report 7151, August (ftp://sequoyah.nist.gov/pub/nist_internal_reports/ir_7151/ir_7151.pdf) C. Watson, et al; NIST Fingerprint Image Software. ( N. Nill, Test Procedures For Verifying IAFIS Image Quality Requirements For Fingerprint Scanners And Printers, MTR 1 February /77

72 05B , MITRE, April ( [NIST-IQS] [NYST] [SDK] [Schalkoff] [Shen] Hicklin, Reedy; Implications of the IDENT/IAFIS Image Quality Study for Visa Fingerprint Processing, October ( Description of Nystagmus, Aniridia Network. C. Watson, et al; NIST Fingerprint SDK (Software Development Kit) Testing, NIST Interagency Report June 2004, ( R. J. Schalkoff, Digital Image Processing and Computer Vision, John Wiley & Sons, New York, Shen, Surette, and Khanna; Evaluation of Automated Biometrics- Based Identification and Verification Systems ; Proceedings of the IEEE Special Issue on Automated Biometric Systems, New Jersey, September [SlapSeg] Ulery, Hicklin, Watson, Kwong; Slap Segmentation Evaluation 2004, NIST Interagency Report March ( [TDQM] MIT's Total Data Quality Management Program, [van der Putte] T. van der Putte and J. Keuning; Biometrical Fingerprint Recognition: Don't Get Your Fingers Burned ; Proceedings of IFIP TC8/WG8.8 Fourth Working Conference on Smart Card Research and Advanced Applications, pages , Kluwer Academic Publishers, ( ion.pdf) [Vigh] [VTB] M. Vigh, Evidence Bungled in Slaying, The Salt Lake Tribune, 19 Feb M. Garris, et al; Studies of Fingerprint Matching Using the NIST Verification Test Bed (VTB), NIST Interagency Report (ftp://sequoyah.nist.gov/pub/nist_internal_reports/ir_7020.pdf) 1 February /77

73 [Wayman] [Wertheim] [Wein04] [Wein05] J. L. Wayman, Fundamentals of Biometric Authentication Technologies, from National Biometric Test Center, Collected Works , San Jose State University, ( P. Wertheim; Latent Fingerprint Fabrication ; ( House Select Homeland Security Subcommittee on Infrastructure and Border Security and Subcommittee on Intelligence and Counterterrorism Hold Joint Hearing on Disrupting Terrorist Travel ; FDCH TRANSCRIPTS, Congressional Hearings, Sept. 30, 2004 ( in.htm) Wein and Bavega; Using Fingerprint Image Quality to Improve the Identification Performance of the US-VISIT Program ; Proceedings of the National Academy of Sciences; May 24, 2005, vol. 102, no. 21. ( 1 February /77

74 Glossary Term AFIS Definition Automated Fingerprint Identification Systems Aging The elapsed time between collections of biometric samples. Binning errors Consolidation DHS Wrongful classifications of biometric data that cause false nonmatch errors. Binning schemes (also known as data categorization or partitioning) are used to reduce system workload. These schemes reduce workload by categorizing search and gallery samples and limiting searches to specific categories of gallery samples. Binning errors for either search or gallery samples cause false reject errors. The process of detecting and removing unconsolidated records from a database. U.S. Department of Homeland Security EFTS False Accept Rate (FAR) False Match Rate (FMR) The FBI's Electronic Fingerprint Transmission Specification, which is the required standard for transmission of fingerprints to the FBI and many other agencies. Appendix F of this document is the Image Quality Standard for fingerprint scanners. Proportion of biometric system match results that are not true matches. Also known as a False Positive. Equivalent to FMR, but generally used for overall systems rather than individual matcher stages. Equivalent to FAR, but generally used for individual matcher stages rather than overall systems. 1 February /77

75 Term False Non-match Rate (FNMR) False Reject Rate (FRR) Failure to Acquire (FTA) Definition Proportion of match results that the system indicates there is no matching biometric sample in the gallery when there is one or more. Equivalent to FRR, but generally used for individual matcher stages rather than overall systems. False Reject Rate. Equivalent to FNMR, but generally used for overall systems rather than individual matcher stages. Inability of a device to capture the biometric data. Closely related to FTE. Failure to Enroll (FTE) Gallery Ground Truth Groundtruthing Human factors (or Ergonomics) IAFIS IDENT Identification Proportion of samples that are of such poor quality that they cannot be processed by a feature extractor, or for which a matcher cannot find any similarity between the sample and its mate. Set of retained biometric samples. May be described as a database. In this context, ground truth refers to biometric data that has been verified by experts (possibly using an automated tool) is being error free. An example is fingerprint minutia that have been identified and adjudicated by multiple fingerprint experts. A second example is a database where all metadata errors have been detected and classified or removed. The process of data analysis conducted to identify and quantify metadata errors. The scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the profession that applies theory, principles, data and methods to design in order to optimize human well-being and overall system performance. (definition adopted by the International Ergonomics Association in 2000) ( The FBI s Integrated Automated Fingerprint Identification System The DHS (formerly INS) Automated Biometric Identification System; the biometric matching system used by US-VISIT. Process of probing a gallery, in a many-to-one match, with the implication that an unknown identity is being determined. 1 February /77

76 Term Definition Instance An individual physical feature captured in a biometric system. For fingerprints, each of the ten fingers is a different instance; for irises, the right and left eyes are different instances. IQM Image quality metric Metadata (Greek: meta- + Latin: data "information"), literally data about data, is information that describes another set of data. ( Misidentified records A metadata error in which records from different people are (misidentifications) listed under the same name or ID Modality Different types of biometrics. For example, fingerprints and iris are two biometric modalities. NCIC 2000 National Crime Information Center 2000 Negative ID systems A biometric system in which it is to the subject s disadvantage to be recognized. NIST National Institute of Standards and Technology Positive ID systems Probe ROC A biometric system in which it is to the subject s benefit to be recognized. The biometric sample to be identified or verified by matching against the gallery. Receiver Operating Characteristic. In biometrics, a chart that plots False Accept Rate against True Accept Rate Sample Six Sigma Subject Total Quality Management (TQM) A biometric sample is a representation of a physical human feature. For biometrics such as face, fingerprint, or iris, sample and image are equivalent terms. Six Sigma is a quality management program that measures and improves the operational performance of a company by identifying and correcting defects in the company's processes and products. ( Person whose biometric sample is collected. A management strategy aimed at embedding awareness of quality in all organizational processes. ( 1 February /77

77 Term True Accept Rate (TAR) True Match Rate (TMR) Definition True Accept Rate (True Positive): Percent of biometric system match results that are true matches. Note that TAR = 1 FRR. Equivalent to TMR, but generally used for overall systems rather than individual matcher stages Equivalent to TAR, but generally used for individual matcher stages rather than overall systems. Unconsolidated records US-VISIT Verification WSQ A database error in which the same person has records under different names or IDs United States Visitor and Immigrant Status Indicator Technology (DHS) Process of comparing two samples to determine whether they match, with the implication that a claimed identity is being verified. Wavelet Scalar Quantization. A standard image compression method used for fingerprint images. 1 February /77