Abstract
Face recognition, as a process of the human visual system, analyses facial properties and contextual information such as body shape. Automated recognition replicates the human process and analyses a face image, which is typically acquired with a visible spectrum sensor. When dealing with automated operational systems, the quality of the captured face image is relevant as it affects the recognition accuracy. Thus, it is necessary to measure the utility of a face sample with both a quality score and complementary measures that can provide actionable feedback. This Perspective addresses challenges and discusses solutions for the optimization of biometric recognition systems specifically related to face image analysis. One of these challenges is the vulnerability to presentation attacks. Consequently, for reliable recognition in non-supervised environments, robust presentation attack detection is required. Moreover, biometric templates must be protected. Finally, acceptability of biometric systems requires fairness of the biometric algorithms and artificial neural networks used.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to the full article PDF.
USD 39.95
Prices may be subject to local taxes which are calculated during checkout




Similar content being viewed by others
References
International Organization for Standardization. ISO/IEC 2382-37:2022. Information Technology — Vocabulary — Part 37: Biometrics (ISO, 2022).
Drozdowski, P., Rathgeb, C. & Busch, C. Computational workload in biometric identification systems: an overview. IET Biom. 8, 351–368 (2019).
International Organization for Standardization. ISO/IEC 19795-1:2021. Information Technology — Biometric Performance Testing and Reporting — Part 1: Principles and Framework (ISO, 2021).
Meints, M. et al. Biometric systems and data protection legislation in Germany. In Proc. 2008 International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIHMSP), 1088–1093 (IEEE, 2008).
Funk, W., Arnold, M., Busch, C. & Munde, A. Evaluation of image compression algorithms for fingerprint and face recognition. In Proc. IEEE Information Assurance Workshop (IEEE, 2005).
International Organization for Standardization. ISO/IEC 39794-5:2019 Information Technology — Extensible Biometric Data Interchange Formats — Part 5: Face image data (ISO, 2019).
International Organization for Standardization. ISO/IEC FDIS 29794-5 Information Technology Biometric Sample Quality. Part 5: Face image data (ISO, 2024).
Tabassi, E. & Wilson, C. A novel approach to fingerprint image quality. In 2005 International Conference on Image Processing (ICIP 2005), 37–40 (IEEE, 2005).
Olsen, M., Sˇmida, V. & Busch, C. Finger image quality assessment features — definitions and evaluation. IET Biom. 5, 47–64 (2016).
Tabassi, E. et al. NIST Interagency Report 8382 (National Institute of Standards and Technology, 2021).
European Council. Regulation 2017/2226 of the European Parliament and of the Council of 30 November 2017 on establishing an Entry/Exit System (EES) to register entry and exit data and refusal of entry data of third-country nationals (European Council, 2017).
European Council. Commission Implementing Decision 2019/329 of 25 February 2019 laying down the specifications for the quality, resolution and use of fingerprints and facial image for biometric verification and identification in the Entry/Exit System (EES) (European Council, 2019).
Schlett, T. et al. Face image quality assessment: a literature survey. ACM Computing Surveys (CSUR). 54, (2022).
International Civil Aviation Organization. NTWG: Machine Readable Travel Documents Part 3 — Specifications for Electronically Enabled MRtds with Biometric Identification Capability (ICAO, 2021).
International Organization for Standardization. ISO/IEC 19794-5:2011 Information Technology — Biometric Data Interchange Formats — Part 5: Face Image Data (ISO, 2011).
ICAO. Machine Readable Travel Documents. https://www.icao.int/publications/documents/9303_p9_cons_en.pdf (2021).
International Organization for Standardization. ISO/IEC 29794-1 Information Technology — Biometric Sample Quality — Part 1: Framework (ISO, 2024).
Meng, Q., Zhao, S., Huang, Z. & Zhou, F. MagFace: a universal representation for face recognition and quality assessment. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2021).
Boutros, F., Fang, M., Klemt, M., Fu, B. & Damer, N. CR-FIQA: face image quality assessment by learning sample relative classifiability. In Conference on Computer Vision and Pattern Recognition (CVPR), 5836–5845 (IEEE, 2023).
Deng, J., Guo, J. & Zafeiriou, S. ArcFace: additive angular margin loss for deep face recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2019).
Huang, G. B., Ramesh, M., Berg, T. & Learned-Miller, E. Labeled faces in the wild: a database for studying face recognition in unconstrained environments. https://api.semanticscholar.org/CorpusID:88166 (2008).
Schlett, T., Rathgeb, C., Tapia, J. & Busch, C. Considerations on the evaluation of biometric quality assessment algorithms. IEEE Trans Biometrics Behav. Identity Sci. https://doi.org/10.1109/TBIOM.2023.3336513 (2023).
Chandaliya, P., Raja, K., Raghavendra, R. & Busch, C. Unified face image quality score based on ISO/IEC quality components. In Proc. International Conference of the Biometrics Special Interest Group (BIOSIG), 1–11 (2023).
Grimmer, M., Rathgeb, C., Veldhuis, R. & Busch, C. Neutrex: a 3D quality component measure on facial expression neutrality. In Proc. International Joint Conference on Biometrics (IJCB), 1–8 (IEEE, 2023).
Grimmer, M., Veldhuis, R. & Busch, C. Efficient expression neutrality estimation with application to face recognition utility prediction. In Proc. International Workshop on Biometrics and Forensics 1–8 (IWBF, 2024).
Funk, W., Arnold, M., Busch, C. & Munde, A. Evaluation of image compression algorithms for fingerprint and face recognition systems. In Proc. IEEE SMC Information Assurance Workshop 72–78 (2005).
Schlett, T., Schachner, S., Rathgeb, C., Tapia, J. & Busch, C. Effect of lossy compression algorithms on face image quality and recognition. In Intl Conference on Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, 2023).
Raghavendra, R. & Busch, C. Presentation attack detection methods for face recognition systems: a comprehensive survey. ACM Comput. Surv. 50, 1–37 (2017).
Zwiesele, A., Munde, A., Busch, C. & Daum, H. BioIS study — comparative study of biometric identification systems. In 34th Annual 2000 IEEE International Carnahan Conference on Security Technology (CCST) (IEEE, 2000).
Matsumoto, T. et al. Impact of artificial ‘gummy’ fingers on fingerprint systems. SPIE Conf. Opt. Security Counterfeit Deterrence Tech. IV 4677, 275–289 (2002).
Schuckers, S. et al. Issues for liveness detection in biometrics. In Proc. Biometric Consortium Conference. 6911 (NISTIR, 2002).
International Organization for Standardization. ISO/IEC 30107-1. Information Technology — Biometric Presentation Attack Detection — Part 1: Framework (ISO, 2023).
International Organization for Standardization. ISO/IEC 30107-2. Information Technology — Biometric Presentation Attack Detection — Part 2: Data Formats (ISO, 2017).
International Organization for Standardization. ISO/IEC 30107-3. Information Technology — Biometric Presentation Attack Detection — Part 3: Testing and Reporting (ISO, 2023).
International Organization for Standardization. ISO/IEC SC37 SD11. General Biometric System (ISO, 2008).
Rathgeb, C., Drozdowski, P. & Busch, C. Detection of makeup presentation attacks based on deep face representations. In Proc. Intl Conference on Pattern Recognition (ICPR), 3443–3450 (2020).
Rathgeb, C., Drozdowski, P. & Busch, C. Makeup presentation attacks: review and detection performance benchmark. IEEE Access. 8, 224958–224973 (2020).
Rathgeb, C., Tolosana, R., Vera, R. & Busch, C. (eds) Handbook of Digital Face Manipulation and Detection: from DeepFakes to Morphing Attacks. 1st edn (Springer, 2022).
Rathgeb, C., Dantcheva, A. & Busch, C. Impact and detection of facial beautification in face recognition: an overview. IEEE Access. 7, 152667–152678 (2019).
Khodabakhsh, A., Raghavendra, R., Raja, K., Wasnik, P. & Busch, C. Fake face detection methods: can they be generalized? In 2018 International Conference of the Biometrics Special Interest Group. 1–6 (BIOSIG, 2018).
Scherhag, U., Rathgeb, C., Merkle, J., Breithaupt, R. & Busch, C. Face recognition systems under morphing attacks: a survey. IEEE Access 7, 23012–23026 (2019).
Ibsen, M. et al. Conditional face image manipulation detection: combining algorithm and human examiner decisions. In Proc. 12th Workshop on Information Hiding and Multimedia Security (ACM IH & MMSEC, 2024).
Davis, J. et al. The super-recogniser advantage extends to the detection of digitally manipulated faces. Preprint at https://osf.io/preprints/psyarxiv/ye7ph (2024).
Ferrara, M., Franco, A., Maltoni, D. & Busch, C. Morphing attack potential. In 10th International Workshop on Biometrics and Forensics (IWBF, 2022).
FRONTEX Report. International Conference on Biometrics for Borders Morphing and Morphing Attack Detection Methods 2020, https://www.frontex.europa.eu/assets/Publications/Research/International_Conference_on_Biometrics_for_Borders.pdf (2024).
Godage, S. et al. Analyzing human observer ability in morphing attack detection — where do we stand? IEEE Trans. Technol. Soc. 4, 125–145 (2023).
Nichols, R., Rathgeb, C., Drozdowski, P. & Busch, C. Psychophysical evaluation of human performance in detecting digital face image manipulations. IEEE Access. 10, 31359–31376 (2022).
Raja, K. et al. Morphing attack detection — database, evaluation platform and benchmarking. IEEE Trans Inf. Forensics Secur. 16, 4336–4351 (2020).
Zhang, H. et al. MIPGAN — generating strong and high quality morphing attacks using identity prior driven GAN. IEEE Trans. Biometr. Behav. Identity Sci. 3, 365–383 (2021).
Ngan, M., Grother, P., Hanaoka, K. & Kuo, J. Face Analysis Technology Evaluation (Fate) Part 4: Morph — Performance of Automated Face Morph Detection. NIST Interagency Report 8292 (National Institute of Standards and Technology, 2023).
Raghavendra, R., Raja, K. & Busch, C. Detecting morphed face images. In 2016 IEEE 8th International Conference on Biometrics: Theory, Applications and Systems (BTAS) (IEEE, 2016).
Scherhag, U., Kunze, J., Rathgeb, C. & Busch, C. Face morph detection for unknown morphing algorithms and image sources: a multi-scale block local binary pattern fusion approach. IET Biom. 9, 278–289 (2020).
Scherhag, U., Budhrani, D., Gomez-Barrero, M. & Busch, C. Detecting morphed face images using facial landmarks. In Intl Conference on Image and Signal Processing (ICISP) (Springer, 2018).
Debiasi, L., Scherhag, U., Rathgeb, C., Uhl, A. & Busch, C. PRNU-based detection of morphed face images. In 6th International Workshop on Biometrics and Forensics, 1–6 (2018).
Scherhag, U., Debiasi, L., Rathgeb, C., Busch, C. & Uhl, A. Detection of face morphing attacks based on PRNU analysis. IEEE Trans. Biometr. Behav. Identity Sci. 1, 302–317 (2019).
Raghavendra, R. et al. Transferable deep-CNN features for detecting digital and print-scanned morphed face images. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1822–1830 (IEEE, 2017).
Scherhag, U., Rathgeb, C., Merkle, J. & Busch, C. Deep face representations for differential morphing attack detection. IEEE Trans. Inf. Forensics Secur. 15, 3625–3639 (2020).
Ngan, M., Grother, M. & Hanaoka, K. Face Recognition Vendor Test — Part 4A. NIST Interagency Report 8430 (National Institute of Standards and Technology, 2022).
Scherhag, U. et al. Biometric systems under morphing attacks: assessment of morphing techniques and vulnerability reporting. In Intl Conference of the Biometric Special Interest Group BIOSIG 2017, 1–7 (IEEE, 2017).
International Organization for Standardization. ISO/IEC DIS 20059. Information Technology — Methodologies to Evaluate the Resistance of Biometric Recognition Systems to Morphing Attacks (ISO, 2024).
Joshi, I. et al. Synthetic data in human analysis: a survey. IEEE Trans. Pattern Anal. Machine Intell. 46, 4957–4976 (2024).
Goodfellow, I. J. et al. Generative adversarial nets. In Proc. 27th International Conference on Neural Information Processing Systems, Vol. 2, 2672–2680 (MIT Press, 2014).
Karras, T. et al. Alias-free generative adversarial networks. In Proc. 35th Intl Conf. Neural Inform. Processing Syst. (NIPS ‘21) 66, 852–863 (Curran Assoc., 2024).
Dhariwal, P. & Nichol, A. Diffusion models beat gans on image synthesis. Adv. Neur. Inf. Proc. Syst. 34, 8780–8794 (2021).
Melzi, P. et al. Gandiffface: controllable generation of synthetic datasets for face recognition with realistic variations. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 3078–3087 (IEEE Computer Society, 2023).
O’Sullivan, D. A high school student created a fake 2020 candidate. Twitter verified it. CNN https://edition.cnn.com/2020/02/28/tech/fake-twitter-candidate-2020/index (28 February 2020).
Oltermann, P. European politicians duped into deepfake video calls with mayor of Kyiv. Guardian https://www.theguardian.com/world/2022/jun/25/european-leaders-deepfake-video-calls-mayor-of-kyiv-vitali-klitschko (25 June 2022).
Ibsen, M., Rathgeb, C., Marcel, S. & Busch, C. Multi-channel cross modal detection of synthetic face images. In Proc. International Workshop on Biometrics and Forensics (IWBF), 1–8 (2024).
Breebart, J., Busch, C., Grave, J. & Kindt, E. A reference architecture for biometric template protection based on pseudo identities. In BIOSIG 2008: Biometrics and Electronic Signatures, 25–37 (2008).
International Organization for Standardization. ISO/IEC JTC1 SC27 Security Techniques: ISO/IEC 24745:2022. Information Technology — Security Techniques — Biometric Information Protection (ISO, 2022).
Rathgeb, C., Breitinger, F. & Busch, C. Alignment-free cancelable iris biometric templates based on adaptive Bloom filters. In 2013 International Conference on Biometrics (ICB), 1–8 (2013).
Gomez-Barrero, M., Rathgeb, C., Galbally, J., Fierrez, J. & Busch, C. Protected facial biometric templates based on local Gabor patterns and adaptive Bloom filters. In 2014 22nd International Conference on Pattern Recognition (ICPR), 4483–4488 (2014).
Kolberg, J., Drozdowski, P., Gomez-Barrero, M., Rathgeb, C. & Busch, C. Efficiency analysis of post-quantum-secure face template protection schemes based on homomorphic encryption. In International Conference of the Biometrics Special Interest Group (BIOSIG) (2020).
Guo, E. & Noori, H. This is the real story of the Afghan biometric databases abandoned to the Taliban. MIT Technol. Rev. https://www.technologyreview.com/2021/08/30/1033941/afghanistan-biometric-databases-us-military-40-data-points/ (30 August 2021).
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N. & Busch, C. Demographic bias in biometrics: a survey on an emerging challenge. Trans. Technol. Soc. 1, 89–103 (2020).
Cavazos, J. G. Phillips, P. J., Castillo, C. D. & O’Toole, A. J. Accuracy comparison across face recognition algorithms: where are we on measuring race bias? IEEE Trans. Biometr. Behav. Identity Sci. 3, 101–111 (2021).
Bacchini, F. & Lorusso, L. Race, again: how face recognition technology reinforces racial discrimination. J. Inf. Commun. Ethics Soc. 17, 321–335 (2019).
International Organization for Standardization. ISO/IEC FDIS 19795-10. Information Technology — Biometric Performance Testing and Reporting — Part 10: Quantifying Biometric System Performance Variation Across Demographic Groups (ISO, 2024).
Grother, P., Ngan, M. & Hanaoka, K. NIST Interagency Report 8280. Face Recognition Vendor Test — Part 3 (National Institute of Standards and Technology, 2019).
Fitzpatrick, T. The validity and practicality of sun-reactive skin types I through VI. Arch. Dermatol. 124, 869–871 (1988).
Howard, J., Sirotin, Y., Tipton, J. & Vemury, A. Reliability and validity of image-based and self-reported skin phenotype metrics. IEEE Trans. Biometr. Behav. Identity Sci. 3, 550–560 (2021).
Drozdowski, P., Rathgeb, C. & Busch, C. The watchlist imbalance effect in biometric face identification: comparing theoretical estimates and empiric measurements. In International Conference on Computer Vision Workshops (ICCVW), 1–9 (IEEE, 2021).
Howard, J. et al. Evaluating proposed fairness models for face recognition algorithms. In Proc. International Conference on Pattern Recognition (IEEE, 2022).
Rathgeb, C., Drozdowski, P., Frings, D. C., Damer, N. & Busch, C. Demographic fairness in biometric systems: what do the experts say? IEEE Technol. Soc. Mag. 41, 71–82 (2022).
Kotwal, K. & Marcel, S. Fairness index measures to evaluate bias in biometric recognition. In Proc. International Conference on Pattern Recognition (IEEE, 2022).
Terhörst, P., Kolf, J., Damer, N., Kirchbuchner, F. & Kuijper, A. Post-comparison mitigation of demographic bias in face recognition using fair score normalization. Pattern Recognit. Lett. 140, 332–338 (2020).
Kolberg, J., Schäfer, Y., Rathgeb, C. & Busch, C. On the potential of algorithm fusion for demographic bias mitigation in face recognition. IET Biom. https://doi.org/10.1049/2024/1808587 (2023).
Terhörst, P., Kolf, J., Damer, N., Kirchbuchner, F. & Kuijper, A. Face quality estimation and its correlation to demographic and non-demographic bias in face recognition. In 2020 IEEE International Joint Conference on Biometrics (IJCB), https://doi.org/10.1109/IJCB48548.2020.9304865 (IEEE, 2020).
Babnik, Z. & Struc, V. Assessing bias in face image quality assessment. In 30th European Signal Processing Conference (EUSIPCO), https://doi.org/10.23919/EUSIPCO55093.2022.9909867 (IEEE, 2020).
Acknowledgements
This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts, within their joint support of the National Research Center for Applied Cybersecurity ATHENE.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The author declares no competing interests.
Peer review
Peer review information
Nature Reviews Electrical Engineering thanks M. Tanveer, Aythami Morales and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Related links
BOEP platform: https://biolab.csr.unibo.it/fvcongoing/UI/Form/BOEP.aspx
Coded Bias: https://en.wikipedia.org/wiki/Coded_Bias
Open Source Face Image Quality framework: https://bsi.bund.de/dok/OFIQ-e
Glossary
- Artefacts
-
Artificial objects or representation presenting a copy of biometric characteristics or synthetic biometric patterns.
- Biometric applications
-
Applications that include the policies that govern the operation of the biometric system.
- Biometric attendant
-
Agent of the biometric system operator who directly interacts with the biometric capture subject.
- Biometric capture
-
Obtaining and recording of, in a retrievable form, signal(s) of biometric characteristic(s) directly from individual(s), or from representation(s) of biometric characteristic(s).
- Biometric capture device
-
Device that collects a signal from a biometric characteristic and converts it to a captured biometric sample.
- Biometric capture process
-
Series of actions undertaken to effect a biometric capture.
- Biometric capture subjects
-
Individuals who are the subject of a biometric capture process.
- Biometric characteristics
-
Biological and/or behavioural characteristics of an individual from which distinguishable, repeatable biometric features can be extracted for the purpose of biometric recognition.
- Biometric concealer
-
Subversive biometric capture subject who performs a biometric concealment attack.
- Biometric features
-
Numbers or labels extracted from biometric samples and used for comparison.
- Biometric impostor
-
Subversive biometric capture subject who performs a biometric impostor attack.
- Biometric probe
-
Biometric sample or biometric feature set input to an algorithm for comparison to a biometric reference(s).
- Biometric recognition
-
Automated recognition of individuals based on their biological and behavioural characteristics.
- Biometric reference
-
One or more stored biometric samples, biometric templates or biometric models attributed to a biometric data subject and used as the object of biometric comparison.
- Biometric samples
-
Analog or digital representations of biometric characteristics prior to biometric feature extraction.
- Biometric utility
-
Degree to which a biometric sample supports biometric recognition performance.
- Bona fide presentation
-
Biometric presentation without the goal of interfering with the operation of the biometric system.
- Canonical face image
-
Face image conformant to an external standard or specification of a reference face image.
- Comparison score
-
Numerical value (or set of values) resulting from a comparison.
- Error-versus-discard-characteristic curve
-
Method to evaluate the efficacy of quality assessment algorithms by quantifying how efficiently discarding samples with low quality scores results in improved (reduced) false non-match rate.
- False match rate
-
(FMR). Proportion of the completed biometric non-mated comparison trials that result in a false match.
- False non-match rate
-
(FNMR). Proportion of the completed biometric mated comparison trials that result in a false non-match.
- Morph image
-
Face image representing two or more parent facial images.
- Morphing attack
-
Biometric image manipulation attack through merging two or more facial images by means of morphing.
- Morphing attack detection
-
(MAD). Detecting traces of a face image morphing attack conducted by some algorithms and/or human examiner.
- Presentation attack
-
Presentation to the biometric capture subsystem with the goal of interfering with the operation of the biometric system.
- Presentation attack detection
-
(PAD). Automated discrimination between bona fide presentations and biometric presentation attacks.
- Presentation attack instrument
-
Biometric characteristic or object used in a biometric presentation attack.
- Quality components
-
Measurements on the biometric sample that may contribute to the computation of a unified quality score.
- Quality measures
-
Quality scores or quality components.
- Quality score
-
Quantitative value of the fitness of a biometric sample to accomplish or fulfil the comparison decision.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Busch, C. Challenges for automated face recognition systems. Nat Rev Electr Eng 1, 748–757 (2024). https://doi.org/10.1038/s44287-024-00094-x
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1038/s44287-024-00094-x


