Open Set Recognition Techniques in Machine Learning

Open set recognition addresses the practical limitation of conventional classification systems, which assume that all possible categories are known at training time. In real-world deployments, models frequently encounter inputs from previously unseen classes or sources. To mitigate misclassification of such instances, open set methods couple standard discrimination with mechanisms for detecting and rejecting unknown samples. Contemporary strategies fall broadly into thresholding techniques that monitor output confidences, feature-space regularisation that shapes latent representations to maximise separability between known and unknown regions, and generative or prototype-based frameworks that explicitly model class boundaries. Recent advances have further introduced uncertainty quantification to express predictive confidence, online adaptation to refine decision boundaries during deployment, and hybrid architectures that combine discriminative backbones with auxiliary rejection modules. These developments enhance reliability in critical applications ranging from medical diagnostics to remote sensing, where the cost of undetected novel inputs may be substantial.

Research from Nature Portfolio

Recent studies have advanced the integration of uncertainty measures into deep models for robust open set performance. One approach employs an uncertainty-inspired framework that computes both class probabilities and a dedicated uncertainty score, enabling high-precision recognition of retinal anomalies while flagging truly novel or low-quality inputs for manual review. This dual output delivers markedly improved F1 scores on both in-domain and unseen categories compared with standard classifiers. Earlier foundational work introduced a prototype-based deep network that jointly learns class prototypes and adaptive radii to guide feature discrimination. This method not only detects unknowns via multi-class triplet thresholding but also incorporates new categories dynamically through distance-based weight initialisation, significantly reducing the sample and time requirements for fine-tuning on novel classes.

Research from all publishers

Beyond Nature outlets, a recent Dirichlet prior network model has been proposed for out-of-distribution detection in remote sensing. By maximising the representational gap between in-domain and out-of-domain examples, the network quantifies distributional uncertainty and achieves reliable separation of unseen classes under varied geographic and sensor conditions. In the field of synthetic aperture radar, a random sampling combination strategy constructs pseudo-unknown targets to transform open-world recognition into a closed-world problem augmented with an explicit unknown class. This enables joint feature learning and unknown detection while maintaining computational efficiency. As a seminal contribution, weightless neural networks have also been adapted for open set tasks through elaborate distance-based computations and rejection thresholds, demonstrating that non-parametric architectures can effectively identify and reject samples from untrained classes across diverse experimental settings.

Open Set Recognition Techniques in Machine Learning publication trend

The graph below shows the total number of articles in open set recognition techniques in machine learning across all publications each year (not limited to Nature Index journals).

Technical terms

Open set recognition: A classification paradigm that requires a model to correctly label inputs from known classes while rejecting samples from unseen classes.

Out-of-distribution detection: The process of identifying inputs whose statistical properties deviate from those observed during training, signalling potential unknown classes.

Uncertainty score: A quantitative measure of a model’s confidence in its prediction, used to determine whether to accept or reject an input.

Prototype learning: A technique that represents each class by one or more reference points in feature space, guiding classification and the detection of anomalous samples.

References

1. Uncertainty-inspired open set learning for retinal anomaly identification. Nature Communications (2023).
2. P-ODN: Prototype-based Open Deep Network for Open Set Recognition. Scientific Reports (2020).
3. An Advanced Dirichlet Prior Network for Out-of-Distribution Detection in Remote Sensing. IEEE Transactions on Geoscience and Remote Sensing (2022).
4. SAR Target Recognition via Random Sampling Combination in Open-World Environments. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2022).
5. Weightless neural networks for open set recognition. Machine Learning (2017).