Scene Graph Generation for Human-Object Interaction Detection

Scene graph generation for human-object interaction detection seeks to convert visual data into a structured graph in which nodes represent humans and objects, and edges denote the nature of their interactions. By detecting action predicates and spatial relations, this approach provides a rich semantic representation of a scene, enabling applications such as assistive robotics, intelligent surveillance, and human-centred image retrieval. Key challenges include disambiguating subtle interactions, managing the long-tailed distribution of predicates, and capturing fine-grained body-part dynamics. Modern frameworks combine convolutional backbones for object localisation with relational reasoning modules—often based on graph neural networks or attention-driven transformers—to infer triadic triplets of ⟨human, predicate, object⟩. Advances in contextual modelling and bias mitigation have driven recent progress, enhancing both accuracy and generalisation across diverse environments.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Scene Graph Generation for Human-Object Interaction Detection publication trend

The graph below shows the total number of articles in scene graph generation for human-object interaction detection across all publications each year (not limited to Nature Index journals).

Technical terms

Scene graph generation: The task of producing a graph-structured representation of an image where nodes are objects or entities and edges encode semantic or spatial relationships.

Human-object interaction detection: A specialised form of visual relation detection focused on identifying actions or predicates linking human agents with objects in images.

Predicate: The relational label or action term that describes how two entities, typically a human and an object, are connected (for example, “holding” or “riding”).

Graph neural network: A deep learning model that operates on graph-structured data by propagating information along edges and aggregating feature representations at nodes.

Long-tailed distribution: A statistical phenomenon in which a small number of predicate classes are very common in training data, while many others appear infrequently, leading to imbalance in learning.

References

1. Scene Graph Generation: A comprehensive survey. Neurocomputing (2024).
2. Detecting human—object interaction with multi-level pairwise feature network. Computational Visual Media (2020).
3. Skew Class-Balanced Re-Weighting for Unbiased Scene Graph Generation. Machine Learning and Knowledge Extraction (2023).