Multimodal Discourse Analysis and Semiotic Practices

Multimodal discourse analysis examines how meaning is constructed through the interplay of diverse semiotic modes such as language, image, gesture, sound and spatial design. Grounded in social semiotics, systemic functional linguistics and cognitive semiotics, the field investigates how communicative artefacts integrate multiple resources to achieve coherence, persuasion or aesthetic effect. Researchers analyse texts ranging from diagrams and advertisements to dance performances and interactive interfaces, revealing how mode combinations shape interpretation, influence attention and support learning. Recent work has emphasised computational annotation, empirical testing of multimodal hypotheses and application to digital platforms, extending classical theories of cohesion and discourse structure into data‐driven and experimental domains. Globally, these methods inform education, advertising, interface design and cultural studies, offering practical guidelines for creating accessible scientific visuals, effective public messaging and inclusive user experiences.

Research from Nature Portfolio

Recent studies have advanced the empirical study of non-verbal semiotic systems by treating classical ballet as a movement-based discourse. By capturing motion data and aligning it with narrative interpretations, researchers have proposed a systematic framework for linking physical gesture sequences to discourse functions. This approach draws on formal semantics and functional multimodality to enable quantitative analysis of choreography as communicative action. The study demonstrates how principles from linguistics can be imported to analyse embodied sign systems, laying groundwork for broader empirical investigations of dance and other performative modes.

Multimodal Discourse Analysis and Semiotic Practices publication trend

The graph below shows the total number of articles in multimodal discourse analysis and semiotic practices across all publications each year (not limited to Nature Index journals).

Technical terms

Multimodal discourse analysis: Investigation of how multiple semiotic modes combine to create meaning in communicative artefacts.

Semiotic resource: Any sign-making element (e.g. visual, verbal, gestural) used to construct and convey meaning.

Rhetorical Structure Theory (RST): A framework for describing discourse relations and hierarchical organisation among text or image segments.

Annotation schema: A structured set of labels and rules applied to data for consistent coding of multimodal features.

Genre: A class of communicative artefacts sharing conventionalised structures, purposes and semiotic strategies.

References

1. Towards semiotically driven empirical studies of ballet as a communicative form. Humanities and Social Sciences Communications (2022).
2. AI2D-RST: a multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation (2020).
3. Multimodal Coherence Revisited: Notes on the Move From Theory to Data in Annotating Print Advertisements. Frontiers in Communication (2022).
4. Corpus-based insights into multimodality and genre in primary school science diagrams. Visual Communication (2023).