Multimodality is an inter-disciplinary approach that understands communication and representation to be more than about language. It has been developed over the past decade to systematically address much-debated questions about changes in society, for instance in relation to new media and technologies.

Multimodal approaches have provided concepts, methods and a framework for the collection and analysis of visual, aural, embodied, and spatial aspects of interaction and environments, and the relationships between these (1,2).

Theoretical assumptions

Three interconnected theoretical assumptions underpin multimodality.

First, multimodality assumes that representation and communication always draw on a multiplicity of modes, all of which contribute to meaning. It focuses on analyzing and describing the full repertoire of meaning-making resources that people use (visual, spoken, gestural, written, three-dimensional, and others, depending on the domain of representation) in different contexts, and on developing means that show how these are organized to make meaning.

Second, multimodality assumes that resources are socially shaped over time to become meaning making resources that articulate the (social, individual/affective) meanings demanded by the requirements of different communities. These organized sets of semiotic resources for making meaning are referred to as modes which realize communicative work in distinct ways – making the choice of mode a central aspect of interaction and meaning. The more a set of resources has been used in the social life of a particular community, the more fully and finely articulated it will have become. In order for something to ‘be a mode’ there needs to be a shared cultural sense within a community of a set of resources and how these can be organized to realize meaning.

Third, people orchestrate meaning through their selection and configuration of modes, foregrounding the significance of the interaction between modes. Thus all communicational acts are shaped by the norms and rules operating at the moment of sign making, and influenced by the motivations and interests of people in a specific social context.

Core concepts

Four core concepts are common across multimodal research: mode, semiotic resource, modal affordance and inter-semiotic relations. Within social semiotics, a mode is understood as an outcome of the cultural shaping of a material through its use in the daily social interaction of people. The semiotic resources of a mode come to display regularities through the ways in which people use them and can be thought of as the connection between representational resources and what people do with them. The term modal affordance refers to the material and the cultural aspects of modes: what it is possible to express and represent easily with a mode. It is a concept connected to both the material as well as the cultural and social historical use of a mode. Modal affordance raises the question of what a mode is ‘best’ for what. This raises the concept of inter-semiotic relationships, and how modes are configured in particular contexts. These four concepts provide the starting point for multimodal analysis.

What is the focus and scope of multimodal research?

Multimodality can be used to build inventories of the semiotic resources, organizing principles, and cultural references that modes make available to people in particular places and times: the actions, materials and artifacts people communicate with. This has included contributions to mapping the semiotic resources of visual communication and colour, gesture and movement, gaze, voice and music, to name a few (1).

Multimodal studies have also been conducted that set out to understand how semiotic resources are used to articulate discourses across a variety of contexts and media for instance school, workplaces, online environments, textbooks and advertisements. The relationships across and between modes in multimodal texts and interaction are a central area of multimodal research.

Multimodal research makes a significant contribution to research methods for the collection and analysis of digital data and environments within social research. It provides novel methods for the collection and analysis of types of visual data, video data and innovative methods of multimodal transcription and digital data management (3).

References

1 Jewitt, C. (ed.) (2009) The Routledge Handbook of Multimodal Analysis, London:Routledge.
2 Kress, G. (2009) Multimodality: a Social Semiotic Approach to Contemporary Communication, London: Routledge.
3 Bezemer, J. and Mavers, D. (2011) Multimodal Transcription as Academic Practice, International Journal of Social Research Methodology Vol. 14, No. 3, May 2011, 191-206.