Using Multimodal Factual Texts

This article is drawn from extracts from PETAA Paper 184, Using Multimodal factual texts during the inquiry process, written by Michele Anstey and Geoff Bull

Understandings about semiotic systems for effective use of multimodal factual texts

What do students need to know and understand about the semiotic systems in order to use multimodal factual texts to learn? There are five semiotic systems (Kress and van Leeuwen 2006) that can contribute to meaning making in a multimodal text. These are:

1. Linguistic eg vocabulary, generic structure and grammar of oral and written language
2. Visual eg the colour, vectors and viewpoints of still and moving images
3. Audio eg volume, pitch and rhythm of music and sound effects
4. Gestural eg the movement, speed and stillness of facial expression and body language
5. Spatial eg the proximity, direction and position of layout and organisation of objects in space (see Bull and Anstey, 2010:2)

Students are taught how text structure, vocabulary, sentence and clauses are carefully selected and combined to achieve a particular purpose and meaning in linguistic text. Similarly, students need to understand how the codes and conventions of images, audio, gesture and space are selected and combined to make meaning. When reading (consuming) or writing (producing) factual multimodal texts, students also need to understand how all the semiotic systems come together to make meaning. For example, the gradual increase in volume and change of pace in music can influence mood and therefore the way a scene or information in a film is interpreted. Similarly the selection, cropping and placement of an image in relation to linguistic text can create nuances of, or even opposing meanings, to the written text. When students are accessing information to learn, or reporting their learning through the production of a multimodal text, these understandings and analyses are critical.

This table shows some codes and conventions for each semiotic system (except for linguistic) and give a starting point for the metalanguage teachers and students will need to use when analysing and describing how meaning is being made through the use of various semiotic systems in a factual multimodal text and what each semiotic system is contributing to the overall meaning.