PETAA PAPER 181

When image and text meet: teaching with visual and multimodal texts

Jon Callow

Original caption: ‘Before it's too late.’ Image source: World Wildlife Federation (WWF) Climate Change

A seamless world

Being immersed in a visual culture, we can sometimes not even notice how we interact with images and words. The photo above of the logged forest catches our eye, summoning the image of a pair of lungs. The poster comes together with the conservation group’s logo and the tag line ‘Before it's too late’, making a powerful point.

Engaging with various texts can sometimes seem a simple, everyday act, belying the complexity involved. Until something catches our eye, jarring us, or delighting us with its simplicity, its beauty or its drama, we may not stop and reflect more deeply about what we have seen or experienced. It is when we do stop that we wonder how the author, artist or designer communicated their message so effectively.

The concept of multimodal texts is integrated in the Australian Curriculum: English, across the language, literature and literacy strands. From understanding social contexts, expressing and developing ideas through language and literature, to creating new texts, the multimodal nature of English is deeply embedded in the rationale and strands of the curriculum. Teamed with a knowledge of written language is the inclusion of a visual language thread (ACARA, 2011:66), as well as assessment linking to multimodal texts, expressed in the achievement standards from Foundation to Year 10.

A key concept to understand when working with multimodal texts is that the whole is greater than the sum of the parts. This means that we need to make sure we don’t just read a text like a picture book, talk about one or two literary features, discuss which picture we like the best then move on to the next activity. Multimodal texts make meaning because all the elements work together to create a whole text. This is the case for picture books, using only image and text, or for video and multimedia, where image, gesture, movement, words and sound all work together to create the final piece.

Consider the image for the advertisement below. Initially we are drawn to the fin on the left, perhaps reacting with a sense of fear. The caption reinforces this, yet when we look to the right, the fin is gone, but the caption suggests this is ‘more horrifying’. Isn’t this a contradiction? Is it because the shark is under the water? What could the advertisement mean? Reading the text on the bottom, the purpose becomes clearer — the loss of wildlife has an impact on the ecosystem. This is the same ecosystem on which humans also depend. The poster cleverly plays with fear of sharks and the juxtaposition of image and word, in order to convey its message.

Caption: ‘Exploiting the ecosystem also threatens human lives.’  Image source: World Wildlife Federation (WWF) Sharks

We can learn through multimodal texts, where our emotions and intellect are engaged but we can also learn about how texts like this work, by understanding how both the written and visual elements work (Callow, 1999; Kress & Van Leeuwen, 2006).

Visually the fin is the most salient element in the poster, with the adjective ‘horrifying’ shaping our interpretation and response, as well as augmenting the image. Reading left to right we move from the shark being present — the ‘given’, to its apparently sudden disappearance — the ‘new’. This new information is possibly more unsettling. The dark blue colours further add to a fearful atmosphere, as well as the fact that the fin in the first photo (and by association the shark) is framed to appear quite close to us as the viewer. The placement of the large photos at the top, with the more detailed text and logo down the bottom is common. Many advertisements put their main ideas at the top, considered the ‘ideal’ in western culture, while the more everyday aspects of the product, such as the details, or specific features, are often at the bottom, symbolically grounded in what we can term the ‘real’. The written text here is also powerful — ‘Exploiting the ecosystem also threatens human lives’. Strong action verbs like exploiting, and threatens create an immediate emotional reaction, especially when linked to the noun group human lives. From a critical literacy aspect, we can also ask about the world-view of this organisation as advocates for wildlife. Is the information true — would the loss of sharks really impact us as humans? Should we think about supporting this organisation? Reflecting critically allows us the space to step back and consider the points of view presented. (Callow, 2008.) Exploring multimodal texts can range from advertisements, posters and videos to one of the mainstays of any English classroom — the picture book.

Modelled viewing and picture books

Teaching viewing using picture books provides both exposure to quality literature as well as a stimulating variety of visual images and techniques. The following provides a guide for choosing and using pictures books with a focus on the visual and written text interplay.

Read and enjoy: Having chosen a piece of quality literature, make sure you read it and enjoy it, as your own knowledge and enthusiasm is important and will always positively influence your teaching.

Choose a key page from the story: There are often key moments in a story, which are represented in word and image. When planning, choose a key page, note down the main ideas and the visual and verbal features that are important in creating the ideas and feel of the page (refer to the samples below).

Literary themes and features: As you are teaching a whole text, consideration of the broader literary features should inform your lesson planning (Saxby,1997). These elements are also key aspects of the Australian Curriculum: English. Features include:

• Theme — ideas that hold the story together, such as friendship, survival, family, courage
• Plot — traditional narrative structures or variations on this
• Characters — major and minor characters, heroes and villains, animals as characters
• Point of view — first person or third person
• Place or setting — realistic (urban, city, bush), fantasy or imaginary, historical.

Table 1 below gives a summary of features that may be present. Only choose the features that are relevant or strongly represented on the page.

Visual (Concepts of print and screen; Visual Language*)	Multimodal meanings Guiding principles to consider when image and text meet	Written (Structures of different types of texts; Sentence and clause level grammar; Word level grammar; Vocabulary*)
Happenings Actions and settings – how line and shape represent people, objects and events, including speech and thought Conceptual and symbolic – images showing ideas and conceptual themes	In the context of the unfolding story, how does the page develop the characters, story events and build themes and concepts? How do the visual and written work together?	Events and Ideas Sentence and clause structures – building happenings and ideas through different verb types and adverbs/ adverbials – description of characters and setting through noun groups and adjectives
Reacting and relating Shot distance – long, mid or close Angle – high, low or eye level Gaze of characters – offers and demands Colours – moods and symbolism Shapes and textures and their impact Realism of images – photo, artwork, drawings, diagrams	How do the visual and written text shape our feelings and reactions to the characters? Do we react differently to the visuals than to the written text? Do the literary features complement or contradict the visual elements?	Interacting Statements, questions and commands Attitudes and opinions Modality – degrees of certainty Literary features – alliteration , repetition, rhyme, personification, metaphor; interesting vocabulary choices
Design and layout Balance – symmetry and contrast of elements Top/bottom, left/right, centre/margin – placement of elements Salience – what attracts attention Reading paths – vectors directing our gaze Framing – how text and image may frame each other; image cropping  ​	How does the layout of the visual and written text influence the reader/viewer? Does the layout privilege particular visual or written elements? Why might the author/illustrator have made these choices?	Text organisation Genre/ text type – literary (narrative, ballad, recount etc.) or factual Paragraphs Sentence order and structure

Table 1: Multimodal meanings — image and text

* denotes thread from Australian Curriculum: English. For more details on visual grammar, see Callow, 1999, and for more details on written grammar, see Derewianka, 2011

Scaffolding multimodal meanings

The middle column in the table brings together the visual and written aspects of the texts to ensure that the multimodal meanings are explored. Rather then just listing specific visual features and grammatical aspects, we need to take that extra step and explore how they work together (Walsh, 2010). Pedagogically, this will include reading the whole text, revisiting key pages and events in the story, explicitly teaching an aspect of the visual and written text and how they work together, then consolidating the concepts and skills in a concluding activity.

In a modelled viewing lesson (whole class or small group), a teacher demonstrates key visual and written features as well as involves their learners in a variety of discussions and hands-on activities. Within one lesson, a teacher may move across a scaffolding continuum, from more teacher-regulated practices (reading the text, making statements about key features, asking focused questions about features) to more scaffolded activities, where students take some responsibility for practising the demonstrated skills or knowledge (Hammond, 2001). Ideas for activities can include:

• sorting a selection of images into shot distances or types of angles
• listing key features such as salient points, characters that demand viewer attention or effective use of colour
• re-drawing a character so that they either look at the viewer (demand)or away from them (offer)
• creating a frozen moment scene from the story and using a digital camera to experiment with angles, framing or shot distance.

Visual literacy an interactive whiteboards

The interactive tools and techniques available on interactive white boards extend the creative ways of teaching about multimodal and visual texts (see Goodwin, K. e-update 017, 2011)

Pens and highlighters: Ask students to annotate an image, circle key features or draw vectors and reading paths.

Drag and drop: Match labels to parts of the page such as description of angles or shot distances, noun groups or verb types.

Manipulate elements: Using elements from a page (such as characters or objects), re-design the page to tell a different story, re-size them to make one element salient, or create text boxes to manipulate the font, size and the position of text on the screen. 

Conclusion

Working with multimodal texts provides opportunities to develop our learners’ love and interest in literature, enhance their knowledge of visual and written language, as well as develop their visual literacy skills as part of engaging with and enjoying the rich visual culture we live and learn in.

Dr Jon Callow
May 2012

About the author

Dr Jon Callow is an experienced educator and teacher, having worked in primary schools, universities and in professional development for teachers. He currently teaches at the University of Sydney, in the areas of English and multiliteracies and he has worked alongside teachers in their classrooms, both in Australia and the United States. His research interests include visual literacy, children’s literature, social justice and pedagogy and digital media.

References

• The Australian Curriculum: English (Version 5.1), ACARA
• Callow, J (1999), Image Matters – Visual Texts in the Classroom, PETAA, Newtown, Sydney.
• Callow, J (2008), ‘Show Me: Principles for Assessing students’ visual literacy’, (.pdf 339 kB) The Reading Teacher, 61(8), 616–626.
• Derewianka, B 2011, A new grammar companion for teachers, PETAA, Sydney.
• Goodwin, K (2011), ‘Engaging students in literacy learning with interactive whiteboards’, e:update, 017, PETAA, Newtown, Sydney
• Hammond, J (2001), Scaffolding : teaching and learning in language and literacy education, PETAA, Newtown, Sydney
• Kress, GR & van Leeuwen, T 2006, Reading images: The grammar of visual design, 2nd edition, Routledge, London; New York.
• Saxby, H M (1997), Books in the life of a child : bridges to literature and learning, Macmillan Education Australia, South Melbourne.
• Walsh, M (2010), Multimodal literacy: researching classroom practices, PETAA, Newtown, Sydney

How this content relates to AITSL teacher standards

• Professional knowledge
• Professional practice
• Professional engagement