Adventures in Multimodality (AIM):
The scholarly interest in how pictures and moving images present information in narratives and arguments has become ever more prominent in recent decades. This tendency straddles all media, and pertains to genres including art film, advertising, comics, facebook, instagram, and many more. Increasingly, the sonic modality also helps frame, or spin, information, through both sound effects and music.
The visual mode (static and moving images) is for most of us the central mode, and is studied in its interaction with other modes. Rather than lose ourselves in endless definitional problems about what counts as a mode, we have decided to be practical and distinguish the following modes: written language; spoken language; static and moving images/visuals; music; sound; and gestures. But we also acknowledge gestures as a separate mode, as well as smell, taste, and touch — although these latter are hitherto vastly undertheorized.
Assuming that all discourses are goal-driven, the group’s agenda is to record and analyze how multimodal discourses are structured and (attempt to) achieve their rhetorical, narrative and/or aesthetic aims. The research group shares a commitment to uncover pertinent variables, develop hypotheses, and systematically test these — all in the service of finding significant patterns.
Apart from “mode,” we consider “genre” and “medium” fundamental to our scholarly work. Our claim is that “genre” is the most important (albeit often subconsciously accessed) “interface” between “text” and “context.” And we take Marshall McLuhan’s “the medium is the message” as entailing that each medium has specific affordances and constraints. Hence shifting information from one medium to another inevitably affects the contents of this information. Although we study a medium primarily from discourse-internal perspectives, we acknowledge that such studies inevitably require an awareness of a medium’s pragmatic aspects: what institution presents the medium? how is it financed? what techniques are involved? In addition, we are interested in how an understanding of evolutionary, socio-biological, and neurological approaches can contribute to insights in the interpretation of multimodal artefacts and multimodal communication — and vice versa. Most of us want to be what Nassim Nicholas Taleb, in The Black Swan (2010 [2007], Penguin) calls “skeptical empiricists” — people who believe, among other things, that one shouldn’t go from books to problems, but from problems to books (p. 291).
Objects of research are primarily those drawn from forms of communication (information graphics, maps, logos, websites, cartoons) and “low” art (feature film, animation, comics, photography, comics), but also include games and jokes. At the centre of each project is a problem that needs to be solved; this problem suggests the usefulness of a specific paradigm or method — not the other way round. Ideally, the solution(s) to the problem can be shown to be truly relevant to one or more expert communities and, preferably, also to one or more lay communities. Each project has a clearly explained and justified corpus and methodology, both of which need to arise naturally from the problems identified and the research questions formulated. The questions “What, how, when, where?” necessarily precede the question “why?” We are also interested in ways to make theory feed into practice (however defined) and vice versa.
Here are some of our projects:
1. Structural & Creative Metaphor and Multimodal Discourse: We think metaphorically. How does this transpire in images? sounds? gestures? combinations of these modes?
2. Multimodality and Relevance Theory. How can Sperber & Wilson’s theory be applied to (audio)visual mass-communication?
3. Rhetorical Analysis and Multimodal Argumentation. Can pictures be used to “prove” anything? Can they independently mount arguments?
4. Figures of Depiction. Which “figures of speech” (metaphor, irony, hyperbole) have (audio)visual equivalents? How can they be identified and characterized ?
5. Multimodality and Comics, Cartoons, & Animation films. How can do these completely “man-made” media convey narrate and argue without language?
As a research group — consisting of academic lecturers and researchers, post-graduate students, graduate students and other sympathizers — we get together about once every two months in order to share and discuss our research and monitor how individual projects are advancing. Therefore, this blog is intended to serve as a platform for archiving what has been presented in the meetings as well as to make available complementary material (essays, presentations, images that can be downloaded). News, book reviews and articles, among others, will be posted. AIM hopes to be a beacon for those interested in the fast-developing field of multimodal metaphor, story-telling, and rhetoric.
Here are some guidelines about the structure of the blog. The content can be accessed in two ways. First, it is possible to search within specific categories and subcategories (using the category menu in the right column). There the contents have been categorized as follows:
(1) Abstracts and papers: This category allows searching for abstracts classified by specific topics or subcategories. Examples are advertising and film. In many cases it is possible to download a pre-proof version of the full paper.
(2) Book reviews: This category contains reviews of relevant texts in the field of multimodality.
(3) Group presentations: This category allows accessing the abstracts and slides used by AIM researchers during the group meetings.
(4) Student work: One of the regular assignments for students in ChF’s metaphor course is: create your own visual or multimodal metaphor and comment on it. Some good examples are uploaded.
A second way of accessing the content is simply by keeping your eyes on the “New” section (also in the right column) for the latest posts in all categories.
We keep thinking about how we can further improve the quality and variety of the blog’s contents, and welcome suggestions and comments.
Finally, credits for coming up with the name and the AIM acronym are due to John Haltiwanger (thanks John!). Natalia Sanchez Querubin designed the blog and created the AIM picture. She and I share the administrative responsibilities.
Charles Forceville (Dept. of Media Studies, Universiteit van Amsterdam), last updated 29 May 2014.
Adventures in Multimodality (AIM) 
It sounds like we are working in similar fields. We’d be interested in your thoughts on our new Embodiment Blog and love to collaborate if you’re interested.