Adventures in Multimodality (AIM)

The Adventures in Multimodality group considers the concepts “medium,” “mode/ modality,” and “genre” fundamental to its scholarly work. We take Marshall McLuhan’s “the medium is the message” as entailing that each medium has specific affordances and constraints. Hence shifting information from one medium to another inevitably affects the contents of this information. Although we study a medium primarily from discourse-internal perspectives, we acknowledge that such studies inevitably require an awareness of a medium’s pragmatic aspects: what institution presents the medium? how is it financed? what techniques are involved? In addition, we are interested in how an understanding of evolutionary, socio-biological, and neurological approaches can contribute to insights in the interpretation of multimodal artefacts and multimodal communication — and vice versa. Most of us want to be what Nassim Nicholas Taleb, in The Black Swan (2010 [2007], Penguin) calls “skeptical empiricists” — people who believe, among other things, that one shouldn’t go from books to problems, but from problems to books (p. 291).

Communication often combines more than one mode/modality. The discussion what should count as a mode is still in full swing, but AIM researchers for the time being distinguish the following modes: written language; spoken language; static and moving images/visuals; music; sound; and gestures.Despite a fast growing number of publications, theorizing pertaining to multimodal discourse is still weak. Much research takes verbal language as the model and overextends the analogy with other modes. Moreover, the concept of “mode” is insufficiently defined or circumscribed and very little research discusses the interaction between modes.  It is moreover a source of worry and surprise that knowledge from studies investigating artistic discourse (including narratology) is often neglected.

AIM positions itself in the field through methods that deem genre an indispensable “interface” between text and context and considers pattern finding, however defined, as essential.  The visual mode (static and moving images) is the central mode, and is studied in its interaction with other modes (language, music, sound, gestures). Objects of research are primarily those drawn from forms of communication (information graphics, maps, logos, websites, cartoons) and “low” art (feature film, photography, comics), but also include games and jokes. Results are formulated in such a way that they can be (dis)confirmed, refined, improved on and, where appropriate, experimentally tested.

At the centre of each project is a problem that needs to be solved; this problem suggests the usefulness of a specific paradigm or method — not the other way round. Ideally, the solution(s) to the problem can be shown to be truly relevant to one or more expert communities and, preferably, also to one or more lay communities. Each project has a clearly explained and justified corpus and methodology, both of which need to arise naturally from the problems identified and the research questions formulated. The questions “What, how, when, where?” necessarily precede the question “why?”

As a research group — consisting of academic lecturers and researchers, post-graduate students, graduate students and other sympathizers — we get together about once a month in order to share and discuss our research and monitor how individual projects are advancing. Therefore, this blog is intended to serve as a platform for archiving what has been presented in the meetings as well as to make available complementary material (essays, presentations, images that can be downloaded).  News, book reviews and articles, among others, will be posted. AIM hopes to be a beacon for those interested in the fast-developing field of multimodal metaphor, story-telling, and rhetoric.

Here are some guidelines about the structure of the blog. The content can be accessed in two ways. First, it is possible to search within specific categories and subcategories (using the category menu in the right column). There the contents have been categorized as follows:

(1) Abstracts and papers: This category allows searching for abstracts classified by specific topics or subcategories. Examples are advertising and film. In many cases it is possible to download a pre-proof version of the full paper.

(2) Book reviews: This category contains reviews of relevant texts in the field of multimodality.

(3) Group presentations: This category allows accessing the abstracts and slides used by AIM researchers during the group meetings.

A second way of accessing the content is simply by keeping your eyes on the “New” section (also in the right column) for the latest posts in all categories.

We keep thinking about how we can further improve the quality and variety of the blog’s contents, and welcome suggestions and comments.

Finally, credits for coming up with the name and the AIM acronym are due to John Haltiwanger (thanks John!). Natalia Sanchez Querubin designed the blog and created the AIM picture. She and I share the administrative responsibilities.

Charles Forceville (Dept. of Media Studies, Universiteit van Amsterdam), last updated 10 May 2011

http://home.medewerker.uva.nl/c.j.forceville/

For more detailed information regarding specific projects, publications and bibliography, please download the following PDF.

Structure_and_Rhetoric_of_Multimodal_Discourse_intro_5_October