This project "Visual narratives as a window into language and cognition" (nicknamed "TINTIN") is funded by an ERC Starting Grant. We will build tools for analyzing visual and multimodal information, and then incorporate these data into a database. All of these tools and data will be made publicly accessible for the general public and other researchers to explore the properties of comics around the world. Our specific project will study whether there are cross-cultural patterns in the visual languages used in comics of the world, and whether those patterns connect to the spoken languages of their authors.
This project is a follow up and expansion from the Visual Language Research Corpus which analyzed cross-cultural variation in comics from Asia, Europe, and the United States. However, the TINTIN project will be launching a more sophisticated annotation scheme and downloadable open software annotation tools, our Multimodal Annotation Software Tool (MAST). The corpus that we will build with this tool will also be open for researchers to access and to contribute to. We will analyze diverse comics from across the world, including those from typically under-represented regions.
Here's the official description of the TINTIN project:
"Drawn sequences of images are a fundamental aspect of human communication, appearing from instruction manuals and educational material to comics. Despite this, only recently have scholars begun to examine these visual narratives, making this an untapped resource to study the cognition of sequential meaning-making. The emerging field analysing this work has implicated similarities between sequential images and language, which raises the question: Just how similar is the structure and processing of visual narratives and language?
I propose to explore this query by drawing on interdisciplinary methods from the psychological and linguistic sciences. First, in order to examine the structural properties of visual narratives, we need a large-scale corpus of the type that has benefited language research. Yet, no such databases exist for visual narrative systems. I will thus create innovative visual annotation tools to build a corpus of 1,500 annotated comics from around the world (Stage 1). With such a corpus, I will then ask, do visual narratives differ in their properties around the world, and does such variance influence their comprehension (Stage 2)? Next, we might ask why such variation appears, particularly: might differences between visual narratives be motivated by patterns in spoken languages, thereby implicating cognitive processes across modalities (Stage 3)?
Thus, this proposal aims to investigate the domain-specific (Stage 2) and domain-general (Stage 3) properties of visual narratives, particularly in relation to language, by analysing both production (corpus analyses) and comprehension (experimentation). This research will be ground-breaking by challenging our knowledge about the relations between drawing, sequential images, and language. The goal is not simply to create tools to explore a limited set of questions, but to provide resources to jumpstart a budding research field for visual and multimodal communication in the linguistic and cognitive sciences."
Our current research team consists of several core staff and various collaborators around the world who will help find and analyze comics for our corpus and conduct experiments. We welcome additional collaborations, so if you are interested in working with us on this project, please inquire with Neil Cohn for details.
At Tilburg University, we collaborate with faculty members Joost Schilperoord and Myrthe Faber.
Bruno Cardoso is a postdocoral fellow who is programming our Multimodal Annotation Software Tool (MAST).
Bien Klomberg and Irmak Hacımusaoğlu are PhD students analyzing cross-cultural visual language typology and conducting experiments. Klava Fadeeva is a Masters Student who has helped gather comics for our corpus.
Fernando Casanova (University of Murcia, Spain) is a visiting PhD student who studies interjections in cross-cultural comics.
Heinz Insu Fenkl (State University of New York, New Paltz)
Leandro Kruszielski (Universidade Federal do Paraná, Brazil)
Anna Marta Marini (Instituto Franklin-UAH)
Marianna Pagkratidou (TU Dublin University)
Michael Ranta (Lund University)
Kazuki Sekine (Waseda University, Japan)
Gaurav Singh (Independent researcher)
Dušan Stamenković and Miloš Tasić (University of Niš, Serbia)
Michał Szawerna (University of Wroclaw)
Eszter Szép (Independent researcher and curator)
Yen Na Yum (Hong Kong Institute of Education)
Our multicultural research corpus has benefited from contributions and donations from several creators and companies. If you would like your comics to be analyzed within our corpus, please contact me!
AzCorp Entertainment (Mobile App)
Comics Factory (Yulia Magera)
Mighty Punch Studios (Facebook)
Two Gargoyles Comics