Research Papers

Research Papers

This page features downloadable academic papers on visual language research. Also, see my blog for more accessible writing for laypeople, along with more casual and exploratory writing, along with updates about ongoing research. For more detailed information about visual language and its study, check out my books, The Visual Language of Comics and Who Understands Comics?

Papers are organized here by their major topics as best as possible. Because aspects of visual language are interconnected, some papers may discuss multiple topics. For a chronological listing of papers, see my vitae, and for articles specifically about cross-cultural analyses of comics see pages for the VLRC and TINTIN Project.

Visual Language Theory

The study of visual language covers a wide range of topics, just like that of spoken or signed languages. These papers discuss the broader relationship between visual language, comics, and lingusitics, and how to study them.

  • Cohn, Neil and Joseph P. Magliano. 2020. Editors’ Introduction and Review: Visual Narrative Research: An Emerging Field in Cognitive Science. Topics in Cognitive Science. 12(1): 197-223 (Online version & PDF)
  • Cohn, Neil. 2020. Visual narrative comprehension: Universal or not? Psychonomic Bulletin and Review. 27(2): 266-285 (Online version & PDF)
  • Cohn, Neil. 2018. Visual Language Theory and the scientific study of comics. In Wildfeuer, Janina, Alexander Dunst, Jochen Laubrock (Ed.). Empirical Comics Research: Digital, Multimodal, and Cognitive Methods. (pp. 305-328) London: Routledge. (PDF)
  • Cohn, Neil. 2015. Climbing trees and seeing stars: Combinatorial structure in comics and diverse domains. In Toivonen, Ida, Piroska Csúri & Emilie van der Zee. (Ed). Structures in the Mind: Essays on Language, Music, and Cognition in Honor of Ray Jackendoff (pp. 379-392). Cambridge: MIT Press. (PDF
  • Cohn, Neil. 2014. Building a better “comic theory”: Shortcomings of theoretical research on comics how to overcome them. Studies in Comics. 5(1): 57-75. (PDF)
  • Cohn, Neil. 2012. Comics, linguistics, and visual language: The past and future of a field. In Bramlett, Frank (ed). Linguistics and the Study of Comics (pp. 92-118). New York: Palgrave Macmillan. (PDF)  
  • Cohn, Neil. 2010. Japanese Visual Language: The structure of manga. In Manga: An Anthology of Global and Cultural Perspectives, edited by T. Johnson-Woods (pp. 187-203). New York: Continuum Books. (PDF)  
  • Cohn, Neil. 2005. Un-defining “comics.” International Journal of Comic Art. 7(2): 236-248. (PDF)

Visual Narrative Grammar

My main project in visual langauge research has always been exploring how sequences of images create meaning in the mind and brain. I am primarily interested in how such competence is structured, how it is processed in the brain, and how that system relates to other cognitive domains like verbal discourse and narrative, event structure, and even musical cognition. Because the theory of narrative grammar is fairly complex, students and researchers interested in the diagnostics and procedures of implementing it should read this tutorial on How to analyze visual narrative (PDF).

Review papers – Give an overview of the theory of visual narrative structure and its comprehension

  • Coderre, Emily L. and Neil Cohn. 2023. Individual differences in the neural dynamics of visual narrative comprehension. Psychonomic Bulletin and Review. (Online version)
  • Cohn, Neil and Martin Paczynski. 2019. The neurophysiology of event processing in language and visual events. In Truswell, Robert (Ed.). Handbook of event structure. (pp. 624-637). Oxford: Oxford University Press. (PDF)  
  • Cohn, Neil. 2019. Structural complexity in visual narratives: Theory, brains, and cross-cultural diversity. In Grishakova, Marina and Maria Poulaki (Ed.). Narrative Complexity and Media: Experiential and Cognitive Interfaces. (pp. 174-199). Lincoln: University of Nebraska Press. (PDF)
  • Cohn, Neil. 2020. Your brain on comics: A cognitive model of visual narrative comprehension. Topics in Cognitive Science. 12(1): 352-386 (Online version)
  • Cohn, Neil. 2019. Visual narratives and the mind: Comprehension, cognition, and learning.In Federmeier, Kara D. and Diane M. Beck (Eds). Psychology of Learning and Motivation: Knowledge and Vision. Vol. 70. (pp. 97-128). London: Academic Press (PDF)

Theory papers – The theory of Visual Narrative Grammar posits a system that organizes information into narrative structure using a construction grammar. These papers outline this overall theory and/or its application to computational models.

  • Cohn, Neil and Joost Schilperoord. 2023. Visual languages and the problems with ideographies: A commentary on MorinBrain and Behavioral Sciences. 46: e240
  • Klomberg, Bien, Irmak Hacımusaoğlu, Lenneke Lichtenberg, Joost Schilperoord, and Neil Cohn. 2023. Continuity, Co-reference, and Inference in Visual Sequencing. Glossa: a journal of general linguistics 8(1). (Read online)
  • Martens, Chris, Rogelio E. Cardona-Rivera, and Neil Cohn. 2020. The Visual Narrative Engine: A Computational Model of the Visual Narrative Parallel Architecture. Proceedings of the 8th Annual Conference on Advances in Cognitive Systems (PDF
  • Cohn, Neil. 2019. Being explicit about the implicit: Inference generating techniques in visual narrative. Language and Cognition. 11(1): 66-97. (Online version
  • Cohn, Neil. 2018. In defense of a “grammar” in the visual language of comics. Journal of Pragmatics. 127: 1-19. (PDF)
  • Cohn, Neil. 2016. From Visual Narrative Grammar to Filmic Narrative Grammar: The narrative structure of static and moving images. In Wildfeuer, Janina & John A. Bateman (eds.). Film text analysis: New perspectives on the analysis of filmic meaning. (pp. 94-117). London: Routledge. (PDF)
  • Cohn, Neil. 2015. How to analyze visual narrative. Visual Language Lab Resource (PDF)
  • Cohn, Neil. 2015. Narrative conjunction’s junction function: The interface of narrative grammar and semantics in sequential images.  Journal of Pragmatics 88: 105-132. (PDF)
  • Cohn, Neil. 2014. The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology. 5:680. (Online version & PDF, Japanese PDF)
  • Cohn, Neil. 2013. Visual narrative structure. Cognitive Science 37(3): 413-452. (Online version
  • Cohn, Neil. 2010. The limits of time and transitions: Challenges to theories of sequential image comprehension. Studies in Comics 1(1): 127-147. (PDF)
Visual Narrative Engine

Experimental and corpus papers – We have explored the theory of Visual Narrative Grammar, and the broader processing of sequential images, through experimentation using both behavioral and neurocognitive measures.

  • Patrick, Morgan, Neil Cohn, John Mertus, and Sheila Blumstein. Sequences in harmony: Cognitive interactions between musical and visual narrative structure. Acta Psychologia. 238:103981 (Read online)
  • Cohn, Neil, Irmak Hacımusaoğlu, and Bien Klomberg. 2022. The framing of subjectivity: Point-of-view in a cross-cultural analysis of comics. Journal of Graphic Novels and Comics. (Read online)
  • Klomberg, Bien and Neil Cohn. 2022. Picture perfect Peaks: Comprehension of inferential techniques in visual narratives. Language and Cognition. 1-26. (Read online)
  • Klomberg, Bien, Irmak Hacımusaoğlu, and Neil Cohn. 2022. Running through the Who, Where, and When: A cross-cultural analysis of situational changes in comics. Discourse Processes. (Read online)
  • Coopmans, Cas and Neil Cohn. 2022. An electrophysiological investigation of co-referential processes in visual narrative comprehension. Neuropsychologia. 172:108253 (Online version)
  • Cohn, Neil. 2021. A starring role for inference in the neurocognition of visual narratives. Cognitive Research: Principles and Implications. 6:8 (Online version)
  • Cohn, Neil, and Tom Foulsham. 2020. Zooming in on the cognitive neuroscience of visual narrative.  Brain and Cognition 146:105634. (PDF
  • Foulsham, Tom and Neil Cohn. 2020. Zooming in on visual narrative comprehension. Memory & Cognition. 49(3): 451-466 (PDF
  • Manfredi, Mirella, Neil Cohn, Pamela Sanchez Mello, Elizabeth Fernandez, and Paulo Sergio Boggio. 2020. Visual and verbal narrative comprehension in children and adolescents with autism spectrum disorders: an ERP study. Journal of Autism and Developmental Disorders (Online version)
  • Coderre, Emily L., Neil Cohn, Sally K. Slipher, Mariya Chernenok, Kerry Ledoux, and Barry Gordon. 2018. Visual and linguistic narrative comprehension in autism spectrum disorders: Neural evidence for modality-independent impairments. Brain and Language. 186: 44-59. (PDF)
  • Cohn, Neil, Martin Paczynski, and Marta Kutas. 2017. Not so secret agents: Event-related potentials to semantic roles in visual event comprehension. Brain and Cognition. 119: 1-9. (PDF)
  • Cohn, Neil and Marta Kutas. 2017. What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing. Cognitive Research: Principles and Implications. 2(27): 1-13. (PDFOnline version)
  • Cohn, Neil and Patrick Bender. 2017. Drawing the line between constituent structure and coherence relations in visual narratives. Journal of Experimental Psychology: Learning, Memory, and Cognition. 43(2): 289-301. (PDF)
  • Foulsham, Tom, Dean Wybrow, and Neil Cohn. 2016. Reading without words: Eye-movements in the comprehension of comic strips. Applied Cognitive Psychology. 30: 566-579 (PDF)
  • Hagmann, Carl Erick, and Neil Cohn. 2016. The pieces fit: Constituent structure and global coherence of visual narrative in RSVP. Acta Psychologica. 164:157-164. (PDF)
  • Cohn, Neil, and Marta Kutas. 2015. Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia 77: 267-278. (PDF)
  • Cohn, Neil, and Eva Wittenberg. 2015. Action starring narratives and events: Structure and inference in visual narrative comprehension. Journal of Cognitive Psychology. 27(7): 812-828. (PDF)
  • Cohn, Neil, Ray Jackendoff, Phillip Holcomb, and Gina Kuperberg. 2014. The grammar of visual narratives: Neural evidence for constituent structure in visual narrative comprehension. Neuropsychologia. 64: 63-70. (PDFVideo summary)
  • Cohn, Neil. 2014. You’re a good structure, Charlie Brown: The distribution of narrative categories in comic strips. Cognitive Science, 38(7): 1317-1359. (Read online
  • Cohn, Neil, and Martin Paczynski. 2013. Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative. Cognitive Psychology. 67 (3):73-97. (PDF)
  • Cohn, Neil, Martin Paczynski, Ray Jackendoff, Phillip Holcomb, and Gina Kuperberg. 2012. (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension. Cognitive Psychology65(1): 1-38. (PDF)
  • Cohn, Neil, Amaro Taylor-Weiner, and Suzanne Grossman. 2012. Framing attention in Japanese and American comics: Cross-cultural differences in attentional structure. Frontiers in Cultural Psychology. 3: 1-12. (PDF)
  • Cohn, Neil. 2011. A different kind of cultural frame: An analysis of panels in American comics and Japanese manga. Image [&] Narrative 12 (1): 120-134. (PDF)

Multimodality

Visual languages rarely appear in isolation, and usually accompany written language. These papers explore these types of multimodal interactions.

  • Cohn, Neil and Joost Schilperoord. 2022. Reimagining language. Cognitive Science. 46(7): 1-6, e13164. (PDF)
  • Schilperoord, Joost and Neil Cohn. 2022. Before: Unimodal linguistics, After: Multimodal linguistics: A parallel architecture account of a multimodal construction. Cognitive Semantics. 8: 109-140. (Read online)
  • Cohn, Neil and Joost Schilperoord. 2022. Remarks on multimodality: Grammatical interactions in the Parallel Architecture. Frontiers in Artificial Intelligence. 4:778060 (Read online)
  • Manfredi, Mirella, Neil Cohn, Beatriz Ribeiro, Pamella Sanchez Pinho, Elisabete Fernandes Rodrigues Pereira, and Paulo Sergio Boggio. 2021. The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder. Brain and Cognition 151:105730 (PDF)
  • Yum, Yen Na, Neil Cohn, and Way Kwok-Wai Lau. 2020. Effects of picture-word integration on reading visual narratives in L1 and L2. Learning and Instruction. 71:101397. (PDF)
  • Manfredi, Mirella, Neil Cohn, Mariana De Araújo Andreoli, and Paulo Sergio Boggio. 2018. Listening beyond seeing: Event-related potentials to audiovisual processing in visual narrative. Brain and Language.185: 1-8. (PDF)
  • Cohn, Neil, Ryan Taylor, and Kaitlin Pederson. 2017. A picture is worth more words over time: Multimodality and narrative structure across eight decades of American superhero comics. Multimodal Communication. 6(1): 19-37. (PDFVideo Presentation)
  • Manfredi, Mirella, Neil Cohn, and Marta Kutas. 2017. When a hit sounds like a kiss: an electrophysiological exploration of semantic processing in visual narrative. Brain and Language. 169: 28-38. (PDF)
  • Pratha, Nimish K., Natalie Avunjian, and Neil Cohn. 2016. Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication. 5(2): 93-109. (PDFVideo Presentation)
  • Cohn, Neil. 2016. A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition 146: 304-323. (PDF)
Multimodal Parallel Architecture

Emoji

Our research on emoji has largely explored the degree to which emoji exhibit “linguistic” properties and what are the limits on how emoji are structured. Specifically, we’ve explored whether people use a grammar in sequencing emoji, how emoji interact with verbal grammar, and how well do people recognize the vocabulary of emoji.

  • Weissman, Ben, Jan Engelen, Elise Baas and Neil Cohn. 2023. The Lexicon of Emoji? Conventionality Modulates Processing of Emoji. Cognitive Science, 47: e13275. (Online paper)
  • Cohn, Neil, Jan Engelen, and Joost Schilperoord. 2019. The grammar of emoji? Constraints on communicative pictorial sequencing. Cognitive Research: Principles and Implications. 4(1):33. (Online version & PDF)
  • Cohn, Neil, Tim Roijackers, Robin Schaap, and Jan Engelen. 2018. Are emoji a poor substitute for words? Sentence processing with emoji substitutions. In T.T. Rogers, M. Rau, X. Zhu, & C. W. Kalish (Eds.), Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 1524-1529) Austin, TX: Cognitive Science Society. (PDFPoster version)
Emoji Grammar

Drawing

These papers discuss the cognition of drawing: how we create meaning by making graphic marks. They explore how drawing is structured in the mind/brain, how this ability is developed, and the effect of culture on these abilities.

  • Cohn, Neil. 2014. Framing “I can’t draw”: The influence of cultural frames on the development of drawing. Culture & Psychology. 20(1): 102-117. (PDF)
  • Cohn, Neil. 2012. Explaining “I can’t draw”: Parallels between the structure and development of language and drawing. Human Development 55(4): 167-192. (PDF)

Visual Vocabulary

These papers cover aspects of visual language “vocabulary”—the meaningful elements of images.

  • Szawerna, Michał and Neil Cohn. Iconicity in the visual lexicons of comics. Olga Fischer, Kimi Akita, and Pamela Perniss (eds). The Oxford Handbook of Iconicity in Language. Oxford, UK: Oxford University Press (PDF)
  • Hacımusaoğlu, Irmak and Neil Cohn. The meaning of motion lines?: A review of theoretical and empirical research on static depiction of motion. Cognitive Science. 47 (11):e13377 (Open online)
  • Atilla, Fred, Bien Klomberg, Bruno Cardoso, Neil Cohn. 2023. Background Check: Cross-Cultural Differences in the Spatial Context of Comic ScenesMultimodal Communication.
  • Hacımusaoğlu, Irmak and Neil Cohn. 2022. Linguistic Typology of Motion Events in Visual Narratives. Cognitive Semiotics. 1-26. (Open online)
  • Cohn, Neil and Tom Foulsham. 2022. Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji. Memory and Cognition. (Open online)
  • Schilperoord, Joost and Neil Cohn. 2021. Let there be … visual optimal innovations. Making visual meaning through Michelangelo’s Creation of AdamVisual Communication. 14703572211004994 (Open online)
  • Cohn, Neil. Combinatorial morphology in visual languages. In Booij, Geert (Ed.). The Construction of Words: Advances in Construction Morphology. (pp. 175-199). London: Springer. (PDF)
  • Cohn, Neil, Vivian Wong, Kaitlin Pederson & Ryan Taylor. 2017. Path salience in motion events from verbal and visual languages. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. J. Davelaar (Eds.), Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1794-1799). Austin, TX: Cognitive Science Society. (PDF)
  • Cohn, Neil, Beena Murthy, and Tom Foulsham. 2016. Meaning above the head: Combinatorial constraints on the visual vocabulary of comics.  Journal of Cognitive Psychology. 28(5): 559-574. (PDF
  • Cohn, Neil and Sean Ehly. 2016. The vocabulary of manga: Visual morphology in dialects of Japanese Visual Language. Journal of Pragmatics. 92: 17-29. (PDFListing of visual morphemes in JVL)
  • Cohn, Neil, and Stephen Maher. 2015. The notion of the motion: The neurocognition of motion lines in visual narratives.  Brain Research. 1601: 73-84. (PDF)
  • Cohn, Neil. 2013. Beyond speech balloons and thought bubbles: The integration of text and image. Semiotica. 2013(197): 35-63. (PDF
  • Cohn, Neil. 2010. Extra! Extra! Semantics in comics!: The conceptual structure of Chicago Tribune advertisements. Journal of Pragmatics 42 (11): 3138–3146. (PDF)
  • Cohn, Neil. 2007. A visual lexicon. Public Journal of Semiotics. 1(1): 35-56. (PDF)

Page Layout

These papers discuss aspects of comic page layouts—their “External Compositional Structure.”

  • Cohn, Neil, Jessika Axnér, Michaela Diercks, Rebecca Yeh, and Kaitlin Pederson. 2019. The cultural pages of comics: Cross-cultural variation in page layouts.  Journal of Graphic Novels and Comics. 10(1): 67-86 (PDF)
  • Pederson, Kaitlin and Neil Cohn. 2016. The changing pages of comics: Page layouts across eight decades of American superhero comics. Studies in Comics. 7(1): 7-28. (PDFVideo Presentation)
  • Cohn, Neil and Hannah Campbell. 2015. Navigating comics II: Constraints on the reading order of page layouts. Applied Cognitive Psychology. 29: 193-199. (PDF
  • Cohn, Neil. 2013. Navigating comics: An empirical and theoretical approach to strategies of reading comic page layouts. Frontiers in Cognitive Science. 4: 1-15. (Online version & PDF)