Today and tomorrow colleagues and I are organising a two-day workshop on the use of LLMs for linguistics research at @UniKoeln: https://sfb1252.github.io/llm-workshop/. This was originally intended as a small, university-internal event, organised as part of the CRC "Prominence in Language", but we have found that there is a lot of interest from colleagues and students about this topic so we are delighted to have 15 poster presentations, in addition to four brilliant keynote speakers, and a concluding panel discussion. 1/🧵
Discussion
Loading...
Discussion
But what do we mean by LLMs exactly? To ensure that everyone is on the same page, co-organiser @nilsreiter is giving a pre-workshop lecture "An Introduction to Large Language Models: LLMs 101". Attendees get introduced to important distinctions such as user prompt vs. system prompt, pre-training vs. fine-tuning, and commercial vs. locally-run open-source models. 2/🧵
Our first keynote speaker is @tschfflr from the Ruhr University Bochum. Tatjana's talk is entitled "Linguistic analysis with LLMs? The case of discourse and pragmatics" and she's brought along no fewer than three studies from her team's work on metaphor detection in religious language, the semantics and pragmatics of emojis, and the detection of hate and candy speech. Her implications for #linguistics research are both very cautious and optimistic as to the future of linguistics. 3/🧵
Tatjana presenting a slide on annotation, data and corpora.
Tatjana speaking at the lectern with an opening slide featuring the image of a robot writing with a feather
Our second keynote speaker of the day is a Mercator Fellow of the CRC "Prominence in Language" @dingemansemark. Mark's asks "What makes LLMs so irresistible?" and begins with our ancient "amazing ability to make sense of nonsense" citing bone-throwing, hand-reading, and horoscopes, before moving to ELIZA and ChatGPT. Mark stresses the need to distinguish between 'understanding' and 'regurgitating', and 'predicting' and 'responding' and introduces what he sees as useful mental models to describe LLMs such as 'stochastic parrots' and 'mansplaining as a service'. In the second half, Mark walked us through the many problems he sees with a recent publication on the supposed metalinguistic abilities of LLMs. Much food for thought for the discussion over coffee and the poster session! 4/🧵
Mark presenting a slide with a quote:
"Decisions made by the general public about emergent technologies depend much more on what that public attributes to such technologies than on what they actually are or can and cannot do"
Joseph Weizenbaum
Computer Power and Human Reason
1976
Mark presenting at the lectern.
It was a busy and lively poster session with presenters ranging from M.A. students to professors (check out the full list: https://sfb1252.github.io/llm-workshop/#poster-presentations), lots of discussions on the use of #LLMs for #linguistics research fulled by copious quantities of coffee and tray-fulls of cup cakes! We now take a short break before continuing the conversations over dinner. 5/🧵
Two people looking at poster, whilst holding a cup of coffee. Cup cakes in the background.
A person presenting a poster and two people listening.
Three people discussing a poster.
Tatjana and Jürgen in front of poster about "Simulating Human Judgments
The Potential of LLMs in Linguistic Experiments on Prominence"
Oups, I nearly forgot to present our own poster... Luckily, Tiziana Ilie, M.A. student and first author on this project, did a grand job presenting our work on how DeepL deals with pronomial pickups of the hybrid noun "Mädchen". For this study, we used the #OpenSource model OLMo-2-7B to generate our experimental stimuli. Our abstract is here: https://paths.to/m%C3%A4dchen-poster and we will have a preprint out soon so watch this space...
We began day 2 of our Large Language Models (LLM) for linguistics research workshop @UniKoeln with a fascinating keynote by Charlotte Pouw on "Interpreting models for speech generation and understanding using methods from #psycholinguistics". Charlotte shared insights from two studies from her #PhD research. Check out the papers: https://aclanthology.org/2025.conll-1.9/ and https://direct.mit.edu/coli/article/50/4/1557/123790. 7/🧵
~1 more replies (not shown)
Our final keynote speaker is also a doctoral researcher, Sam Boeve, presenting "From Tool to Theory: LLMs in Psycholinguistics". Sam presented three case studies, the last of which is based on his recent work on Dutch LLM's surprisal estimates in sentence, paragraph and book reading (https://doi.org/10.3758/s13428-025-02774-4). As previous research with English has shown, it turns out that not one size fits all and larger models are not necessarily better. 8/🧵
A systematic evaluation of Dutch large language models’ surprisal estimates in sentence, paragraph and book reading
We concluded our two-day workshop on LLMs for linguistic analyses at the CRC "Prominence in Language" @UniKoeln with a panel discussion with our four keynote speakers. The symposium's social media correspondent was also tasked with moderating the discussion, hence the lack of photo 🙃
I'm definitely biased as to how it went, so all I can tell you is that I very much enjoyed the discussion. I thought it was refreshing to have a forum in which people could disagree and exchange diverging views on such important issues as the inherent biases of LLMs, their cognitive plausibility, the dangers of using commercial models and associated products for research, the responsibility of linguists in countering the hype, and the risk of linguistics research becoming even WEIRDer than it already is as a result of increased LLM usage.
A huge thank you to our four invited speakers, everyone who contributed to the discussion, and everyone who attended! Safe travels! 🚆 🚲 9/🧵
P.S.: There were a lot of muffins and mini-croissants left, but I came equipped with tupperware, which the students of my afternoon class emptied in no time. 🥐 🧁
Poster of the event:
LARGE LANGUAGE MODELS
FOR LINGUISTIC ANALYSES:
APPLICATIONS AND LIMITATIONS
24.-25.11.2025
University of Cologne, Main building, Neuer Senatssaal
WHO?
Tatjana Scheffler: „Linguistic analysis with LLMs? The case of discourse and pragmatics"
Mark Dingemanse: „What makes LLMs so irresistible?"
Charlotte Pouw: „Interpreting models for speech generation and understanding using methods from psycholinguistics"
Sam Boeve: „From Tool to Theory: LLMs in Psycholinguistics"
Photos of our four invited speakers.
bonfire.cafe
A space for Bonfire maintainers and contributors to communicate
Automatic federation enabled