#Tag · bonfire.cafe

Can #AI reasoning models infer people's underlying reasons in unstructured chat data from group decisions?

Across multiple prompting steps, #GTP5 usually did NOT select the same underlying reason as a human rater: https://doi.org/10.48550/arXiv.2601.05582

#AI #cogSci #textAnalysis #psychometrics

2 media

Figure 4 Overview of prompt engineering techniques at each step: ND, ZS, and CoT for Step 1, and CoT, SR, PD, and MoRE for Steps 2-4.

"Step 4 presents the greatest challenge. The model is tasked with inferring the underlying reasons behind each perception, based on indirect cues embedded in the conversation. Across all prompting strategies, performance remains low (Positive-F1 = 0.37–0.40), indicating that the model infrequently selects the same reasoning factors as humans. These low scores suggest that extracting latent intent from unstructured group dialogue may go beyond what the current LLM is able to interpret."

"These results suggest that the gap between LLM and human annotation widens progressively as the task shifts from surface-level extraction to the interpretation of expressed content, and finally to inference of unstated intent." — "Step 4 presents the greatest challenge. The model is tasked with inferring the underlying reasons behind each perception, based on indirect cues embedded in the conversation. Across all prompting strategies, performance remains low (Positive-F1 = 0.37–0.40), indicating that the model infrequently selects the same reasoning factors as humans. These low scores suggest that extracting latent intent from unstructured group dialogue may go beyond what the current LLM is able to interpret." "These results suggest that the gap between LLM and human annotation widens progressively as the task shifts from surface-level extraction to the interpretation of expressed content, and finally to inference of unstated intent."

arXiv.org

Can large language models interpret unstructured chat data on dynamic group decision-making processes? Evidence on joint destination choice

Social activities result from complex joint activity-travel decisions between group members. While observing the decision-making process of these activities is difficult via traditional travel surveys, the advent of new types of data, such as unstructured chat data, can help shed some light on these complex processes. However, interpreting these decision-making processes requires inferring both explicit and implicit factors. This typically involves the labor-intensive task of manually annotating dialogues to capture context-dependent meanings shaped by the social and cultural norms. This study evaluates the potential of Large Language Models (LLMs) to automate and complement human annotation in interpreting decision-making processes from group chats, using data on joint eating-out activities in Japan as a case study. We designed a prompting framework inspired by the knowledge acquisition process, which sequentially extracts key decision-making factors, including the group-level restaurant choice set and outcome, individual preferences of each alternative, and the specific attributes driving those preferences. This structured process guides the LLM to interpret group chat data, converting unstructured dialogues into structured tabular data describing decision-making factors. To evaluate LLM-driven outputs, we conduct a quantitative analysis using a human-annotated ground truth dataset and a qualitative error analysis to examine model limitations. Results show that while the LLM reliably captures explicit decision-making factors, it struggles to identify nuanced implicit factors that human annotators readily identified. We pinpoint specific contexts when LLM-based extraction can be trusted versus when human oversight remains essential. These findings highlight both the potential and limitations of LLM-based analysis for incorporating non-traditional data sources on social activities.

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de · 2 weeks ago

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, - #CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

Figure 1b. Number of observations per sample,
before and after filtering for data quality (N = 460). — Figure 1b. Number of observations per sample, before and after filtering for data quality (N = 460).

Reflection-philosophy order effects and correlations across samples

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de replied · 2 weeks ago

We've found recruiting people for online #research via #onlineAdvertising yielded good results on overt and covert #dataQuality measures (perhaps because participation incentives aren't financial):

Attention checks passed ≅ 2.6 out of 3

ReCAPTCHA (v3) ≅ 0.94 out of 1.0

Sample size > 5000 (from six continents)

https://doi.org/10.1017/S0034412525000198

#surveyMethods #cogSci #psychology #xPhi #QualityControl #econ #marketing

Analytic atheism and analytic apostasy across cultures

Fabrizio Musacchio

@pixeltracker@sigmoid.social · last month

🧠 New paper by Huang et al.: By using #pharmacological #fMRI and dynamic #connectome-based #PredictiveModeling, they show how #cortisol reshapes whole-brain #NetworkDynamics during emotional memory encoding. Trial-level analyses reveal distinct but increasingly integrated #arousal and #memory networks under #stress, supporting a hormonally driven "memory formation mode".

🌍 https://doi.org/10.1126/sciadv.adz4143

#Neuroscience #CognitiveNeuroscience #BrainNetworks #CogSci

Fig. 2. Analysis design. (A) Schematic of the trial-level phase synchrony extraction approach. (B) Schematic of dCPM. Edges that are significantly correlated with memory/arousal are selected. A linear model is then trained to predict memory/arousal on the left-out trial. This model is separately applied to all four study conditions (pill × emotionality) to yield four predictive networks. R, remembered.

Dynamic brain mechanisms supporting salient memories under cortisol

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de · last month

Are more reflective thinkers better at games that require empathy or perspective-taking?

In two experiments, reflective thinkers performed better on such games, seemingly because they paid more attention to the other players' incentives.

https://doi.org/10.1017/S1930297500007373

#econ #cogSci #psychology #relationships #negotiation #diplomacy #geopolitics #intelligence #defense #security #policy

4 media

"Nash equilibrium is a game solution in which none of the players has a self-interested incentive to deviate from its own strategy after considering the counterpart’s choice. In Experiment 1, we used a particular class of game called dominance-solvable. These games contain an option which is better than another one for a player, independently of the action the counterpart will take. We refer to this option as a dominant strategy.3 In Experiment 1, we used two classes of dominance-solvable games characterized by diﬀerent equilibrium structures, creating sixteen 2x2 games for each class (for a full list of game matrices, see Figure A1 in section A.1, Appendices). The two classes of games (Figure 1) were: (1) dominance solvable “self” games (DSS), in which only the participant had a strictly dominant strategy; (2) dominance solvable “other” games (DSO), in which only the opponent had a strictly dominant strategy."

"In order to explore the type of visual analysis performed by participants, transitions were divided in ﬁve major types (Figure 2), following the classiﬁcation of Devetag and colleagues (2016):

1) own-payoﬀs within-action transitions: transitions between player’s own payoﬀs within a single row (necessary to identify the action with the highest average payoﬀ).

2) own-payoﬀs between-action transitions: ....

3) other-payoﬀs within-action transitions: ...

4) other-payoﬀs between-action transitions: transitions between the counterpart’s payoﬀs within a single row (necessary to identify the presence of counterpart’s dominant choices).

5) intra-cell transitions: transitions between the payoﬀs of the two players, within the same cell (necessary to compare the two players’ payoﬀs given a speciﬁc combination of choices)." — "In order to explore the type of visual analysis performed by participants, transitions were divided in ﬁve major types (Figure 2), following the classiﬁcation of Devetag and colleagues (2016): 1) own-payoﬀs within-action transitions: transitions between player’s own payoﬀs within a single row (necessary to identify the action with the highest average payoﬀ). 2) own-payoﬀs between-action transitions: .... 3) other-payoﬀs within-action transitions: ... 4) other-payoﬀs between-action transitions: transitions between the counterpart’s payoﬀs within a single row (necessary to identify the presence of counterpart’s dominant choices). 5) intra-cell transitions: transitions between the payoﬀs of the two players, within the same cell (necessary to compare the two players’ payoﬀs given a speciﬁc combination of choices)."

"the model that best predicts the proportion of equilibrium choices included only the CRT score (R 2 = .11, F (1, 46) = 5.59, B = 0.33, p = .022), while we did not ﬁnd any eﬀect of ﬂuid intelligence or working memory".

"the average causal mediation effect of proportion of other-payoﬀs within-action transitions on the relation between CRT score and proportion of equilibrium responses is statistically signiﬁcant (p = .002, based on 10000 bootstrap samples), accounting for an estimated 58 % of the total eﬀect between CRT score and proportion of equilibrium responses (Table 4)."

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de · 2 months ago

🧠🏔️ I’m sharing presentations from the Society for Judgment and Decision Making conference in #Denver at the URL below:

https://bsky.app/profile/byrdnick.com/post/3m6agtctbhc2b

My poster is about #argumentMapping and #learningScience. You will also find presentations about how to advance #cogSci with #AI tools, do #ProcessTracing in #Qualtrics without #coding, and avoid backfiring in #healthcare #nudges.

Follow to fight FOMO and enjoy #openAccess conferencing.

#SJDM25 #psychology #SciComm

Byrd, N. (2025, July). Map My Words—Using Waitlist Controlled Trials To Test Whether Argument Mapping Improves Individuals’ Persuasive Writing or Critical Thinking. Experimental Argument Analysis, University of East Anglia. https://www.researchgate.net/publication/390977878

Abstract. Argument mapping is the practice of diagraming the logical relationships between each proposition in an argument, including objections and counter-objections. Some studies find that courses that teach students how to map arguments exhibit better critical thinking and persuasive writing than students in other courses. These promising results are sometimes from surveyors of argument mapping goods and services, and they garner plenty of attention in fields that champion careful thinking and communication. However, the total evidence is mixed, null results are often never published, and many promising studies have not controlled for known confounds. Two waitlist control trials (N = 83) attempted to address t — Byrd, N. (2025, July). Map My Words—Using Waitlist Controlled Trials To Test Whether Argument Mapping Improves Individuals’ Persuasive Writing or Critical Thinking. Experimental Argument Analysis, University of East Anglia. https://www.researchgate.net/publication/390977878 Abstract. Argument mapping is the practice of diagraming the logical relationships between each proposition in an argument, including objections and counter-objections. Some studies find that courses that teach students how to map arguments exhibit better critical thinking and persuasive writing than students in other courses. These promising results are sometimes from surveyors of argument mapping goods and services, and they garner plenty of attention in fields that champion careful thinking and communication. However, the total evidence is mixed, null results are often never published, and many promising studies have not controlled for known confounds. Two waitlist control trials (N = 83) attempted to address t

https://bsky.app

View

Fabrizio Musacchio

@pixeltracker@sigmoid.social · 3 months ago

🧠 New paper by Aidan J. Horner (2025, Trends in Cognitive Sciences) introduces a 3D neural #StateSpace for #episodic memories. It replaces linear #SystemsConsolidation models with a dynamic framework where #hippocampal, #neocortical, and episodic specificity dimensions evolve independently and non-linearly, allowing memories to shift, reverse, or re-engage hippocampal circuits.

🌍 https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(25)00284-0

#Neuroscience #CognitiveScience #Hippocampus #CogSci #compneuro #memory

2 media

Figure 1. A modal neurocognitive model of episodic retrieval and systems consolidation. (A) When experiencing an
event, a distributed pattern of activity occurs along neocortical processing pathways (pathways shown here are intentionally
‘generic’ but could map to ventral and dorsal visual pathways as well as other sensory processing pathways; e.g., auditory).
These pathways converge on the hippocampus and are bound into a memory trace allowing for their retrieval later in time. At
encoding, all neocortical representations are active, forming a new hippocampal memory trace. At retrieval, a subset of neocor-
tical representations will be active, allowing for the retrieval of the hippocampal memory trace and subsequent reinstatement of
the remaining neocortical representations. (B) Over time, connections between neocortical regions form, allowing for their
retrieval without the involvement of the hippocampus. Blue lines indicate pre-existing connections/representations;
yellow lines indicate newly formed connections/representations; grey lines indicate previously formed connections that have
weakened. — Figure 1. A modal neurocognitive model of episodic retrieval and systems consolidation. (A) When experiencing an event, a distributed pattern of activity occurs along neocortical processing pathways (pathways shown here are intentionally ‘generic’ but could map to ventral and dorsal visual pathways as well as other sensory processing pathways; e.g., auditory). These pathways converge on the hippocampus and are bound into a memory trace allowing for their retrieval later in time. At encoding, all neocortical representations are active, forming a new hippocampal memory trace. At retrieval, a subset of neocor- tical representations will be active, allowing for the retrieval of the hippocampal memory trace and subsequent reinstatement of the remaining neocortical representations. (B) Over time, connections between neocortical regions form, allowing for their retrieval without the involvement of the hippocampus. Blue lines indicate pre-existing connections/representations; yellow lines indicate newly formed connections/representations; grey lines indicate previously formed connections that have weakened.

Figure 2. The state space comprises three dimensions: the degree to which reinstatement is driven by a hippocampal
mechanism (Hippocampal; weak to strong); the degree to which reinstatement is driven by a nonhippocampal mechanism
(Nonhippocampal; weak to strong); and the degree to which reinstatement is episodically speci c (Speci city; low to high). — Figure 2. The state space comprises three dimensions: the degree to which reinstatement is driven by a hippocampal mechanism (Hippocampal; weak to strong); the degree to which reinstatement is driven by a nonhippocampal mechanism (Nonhippocampal; weak to strong); and the degree to which reinstatement is episodically speci c (Speci city; low to high).

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de · 3 months ago

Prayer's effects on health have been studied many times.

Small studies initially reported some benefit, but more rigorous studies found null or even harmful effects.

In sum, prayer seemed ineffective, recommending focus on more promising interventions: https://doi.org/10.1002/14651858.CD000368.pub3

"Ten studies are included in this updated review (7646 patients). ...overall there was no clear effect of intercessory prayer on death... (6 RCTs, n=6784, random-effects RR 0.77 CI 0.51 to 1.16, I(2) 83%). For general clinical state there was also no significant difference between groups (5 RCTs, n=2705, RR intermediate or bad outcome 0.98 CI 0.86 to 1.11). Four studies found no effect for re-admission to Coronary Care Unit (4 RCTs, n=2644, RR 1.00 CI 0.77 to 1.30). Two other trials found intercessory prayer had no effect on re-hospitalisation (2 RCTs, n=1155, RR 0.93 CI 0.71 to 1.22).

...although some of the results of individual studies suggest a positive effect of intercessory prayer, the majority do not and the evidence does not support a recommendation either in favour or against the use of intercessory prayer. We are not convinced that further trials of this intervention should be undertaken and prefer to see resources ...used to investigate other questions in health care." — "Ten studies are included in this updated review (7646 patients). ...overall there was no clear effect of intercessory prayer on death... (6 RCTs, n=6784, random-effects RR 0.77 CI 0.51 to 1.16, I(2) 83%). For general clinical state there was also no significant difference between groups (5 RCTs, n=2705, RR intermediate or bad outcome 0.98 CI 0.86 to 1.11). Four studies found no effect for re-admission to Coronary Care Unit (4 RCTs, n=2644, RR 1.00 CI 0.77 to 1.30). Two other trials found intercessory prayer had no effect on re-hospitalisation (2 RCTs, n=1155, RR 0.93 CI 0.71 to 1.22). ...although some of the results of individual studies suggest a positive effect of intercessory prayer, the majority do not and the evidence does not support a recommendation either in favour or against the use of intercessory prayer. We are not convinced that further trials of this intervention should be undertaken and prefer to see resources ...used to investigate other questions in health care."

Intercessory prayer for the alleviation of ill health

Nick Byrd, Ph.D.

@ByrdNick@nerdculture.de replied · 3 months ago

Notably, some of the investigators had "a prior belief in the positive effects of #prayer" (Benson and Byrd) and nonetheless found plenty of null results, or results that hovered on either side of the null (p. 16, 👆).

🔓 https://pmc.ncbi.nlm.nih.gov/articles/PMC7034220/pdf/CD000368.pdf

#religion #health #cogSci #stats

“The Benson 2006 and Byrd 1988 studies were undertaken by researchers with a prior belief in the positive eﬀects of prayer. The beliefs of the researchers in the other trials are not clear.”

Roberts, L., Ahmed, I., Hall, S., & Davison, A. (2009). Intercessory prayer for the alleviation of ill health. The Cochrane Database of Systematic Reviews, 2009(2), CD000368. https://doi.org/10.1002/14651858.CD000368.pub3 — “The Benson 2006 and Byrd 1988 studies were undertaken by researchers with a prior belief in the positive eﬀects of prayer. The beliefs of the researchers in the other trials are not clear.” Roberts, L., Ahmed, I., Hall, S., & Davison, A. (2009). Intercessory prayer for the alleviation of ill health. The Cochrane Database of Systematic Reviews, 2009(2), CD000368. https://doi.org/10.1002/14651858.CD000368.pub3

Preparing to download ... (PDF)

Chip Butty and 1 other boosted

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪

@ElenLeFoll@fediscience.org · 3 months ago

We are delighted to announce that the new, multidisciplinary Diamond #OpenAccess journal **Replication Research (R2)** is now accepting submissions for replications and reproductions in a broad range of disciplines including #linguistics, #CogSci, and #DigitalHumanities! 🎉

https://www.uni-muenster.de/Ejournals/index.php/replicationresearch/index

#OpenScience #replication #academia #AcademicChatter

https://replicationresearch.org
Open for submissions!

DIAMOND OPEN ACCESS JOURNAL

MULTIDISCIPLINARY

REPETITIVE RESEARCH

ROBUST AND TRANSPARENT

Psychology
Linguistics
Political Science
Neuroscience
Management Science
Health Science
Geoscience
and many more!

Citable Reproducibility Checks
Open Data, Materials, and Code
Registered Reports
Open Peer Review Reports

Free to Read
Free to Publish
Researcher-Owned
Preprint Friendly
Non-Commercial
Mentorship Program
Optional Manuscript Templates

Replications
Reproductions
Multiverse analyses
Many analyst studies
Conceptual contributions
Tutorials
Reviews — https://replicationresearch.org Open for submissions! DIAMOND OPEN ACCESS JOURNAL MULTIDISCIPLINARY REPETITIVE RESEARCH ROBUST AND TRANSPARENT Psychology Linguistics Political Science Neuroscience Management Science Health Science Geoscience and many more! Citable Reproducibility Checks Open Data, Materials, and Code Registered Reports Open Peer Review Reports Free to Read Free to Publish Researcher-Owned Preprint Friendly Non-Commercial Mentorship Program Optional Manuscript Templates Replications Reproductions Multiverse analyses Many analyst studies Conceptual contributions Tutorials Reviews

Ross Gayler

@RossGayler@aus.social · 3 months ago

From Nao Tsuchiya @ Monash University

YouTube videos for the 2025 summer school for Qualia Structure & Integrated Information Theory

Dear all, (who contributed to our summer school projects in the past!)

We are starting to upload 2025 Qualia Structure IIT summer school lecture videos in this Youtube Channel
https://www.youtube.com/watch?v=cwPGZ1CacVU&list=PLEP8weJRxEPbn_D22O-RmWIZbxOPDO-Vu

PLEASE spread the words to your local networks (email list, social media, etc).

It would be really great if we can reach STUDENTS (from high school to PhD), early career research or more. (Nowadays, it's very difficult to reach the relevant audience!)

We are particularly interested in reaching the younger generation, who might be interested in joining the future summer school (we will organize it again, possibly in 2026 Sep or 2027 Feb).

For those who have not been exposed to the basic and interdisciplinary consciousness research, Day 1 talk by Christof Koch will be a great introduction.

For those who are interested in Integrated Information Theory, the gentle introduction by Matteo Grasso will be particularly informative.

#connectionist #consciousness #videos #CognitiveScience #CogSci #Summerschool

Ross Gayler

@RossGayler@aus.social · 3 months ago

A lot of #MachineLearning and #PredictiveModelling in #statistics is based on minimisation of loss with respect to a training data set. This assumes that the training data set as a whole is representative of potential training sets. Consequently, this implies that loss minimisation is not an appropriate approach (or way of conceptualising the problem) in problems where the training data sets are not representative of the potential testing sets. (As a working title, let's call this issue "radical nonstationarity".)

I recently read Javed & Sutton 2024 "The Big World Hypothesis and its Ramifications for Artificial Intelligence" (https://web.archive.org/web/20250203053026/https://openreview.net/forum?id=Sv7DazuCn8) and think it describes a superset of this issue of radical nonstationarity. I strongly recommend this paper for motivating why loss minimisation with respect to a training data set might not always be appropriate.

Imagine an intelligent agent existing over time in a "big world" environment. Each observation records information about a single interaction of the agent with it's environment, and this observation only records the locally observable part of the environment. The agent may be moving between locations in the environment that are radically different with respect to the predictive relationships that exist and the variables that are predictive of the outcome of interest may vary between observations. Nonetheless, there is some predictive information that an intelligent agent could exploit. The case where everything is totally random and unpredictable is of no interest when the focus of research is an intelligent agent. In such a world minimising loss with respect to the history of all observations seen by the agent or even a sliding window of recent history seems irrelevant to the point of obtuseness.

One possible approach to this issue might be for the agent to determine, on a per-observation basis, the subset of past observations that are most relevant to making a prediction for the current observation. Then loss minimisation might play some role in determining or using that subset. However, that use of a dynamically determined training set is not the same thing as loss minimisation with respect to a statically given training set.

I am trying to find pointers to scholarly literature that discusses this issue (i.e. situations where minimisation of loss with respect to some "fixed" training set). My problem is that I am struggling to come up with search terms to find them. So:

* Please suggest search terms that might help me find this literature
* Please provide pointers to relevant papers

#PhilosophyOfStatistics #PhilosophyOfMachineLearning #CognitiveRobotics #MathematicalPsychology #MathPsych #CognitiveScience #CogSci #CognitiveNeuroscience #nonstationarity #LossMinimisation

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪

@ElenLeFoll@fediscience.org · 3 months ago

https://www.uni-muenster.de/Ejournals/index.php/replicationresearch/index

#OpenScience #replication #academia #AcademicChatter

Ulrike Hahn boosted

Tom Stafford

@tomstafford@mastodon.online · 6 months ago

Encouraged by the reaction to yesterday's (very short) post : https://tomstafford.substack.com/p/ai-will-be-the-biro-of-thought

I have a set of similar pre-baked talking points on how to make sense of the AI/LLM revolution, so this may be the first in a series

#AI #LLMs #ChatGPT #CogSci

Tom Stafford

@tomstafford@mastodon.online · 6 months ago

Encouraged by the reaction to yesterday's (very short) post : https://tomstafford.substack.com/p/ai-will-be-the-biro-of-thought

I have a set of similar pre-baked talking points on how to make sense of the AI/LLM revolution, so this may be the first in a series

#AI #LLMs #ChatGPT #CogSci

Ulrike Hahn boosted

Richard Dallaway

@d6y@mastodon.green · 6 months ago

Sad news: “BODEN — Professor Margaret (Maggie) Boden, renowned cognitive scientist and long-time member of the University of Sussex, died peacefully in Brighton on 18th July 2025, aged 88.”

https://www.theargus.co.uk/memorials/death-notices/death/30683058.margaret-maggie-boden/notice/

#cogsci

Richard Dallaway

@d6y@mastodon.green · 6 months ago

Sad news: “BODEN — Professor Margaret (Maggie) Boden, renowned cognitive scientist and long-time member of the University of Sussex, died peacefully in Brighton on 18th July 2025, aged 88.”

https://www.theargus.co.uk/memorials/death-notices/death/30683058.margaret-maggie-boden/notice/

#cogsci