#Tag · bonfire.cafe

How do academics judge journals?

For > 1000 in #socialSciences and #humanities
- claiming fair/transparent #peerReview, having an accurate title, and disclosing author contact info increased credibility.
- reflective thinking predicted lower credibility.

https://doi.org/10.1002/leap.2042

4 media

"Short scenarios were created using the positive and negative aspects of journal characteristics. Each scenario describes a situation in which one is confronted with an imaginary scientific journal. The journal is described briefly based on the aforementioned characteristics. A total of eight scenarios were created (all available in Table A2 and in Table S3 at https://osf. io/vf3tx/), combining all the positive and negative aspects of the three characteristics (Table 3). An example of a single scenario with two positive and one negative characteristic (Scenario 4) is presented in Table 4."

"all main effects and interactions are s tically significant, with large effect sizes for the main effects and two-way interactions, and a medium effect size for the three-way interaction (Cohen 1988). Each positive journal characteristic significantly increased perceived credibility. When two or three positive features were combined, an interaction effect occurred whereby the shift in the journal's credibility was additionally strengthened, i.e., it was greater than the cumulative effect of the individual characteristics (Figure 1). The post hoc Tukey HSD test shows that most scenario pairs differed significantly in credibility ratings."

"The effect of publishing expectations was statistically significant: scholars with higher publishing expectations (HPE) gave lower credibility ratings than scholars with LPE, suggesting a more critical evaluation. However, the effect size was small (ηₚ 2 = 0.014). No significant difference was found between [social science] and [humanities] scholars.

The number of positive characteristics had a large effect on credibility ratings, with a greater number of positive characteristics leading to notably higher credibility ratings. As illustrated in Figure 2 , this increase follows an exponential trend, consistent with the interaction effects shown in Figure 1, where combinations of positive attributes enhance credibility beyond the sum of the individual attributes." — "The effect of publishing expectations was statistically significant: scholars with higher publishing expectations (HPE) gave lower credibility ratings than scholars with LPE, suggesting a more critical evaluation. However, the effect size was small (ηₚ 2 = 0.014). No significant difference was found between [social science] and [humanities] scholars. The number of positive characteristics had a large effect on credibility ratings, with a greater number of positive characteristics leading to notably higher credibility ratings. As illustrated in Figure 2 , this increase follows an exponential trend, consistent with the interaction effects shown in Figure 1, where combinations of positive attributes enhance credibility beyond the sum of the individual attributes."

"The correlation between [reflection test] scores and the average rating across all scenarios was negative and low, but statistically significant (Table 8), suggesting that more reflective participants tended to rate journals more critically. Specifically, a significant negative correlation was obtained for journals with zero or one positive journal characteristics and at the same time with three or two negative attributes. However, the ratings of journals with two and three positive attributes did not correlate with the [reflection test]."

"Exploratory Factor Analysis ...yielded a two-factor solution accounting for 59.39% of the total variance. Component 1 (eigenvalue of 3.593) explained 44.91% of the variance, whilst Component 2 (eigenvalue of 1.158) explained 14.48%. The structure matrix (see Table 9) shows that Component 1 included low-quality journals, and and Component 2 included high-quality journals. Journals with mixed characteristics loaded on both components" — "The correlation between [reflection test] scores and the average rating across all scenarios was negative and low, but statistically significant (Table 8), suggesting that more reflective participants tended to rate journals more critically. Specifically, a significant negative correlation was obtained for journals with zero or one positive journal characteristics and at the same time with three or two negative attributes. However, the ratings of journals with two and three positive attributes did not correlate with the [reflection test]." "Exploratory Factor Analysis ...yielded a two-factor solution accounting for 59.39% of the total variance. Component 1 (eigenvalue of 3.593) explained 44.91% of the variance, whilst Component 2 (eigenvalue of 1.158) explained 14.48%. The structure matrix (see Table 9) shows that Component 1 included low-quality journals, and and Component 2 included high-quality journals. Journals with mixed characteristics loaded on both components"

From Policy to Psychology: Analysing Evaluations of Journal Credibility Through Institutional and Cognitive Lenses

CWTS | Leiden University boosted

Dmitry Kochetkov

@dkochetkov@social.cwts.nl · last week

The paper itself went through 3 rounds of open peer review on MetaROR, with reviews published alongside the article. This transparent process exemplifies the innovations it examines!

View open reviews: https://metaror.org/article/peer-review-at-the-crossroads-3/

#PeerReview #OpenScience #ScholarlyCommunication

CWTS | Leiden University boosted

Dmitry Kochetkov

@dkochetkov@social.cwts.nl · last week

New paper out: "Peer Review at the Crossroads"

I analyzed 4 peer review models from traditional gatekeeping to Publish-Review-Curate. Key finding: no single model is ideal, but PRC shows promise for transparency & accessibility.

Open access: https://doi.org/10.1002/leap.2046

#PeerReview #OpenScience #ScholarlyCommunication

Peer Review at the Crossroads

Dmitry Kochetkov

@dkochetkov@social.cwts.nl · last week

New paper out: "Peer Review at the Crossroads"

I analyzed 4 peer review models from traditional gatekeeping to Publish-Review-Curate. Key finding: no single model is ideal, but PRC shows promise for transparency & accessibility.

Open access: https://doi.org/10.1002/leap.2046

#PeerReview #OpenScience #ScholarlyCommunication

Peer Review at the Crossroads

Dmitry Kochetkov

@dkochetkov@social.cwts.nl · last week

The paper itself went through 3 rounds of open peer review on MetaROR, with reviews published alongside the article. This transparent process exemplifies the innovations it examines!

View open reviews: https://metaror.org/article/peer-review-at-the-crossroads-3/

#PeerReview #OpenScience #ScholarlyCommunication

Dmitry Kochetkov

@dkochetkov@social.cwts.nl · last week

New paper out: "Peer Review at the Crossroads"

I analyzed 4 peer review models from traditional gatekeeping to Publish-Review-Curate. Key finding: no single model is ideal, but PRC shows promise for transparency & accessibility.

Open access: https://doi.org/10.1002/leap.2046

#PeerReview #OpenScience #ScholarlyCommunication

Peer Review at the Crossroads

Quixoticgeek and 2 others boosted

El Duvelle

@elduvelle@neuromatch.social · 2 weeks ago

If a journal asks me to do a peer-review for them, which I accept, then it makes a decision before the review deadline, without waiting for my review, and without letting me know that I do not need to do the review.. while I use my spare time to complete the peer-review by the deadline... then I am not going to do any more peer-review for that journal 🤷

( #PLoSOne in this case 🫤)
#AcademicChatter #PeerReview

El Duvelle

@elduvelle@neuromatch.social · 2 weeks ago

( #PLoSOne in this case 🫤)
#AcademicChatter #PeerReview

Djoerd Hiemstra 🍉 boosted

Ingo Frommholz

@ingo@idf.social · 3 weeks ago

The growing crisis of “AI slop” that endangers scientific publishing.

#FakeScience #InformationOverload #AIEthics #ScientificIntegrity #ResearchQuality #PeerReview #AcademicPublishing #OpenScience #TrustInScience

https://www.theatlantic.com/science/2026/01/ai-slop-science-publishing/685704/

The Atlantic

Science Is Drowning in AI Slop

Peer review has met its match.

Ingo Frommholz

@ingo@idf.social · 3 weeks ago

The growing crisis of “AI slop” that endangers scientific publishing.

#FakeScience #InformationOverload #AIEthics #ScientificIntegrity #ResearchQuality #PeerReview #AcademicPublishing #OpenScience #TrustInScience

https://www.theatlantic.com/science/2026/01/ai-slop-science-publishing/685704/

The Atlantic

Science Is Drowning in AI Slop

Peer review has met its match.

Hilda Bastian

@hildabast@mastodon.online · 3 weeks ago

A lot to think about in my annual roundup of peer review research:

— AI, mostly bad, but some suggestions of ways it may help

— State of evidence on financial incentives for peer review

— Studies on the influence of editors' biases

& more on equity and diversity, & librarians & systematic reviews

https://absolutelymaybe.plos.org/2026/02/02/5-things-we-learned-about-journal-peer-review-in-2025/

#PeerReview #MedLibs

Absolutely Maybe

5 Things We Learned About Journal Peer Review in 2025 - Absolutely Maybe

Back in 2019 I wrote a couple of posts summarizing what we had learned from research about peer review at journals. Since…

Marcel Stimberg and 4 others boosted

JOSS

@joss@fosstodon.org · 2 months ago

We've shipped a major update to the JOSS submission scope requirements, affecting what is eligible for submission and what information we require from authors.

You can read more about the changes and our motivations here: https://blog.joss.theoj.org/2026/01/preparing-joss-for-a-generative-ai-future #joss #opensource #openscience #peerreview

JOSS

@joss@fosstodon.org · 2 months ago

We've shipped a major update to the JOSS submission scope requirements, affecting what is eligible for submission and what information we require from authors.

You can read more about the changes and our motivations here: https://blog.joss.theoj.org/2026/01/preparing-joss-for-a-generative-ai-future #joss #opensource #openscience #peerreview