Reviewed by Bri

From the abstract of this paper, presented at the May 2026 ICWSM (International AAAI Conference on Web and Social Media):^[1]

"Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs' capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia's Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors' simpler neutralizations, resulting in high-recall – low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. ... Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult."

With a paper title that harkens back to James C. Scott's Seeing Like a State, one might expect this paper to have something to do with political theory, and it does to an extent – an intriguing hypothesis that norms used in a community can be intentionally applied as constraints for the production of a large language model to good effect. Or, as the authors state:

Large language models (LLMs) are trained on large, broad corpora but then used within smaller communities that have their own norms. To steer models towards specific norms and values, there is a growing trend of stating high-level rules as prompts.
— Introduction to the paper

The experiment, as outlined in the abstract above, centered on "fixing" biased edits, and introduced another bit of word play: "high recall but low precision editing" (emphasis added by reviewer) is (perhaps) a new term, mixing terminology that usually applies to classifier measurements with the act of text generation which large-language models excel at. It emphasized to this reviewer the authors' intent to bring new perspectives to machine learning. In any event, the interpretation of the LLM results was that they sometimes made over-broad changes (perhaps beyond what a skilled human community member would have done with the same constraints).

The authors found that LLMs such as ChatGPT struggle with identifying neutrality violations on Wikipedia: "better than random individuals...but worse than expert editors". (The study, first published in preprint form in 2024,^{[supp 1]} examined the by now rather dated version 3.5 of ChatGPT, released in November 2022, alongside the only somewhat newer GPT-4 model released in 2023, and an open-weights LLM by Mistral AI.) One editing model they experimented with included humans finding violations and AI fixing the violation, with mixed results: the AI tended to do things a human expert would not, including adding new content – perhaps introducing a risk of AI hallucination, or at least authorship and editor agency concerns.

"Comparing AI neutralizations to Wikipedia editor neutralizations" (figures from the paper)

The dryly put statement in the paper "LLMs may reduce editor agency and increase moderation workload" would seem to not exactly align with the level of concern the English Wikipedia community expressed in enacting its recent ban on incorporating unreviewed LLM outputs. This cleanup and moderation workload (including not insignificant detection of undeclared use of LLMs) were key frustrations that were expressed during the highly engaged community discussion. (See related Signpost coverage.)

Also of potential interest to the Wikipedia community a door left open in this finding presented by the authors:

Crowdworkers prefer AI edits over human edits on both fluency and neutrality. We note that participants were not Wikipedia editors. Their judgment may be more representative of Wikipedia readers.

If future decisions to adopt, or not adopt, various forms of AI technology are made by the current community, and not readers or some other authority, could further research address what that AI would look like, in order to meet the community's requirements? There may be very human concerns about community-building at play here, such as making conservative edits to others' writing in order to preserve collegiality and the needs of the community, even when at odds with the "best" presentation of a neutral point of view. How would an AI respond to prompts to make not just correct, but civil edits?

See also the authors' presentation at the June 2025 Wikimedia Research Showcase

• Wikimania 2026 will take place from July 21-25 in Paris, France (see this issue's News and notes for more detail, including about online participation). The program contains various research-themed presentations (on the "Schedule" page, click "Tracks" and select "Research" in the dropdown menu).

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

Compiled by Tilman Bayer

From the abstract:^[2]

"This study examines two decades of user blocking on the English Wikipedia to understand how a volunteer-run, non-profit platform has adapted its content moderation practices in response to increasing visibility amid declining participation. Analyzing more than 20 million block log entries from 2004 to 2024, the study identifies shifts in block frequency, duration, and stated rationales. A significant increase in preemptive, automated blocking of open proxies since 2020 accounts for most block activity, but excluding these reveals a broader trend toward longer blocks and vaguer rationales such as “disruption.” These patterns suggest that volunteers are scaling labor through automation and normative adjustment, trading openness for efficiency and stability.

See also Bluesky thread by the author, and post at The Conversation, covered in this issue's In the media.

From the abstract:^[3]

"International Auxiliary Languages (IALs) are constructed languages designed to facilitate communication among speakers of different native languages while fostering equality, efficiency, and cross-cultural understanding. This study focuses on analyzing the editions of IALs on Wikipedia, including Simple English, Esperanto, Ido, Interlingua, Volapuk, Interlingue, and Novial. We compare them with three natural languages: English, Spanish, and Catalan. Our aim is to establish a basis for the use of IALs in Wikipedia as well as showcase a new methodology for categorizing wikis. We found in total there are 1.3 million articles written in these languages and they gather 15.6 million monthly views. Although this is not a negligible amount of content, in comparison with large natural language projects there is still a big room for improvement. We concluded that IAL editions on Wikipedia are similar to other projects, behaving proportionally to their communities' size. Therefore, the key to their growth is augmenting the amount and quality of the content offered in these languages.

From the abstract:^[4]

"we aim to develop a novel system to detect knowledge manipulations in Wikipedia edits. We incorporate the RWFork dataset, which consists of changes made by the Russian government-backed Wikipedia fork to comply with state-specific laws and narratives. Our methodology includes using advanced multilingual language models, metadata-driven features for modeling, and fairness-aware metrics for evaluation, ensuring the system is robust and transparent. We analyze patterns in RWFork’s content modifications to develop a scoring mechanism for edits with manipulative content identification. This score can be integrated into existing vandalism detection systems or similar applications to improve their accuracy and reliability to Russian state propaganda."

From the abstract of the lead author's related thesis:^[5]

"we build a system that aims to detect knowledge manipulation in Wikipedia content edits. Our approach is based on a detailed analysis of changes made in a Russian government-backed fork of Wikipedia, created to reflect Russian state point of view. We apply our system to Russian and Ukrainian Wikipedia to check their resilience to such manipulative edits. Moreover, we show that the models developed in this work can be effectively applied beyond the Wikipedia setting."

See also our earlier coverage of research involving some of the same authors: "Knowledge manipulation on Ruwiki, the Russian Wikipedia fork"

From the abstract:^[6]

"The article examines the exploitation of the Croatian Wikipedia (Hr.WP) as a platform for political activism and historical distortion, specifically through right-wing administrators manipulating entries. [...] The Hr.WP case exemplifies disinformation not only as content manipulation, but also as process manipulation weaponising neutrality and verifiability policies to suppress dissent and enforce a single ideological position.The research highlights the need for stronger selection, monitoring, and accountability mechanisms for Wikipedia administrators, who, regardless of their volunteer status, must uphold professionalism, neutrality and transparency. It also provides suggestions for pedagogical interventions grounded in CIL.The study offers a novel conceptualisation of disinformation in participatory knowledge systems, revealing how Wikipedia’s governance failures can enable its institutionalisation."

From the abstract:^[7]

"This paper examines Wikipedia’s participatory governance model as a framework for informing European digital public sphere development. Through analysis of Wikipedia’s two-decade experience with community-driven content moderation, reliable source verification, and decentralized decision-making, the study demonstrates how public-interest platforms can maintain information quality while fostering democratic participation. Drawing on Henry Jenkins’ participatory culture theory, the research shows how Wikipedia’s collaborative editing processes naturally develop users’ media literacy competencies through active engagement rather than passive consumption. The paper analyses Wikipedia’s recent regulatory experiences under the EU Digital Services Act and European Media Freedom Act, highlighting both compliance challenges and opportunities for policy learning."

From the abstract:^[8]

"This brief study examines the types of sources cited on German-language Wikipedia pages about German Christmas markets, because ChatGPT-5.2 draws on Wikipedia to check information from other online sources. The analysis covered the German section on the main Wikipedia page on Christmas markets as well as 35 linked pages for individual markets. References listed under literature, web links, and numbered citations were recorded and grouped by source type. The results show that individual market pages rely mainly on newspaper articles and websites run by market organisers or destination marketing bodies, while formal publications, and archival sources are used less frequently."

This study defines an "Epistemic Violence Index" (EVI) based on Wikipedia link graphs. From the abstract:^[9]

"[...] digital content overwhelmingly represents knowledge produced in English and within the majority world, reflecting only a fraction of the knowledge created throughout history across diverse cultures. Epistemic violence remains pervasive in much of the moderated content online, yet its extent is challenging to measure. This paper introduces a novel approach to address this gap by proposing an Epistemic Violence Index applied to Wikipedia biographies of Latin American women scientists and writers. Our study involves constructing a graph representation of the Wikipedia network connections for leading female figures in science and literature from the 19th and 20th centuries. The analysis highlights their connections with influential voices both within the region and in the majority world, evaluating the reciprocity and imbalance of these relationships. By leveraging these graphs, we compute an Epistemic Violence Index based on an intersectional set of variables, including gender identity, socio-economic status, and race, providing an initial step toward quantifying and addressing this persistent issue."

From the paper:

"The EVI is [...] calculated as a weighted average of six distinct measures, each designed to capture specific aspects of network dynamics: visibility disparity, lack of reciprocity, marginalization, lack of influence, exclusion from tightly knit subgroups, and overall lack of connections. These measures are derived from appropriately normalized standard centrality indices, such as degree centrality, betweenness centrality, Eigenvector centrality, and clustering coefficient [...]

Supplementary references and notes: