 Article display preview: | This is a draft of a potential Signpost article, and should not be interpreted as a finished piece. Its content is subject to review by the editorial team and ultimately by JPxG, the editor in chief. Please do not link to this draft as it is unfinished and the URL will change upon publication. If you would like to contribute and are familiar with the requirements of a Signpost article, feel free to be bold in making improvements!
| ||||
A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.
This report[1][supp 1] from the AISciComm project discusses uses and perceptions of AI by Wikipedians in Switzerland. 159 Wikipedians, contacted through a banner and Wikimedia CH's communication channels, answered a questionnaire about their uses and perceptions of AI. This questionnaire was also answered by scientists, journalists, science communicators, science content creators (also referred to as influencers in the report) and regular citizens.
By AI, the report refers to Generative AI and Large Language Models. The tools most commonly used by participating Wikipedians are DeepL Translator and ChatGPT. The results are a fairly short and easy read, but here are the main findings, as presented by the authors and as they struck me (with my bolding).
[T]he survey suggests that Wikipedians in Switzerland are not generally unfamiliar with AI, but notably cautious about using it for their contributions to Wikipedia. While almost 90% of respondents say they use AI outside their Wikipedia work, 63.9% say they never use AI for their Wikipedia contributions. Compared with other surveyed groups, Wikipedians also report much lower AI use. (p.2)
When respondents do use AI for Wikipedia, they mainly use it as a support tool, especially for translation and language improvement. At the same time, they perceive the Wikipedia community as actively engaging with and reflecting on the use of AI [through] community discussions, [and] procedures or guidelines. (p.2)
On the other hand, most respondents did not receive any training on the use of AI (p.6).
The survey also suggests that caution is not due to a lack of competence. Many respondents say AI is easy to use, but they are less convinced that it is useful for Wikipedia. (p.2)
At the same time, respondents expect AI to matter considerably for public communication about science in general. [..] However, they do not see AI as replacing Wikipedia [...] respondents appear to take a cautious, competent, and normatively reflective position toward AI, especially where science-related information is concerned. (p.3)
Figure 1 shows that Wikipedians and scientists are far more cautious than journalists, science communicators and science influencers. This discrepancy may raise concerns about the secondary press coverage of academic research we rely on. That said, I suspect that the study could have painted a different landscape if it asked about using generative AI as a search engine, and that it doesn't account for our technical community due to its focus on content editors. It still offers an interesting window into the norms developed by Wikipedians.
The reasons underlying the development of LLM-related norms is further elucidated in a journal article], "LLMs in Wikipedia: Investigating How LLMs Impact Participation in Knowledge Communities"[2] by Moyan Zhou, Soobin Cho and Loren Terveen at University of Minnesota. They approach the subject qualitatively, through 16 interviews[supp 2], seeking to answer three questions:
They highlight the paradox that LLMs may be a very helpful tool for experienced editors but do a disservice to newcomers:
Experienced editors enhance their participation through LLMs, as they expand the range of their contributions in Wikipedia, leverage strategies to align LLM-generated content with community norms, and thus receive positive responses from other members. In contrast, LLMs raise the demands for new editors to participate in the community. LLMs lower the barriers for entry, and new editors tend to rely on LLMs to fill in their knowledge gaps. However, LLMs compel them to make judgments about AI-generated content, which requires skills they have not yet developed. This challenges them to produce high quality content, leading to rejections from others.
In other words, LLMs encourage new editors to tackle complex writing projects before understanding of the community and its norms. This creates a double frustration for experienced editors who have to edit or correct AI-aided content, and for newcomers who immediately receive negative feedback. This alone might explain why Wikipedians, due to the collaborative nature of their writing, are more cautious about the use of LLMs. However, experienced editors using LLM received more positive feedback about their contributions.
The paper also offers an expanded categorization of use cases for LLMs:
| Generate | Search | Refine |
|---|---|---|
|
|
|
The results also point to a change of role for editors, from content writers to evaluators, verifiers and editors of the content generated by LLMs. It further argues that "the general sensitivity and confusion around LLMs may stem from a lack of consensus on what it means to use LLMs for editing", which aligns with the report reviewed above.
The article concludes with a discussion of the "paradox of participation" uncovered by the study, which should be of interest to those working on new editors retention:
Our findings reveal a paradox of participation: LLMs simultaneously lower barriers to entry while increasing the demands of contributing, especially for newcomers who already struggle to engage with the community.
We discuss this paradox of participation in three interrelated elements:
- LLMs interrupt traditional learning pathways for newcomers that support gradual skill acquisition for newcomers.
- LLMs shift the focus from peripheral tasks to editorial judgment, requiring newcomers to make normative decisions before developing core competencies.
- As a result, LLMs exacerbate a participation divide, enabling experienced editors to thrive while marginalizing newcomers.
These remarks align with the recommendation not to have newcomers create new pages during an editathon, even though it long was a main way of organizing these events. Finally, it also introduces design recommendations for tools aiding new editors, which the researchers have developed and reported on in a different paper[supp 3] we will soon review.
Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.
From the abstract:
...
From the abstract:
...
From the abstract:
...
Discuss this story
(This allows for greater visibility of discussions, makes archiving easier, and prevents discussions becoming disconnected from articles during the publication process)