A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

Reviewed by E_mln_e

This report^{[1][supp 1]} from the AISciComm project discusses uses and perceptions of AI by Wikipedians in Switzerland. 159 Wikipedians, contacted through a banner and Wikimedia CH's communication channels, answered a questionnaire about their uses and perceptions of AI. This questionnaire was also answered by scientists, journalists, science communicators, science content creators (also referred to as influencers in the report) and regular citizens.

By AI, the report refers to Generative AI and Large Language Models. The tools most commonly used by participating Wikipedians are DeepL Translator and ChatGPT. The results are a fairly short and easy read, but here are the main findings, as presented by the authors and as they struck me (with my bolding).

[T]he survey suggests that Wikipedians in Switzerland are not generally unfamiliar with AI, but notably cautious about using it for their contributions to Wikipedia. While almost 90% of respondents say they use AI outside their Wikipedia work, 63.9% say they never use AI for their Wikipedia contributions. Compared with other surveyed groups, Wikipedians also report much lower AI use. (p.2)

When respondents do use AI for Wikipedia, they mainly use it as a support tool, especially for translation and language improvement. At the same time, they perceive the Wikipedia community as actively engaging with and reflecting on the use of AI [through] community discussions, [and] procedures or guidelines. (p.2)

On the other hand, most respondents did not receive any training on the use of AI (p.6).

The survey also suggests that caution is not due to a lack of competence. Many respondents say AI is easy to use, but they are less convinced that it is useful for Wikipedia. (p.2)

At the same time, respondents expect AI to matter considerably for public communication about science in general. [..] However, they do not see AI as replacing Wikipedia [...] respondents appear to take a cautious, competent, and normatively reflective position toward AI, especially where science-related information is concerned. (p.3)

Figure 1 shows that Wikipedians and scientists are far more cautious than journalists, science communicators and science influencers. This discrepancy may raise concerns about the secondary press coverage of academic research we rely on. That said, I suspect that the study could have painted a different landscape if it asked about using generative AI as a search engine, and that it doesn't account for our technical community due to its focus on content editors. It still offers an interesting window into the norms developed by Wikipedians.

The reasons underlying the development of LLM-related norms is further elucidated in a journal article], "LLMs in Wikipedia: Investigating How LLMs Impact Participation in Knowledge Communities"^[2] by Moyan Zhou, Soobin Cho and Loren Terveen at University of Minnesota. They approach the subject qualitatively, through 16 interviews^{[supp 2]}. Editors primarily used ChatGPT. The paper seeks to answer three questions:

1. How does using LLMs influence the ways editors contribute content?
2. What strategies do editors leverage to conform to community norms for LLM-assisted contributions?
3. How do other editors in the community respond and engage with LLM-assisted contributions?

They highlight the paradox that LLMs may be a very helpful tool for experienced editors but do a disservice to newcomers:

Experienced editors enhance their participation through LLMs, as they expand the range of their contributions in Wikipedia, leverage strategies to align LLM-generated content with community norms, and thus receive positive responses from other members. In contrast, LLMs raise the demands for new editors to participate in the community. LLMs lower the barriers for entry, and new editors tend to rely on LLMs to fill in their knowledge gaps. However, LLMs compel them to make judgments about AI-generated content, which requires skills they have not yet developed. This challenges them to produce high quality content, leading to rejections from others.

In other words, LLMs encourage new editors to tackle complex writing projects before understanding of the community and its norms. This creates a double frustration for experienced editors who have to edit or correct AI-aided content, and for newcomers who immediately receive negative feedback. This alone might explain why Wikipedians, due to the collaborative nature of their writing, are more cautious about the use of LLMs. However, experienced editors using LLM received more positive feedback about their contributions.

The paper also offers an expanded categorization of use cases for LLMs:

The results also point to a change of role for editors, from content writers to evaluators, verifiers and editors of the content generated by LLMs. It further argues that "the general sensitivity and confusion around LLMs may stem from a lack of consensus on what it means to use LLMs for editing", which aligns with the report reviewed above. It is also notable that editors in this study do not report using AI to generate images, while some reported that they did in the survey reviewed above.^{[supp 3]}

The article concludes with a discussion of the "paradox of participation" uncovered by the study, which should be of interest to those working on new editors retention:

Our findings reveal a paradox of participation: LLMs simultaneously lower barriers to entry while increasing the demands of contributing, especially for newcomers who already struggle to engage with the community.

We discuss this paradox of participation in three interrelated elements:

• LLMs interrupt traditional learning pathways for newcomers that support gradual skill acquisition for newcomers.
• LLMs shift the focus from peripheral tasks to editorial judgment, requiring newcomers to make normative decisions before developing core competencies.
• As a result, LLMs exacerbate a participation divide, enabling experienced editors to thrive while marginalizing newcomers.

These remarks align with the recommendation not to have newcomers create new pages during an editathon, even though it long was a main way of organizing these events. Finally, it also introduces design recommendations for tools aiding new editors, which the researchers have developed and reported on in a different paper^{[supp 4]} we will soon review.

This 2024 paper^[3] by Kaylea Champion and Benjamin Mako Hill (published in New Media & Society)

"... use[s] a detailed longitudinal dataset from English Wikipedia to show that more experienced contributors—including those who contribute without an account—tend to contribute to underproduced goods. A within-person analysis shows that contributors’ efforts shift toward underproduced goods over time."

The authors are motivated by the question "how well [Wikipedia] meets the needs of its users". They start out by recapping a decade-old paper that had identifed a "Misalignment Between Supply and Demand" of English Wikipedia articles (see also our review at the time: "Popularity does not breed quality (and vice versa)"):

Warncke-Wang et al. (2015) found that over 40% of views to English Wikipedia were to articles that were lower quality than one might expect given their popularity. Countries, religions, LGBT (lesbian, gay, bisexual, and transgender) topics, psychology, pop and rock music, the Internet and technology, comedy, and science fiction were found to be disproportionately affected. [...]

To quantify misalignment, Warncke-Wang et al. (2015) propose that if a system is aligned, goods in the highest demand should be the highest quality while those in the lowest demand should be the lowest quality. When quality is low relative to demand, goods are described as underproduced. When quality is high relative to demand, they are overproduced.

Adapting the approach of that 2015 paper and some of their own earlier work on open source software, the authors construct a quantitative "underproduction factor" for Wikipedia articles. An article's popularity is measured via pageviews, and its quality by using the Wikimedia Foundation's ORES score ("a machine learning-based quality measure that reflects the structural characteristics of articles, such as the presence of references, section headings, and inter-wiki links"). Articles are ranked by popularity and quality (on a monthly level), and an article's "underproduction factor" in a given month is then calculated as

the negative log of the ratio between quality rank and popularity rank. Both ranks were assigned such that high quality and high popularity receive high ranks, while low quality and low popularity receive low ones. A perfectly aligned article—one with the same popularity and quality rank—would have an underproduction factor of 0 since log(1) is 0. An overproduced article of high quality and low popularity would have a negative underproduction factor. An underproduced article of low quality and high popularity would have a positive underproduction factor.

The paper's main research questions ask how editors' experience levels relate to the "underproduction factor" of the articles that they edit. Experience is quantified using an editor's edit count (at the time of editing the article).

Analyzing this data using severel hierarchical regression models, the authors find evidence for two of their hypotheses:

[Suppported:] Hypothesis 1 (H1). Individuals with less experience will contribute to less underproduced goods than individuals with more experience. [...]
[Suppported:] Hypothesis 3A (H3A). An individual will shift toward underproduced goods as they accumulate experience

The analysis also includes contributions made without an account (known as IP editing during the timespan analyzed in the paper: 2007-2021). Champion and Hill have long been vocal defenders of Wikipedia's practice of allowing such edits, e.g. in 2019 as the authors and first signatories of a statement on Meta-wiki compiling arguments in favor of it. Their second hypothesis in this paper would have yielded an additional argument:

[Not supported:] Hypothesis 2 (H2). Contributing without an account is associated with more underproduced goods than contributing with an account.

The paper offers a theoretical motivation for this expectation, observing that "contributing without an account makes exchange of social rewards more difficult" and IP editors might thus be more motivated by public interest instead. However, this hypothesis was not borne out by the data ("Overall, [our] results contradict our proposal in H2"). Likewise, the authors only found "mixed results" for the analogue of H2A for IP editors:

[Not supported:] Hypothesis 3B (H3B). An individual not using an account will shift toward underproduced goods at lower experience levels than one using an account.

In any case, an obvious problem with relying on the number of edits previously made under an IP address as a metric for the editing experience of the person making another edit under that IP is that they might have well have previously edited using different IPs or account. The authors acknowledge this limitation ("some IP-based revisions are likely to be authored by experienced Wikipedians"), but don't offer a convincing defense of their choice in this regard.

In the paper's extensive "Discussion" section, the authors take their quantitative findings as basis for many interesting (even if largely somewhat speculative) remarks and suggestions for future research. For example, they point out a relation to the old debate of whether Wikipedians are "born" or "made":

Our results in H1 and H3A suggest that as they increase in experience, contributors tend to shift their attention to underproduced content. This is consistent with the “made” argument that accumulating experience is associated with editing underproduced articles and, presumably, with increased exposure to socialization. Our results in H2 and H3B call this interpretation into question. Socialization suggests a social process, but contributors who participate without an account have less opportunity to build a social identity in Wikipedia.

As potential answers to the question raised in the paper's title ("Countering underproduction of peer produced goods"), the paper offers that

One approach to addressing underproduction is to increase retention overall. Another approach is nudging contributors toward more severe areas of underproduction earlier: both when they are newer to the platform, and more quickly in response to emerging needs.

As examples for implementing the latter approach, the authors name recommender systems and "Creating interest groups within a larger project" such as WikiProject Women in Red, WikiProject Women Scientists, and WikiProject Vital Articles.

• TKTK
• ...

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

Compiled by Tilman Bayer

From the abstract:

...

From the abstract:

...

From the abstract:

...

Supplementary references and notes: