Feminist critique of Wikipedia's epistemology, Black Americans vastly underrepresented among editors, Wiki Workshop report: Summaries of 26 new research publications
A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.
This paper[1] by Menking and Rosenberg, published in the journal Science, Technology, & Human Values, is a recondite article. Its depth is both a strength (diligent study of the article will likely enhance Wikipedians' understanding of potential problems, such as our assumptions about what constitutes a reliable source or our epistemological presumptions), and a weakness (most Wikipedians will not read it because it is so dense).
I tried (several times) but cannot improve on the authors' summary. Here, then, is an excerpt from the article abstract:
A repository for established facts, Wikipedia is also a social space in which the facts themselves are decided. As a community, Wikipedia is guided by the five pillars—principles that inform and undergird the prevailing epistemic and social norms and practices for Wikipedia participation and contributions. We contend these pillars lend structural support to and help entrench Wikipedia’s gender gap as well as its lack of diversity in both participation and content. In upholding these pillars, Wikipedians may unknowingly undermine otherwise reasonable calls for inclusivity, subsequently reproducing systemic biases. We propose an alternative set of pillars developed through the lens of feminist epistemology ... Our aim is not only to reduce bias, but also to make Wikipedia a more robust, reliable, and transparent site for knowledge production.
Background reading that will enhance understanding of this Menking & Rosenberg (2021) article:
The article has generated some engaging discussions on Wikipedia talk pages, for example:
Last month, the Wikimedia Foundation published the results of its annual "Community Insights", a global survey of 2,500 Wikimedians (including active editors and program leaders) conducted in September/October 2020.[2]
For the first time, the survey asked about race and ethnicity, confined to two countries where such categories are widely used and accepted: the US (195 responses) and the UK (67). Among US contributors, the findings shows striking gaps among Black/African American editors (0.5% compared to 13% among the general population) and American Indian/Alaska Native editors (0.1% vs 0.9%). Hispanic/Latino/a/x editors show a lesser but still large gap (5.2% vs. 18%). White/Caucasian (89% vs. 72%) and especially Asian Americans (8.8% vs. 5.7% among the general population) are over-represented among contributors in the US.
In the UK, the survey similarly found "significant underrepresentation" of Black or Black British editors (0.0% vs. 3.0% in the general population), whereas the percentage of white editors was close to the general population.
A racial or ethnic gap among Wikipedians in the US has long been anecdotally observed or conjectured (see e.g. this 2010 thread which also contained some informed speculation about possible reasons), but this marks the first time that it is backed by empirical survey data, related to the fact that the Wikimedia Foundation's annual surveys are global in nature and there are no internationally accepted definitions of race and ethnicity (or worse, survey questions of this nature would be considered offensive in many countries) [1][2].
Correspondingly, there has been few research about possible reasons for such gaps. An exception is the 2018 paper "The Pipeline of Online Participation Inequalities: The Case of Wikipedia Editing"[supp 1], which we previously reviewed here with a more general focus, but it also contains some insights about reasons why African Americans contribute at a lower rate.
While this study did not find a significant racial disparity among the earlier parts of the pipeline (measuring whether survey respondents had heard of Wikipedia or had visited Wikipedia), when it comes to "know[ing] that Wikipedia can be edited [...] age, gender, and several racial/ethnic identity categories (Black, Hispanic, Other) emerge as salient explanatory factors where they did not before. Income no longer explains the outcome. Education level associates strongly with knowing Wikipedia can be edited." However, racial and ethnic background factors "do not associate with who contributes content" (i.e. the last part of the pipeline). This points to raising awareness of Wikipedia's editability as a potential strategy for reducing these gaps, although this would not address "the importance of education and Internet skills" gaps for closing knowledge gaps that the authors highlight in their overall conclusions.
Conversely, a 2020 paper answered the question "What drives Black contributions to Wikipedia?"[3][4] with the following conclusions, based on a survey of 318 Black Wikipedia editors in the US:
First, black altruism [measured via the survey question "Writing/editing Wikipedia is a way for me to help the Black community," "Sharing my knowledge through Wikipedia improves content on the Black community," and "It is important that Blacks write/edit Wikipedia"] indirectly influences Wikipedians’ content contribution through their perception of information quality regarding Black content in the Wikipedia universe. In other words, the higher their altruistic tendencies, the stronger their perceptions of information quality. [...] We argue that black altruism mediated through perceptions of information quality [...] helps extend black digital culture in the online encyclopedia. Therefore, we suggest one of the gratifications of Black Wikipedia contribution is likely the production and dissemination of Black cultural information content. [...]
Secondly, our findings suggest that Black Wikipedians’ perceived social presence, is also a significant driver of content contribution. That is to say, Wikipedia contribution uplifts self-esteem and feelings of self-enhancement among Black authors and editors. [...]
Lastly our findings, demonstrated that entertainment is not a factor that significantly motivates Black Wikipedians’ content contribution in the current study. We suggest this may be because Wikipedia content contribution, like long-form blogging, requires “more sustained time, editing, maintenance” (Steele, 2018 p. 116), intellectual endeavor, and a higher degree of technological proficiency than other types of social media activities. Nevertheless, our results are a sharp contrast to previous studies which suggest that amusement is an influential factor in Wikipedia contributions [...]"
Survey respondents were recruited in 2017 via Qualtrics "based on predefined characteristics such as individuals who identified as Black/African American, resided in the United States, and had made at least one edit/contribution to Wikipedia's English edition over the last three years". Interestingly, the resulting sample of 318 Black Wikipedia contributors was much larger than that of the WMF Community Insights survey, which (barring some extreme downward adjustments during the weighting process) appears to have consisted of a single Black respondent in the sample, considering the stated percentage of 0.5% among 195 US-based respondents.
The annual WikiWorkshop, part of The Web Conference, took place as an online event on April 14, 2021, featuring the papers listed below. The organizers reported that 78% of attendees were non-native English speakers, 66% first-time attended Wiki Workshop for the first time, 53% were academic researchers and 34% students.
"we explore the creation and collection of references for new Wikipedia articles from an editor’s perspective. We map out the workflow of editors when creating a new article, emphasising on how they select references."
" we hypothesize that administrators’ community governance policy might be influenced by general trust attitudes acquired mostly out of the Wikipedia context. We use a decontextualized online experiment to elicit levels of trust in strangers in a sample of 58 English Wikipedia administrators. We show that low-trusting admins exercise their policing rights significantly more (e.g., block about 81% more users than high trusting types on average). We conclude that efficiency gains might be reaped from the further development of tools aimed at inferring users’ intentions from digital trace data."
"Like most major KBs [knowledge bases, Wikidata is] incomplete and therefore operates under the open-world assumption (OWA) – statements not contained in Wikidata should be assumed to have an unknown truth. The OWA ignores however, that a significant part of interesting knowledge is negative, which cannot be readily expressed in this data model. In this paper, we review the challenges arising from the OWA, as well as some specific attempts Wikidata has made to overcome them. We review a statistical inference method for negative statements, called peer-based inference, and present Wikinegata, a platform that implements this inference over Wikidata. ... Wikinegata is available at https://d5demos.mpi-inf.mpg.de/negation ."
"This paper analyzes the various associating factors throughout this journey including the number of active editors, number of content pages, pageview, etc., along with the connection to outreach activities with these parameters."
"This paper introduces WikiShark (www.wikishark.com) – an online tool that allows researchers to analyze Wikipedia traffic and trends quickly and effectively, by (1) instantly querying pageview traffic data; (2) comparing traffic across articles; (3) surfacing and analyzing trending topics; and (4) easily leveraging findings for use in their own research."
"... we investigate the impact of gradual edits on the re-positioning and organization of the factual information in Wikipedia articles [...] we show that in a Wikipedia article, the crowd is capable of placing the factual information to its correct position, eventually reducing the knowledge gaps. We also show that the majority of information re-arrangement occurs in the initial stages of the article development and gradually decreases in the later stages."
"We are interested in soft (approximate) constraints expressed as dependencies (or logical rules), such as the constraint that “a person cannot be born after one of her children”. Such rules have proven to be useful for error detection [4], adding missing facts [3], executing queries faster, and reasoning [1]. Not only these rules are not stated in Wikidata, but, to the best of our understanding, a way to express them as constraints is still to be defined in the repository ... there are very few rules that are exact, i.e., true for each and every case. As an example, consider a rule stating that “a country has always one capital”. This is true for most countries, but there are 15 countries that have two or more capitals. Therefore, the rule has a very high confidence, but it is not exact ... The goal of our work is to create a large collection of rules for Wikidata with their confidence measure. In this abstract, we report on two directions we have been exploring to obtain such rules, our results, and how we believe the Wikimedia community could benefit from this effort."
"we highlight the possibilities of taking advantage of structured data from Wikidata for evaluating new biographical articles, so facilitating users to get engaged into diversity challenges or track potential vandalism and errors"
Related code: https://github.com/toniher/wikidata-pylisting
"we present a novel approach based on the structural analysis of Wikigraph to automate the estimation of the quality of Wikipedia articles. We examine the network built using the complete set of English Wikipedia articles and identify the variation of network signatures of the articles with respect to their quality. Our study shows that these signatures are useful for estimating the quality grades of un-assessed articles with an accuracy surpassing the existing approaches in this direction."
"This work [i.e. research proposal] describes a project towards achieving the next generation of models, that can deal with open-domain media, and learn visio-linguistic representations that reflect data’s context, by jointly reasoning over media, a domain knowledge-graph and temporal context. This ambition will be leveraged by a Wikimedia data framework, comprised by comprehensive and high-quality data, covering a wide range of social, cultural, political and other type of events"
"we propose a language-agnostic approach based on the links in an article for classifying articles into a taxonomy of topics that can be easily applied to (almost) any language and article on Wikipedia. We show that it matches the performance of a language-dependent approach while being simpler and having much greater coverage."
See also: online demo, data dumps, model details
"We evaluate the quality and time-savings of AI-generated formula and identifier annotation recommendations on a test selection of Wikipedia articles from the physics domain. Moreover, we evaluate the community acceptance of Wikipedia formula entity links and Wikidata item creation and population to ground the formula semantics. Our evaluation shows that the AI guidance was able to significantly speed up the annotation process by a factor of 1.4 for formulae and 2.4 for identifiers. Our contributions were accepted in 88% of the edited Wikipedia articles and 67% of the Wikidata items. The >>AnnoMathTeX<< annotation recommender system is hosted by Wikimedia at annomathtex.wmflabs.org. In the future, our data refinement pipeline will be integrated seamlessly into the Wikimedia user interfaces."
"Wikidata recently supported entity schemas based on shape expressions (ShEx). They play an important role in the validation of items belonging to a multitude of domains on Wikidata. [...] In this article, ShExStatements is presented with the goal of simplifying writing the shape expressions for Wikidata."
"we investigate the state-of-the-art of machine learning models to infer sociodemographic attributes of Wikipedia editors based on their public profile pages and corresponding implications for editor privacy. [...] In comparative evaluations of different machine learning models, we show that the highest prediction accuracy can be obtained for the attribute gender, with precision values of 82% to 91% for women and men respectively, as well as an averaged F1-score of 0.78. For other attributes like age group, education, and religion, the utilized classifiers exhibit F1-scores in the range of 0.32 to 0.74, depending on the model class."
"We managed to align more than 37,000 articles across Wikipedia and Conservapedia; of these, about 28,000 pages share an identical title, while the remaining ones are aligned based on redirect pages. In total, the whole corpus contains 106 million tokens and 558,000 unique words. [...] We can notice marked differences in word usage in the two resources: Wikipedia authors tend to use more objective/neutral words (affordable care, american politician), in addition to many non-political terms. In Conservapedia [there] prevail derogatory terms such as rino, which stands for “Republican In Name Only", and Democrat Party, but also topics of high concern to the conservative community such as the homosexual agenda, communist manifesto, and fetal tissue."
"we study the content editor and viewer patterns on the COVID-19 related documents on Wikipedia using a near-complete dataset gathered of 11 languages over 238 days in 2020. Based on the analysis of the daily access and edit logs on the identified Wikipedia pages, we discuss how the regional and cultural closeness factors affect information demand and supply."
"As part of our research vision to develop resilient bias detection models that can self-adapt over time, we present in this paper our initial investigation of the potential of a cross-domain transfer learning approach to improve Wikipedia bias detection. The ultimate goal is to future-proof Wikipedia in the face of dynamic, evolving kinds of linguistic bias and adversarial manipulations intended to evade NPOV issues."
"[Wikipedia article deletion] decisions (which are known as “Article for Deletion”, or AfD) are taken by groups of editors in a deliberative fashion, and are known for displaying a number of common biases associated to group decision making. Here, we present an analysis of 1,967,768 AfD discussions between 2005 and 2018. We perform a signed network analysis to capture the dynamics of agreement and disagreement among editors. We measure the preference of each editor for voting toward either inclusion or deletion. We further describe the evolution of individual editors and their voting preferences over time, finding four major opinion groups. Finally, we develop a predictive model of discussion outcomes based on latent factors."
Among the findings are that "Editors who joined before 2007 tend to overwhelmingly belong to the more central parts of the network" and that "user preferences [for keep or delete] are relatively stable over time for ... more central editors. However, despite the overall stability of trajectories, we also observe a substantial narrowing of opinions in the early period of an AfD reviewer tenure. ... Strong deletionists exhibit the least amount of change, suggesting the possibility of lower susceptibility, or higher resistance, to opinion change in this group." Overall though, the authors conclude that "differences between inclusionists and deletionists are more nuanced than previously thought."
" we use general and health-specific features from Wikipedia articles to propose health-specific metrics. We evaluate these metrics using a set of Wikipedia articles previously assessed by WikiProject Medicine. We conclude that it is possible to combine generic and specific metrics to determine health-related content’s information quality. These metrics are computed automatically and can be used by curators to identify quality issues."
.png){kind=spacer}
"... we present an approach to characterize Wikipedia’s editor drop-off as the transitional states from activity to inactivity. Our approach is based on the data that can be collected or inferred about editors’ activity within the project, namely their contributions to encyclopedic articles, discussions with other editors, and overall participation. Along with the characterization, we want to advance three main hypotheses, derived from the state of the art in the literature and the documentation produced by the community, to understand which interaction patterns may anticipate editors leaving Wikipedia: 1) abrupt interactions or conflict with other editors, 2) excess in the number and spread of interactions, and 3) a lack of interactions with editors with similar characteristics."
The paper is part of an ongoing research project funded by a €83,400 project grant from the Wikimedia Foundation. Some related code can be found at https://github.com/WikiCommunityHealth/ .
Besides presentations about the papers listed above, the Wiki Workshop event also saw the announcement of the first "Wikimedia Foundation Research Award of the Year" ("WMF-RAY", cf. call for nominations), with the following two awardees:
"Content Growth and Attention Contagion in Information Networks: Addressing Information Poverty on Wikipedia"[25] (also presented at last year's Wikiworkshop), a paper which according to the laudators
"demonstrates causal evidence of the relationship between increases in content quality in English Wikipedia articles and subsequent increases in attention. The researchers conduct a natural experiment using edits done on English Wikipedia via the Wiki Education Foundation program. The paper shows that English Wikipedia articles that were improved by students in the program gained more viewers than a group of otherwise similar articles. It also found that this effect spills over into a range of articles linked to from the improved articles."
"Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages"[26] and Masakhane (which describes itself as "A grassroots NLP community for Africa, by Africans"). The paper
describes a novel approach for participatory research around machine translation for African languages. The authors show how this approach can overcome the challenges these languages face to join the Web and some of the technologies other languages benefit from today."
While the research does not seem to have concerned Wikipedia directly, the laudators find it an "inspiring example of work towards Knowledge Equity, one of the two main pillars of the 2030 Wikimedia Movement Strategy" and expect the project's success
"will directly support a range of Wikimedia Foundation and Wikimedia Movement goals including the newly-announced Abstract Wikipedia which will rely heavily on machine translation too."
Consistent with its title, the paper features an impressive list of no less than 48 authors (with the cited eprint having been submitted to arXiv by Julia Kreutzer of Google Research).
(but still available via archive.org)
Discuss this story
Initial discussion
These papers would be more convincing if they were written in something resembling the English language. "An alternative set of pillars developed through the lens of feminist epistemology" is about as meaningful as "Colorless green ideas sleep furiously". The Blade of the Northern Lights (話して下さい) 21:25, 27 June 2021 (UTC)[reply]
{{reply to|Chess}}on reply) 21:37, 27 June 2021 (UTC)[reply]{{reply to|Chess}}on reply) 22:50, 27 June 2021 (UTC)[reply]{{reply to|Chess}}on reply) 00:59, 29 June 2021 (UTC)[reply]I'm not a huge fan of the conclusions in the paper about why black people participate in Wikipedia less. Considering that they participate at, what, 1/20th of the rate of white people, we could say that we're only engaging 1/20th of the black people who would potentially be interested in editing. Of the ones we do engage, they're interested in black altruism - but what about the ones we don't? I think that's the more important question. The fact that the ones we do engage tend to not cite "entertainment" as a reason is somewhat interesting I think - perhaps we should look at that among non-editors? Since I'd assume there is a large base of black people who would potentially find Wikipedia fun to edit, yet for some reason have avoided it. I do appreciate the effort to look more into demographics, I'm only concerned about a form of survivorship bias interfering with getting useful information. Elli (talk | contribs) 22:37, 27 June 2021 (UTC)[reply]
Yeah this reeks of bad conclusions based on poor evidence. Maybe consider systemic and socioeconomic factors before torpedoing the 5 Pillars and how wiki has functioned decently for a long time. ~Gwennie🐈｟💬 📋｠ 22:41, 27 June 2021 (UTC)[reply]
WhatamIdoing's discernment
In the talk page discussion I referenced in my review—WP:5P sidetrack (part II)—WhatamIdoing wrote (on 21 Apr 2021 @ 03:23 UTC): The discussion continues after this, with additional insightful posts by other smart Wikipedians. Mark D Worthen PsyD (talk) [he/his/him] 14:46, 29 June 2021 (UTC)[reply]What if they were right?
I'm inclined to reject the argument, but I think it is good intellectual discipline to seriously consider the possibility that they are basically right about the 5P being responsible for the diversity gap in the editor population. I think, though, even if they were right, it would be a mistake to attempt a radical reengineering of core principles: to use nautical terms, the Wikipedia community, at least for en.WP, is the analog of an oil tanker with a turning circle that takes hours to execute. I doubt the enterprise could survive such an effort and remain fruitful. If we did want to remake WP on new principles, I think it could only work in the context of a new project. — Charles Stewart (talk) 17:24, 1 July 2021 (UTC)[reply]
{{reply to|Chess}}on reply) 01:11, 2 July 2021 (UTC)[reply]So what, if knowers are situated? Why can't knowers reflect on what they are doing and put their personal beliefs aside when writing an encyclopedia? Is it so hard to just describe the debates rather than engaging in the debates themselves? Is it really unrealistic to expect that editors can look after each others' edits on controversial articles and make sure that only description of the debates are being done? We aren't trying to write using a view from nowhere (as Thomas Nagel would put it). What we are doing is presenting from every angles proportionally to the weight they have on the composition of views held by expert researchers. Also, feminist epistemology is originally concerned with researchers, not with the summarizers (i.e. encyclopedia writers) of the findings of those researchers. So the concepts in that area aren't automatically applicable to Wikipedia. So why are the authors referring to 'the truth' when Wikipedia doesn't lead, it only follows?
VarunSoon (talk) 03:39, 2 July 2021 (UTC)[reply]
{{reply to|Chess}}on reply) 23:37, 16 July 2021 (UTC)[reply]{{reply to|Chess}}on reply) 00:52, 17 July 2021 (UTC)[reply]Survey percentages
I don't understand how if there were 195 respondents reporting their race/ethnicity in the US, First Nation people can make up 0.1%. Even if there was just one such person in the sample, that would be 0.5%. What am I missing? --Andreas JN466 09:00, 29 June 2021 (UTC)[reply]
Publically available version of Menking and Rosenburg paper
The article links to a paywalled version of the paper, which is available OA at https://journals.sagepub.com/doi/pdf/10.1177/0162243920924783?casa_token=EfdSjisfZf8AAAAA:EB-0LLFClccB0CVNc8io5W46u4DoBWAx9gX-bBDf3PHbsRq3xDMbs1Fh_uePmIJ4RpxXh1WGZg9j
The link should be updated. — Charles Stewart (talk) 10:22, 1 July 2021 (UTC)[reply]
What inevitably mathematically dominated the study
With our sports SNG "did it for a living for one day" criteria to bypass GNG, we have an immense amount of articles (many permastubs) in this numerically male dominated (and even more so collectively over history) field which heavily influence overall numbers in such studies. I hit "random article" a few hundred times and 43% of ALL of the articles about men were about sports figures. This mathematically dwarfs any other category, with politicians being a distant second at 11%. So sports figures would have mathematically dominated that study. North8000 (talk) 00:05, 16 July 2021 (UTC)[reply]
{{reply to|Chess}}on reply) 01:20, 17 July 2021 (UTC)[reply]This is premature but I wanted to post something. I did a more careful sample (so far 200 articles) Of the articles about individual people (59) , I divided them into recent (active in the last 15 years) and not recent. Here was the breakdown of articles on individual people:
North8000 (talk) 21:12, 16 July 2021 (UTC)[reply]