 Article display preview: | This is a draft of a potential Signpost article, and should not be interpreted as a finished piece. Its content is subject to review by the editorial team and ultimately by JPxG, the editor in chief. Please do not link to this draft as it is unfinished and the URL will change upon publication. If you would like to contribute and are familiar with the requirements of a Signpost article, feel free to be bold in making improvements!
| ||||
How is Wikipedia doing on its most important articles?
Recently, Looie496 conducted an analysis of how many important FAs are made each year. He concluded that we have produced very few since 2008. In the subsequent discussion, Wehwalt had an interesting mathematical challenge: how often do readers see articles of what quality?
I have collected data manually on page views and looked at cases to analyze quality versus importance. 40+ pages of Excel and 100+ pages of PowerPoint later, the result is here: Wikipedia's poor treatment of its most important articles (PowerPoint, pdf).
We are used to thinking of Wikipedia in terms of numbers of articles, but readers do not look at articles equally often. Answering Wehwalt’s question is not an easy task that one can accomplish by manual methods. This month, Harvard researcher, Andreea Gorbatai, has combined quality and page view statistics from two databases to calculate what percent of eyeballs are hitting the respective categories of article ranked by quality on the assessment scale. She found that only 3% of views are on “high quality” articles (Good articles, A-class articles, and Featured articles). 28% of views are on “medium quality” (C- and B-class articles). 69% of views are on “low quality” (Start-class, Stub-class and unassessed articles). Manual examination of 10 unassessed articles showed they were of Starts or Stub quality, so this group is assigned to low quality on the scale. Essentially, what the average reader sees when he hits Wikipedia is…low quality.
In a pee-reviewed conference paper, presented at WikiSym 2011, "Exploring underproduction in Wikipedia", Gorbatai had drawn attention to major concerns with Wikipedia’s ability to produce high quality important articles. She examined both articles that had a high number of page views (over 50,000 per month) and articles that were subjectively rated important (top/high, by WikiProjects). According to both metrics, only a few percent of "important" articles were assessed as high quality, and there were 10-50 times as many low quality articles as high quality articles.
Looking at a community-compiled list of the most important 1000 Wikipedia articles, the Vital Articles (VAs), we find issues as well. 85% of these super-important articles are below Good article (GA) standard. Even worse, the number of VAs of GA standard or above is slowly dropping over the last 4 years.
Ten years into the otherwise successful Wiki enterprise, we still have not produced high quality articles on our most important topics.
Moving to objective page view analysis, we see some interesting results. Comparing the VAs to Featured articles (FAs), the former have 20 times the median monthly page view: 66,000 views versus 3300. GAs are even lower at 900 median monthly views. The Vital Articles are not just subjectively important for educational or cultural reasons…they are popular!
FAs and GAs are becoming more obscure lately. The 2011 FAs had a median page view of 1000 views per month. GA median for a recent sample was only 600.
Part of the problem is overconcentration on certain topics. 10% of the GA+ articles come from four peculiar categories: tropical cyclones (hurricanes), mushrooms, trains, and US roads. What would an outside scholar think of this? Those four categories together account for only 3 of the 1000 VAs (0.3%). Could these topics be pushed because they are easier to mechanically write award winners on?
Tropical cyclones in particular show a strange pattern: GAs account for 30% of the project’s articles and outnumber B- and C-class articles (very unusual). There is also a higher % of low priority GAs than there are overall in the WikiProject. For Project Tropical cyclones, “low priority” gets high priority!
While having an incredible 640+ GA/FAs, the project still has half of its top/high articles (only 13 total) below GA. This includes such famous storms as Andrew, Camille, and Hugo. And Andrew has 26,000 views per month, while a typical tropical cyclone GA has only 260, less than the average Wikipedia stub. WikiProject Tropical cyclones appears to be a factory for making GA plus signs, not a project to serve encyclopedic readers.
There are also some broad categories that appear to be underserved. There are only 6 automobile FAs, 4 fashion FAs, and 6 aircraft FAs (2 added recently). Consider at the same time that there are 67 FAs on battleships.
Why have we piled up so many obscure FAs and still not achieved the goal that ALoan called out for us in 2006: to bring the most important articles to FA?
| “ | We should aim to do [Feature] all chemical elements, all major solar system objects, all of the plays by Shakespeare, all Roman emperors (well, the Julio-Claudians, at least), all countries, all capital cities, all currencies, the few longest rivers and highest mountains on each continent, all heads of state, all winners of a Nobel or Booker or Pulitzer prize... | ” |
| — ALoan (retired) | ||
The Featured Article program is intriguing. There are top minds there, writing meaty, researched prose that is highly polished. However, there are reasons for concern. Not only is article relevancy (by page views) dropping precipitously, but the production is dropping. And this is happening at the same time that GA is growing as a program. Nor have GA standards gone down from 2007. They have gone up.
Yet at the top end of content production, there are little efforts to recruit new FA writers and reviewers. Even if the standards for writing and reviewing are very high, we still need to find more – either from within Wikipedia or beyond it – or somehow train the ones who are not good enough now. We can't rely on the regulars moving their time around. We need to grow the pie.
FA writers are generally known by their number of FAs. Many show their stars prominently in the right corner of their user pages. And there is a ranking List of Wikipedians by featured article nominations (WBFAN) showing who has the most. Look at discussion in RFA statements, ARBCOM cases, and with resignations. What is noted are the number of FAs, not the import. It is an unconscious mindset that we have. To think in terms of sheer numbers of articles, not the web traffic viewing them or their subjective importance to an encyclopedia (for reference, for education).
The FA star icons don't say how important a particular FA was. They all look the same. What happens if we look article relevance to readers as well as just number of FAs? With that second metric, we can then segment the writers into 4 boxes:
It is interesting to compare the most prominent author of high importance articles at low production rates, Garrondo, with the most prominent author of low importance articles at high production rates, Ucucha. Garrondo has written one FA, Parkinson’s disease in 2011. Ucucha has produced 14 FAs on rare, Latin-named, mammal species. Garrondo has a lousy strategy for climbing up the WBFAN. However, when we look at the impact of the two editors' articles for the readers, there is little question. Because the single Parkinson’s disease article has 180 times the views as Ucucha’s average article, Garrondo achieved 13 times the total contribution to reader-viewed FA content.
The problem is all our systems of rewards, all our tracking systems, all our unconscious assumptions, talk page remarks, and so on simply talk about number of stars…instead of the importance of them. We are incentivising the high production of low importance articles and discouraging the opposite. Yet the latter strategy is the more efficient way to serve the readers.
The Wikimedia Foundation has listed quality as one of its top 5 priorities. However, the target is timid: a 25% increase in the percentage of FA/GAs in five years. We will easily achieve that just based on current production. And their initiatives to support quality appear perfunctory. Furthermore, recent statements suggest an even more passive acceptance of current quality. Yes, other priorities exist. We need to keep the servers turned on, look better on smartphones, and stream video in a format that the civilized world uses.
And undeniably, the drop in participation is a big “oh shit” elephant in the room. The CEO is under board pressure to turn it around. But that is a very difficult, amorphous problem. And…it is really “back office”. Readers care about the actual articles, not our community. And donors give money to WMF because they want a better Wikipedia for readers, not editors.
We have come a long way since 2000, when Jimbo and Larry started Wikipedia. Who could have predicted the incredibly Internet presence it would become, with a nonprofit support structure growing towards a $50 million operating budget? What is the next real differentiating step we can take in our evolution? That step is quality. And the way to most efficiently drive improved quality is by concentrating on high importance articles. The results will resonate with readers and donors and the press.
TCO has addressed criticism and responses to his original report in userspace.
Agreed that there are things the community, the Foundation and the Wikimedia chapters could do to make it easier to write Good and Featured Articles. But the biggest problem with the analysis above is the assumption that we have teams of editors who will exhaustively and competently research a topic, take months to write and vet an article and push it through FAC, and then track the Featured Article for the rest of time ... all so that they'll win a star or a contest prize. As far as I can tell, there's nothing that the Foundation or any Wikipedian has to offer that can mechanistically determine which FAs will be written. The motivators are collaboration, reputation and pride of accomplishment.
I understand the question to be Are important articles underrepresented in highly rated articles?
When a reader wants to know what historic facts exist about Jesus, or the population of China, they simply want to know the information and they'd like it to be true. They are unlikely to not read the article because there is not a gold star on it. WMF's participation strategic goal reiterates "It is a core principle of Wikimedia that 'anyone can edit,' and we want to have a huge and diverse set of contributors." Feature article criteria state "its prose is engaging, even brilliant, and of a professional standard." That's not "anyone," that's a select few. As long as articles are reliable and neutral, readers are being served.
Since TCO has mentioned me on the publication, I feel I have to say something about this matter. While I fully support TCO's views, I understand that Wikipedia is a volunteer-based encyclopedia, made up of (obviously!) generous volunteers who do not ask anything in return. As such, any Wikipedia can contribute constructively to any article of their liking, without getting asked to work on a particular article. Once someone is pressured into writing a VA, the volunteer atmosphere of this Project will be lost. The most that anyone can do to address the problem is to raise awareness of the issue (something TCO has done really well) and hope that some day someone will coming along, pick some vital article and improve it and see it mature into an FA. I believe that to be very unlikely, and that TCO's publication will remain just that, instead of the desired clap which starts an avalanche of vital GAs/FAs getting promoted.
I'm an occasional FAC reviewer and played a small part in the British museum collaboration that took Hoxne Hoard to FA. I'm also a fairly active editor in the broader subject of Quality improvement, amongst other things I initiated the Death anomalies project which has so kindly been supported by Signpost and indeed by Jimbo. I'm also a longterm member of the typo team and have personally eradicated certain typos from the pedia including the Olympic sport of discuss throwing. But I'm not convinced that this report is a good basis to approach our quality issues. We are a volunteer community and we have no deadline. If Wiki-Project Hurricanes and Funghi are further advanced at getting articles to FA than other projects then we should praise the editors who made that happen. Not denigrate them, their subjects and their motivations by describing them as star collectors or other editors as dabblers. Nor is it sensible to discuss the role of FA in improving Wikipedia's quality without considering the improvements that happen even to articles that don't make the grade; The halo effect of improvements to related articles; Or indeed the improved editing of those who've learned things by participating at FAC. As for the recommendation to employ a couple of outside experts to improve Wikipedia editing quality, I can think of few ideas more likely to offend and demotivate our best editors. This report is overly focussed on the concept of less than one percent of our articles that are somehow deemed to be "valued articles" and on the somewhat related metric of recent page views, ignoring the impracticality of agreeing a global 10,000 important articles or the effect of current publicity on page views (One widely cited example being that House is deemed to be a high interest valued article, despite the obvious leakage of searches related to House (TV series) - which is an FA). I can understand the temptation to totally discard this report for those flaws. But there are some potential good ideas in there. One is to offer JStor subscriptions to FA writers, it wouldn't cost the Foundation much to give the FA delegates a bunch of paywall subscriptions that they could dispense as they saw fit. Another is to work with the Wikicup, for example if the cup were to award points for raising a valued article to C, B or A class you might well see a big jump in quality of those articles. There is an awful lot that the WMF and the chapters can do, but I would suggest they would be more effective when working with the grain of the community.
On a broader note, with the vast majority of articles currently below GA standard, a more logical quality strategy would be to leave FA little changed as a source of exemplar articles for the mainpage, and instead concentrate on upgrading a much larger number of articles to a somewhat higher standard than they are now. This could be done through a combination of wikignomes fixing minor errors on huge numbers of articles and interested editors improving the quality of articles that they care about. From my experience this is the strategy that the community appears to actually be following. Ways in which the Foundation and the chapters could help this process include:
