MediaWiki problems but good news for Toolserver stability: Deployments of MediaWiki 1.21wmf5 cause widespread problems for users across wikis when HTML and CSS updates came temporarily out of sync. On the first wikis targeted for deployment, this was caused by the different cache invalidation rates for HTML (typically one month) and CSS (typically five minutes). The retrospective on the problem highlighted the fact that that the test wiki – the WMF's answer to a production environment that individual developers can no longer practically emulate themselves – actually demonstrated the exact problem that would later manifest itself on production wikis. It went unnoticed.
The mixture of new CSS and old HTML, or old CSS and new HTML (pictured), caused rendering problems across a large number of wikis this week.
Deployments of MediaWiki 1.21wmf5 caused widespread problems for users across wikis when HTML and CSS updates were temporarily out of synchronisation. On the first wikis targeted for deployment, this was caused by the different cache invalidation rates for HTML (typically one month) and CSS (typically five minutes). The retrospective investigation of the problem highlighted the fact that the test wiki – the WMF's answer to a production environment that individual developers can no longer practically emulate themselves – actually demonstrated the exact problem that would later manifest itself on production wikis. It went unnoticed.
Despite attempts to ensure that future deployments did not cause the same problem, similar symptoms were immediately reported after Monday's deployment to the English Wikipedia. The issue was later identified as a repetition of a malfunction of ResourceLoader cache management dating back months (wikitech-l mailing list).
In unrelated news that generated similar reporting, the German Wikipedia (dewiki) was also briefly broken by a botched deployment of version 5 of the Article Feedback Tool. During its first deployment outside of the English Wikipedia, an oversight meant that the database schema changes supposed to precede the deployment had not been carried out, leaving dewiki users unable to access key pages on the site. The problem led to a review of the guidance given to staff attempting deployments, which was found to be out of date in a number of places.
Business as usual for Toolserver as volunteer admin decides to stay
Appropriately, as WMDE look to put the thorny issue of Toolserver twilight funding behind them, the latest issue of the German-language community quarterly Wikimedium focussed on why the chapter encouraged donations.
In the latest episode of a saga dating back nearly two years, German systems administrator DaB. confirmed that he will not follow through on a previous threat to stop devoting his spare time to keeping the Toolserver up and running (toolserver-l mailing list; see also previous extensive Signpost coverage for context).
DaB., like many others involved with the Wikimedia Deutschland (WMDE)-run Toolserver, is concerned that the web tool and bot hosting facility will be switched off, or more likely simply left to die, before its replacement – Wikimedia Labs – has been fully prepared for the transition. If it is, DaB. argues, tool creators will be forced into attempting difficult transitions, a not unlikely scenario if development work to improve Wikimedia Labs overruns. Tools will necessarily be lost in the process, he has repeatedly warned (also toolserver-l).
As reported last week, the issue came to a head when DaB.'s ultimatum attempt to get WMDE to fund new hardware for the Toolserver to ensure its continued availability came before the chapter's AGM. After much deliberation, the chapter settled on a "wait-and-see" policy that aimed to balance DaB.'s concerns with the realities of the WMF-WMDE relationship. That policy centres on a request for a binding roadmap from the Wikimedia Foundation, accompanied by WMDE's own assurance that all "necessary resources" (notwendigen Mittel) will be made available to ensure the "almost uninterrupted functionality" (annähernd störungsfreie Funktionalität) of the Toolserver. WMDE also agreed to bear the brunt of any attempt by the WMF to withdraw their support-in-kind for the Toolserver, currently valued at $65,000/year. Writing after the AGM, DaB. said that he was "disappointed" and would need time to consider whether the amended motion was sufficient to entice him to stay.
On Monday DaB. announced his decision: he would stay for another twelve months, though he remains unconvinced that the need for a functioning Toolserver will cease any time soon. He cited a number of factors for staying, including a need to continue advocating for financial support and his personal pride in the project. Though the path ahead would be "rocky", he wrote, "the Toolserver and you all are worth my time, the Toolserver is still helpful, and there is always the possibility that WMDE will [make the right decisions]." As of time of writing, DaB. and other involved parties were still struggling to get the Toolserver's set of replicated databases back up to date, a process that requires technical assistance from the WMF.
In brief
Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.
MediaWiki 1.20.1, 1.19.3, 1.18.6 and 1.20.2 released: MediaWiki 1.20.1, 1.19.3 and 1.18.6 were released on November 30 as security releases (mediawiki-announce mailing list). A bug which may lead to user account takeover has been fixed in these releases. Five days later, MediaWiki 1.20.2 was released correcting further issues from the 1.20.1 security release (mediawiki-announce mailing list). All represent releases to external sites; Wikimedia wikis had already been patched.
Bugzilla upgraded: Wikimedia's installation of issue tracking software Bugzilla was upgraded to version 4.2.4 from 4.0.9 (wikitech-l mailing list). Benefits include optional HTML emails and improved search. Overall, however, users are expected to witness no significant changes to the user interface.
WMF analyses decentralisation issue: Wikimedia Foundation designers were forced to come to terms with the level of decentralisation among Wikimedia wikis this week when it was revealed that 205 have overridden one or more of the default account creation messages (wikitech-l mailing list). The decision to override, in many cases taken years ago, means that central (WMF-led) improvements to the messages will not naturally propagate outwards to those wikis, the list of which includes the Russian and Hebrew Wikipedias. Nor is it clear what action is appropriate. "These customizations didn't go through the thorough user testing process of the kind that [WMF-led changes do], but they are nevertheless a wonderful source for ideas for improvements that can be useful for any language, and for identifying things that communities need and the software doesn't provide out of the box," stressed WMF Internationalisation specialist Amir Aharoni.
MediaWiki groups: The Wikimedia Foundation will help promote and support small, informal MediaWiki user groups under a draft proposal fleshed out this week. Groups, whether organised around a topic ("MediaWiki Lua Group") or a region ("MediaWiki Bangalore Group"), would be filled with approved MediaWiki "reps", who could then give feedback and discuss important issues. The proposal is a response, if nothing else, to the problems the WMF has faced engaging MediaWiki's end users in its design and development processes. The groups will be backed by merchandise and small financial support, where necessary.
Mike Wang joins Labs team: Florida-based systems administrator Mike Wang has joined the Foundation's Labs team as an engineering consultant. The Chinese-born developer's experience with maintaining Unix systems is likely to provide a useful basis for growing the Labs team's user support capacity as it looks to ramp up to Toolserver migration (see above). As rounded up by WMF Engineering Coordination Manager Sumana Harihareswara this week, the Foundation has at least 24 places left to fill. Of course, those positions also give an idea of what the Foundation's Engineering Department considers its current priorities, namely search, mobile and continuous integration.
Discuss this story
WMDE needs to grow the fuck up and realize that Labs isn't what they thought it was, admit that they've been wrong, and properly fund toolserver. The WMF needs to grow the fuck up, get over the fact that they dislike sharing power with the chapters, and give WMDE some money to help run the toolserver. This kind of childish bullshit is precisely the reason that confidence in the WMF and some of the larger chapters is so low. No one editing on any of the projects, the people doing the actual work that the WMF and WMDE supposedly support, gives a shit about the monumentally large egos of the battling staff members, or the incessant pissing matches that serve only to fuel those egos. What we do care about is that if toolsever fails, projects stop working correctly. Figure out something, pretend to be nice to each other, and don't let this happen again. Everyone that isn't in the WMF or WMDE leadership is tired of it. Sven Manguard Wha? 19:20, 7 December 2012 (UTC)[reply]