Wikipedia suffered a major interruption last Monday when the Wikimedia servers lost power due to circuit breakers tripping at their colocation facility. Read-only service was restored after several hours, although it took more than a day for editing to resume, but in the end no significant data was lost.
The servers crashed at about 22:14 (UTC) on Monday, 21 February, due to a loss of power at Neutelligent, the Tampa, Florida company that provides hosting services for the primary set of Wikimedia servers. After some scrambling to determine why network access had been lost, Jimbo Wales contacted Neutelligent and learned that the cause was the tripping of circuit breakers.
In spite of the fact that some of the servers had a second power supply on a separate circuit, this circuit breaker tripped as well, leaving all of the servers entirely without power. Speculative theories suggested that because the tripping mechanism is magnetic, one circuit breaker tripping could also have caused others around it to trip, or that the servers all simultaneously switching over to the new circuit might have overloaded it as well. However, the actual reason for the circuit breakers tripping remains unknown.
Once power was restored, the developers then faced the challenge of bringing the servers back online from an unplanned shutdown. This was complicated by the fact that all of the databases actively replicating at the time of the crash ended up being corrupted. Fortunately, one slave database server had been halted earlier and was storing updates in a log rather than actively applying them to its copy of the database.
This intact copy was used to restore read-only service initially, while the logs were being applied to bring the database back to its current state before the power went out (at most, a few edits in process at the moment of the crash may have been lost). Additional technical details can be found in the report on Meta. Wiki editing was restored at 22:26 (UTC) on Tuesday, 22 February, just over 24 hours after the initial crash.
In order to get back to a full level of service, the database needed to be copied again to several servers, and features such as watchlists and contribution histories remained mostly unavailable over the next day. Performance also stayed rather slow for several days.
The incident had several points in common with the power outage last month that took down another community website with a sizable database, LiveJournal. The LiveJournal servers, hosted by Internap in Seattle, Washington, crashed on 14 January when someone improperly pushed the EPO (Emergency Power Off) button at the facility. Like Wikimedia, LiveJournal did not have its own uninterruptible power supply for the servers as an additional form of backup; whether the local fire code would allow this has not been determined.
LiveJournal, which uses a MySQL database like Wikimedia, needed a similar amount of time to restore service to users after its power outage. They also had to deal with corruption in the databases that had been shut down. Based on LiveJournal's experience, some developers suggested that the data corruption in the active copies of the database might have been due to the RAID hardware giving the database incorrect reports that information had been written to the hard disk when it was actually being stored in cache first.
Brad Fitzpatrick, founder of LiveJournal, expressed his sympathies over the situation as the developers were working to restore service.
Pages related to the ongoing Wikimedia fundraiser (see archived story) were naturally disabled along with everything else. As an emergency measure, a static version of the main fundraising page was temporarily hosted on Angela's website, but the traffic there, with a Slashdot story about the crash also contributing to the load, soon exceeded her bandwidth quota. About eight hours after power was lost, Wikimedia was able to make another backup of the fundraising page available on its own servers.
Wikimedia CFO Daniel Mayer reported that donations were not significantly affected by the downtime. PayPal donations for Monday and Tuesday remained fairly steady at levels somewhat below the initial surge from the start of the fundraiser. Mayer did indicate that donations increased after the Slashdot story was posted, about four hours after the power was cut off.
The Arbitration Committee devised a way to handle violations of the No legal threats policy last week, a type of conduct for which it had not previously needed to construct a specific remedy. New cases also came up as problems over circumcision-related articles continued, this time involving a different editor, and a privately negotiated arrangement with Anthony DiPierro collapsed and went into arbitration.
While threats of legal action had been cited in the original decision against RK, no specific remedy was applied on that point beyond the general four-month ban RK received. In the case of WikiUser, however, the arbitrators had to consider evidence that he had directly stated that he planned on suing Wikipedia.
In June 2004, only a few days after first beginning to edit Wikipedia and getting into a dispute with several users over the England article, WikiUser resorted to a request for arbitration without any of the usual preliminaries for resolving disputes. The request was dismissed, and when WikiUser tried to bring it again, he claimed to be collecting evidence and building his case for a lawsuit. An edit on his talk page also threatened to sue the "Nazis" who own Wikipedia.
Besides all of this, the arbitrators found evidence of personal attacks and determined that WikiUser had significantly disrupted Wikipedia in a variety of ways, including vandalism and edit warring. These problems, rather than the legal threats themselves, led first to a request for comment and then finally the arbitration case. The personal attacks and disruption each earned a one-year ban, with the bans to run concurrently.
After the year was concluded, the ruling provided that the ban for making legal threats would also end, assuming that WikiUser did not make further threats or take other legal action. The ban would have to remain in place if legal action was initiated; arbitrator Sannse made the point that she considered this a protective measure rather than a punitive one. Meanwhile, if WikiUser returns at the end of one year, the decision includes a six-month probation period during which he can be blocked for disruptive behavior.
The ban of Robert the Bruce (arbitration) apparently was not sufficient encouragement for some of the activists on the circumcision articles to improve their behavior, and this time one of the editors on the opposite side of the controversy, Robert Blair, became the subject of an arbitration case requested by Alteripse. The arbitrators voted for a temporary injunction prohibiting Blair from editing articles in the subject area, as they had previously with Robert the Bruce, which was scheduled to go into effect on Monday.
Meanwhile, a dispute that had narrowly avoided arbitration on previous occasions came to a head over Anthony DiPierro's efforts to preserve articles that have gone through the Votes for deletion process. DiPierro was subject to a standing order allowing him to be blocked for making provocative edits, to which he had agreed rather than go to arbitration over a dispute with Raul654. Recently, problems had centered on Votes for undeletion and DiPierro's use of his user space to store deleted articles.
The arbitrators voted to accept the case on Saturday, and DiPierro withdrew from the agreement to the standing order after the case was accepted. In addition to Raul654, four other arbitrators decided to recuse themselves due to various levels of personal involvement in the matter, leaving only six arbitrators to hear the case with mav on a temporary leave of absence from arbitration duties.
The volume level in the ongoing public debate over Wikipedia picked up again considerably last week. Arguments and rebuttals continued to circulate while Wikipedia editors reacted to these discussions by making changes in the encyclopedia as appropriate.
The second issue of the fledgling Free Software Magazine published an article last week entitled "The FUD-based Encyclopedia", written as a response to last year's article from former Britannica editor Robert McHenry, "The Faith-Based Encyclopedia". Author Aaron Krowne, one of the founders of PlanetMath, gave the glowing review, "I cannot remember the last time I accessed a Wikipedia article that was not of apparent professional quality." Krowne set out to rebut McHenry's commentary point by point, and the arguments were further aired in a Slashdot discussion about the article.
An article that was "not of apparent professional quality" was not nearly so hard to find for a blogger who posts under the name Callimachus, however. On a blog called Done With Mirrors, he told of finding the Cantor Fitzgerald article (actually, this is a redirect to Cantor Fitzgerald Securities) based on a Google search, and proceeded to quote it in full as it existed last Friday. At the time, it had three very short paragraphs, one "See also" link, and three external links. His verdict was that "this seems a curiously incomplete full entry for a major company that has been around since 1945." (Neither Britannica nor Encarta has so much as an article on the firm.)
Not only was the article sketchy, Callimachus commented, "It all looks like a lot of conspiracy theory hoo-ha." The article briefly mentioned the company's employee losses in the September 11, 2001 attacks (most of the article's history involved this, as a list of casualties was created, then later removed to the September 11 memorial wiki). It also mentioned an affiliated business called eSpeed, through which Cantor Fitzgerald worked on a wargaming exercise with the US Naval War College, with some rather "clumsy innuendo" suggesting that terrorists might find such an organization a logical target. (The information about eSpeed was added in 2003 as an "ironic fact" by an IP address in the 142.177 range associated with since-banned user EntmootsOfTrolls.)
Instapundit stirred up additional traffic on Friday by linking to Callimachus' post, and later updating it with a link to older criticism from Judith Weiss of Kesher Talk, who has been collecting examples of complaints about Wikipedia. The conspiracy theory material was then removed, but later readers coming from Instapundit still left their commentaries on the talk page.
This followed an earlier posting 20 February on Instapundit calling attention to the nomination for deletion of an article about blogger La Shawn Barber. While Barber said she did not want to be in Wikipedia, other bloggers and Instapundit readers campaigned for her inclusion, although many of the votes on both sides were being questioned. Votes of established users were fairly evenly split on whether to keep the article.
In the middle of these arguments and counterarguments over critiques of Wikipedia, the article Criticism of Wikipedia was suggested for deletion for the second time. The first vote for deletion, which followed the article's initial creation in December, resulted in no action after a majority voted to keep it. The Wikipedia article also has a shorter "Criticisms" section, which now opens with a link to the Criticism of Wikipedia article. This practice is common for lengthy articles that spawn articles on more specific subtopics, and in fact the Wikipedia article is currently about 32 Kb in length.
The idea of merging the "Criticisms" content back into the main article was suggested by several people in the original vote, and Pjacobi mentioned this in opening the second vote for deletion, adding, "death to all POV forks". On the second attempt, the response was overwhelmingly to keep the article, although it was not always clear whether people saw this as an issue with article length or a way to siphon off a seemingly intractable dispute from the main article. Andries called Pjacobi's concept of what constitutes a POV fork "misguided", but Kappa stated a belief that the article was "specifically created to allow the main article to have a pro-wikipedia POV". A lone voice for a different solution came from Raul654, who said it was navelgazing that belonged on Meta instead.
The race for a better search engine was cast in a different light last week, as analyst David Coursey explored the possibilities of "vertical search", and Wikipedia's position in this area could lead to new competition.
In an opinion column last Thursday on eWeek, "Search Engines Succeed at Stoking Frustration", Coursey comments on his increasing difficulties finding useful answers through Google search results. Taking a cue from another reporter's article about vertical search, he suggests, "Sometimes I also think that for common searches I might be better off with a service that actually involves humans in finding answers online, if any of those still exist."
Vertical search is a term used in a recently-released study from JupiterResearch, referring generally to narrower search engines that are focused on specific categories. It relies less on the web crawler approach used by the broad search engines ("horizontal search"), and could offer an alternative to advertisers dealing with the prices of popular keywords at the big search companies. While the concept is not clearly defined, it seems that hyperlinking is one feature that could easily contribute to the "vertical" aspect.
Coursey specifically mentions Wikipedia twice in his analysis, saying that this kind of service "is not too far off the Wikipedia track." Although he doesn't mention the discussions of Google hosting Wikipedia — he might not know about it, or may consider it too speculative to comment on — the negotiations are interesting to consider as a possible alliance between horizontal and vertical search.
In discussing the concept, Coursey considers Wikipedia in tandem with About.com, as if the two would be natural competitors in this field. This view takes the place of the more traditional perspective of Wikipedia as simply an encyclopedia, with the Encyclopædia Britannica as its natural competitor. Worth noting is that while Wikipedia has long ago surpassed Britannica in terms of Alexa traffic rankings, About.com remains ahead of Wikipedia by a decent margin.
Another interesting consideration is the fact that The New York Times announced on 17 February that it would purchase About.com from its previous owner PRIMEDIA, a magazine publishing company, for 410 million dollars (US). Google along with Yahoo!, Ask.com, and AOL were reportedly among the other bidders.
Some reports saw this as the New York Times' strategy to get into the blogging world, characterizing About.com as analogous to a network of 500 bloggers. Doing the math this way, reviews were mixed as to whether the price paid ($820,000 per "blog") was possibly too steep.
The cost of doing business in this area was one issue debated at the recent Harvard conference on blogging and journalism (see archived story), as more traditional media consider how to react to the blogging phenomenon. The conference included a heated discussion involving Jill Abramson, managing editor of the Times, and Dave Winer over the importance of having a media organization with the resources to fund a bureau in Baghdad, as compared to a blog network.
At one point, Jimmy Wales chimed in to mention how Wikipedia had come up with a product that could compete with the Encyclopædia Britannica, with its multimillion-dollar budget, while spending a fraction of the amount. While a simple analogy might tie this to Wikinews as a developing project that can compete with other news organizations, the possibility also exists of news companies expanding in ways that compete with the Wikimedia Foundation's other projects. With the Times' purchase of About.com, Wikipedia may also be moving closer toward head-to-head competition with media organizations.
