Wikipedia Found to Influence Judges' Legal Writing
Research shows that judges in Ireland were influenced by Wikipedia's coverage of relevant case law.
This article originally appeared in Slate on August 17, 2022.
A new study finding that Irish High Court judges were influenced by the existence and specific language of Wikipedia pages seems to have put the Irish judiciary on the defensive. Responding to the paper, a High Court judicial source told the Irish Times that the idea that a Wikipedia article would be relied upon by a judge to support their reasoning for a decision “is clearly wrong.” As one judicial source put it, “Judgments are written based on the legal authorities and submissions put before the court, not on a Google search or Wikipedia.”
Then again, it’s hard to dispute the outcome of the peer-reviewed study, which was published online July 27 as a preprint and is forthcoming in the Cambridge Handbook of Experimental Jurisprudence.
The research team arranged for 154 new Wikipedia articles to be created for Irish Supreme Court cases. After creating the new Wikipedia entries, they found that High Court judicial decisions cited those cases 20 percent more often than those without Wikipedia articles. (Perhaps confusingly, the High Court is the lowest level of “superior” court in Ireland, comparable to a federal district court in the United States.) They also found that the language judges used in their written opinions began to echo the language from the new Wikipedia pages.
At first blush, it might seem that Irish judges need a refresher of what so many of us learned back in high school: that Wikipedia is not itself a primary source for serious research, and that you’ll find yourself in trouble if you copy/paste it to your “paper.” But in fact, there’s a lot more going on here. When people go to court in democratic countries, they expect that the judge’s decision will be governed by the “rule of law,” by authoritative texts such as a criminal statute, a prior judicial decision, or the national constitution. Nobody expects to be governed by the “rule of Wikipedia.” Then again, in an era when Wikipedia is widely considered the gold standard for online information resources, it’s worth asking: Is there any meaningful distinction?
At the start of their experiment in early 2019, there were only nine Irish Supreme Court decisions that had associated Wikipedia articles, making it an ideal data set for their purposes. (For context, the Irish Supreme Court issued 107 judgements in 2021 alone, according to its website.) The authors shared with me a few theories about why there were so few Wiki entries in this space. First, the Wikipedia-editing community in Ireland is relatively small, and there was likely not a huge interest in creating these pages by people outside the legal community. Moreover, Wikipedia has matured to the point that new contributors often feel like their work is more likely to be challenged by experienced editors. It’s possible that some Irish Wikipedia newbies considered making these pages in the past but were reluctant to stick their necks out and go for it.
Until this study, that is. Beginning in early 2019 and continuing through early 2020, law students were assigned to prepare draft Wikipedia articles on 154 Irish Supreme Court decisions. The researchers then sorted the Irish cases into pairs with similar features, such as area of law under review and the year of the decision. After that, they essentially flipped a coin, publishing half of the articles from each pair to Wikipedia (a total of 77) while the other half were held back as a control group. The advantage of this experimental design, the authors said, was to distinguish causation from correlation. “Using the benchmark of empirical proof, a randomized field experiment, this research shows that judges do rely on Wikipedia,” wrote the authors, noting that both the uptick in case citations and the linguistic effect were statistically significant.
Before going further, it’s worth noting the critical role that citations play in common-law legal systems like Ireland, the United Kingdom, and the United States. Instead of jumping to conclusions, common law judges must publicly explain their legal reasoning by referring to decisions from courts that are ranked higher in the hierarchical court structure. Thus, it’s common practice for judges to namecheck Supreme Court cases in their writing. Again, what’s striking here is that the judges began citing the cases with Wiki-summaries more often than the control group of similar cases that lacked Wikipedia pages.
No doubt Wikipedia is affecting Yankee judges, too. More than 3,300 decisions from the U.S. Supreme Court have Wikipedia pages. The Wikipedia article dedicated to the recent Dobbs decision, which overturned the abortion rights precedent set forth in Roe v. Wade, has racked up 1.7 million direct page views and represents the collaborative work product of more than 370 Wikipedia editors. Judges in the United States have also taken to citing Wikipedia itself. Citations to Wikipedia in U.S. judicial opinions first appeared in 2004, and since then, judges have at times leaned on Wikipedia to gauge the popular understanding of a word or phrase. Case in point: A judge in Illinois deferred to Wikipedia’s definition of “crotch rocket” as meaning “a ‘super sport’ or ‘super bike’ capable of great acceleration with a distinctive seat shape.”
Shortly after the judicial experiment was published online in July, a concerned Wikipedia editor posted to the site’s Administrator’s Noticeboard—a place for Wikipedia users to flag concerns for the site’s community-selected leaders—to state that they were “non-plussed about Wikipedia being used this way.” (It’s a rare example of someone using the word nonplussed correctly.) For background, a few past experiments involving Wikipedia have been disastrous, such as the University of Toronto professor who in 2013 told 1,700 of his students to edit and improve Wikipedia articles on topics related to his lectures; experienced editors later found that 85 percent of the resulting changes were plagiarized. Other Wikipedians told me that they worry about so-called breaching experiments—attempts to deliberately place disinformation onto the site to test how quickly it is discovered and removed. This practice is considered unethical in part because of the collateral damage—that is, the innocent Wikipedia readers who would be misled during the trial period.
However, the consensus among Wikipedia editors seems to be that this particular experiment was acceptable in terms of ethics and might even be a model for future research. “We want to improve Wikipedia, we want to protect it,” said co-author Brian McKenzie. “That was our starting point.” The student contributors were trained about Wikipedia’s policies and norms—including guidance on why it might be better to choose a pseudonym that protects your real name. None of the contributors received financial compensation for their articles. Before the draft was published on Wikipedia (and made available to the site’s billions of potential readers), it was first reviewed by an experienced Wikipedia editor who then shepherded it onto the platform. Most importantly, McKenzie told me, the experiment added value to a public information resource in an area of knowledge that had previously been poorly represented. “We’re filling a lacuna here,” said McKenzie. (The paper was jointly authored by Neil C. Thompson of MIT; Brian Flanagan, Edana Richardson, and McKenzie of Maynooth University in Ireland; and Xueyun Luo of Cornell University.)
When I spoke to Rebecca O’Neill, a project coordinator for Wikimedia Community Ireland, she expressed concerns about the editorial angle some publications have used to describe the research and its reception on Twitter. Some people seem to have focused less on the judicial behavior at issue and more on the trustworthiness of Wikipedia, which is ironic considering that the Wikipedia articles created for this experiment are of very high quality.
Back to the judges, then. How might it happen that an Irish judge—or an American judge for that matter—could be influenced by Wikipedia? Although the paper doesn’t offer details from specific cases, it seems worthwhile to consider a plausible hypothetical scenario. Consider an imaginary Irish judge on the High Court. Our judge is ruling on a case involving the propriety of law enforcement handcuffing someone who has been arrested for marijuana. To help with writing her official opinion, the judge (or someone on her staff) Googles some previous Irish Supreme Court decisions involving handcuffs and arrests. Google surfaces an info box for DPP v. Peter Cullen, which features a snippet of text from the case’s Wikipedia’s article. (Pausing here to stress that Wikipedia content affects people indirectly through search even if they never advance to the Wikipedia domain itself).
For purposes of our hypothetical, let’s say that the judge scrolls down to click the first search result, which is the Wikipedia page for the Cullen case. The judge reads the Wikipedia summary of the 2014 case holding that an arrest will be thrown out if it is shown that it was unnecessary to place handcuffs on the accused. The Wikipedia page describes the facts of Cullen’s drunk driving case and the ruling that his handcuffing was improper because he showed no tendency toward violence when he was arrested for the DUI.
Our hypothetical judge is not aware that this Wikipedia article was one of those added during the researcher’s field experiment. Most of the content was written by Lawstudent1999. After skimming the Wikipedia article, the judge in our scenario (a) cites DPP v. Cullen in her opinion as a reason for deciding that the handcuffing of the accused invalidated their arrest, and (b) paraphrases some text from the Wikipedia entry into her own judicial decision, leading to the language overlap.
I wondered whether that linguistic echo from Wikipedia to the judge’s opinion might say something about the general style of legal writing, which is often formulaic and bland. But Thompson, the lead author of the paper, told me that the analysis distinguished the baseline effect (the ordinary dullness of legal writing) versus the incremental effect (the ways that Wikipedia specifically changed the judges’ writing). “The crucial thing here is that we are not just saying there is ‘some’ similarity between these things. We are saying that there is ‘more’ similarity when we have published the Wikipedia articles than for the ones we did not publish,” Thompson said.
Overall, it sure looks like the Irish judges were influenced by Wikipedia, but before you judge the judges, it’s worth considering their working conditions. Remember that the effect from the study was only found in the Irish High Court, which is (again, a little confusingly) lower in the hierarchy than the Court of Appeals and the Supreme Court. According to judicial sources, these judges have very high caseloads and are required to release their opinions under challenging time constraints. Ms. Justice Mary Irvine, the first woman to be appointed president of the High Court, argued in her retirement speech in July that there is a chronic lack of judges on the High Court. Given the lack of resources, it makes intuitive sense that judges and judicial staff at this level would check Wikipedia before dashing off a hastily written opinion.
But there’s still the basic question: Why is it a problem if judges are influenced in some fashion by Wikipedia? Remember that in this experiment the newly created Wikipedia articles were accurate and substantive. What’s wrong with judges using them?
For Flanagan, a law professor at Maynooth and a co-author of the study, the chief concern is Wikipedia’s opacity. “Traditionally, litigants expect that only recognized professional expertise will be used and that it will be reflected in the reasons for a court’s decision,” Flanagan said. “The problem with this is that the judge doesn’t know whether the author of the Wikipedia article has any expertise.” That’s a fundamental distinction between Wikipedia and a legal textbook.
And of course, it’s a much bigger deal if anonymous Wikipedia authors are indirectly affecting someone’s constitutional rights as opposed to recapping their favorite TV series or summarizing other quirky subjects. (By the way, it’s not just law. Earlier research has shown that adding Wikipedia articles on chemistry topics changes how those topics are discussed in the scientific literature.)
Flanagan and his co-authors also pointed to concerns about a “cynical litigant” attempting to manipulate the encyclopedia’s content pre-trial to influence the judge to rule in their favor. Think Inception, but make it Wikipedia-style. When I spoke to Wikipedians about this, they were skeptical. If the litigant was making material changes, this behavior would likely be detected by monitoring tools or by the vigilance of other editors, such as the users in the WikiProject Law workgroup.
Finally, there’s the issue of what’s missing on Wikipedia. What if a hypothetical judge found a case with a Wikipedia article but did not find a more helpful or relevant case that was lacking one? (In case you were concerned, the authors confirmed to me that the articles in the control group would be published later this year.) Of course, knowledge gaps present a recurring challenge for Wikipedia. The project is always evolving; indeed, the appeal for some contributors is that the work of adding knowledge will never be complete. Some Wikipedians told me that rather than gasping about judges using Wikipedia—that is, acting like their fellow humans—the critics should get to work to ensure the site includes the knowledge people need.
To that point, the authors of the new paper conclude that legal professionals should dedicate sustained attention to Wikipedia, developing initiatives to buttress and confirm that Wikipedia content about legal cases is monitored by people with legal training. In the meantime, I keep returning to the remarks from one Wikipedia editor in a discussion page on the site. “For the people of Ireland, this shows their judges are relying on Wikipedia. For the people of Wikipedia, it shows the same thing,” the user wrote. “What we write here can change the world. It’s paramount we get it right.”
My novel, THE EDITORS, is inspired by Wikipedia and will be published August 6, 2024. Pre-order: Barnes & Noble, Bookshop, Amazon.