The emerging formalist-pragmatist divide on the Supreme Court (Biestek v. Berryhill forecast)

Unheralded case may be emblematic of the Supreme Court’s 2018 Term

Based on the ebb and flow of oral argument in Biestek v. Berryhill, we predict a formalist-pragmatist division between the justices, rather than one based upon the usual liberal-conservative fault lines.

Biestek was a relatively unheralded and unremarkable case about administrative procedure and the denial of disability benefits. The hearing generated very little press coverage and may have even seemed a little dull to spectators without a keen interest in the outcome or in administrative law. However, for this very reason, Biestek may end up being emblematic of the 2018 Term, a Term in which the Court appears to be trying to stay out of the limelight and avoid the impression that every case is a pre-determined political contest.

The results of our predictive model based on the oral argument transcript are shown in the figure below. The model captures the formalist-pragmatist division as reflected on the opposing views of conservative Justice Gorsuch and equally conservative Justice Alito.

Supreme Court Oral Argument Prediction for Biestek v. Berryhill (December 4, 2018)

However, the model does not capture every nuance of the argument.

“I have evidence, but it’s a secret”

Some additional background helps explain the issues. Biestek applied to the Social Security Administration for Supplemental Social Security Income and benefits in 2010. In a hearing to review the denial of his claim, a vocational expert testified that there were jobs available to Biestek in spite of his disability. The vocational expert based her testimony on private data that she refused to produce. Nonetheless, based on this secret evidence, the Administrative Law Judge (“ALJ”) found there was work available to Biestek and denied his claim.

The Sixth Circuit affirmed on the basis that the strict requirements of the Federal Rules of Evidence do not apply to Social Security disability proceedings. Petitioner acknowledges that the Federal Rules do not apply, but contends that the ALJ’s decision failed the substantial evidence threshold. Thus, the essential question in the case is whether a man should be denied disability benefits when the expert evidence against him is based solely on data that the expert declines to reveal.

Or, as Justice Gorsuch put it: 

If we were in federal district court [and] if on the key issue in the case the evidence depended upon the testimony of an expert, and the expert said, ah, I’m not going to give you my underlying data, it’s secret, I don’t think we would hesitate to find that no rational jury could sustain a verdict in favor of the party propounding that expert. Why isn’t the same true here?

The arguments against the Petitioner are largely pragmatic. As Anthony Yang explained on behalf of the government:

Each year, there are about 2.6 million initial disability claims that are filed with SSA, and at the third level of review, the SSA conducts 670,000 hearings. That’s about 200 — 2500 a day. Over 1 million people are waiting for just a response for their hearing, and they wait, on average, about 605 days. There is no adjudicatory process on a scale comparable to this.

The Chief Justice and Justice Alito did not seem convinced that the secret basis of the vocational expert’s opinion was any great problem. Justice Breyer, on the other hand, had clear reservations about the reliability of the testimony of vocational experts but thought that this issue would be better addressed by a test case of some kind.  


The case is hard to predict. Justices Gorsuch, Ginsburg, and Sotomayor seemed clearly inclined to the view that secret evidence was no basis for an administrative decision. However, Justice Kagan raised telling objections on both sides of the argument and Justice Kavanaugh’s limited comments to the Respondent did not contribute much.  


Petitioner: Gorsuch, Ginsburg, Sotomayor, Kavanaugh, Kagan

Respondent: Alito, Roberts, Breyer, Thomas

Most likely to switch: Kavanaugh, Kagan, Breyer

Timbs forecast: Many roads to incorporation

In Timbs v. Indiana, the Court saw a particularly lopsided oral argument in favor of Petitioner, who argued that the  Eighth Amendment’s excessive fines clause is incorporated against the states under the Due Process Clause of the Fourteenth Amendment (or alternatively, the Privileges and Immunities Clause).

The Supreme Court continues to ignore pressing social issues

There are many reasons why Petitioner Tyson Timbs is a sympathetic litigant, and why his constitutional argument is persuasive. He is the prototypical victim of the opioid epidemic: having become addicted to hydrocodone for foot pain, he took to dealing small amounts of heroin to fund his habit. He used his one considerable asset, a $42,000 SUV bought with money inherited from his father’s $73,000 estate, to drive to the sites of two deals of value $225 and $160. Timbs was convicted and put under house arrest, and subject to fines totaling less than $1000. But the state of Indiana seized his $42,000 SUV as an instrumentality of the crime, subject to in rem forfeiture.

According to Indiana, there is no meaningful limit to stop police directly profiting from seizing cars used in drug crimes, or even cars going 3 miles over the speed limit. According to Timbs, the constitutional guarantee against excessive fines should be incorporated against the states because freedom from excessive fines is “fundamental to our scheme of ordered liberty.” This constitutional protection is vital with respect to fines because, unlike other forms of punishment, fines are particularly prone to abuse as a potential profit center for the state. Furthermore, as Timbs’ brief argues persuasively, such forfeitures have been used for centuries to oppress the poor, perpetuate Jim Crow long after the Civil War, and give problematic police departments, such as that in Ferguson, Missouri, a mechanism of harassing racial minorities.

Yet, as is so often the case in criminal procedure cases, few of these compelling arguments appeared to shape the justices’ inquiries in the case. Only Justice Sotomayor made passing reference to these pressing social concerns.

The real point of interest in the oral argument washow justices from various points on the ideological spectrum seemed to be converging on a decision in favor of Petitioner while maintaining starkly different motivations and reasoning. It was particularly fascinating to see that the differences among the conservative justices and among the liberal justices were more sharply drawn than the distance between the conservative and liberal camps.

A variety of ways of getting to a new state constitutional right

Among the conservatives, Justice Gorsuch and Justice Kavanaugh seemed to be competing as to who would come out swinging most against the state. Kavanaugh repeatedly asked a version of the question “aren’t all –all the Bill of Rights at this point . . . incorporated?” Gorsuch clearly won,however, immediately jumping into a 138 word monologue, demanding that Respondent advocate “at least agree” that the Excessive Fines Clause is incorporated against the states, and subsequently goading the advocate, asking if he “really wants” the Court to address a secondary question.

In contrast, Chief Justice Roberts and Justice Alito pushed hard against Petitioner’s argument, questioning whether any fine can be excessive in comparison to prison time. The figure below reflects this division, with Roberts and Alito favoring Respondent. However, there is reason to think that they may not vote that way: for institutional reasons, the Chief is unlikely to want to be on the losing side of a case recognizing a constitutional right. Also, by the end of the argument even Alito seemed troubled by the implication of Respondent’s position that individuals pursuing Second Amendment rights would have to establish each aspect of the right bothat the federal and state level.

ScotusOA Predictive Model for Timbs v. Indiana

Model predictions for Timbs versus Indiana showing votes likely in favor of Petitioner and respondent.  Ranging from Gorsuch 95% Petitioner to Roberts 93% Respondent.
Supreme Court Oral Argument Prediction for Timbs v. Indiana (November 28, 2018)

There was also division among the liberal justices,but over the reasons for and extent of support for Petitioner. As mentioned,Sotomayor did consider the practical reality of civil and criminal forfeiture;Justice Kagan, in contrast, repeatedly raised a question of great interest only to constitutional scholars: at what level of generality should a constitutional right be recognized? Justice Breyer raised strong institutional concerns with Petitioner’s position,particularly regarding the role of stare decisis and the logical conundrum of protecting against excessive fines given the lack of a meaningful proportionality requirement in sentencing. In contrast, Justice Ginsburg so clearly came to the aid of Petitioner’s advocate, Wesley Hottot, that he subsequently committed the faux pas of referring to her as having “assisted”him earlier. Based solely on the oral argument, one might think Breyer was a vote for Respondent, but although he was clearly weighing some broader institutional issues, our reading of Breyer’s constitutional theory leads us to the opposite conclusion. Breyer’s congenial verbosity makes him harder to predict than many of the other justices.

Unanimous and lopsided votes are difficult to predict

Thus, while the numbers show a 5:3 split in favor of Petitioner, we expect that the case will be much more lopsided, perhaps even unanimous, since even ultra-conservative, ever-silent Justice Thomas called for the Court to hear the issue. Very lopsided and especially unanimous cases are harder to predict on the numbers from oral argument: even if the justices all expect to vote in favor of one side, we would still expect them to talk during that advocate’s time, since modern norms of Supreme Court oral argument involve a lot of justice speech. This makes the numbers less reliable in lopsided cases, showing the importance of both qualitative and quantitative analysis. We do, however think that the figure reflects a real division between the justices, particularly between the Chief and Alito versus the rest of the Court. This division just may not be reflected in the final votes. We expect that either Roberts and/or Alito will vote for Petitioner on the question presented but  write a narrowing concurrence.

Prediction: 8:1 or 9:0 for Petitioner, Timbs

Most likely to switch: Alito

Visualizing the content of justices’ and advocates’ cross-references

In our previous two posts (this one and this one), we examined the implications of justices cross-referencing other justices at oral argument. In this post, we take the analysis a step further and focus on the content of those cross-references.

The figure below is a word cloud derived from every sentence in the Supreme Court oral argument transcripts from the 1985 to the 2017 Terms in which one justice refers to another sitting justice by name. (We also treated references to “the chief” and “the Chief Justice” as a reference to Rehnquist or Roberts, depending on the era.) The word cloud provides a visualization of the relative importance of the 100 most frequent words in that set, excluding common words such as and, the, it, to, …

Supreme Court Oral Argument, 1985 – 2017 Terms: Justice cross-reference word cloud 

Word clouds are the text mining equivalent of a Rorschach test. The frequency of each individual word in a “bag of words” is of course an objective fact; but the significance each viewer gives to that fact depends on his or her expectations. Our interpretation of the word cloud above is that when justices are referring back to other justices in the modern era, they are overwhelmingly attempting to go back to a specific question raised by that justice. “Question” is the dominant token in this word cloud, occurring more than twice as often as the next two tokens, “answer” and “think.” Likewise, “answer” suggests a reference is being made to an earlier question; “think” is more generic.

Supreme Court Oral Argument, 1985 – 2017 Terms: Advocate-justice cross-reference word cloud 

Looking at the same visualization for advocates in the above word cloud, we see questions are still important, but not so clearly dominant. In the advocate word cloud, “question” is only the third ranked token, after “think” and “court.”

Previously, we have shown that the modern era of oral argument is defined more by comments than by questions; we have also shown that questions are more often asked of one’s friends, and comments are put to one’s foes. Finally, we have shown that judicial cross-references are generally made between two justices who ultimately agree with one another in the case at hand. As such, it follows logically that justices’ cross-references tend to focus substantially on fellow justices’ questions. This content analysis further buttresses this prior evidence of the relationship between questions and agreement, comments and disagreement, and judicial cross-references and agreement.

Cross-referencing one’s friends: A universal pattern among the justices

In our previous post, we showed that when one justice refers to one another by name at oral argument, it usually signals agreement. We also showed the practice of justice name checking each other has increased over time. The next natural question is whether this strategy is one utilized by all of the justices or only a select few. We might expect that there would be significant differences amongst the justices, since previously we observed significant variation in naming patterns among the justices when referring advocates. Do the same patterns emerge when justices use each other’s names?

Which justices cross-reference the most?

To explore these questions, the following table shows the cross-reference tendencies of a selection of justices serving from 1985 to 2017.

Speaker Makes Reference Is Referenced Agreement Ratio
Alito 0.48 1.13 2.60
Breyer 0.78 2.86 2.64
Ginsburg 0.77 1.69 2.77
Gorsuch 2.13 0.05 1.55
Kagan 1.47 0.90 3.46
Kennedy 2.05 1.30 2.71
O’Connor 0.29 2.22 4.05
Rehnquist 0.82 1.04 3.05
Roberts 1.04 1.12 3.22
Scalia 0.57 2.51 2.43
Sotomayor 1.62 0.67 3.41
Souter 1.47 0.53 3.02
Stevens 0.88 1.54 1.62
Thomas 0.76 0.02 3.00

The first column of the table above shows the rate per thousand words by which each justice references another justice. The results are very different for which justices cross-reference their colleagues than the advocates. Justices Gorsuch, Kennedy, Sotomayor, Souter, Kagan, and Chief Justice Roberts lead the pack in naming their brethren more than once per thousand words. In contrast, when using the names of advocates, Kagan, Rehnquist, Ginsburg, Scalia, and O’Connor make up the top 5. Only Kagan is on both top referencer lists. Kennedy, second only to Gorsuch in judicial cross-references, was barely discernible when measuring the use of advocates’ names. Clearly, then, using names serves a very different function when the justices are referring to colleagues than when referring to advocates.

Which justices are referenced most often?

The second column shows the tendency of each justice to be referred to by another justice. Some interesting patterns emerge here, too. First, Breyer and Scalia are way ahead of the others on this dimension. This could be because, as one of us (Jacobi) has argued elsewhere, their ideological and methodological distance led to such intense disagreements that their mutual interruptions dwarfed any others between any judicial pair. If that is what is driving these two justices’ high rates of cross-references, it would suggest that sometimes cross-references do indicate disagreement. However, it is also possible that these two justices constitute the contrasting personifications of not only liberal and conservative ideology but also of their methodologies, since Breyer and Scalia have each written influential books on their very contrasting constitutional views.

In fact, the data supports both of these interpretations. With 235 cross-references, Breyer referenced Scalia far more than he referenced any other justice—other conservatives, such as Alito and Roberts, were referenced by Breyer only 77 and 26 times, respectively (although they served with Breyer for less time). Similarly, Scalia most often referenced Breyer, with 182 references; in contrast, he referenced Ginsburg only 94 times, despite serving one year longer with her, and referenced Stevens 116 times, while serving even longer with him. Scalia and Breyer were exceptionally focused on each other, and seldom for reasons of agreement.

Yet, unlike with interruptions, Scalia and Breyer did not leave the other justices in their wakes. Kennedy was an active and bipartisan cross-referencer, invoking Roberts 239 times, Ginsburg 204 times, Breyer 201 times and Scalia 239 times. Other high pairings were Stevens referencing Scalia, at 225 times, Souter referencing Scalia, at 247 times, and Souter referencing Breyer, at 227 times. Significantly, though, Kennedy’s references to Ginsburg and Roberts were the only instances of any justice other than Breyer or Scalia being referenced more than 200 times by any other justice. Scalia and Breyer were exceptional not only in referencing each other, but in being referenced by others. As such, this supports the idea that these two justices’ high reference rates are a sign of their influence, even more so than the medians on the Court during this time, Kennedy and O’Connor.

The universality of cross-referencing one’s friends

The third column shows what we call “the agreement ratio”—the rate at which a justice is referenced (column 1) where that justice and the referencing justice are in agreement versus where they disagree (it does not relate to column 2). This is the most striking result: every single justice, even Justice Thomas, who talks so infrequently, displays the same trend, of cross-referencing other justices with whom the justice eventually will agree in the case in which they make the cross-reference. Once again, then, we have a potentially powerful predictor of eventual outcomes in Supreme Court cases, stemming from in-depth analysis of what happens in the oral argument of any given case.

Altogether, we can see that all of the contemporary justices cross reference each other overwhelmingly when they are in agreement. But the consistency only applies to doing the referencing: when it comes to being referenced, there are tiers of influence among the justices. The justices cross-reference selectively, particularly cross-referencing Justices Breyer and Scalia, two of the leading thinkers on the contemporary Court of their respective ideologies and methodologies.

What does it mean when justices reference each other at oral argument?

What’s in a name?

Previously, we showed that justices often refer to advocates by name as a form of “throat clearing,” that is, using prefatory words to ease into interrupting an advocate. Since interruptions are highly predictive of justices voting against the person they interrupt, that may suggest justices use of an advocate’s name is likely to signal disagreement. Is the pattern the same when justices use each other’s names? There is reason to think not: the justices may refer to each when they want an advocate to return to a question of shared interest. In this post, we begin to explore the phenomena of justice-to-justice name checking and conclude that, although there is significant variation between justices and over time, the practice is highly correlated with, and predictive of, agreement among the justices.

Judicial cross-references over time

As we have shown elsewhere, oral argument has changed significantly in recent decades, so the first natural question is whether judicial cross-references have changed also. The following figure shows a distinct shift in the practice of justice-to-justice name-checking over time. The figure shows the rate of justices referring to each other by name at oral argument from the 1985 to 2017 Terms, normalized per thousand words spoken by the justices.

Justices Naming Other Justices at Supreme Court Oral Argument (1985-2017 Terms)

It is important to normalize figures by word count in order to adjust for the fact that the justices talk more during oral argument than they used to. In the mid-1990s, the justices began to play a far more active role in oral argument; the justices now talk for about 13 minutes more in the course of a 60-minute proceeding than they did in the pre-1995 era. . Yet even controlling for that enormous upward shift in judicial verbosity, justices referring to each other by name has measurably increased over the last 30 years. Since 2007, the rate of justice-to-justice name checking has been consistently above one per thousand words. Given the competitive cut and thrust of oral argument, that makes name-checking a significant part of the dialogue among the justices.

Why the increase has occurred is a matter of interpretation. At least part of the cause is likely to be a response to the other dramatic changes we have observed in oral argument, in particular the significant increase in judicial activity at oral argument. With justices talking so much more and interrupting each other and the advocates more often, and the advocates interrupting back at greater rates, it is easy to have questions go unanswered or partially answered. As such, we would expect justices to refer back to earlier questions that they want to hear the answer to.

That raises the associated question of when the justices refer back to each other’s questions. If the above interpretation is correct, it suggests that Justice A will be refer to Justice B when he or she wants an advocate to complete a previously disrupted answer to Justice B’s question. That in turn suggests that cross-references will occur among justices who are likely to agree rather than to disagree.

The relationship between agreement and justice-to-justice name checking

The following figure explores that agreement-based intuition. It shows the rate of judicial cross-references, once again normalized per thousand words spoken by the justices, but this time divided among whether the cross-references occur between justices who ultimately vote together, or those who ultimately disagree, in the case at hand.

Justices Naming Other Justices at Supreme Court Oral Argument (1985-2017 Terms) (Agree, Disagree)

The results support the interpretation above: justices cross reference each other about three times as often when they will ultimately agree with the person they are cross-referencing than when with they will ultimately disagree. This means that whereas we have a “disagreement gap” for justices interrupting advocates, we have an “agreement gap” between justices cross-referencing each other. (Note we could also develop a disagreement gap between justices, as Jacobi and Rozema’s research shows interruptions are associated with future disagreement among pairs of justices).

Note also that the very large increase seen in the first figure is primarily being driven by an increase in justices cross-referencing those with whom they agree, rather than among justices who disagree. Put another way, while there has always been a tendency to more often cross reference one’s friends than one’s foes, the difference is considerably increasing in the last three decades. This result suggests that cross-references may be a counter-strategy to the more conflictual strategies we have previously highlighted, particularly of taking up the time of the advocate with whom a justice disagrees. By pointing an advocate back to a question of a future ally, the justice is emphasizing what he or she agrees with, rather than solely what he or she disagrees with.

In a time of increasing partisan division on the Court, mirroring the increased partisan conflict in Congress and among the public, here at least we have a manifestation of agreement rather than disagreement. Of course, agreeing with one’s allies is a form of tribalism, as much as disagreeing with one’s foes, but next week we explore in more detail which justices specifically cross reference which other justices, and the broader implications of such behavioral patterns.

The importance of empirical analysis (with forecasts of Bucklew & Madison)

The two Eighth Amendment cases show hard cases make for hard predictions

Bad facts make bad law. So too, hard cases make for hard predictions. The Court’s two current death penalty cases, illustrate both phenomena. But these cases also illustrate the added value of empirical analysis of oral argument over purely qualitative or impressionistic readings.

The Supreme Court’s two current pending death penalty cases both have very peculiar facts. In Bucklew v. Precythe, argued last week, the prisoner argues that execution by lethal injection would be cruel and unusual given his particular medical history. Madison v. Alabama, argued on October 2nd, presents the question of whether a prisoner can be executed for a crime he cannot recall.

The Supreme Court had the option of taking a case raising a question with potentially much broader significance for Eighth Amendment jurisprudence. Hidalgo v. Arizona challenged whether Arizona’s death penalty legislation insufficiently narrowed application of the ultimate punishment. True to form, the Supreme Court avoided that important question of the frequency of the ultimate punishment, and focused instead on two cases raising as applied challenges, with limited applications for the 2,743 people currently on death row.

Forecast of Bucklew v. Precythe

Argument in Bucklew v. Precythe involved delving into the grisly facts of Petitioner’s medical condition and the painful possibilities involved in his execution. Along the way, the argument saw some unusual judicial behavior. Immediately, Justice Sotomayor began grilling Petitioner’s advocate for the lack of a clear record on those facts, and whether the horrific nature of his death could simply be avoided by the trach already installed in his throat.

ScotusOA prediction based on the oral argument in Bucklew v Precythe

Commentators on the First Mondays podcast suggested that this interplay may indicate that Petitioner’s advocate may have lost Justice Sotomayor, which would make it “tough” to win, given her general reliable pro-defendant vote. This much-qualified prediction misunderstands what is going on. Although Sotomayor was very active in questioning the Petitioner, and that level of activity often signals disagreement, she was even more active during Respondent’s time. Sotomayor spoke 17 times to Petitioner and 18 times to Respondent; but when she spoke to Respondent, she had much more to say—921 words versus 497 words. In spite of the opening salvo, by the end of the argument Sotomayor registered a significant disagreement gap in favor of Petitioner. Thus, even though she described herself as “upset” with Petitioner’s advocate, our analysis, represented in the figure below, predicts she is 90% likely to vote for Petitioner, and is the Justice most likely to do so.

Sotomayor normally reserves her toughest questions for the prosecution side in criminal cases—perhaps reflecting, as well as her liberal ideology, that as a former prosecutor herself she has particularly high standards for the profession. That fact made her sharp critique of the capital defendant seem more important than it really was. Our empirical approach to reviewing oral argument helps put episodes like this in a more balanced context and avoid salience bias (a recognized common behavioral irrationality that causes people to focus on prominent information and ignore potentially more significant but less noticeable contra-indicators).

A far clearer signal of a Justice breaking with expectations was Justice Kavanaugh. He spoke only to Respondent’s advocate and asked tough questions as to whether there was any limit on the potentially “gruesome and brutal pain” the state is permitted to impose in executing Bucklew, and demanded a yes or no answer from the state’s advocate.

In contrast, the numbers on Justice Gorsuch are likely misleading. He spoke only twice to Respondent’s advocate and once to Petitioner’s advocate, making his signals very weak. Equally importantly, both questions he put to Respondent’s advocate were open-ended questions asking him to get to a point he had said he would make—that is, a seemingly friendly inquiry. Given Gorsuch voted to deny the stay of execution, we expect he will join the other conservatives in voting with Respondent. (Although Justice Thomas was his normal reticent self, he too had joined the dissent from the stay of execution).

With Justice Ginsburg unusually silent (even before breaking her ribs), we can only go on her prior voting record, which is generally pro-defendant. All of this would lead to a prediction of a 5:4 vote for Petitioner, however the prediction depends on the untested vote of Justice Kavanaugh—a tough prediction, indeed.

Forecast of Madison v. Alabama

Oral argument in Madison v. Alabama, on the question of whether someone who cannot remember the crime he committed due to multiple strokes, also involved descriptions of peculiar facts. According to Petitioner’s advocate, inmate Madison regularly soils himself because he cannot remember that he has a toilet in his cell. The law on the issue was as messy as the facts. The following was a typical confused interaction:

Sonia Sotomayor: Mr. Stevenson, part of the problem is the use of the word “loss of memory.” And I — in your briefs, you seem to go back and forth on this. Are you conceding that amnesia about the incident alone, where you can function in every other way in society, would you be incompetent then?

Bryan A. Stevenson: No.

Sonia Sotomayor: To be executed?

Bryan A. Stevenson: Yes, that’s right.

On another point Justice Alito complained: “No, I don’t understand — I don’t understand your answer.”

And Chief Justice Roberts questioned whether there was an issue at all in the case, saying to Petitioner’s advocate: “There are two questions. You concede on one, and the state concedes on the other.” Petitioner had conceded that simply not remembering the crime is not enough to avoid execution, and the state had admitted that if the person is incompetent, they cannot be executed.

The figure below shows our predictions for the case.

ScotusOA prediction based on the oral argument in Madison v Alabama

Thomas was silent as always and Gorsuch was silent in this case also. Furthermore, both Breyer and Roberts presented ambiguous signals. Again, Thomas, Alito, and Gorsuch had opposed the imposition of the stay, suggesting again support for Respondent. This time, however, Roberts had not joined that order. His far more mixed signal in this argument supports Amy Howe’s prediction that as Chief Justice, he “might have a greater incentive than his colleagues to avoid deadlocking on Madison’s case” (Kavanaugh was not yet on the Court when the case was heard) and so could provide the fifth vote for a very narrow victory for Madison by remanding the case back to state court to consider the specific question of whether Madison is incompetent because of his dementia.

We think this analysis makes sense of Roberts’ tentative signal. But we would go further and say that Roberts will only vote in favor of Petitioner for a very narrow ruling, otherwise he will vote for Respondent. We believe Roberts’ ambiguous signal indicates his concern over the potential for this second very specific medical circumstance to create an enormous slippery slope to a flood of future death penalty challenges. Whereas Bucklew’s case is limited to an n of approximately one, making it hard to justify the Court using one of its approximate 75 spots to effectively act as a court of last instance, in contrast, Madison’s case could drastically change death penalty jurisprudence. With death penalty appeals dragging the ordinary execution process out over decades, the chances of other inmates developing memory-related medical issues are very high. A broader ruling for Petitioner could spawn a tide of challenges that would make Atkins IQ challenges seem a narrow set. As such, we expect Roberts to either support a very narrow ruling for Petitioner or rule in favor of Respondent. The empirics indicate he is right on the borderline.

Thus, we predict a 5:3 win for Petitioner on narrow grounds, or a 4:4 default win for Respondent.


Prediction for Bucklew: 5:4 vote for Petitioner (Bucklew)

For Petitioner: Sotomayor, Kagan, Breyer, Kavanaugh, Ginsburg

For Respondent: Roberts, Alito, Thomas, Gorsuch

Most likely the switch: Kavanaugh


Prediction for Madison:

5:3 vote for Petitioner (Madison) and remand


4:4 vote for Respondent (Alabama)

For Petitioner: Sotomayor, Ginsburg, Kagan, Breyer, (Roberts)

For Respondent: Alito, Thomas, Gorsuch, (Roberts)

Most likely to switch: Roberts

Conservative Principles v. Conservative Outcomes: The Clash of the Two Trump Appointees on the ACCA

Gorsuch versus Kavanaugh: what is a conservative?

The Supreme Court heard two cases about the reach of the Armed Career Criminal Act (ACCA) in October that highlight stark differences between the two Trump appointees. Justice Gorsuch appears to be, as promised, a conservative in the mold of the late Justice Scalia: sitting far to the right on the Court but willing to side with the liberals when issues of methodology or fairness in criminal cases demand. In contrast, Justice Kavanaugh was uncritical of the potential harshness of the government’s position in each case, seemingly focused on promoting the law and order outcome rather than refining the means of analysis.

The two cases involve complex distinctions about the applicability of the much-litigated ACCA, which imposes a 15 year sentence enhancement for persons convicted of three crimes within certain specified categories, at least one of which involves weaponry. Burglary is one recognized category, and U.S. v. Stitt queries whether burglary of a residence can include burglary of a vehicle—which previous cases have said are otherwise not covered—if those vehicles can be adapted for residency or are otherwise being used as residences. Robbery is not recognized as a category under the ACCA, but Stokeling v. U.S. raises the question of whether robbery can nevertheless be covered by the sentence enhancement as a crime that has “an element of use, attempted use, or threatened use of physical force” against another, even if it only involves using slight force to overcome minimal victim resistance.

Early in the argument in Stitt, Gorsuch started listing problems with the government’s position, drawing on a variety of approaches, from textualism—including drawing significance from the state legislature’s use of a disjunctive and the government’s interpretation raising problems of surplusage—to whether congressional intent could be inferred if the legislation covered so few states that defined burglary in such a way. He made similar points in Stokeling, along with raising concerns about the meaning of the legislation at the time of enactment, the traditional use of the term ‘robbery’ at common law, as well as more general critiques in Stitt about widely held dissatisfaction with the Court’s jurisprudence on the ACCA.

In contrast, Kavanaugh focused on carefully maneuvering around prior precedent to make a stronger case for the government. For instance, in Stitt, he appeared quite well-prepared, tailoring a path for getting around what had appeared to be a bright line rule against the inclusion of vehicles in burglaries. For a number of justices, ACCA cases raise questions of fairness, notice, due process, and proportionality, as well as difficult questions of statutory interpretation. Kavanaugh seemed unfazed by these worries, saying for instance of the notice question: “if you’re convicted three times of burglary for burglarizing an RV, you’re on notice, presumably. . . I don’t understand the notice point.

These differences are likely to be determinative, and we predict that the two Trump appointees will diverge in their votes on these two cases, as the following two figures show.

Stokeling v US Predictions

Figure 2

US v. Stitt Predictions

The figures above show the results of a new predictive model we have developed. The model transforms statistical observations about oral argument directly into predicted probabilities based on the prior behavior and voting patterns of each justice. We blended an average of Gorsuch, Roberts, and Alito for the 2017 Term to estimate baseline parameters for the newly appointed Kavanaugh.

In Stokeling (the robbery case), the model predicts a winning coalition for the Petitioner (the criminal defendant) of Justices Sotomayor, Kagan, Gorsuch, and Ginsburg, with probabilities of 88%, 87%, 84%, and 68% respectively. Justice Breyer was uncharacteristically silent but we are somewhat confident he will make up the fifth vote (Breyer is only silent in about 7% of arguments). The model predicts at least a three justice minority consisting of Justice Alito, Chief Justice Roberts, and Kavanaugh, with 91%, 66%, and 61%, respectively. These numbers struck us as understatements after we listened to the argument.

In Stitt (the burglary case), we believe that the Chief and Justice Thomas will vote for the Petitioner (the state), although Thomas said nothing and Roberts said nothing of substance during the oral argument. Assuming Roberts will vote consistently in the two cases and that Thomas will vote against the defendant in both, the model predicts a winning coalition of Kavanaugh, Kagan, Alito, Roberts, and Thomas, at rates of 88%, 73%, and 58% for Kavanaugh, Kagan, and Alito, respectively. The model also predicts at least a three justice minority consisting of Ginsburg, Gorsuch, and Sotomayor (all north of 85%), but it is equivocal about Breyer, as indicated by the fact that he shows up on both sides of the 50/50 dividing line.

One of the striking aspects of the two arguments is that Justice Kagan seems likely to vote a split ticket, holding for the criminal defendant in one case and for the government in the other. For Kagan, the central question in both cases appeared to be the feasibility of line-drawing. The difference between the two cases is that Kagan appeared comfortable with the distinction between mobile homes and other vehicles in Stitt, but not with the different shades of robbery implemented by the various states in Stokeling. It is also possible that Breyer will split the same way for largely the same reasons, but this is far from clear from his numbers. We expect the rest of the justices to vote consistently pro- or anti-criminal defendant in both cases.


Stokeling prediction: 5:4 for Petitioner (Stokeling)

For Stokeling: Sotomayor, Kagan, Gorsuch, Ginsburg, and Breyer

For the government: Alito, Chief Roberts, Kavanaugh, and Thomas

Most likely to switch: Breyer


Stitt prediction: : 5:4 for Petitioner (the state)

For the government: Kavanaugh, Kagan, Alito, Roberts, and Thomas

For Stitt: Ginsburg, Gorsuch, Sotomayor, and Breyer

Most likely to switch: Breyer

Using interruptions to predict Supreme Court cases

Reading Justice Gorsuch

If you wanted to know which way Justice Gorsuch was going to vote in the 2017 Term, you could have placed your bets with 86% accuracy by observing just one statistic from oral argument—how many times Gorsuch interrupted each side.

In the 2017 Term, Gorsuch heard 60 arguments and voted 40 times for the Petitioner and 19 times for the Respondent (Washington v. United States was affirmed by an equally divided Court, and by convention no individual justice votes are recorded). On average, he initiated 1.6 more interruptions during the Respondent’s time than during the Petitioner’s (a few of these interruptions were of other justices but most were of the advocate). But like Chicago’s annual average temperature of 49°, that number masks all the interesting variation. In those cases where Gorsuch voted for the Petitioner, he interrupted the Respondent an average 3.8 more times than the Petitioner. Conversely, when voting for the Respondent, he interrupted the Petitioner 3.1 more times on average.

One way to visualize this kind of data is with a histogram.

Gorsuch Interruption Gap, Pro-Petitioner and Pro-Respondent Histograms 2017 Term

In the figure above, we have overlaid two different histograms of the interruption gap (interruptions of Petitioner minus those Respondent) for Gorsuch in the 2017 Term. The orange bars indicate the interruption gap in cases where the Justice voted for the Petitioner: it is clear that in the vast majority of these cases, Gorsuch interrupted less during the Petitioner’s time than Respondent’s. The blue bars show the interruption gap for the pro-Respondent cases. Here, in contrast, most of the weight of the figure is above zero—i.e. Gorsuch interrupted the Petitioner more often, though the shape of the blue distribution is flatter. We should note that because Gorsuch voted for the Petitioner twice as often as he voted for the Respondent (as the Court in general does), an orange bar represents twice as many cases as would a blue bar of the same height.

In the next figure, we further explore the distribution of Gorsuch’s interruption voting patterns through a boxplot.

Gorsuch Interruption Gap Boxplots 2017 Term

Boxplots are cool (trust us!) but they require a little explaining. The “box” part of a boxplot indicates the range separating the top 25% of the data from the bottom 25% (the interquartile range); think of this as the space between the 25 and 75 yard lines on a football field. The vertical line segmenting this box is the median of the data. So, in the figure above, a quick comparison of the light blue pro-Respondent box with the orange pro-Petitioner box shows that there is no overlap between the middle 50% of each of these categories. What is more, the orange box is entirely to the left of zero on the interruption gap scale and the blue box is entirely to the right. The white space between the boxes tells you at a glance that 75% of cases in the orange category have a lower interruption gap than 75% of cases in the blue category. And thus, when Gorsuch interrupts one side more than the other, he is usually showing his hand and telegraphing which way he will vote.

The whiskers extending away from the boxes extend close to the extremes of the data: they indicate about how far the data extends before you get to true outliers (in a normal distribution, 0.7% of observations lie beyond the upper and lower limits). The fact that the whiskers cross the zero point confirms that although the interruption gap is a very strong predictor for Gorsuch, it is not infallible. In fact, in the 2017 Term, relying on the interruption gap to predict Gorsuch’s votes would have been misleading in just 7 cases.

Some broader implications

The Supreme Court is under close scrutiny like never before. In this era of intense political polarization, the Court merits attention for reasons that go beyond the outcomes of individual cases or issues of methodology and jurisprudence. The justices themselves have become icons—or demons—representing the moral future of the country. Ruth Bader Ginsburg’s visage appears on cups, T-shirts, and various other paraphernalia, and the late Antonin Scalia is fetishized equally by the right. The two most recently appointed associate justices were subject to extraordinary confirmation processes. Gorsuch holds a “stolen” seat in the eyes of some, and Kavanaugh is defective in character and temperament, according to others. In a divided country where justices are feted and demonized with equal fervor, the public is now paying close attention to every aspect of the Court’s decision-making process, down to the meanings attributed to every word uttered. Oral argument deserves attention because it is the one public part of the Court’s process, and because it is public it gives us an opportunity to more rigorously assess aspects of judicial character that would otherwise be matters of supposition. One way to explore those more amorphous judicial traits is to look at the language justices use in the relatively unguarded arena of oral argument, as compared to tightly scripted case opinions.

Websites such as SCOTUSBlog provide excellent coverage of current Supreme Court cases using traditional modes of legal analysis: close reading and expert opinion. In addition, reporters such as Marcia Coyle, Dahlia Lithwick, Adam Liptak, and Nina Totenberg provide qualitative analysis of Supreme Court oral arguments, and there also are podcasts such as Amicus, First Mondays, and even an army of teenagers who write about the Supreme Court at the website High School Scotus. There is also excellent analysis by other empirically minded scholars, such as Tim Johnson, who has shown that oral argument can be important in judicial decision-making, and some have used transcript data in their research.

But ScotusOA offers something new. Unlike other commentators, we make voting predictions for each justice as the cases are heard. Predicting outcomes is always treacherous, as even a largely accurate model will have variation and noise: even an attempt to model prior data, where the results are known, will lead to imperfect “predictions.” Consequently, we buttress our empirical analysis by listening to the oral arguments, in order to follow the nuances of each case and to understand what the justices think they are communicating.

We also base our predictions on justice-specific models of features of oral argument that our early research has told us are important in the aggregate (Jacobi & Schweers, “Justice, Interrupted”; Jacobi & Rozema, “Judicial Conflict and Voting Agreement”; Jacobi & Sag, “Justices as Advocates”). The challenge is to develop a model that is a good predictive tool in individual cases. To do that, we have developed multiple models for each justice to predict their votes based on oral argument. What is striking about Justice Gorsuch is that he is most easily predicted using just one variable, the interruption gap.

Nielsen v. Preap forecast: Kavanaugh shows his colors and they are bright red

On a major issue for the Trump administration, Kavanaugh signals he will be loyal

At oral argument in Nielsen v. Preap last week, the Supreme Court considered whether non-citizens released from criminal custody should be subject to mandatory detention when the Department of Homeland Security fails to take them into custody immediately. Coming on only his second day on the Bench, the natural question is: how did Justice Kavanaugh behave, and what does it tell us of his likely vote in the case? Given the Trump administration’s crackdown on illegal immigrants, Nielsen v. Preap presented Kavanaugh with his first opportunity to repay those who stood by him in the face of allegations of sexual misconduct, lack of candor, and an injudicious temperament.

After his confirmation hearings, many expressed concern that Kavanaugh could not be impartial, given the arguably unjudicial anger he expressed, with even the nonpartisan ABA reconsidering its review of him based on concerns about his temperament. In response, Kavanaugh said that he could be impartial. It is difficult to infer how the nomination hearings have affected Kavanaugh because he was always expected to be both conservative and partisan. But whether the cause is ideological alignment, inherent partisanship, or indebtedness and resentment in response to the nomination process, our empirical analysis of the Preap oral argument suggests that Kavanaugh will be loyal to the Republicans. The following figure shows the disagreement gap—the differential between a justice’s treatment of the side he or she ultimately votes with and against—in terms of both justice words and speech episodes in the Preap argument.

United States v. Preap, Disagreement Gap in Speech Events and Words

The disagreement gap is a better predictor for some justices than others. As we noted in a previous post, Justice Gorsuch and Justice Alito have particularly obvious tells in this regard: the more each talks to an advocate, the more likely each is to vote against him or her. The early numbers on Kavanaugh suggest that he will leave Gorsuch and Alito in the dust in this regard. At first in Preap, Kavanaugh seemed to be still finding his feet, with his initial attempts to ask a question thwarted by his colleagues. But once he finally took the floor, he signaled his favor for the Government. He had as much to say to the respondents as anyone—Gorsuch and Kavanaugh both had 13 speech events directed at the Respondents—and he had the highest disagreement gap, with nothing at all to say to appellants. The extreme tilt in his participation on this issue and the proportion of his questions and non-questions (a 5:11 ratio, if you were curious), as well as their content, suggest he is a safe vote for the government and for the Trump administration’s immigration policies.

The rest of the Court

As for the rest of the Court in Preap, the split was fairly predictable, with the liberals strongly supporting the Respondent immigrants and the conservatives showing equal support for the government. On a highly polarized Court, the biggest surprise is the ambiguous signals from Justice Ginsburg.

The issue presented is one of statutory interpretation, since the statute refers to detention “when” the person is released; however, with some of the non-citizens represented in this case having been free for years, one for 11 years, the liberal side of the Court could not help querying whether a constitutional claim also arose (much to the annoyance of Justice Alito). As the figures above show, however, this strategy is unlikely to be successful. We predict a 5:4 win for the government, falling along standard ideological lines (as one of us predicted on CBS on Tuesday, prior to the argument).

The Advocates

The performance of the advocates in Preap is worth noting. Assistant Solicitor General, Zachary D. Tripp, was unusually weak for that office. He struggled to answer basic questions, such as where the government draws authority to detain non-arrestees and whether the same rule would apply to somebody 50 years past attention rather than 11 years. He also interrupted and spoke over the justices an extraordinary 23 times. In contrast, advocate for Respondents, Cecillia D. Wang, began exceptionally strongly, speaking for 4 minutes and 15 seconds before any Justice asked her a question. She was interrupted only 28 times, compared to Tripp’s 52 times, another sign of strength, and she interrupted only 8 times (these were mostly cross-talk).

However, Wang struggled a little later in her argument, repeatedly insisting that the statute’s use of the term “when” a person is detained necessarily requires that detention be initiated on the same day as release from arrest. That strict timetable clearly alarmed the pragmatists on the Court, both left and right. Instead, she should have responded that 24 hours would be the ideal rule, but no matter what the rule, 11 years is clearly unreasonable. Perhaps having had such a perfect beginning, she was shooting for the moon, but in the process she may have lost any chance she had to bring on board the conservative pragmatists, Chief Justice Roberts and Justice Alito.

Justice Gorsuch attempted to flex his formalist muscles, debating Wang on questions of grammar, but it is foolish to think that formalism can decide this question. The term “when” is inevitably context dependent: the time constraint implicit in the command “make the tea when the kettle boils” raises a very different expectation than the instruction “come visit when you are in town.” The government would have the Court believe that tea can be made with water boiled 11 years previously.

Beyond Preap: Kavanaugh’s first week on the Court

Considering all four cases the Court heard in Kavanaugh’s first week on the Bench, the newest justice may not be ideologically moderate, but so far he has been moderate in his behavior, in some senses. As seen on the figure below, on average this week, Kavanaugh talked slightly less than the median level among the justices and interrupted at the median. We saw the same trend in the Preap case.

Kavanaugh’s first week, by the numbers

However, these numbers might conflict with the subjective experience of listening to the oral argument because Kavanaugh barely spoke at all in Air And Liquid Systems v Devries and spoke only to one side in both Preap and in United States v. Stitt. It was only in Stokeling v. United States that Kavanaugh had a significant amount to say to both sides of the argument. It seems likely that when Kavanaugh settles in to the Court he will become even more vocal, and it would not be surprising to see him continue to strongly favor one side in any given case. We also expect that Kavanaugh will interrupt even more in the weeks and years to come.

Prediction: 5:4 for the government

For the government (Petitioner): Kavanaugh, Gorsuch, Thomas, Roberts, Alito

For Preap (Respondent): Kagan, Sotomayor, Breyer, Ginsburg

Likely to switch: none

Gundy v. U.S. Forecast: A highly unusual meeting of minds

Gorsuch and Sotomayor find uncommon common ground

Last week’s oral argument in Gundy v. U.S. leads us to predict a unique alliance: a dissenting coalition of the liberal Justice Sotomayor and the conservative Justice Gorsuch. Gundy addressed whether the Sex Offender Registration and Notification Act (SORNA) unconstitutionally delegated unconstrained authority to the Attorney General to determine its pre-authorization application. Petitioner’s theory of the case has the potential to radically reshape the Court’s approach to the modern administrative state.

The following two figures show the disagreement gap—the differential between a justice’s treatment of the side he or she ultimately votes with and against—in terms of both justice words and speech episodes in Gundy’s oral argument. Our qualitative and quantitative analysis leads us to predict a 5:3 ruling in favor of the government, with Sotomayor and Gorsuch joined by the silent Justice Thomas, based on his previous positions. This alignment has never occurred before. For context, Sotomayor and Thomas have only once dissented in isolation from the other justices and have formed two of three dissenters in just three cases previously, but never with Gorsuch as the third vote. Sotomayor has never been part of a two or three justice dissent with Gorsuch. If we are correct, this will be a truly unique alignment among the justices.

Gundy v. US, Disagreement Gap (Words)

Gundy v. US, Disagreement Gap (Speech Events)

Listening to the oral argument, the reasons for this unusual alignment become clear: Sotomayor and Gorsuch have found common ground even though each is clearly guided by their contrasting ideologies. Sotomayor’s questions indicate a liberal distrust of the criminal justice process: she considered that who is covered by a statute is “at the core of what a law is” and here defines who is a criminal. In contrast, Gorsuch’s questions and comments show he is motivated by conservative distrust of government: he worried about the “blank check” Congress gave to the Attorney General. Neither justice seemed too bogged down in concerns about how limiting delegations like the one in SORNA would imperil the broader edifice of the administrative state.

We argued recently that you should prefer to play poker with Gorsuch than with any other justice on the Court, as the early data suggests his vote on case outcomes is the easiest to predict by the bias in his questions and comments at oral argument. The comments of two of his fellow justices and an advocate he was trying to help, as well as the laughter of the courtroom, implied agreement with that conclusion, at least in the case at hand. Gorsuch gave two very detailed leading speech episodes suggesting he agreed with Petitioner’s argument, ending with the rhetorical question “What’s vaguer than a blank check to the Attorney General of the United States to determine who he’s going to prosecute?” In response, Petitioner’s advocate Baumgartel repeatedly answered simply “yes.” Justice Ginsburg then pointedly leave quipped “That’s your argument stated very, very concisely,” to which Baumgartel piled on, saying “I’ll cede my time.” The courtroom laughed, and Justice Kagan, who had ceded the floor to Gorsuch after he interrupted her, continued the joke, saying “Well, then I’ll take back my time.” Laughter followed once more.

That Gorsuch was so clearly willing to rule against the government in Gundy is little surprise: along with Thomas, Gorsuch was predicted to be most likely to take that position based on his prior statements, with some suggesting  Ginsburg could possibly also agree with those two conservative justices. But much more surprising is our prediction here that it is Sotomayor, not Ginsburg, who is the most likely liberal justice to vote with Gorsuch, as the two figures above show.

Ideology was not all that mattered in Gundy’s oral argument. Justice Kagan is almost as liberal as Sotomayor, yet she measured very clearly in the other direction. On the numbers, she was even more predictable than Gorsuch, but her more subtle style makes her views less obvious to a casual observer. The other liberal justice, Justice Breyer, appeared on the figures above to be sending mixed or unclear signals. In fact, we are confident that he will join the strongest advocates for the government’s position: in order, Justices Kagan, Alito, and Ginsburg. Early in the argument, Breyer got his own laugh by advising Baumgartel that she was wise not to try to make an argument to strike down all 300,000-odd legislative delegations to agencies. A highly pragmatist justice, we predict Breyer will rule for the government for this very practical reason, but he is torn in this case between his liberal ideology and his pragmatism. He described the concern “gnawing” at him: the “danger when you combine [in one person] prosecuting a person with the writing of the law under which you prosecute.” With his ideology and his legal methodology pulling him in opposite directions, Breyer spoke similar amounts to each side. Yet, ultimately, he showed his hand, beseeching the government advocate Wall to help him write the rule to avoid such a danger.

The advocates seem aware of the dissonance between ideology and methodology that the case presents for some of the justices. Wall finished his argument with a long description of the many harms that would flow from ruling for the Petitioner, an attempt to ensure that Breyer, Alito, and the Chief stayed with him for pragmatic reasons. In rebuttal, Baumgartel tried to pick off Alito and Roberts by saying that the state legislative equivalent can avoid those harms. At least Breyer and Alito seem unpersuaded by this. The hardest vote to read is Chief Justice Roberts, who did question both sides closely, measuring as mixed on both measures, one in each direction.

Forecast: 5:3 for the U.S.

For the U.S.: Kagan, Alito, Ginsburg, Breyer, Roberts

For Gundy: Gorsuch, Thomas, Sotomayor

Most likely to concur or switch: Roberts

by Tonja Jacobi and Matthew Sag