The Error of Equal Error Rates

1999 D.H. Kaye(*)

The final version of this papers is published in Law, Probability & Risk, Vol. 1, No. 1, July 2002, at 3-8.

In most civil litigation, plaintiffs are required to prove their cases by a bare "preponderance of the evidence," and this is usually taken to mean that the probability that their version of the disputed events exceeds one-half. This p > rule can be shown to minimize the expected "losses,"(1) "disutilities,"(2) "regret,"(3) "costs,"(4) or "errors"(5) in verdicts that turn on these events.(6) This analysis has been called the Bayesian Decision Theory justification or explanation of the p > rule,(7) and it has been presented, perhaps too cavalierly, as the "reigning theory" of the law's burdens of persuasion.(8)

Some distinguished commentators have defended the p > rule on other grounds. They see it as advancing a different objective--allocating erroneous verdicts across plaintiffs and defendants in equal numbers. In a rich and thoughtful textbook on the law of evidence,(9) Professors Allen, Kuhns, and Swift identify "the premises underlying the preponderance rule"(10) as follows:

The preponderance rule incorporates an underlying assumption concerning the participants in litigation: that plaintiffs as a class and defendants as a class generally ought to be treated equivalently. The reason for this assumption is that before a case is resolved, one cannot know who should win; it is as likely that the defendant should win as the plaintiff. . . . Without knowing the facts, it seems just as likely that the defendant is refusing to pay what is owed as that the plaintiff is attempting to obtain an undeserved benefit.

The preponderance of the evidence standard generalizes this basic point into a statement that the law should treat plaintiffs as a class and defendants as a class equivalently (with defendants and the status quo favored if there is a "tie" -- that is, if the evidence does not preponderate in either direction). As Professor Kaye has demonstrated algebraically, if certain conditions are met the preponderance of the evidence standard should result in about the same numbers of errors being made for plaintiffs as for defendants.(11)

I fear that this passage misstates the underpinnings and workings of the burden of persuasion.(12) There is no obvious reason to pursue equal numbers of actual errors, and even if that policy had a sound foundation, it would tend to support departing from the p > rule rather than adhering to it.(14) Accordingly, the equal-error theory of the civil burden of persuasion is singularly unpersuasive. To establish these points, I proceed in three short, but decisive steps. Part I questions the premise that jurors do or should presume that the prior probabilities for plaintiffs and defendants are equal. Part II probes the connection between equal prior probabilities and the p > rule. Finally, Part III identifies the aspect of equality that underlies the p > rule.

I. Equality as Equal Prior Odds

According to Professor Allen and his colleagues, the reason to treat plaintiffs and defendant equally is that "before a case is resolved, one cannot know who should win; it is as likely that the defendant should win as the plaintiff."(14) It is difficult to know what to make of this suggestion. It might be an empirical assertion that prior to introducing any evidence, the probability that the facts are such that a plaintiff should prevail is one-half. This would be so if "in the set of all cases going to trial there are approximately as many deserving plaintiffs as deserving defendants."(15) However, no data are available to test this hypothesis,(16) and even if it were a fact, what would it prove? No obvious inference as to how a case ought to be decided after all the evidence is in seems to flow from the fact that a case is in equipoise before the trial begins.

Perhaps, then, the claim is not empirical, but normative.(17) Maybe Professor Allen and his colleagues are claiming that ignorance of the facts of a case should mean that the probabilities are in equipoise. Again, no argument for this claim is made, and none is apparent. Suppose plaintiff sues in tort, claiming that he bought an automobile with a defective steering column that caused the vehicle to crash. The defendant manufacturer, let us assume, denies only part of the complaint--that there was a defect in the steering. Even if one knows nothing else about the case, one should not assume that the car's steering wheel is as likely be to defective as it is to be non-defective. Equal treatment does not mean ignoring the reality that most cars that are involved in accidents do not have defective steering columns. Plaintiff may be able to prove that his car had this unusual characteristic, but until he produces some admissible evidence of that fact, a juror need not believe that the claim has a 50-50 chance of being true. Laplace's principle of insufficient reason hardly seems compelling in this context.(18)

II. Equal Prior Odds and the p > Rule

Even if some convincing argument were available to show that jurors should enter the jury box with prior probabilities of one-half, the statement that "the preponderance of the evidence standard generalizes this basic point into a statement that the law should treat plaintiffs as a class and defendants as a class equivalently"(19) would remain mysterious. According to Profesor Allen and his colleagues, the generalization consists of an article that "demonstrated algebraically" that "if certain conditions are met, the preponderance of the evidence standard should result in about the same numbers of errors being made for plaintiffs as for defendants."(20) But the article contains no such demonstration.(21) Instead, it shows that in cases with a single plaintiff, a single defendant, and a single disputed factual issue, the p > rule minimizes the expected number of dollars that come out of the wrong pockets.(22)

Professor Allen and his colleagues misread the article, suggesting that its analysis "can be understood without the mathematics," simply by assuming "that in the set all cases going to trial there are approximately as many deserving plaintiffs as deserving defendants," that jurors "will make a rough probability assessment of the strength of each case," and that "the probability assessments for these two sets are in a normal distribution over the range of 0.0 to 1.0."(23) They think that "then the number of errors made for plaintiffs will approximate the number of errors made for defendants, and the preponderance of the evidence standard will have done its job."(24) Of course, the conclusion does not follow even from these artificial premises,(25) and the premises are neither used nor required to establish the expected-error-minimizing quality of the p > rule.(26)

III. Equality and the p > Rule

Although the equal-error analysis does not offer an adequate positive or normative account of the p > rule, this is not because the p > rule neglects the norm of equality. To the contrary, as Professors Allen, Kuhns, and Swift assert at the outset of their discussion, the rule implements the principle of equal treatment of plaintiffs and defendants. However, it does this not by allocating errors in any particular way, but rather by attending to the expected loss associated with the decision rule. The basic premise is not that we should equalize the numbers of errors of each type, but rather that we should be equally concerned with each type of error. When the loss resulting from a mistaken verdict for plaintiff is set equal to that for a mistaken verdict for defendant, the p > rule immediately follows.(27)

This form of equality is appropriate because every error is equally serious-- regardless of who it favors or hurts--and because errors for different parties in different cases do not cancel each other out. When these conditions hold, we should strive to minimize the total number of errors without worrying about whose ox is gored by any asymmetry in the distribution of errors.

In adopting and applying the p > rule, therefore, the law treats plaintiffs and defendants equally. The prior probabilities may not be equal, and the numbers of errors of each type may not be equal. But to demand that the burdens of factual mistakes at trials fall with mathematical equality on plaintiffs and defendants would be to pursue an outcome-based egalitarianism that has no defensible foundation. By giving equal weight to each type of error and letting the errors fall where they may, we afford plaintiffs and defendants equal concern and respect.(28)


* Regents Professor, Arizona State University, College of Law.

1. D.H. Kaye, Clarifying the Burden of Persuasion: What Bayesian Decision Rules Do and Do Not Do, 3 Int'l J. Evid. & Proof 1 (1999).

2. John Kaplan, Decision Theory and the Factfinding Process, 20 Stan. L. Rev. 1065 (1968).

3. Richard Lempert & Stephen Saltzburg, A Modern Approach to Evidence 163 (2d ed. 1983); Richard Lempert, Modeling Relevance, 75 Mich. L. Rev. 1021 (1977).

4. Richard A. Posner, An Economic Approach to the Law of Evidence, 51 Stan. L. Rev. 1477, 1504 (1999) (asserting that the p > rule follows from the presumed fact that, on average, the costs of an erroneous verdict are equal as between civil plaintiffs and defendants).

5. David Kaye, Naked Statistical Evidence, 89 Yale L.J. 601 (1980) (book review).

6. See infra note 26; authorities cited, supra notes 1-3 & infra note 7.

7. See, e.g., D.H. Kaye, What is Bayesianism?, in Probability and Inference in the Law of Evidence: The Limits and Uses of Bayesianism 1 (Peter Tillers & Eric D. Green eds., 1988).

8. D.H. Kaye, Apples and Oranges: Confidence Coefficients Versus the Burden of Persuasion, 73 Cornell L. Rev. 54 (1987).

10. Ronald J. Allen et al., Evidence: Text, Cases, and Problems 828 (2d ed. 1997).

10. Id.

11. Id. at 828. The reference is to David Kaye, The Limits of the Preponderance of the Evidence Standard: Justifiably Naked Statistical Evidence and Multiple Causation, 1982 Am. B. Found. Res. J. 487, reprinted in Evidence and Proof (W. Twining & A. Stein, eds., 1992).

12. A related mistake appears in an early edition of the influential textbook by Professor (now judge) Richard Posner. See Richard A. Posner, Economic Analysis of Law 432 (2d ed. 1977) (asserting that the p > rule "implies that of cases decided erroneously, about half will be won by undeserving plaintiffs and about half lost by deserving plaintiffs").

13. See Michael Finkelstein, Quantitative Methods in Law: Studies in the Application of Mathematical Probability and Statistics to Legal Problems (1978) (asserting that error-equalization sometimes justifies increasing the threshold of the civil burden to some higher probability than one-half). This may be the authors' point. See Allen et al., supra note 8, at 829 ("We can use these same graphs to demonstrate why alternative, higher burdens of persuasion are occasionally relied on in civil cases."). If so, however, the argument is odd. It begins by asserting that equalization is the reason for a broadly applicable p > rule, but specifies a very narrow range of circumstances in which the p > rule achieves equality. If these difficult-to-satisfy conditions were the only such circumstances in which the rule leads to equal error rates, however, the equalization rationale would not be much of an explanation for the rule.

14. Allen et al., supra note 8, at 828.

15. Id.

16. Such data would have to include information from which to determine the true facts in each case--surely, a Herculean task.

17. Cf. Posner, supra note 4 (making such a claim).

18. Cf. David Kaye, Playing Games with Justice: Rawls and the Maximin Rule, 6 Social Theory and Practice 33 (1980).

19. Allen et al., supra note 8, at 828.

20. Id.

21. Furthermore, in other writings I have contrasted the expected-error-minimizing p > rule to an expected-error-equalizing standard and have disavowed the idea that error equalization should influence burdens of persuasion or the use of statistical evidence. See Kaye, supra note 4; Kaye, supra note 8, at 72-73; D.H. Kaye, Hypothesis Testing in the Courtroom, in Contributions to the Theory and Application of Statistics (Alan E. Gelfand ed., 1987).

22. By "dollars that come out of the wrong pockets," I mean the sum of (a) dollars paid to plaintiffs who should not have been compensated, and (b) dollars retained by defendants who should have compensated plaintiffs.

23. Id.

24. Id. at 828-29.

25. For example, normality is neither a necessary nor a sufficient condition for error equalization. Allen et al. recognize the latter point when they speak of "shaded areas under the two graphs [being] of approximately equal size." Id. at 829. For a more precise statement of the conditions required for error equalization, see Kaye, supra note 4.

26. See Kaye, supra note 1; infra note 26.

27. Although many derivations are available in the literature, it may be convenient to provide a version tailored to make this point. Let p be the probability that the true facts are such as to warrant recovery for plaintiff, and let L be the loss associated with a false positive (a verdict of liability when the facts do not warrant recovery) as well as that of a false negative (a verdict of no liability when the facts do warrant recovery). Because 1-p is the probability that the facts do not warrant recovery, and the expected loss is the loss discounted by the probability that it will occur, the expected loss of a verdict for plaintiff is (1-p) L. Likewise, because p is the probability that the facts justify a verdict for plaintiff, the expected loss of a verdict for defendant is p L. Thus, the expected loss of a verdict for defendant is greater than that of a verdict for plaintiff when pL > (1-p)L, i.e., when p > . In all these cases, a verdict for plaintiff minimizes the expected loss. Similar reasoning shows that in all cases in which p < , a verdict for defendant minimizes expected loss. Hence, the p > rule readily follows from the assumption of the equality of the costs of false positive and false negative errors. (The case of p = has no effect on minimizing expected losses; some supplementary argument, such as avoiding transaction costs or a preference for the status quo, is required to choose between a p> rule and a p rule.)

28. As this phrase suggests, one might argue that if potential plaintiffs and defendants were to choose, behind a veil of ignorance, between an expected-error-minimizing and an expected-error-equalizing burden of persuasion, they would adopt the former. See Ronald Dworkin, Taking Rights Seriously 150 (1977); John Rawls, A Theory of Justice (1972).

updated 9 March 2000