Bayes, Burdens, and Base RatesBy D.H. Kaye
Regents' Professor, Arizona State University
[-260-] In 1997, Professor Ronald Allen wrote:
Evidence has also experienced the demise of legal theorems. The best example is the proofs that employing the civil burden of persuasion of a preponderance of the evidence will minimise or optimise errors. These are all false as general proofs (although not as special cases), and all for the same reasons. They neglected base rates and the accuracy of probability assessments of liability, and virtually any relationship at all can exist between subjective assessments of liability and the truth of factual assertions at trial.1
Sandwiched around this claim were analogies to Darwin and Newton, and talk of the tension between 'algorithms' and 'formalisms' on the one hand, and 'judgment', 'justice', and 'equity', on the other.2
When I observed that the proofs establish that a Bayesian decision rule minimises expected losses 'for all possible base rates',3 Professor Allen wrote that I was 'fascinat[ed] with algorithms',4 'blind . . . to the deeper implications of the work' that I had cited,5 and that even though I teach evidence law, I did [-261-] not recognise the possibility that a juror may misjudge the probability of a disputed set of facts.6 Ad hominens aside, he dismissed my correction of his claim about 'base rates' as 'astonishing'7 and, relying on a numerical example, persisted in the view that '[t]he base rates and the particular probabilities have to be in particular relationships in order for any rule to minimise expected losses'.8
In the latest iteration of his self-proclaimed 'devastating' critique,9 Professor Allen further complains that I have 'convert[ed] what is obviously a true statement about reality into an erroneous statement about mathematics', quoted him out of context,10 made an 'old philosophical error of a category mistake',11 and manufactured 'a silk purse out of a pigskin'.12 He insists that he made no 'mathematical mistakes'13 because he was speaking of 'the real world',14 and he recites a litany of well-known objections that continue to divide (but not to devastate) philosophers interested in inductive proof and scientific theory.15 For a recent effort by a noted epistemologist to apply Bayesian ideas to legal fact-finding, see Goldman (1999).16
[-262-] Despite his initial expressions of astonishment that I would write that a decisionmaker's expected losses are minimized for all possible base rates, Professor Allen now asserts that he never meant to deny that the preponderance-of-the-evidence standard always minimises the total number of expected errors in verdicts:17 his expressions of incredulity were only meant to say that the p > .5 rule does not necessarily minimise someone else's expected loss function, that person being an observer who knows more than the jurors do about the probabilities in each case. Read this creatively, the claim is true--but trivial. That jurors can make mistakes in their 'subjective assessments of liability' signals no 'demise of legal theorems',18 and it hardly suggests that jurors should not be instructed to return a verdict for plaintiff if and only if the jurors' best judgment of the odds that the facts are such as to create legal liability favours the plaintiff.
What then separates the views that Professor Allen has presented about the burden of persuasion from mine? We agree (and always have) that if juries are not very good at estimating the probabilities of events, then making decisions in accordance with their probability judgments is not guaranteed, even in the long run, to produce the smallest number of factually erroneous verdicts. The crucial issue is what to do about the ineluctable risk of error. I believe that the law's best strategy is to formulate rules of evidence that promote the accuracy of factual determinations and to encourage triers of fact to decide in accordance with their best-informed and best-justified assessments of the probabilities that the totality of the material facts is such as to warrant recovery. Unless jurors are systematically wrong in their assessments of the probabilities--'miscalibrated' is the technical term--then this decision rule not only minimises expected errors as seen by the triers of fact, but also the actual errors as seen by an omniscient observer. Whether jurors are well-calibrated over all types of civil cases and parties is an empirical question to which I have no definitive answer. In the absence of any data, I would let jurors make their best judgments of the odds and return their verdict in the way that would let them be right more often than wrong if those assessments are on the correct side of the 50-50 threshold.
[-263-] Before engaging in a debate with Professor Allen, I had thought this position to be boringly uncontroversial--except for his brilliant but ultimately unpersuasive exposition of a 'relative plausibility' theory of civil litigation.19 In his more recent work, however, Professor Allen paints with a broader brush, dismissing 'the Bayesian decision rule as a sensible explication of current law, . . . for the law is deeply inconsistent with it . . .'.20 Unfortunately, his arguments for this negative assessment are either trite, superficial, inconclusive, or not directed to the decision rule itself. For example, he writes that 'a decision-maker following the prescriptions of the formalisms Professor Kaye discusses could go merrily along its way doing whatever it liked with respect to expected utility and making nothing but mistakes'.21 He thinks this is so because '[t]he subjective states of mind relevant to decision theory are radically subjective in that the only relevant question is the state of mind of some person' and 'the connection of this state of mind to the outside world is irrelevant, and indeed could easily be random'.22 This is a repetition of an attack advanced by L. Jonathan Cohen.23 It fares no better in [-264-] Allen's hands.24 Whether partial beliefs should be 'coherent' in the sense prescribed by the axioms of subjective expected utility theory is one question. Which partial beliefs one should hold out of the many coherent possibilities is another question. To confuse a question of logic with a question of epistemology is a category mistake. There is no inconsistency in maintaining that coherence is a property worth striving for, but that it is not a sufficient criterion for legal fact-finding.25
Another category mistake lurks in the response to my suggestion that Bayesian decision theory could support the conclusion that it may be best to have jurors use imperfect heuristics and shortcuts (roughly analogous to rounding off and performing mental arithmetic instead of labouring with long division) to form their partial beliefs.26 According to Professor Allen, this approach 'encourages widely idiosyncratic responses to the evidence that are constrained only by the fact-finder's imagination and a mild requirement of consistency . . .'.27 But nothing in the suggestion that jurors might be well advised to assess the probability of a complex proposition without decomposing it into more elementary propositions and examining their links to every piece of evidence encourages jurors to be irresponsible. In this context, Bayesian decision theory is a meta-decision procedure. Together with background knowledge, it generates a conclusion as to how jurors should go about arriving at the probability of a complex proposition. Also operating at a meta-level, it generates a decision rule for returning a verdict once the probability of the dispositive proposition is in hand. Because decision theory applies at this higher-order level, it is misguided to demand a complex and [-265-] realistic example of how individual jurors would benefit from calculating subjective expected utilities.28 Any such example would apply at a lower level at which the theory demands no such computations.29 The Bayesian analysis merely explains why applying the p > .5 rule to the best available probabilities is a sensible lower-order decision rule. Likewise, to use probability theory, the propositional calculus, or other tools of mathematics or logic to analyse rules of evidence or the reasoning of counsel or courts does not mean that lawyers should submit briefs replete with equations.
Professor Allen also is chagrined at '[t]he total failure of those writing from Professor Kaye’s formalist perspective to address the actual conditions of trial . . .' .30 Over the years, I have written about empirical research on the jury system, the reactions of actual jurors to missing evidence, the inferences that jurors might draw from unusual accidents, the effects of jury size on verdicts, the differences between scientific and legal proof, jurors' understanding of probabilities, and the problems of presenting scientific evidence to jurors.31 Perhaps Professor Allen does not count these discussions because not all were written from the 'formalist perspective'. Although I do not consider myself to be a 'formalist' in the sense that Professor Allen uses the term, I would not agree that this alleged failure 'constitutes substantial evidence that the formalists have nothing to say about the matter, that even they do not see how their formalisms map onto trials in any significant way'.32 The mapping is this: If minimising expected loss is the criterion that the legal system desires, and if the loss is the same for errors that favour plaintiffs as it is for errors that favour defendants, then in simple cases of a single plaintiff and defendant, the p > .5 rule is optimal. If minimising actual errors is the criterion, and if actual jurors are well-calibrated, then the same rule is optimal.33 These 'formal' results apply to 'the actual conditions of trial'.
[-266-] Knowing this much seems significant to me, but this is a question of taste. Since no one really knows whether jurors are well-calibrated and how they depart from Bayesian and epistemic ideals in the many situations that can arise in the courtroom, the drafters of legal rules must make their best guess as to which rule (or rules) will minimise actual errors (or fulfill any other value that should govern the trial process). If Professor Allen is right, then even though 'the system of litigation has been experimenting with thousands, probably hundreds of thousands, of instructions for centuries',34 it should abandon or modify these approaches to implement his relative-plausibility principle. If I am right, then there is some small benefit to looking at these approaches with the mathematics of expected values to understand or to criticise some of these instructions.
Professor Allen is at liberty to dismiss studies of burdens of persuasion that ask whether these rules are or should be structured to minimise expected losses as 'uninteresting'35 or unconvincing.36 But, on the arguments he has collected, he is not entitled to reject them as 'deeply inconsistent' with the operation of the legal system or to misrepresent their contents.37 He is free to contend, as have many before him, that importing the full machinery of Bayesian decision theory into trials is too complicated or too esoteric to advance the trial process.38 One can offer a Bayesian analysis of the burden of persuasion without endorsing these or any other proposals to encourage [-267-] jurors to reason in an explicitly Bayesian fashion.39 In some contexts, however, I have argued in favour of allowing Bayesian computations to be put before jurors. This is not because of any 'formal' commitment to make jurors into Bayesians, but rather to assist jurors in appreciating the probative value of particular scientific evidence.40 In short, I am pleased to learn that Professor Allen and I always have agreed about the properties of decision rules.41 Furthermore, I appreciate his invitation to supply a detailed refutation of each and every point that he can raise or repeat. However, I trust that I can decline this invitation without again being pilloried as 'blind to . . . deeper implications' and having 'nothing to say about . . . fundamental matters'.42
* I am grateful to Ronald J. Allen for providing me with a prepublication copy of the paper discussed here and to Brad Armendt for comments on my previous rejoinder to Professor Allen.
1. Ronald J. Allen 'Rationality, Algorithms, and Juridical Proof: A Preliminary Inquiry' (1997) 1 E & P 254 (hereinafter Allen I).
2. Ibid. at 255.
3. D.H. Kaye, 'Statistical Decision Theory and the Burdens of Persuasion: Completeness, Generality, and Utility' (1997) 1 E & P 313 at 314 (hereinafter Kaye I).
4. Ronald J. Allen, ‘Reasoning and Its Foundation: Some Responses' (1997) 1 E & P 343 at 345 (hereinafter Allen II).
5. Ibid. at 347.
7. Ibid. at 346.
9. Ronald J. Allen, 'Clarifying the Burden of Persuasion and Bayesian Decision Rules: A Response to Professor Kaye' 4 E & P 246 (hereinafter Allen III).
10. This complaint has no bearing on the merits of Bayesian decision theory as a device for understanding the civil burden of persuasion, and I do not wish to burden readers of this journal with a tedious refutation of the charge. For the reasons given at http://homepages.law.asu.edu/~kayed/pubs/evid/context.htm, however, I do not think it has much basis in fact.
11. The 'philosophical error' appears to consist of my failure to recognise that Allen was using words ike 'expected loss' in an idiosyncratic way. For Allen to denominate an alleged mistake of interpretation as a 'category mistake' also is peculiar.
12. Allen III, above note 9.
14. Ibid. ('Transposing my comments about the real world into the mathematical world that Professor Kaye wishes to discuss does indeed transmute them into 'sheer fantasy', as he alleges . . .').
15. Ibid. See, e.g., Patrick Maher, Betting on Theories (Cambridge University Press: Cambridge, 1993). ndeed, Professor Allen relies heavily on John Earman, who is critical but not dismissive of the Bayesian effort. 'The upshot of my xamination of Bayesian conformation theory', Earman writes, 'is neither a simple thumbs up nor a simple thumbs down.' John arman, Bayes or Bust? A Critical Examination of Bayesian Confirmation Theory (MIT Press: Cambridge, 1992) 5. After observing that '"Bayesianism" is . . . a leading school of statistics and . . . arguably the predominant view among philosophers of science concerning the confirmation of scientific hypotheses and scientific inference in general', ibid. at 1, Earman adds:
Bayesianism is the only view presently in the offing that holds out the hope for a comprehensive and unified treatment of inductive reasoning . . . Whatever one's ultimate decision about Bayesian confirmation theory, it possesses the one unmistakable characteristic of a worthy philosophical doctrine: the harder it is pressed, the more interesting results it yields.
16. A. I. Goldman, 'Truth and Testimony: Evidence in the Law' (paper presented at the University of North Carolina Conference on Epistemology and Legal Evidence, September 1999).
17. He professes that his understanding of the mathematical proofs found in the legal literature is--and always has been--that they establish that the p > .5 rule minmises expected losses as that phrase is used by all statisticians and decision theorists, and that this is true regardless of base rates and calibration problems.
18. See above text accompanying n. 1.
19. Professor Allen explains that he has 'argued, in opposition to the formalists such as Professor Kaye, that decision is reached at trial; it is reached through the comparison of the relative plausibility of the stories advanced by the parties rather than through application of any formalistic tools to the plaintiff's (or state's) case'. Allen III, above n. 9. Although I have never maintained that, as a general matter, jurors do or should reach their decisions 'by application of any formalistic tools' (see below), I am not enamored of Professor Allen's 'relative plausibility' theory. This approach differs from the p > .5 rule in that it would allow recovery in some cases in which p < .5 because the defendant's 'story' does not include some alternatives with non-zero probability. For a critical analysis of the premises and consequences of the 'relative plausibility' theory, see R. Lempert 'The New Evidence Scholarship: Analyzing the Process of Proof' in P. Tillers and E. D. Green (eds) Probability and Inference in he Law of Evidence: The Limits and Uses of Bayesianism (Kluwer Academic Publishers: Dordrecht, (1988) 61 at 80-87 (concluding at 83 that 'more than any other recent writer on this topic, Professor Allen is willing to let some formal model of rational decisionmaking dominate our views of how rials should proceed'). Judge Richard Posner also rejects Allen's thesis (but without any real analysis of it) in favour of the onventional view that even when plaintiff has the more probable story, 'plaintiff should lose because he has failed to prove that is story is more likely than not true.' R.A. Posner, 'An Economic Approach to the Law of Evidence' (1999) 51 Stanford Law eview 1477 at 1513.
20. Allen III, above n. 9.
22. Ibid. Of course, people whose beliefs are but randomly related to the real world will not last long in this world.
23. L. J. Cohen 'The Role of Evidential Weight in Criminal Proof' in P. Tillers & E. D. Green (eds) Probability and Inference in the Law of Evidence: The Limits and Uses of Bayesianism (Kluwer Academic Publishers: ordrecht, 1988) 113 at 125 ('A subjectivist interpretation is inherently incapable of' elucidating 'what reasonings are appropriate to jurors and advocates' because 'on a subjectivist interpretation each person's judgment of the probability . . . does not more than describe that person's own state of mind').
24. Grandiose pronouncements, like 'formal methods of proof cannot substitute for substantive knowledge in any juridical system constructed in the light of western, post-enlightenment thought', Allen III, above n. 9, add nothing of value to the argument. As explained in D. H. Kaye, 'Clarifying the Burden of Persuasion: What Bayesian Decision Rules Do and Do Not Do' (1999) 3 E & P 1 (hereinafter Kaye II), knowledge is as vital in ascertaining the pertinent probabilities in law as it is in any other actual application of decision theory. No one claims that wise decisions can be achieved simply by manipulating symbols, and I am a loss to understand why Professor Allen is so intent on dismissing efforts to reason rigorously about one small facet of legal decision-making as a useless and empty 'formalism'.
25. No one familiar with the literature on the foundations of probability and decision-making could conclude that Bayesians are all radical subjectivists, and no one who has read my previous writing could believe that I advocate a radical subjectivist theory of decision-making. See D. H. Kaye 'Do We Need a Calculus of Weight to Understand Proof Beyond a Reasonable Doubt?' in P. Tillers & E. D. Green eds) Probability and Inference in the Law of Evidence: The Limits and Uses of Bayesianism (Kluwer Academic Publishers: Dordrecht, 1988) 129 at 137 ('I agree with Cohen (and many other philosophers) that we ought to demand more than this.'); D. H. Kaye 'Paradoxes, Gedanken Experiments and the Burden of Proof: A Response to Dr. Cohen's Reply' (1981) Arizona State Law Journal 635 at 644 ('the notion that subjective probabilities are degrees of belief nd that such "partial beliefs" can be rationally justified is nothing new').
26. Kaye II, above n. 24, at 21; cf. V. Walker 'Review-Essay' (1999) 39 Jurimetrics Journal 394 (describing logic and probability as a tool that may be more useful for some tasks than others).
27. Allen III, above n. 9.
28. Ibid. (complaining that the didactic example of taking an umbrella in case it rains is too simple to model the decision that a jury must make and that '[Kaye] cannot even come up with a remotely realistic example to demonstrate any of his points').
29. The situation is analogous to the difference between act utilitarianism and rule utilitarianism. Act utilitarianism requires moral agents to weigh the expected utilities of each act to decide whether that act is right. As such, it is subject to the criticism that this demand is unrealistic. Rule utilitarians use the principle of utility differently. They argue that the principle of utility should be used to generate a set of more specific rules that can be applied to approximate the results that would be attained by act utilitarians with sufficient time and computational resources. Whatever one thinks about rule utilitarianism (or the facet of 'Bayesianism' that I have defended), to criticise it on the ground that there are no 'remotely realistic' examples of individuals performing a full utilitarian calculation (or of fact-finders using the full-scale apparatus of Bayesian inference and decision) is to commit a category mistake.
30. Allen III, above n. 9.
31. See, e.g., H. Zeisel and D. Kaye, Prove It with Figures: Empirical Methods in Law and Litigation (Springer-Verlag: New York, 1997).
32. Allen III, above n. 9.
33. See Kaye II, above n. 24.
34. Allen III, above n. 9.
35. See ibid., n. 19.
36. See ibid., where Professor Allen writes that an unresolved 'formal problem' with the decision theoretic explanation of the p > .5 rule is that 'the burden of persuasion instruction that [Kaye] (mistakenly) defends is [not] universal', and '[i]nstructions on burdens of persuasion are enormously complicated, with many direct variations and even more indirect variations imposed through presumption and inference instructions.' A rather different characterisation of the rule can be found in Ronald J. Allen 'Burdens of Proof, Uncertainty, and Ambiguity in Modern Legal Discourse' (1994) 17 Harvard Journal of Law and Public Policy 627 ('The nearly universal standard in civil cases requires the person bearing the burden to establish the relevant elements to greater than a .5 probability.').
37. For instance, he has written that my work has shown that the p > .5 rule produces an equal or symmetrical distribution of errors over plaintiffs and defendants. See Allen, above n. 36 at 634, 641; R. Allen et al., Evidence: Text, Cases, and Problems, 2nd edn (Aspen Law and Business: New York, 1997) 828. In reality, my work has shown that the rule produces a distribution of errors that can be grossly lopsided--for the distribution does depend on the base rates--but that nevertheless minimises the expected sum of the errors of both types--whatever the base rates may be.
38. Some statisticians have proposed that jurors be lectured in probability theory to handle evidence that lends itself to quantitative characterisations. E.g. D. Berry 'DNA, Statistics, and the Simpson Case' (Fall 1994) Chance 9. Similarly, some decision theorists have urged that the burden of persuasion be articulated in quantitative terms so that jurors might consult a 'probability wheel' to test their global assessments of the relevant probabilities and verdict. H. D. Saunders & J. G. Genser 'Trial and Error' (September/October 1999) The Sciences 18. For criticism of such efforts to make the decision-theoretic framework a part of the trial process, see L. Tribe 'Trial by Mathematics: Precision and Ritual in the Legal Process' (1971) 84 Harv LR 1329.
39. See, e.g., Lempert, above n. 18, at 64-65; R. Lempert 'Modeling Relevance' (1977) 75 Mich. LR 1021.
40. See, e.g., I. M. Ellman and D. Kaye 'Probabilities and Proof: Can HLA and Blood Group Testing Prove Paternity?' (1979) 54 New York University Law Review 1131; D. H. Kaye, 'DNA Evidence: Probability, Population Genetics, and the Courts' (1993) 7 Harvard Journal of Law and Technology 101.
41. Less enlightening is Professor Allen's proclivity for shedding straw men. See Allen III, above n. 9 ('[Kaye] is arguing that jurors would do well to employ Bayes' theorem in their reasoning about evidence').
updated 22 October 2001