Click to return home

The Problem of Nonresponse:
Remarks on a Recent Survey
of Clinical Education

D.H. Kaye(1)


Note: This comment is published in the Journal of Legal Education, Volume 43, pages 594-595, Dec. 1993. A reply from the authors of the study that is criticized appears at pages 596-597.


The Journal of Legal Education recently published a "Report of the Committee on the Future of the In-House Clinic."(2) Section two, the "Report of the Data Collection Subcommittee," describes the results of a 1987 survey of 175 law schools or clinicians at these schools. These findings should be of interest to legal educators, but if one were to judge by methodology alone, it would not be obvious that they provide an accurate picture of modern clinical legal education.

The committee attempted to obtain a stratified random sample of law schools,(3) but it received responses from only 40% of the schools it contacted. The nonresponse rate of 60% poses a serious risk that the resulting sample is biased in unknown ways.(4) For instance, it could be that the schools choosing to respond tend to have better clinical programs (that they are happy to publicize) or to differ in other respects that are related to the objectives of the study. Self-selected samples often differ from the population in unexpected ways.(5)

The mere fact that the relatively few responding schools seem roughly similar to the population of law schools in two respects(6) is hardly sufficient to dispel this concern. Two examples, one old and one new, show that such similarities offer no sure protection against bias. In the late 1920s, two prominent Italian statisticians, Corrado Gini and Luigi Galvani, faced the problem of deciding which few returns from the 1921 Italian census should be preserved while the remainder were discarded. They choose to retain returns from districts that were representative of the country's average on seven variablesand discovered too late that those districts were not representative of the country as a whole on other variables.(7)

More recently, a behavioral researcher sent out 100,000 questionnaires to explore how women viewed their relationships with men.(8) She amassed a huge collection of anonymous letters from thousands of women disillusioned with love and marriage. Recognizing that the response rate was very small (although the sample was huge), she defended the sample as representative because "those participating according to age, occupation, religion, and other variables known for the U.S. population at large in most cases quite closely mirrors that of the U.S. female population."(9) Nevertheless, the results from this sample differ dramatically from those in more scientific polls on the same topic with better response rates.(10)

When the response rate to a survey of law schools is low, the best strategy is to make it highto contact the nonrespondents and convince them to cooperate. Although there are statistical procedures to impute missing data when this strategy fails,(11) the next best approach probably is to present the survey for what it isa description of a predominantly self-selected sample.(12) Although large non-response rates may not keep magazines like U.S. News and World Report from incorporating survey data in its dubious rankings of law school, committees and subcommittees of legal educators should be more demanding in their surveysor more guarded in their descriptions of them.

Notes

1. D.H. Kaye is Regents' Professor, College of Law, Arizona State University 85287-7906. [BACK]

2. 42 J. Legal Educ. 508 (1992) [hereinafter Report]. [BACK]

3. Id. at 519 n.2. [BACK]

4. See, e.g., David Freedman, Robert Pisani & Roger Purves, Statistics 304 (New York 1978) ("Nonrespondents can be very different from respondents. When there is a high nonresponse rate, look out for nonresponse bias."). [BACK]

5. The subcommittee remarks that "the size of the sample obtained gives some confidence that the results are not likely to be too prone to error." Id. However, "[w]hen a selection procedure is biased, taking a large sample doesn't help. This just repeats the basic mistake on a larger scale." Freedman, Pisani & Purves, supra note 3, at 303. Because the non-response problem is so fundamental, I shall not belabor the subcommittee's misinterpretation of the 0.05 significance level as establishing "95-percent [certainty] that the result could not be the product of chance." Id. at 523. [BACK]

6. These are size (small, medium or large) and ownership (public or private). Report, supra note 1, at 519. [BACK]

7. C. Gini & L. Galvani, Di una Applicazione del Metodo Rappresentive All'ultimo Censimento Italiano Della Popolazione (10 Decembri, 1921), 4 Annals di Statistica (Ser. 6) 1 (1929); C. Gini, Une Application de la Méthode Répresentative aux Materiaux du Dernier Recensement de la Population Italienne, 23 Bull. Int'l Statistical Institute 198 (1928), described in Judith M. Tanur, Samples and Surveys, in Perspectives on Contemporary Statistics 55, 58 (David C. Hoaglin & David S. Moore eds., Wash. DC, 1992). [BACK]

8. Shere Hite, Women and Love: A Cultural Revolution in Progress (New York 1987). [BACK]

9. Id. at 777. [BACK]

10. See Chamont Wang, Sense and Nonsense of Statistical Inference: Controversy, Misuse and Subtlety 176 (New York 1993). [BACK]

11. E.g., U.S. Dep't of Commerce, Bureau of the Census, Imputation and Editing of Faulty Survey or Missing Survey Data (F. Aziz & F. Scheuren eds., 1978); Donald B. Rubin, Multiple Imputation for Nonresponse in Surveys (1987). [BACK]

12. As such, there is little point in performing a chi-square or any other kind of test to establish the trivial hypothesis that all schools do not perceive all problems of in-house clinics as equally challenging. Report, supra note 1, at 523-24. [BACK]