Ranking Legal Periodicals and Some Other Numeric Uses
of the Westlaw and Lexis Periodical Databases
John Doyle
Associate Law
Librarian,
Summary.
This article is an excursion into some arcane realms of the JLR and ALLUS databases on Westlaw and Lexis and their use for extracting statistical information about legal periodicals and articles. Aside from various issues such as citation decay, footnote density, and prolix authors, this article delves into the prestige of legal periodicals which are reflected by (and encouraged by) ranking studies, and in particular the Washington and Lee citation count study which is based on citations from the JLR and ALLCASES databases.
Keywords. Legal periodicals, law reviews, ranking, Westlaw, Lexis.
Introduction
This article is an exploration of the opportunities and
difficulties in extracting numeric information from the full-text legal
periodical databases on Westlaw and Lexis.
Comments are offered as to legal periodical rankings, and on some
numerical aspects of legal periodicals such as, footnote usage, article size,
and rates of decay in citations to older articles. Appended is a copy of the
Criticism of Law Reviews
Baiting law reviews has been a sport for generations. George P. Costigan, Jr., then Editor-in-Chief of the Illinois Law Review, said in 1915 that "the multiplication of law reviews connected with law schools has been due largely to the desire of those schools to advertise themselves as wide awake...."[1] Harold Havighurst, Dean of Northwestern University School of Law, said in 1956 that "law reviews are unique among publications in that they do not exist because of any large demand on the part of the reading public. Whereas most periodicals are published primarily in order that they may be read, the law reviews are published primarily in order that they may be written."[2] And for an Australian viewpoint in 2002, consider: "Law reviews have become the public face of an unpleasant and inappropriate form of academic life that degrades scholars, wastes valuable time and money and devalues the importance of good teaching and collegiality in law schools."[3] Given this kind of criticism, what value do individual journals provide, whether printed or electronic, and in particular, what value do the general law reviews provide? If we see an advantage to a plethora of generalized and specialized journals it must substantially lie in the gate-keeping function of article selection. It's easier to justify the existence of subject oriented journals, whose editorial staff will likely have an above average interest in the subject matter of submitted articles, with a market of specialized practitioners that may even be eager to see the next issue, and with an archival function of storing like articles together. It is unfortunate that, in large part for historical reasons of status, the specialized journals are not generally cited as often as are the general law reviews. Examples where a law school's specialized journal is cited more often than the law school's general journal are quite rare. One instance is the John Marshall Journal of Computer & Information Law, and the John Marshall Law Review. There are no specialized journals in the top-30 journals in the W&L study, and only 5 in the top-50. The gate-keeping function of duplicative journals is simply based on prestige differences--that it's considered better, for example, to publish in the Columbia Law Review than in most any other journal except Harvard Law Review or Yale Law Journal. Thus, it's possible to tell something about the probable quality of an article by the journal in which it's published. This is useful filtering information, especially in an online database search and retrieval environment where, unless author prominence is known, the numerous articles obtained as the result of a search may immediately be distinguishable by nothing other than their date of publication.
Citation Count Ranking of Legal Periodicals
Seemingly there is no end to the published literature on citation counting and other methods of ranking legal periodicals. References to a few of these are given here, but no extensive analysis is attempted. Finet in 1990 in the Legal Reference Services Quarterly[4], and Brown in 2002 in the Law Library Journal[5] surveyed the literature of legal periodical citation ranking and many of the general citation studies are referenced there. Some other studies are, Joergensen[6] in 2002 (second tier law reviews), DeLeve[7] in 2003 (seven mid-south law reviews), and Manz in 2002[8] (U.S. Supreme Court citations - containing a useful bibliography of court citation studies).
The methodology of the two survey articles of Finet, and Brown, mentioned in the previous paragraph, lists the rankings from previous studies, and then uses each journal's rank number from the various studies to arrive at an average ranking for each journal. Attempts to average such various rankings are problematic because they inter-mingle court and journal citation studies, and because the citation studies have been published at various dates with differing inclusions of cited journals. As an example, Hofstra Law Review is ranked 29th in Brown’s study which, based on the W&L study is a doubtfully high ranking. A defect in such an averaging methodology is the lack of accumulation into a journal’s average score when one of the averaged ranking studies does not include the journal. For most of the utilized rankings, if a journal isn’t included, then it must have a worse ranking than all the others that are included. For example, Hofstra Law Review isn't in the top-30 general law reviews of Shapiro’s ranking in The Most-Cited Law Reviews (2000),[9] Shapiro must therefore rank it 31st or lower, but this piece of ranking information is not utilized to reduce the rank of Hofstra Law Review in Brown’s 2002 survey.
Another problem with mingling case citation studies and
journal citation studies is that cases cite to journals nowhere nearly as
frequently as do journal articles. So
that in studies of citations from cases to journals there may be many journals
that are each only cited once although it seems statistically insignificant if
a journal is cited either on zero or on one occasion. Such an averaging study
may generate a higher ranking for a journal with zero citations to it (where
the journal is not then included in one or more of the underlying studies),
than it will for a journal that has one citation (and therefore may be ranked
low in a study). However, more
frequently the case citation studies will result in only a small number of
citing references, particularly if the survey covers a limited number of
journals, or a short date period, and it's easy for a journal to gain a higher
ranking than would generally be justified. For example, in Daniels’[10]
study of citations from U.S. Supreme Court (1978 term) cases, a number of
journals are ranked 25th with just 2 citations to each, but the 25th
ranked journal in Shapiro’s 2000 study had 1,005 journal citations to it. As a
relative comparison between how often journals are cited by cases and by
articles the W&L study (as at January 26, 2004) shows that U.S. journals
(1996-2003) are cited on average by
457 articles and by 14 federal/state cases, and that non-U.S. journals
(1996-2003) are cited on average by
31 articles and by 0.08 cases. Looked
at comparatively that data shows that
An interesting and
innovative use of journal rankings can be seen at HeinOnline.
In January 2004 HeinOnline introduced a new
search interface which makes available for separate searching three subsets of
their "Law Journal Library," being, "Most-Cited Law
Journals," "Core U.S. Law Journals," and "International
& Non-U.S. Law Journals." The
"Most-Cited Law Journals", contains, at the time of writing, a
searchable set of the top-27[12]
legal periodicals. Unlikely to be attempted, but it is possible to imagine a
search process being developed by a full-text search system that would allow a
searcher to limit results to the top-n journals (whatever number "n"
is chosen by the searcher to be).
Usage Count Ranking
of Periodicals
There are numerous reasons why a citation might appear in an
article, not all of them related to the intellectual quality of the cited work.
The higher the footnote (or citation) density in an article the more likely it
is that citations are of an informative, and not a supportive nature, and law
as a subject matter has a higher citation density than many other subjects. At
the extreme, the upper record for footnote usage is held by, Arnold S. Jacobs, An Analysis of Section 16 of the Securities
Exchange Act of 1934, 32
N.Y.L. Sch. L. Rev. 209 (1987) with 4824
footnotes. It is a useful function for
collections of citations to offer a repository of pointers to related research
but such usage diminishes the argument that citation counting can indicate how
often a cited article is actually used. One alternative to citation counting is
usage counting. Usage studies have been
done in libraries by counting volumes to be re-shelved, and by using reserve
room circulation statistics. Aside from
any physical problems in accurately recording in-house use, a sizeable problem
of bias is introduced by a study of usage from only one library. As an example, the
Aside from counting citations and usage, legal periodicals
have been ranked by author prominence,[14]
by expert opinion, and no doubt, in everyday practice, by the prestige
associated with the journal’s publisher.
Gregory Scott Crespi has authored two similar articles that rank by
expert opinion the journals of, in one study, international and comparative law
(1997),[15]
and in the other, environmental law, natural resources law, and land use
planning (1998).[16]
Crespi’s method in both studies was to send a survey instrument to teachers
listed in the AALS Directory of Law Teachers who had taught any of the subjects
for at least 10 years. Taking the
international and comparative law study as an example, the respondents were
given a list of 88 journal titles in the subject area and asked to ideally
assign a numerical value (1 highest, 10 lowest), to the 10 journals that the
expert regarded as having the strongest academic reputation. Crespi assigned each journal 10 points for
first place in each survey down to 1 point for last place (it’s not entirely
clear from the methodology, but journals not listed in a respondent’s 10
choices were evidently assigned a score of 0). The results were then averaged
for each journal. The listing below
shows some ranking differences between the W&L citation study, and Crespi’s
expert opinion study. The numbers in
parentheses are the international and comparative law rankings in each
survey. For reasons stated below I do
not believe that the lower expert rankings are reliable, but the higher ranked
expert scores do say something interesting about prestige opinions. I have no cavil with the American Journal
of International Law, which is the premier journal. Fordham International Law Journal,
which is very heavily cited, we are told is all but ignored by the
experts. The American Journal of
Comparative Law is ranked 2nd by expert opinion, while only 7th
in citations, no doubt because, in general, comparative law articles are cited
less often than international law articles, and it is the first ranking journal
in its own category of comparative law.
The journals ranked 3rd, 4th, and 5th
in Crespi’s study are interesting because they are the big-three of Harvard,
|
Journals |
W&L Study: 1996-2003 Journal Citations |
Crespi Study: Average of expert ranking scores |
|
Am. J. Int'l L. |
1737 (1) |
7.46 (1) |
|
Fordham Int'l L.J. |
1175 (2) |
0.54 (17) |
|
|
1061 (3) |
3.64 (6) |
|
Vanderbilt J. Transnat'l L. |
981 (4) |
1.51 (10) |
|
Am. U. Int'l L. Rev. |
922 (5) |
0.37 (24) |
|
|
898 (6) |
2.62 (7) |
|
Am. J. Comp. L. |
890 (7) |
6.09 (2) |
|
Harv. Int'l L.J. |
853 (8) |
4.7 (3) |
|
Int'l Law. |
850 (9) |
1.65 (9) |
|
|
739 (10) |
--- |
|
Colum. J. Transnat'l L. |
734 (11) |
3.93 (4) |
|
Yale J. Int'l L. |
695 (12) |
3.68 (5) |
There is a methodological problem in averaging the expert rankings because of the understandable limitation that the experts were only asked to rank their top ten choices, and not the other 78 journals. There is no problem with the top few journals which were probably found on everyone’s list, but journals that failed to appear in some of the respondent’s top-10 have an unreliable average because Crespi, in effect assigned a rank of 11th to all of the 78 journals that each respondent didn’t rank, despite the possibility, for example, that the expert might have estimated a journal's true reputation as more in the order of 14th, or 30th. Thus, had Crespi asked each expert to list their top-15, instead of top-10 journals, Crespi's published results would have been quite different.
Rankings and Article Submissions
The most widely used rough proxy for an accepted law review
ranking is the ranking of the law schools that publish the journals, which in
the recent past has been the popular law school rankings supplied by U.S. News & World Report[17] Although
there is broad agreement between the U.S. News law school ranking and the
W&L citation count ranking (e.g. Harvard Law School is 3rd with U.S. News,
and its law review 1st in citation counts, Syracuse Law School is 97th, and its
law review 111th), there are considerable departures from correspondence.
In a sense, it doesn't matter what prestige hierarchy is used, as long as there is one to filter articles. Russell Korobkin has said, in relation to law school rankings, that "the existence of rankings fulfills the purpose of rankings, without reference to what the rankings claim to measure or how well they measure it." (emphasis in original)[18] This is certainly true of legal periodical rankings, where it matters nothing to the system as a whole if the first ranked journal is the Harvard Law Review or the Nebraska Law Review, if the latter then the better articles will go there and the Nebraska Law Review will justify its prestige. Other than amusement, the primary purpose of a legal periodical ranking is to stand as a substitute for the quality assessment of articles, so that the quality of an article may be judged by the journal it's published in without needing a thorough evaluation of the article's scholarship. As a corollary, an author submitting an article is primarily looking for prestige and the readership that it brings, and must gravitate toward the higher ranked journals, simply because they are higher ranked. Thus any ranking will do, whether it's based on citations, use, author prestige, expert opinion, or institutional prestige, so long as it's generally accepted. The purpose of a ranking scheme is to create prestige, and in large part what it measures is prestige. This circularity can be seen in deciding as to the single factor scheme on which to base a ranking. Should a citation count study base its ranking on bare numbers of citations, or should it use an "impact factor" modification of those numbers based on a formula for dividing the number of citations by the number of pages, or articles, that each journal published. Some journals publish 2,000 pages a year and others 500 pages a year and an "impact factor" ranking may be fairer to all journals. A larger journal does have to attract a correspondingly larger number of well-written articles in order to maintain the quality of its output and if successful it will certainly be cited more often than a smaller journal of equal quality. The W&L study does not use "impact factor," and is thus not fair to smaller journals, but is looking instead to usage as the basis for reputation, and even though a contributing factor to prestige is volume output, a prestigious and voluminous journal is still prestigious.
Law journal rankings create status, status is hierarchical and often repressive discouraging the lowly from climbing the rankings. With an established, and comprehensive, ranking of journals, any author with a quality work to publish will naturally aim to publish it in, say, the top-30 journals, and therefore, being published in a top-30 journal becomes a guarantee of quality, with the vast bulk of journals having increasing difficulty in attracting any but pedestrian works. A circular process then swells the ranks of the numerous law journals that are not cited by the courts and garner no more than a few cites each year in the legal literature. On this basis, an accepted and comprehensive ranking system can be seen as pernicious, being a barrier to the entry of new journals and potentially killing-off existing low-ranked journals. Korobkin, in his excellent article “Ranking Journals: Some Thoughts on Theory and Methodology” (1999)[19] approves of the prestige-making endeavor believing that, it “can lead to competition among journal editors for that status. This competition can then be channeled in a way that encourages authors to produce more valuable scholarship than they might otherwise.”[20]
The notion of competition among editors has its disturbing side, in that behavior modification to manipulate a journal’s citation ranking may not be a positive feature. Without improving the intellectual quality of a general law review it is possible to increase a journal’s citation ranking by excluding less popular topics, such as wills, or workers' compensation, and including more federal, and constitutional topics.[21] Nevertheless, Korobkin’s argument that an accepted journal ranking would encourage authors to produce more valuable scholarship seems plausible. An author’s satisfaction in publication, often along with other rewards such as employment, salary, tenure, and promotion, relates to the prestige of the journal that publishes the article. Without a definite ranking, competition for articles is unfocused because an author will not know amongst a target group of journals if it’s more prestigious to publish in, for example, the Wisconsin Law Review or in the Houston Law Review. Absent that knowledge, an author will be unconcerned as to which out of a range of journals should best publish the article as no one who reads the article will know the prestige difference between the journals either. With widespread knowledge of a detailed ranking hierarchy there is more incentive for an author to improve the quality of an article submission in order to increase the chance of acceptance by what is known to be a more prestigious journal.
The process of article submission to the general law reviews could be made more efficient by a well-recognized journal ranking, particularly as authors and journals move more toward electronic submission. There is an obvious efficiency advantage to all concerned in articles being submitted electronically, but the ease with which authors can submit electronic copies of articles has overload consequences for article editors.[22] A "Law Review Electronic Submissions" web page at Northern Kentucky University School of Law (http://www.nku.edu/~chase/libesubmission.html) permits easy inclusion of multiple editorial e-mail addresses into one e-mail message. BePress (Berkeley Electronic Press) has an “ExpressO” service (http://law.bepress.com/expresso/), which, for a moderate fee, permits uploading of an article. BePress then submits electronically to the chosen law reviews, and for an extra fee even sends paper copies to law reviews not accepting electronic submissions. In early 2004 it's expected that an e-mail submission function will be added to W&L’s “most-cited” web site, that being a logical extension of the ranking arrangement, permitting authors to select for submission a range of journals that are all similar in rank. Electronic submission services can save authors considerable time and expense, and with little effort an author may submit an article to a couple of hundred law reviews. A solution to the problem of editorial staffs being overwhelmed with articles (when in many cases an author would really prefer to publish elsewhere) is for law reviews to adhere to a single-submission policy, but competition between law reviews largely precludes that avenue. Authors can increase the chance that an article will be published by one of a target group of law reviews by knowing the ranking of the law reviews, selecting a narrow range within the ranking, and informing the chosen law reviews that the article has simultaneously been submitted to only that limited number of journals. There are significant time-savings for law review staff in pre-selection by an author as to an appropriate and limited range of journals to which to offer an article. When given a short time-frame in which to respond to an offer, an author, in accepting such an offer, suffers little risk that a greatly superior offer will be subsequently received as the prestige values of the journals within the chosen law journal group will all be reasonably similar. Thus a consequence of the general acceptance of a detailed legal periodical ranking along with author self-selection, is an increased correlation between a journal's prestige and the quality of the articles that it publishes.
Too Many Articles and Too Many Legal Periodicals
Howard Denemark had these comments in 1996: “Many of the articles on library shelves will be of no use to anyone, but some articles will change the law. If we cannot predict which ones will make a difference—and we cannot—we should be very circumspect about asserting that there are too many legal articles in too many law reviews.”[23] As to there being too many articles, this seems an irrefutable argument. In early 2004 the total of all federal and state cases in Westlaw was a little over 7,620,000, with approaching 4 million of that total published after 1979. [24] These are huge bodies of information, but evidently manageable. By comparison, the articles in JLR amount to fewer than 10% of those 4 million cases.[25]
Over 30 journals (that began publication before 2000) listed
in the W&L study have had less than an average of one journal article in
JLR cite to them in each of the current eight years of the study. Commercial journals, even those with low
citations, will survive if the intended market proves their usefulness by
purchasing the journals in sufficient numbers.
Association publications, whether cited or not, have utility as a
membership tool, and are supported by their membership organization. Law school general and specialized journals
are supported by their institution (and alumni), for educational, and for
traditional and status reasons. There is no guarantee that low citation, and
low prestige, will lead to the death of a journal, although a widely used
ranking that continued to show that a journal ranked low would dry up
contributions and dispirit editors. Any
statement that there are too many legal periodicals is probably intended to say
that library’s must buy too many of them, but library purchases are on the
whole rationally conducted, balancing the cost of purchase and storage against
the use of the journals, and as yet the low cost of most legal journals ensures
their continued purchase. It would be
interesting to see a study of trends in paid subscriptions for all the
|
Year |
Harv. L. Rev. |
Yale L.J. |
S. Ill. U. L.J. |
Crim. L. Bull. |
Sec. Reg. L.J. |
|
1995 |
5204 |
3300 |
438 |
|
|
|
1996 |
5029 |
3300 |
496 |
|
|
|
1997 |
5454 |
3300 |
494 |
884 |
936 |
|
1998 |
4367 |
3300 |
461 |
788 |
947 |
|
1999 |
4574 |
3300 |
436 |
961 |
|
|
2000 |
4223 |
2705 |
426 |
886 |
900 |
|
2001 |
4013 |
2705 |
401 |
890 |
|
|
2002 |
3735 |
2677 |
398 |
825 |
899 |
|
2003 |
3491 |
2577 |
|
724 |
|
Subsidization is the prime reason for the survival of
non-economic marginal journals but since the mid-1990s there has been added to
the mix a no-cost alternative, the exclusively online journal. The W&L study lists 25
Comparing Citations
to General and Specialized Journals
Have citations to the big-three general law reviews Harvard Law Review, Columbia Law Review, and Yale Law Journal fallen off in recent years? The answer is "yes". The following table in column two shows the number of articles in JLR that cite to at least one of these three journals[26], over a number of rotating seven year periods (i.e. where both the citing and cited articles are dated from those seven years). Also shown is the total number of articles in the JLR database for those periods. Had the JLR database remained static in size, it would have been expected that the citations to the big-three journals would have been stable. However, Westlaw has been incrementing the size[27] of its JLR database and thus citations to these journals should have increased in recent years, there being more articles in each of the later seven year periods that could have cited the three journals.
|
Rotating 7 years in JLR |
Articles citing: Harv. L. Rev. Colum. L. Rev. Yale L.J. |
Articles citing: Am. J. Int'l L. J. Leg. Stud. Harv. J. Legis. |
Potential number of citing articles in JLR |
|
1992-1998 |
11,253 |
2,707 |
137,559 |
|
1993-1999 |
11,003 |
2,845 |
151,393 |
|
1994-2000 |
10,980 |
3,092 |
162,662 |
|
1995-2001 |
10,805 |
3,135 |
169,559 |
|
1996-2002 |
10,492 |
3,189 |
176,530 |
There is a 6.76% drop-off in citations to the big-three over these years, despite the pool of potentially citing articles having increased by 28.33%. It seems that these general law reviews suffer dilution with so many competitor publications in which authors may find an outlet for their work, including a burgeoning number of specialized publications.
For the comparable test on specialized journals[28], the publications, American Journal of International law, Journal of Legal Studies, and Harvard Journal on Legislation were chosen because they are cited frequently, and have had a regular publication schedule. The table above shows that citations to those specialized publications had a steady, though by no means spectacular growth. Dilution can be a factor for specialized publications as well, for example, there are presently over sixty U.S. academic journals that focus on international law, and many others specializing in associated subjects such as international trade, international environment, and human rights.
Size of the JLR (Journals and Law Reviews) Database
Although it is generally assumed
in this study that citing references from JLR are from
Many uses of the JLR database[29] as a bibliometric source have problems with the year-by-year size changes in Westlaw's database. The listing below shows, for each date period, the number of articles that Westlaw has coded in the JLR database to those dates (i.e. using the DATE field)[30] - the oldest documents in JLR being from the year 1939.
Total for 1939-2002 331,378
1939-1950 402 1971 246 1981 721 1991 11,476 2001 28,022
1951-1960 2,583 1972 152 1982 2,045 1992 13,607 2002 27,669
1961-1970 2,385 1973 252 1983 2,539 1993 17,042
1974 270 1984 3,652 1994 21,482
1975 257 1985 4,637 1995 22,649
1976 233 1986 5,662 1996 24,347
1977 203 1987 6,871 1997 24,988
1978 213 1988 8,187 1998 26,498
1979 198 1989 8,390 1999 26,550
1980 235 1990 9,715 2000 27,135
As can be seen, JLR has few articles for each of the years before 1981 and a relatively stable number of yearly additions since the mid-1990s. Not only might a citation study that relies on JLR find problems with thirteen thousand articles being added one year and twenty seven thousand in another year, but the changing nature of the types, and sources, of articles may also be a problem.
Decline in Citations to Older Articles
Richard Leiter[31] examined fifteen law reviews, looking at citation patterns from 1996 articles (after online full-text became available), and from 1976 articles (before online availability). His general observation was that "about 70 percent of all articles cited in any law journal article will come from the sixteen-year period immediately preceding the year the article is published"[32] Leiter found that only around 30% of legal journal citations are to journals that pre-date the citing article by more than sixteen years, with or without the availability of full-text research. Given that a similar decay occurred in pre-computer citations, his view was that "... computer-assisted legal research alone is not the cause for the relatively low use of pre-1980 law review articles by modern writers."
As a method for testing Leiter's finding we can look at how many articles, dated 1996 (one of Leiter's test years) cite to any volume of Columbia Law Review, and how many of the articles are citing to volumes from the prior-16 years (1980-1995) and how many to older volumes (1901-1979). This test will give an overlap when articles cite to both newer and older Columbia Law Review articles.[33] The result is that, of the articles from 1996 in JLR:
2158 articles cite to any volume of
303 articles cite to both older and newer volumes
1418 articles cite just newer volumes (1980-1995)
437 articles cite just older volumes (1901-1979 - Colum. L. Rev. began in 1901)
Thus, of the articles in 1996 that cite to Columbia Law Review, 34% cite to Columbia Law Review articles that were published more than 16 years previously, a result in accord with Leiter's findings.
Similar searches show that, out of the articles dated 1996 that cite any volume of Columbia Law Review, that 74% cite 1901-1990 vols., 53% cite 1901-1985 vols., 37% cite 1901-1980 vols., 30% cite 1901-1975 vols., 26% cite 1901-1970 vols., 21% cite 1901-1965 vols. Despite the decline, it still seems impressive that over a quarter of the use of a journal is accounted for by articles published more than 25 years prior to their citation, particularly as none of those articles were conveniently available online in 1996. It seems likely though, that Columbia Law Review experiences a slower decay in citations to its older volumes than occurs with less prestigious journals.
What citation patterns to articles can we see in years subsequent to article publication? To test this, a year-by-year search was conducted of a sample of law review article citations (Harvard Law Review, Yale Law Journal, California Law Review, Virginia Law Review, and Fordham Law Review), using volume-journal-year (e.g. 104 +1 yale l.j. +4 1994)[34] where the citations to those articles came from 19 major general law reviews. What can be seen from the following table, by examining each column, is that in the years that follow article publication (i.e. the citing year rows), annual citations to articles from the cited volumes/year climb to a peak within 2 or 3 years of publication, enter a declining plateau for perhaps 5 years, and decline further over the following 10 years or so to about 50% of peak annual citation use.
A complication is that the chosen 19 journals from which the citations come, have a fluctuating number of additions to JLR each year, 771 articles were dated 1983, whereas 1314 were added for 2002. There was a slow increase in JLR yearly additions for those journals during 1983-1993, and a relatively stable number of additions during 1994-2002 fluctuating around 1320 articles added each year. So, examining the 1982 volumes target column, it can be seen that the 1982 volumes are cited by 64 articles in 1983 and 17 articles in 2002, yet JLR has 70% more 2002 articles than it does 1983 articles. Thus the decay in citation to older articles is more severe than the table indicates. The final two columns (for 1993 and 1994 cited years) show the most reliable decay figures (albeit only for a little less than a decade), as the citing journals set have relatively stable JLR additions for the citing years from 1994 onwards.
|
Citing year |
Publication Year of Cited Articles |
||||||||||||
|
1982 |
1983 |
1984 |
1985 |
1986 |
1987 |
1988 |
1989 |
1990 |
1991 |
1992 |
1993 |
1994 |
|
|
1983 |
64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
1984 |
65 |
61 |
|
|
|
|
|
|
|
|
|
|
|
|
1985 |
37 |
68 |
78 |
|
|
|
|
|
|
|
|
|
|
|
1986 |
69 |
63 |
89 |
65 |
|
|
|
|
|
|
|
|
|
|
1987 |
42 |
62 |
72 |
88 |
97 |
|
|
|
|
|
|
|
|
|
1988 |
50 |
51 |
101 |
79 |
117 |
100 |
|
|
|
|
|
|
|
|
1989 |
37 |
65 |
85 |
90 |
102 |
118 |
87 |
|
|
|
|
|
|
|
1990 |
46 |
46 |
58 |
82 |
69 |
80 |
71 |
105 |
|
|
|
|
|
|
1991 |
29 |
53 |
68 |
59 |
83 |
82 |
68 |
112 |
73 |
|
|
|
|
|
1992 |
32 |
37 |
58 |
58 |
62 |
71 |
77 |
103 |
108 |
95 |
|
|
|
|
1993 |
23 |
42 |
44 |
58 |
47 |
50 |
49 |
76 |
70 |
60 |
80 |
|
|
|
1994 |
36 |
65 |
54 |
51 |
51 |
60 |
58 |
73 |
66 |
82 |
105 |
78 |
|
|
1995 |
47 |
43 |
55 |
47 |
54 |
54 |
63 |
90 |
69 |
81 |
84 |
98 |
116 |
|
1996 |
26 |
46 |
61 |
44 |
59 |
67 |
53 |
87 |
67 |
74 |
98 |
84 |
111 |
|
1997 |
30 |
38 |
50 |
45 |
58 |
64 |
57 |
84 |
63 |
61 |
70 |
92 |
91 |
|
1998 |
31 |
46 |
63 |
54 |
54 |
43 |
70 |
74 |
70 |
62 |
78 |
65 |
84 |
|
1999 |
26 |
38 |
47 |
38 |
35 |
55 |
52 |
78 |
60 |
57 |
70 |
63 |
68 |
|
2000 |
33 |
41 |
54 |
42 |
37 |
36 |
57 |
58 |
57 |
56 |
55 |
51 |
61 |
|
2001 |
25 |
33 |
31 |
42 |
37 |
40 |
42 |
62 |
52 |
48 |
57 |
55 |
59 |
|
2002 |
17 |
35 |
34 |
34 |
32 |
46 |
38 |
54 |
44 |
42 |
42 |
54 |
44 |
Impact of Online
Full-Text on Citation Counts
Because of the increasingly comprehensive full-text legal periodical databases on Westlaw and Lexis, and the ease with which articles may be found and retrieved as the result of word-searches, intuitively it seems that articles that are not included in Westlaw and Lexis are less likely to be cited. The expansion of the full-text legal journal databases (JLR totaled over 350,000 documents by the end of 2003)[35] leveled the playing field by making more articles both readily findable and easily retrievable. It seems likely that users of indexes, faced with the physical task of finding and retrieving articles, would be inclined toward limiting effort by investigating articles from the more prestigious of the journals.
As a simple method for testing if there is a drop-off in citations to articles that pre-date full-text availability on Westlaw and Lexis, the following table looks at citation counts to a number of major law reviews and to their three volumes before, and their three volumes commencing with the oldest volumes available in either Westlaw or Lexis.[36] In order to correct for citation decay over the years, the citations are from articles dated about 19 years after the middle of each date range, so for example citations to California Law Review volumes 67-69 (1979-1981) are from 1999 articles, and citations to volumes 70-72 are from 2002 articles. JLR does have about 4% more articles dated 2002 than dated 1998/1999, so it might be expected that the later published volumes would have approximately 4% more citations to them, but with the notable exception of Yale Law Journal the cited major law reviews show a far more sizeable increase in citations subsequent to Lexis/Westlaw availability. It would be more certain if Yale Law Journal had followed the trend, but the overall increase in citations to volumes just before and just after online availability makes the case that there is substantial benefit, even for the major law reviews, in having articles available full-text in online research services. It can be argued that with online search access at HeinOnline of articles that precede Lexis/Westlaw availability that this
alters the assumption that 1979-1981 articles preceded online search capacity. Whatever may be the situation after 2002, I believe that HeinOnline’s full-text search capacity would not have significantly impacted the citations in 2002 articles. Much of the research used in 2002 articles having been done in 2001, quite early in HeinOnline dissemination, and with every respect for HeinOnline’s useful service, Hein’s search engine was not then as sophisticated as it will no doubt become.
|
|
Pre-dating WL/LX vols. citations |
Included in WL/LX vols. citations |
Percentage Change |
|
|
v67-69 50 |
v70-72 91 |
82% |
|
Cornell Law Review |
v64-66 26 |
v68-70 73 |
181% |
|
|
v78-80 134 |
v82-84 211 |
57% |
|
|
v68-70 31 |
v71-73 68 |
119% |
|
Stanford Law Review |
v31-33 80 |
v35-37 195 |
144% |
|
Yale Law Journal |
v88-90 256 |
v92-94 235 |
(-8%) |
|
|
v78-80 69 |
v81-83 160 |
132% |
Joergensen in 2002[37] examined “second tier” law reviews, and the dates at which they became available on Lexis and Westlaw, determining that citation counts before and after such additions showed that “middle of the road journals are much more widely used when they are available online.”[38] It’s not necessarily intuitive that the same effect occurs with high prestige journals, as it can be speculated that knowledge disseminates more rapidly and more widely from top-ranked journals. However, as the table above shows, for these seven major law reviews there exists an average of 60% more citations to journal volumes published just after full-text availability on Lexis and Westlaw compared with volumes published shortly before Lexis and Westlaw availability.
Length of Articles
Beginning with the June 1995 issue Harvard Law Review began including in its "Information for
Contributors" paragraph this suggestion to authors: "This Review is
particularly interested in articles under 100 double-spaced pages (including
footnotes)...." It can be assumed from this that the editors felt a need
to curb authorial verbosity. How lengthy
are legal periodical articles, or for our purpose, how many words do they
contain? This cannot be determined from
searches in the JLR database. However,
the corresponding Lexis files are richer in field, or segment, information, and
Lexis does have a LENGTH segment that can be used to determine the number of
words (as defined by the Lexis computer program[39])
in law review articles.[40] The data below reports on length as the
number of words in an article, which is a little hard to grasp the meaning of,
so what do the number of words mean in terms of printed pages? Page size, white space, font type and size
will vary, so only a rough idea is possible, but looking at one article in Texas Law Review, two in Harvard Law Review and one in the George Washington Law Review and
comparing the page numbers in the printed journal with the number of words
reported for those articles in Lexis, shows respectively, 558, 475, 478, and
600 words per page. So just as a rough
rule-of-thumb let's consider that printed law reviews contain 500 words per
printed page. The following data, limited to 1995-2002, comes from the Lexis
LAWREV:ALLUS file "US Law Reviews and Journals, Combined."[41] The lengthiest article in the ALLUS file
(1995-2002), at 287,075 words (and 428 printed pages), is Louis Kaplow & Steven Shavell, Fairness Versus Welfare, 114 Harv. L. Rev. 961 (2001).[42] Although very large, it’s not likely to be
the lengthiest of law review articles. The article with the largest number of
footnotes (81
Number of words (greater than or equal to) in law review article
|
|
10,000 |
20,000 |
30,000 |
40,000 |
50,000 |
60,000 |
70,000 |
80,000 |
90,000 |
100,000 |
200,000 |
|
1995 |
5,502 |
1,947 |
634 |
230 |
91 |
45 |
28 |
17 |
9 |
5 |
0 |
|
1996 |
6,116 |
2,159 |
732 |
267 |
96 |
47 |
22 |
13 |
6 |
2 |
1 |
|
1997 |
6,608 |
2,388 |
703 |
253 |
108 |
48 |
26 |
13 |
7 |
4 |
0 |
|
1998 |
6,891 |
2,512 |
844 |
290 |
110 |
54 |
25 |
15 |
10 |
8 |
1 |
|
1999 |
7,344 |
2,583 |
883 |
337 |
125 |
58 |
25 |
11 |
4 |
3 |
0 |
|
2000 |
7,573 |
2,655 |
868 |
318 |
130 |
52 |
23 |
12 |
8 |
3 |
0 |
|
2001 |
7,419 |
2,620 |
854 |
285 |
115 |
67 |
37 |
19 |
14 |
11 |
2 |
|
2002 |
7,465 |
2,641 |
920 |
351 |
141 |
72 |
41 |
29 |
17 |
12 |
0 |
The number of articles with at least 10,000 words, as well as those with at least 20,000 words, have increased in most years, but as the number of articles varies that are added each year to Lexis, the year-by-year figures above are not entirely comparable. To correct for the changing database content the table below shows the number of law review articles available for each year[43], and what percentage of each year’s articles have at least 10,000 or 20,000 words.
|
|
Number of articles available[44] |
Number and percentage of articles with >= 10,000 words |
Number and percentage of articles with >= 20,000 words |
|
1995 |
9,887 |
5,502 (55.65%) |
1,947 (19.69%) |
|
1996 |
11,595 |
6,116 (52.75%) |
2,159 (18.62%) |
|
1997 |
12,354 |
6,608 (53.49%) |
2,388 (19.33%) |
|
1998 |
13,179 |
6,891 (52.42%) |
2,512 (19.06%) |
|
1999 |
14,015 |
7,344 (52.29%) |
2,583 (18.43%) |
|
2000 |
14,910 |
7,573 (50.79%) |
2,655 (17.81%) |
|
2001 |
14,538 |
7,419 (51.35%) |
2,620 (18.02%) |
|
2002 |
13,917 |
7,465 (53.64%) |
2,641 (18.98%) |
There is no discernable trend toward an increase or decrease in the average size of law review articles. Slightly over half of the law review articles added each year contain at least 10,000 words (approximately 20 printed pages) and almost one-fifth contain at least 20,000 words (approximately 40 printed pages).
Footnotes
Westlaw's JLR database can be used to determine the number of footnotes that occur in articles. A search like: FN397 & FN398 & FN399 & FN400 & FN401 & DA(<2003) shows the number of articles in the database, up to the end of 2002, that have more than 400 footnotes. Like most full-text searches in a large database some potential for inaccuracy is inevitable. The repetition in the search of consecutive note numbers[45] minimizes confusion from any isolated footnote references, but there are articles that have more than a single sequence of footnote numbering, and this search method would undercount their footnotes.
No. of No. of
Footnotes Articles
0-4 128,265
5-100 95,180
101-200 58,921
201-300 31,131
301-400 11,218
401-500 3,927
501-600 1,526
601-700 563
701-800 261
801-900 155
901-1000 83
1001-1500 105
1501-2000 16
2001-3000 9
3001-4000 17
4001- 1 (which is 32 N.Y.L. Sch. L. Rev. 209 with 4824 footnotes)[46]
Or in more compact percentage terms:
38.71% of articles have 0-4 footnotes
28.72% 5-100
17.78% 101-200
9.39% 201-300
3.39% 301-400
2.01% 401-
It's also interesting to ask how the median number of
footnotes has changed in recent years.
The median number of footnotes for all articles in the JLR database is
29.[47] The following table shows by yearly dated
additions to JLR the median number of numbered footnotes in articles from each
year. The second column is for all
articles dated with that year in JLR[48],
and the third column gives the median footnote number for articles published in
any of eleven major general law reviews, Harvard,
Date of Median Median
Articles Footnotes Footnotes
For All For Articles
Articles in Major Law
Reviews
1991 71 106
1992 54 107
1993 41 104
1994 28 109
1995 26 111
1996 22 104
1997 20 116
1998 17 111
1999 18 105
2000 19 129
2001 14 112
2002 18 95
When examining these twelve years of additions to the entire JLR database a dramatic drop in the median number of footnotes can be seen. This does not really represent a diminishing use of footnotes, but rather a change in the nature of the JLR database which in the early 1990s was heavily concentrated on law reviews and in later years has been adding a higher percentage of articles from specialized, bar journal, and CLE publications that are less prone to extensive footnoting. The CLE material, in particular, uses few footnotes as their references to legal sources are predominantly in textual citations. CLE publications in the JLR database, as identified by the search CI(PLI CLE ALI-ABA) & DA(<2003), amount to 60,383 articles, or 18% of the entire JLR database through 2002. Of those CLE publications, 65% do not have any numbered footnotes, and 21% have 20 or fewer numbered footnotes. Column 3 in the table above shows a relatively stable number of footnotes as used by articles in major law reviews. The number of footnotes in articles in major law review hovers around a median of approximately 109 footnotes for each year.
The LENGTH segment in Lexis could also be useable by anyone wanting to conduct a footnote density study.
By combining LENGTH and FOOTNOTES segments in a search it's possible to say, for example, that in the Lexis ALLUS file there are 550 articles with at least 30,000 and less than 40,000 words that have more than 500 footnotes.[50]
Corporate Affiliations of Authors
Using full-text journal databases to determine author corporate affiliations is not a very satisfactory undertaking as field information in the Lexis and Westlaw full-text files is inadequate to the task. Nevertheless, there may be some tasks that it's suited to, such as creating preliminary bibliographies of all works associated with a particular university. Lexis has an AUTHOR segment which frequently contains corporate affiliation, assuming the journal lists it (often not the case for "comment," and "note" writers). The problem with such a field is that it can be a lengthy paragraph and the corporate name may be given for numerous reasons not associated with author affiliation. A search utilizing Westlaw's 'FN' asterisk and dagger usage is also possible, such as:
fna fna1 fna2 fna3 fna4 fna5 fna6 fna7 fna8 fna9 fnaa! fnd fnd1 fnd2 fnd3 fnd4 fnd5 fnd6 fnd7 fnd8 fnd9 fndd! +s "columbia law" "columbia university" This particular search constrains the corporate affiliation to the sentence immediately following the footnote, which will miss some relevant entries but is more precise than a lexis AUTHOR search. Affiliation is not necessarily the latest, it could also be, for example, an undergraduate affiliation.
So, similar searches to that above show the following as a few examples:
Harvard 6,453
Yale 3,987
Stanford 2,409
Fordham 1,581
Again, these are rather non-rigorous searches. In terms of precision, the best database to use for corporate affiliations would be ISI's Social Sciences Citation Index (1972- ), but it's coverage of legal journals is not complete. A search of Social Sciences Citation Index on Westlaw (SOCSCISRCH database) with "yale univ" as a corporate source, and "law or legal" as the journal category, as in, cs("yale univ") & sc(law legal), retrieves a list of 1552 documents.
It’s possible to imagine doing a faculty productivity study
using the Lexis LAWREV;ALLUS file. So
that AUTHOR(
Methodology of the W&L Ranking of Legal Periodicals
The legal periodical rankings in the appendix to this article come from the online version at http://law.wlu.edu/library/mostcited/index.asp. The online version is interactive, permitting various lists to be generated based on general/specialized and U.S./non-U.S journals, with sorting by citation counts from cases or articles. This article reproduces separate lists for general-U.S. and specialized-U.S. journals, the lists being sorted by counts of citations found in 1996-2003 journals.
The single most important point to make about the rankings is that they do not deal with all citations to each of the law journals. Counted citations are to those journal volumes that were published in the preceding eight years. The reason for this limit is to prevent a bias in favor of long-published journals. Thus the study is concerned only with current scholarship, and though a journal may have been thought well of and highly cited for fifty years, if the journal volumes published during the last eight years have not often been cited then the journal will have a low ranking.
The "Journals" columns in the rankings show the number of citations to each journal that were found in the full-text Westlaw journals database "Journals and Law Reviews (JLR)". The "Cases" columns show the number of citations to each journal that were found in the full-text Westlaw state and federal case database "Federal & State Case Law (ALLCASES)". Citations are from recent articles, or cases, to recent articles. All citations are from articles, or cases, dated either 1996-2003 or 1995-2002, and the articles they are citing must also have been published during that same period. The list does include cited periodicals that began publication after the survey period began and as ranking results are unfair to those periodicals a warning has been supplied next to the periodical name in the form of a parenthetical date, such as "(2001- )".
Legal periodicals, usually English language, published in the
The text of each search that was run in the JLR and ALLCASES databases can
be seen online at http://law.wlu.edu/library/mostcited/index.asp by
clicking on a journal's citation count number in the Journals or the Cases
column. As an example for Arizona Law
Review: text(38
39 40 41 42 43 44 45 +1 (az
Comparisons between the 1996-2003 and 1995-2002 rankings cannot be made
precisely. Although the number of years covered by each ranking column is
identical (7 years and 10 months - the cut-off date is October 31), the JLR or
ALLCASES databases for the later of the two rotated 8-year periods are slightly
larger then the earlier, and it would be expected that an increase in citations
to each periodical might consequently occur. The Westlaw database sizes[51]
(as of
|
|
1995-2002 |
1996-2003 |
percentage |
|
JLR |
195,313 |
201,829 |
3.3% |
|
ALLCASES |
1,524,774 |
1,574,598 |
3.26% |
Thus, a small percentage increase in the number of documents citing to a particular journal may be totally accounted for by an increased number of documents in the database.
The search pattern for citing references (using the 1996-2003 period as an example) looks for citations within the full-text articles or cases that have one of the volume numbers published for the journal from 1996 onwards (i.e. where the journal has labeled the issue as 1996..2003), followed immediately by the journal abbreviation/name, followed within 6 words with a year designation of 1996..2003. A further condition is that any document (case or article) in which such a citation occurs must be dated (in Westlaw's DATE field) as 1996 onwards, and must have been added to the Westlaw JLR or ALLCASES database subsequent to 1995 and before November 2003.
Westlaw's treatment of periods and spaces within character strings is
difficult to understand and is part of the reason why so many alternate cite
forms were used in the searches. Phrase quotes are not used to surround words
with periods, for example, in looking for "harv.
l. rev.", the search HARV +2 L +1 REV would be used instead. After a "word" ending with a
period, the word is followed by a +2 as in HARV. +2 L. After letters with
embedded periods or single-letters there is no problem with using a +1 as in L
+1 REV. However, a problem occurs if the text incorrectly uses a period after a
complete word, such as "
The search results give only the number of citing articles, and do not show where a citing article or case cites to two, or more, articles in the cited legal periodical. As the searches are full-text searches they are naturally prone to some error due to variant citation form in the citing cases and articles. Searches for citation patterns roughly follow the Harvard Blue Book format, usually, volume-journal-[page]-year. Citations that are not in the usual format for legal citations may not have been found. Effort was made to allow for different forms of journal name citations (e.g. allowing for the ALWD citation format) but not all can be retrieved. No reliance should be placed on the precise numbers of citations found.
Conclusion
Hopefully I have shown that the full-text legal periodical databases on Westlaw and Lexis are a rich source of interesting numeric data. However, the primary purpose of this article has been to reproduce a copy of the online W&L ranking study and to make some comments as to the process and purpose of ranking legal periodicals. Ranking journals on a unitary basis, such as citation count, can be attacked as only superficially related to journal quality, yet as I hope I have shown in discussing expert opinion, even the experts cannot be expected to evaluate a large collection of journals in any objective manner but instead will themselves be guided by some external ranking method, such as law school prominence. An accepted and comprehensive ranking system is a misfortune for low-ranked journals as it tends to perpetuate their position, but does have benefits for authors in guiding their article submission practices, and for readers in filtering out articles of lesser quality. It can be argued that there’s not enough article filtering value in the current prestige scheme to make the whole system worth the cost to journals in duplicative submissions and to authors in trying to work their articles up the hierarchy of journals. A more democratized system is possible, and sooner or later seems inevitable, through article submission to some combination of author web pages, online journals, institutional repositories, or centralized databases, with a coordinated indexing and searching scheme.
[1] George
P. Costigan, Jr., The
Waste of Law Reviews, 10
[2] Harold C. Havighurst, Law Reviews and Legal Education, 51 Nw. U. L. Rev. 22 (1956).
[3] John Gava, Law Reviews: Good for Judges, Bad for Law Schools? 26 Melb. U. L. Rev. 560 (2002).
[4] Scott Finet, The Most Frequently Cited Law Reviews and Legal Periodicals, 9 Leg. Reference Services Q. 227 (1990).
[5] Kincaid C. Brown, How Many Copies Are Enough? Using Citation Studies to Limit Journal Holdings, 94 Law Library J. 301 (2002).
[6] John P. Joergensen, Second Tier Law Reviews, Lexis and Westlaw: A Pattern of Increasing Use, 21 Leg. Reference Services Q. 43 (2002).
[7] Scott DeLeve, A Comparative Citation Analysis of Seven Mid-South Law Reviews, 22 Leg. Reference Services Q. 99 (2003).
[8] William H. Manz, Citations in Supreme Court Opinions and Briefs: A Comparative Study, 94 Law Library J. 26 (2002).
[9] Fred R. Shapiro, The Most-Cited Law Reviews, 29 J. Leg. Stud. 389 (2000).
[10] Wes Daniels, ‘Far Beyond the Law Reports’: Secondary Source Citations in Unites States Supreme Court Opinions, October Terms 1900, 1940, 1978, 76 Law Library J. 1 (1983).
[11] Journals that had no citations to them were excluded from this average. Note that these are not all the citations to journals. The W&L study, in order to prevent a bias in favor of older journals, is looking only for citations to volumes published within the current eight year period. The relatively small number of cases that cite journal articles is all the more striking as there are roughly 1.4 million cases in ALLCASES (1996-2003) and only around 190,000 articles in JLR for the same period.
[12] At the time of writing HeinOnline was using the study at, Kincaid C. Brown, How Many Copies Are Enough? Using Citation Studies to Limit Journal Holdings, 94 Law Library J. 301 (2002), for the list of its top-27 titles (HeinOnline is using a top-30 list, but 3 of the entries are precursor titles to titles which are in the top-27 list).
[13] Nancy P. Johnson, Legal Periodical Usage Survey: Method and Application, 71 Law Library J. 177 (1978).
[14] Tracy
E. George & Chris Guthrie, An Empirical Evaluation of Specialized Law
Reviews,
[15] Gregory Scott Crespi, Ranking International and Comparative Law Journals: A Survey of Expert Opinion, 31 Int’l Law. 869 (1997).
[16] Gregory Scott Crespi, Ranking the Environmental Law, Natural Resources Law, and Land Use Planning Journals: A Survey of Expert Opinion, 23 Wm. & Mary Envtl. L. & Pol’y Rev. 273 (1998).
[17] U.S. News & World Report,
[18] Russell
Korobkin, In
Praise of
[19] Russell
Korobkin, Ranking Journals: Some Thoughts on
Theory and Methodology,
[20]
[21] For a rough comparative idea of how many articles, during 1996-2003, academic legal journals published on a few selected subjects, the following search (limited to journals with "L.J." or "L. Rev." in their title) was run in the Index to Legal Periodicals (ILP) database on Westlaw: ci((l +1 rev)(l.j.)) & in,ti("life insurance") & da(>1995 & < 2003) - substituting for "life insurance" the topic of interest.
Note that the "in,ti" fields are the descriptor (ILP's indexing subjects) and the title fields.
Life Insurance 92
Wills 102
Workers' Compensation 190
Conflict of Laws 425
Environmental Law 727
Copyright 1112
Bankruptcy 1214
Antitrust Law 1718
It might be interesting to do a study, using ILP thesaurus terms, to determine if subject selection by law reviews is correlated with citation rank.
[22] "Thus, if the advent of electronic submissions increases the number of submissions to the point where articles editors are not able to screen articles effectively, it may be in the best interest of authors to target their submissions more restrictively and of law reviews to reward this behavior by a preferential review of such manuscripts." Richard A. Bales, Electronically Submitting Manuscripts to Law Reviews, 30 Stetson L. Rev. 577, 589 (2000).
[23] Howard Denemark, How Valid is the Often-Repeated Accusation
That There are Too Many Legal Articles and Too Many Law Reviews? 30
[24] The ALLFEDS, ALLFEDS-OLD, ALLSTATES, ALLSTATES-OLD search of, 190* 191* 192* 193* 194* 195* 196* 197* 198* 199* 200* COURT 180* 181* 182* 183* 184* 185* 186* 187* 188* 189* 171* 172* 173* 174* 175* 176* 177* 178* 179*
results in (as at
[25] In early 2004 the numbers of post-1980 legal journal articles that are findable are approximately: ILP 372,000; LRI 1,030,000; JLR 354,000; JLR (excluding CLE material) 280,000. The Legal Resource Index (LRI) has by far the largest number of entries, but LRI includes articles from legal newspapers and from non-legal journals. The number from Index to Legal Periodicals (ILP) is the better number to use for academic legal articles, and while it's unfortunate that JLR doesn't contain full-text for more post-1980 academic articles, it does have approximately 75% of them since 1980. From 1995 onwards JLR (excluding CLE materials) has as many entries as are found in the same period for ILP, so for recent academic articles JLR has close to 100% coverage.
[26] The number of articles citing to, Harvard Law Review, Columbia Law Review, or Yale Law Journal, was determined by the following search. This example is for the years 1992-1998. For other years the search would drop off the older volumes and add the more current volumes, and rotate the date restrictions. Note that these searches were run in the web version of Westlaw which only permits a maximum result of 10,000 documents so each search was split in two, by dividing the added-date (AD) field into two units, such as AD(>1991 & <1995), and AD(>1994 & <1999).
text((92 93 94
95 96 97 98 +1 (colum
(105 106 107 108 109 110 111 112 113 114 +1 (harv harvard +2 "l.rev." "l.r." (l +1 rev review) (law +2 rev review)) "harv.l.rev." "harvard.l.rev.")
(101 102 103 104 105 106 107 108 +1 "yale.l.j." "yale.lj" (yale +2 lj "l.j." (l +1 j journal) (law +2 j journal)))
+6 1992 1993 1994 1995 1996 1997 1998)
& DA(> 1991) & AD(>1991 & < 1999)
[27] The size of the JLR database for each rotating seven
years was determined by a search (in Westlaw's dedicated communication
software) for variants of the following (this example is for 1992-1998):
1992 1993 1994 1995 1996 1997
1998 & DA(> 1991) &
AD(>1991 & < 1999)
Note that the result is not the total size of the JLR database at a particular moment, rather the search shows the number of documents which match the seven-year rotating period used in the table.
[28] The number of articles citing to either, American Journal of International Law, Journal of Legal Studies, or Harvard Journal on Legislation, was determined by searches such as the following:
text(
(86 87 88 89 90 91 92 +1 (am american +2 j journal +3 intl int'l international +2 l law) "am.j.int'l.l" "am.j.intl.l." "am.j.int'l.law" "am.j.intl.law")
(21 22 23 24 25 26 27 +1 "j.legal.stud" "j.leg.stud" ((j journal +3 legal) "j.leg" "j.legal" +2 stud studies))
(29 30 31 32 33 34 35 +1 ((harvard harv +2 j journal) "harv.j." "harvard.j." +3 legis legislation) "harv.j.legis" "harvard.j.legis.")
+6 1992 1993 1994 1995 1996 1997 1998
)
& DA(> 1991) & AD(>1991 & < 1999)
[29] A few
statistics (as of
LAWREV; ALLUS JLR (excluding CLE PLI ALI-ABA)
(through 2002) (through 2002)
total size 185,298 272,021
before 1981 1,413 7,630
1981-1990 25,841 41,600
1991 3,368 9,210
1992 3,656 11,311
1993 6,611 14,488
1994 8,981 18,893
1995 11,580 19,961
1996 15,173 20,957
1997 16,641 21,233
1998 17,709 21,679
1999 18,509 21,266
2000 19,364 21,592
2001 18,770 21,240
2002 17,682 20,961
[30] The search for the total number of articles used a string of alternate years from 1930-2003 (as in, 1930 1931 1932 ...) and a DATE restrictor for the appropriate date period, as in: 1930 1931 .... 2002 2003 & DA(2002).
[31] Richard A. Leiter, Use of Law Reviews in Modern Legal Research: The Computer Didn't Make Me Do It!, 90 Law Lib. J. 59 (1998).
[32]
[33] The search for the older years would be:
text(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 +1 (colum columbia +1 "l.rev." "l.r." (l +1 rev review) (law +2 rev review)) "colum.l.rev." "columbia.l.rev." +6 190* 191* 192* 193* 194* 195* 196* 197*) & DA(1996)
Modifying the search for all years, or for more recent years entails changing the volume numbers and the "197*" and the like.
[34] A typical search would be:
da(2002) & ci((harv colum mich stan fordham tex minn va n.y.u. vand ulca cardozo conn tul +2 "l. rev")(yale geo cornell duke emory +2 l.j)) & ((108 +1 harv +2 l +1 rev)(104 +1 yale l.j.)(82 +1 calif +2 l +1 rev)(80 +1 va +2 L +1 rev)(63 +1 fordham +2 l +1 rev)) +4 1994.
The reason for having the "CI" restriction from particular journals is to attempt to limit the effect of the year-by-year expansion of the entire JLR database.
[35] As at
[36] A typical search for citations would be:
text(78 79 80 +1 colum +2 l +1 rev +5 1978 1979 1980) & DA(1998)
In this example Westlaw has Columbia Law Review back to volume 81, and Lexis back to volume 82.
A citing date field of 1998 was used for the older volumes
of Cornell,
[37] John P. Joergensen, Second Tier Law Reviews, Lexis and Westlaw: A Pattern of Increasing Use, 21 Legal Ref. Services Q. 43 (2002).
[38]
[39] Variations in defining what demarcates a “word” will likely cause different counters to come up with different word count results. I compared a Lexis and a Microsoft Word count of one article, 82 Tex. L. Rev., and the result (after cleaning up the Lexis download to eliminate some Lexis editorial additions) was that Lexis reported 2,419 words and Word reported 2,753 words. So it’s possible that Lexis systematically “undercounts”, at least relative to Microsoft Word.
[40] An important note is that the LENGTH segment is only used by Lexis for law review articles, it is not present for bar journal articles. Sadly, the LENGTH segment in Lexis is not presently mathematically searchable, which means that some contorted searches are necessary to determine how many articles have a particular range of lengths. The following search, for example, will show how many articles, dated 1993, have a length of 90,000-99,999 words.
date(=1993) and length(9***0 or 9***1 or 9***2 or 9***3 or 9***4 or 9***5 or 9***6 or 9***7 or 9***8 or 9***9)
[41] Using the LENGTH field is not as automatic as I would have liked because over 300 articles are split, because of their length, into a number of parts, which become separately searchable documents with individual lengths. So it’s necessary to make manual adjustments to the Lexis search results. The means of retrieving the article subparts was to do a series of searches such as: (part 6 of 6 this document has been split) or title(part 6 of 6) or (editors note part 6 of 6), then to use "Get a Document" for each cite, which lists the length for each subpart. There seems to be no consistency in the choice of whether to split a document or not. Just looking at additions for the year 2002, some articles with less than 40,000 words are split and one with over 150,000 words is not.
[42] Just using the LENGTH field the lengthiest article in the entire ALLUS file is reported to be, at 158,630 words, Sarah H. Cleveland, Powers Inherent in Sovereignty: Indians, Aliens, Territories, and the Nineteenth Century Origins of Plenary Power over Foreign Affairs, 81 Tex. L. Rev. 1 (2002). However, making the manual adjustments for split files shows that for the years 1995-2002 there are 7 articles that are larger. There is also a definitional issue as to what constitutes an article, or a part thereof. For example, the Thirteenth Annual Review of Criminal Procedure: United States Supreme Court and Courts of Appeals 1982-1983, 72 Geo. L.J. 185 (1983) comprises 563 pages, but it is included in Lexis as separately titled articles using the sub-part designations of the article.
[43] Unfortunately Lexis has a peculiarity which makes searches for a sequence of yearly ranges a little less than routine, in that Lexis will sometimes code documents with more than one year, for example a search, date(=2001 and =2002), retrieves 435 documents that are coded with both years in the date segment--an objectionable practice for those wanting numerical year-by-year results. This means that a decision must be made, to prevent overlapping numeric results, as to whether to accept the earlier or later date and then to modify date searches. Accepting the earlier date, as is done here, means a search like, date(=1996 and not <1996). so that articles with the years 1995 and 1996 are rejected, but articles with 1996 or else both 1996 and 1997 are retrieved.
[44] As Lexis does not supply a LENGTH segment for bar journals, these available articles are law reviews.
[45] Westlaw, at present, does not allow the use of the "atleast" command in JLR. If it did, then it would have been a better search to have asked for documents where two occurrences of the FN numbers appear, as in: atleast2(FN397) & atleast2(FN398)....
[46] Just by way of comparison with case law, the case
with the most citations contains 1715 footnotes (118 F. Supp. 41), a record
which seems likely to be maintained as the next closest contains less than 900
footnotes.
[47] The median is determined by taking half of the size of the JLR database to the end of 2002 (331,378/2 = 165,689 articles) and determining what footnote search most closely matches that number. Which is fn26 & fn27 & fn28 & fn29 & fn30 & da(<2003), which retrieves 165,406 articles having more than 29 footnotes. Thus there are approximately an equal number or articles that have 0-29 footnotes as have 30 or more footnotes - the median value has been rounded down to the closest integer.
[48] The footnote median for each year's articles is determined by the a search such as: fn25 & fn26 & fn27 & fn28 & fn29 & da(1994), which result most closely matches half of the articles with that year's date.
[49] A typical search would be: fn92 & fn93 & fn94
& fn95 & fn96 & da(2002) & ci("harv. l. rev." "colum.
l. rev." "yale l.j."
"stan. l. rev." "geo. l.j." "
[50] The search would be: footnotes(n497 and n498 and n499 and n500 and n501) and length(3***0 or 3***1 or 3***2 or 3***3 or 3***4 or 3***5 or 3***6 or 3***7 or 3***8 or 3***9). Note that for complete accuracy some manual adjustment might be needed when retrieving very large articles as Lexis may split them into several documents.
[51] The searches to retrieve the total number of
potentially citing articles/cases were:
1996-2003: 1996 1997 1998 1999 2000 2001 2002 2003 & DA(> 1995) &
AD(>1995 & < 11/1/2003)
1995-2002: 1995 1996 1997 1998 1999 2000 2001 2002 & DA(> 1994) &
AD(>1994 & < 11/1/2002)