Examining a Misapplication of Nearest Shrunken Centroid Classification to Investigate Book of Mormon Authorship
Examining a Misapplication of Nearest Shrunken Centroid Classification to Investigate Book of Mormon Authorship
Reviewed by Paul J. Fields, G. Bruce Schaalje, and Matthew Roper
Editor’s note: The above-referenced essay by
Jockers, Witten, and Criddle (hereafter Criddle and associates) was answered by
G. Bruce Schaalje, Paul J. Fields, Matthew Roper, and Gregory L. Snow in a
technical paper entitled “Extended nearest shrunken centroid
classification: A new method for open-set authorship attribution of texts of
varying sizes,” Literary and Linguistic Computing 26/1 (2011):
71–88. We have invited Fields, Schaalje, and Roper to provide both a
popularization of this important essay and a brief history of efforts to use
what is called stylometry to identify the authors of disputed texts. In
addition, because Professor Criddle has been involved in efforts to resuscitate
the Spalding-Rigdon theory of Book of Mormon authorship, Roper and Fields were
also invited to comment on that rather moribund explanation in a separate essay
that immediately follows this one.
In 1834 the first anti-Mormon book
was published in Ohio by E. D. Howe. Relying on testimony claimed to have been
gathered by D. P. Hurlbut, a disgruntled former member of the Church of Jesus
Christ of Latter-day Saints and sworn enemy to Joseph Smith, Howe argued that
the Book of Mormon was based on an unpublished fictional tale by an
unsuccessful amateur novelist, Solomon Spalding. Spalding lived in Conneaut,
Ohio, between 1809 and 1812. Howe claimed that Sidney Rigdon somehow acquired
Spalding’s unpublished manuscript and added religious material, thereby concocting
the Book of Mormon.1
The 1884 recovery of an
original Spalding manuscript bearing little resemblance to the Book of Mormon
led most critics to abandon the Spalding-Rigdon conspiracy theory.2 This manuscript is known today variously as “Manuscript
Story” or the “Oberlin manuscript.” Today, among those who
reject Joseph Smith’s explanation of the Book of Mormon, a majority see Joseph
Smith alone as responsible for the text and believe that the Spalding theory
sheds no light on Book of Mormon origins. A minority of these critics continue
to argue that the Book of Mormon was based on a hypothesized second or third,
now-lost Spalding manuscript, though even the existence of such a manuscript
has never been proved.3
A recent article by three Stanford researchers—Matthew
Jockers, Daniela Witten, and Craig Criddle—is the latest in a series
of stylometric investigations of Book of Mormon authorship.4 The Criddle and associates study applies a statistical methodology developed
for genomics research,5 known as Nearest Shrunken Centroid (NSC) classification, to the question of
Book of Mormon authorship. In contrast to previous wordprint studies, Criddle’s
team concluded that the majority of the chapters in the Book of Mormon were
written by either Solomon Spalding or Sidney Rigdon: “The NSC results are
consistent with the Spalding-Rigdon theory of authorship,” and “our
findings are consistent with historical scholarship indicating a central role
for Rigdon in securing and modifying a now-missing Spalding manuscript”
(p. 482). Although they claim to have discovered evidence for smaller
contributions from Parley P. Pratt and Oliver Cowdery, the authors “find
strong support for the Spalding-Rigdon theory of authorship. In all the data,
we find Rigdon as a unifying force. His signal dominates the book, and where
other candidates are more probable, Rigdon is hiding in the shadows” (p.
We here examine the stylometric analysis presented by Criddle and
associates. We first review past attempts—stylometric and
otherwise—to analyze Joseph’s writing style. We review the strengths and
weaknesses of those attempts and assess past authors’ success in meeting
objections to their findings. We then address the validity of Criddle and associates’
methodology, its utility in dealing with questions of authorship in general,
and its application to authorship of the Book of Mormon in particular. Lastly,
we present the findings of our study extending the NSC methodology, which shows
the naïveté and invalidity of Criddle and associates’ efforts to add a
mathematical patina to an untenable historical hypothesis that has been long
abandoned by virtually all serious scholars, whether believers or skeptics.
Prelude to Stylometry: Joseph Smith’s
In 1976 Elinore Partridge
performed a study of the characteristics of Joseph Smith’s writing style. She
also studied the writings of several of his closest associates—Sidney
Rigdon, Frederick G. Williams, Parley P. Pratt, Oliver Cowdery, and Willard
Richards. Partridge detected a characteristic tone in the Prophet’s writings.
In contrast to the dark visions of Calvinism and the dry,
rational theology of Unitarianism, Joseph Smith’s pronouncements emphasize the
wonder of existence and the love of humanity. Likewise, in contrast to the
threats of wrath, judgment, and damnation, which one can find in the statements
of some of the early church leaders, there is an undercurrent of understanding
and compassion in those of Joseph Smith. Moments of discouragement and anger do
occur; however, even at times when he laments the state of mankind, he tempers
the observations with trust in God, love for his family, and hope for the
future. The love of others, the pleasure in variety, and the joy in living
which is apparent in the language of Joseph Smith give us some real sense, I
believe, of what he must have been like as a leader and a friend.6
Partridge also found significant “markers” of Joseph
Smith’s style that distinguish his writing from that of other Latter-day Saint
leaders of his day. These include a tendency to form a structure of “interconnected
sentences joined, like links in a chain, by simple conjunctions,” a
characteristic that she found could often be detected even after Joseph’s work
had been edited by others.7
Joseph Smith’s writing is characteristically marked by a
series of related ideas joined by simple conjunctions: and, but, for.
In his handwritten manuscripts, he used neither punctuation nor capitalization
as sentence markers. When his writing has been edited, or when someone else
wrote words which he dictated, the result is an unusually large number of
sentences beginning with for, and, or but (almost three out
of five sentences). On the other hand, Sidney Rigdon seldom used these
conjunctions, and almost never used them at the beginning of sentences; on the
average, only about one in twenty sentences begins with and, for, but.
Rigdon’s sentences frequently begin with participial or prepositional phrases;
for example, ‘Having shown . . .’ ‘From the foregoing we learn . . .’ which is
a structure Joseph Smith seldom used. Sidney Rigdon regularly used phrases such
as ‘in order that,’ ‘so that,’ or ‘the fact that,’ to introduce and link ideas.
Joseph Smith almost invariably uses ‘that’ or ‘this’ instead. Joseph Smith’s
images and examples are concrete, specific, and well-detailed, while Sidney
Rigdon’s tend to be abstract and generalized.8
Partridge also noted Joseph Smith’s use of “pronouns and
demonstratives which require specific referents” and the use of a “series
of modifying phrases which must be attached to other words,” features that
she notes “suggest a personality used to seeing things as an
interconnected whole rather than as separate parts.” 9 She saw this as evidence that “Joseph Smith is more comfortable with the
spoken than with the written language. The long interrelated sentences, with no
clear stopping point, are typical of an oral style. The occasional repetitions
or awkward constructions also indicate that he is writing as he speaks.” 10
Interestingly, Partridge also detected evidence that some
elements of Joseph Smith’s style could be found even in works that he oversaw
or directed others to write for him.
Joseph Smith’s influence can be seen in many of the works
which he did not actually write himself. For example, I see signs of his
collaboration in the Lectures on Faith. The sermons and discourses published
in the Times
& Seasons and parts of the History of the Church have
obviously been well polished and heavily edited; however, in details and in
general structures of the sentences, it is possible to identify characteristics
of Joseph Smith’s style. Even when a scribe has obviously altered sentence
structure to conform to a more standard, written style (that is, with definite
marks of punctuation, capitalization, and clearer divisions between sentences),
the interrelationships and internal references characteristic of Joseph Smith’s
style remain. Occasionally, there are certain images and examples which
indicate that a reported version of a sermon or speech has managed to capture
the essential ideas and illustrations of Joseph Smith, although the language
may have been dramatically altered.11
Partridge’s findings suggest that there are distinct and
significant differences between the writing styles of Joseph Smith and Sidney
Rigdon. Consequently, analyses of the writing styles exhibited in the text of
the Book of Mormon might provide insights into the question of the book’s
authorship and particularly into Rigdon’s alleged role in its origin.
Stylometry and the Book of Mormon
Stylometry uses statistical techniques to quantitatively describe
the characteristics of an author’s writing style. It is based on the
fundamental premise that authors write with distinctive word-use habits. For
example, one commonly used method measures the frequency with which an author
uses or does not use certain words or groupings of words. Identifying the
word-use patterns in a text of unknown or questioned authorship and then
comparing those patterns with the patterns in texts of known authorship can
provide supporting evidence for or against an assertion of authorship. Although
the proper term for this type of analysis is stylometry, the term wordprint
analysis is also sometimes used (in a loose comparison to
fingerprint analysis). However, an author’s writing style is not nearly as precise,
distinctive, unalterable, or unchanging as his or her fingerprints, and so the
latter term is a potentially misleading exaggeration.
Over the last thirty years, researchers have conducted five major
and several minor stylometric studies of the Book of Mormon. We will describe
the major studies by Larsen et al., Hilton, Holmes, Criddle et al., and
Schaalje et al.
First Study: Word-Frequency Analysis
In 1980 Wayne Larsen, Alvin
Rencher, and Tim Layton examined word frequencies in a precedent-setting
analysis of the Book of Mormon.12 As indicators of writing style, they used noncontextual words—the
words that play a grammatical role in forming the structure of a message but do
not provide the information content of the message. Examples are a, an, but, however, the, to, with, and without. These words are also called function words since by themselves they do not convey the author’s
message but, rather, provide the framework for the author’s message. Studying
the function words in a text can indicate an author’s personal manner of
expressing his or her ideas since they do not indicate what the author says but
the way he or she says it.
The Larsen et al. researchers used three statistical
techniques—Multivariate Analysis of Variance (MANOVA), Cluster Analysis
(CA), and Linear Discriminant Analysis (LDA)—to test for differences in
the frequencies of noncontextual words. MANOVA is a method of testing for
homogeneity (degree of similarity) within groups of items. CA is a method that
can identify which items are closest to each other among all items compared.
LDA is a method for determining a set of mathematical functions (discriminant
functions) that can be used to classify items into categories based on their
characteristics. The three methods produced consistently congruent results,
which are highlighted below using LDA to
summarize the findings.
In stylometric analysis, LDA can compare the word-frequency
profile in a block of text to the profile of each candidate author and then
assign that block of text to the author with the most similar style. It does
this by measuring how closely the word profile in the test block matches the
average word profile of each author. A plot of the test texts using the
discriminant functions as the axes of the graph can display how well the texts
correspond to each author.
In the Larsen et al. study, the researchers segmented the entire
text of the Book of Mormon into 2,000-word text blocks aligned with each of the
twenty-one purported authors in the book. Then they tested whether there was
evidence that the text blocks displayed a consistent style across the blocks
(indicative of one author for all the texts) or whether there was evidence of
differing styles (congruent with the claim that the Book of Mormon texts came
from different writers).
For comparison they also
examined texts from Joseph Smith, Sidney Rigdon, and Solomon Spalding, along
with texts from Oliver Cowdery, Parley P. Pratt, and W. W. Phelps, all of whom
they referred to collectively as “nineteenth-century authors.”
Larsen’s team showed that the text blocks from the Book of Mormon
were consistently classified as separate from those of the nineteenth-century
authors. This is shown in figure 1. Further, they showed that each Book of
Mormon author is consistently similar to himself but consistently different from
the other authors. This is illustrated in figure 2, which shows the texts
grouped into separate clusters by author. For simplicity in illustrating the
results, figure 2 shows the clusters for only Nephi, Alma, and Mormon—the
three major authors in the Book of Mormon. Joseph Smith’s cluster is also shown
in figure 2, and it stands apart from the Nephi, Alma, and Mormon clusters.
After repeatedly analyzing all the texts and all the candidate
authors, Larsen’s team found the statistical evidence of differences between
the writings of the purported authors to be striking. They concluded that “distinct
authorship styles can be readily distinguished within the Book of Mormon, and
the nineteenth-century authors do not resemble Book of Mormon authors in style.”13
D. James Croft, a statistician at the University of Utah, raised
several questions in critique of the Larsen et al. analysis.14 In essence he asked the following:
1. Is the basic assumption of stylometrics—that authors’ writing
styles can be characterized by measurable features—valid?
2. Does the modern Book of Mormon edition used by Larsen et al. exhibit
the same stylistic patterns as those in the original 1830 edition?
3. Was the phrase “and it came to pass that” recognized by Larsen et al. as a possible indicator of content differences
rather than author differences?
4. Were the results
of the analysis due to style differences among the purported authors or to
topic differences among the texts?
When Croft’s review of the Larsen et al. study was published, it
was accompanied by a well-reasoned reply by the researchers to all the issues
he raised.15 We offer here some additional analysis in further rebuttal.
Croft’s first point—the validity of stylometry—has
been answered by the continuing and increasingly successful use of stylometric
methods similar to those employed by Larsen’s team. The methodology has been
validated repeatedly and is a well-accepted analytical approach in literary
analysis. Even other critics of the Larsen et al. study, such as David Holmes,16 do not dispute the validity of the methodology. However, most stylometry practitioners
would agree with Croft that the methodology has limitations and that it is only
as valid as the stylistic measures used in the analysis.
Croft’s second point—the use of a modern edition of the
Book of Mormon—turns out to be a nonissue when we examine the effects of
editorial changes to the book. Figure 3 overlays plots of word-use frequencies
from sequential 5,000-word blocks of both the 1830 and 1980 editions of the
Book of Mormon. The editorial changes to the Book of Mormon over 150 years
appear to have been made nearly proportional throughout the book since the
patterns present in one edition are almost exactly reproduced in the other. For
the main purpose of the Larsen et al. study, it did not matter which edition
the researchers used.
Croft’s third point—the possible effect of the frequently
occurring phrase “and it came to pass”—is insightful.
However, subsequent studies we have conducted showed no detectable differences
in the results of stylometric analyses that include the words in the phrase “and
it came to pass” as separate words, treat the phrase as one word,
or delete those words entirely from the frequency counts when they occur in
Croft’s fourth point—results due to style or topic
differences—is well-taken. The consistent difference between writings
attributed to Mormon and those attributed to Nephi or Alma could be due to
content differences instead of stylistic differences, since Mormon’s writings
tend to be historical narrative while Nephi’s and Alma’s writings tend to be
doctrinal discourse. However, there can be little question that the Larsen et
al. study showed, at a minimum, that the texts purported to be written by Nephi
and Alma exhibit internally consistent but highly distinct authorship styles as
measured by their use of noncontextual words, even though both authors were
discussing the same topics.
Other criticisms of the Larsen et al. study have come forward
more recently. The 2008 paper by Criddle and associates questioned the Larsen
et al. approach of grouping verses and partial verses into blocks of words “based
on their understanding of speakers (or characters) in the Book of Mormon”
(p. 467). However, this criticism is misguided since such grouping was
appropriate given that Larsen’s team was testing a hypothesis of multiple
authorship. A second point raised by Criddle and associates was that even if
the texts were carefully grouped, they might be “composites containing
different fractional contributions from different nineteenth-century authors”
(p. 467). Although this could be true, the consistent clustering of writings
due to purported Book of Mormon speakers would imply a remarkable compositing
process, a process in which the different nineteenth-century authors contributed
consistent but different proportions of text for each of the purported authors.
Finally, Criddle and associates state that biblical-sounding words such as behold, forth,
lest, nay, O, unto, wherefore, and yea might account for observed
differences between Book of Mormon text blocks and the text blocks of the
nineteenth-century authors in the study. However, the Larsen et al. study did
not use those words, so perhaps Criddle and associates misread the word lists
used by Larsen et al. We discuss in detail the paper by Criddle and associates
later in this article.
On the whole, even after the thoughtful criticism of the Larsen
et al. study is accounted for, the results of that early study continue to
provide persuasive support for the claim that the Book of Mormon is the work of
multiple authors and not the work of any of the likely nineteenth-century
Second Study: Word-Pattern Ratios
In a subsequent study, John Hilton took a different approach to
studying stylometric patterns in the Book of Mormon.17 Intrigued but uncertain of the Larsen et al. results, Hilton set out to see if
he could replicate their results using a study designed to accommodate Croft’s
criticisms. Rather than noncontextual word frequencies as in Larsen et al.,
Hilton used “noncontextual word-pattern ratios.” Word-pattern ratios
measure the rates an author uses words in four categories:
1. Specific words in key positions of sentences (e.g., the as the first word of a sentence)
2. Specific words adjacent to certain parts of speech (e.g., and followed by an adjective)
3. Collocations of words (e.g., and followed by the)
pairs of words (e.g., no and not, all and any)
Hilton used the sixty-five word-pattern ratios developed by A. Q.
Morton that he had shown to be useful in authorship studies for other religious
texts as well as secular texts.18 One of the advantages of using word-pattern ratios is that the potentially problematic
phrase “and it came to pass” can only partly affect one of the
sixty-five word-pattern ratios, so its impact on the analysis was negligible in
Using primarily the printer’s manuscript of the Book of Mormon,
Hilton applied his procedure to 5,000-word blocks of text to ensure the
reliability of the style measures since in larger text blocks an author’s
writing habits and stylistic propensities should assert themselves more
strongly than in smaller texts. He compared texts attributed only to Nephi or
Alma to control for topic differences and then texts known to be authored by
Joseph Smith, Oliver Cowdery, or Solomon Spalding. He compared each author to
himself and then each author to every other author. The result demonstrated that
the stylistic patterns in the Nephi, Alma, Smith, Cowdery, and Spalding texts
were consistent within themselves but distinctly different from one another.
This evidence argues strongly for the assertion that the Nephi
and Alma texts were written by different authors, and against the idea that
Joseph Smith, Oliver Cowdery, or Solomon Spalding was the author of the Nephi
or Alma texts. Hilton stated:
It is statistically indefensible to propose Joseph Smith or
Oliver Cowdery or Solomon Spaulding as the author of the 30,000 words from the
Book of Mormon manuscript texts attributed to Nephi and Alma. Additionally
these two Book of Mormon writers have wordprints unique to themselves and
measure statistically independent from each other in the same fashion that other
uncontested authors do. Therefore, the Book of Mormon measures [as being]
multiauthored, with authorship consistent to its own internal claims.19
Hilton’s results corroborated the Larsen et al. results even
though Hilton used an entirely different technique.
Third Study: Vocabulary Richness Analysis
In 1992 David Holmes published the results of a stylometric
analysis of the Book of Mormon using another approach, one he
had developed as a doctoral student. He attempted to show that measures of “vocabulary
richness” could be used for authorship attribution.20 Vocabulary richness measures attempt to quantify an author’s style based on his
or her lexical variety in word choices. As stylistic features, Holmes computed
a standardized measure of once-used words, a standardized measure of twice-used
words, a measure of lexical repetitiveness, and two estimated parameters for a
theoretical model of word frequencies in writing. The first three measures were
calculated for the total vocabulary in the texts, while the last two were
calculated for nouns only.
Holmes compiled fourteen 10,000-word blocks assigned to six Book
of Mormon authors, combined sections 1 through 51 of the Doctrine and Covenants
into three 10,000-word blocks, combined an assortment of writings by Joseph
Smith into three 6,000-word blocks, included the Book of Abraham from the Pearl
of Great Price as one text, and extracted three 12,000-word blocks from Isaiah.
He then used Principal Components Analysis (PCA) to search for separations
among the clusters of texts.
PCA takes a set of
multidimensional points and projects them into two dimensions. As an analogy,
imagine the outline of a three-dimensional object such as a pencil projected by
an overhead projector onto a flat, two-dimensional screen. Its projected image
could look like a dot or like an arrow, depending whether the pencil is
oriented vertically or horizontally. The PCA procedure determines how to rotate
a set of points so the greatest separation among the points can be seen. This
is a useful way to visually explore the data in two dimensions for possible
relationships among points in many dimensions. The first and second principal
components define the two-dimensional space.
Using PCA applied to the vocabulary richness measures, Holmes
found that the Joseph Smith texts clustered together, the Isaiah texts
clustered together, and all but three of the other texts clustered together.
Figure 4 presents a PCA plot of Holmes’s results. From this he concluded that
the writings of Mormon, Lehi, Nephi, Jacob, and Moroni were not stylistically
Subsequent research has shown that Holmes’s vocabulary richness
stylistic measures are weak discriminators of authorship. For example, when
testing texts of undisputed authorship, correct classification rates
were 96 percent using noncontextual word frequencies, 92 percent for
noncontextual word-pattern ratios, but only 23 percent for vocabulary richness
measures.21 In statistical terms, a method’s ability to find differences is called “power.”
A weak discriminator, such as the vocabulary richness measure, can lack the
power to find differences even when they are present.
When a method cannot find differences that are known to exist in the
data, and then subsequently does not find a difference between two items, such
a result is not convincing evidence that no true difference exists between
those items. Consequently, the correct interpretation of Holmes’s finding is
not that “there are no differences among the tested authors,” but
rather that he “found no evidence of difference.” Not finding
evidence of difference may therefore say little about the subject of the test but can be an indication of the test’s inadequacy. This was the
case for Holmes’s Book of Mormon study—he was using a technique with low
power. Such a situation is analogous to using a low-powered microscope when a
high-powered instrument is needed: his “instrument” was inadequate
for the research he was attempting, leaving him unable to discern features that
were, in fact, present.
Although in concept vocabulary richness analysis seems like
it should be useful, in practice it has been shown to be unreliable. In fact,
after his early work in stylometrics, Holmes subsequently discontinued the use
of vocabulary richness measures and employed other techniques in his work. We
conclude that the Holmes study serves only to show the limitations of
vocabulary richness analysis, while providing no insight into the question of
Book of Mormon authorship.
Fourth Study: Nearest Shrunken Centroid Analysis
Sixteen years after the Holmes study, Matthew Jockers, Daniela
Witten, and Craig Criddle tried to take an innovative approach to authorship
attribution by applying an analytical method developed for the classification
of tumors in genomics research.22 The technique is called Nearest Shrunken Centroid (NSC) classification. It
takes a set of items of known origin(s) and compares them to a set of items of
unknown origin(s) by determining the distances between the centers (centroids)
of the groups of items. The technique seeks to identify the centroids that are
nearest to each other. “Shrinkage” is a statistical technique to
combine all available information in a way that can reduce the uncertainty in
estimating the distances between the centroids. The distance between the
centroids is considered a surrogate for similarity. When centroids are
relatively close to each other, this is taken to indicate relative similarity.
Conversely, when the centroids are relatively far apart, this is taken to
indicate relative dissimilarity. NSC calculates the probability of relative
When applied to stylometry, NSC develops a classification rule
based on stylistic characteristics—such as word frequencies—in a
set of texts with known authorship and then uses that classification rule to
assign texts of questioned authorship to the author whose style is closest. The
closer a test text of an unknown author is to the centroid of a known author’s
texts, the greater the likelihood that the writing style exhibited in the test
text matches the writing style of the known author. The analysis is complex
since each word frequency is a dimension in which “distance” must be
measured. If a researcher uses one hundred word frequencies, the analysis is a
Criddle and associates applied NSC to the Book of Mormon in an
attempt to find evidence in support of the Spalding-Rigdon theory. Their set of
texts for candidate authors included Solomon Spalding, Sidney Rigdon, Parley P.
Pratt, and Oliver Cowdery. They also included Isaiah and Malachi (combined as
one author) as a positive control and Joel Barlow and Henry Wadsworth
Longfellow (nineteenth-century authors) as negative controls. The texts varied
widely in length from 114 to 17,797 words. Their test texts were the individual
chapters of the Book of Mormon, which varied in length from 95 to 3,752 words.
As stylistic features, Criddle and associates used the relative
frequencies of the most common 110 words in the Book of Mormon that were used
at least once by each candidate author. Although their list contained mainly
function words, they retained some lexical words as well. From their analysis
they concluded that the evidence showed with high probability that Spalding and
Rigdon were the principal authors of the Book of Mormon.
However, there were very significant problems with this study. We
will discuss the following problems:
• Failing to include Joseph Smith as a candidate author
• Misapplying a closed-set technique for an open-set problem
• Confusing “closest” to mean “close”
• Misinterpreting relative probabilities as absolute probabilities
• Ignoring a high rate of false classifications
• Using circular statistical thinking
• Disregarding statistical problems of homogeneity and multiplicity
• Confounding the primary candidate author’s differing writing styles
Failing to Include Joseph Smith as a
Considering the lack of unanimity on the question of Book of
Mormon authorship, even among critics, it is strikingly odd that Criddle and
associates would choose to exclude Joseph Smith from the list of potential
authors. A substantial majority of critics favor some version of the Joseph
Smith composition theory, which sees Joseph Smith as the book’s author.
Latter-day Saints, on the other hand, who understand the Book of Mormon as
divinely revealed scripture, acknowledge Joseph Smith as a human mediator of
the revealed word (2 Nephi 31:3; D&C 1:24) and may be interested in the
degree to which the Prophet’s language may have influenced the translation of
the text. Consequently, the question of Joseph Smith’s influence on the text of
the Book of Mormon is one of considerable interest to both Latter-day Saints
and non–Latter-day Saint students of religion.
In an attempt to justify this significant omission, Criddle and
associates noted that Joseph Smith usually dictated his writings to others.
They cite Dean Jessee, the leading authority on Joseph Smith’s personal
writings, who explains (like Partridge above) that Joseph Smith appears to have
been much more comfortable as a speaker than a writer and that, consequently,
the majority of his writings are not in his own hand but in that of scribes (p.
469). However, Criddle and associates make the astonishing assertion that even
Joseph Smith’s holographic writings—those written in his own
hand—are unreliable examples of Joseph’s written style. “In
the case of Joseph Smith, we do not believe that even the small number of
letters written in his own hand can be reasonably attributed to him. Moreover,
were we to concede the reliability of these few letters, we would still not
have enough text to constitute an ample sample of known authorship” (p.
486). The authors make two claims: (1) that the writings of Joseph Smith in his
own hand are not a reliable source of data reflecting his writing style and (2)
that there are not enough of these writings to utilize in a wordprint study.
The first claim is mystifying, and the second claim is unjustified.
First, their hyperskepticism about Joseph Smith’s holographic
writings is not supported by historians. Dean Jessee, whom they cite in support
of this claim, states: “The real importance of Joseph Smith’s holographic
writings (the writings he produced with his own hand) lies in their being his expression
of his own thoughts and attitudes, his own contemplations and reflections. They
not only reveal idiosyncrasies of his education and literary orientation but
also clearly reflect his inner makeup and state of mind—his moods and
feelings. Furthermore, they provide a framework for judging his religious
claims.” 23 In a separate article, Jessee explains, “One of the best avenues, which is
undistorted by clerical and editorial barriers” for studying Joseph Smith
as a speaker and a writer, “is the Prophet’s holographic
writings—those materials produced by his own hand and hence by his own
mind.” 24 Writing that captures an author’s “inner makeup and state of mind” is
essential when performing a stylometric analysis.
The authors’ second
claim—that even if one wanted to use holographic material from Joseph
Smith there would not be enough to be useful—seems disingenuous, given
that they use samples from other candidate authors with sizes as small as only
114 words (p. 471). Available holographic material potentially includes (1)
holographic portions of Joseph Smith’s 1832 history (1,016 words); 25 (2) portions of Smith’s 1832–34 Kirtland,
Ohio, Journal (1,589 words); 26 (3) portions of his 1835–36 Kirtland, Ohio,
journal, which contains seven entries (four manuscript pages) in his own hand
(529 words); 27 (4) three letters partly in the hand of Joseph
Smith and partly in the hand of another writer (899 words); 28 and (5) twenty-four letters entirely written in
Joseph Smith’s handwriting totaling over 12,039 words.29 While these holographic texts are small in
quantity when compared to the entire corpus of historical documents dictated by
or prepared under the direction of Joseph Smith, it seems reasonable to expect
that a serious researcher would use these materials and could thereby obtain a
reliable and adequate sample for the purposes of authorship analysis.
After the paper by Criddle and associates was published and this
most obvious error in their analysis was pointed out, Matthew Jockers attempted
to justify the error in an unpublished manuscript. A review and analysis of
that manuscript is provided in the appendix to this paper.
Misapplying a Closed-Set Technique
for an Open-Set Problem
In their study, Criddle and associates treat the set of candidate
authors as a “closed set,” assuming that they knew with certainty
that the true author was one of the authors in their candidate set. Although
such an assumption would be appropriate when using NSC in the genomic studies
for which it was originally developed, this is not appropriate in most
authorship attribution studies. The case of The Federalist Papers is a situation where the true author was known to be in the candidate
set—the twelve disputed articles were written by either Alexander
Hamilton or James Madison, and by no one else. Such a well-defined closed-set
problem as The
Federalist Papers is a rarity in authorship attribution studies.
Although Criddle and associates show that NSC performed well in
an analysis of The Federalist Papers, this is to be expected
when applying a closed-set procedure to a closed-set problem. The case of the
Book of Mormon is clearly not the same type of problem. In their study, Criddle
and associates did not allow for the possibility that the Book of Mormon was a
translation of writings authored many centuries ago, nor (as discussed in the
previous section) did they consider the option that most secular critics deem
most plausible: that Joseph Smith himself was the author. Not allowing for
either possibility prejudiced their study’s results from the start.
To understand the consequences of naively applying the NSC
classification technique indiscriminately, let’s consider four cases in which
we use a closed set of candidate authors when clearly an open-set should be
First, if we naively apply NSC to The Federalist Papers using Criddle and associates’ set of candidate authors and using their way of
interpreting the results, we find with 99 percent probability that Sidney
Rigdon wrote thirty-four of The Federalist Papers published in 1788, before he was
even born (he was born in 1793). If we ignore important potential authors,
Criddle and associates’ technique will mislead us with a high level of
confidence in a misattribution.
Next, if we propose that the Spalding-Rigdon theory applies to
the King James Bible as well as to the Book of Mormon and then naively apply
NSC to the Bible using Criddle and associates’ closed set of authors, we find
with 99 percent probability that Sidney Rigdon wrote about 30 percent of the
Bible. If one wishes to attach any validity to Criddle and associates’ finding
about Rigdon as an author of the Book of Mormon, he or she must also attach the
same level of validity to Rigdon’s authorship of the Bible.
Similarly, if we concoct the absurd scenario that one or more of
a closed set of five early anti-Mormon writers—Alexander Campbell (1831),
Eber D. Howe (1834), Daniel Kidder (1842), Tyler Parsons (1841), and Walter
Scott (1841)—wrote the Book of Mormon, when we naively apply NSC as
Criddle and associates did in their study, we find that Parsons was the
principal author of the Book of Mormon since NSC attributed 65 percent of the
chapters to him with greater than 99 percent probability.
Finally, applying that naive approach to the paper under review
and using its candidate set of authors, we find with 99 percent probability
that Oliver Cowdery (who died in 1850) wrote the Jockers, Witten, and Criddle
paper published in 2008. Clearly, this approach produces absurd results when naively
employed unless Criddle and associates are willing to disavow authorship of
their own paper!
We can see from these examples how easily researchers could
deceive themselves into thinking they had found evidence in support of a
hypothesized authorship attribution regardless of how impossible or baseless it
might be. We can illustrate this graphically with an additional seemingly
plausible example. Let us propose that the Book of Mormon was written by either
Solomon Spalding or James Fenimore Cooper, the author of Last of the Mohicans.
We base this conjecture on the simple facts that both authors lived during the
same time period (Spalding 1761–1816 and Cooper 1789–1851), both
wrote their documents prior to the publication of the Book of Mormon in 1830
(Spalding’s “Manuscript Story” circa 1800 and Cooper’s Last of the
Mohicans in 1826), both wrote in the same genre (historical fiction), both used
the same geographic setting for their stories (northeastern North America),
and, most importantly, both used Native Americans as their subject matter. Now,
since the Spalding-Rigdon theory alleges that Spalding’s work was the basis for
the historical narrative in the Book of Mormon with Rigdon contributing the
doctrinal content, and since Rigdon is not included in our Spalding-Cooper theory,
we test our theory by examining the writing styles in only the chapters of the
Book of Mormon that are primarily historical in nature. We use the same
noncontextual words as Criddle and associates to determine the word-use
frequencies in the texts.
Figure 5 presents a principal components plot of the Book of
Mormon texts along with Spalding’s “Manuscript Story” texts and
of the Mohicans texts. We can easily see that the writing styles of
nineteenth-century authors Spalding and Cooper are more similar to each other
than they are to the writing styles exhibited in the Book of Mormon. It is also
obvious that Spalding is not a better candidate author for the Book of Mormon
than Cooper, who we know did not contribute to the Book of Mormon. In fact, in
the context of the Book of Mormon, Spalding is more likely to have written Last of the
Mohicans than he is to have influenced the writings in the Book of
From the examples above, it is clear that when any potpourri of
authors is collected and then a closed-set procedure is used to assess
attribution, the style of at least one of the candidate authors will always be
identified as “closest to” the style of the author of the test text.
It is also equally clear that “closest to” (a relative comparison)
does not necessarily mean “close to” (an absolute comparison), and
therefore caution is always necessary in interpreting the results.
Confusing “Closest” to Mean “Close”
The logic of Criddle and associates’ approach is no different
than asking, “Choosing among Boston, New York, and Chicago, which city is
closest to Los Angeles?” and then, upon finding that there is a 99 percent
probability that Chicago is the closest, concluding that “Chicago is the
city in the United States that is closest to Los Angeles.” In addition,
finding that one city of three candidate cities is “closest” to some
target city does not mean the cities are necessarily “close” to each
other. Just as Chicago might be closest to Los Angeles given the closed set
consisting of Chicago, New York, and Boston, certainly Chicago is not closest
given the open set of Chicago, New York, Boston, or any other city in the
Also, since Chicago and Los Angeles are half a continent apart,
few people would say they are “close” to each other, let alone that
they are the same city. In similar fashion, Criddle and associates assert that
when, according to their calculations, the writing style in a test text is “closest”
to one author’s style, then the two styles are “close.” In fact, they
imply that the styles are close enough to be considered identical. This is
Misinterpreting Relative Probabilities as Absolute
Since the NSC technique is a closed-set analysis technique, the
probabilities of closeness of writing style calculated by Criddle and
associates can be interpreted only as relative probabilities. That is to say,
the probabilities are only relative to the authors in the closed set of
candidate authors. Yet Criddle and associates present their calculations as
absolute probabilities, which would require that all possible outcomes had been
included in the computations. They obviously misinterpreted the probabilities
as saying, for example, that there is a greater than 99 percent chance that
Rigdon wrote a particular text, when the correct interpretation is that there
is a greater than 99 percent chance that Rigdon’s writing style is closer to
the style exhibited in a particular text compared to the other author candidates used
in the analysis. They give the false impression that their 99
percent computation is an absolute measure applicable to all possible
candidates, when it applies only to the specific set of authors they choose to
Ignoring a High Rate of False
In determining the reliability
of an analytical technique, a researcher will use “positive controls”
and “negative controls.” In a stylometric analysis, authors will be
included in the candidate set who are known to have contributed some of the
test texts. These serve as positive controls to test if the method can identify
authors for whom some texts should be attributed. Conversely, authors will be
included in the candidate set who are known not to have contributed any of the
test texts. These serve as negative controls to determine whether the method
can exclude authors to whom texts should not be attributed. In the Criddle and
associates study, texts by Isaiah and Malachi were composited together into one
set of texts to use as a positive control since those ancient prophets had
definitely authored some of the chapters in the Book of Mormon. Similarly,
texts by Joel Barlow and Henry Wadsworth Longfellow were included as negative
controls since those poets had definitely not authored any of the chapters.
Although the NSC technique was able to exclude Barlow and
Longfellow, it did poorly with Isaiah and Malachi. NSC correctly classified
twenty of twenty-one Isaiah and Malachi chapters, but it misclassified
forty-two other chapters as being authored by Isaiah and Malachi. A technique
that makes twice as many false classifications as correct classifications for
the control texts cannot possibly be considered to be a reliable technique.
Consequently, whatever other classifications it produced must be viewed as
unreliable and uninterpretable. It is astonishing that Criddle and associates
ignored their technique’s high rate of false classifications.
Using Circular Statistical Thinking
Statistical methods are not foolproof and must be used correctly
to produce reliable results. For example, if a statistical method is used to
fit a straight line to a set of data for two variables, x and y,
the method will give a straight line even if the data follow a curved pattern
(see fig. 6). To deal with this issue, an assessment of the data relative to a
proposed model should be carried out before fitting the model, and a
confirmatory goodness-of-fit test should be done after fitting. Concluding that
a straight-line model is appropriate simply because a straight line can be fit
to a data set is obviously fallacious circular reasoning. Justifying
straight-line predictions of y from values of x by saying that the predictions are correct assuming the straight-line model
to be correct could lead to grossly incorrect predictions.
Criddle and associates made exactly this kind of mistake in their
stylometric analysis. Without checking the fit, they assumed that every chapter
of the Book of Mormon was written by one of their seven candidate authors
(Rigdon, Spalding, Cowdery, Pratt, Isaiah/Malachi, Longfellow, and Barlow).
Then, using NSC to assign each of the Book of Mormon chapters to one of their
set of authors, they concluded that since almost all of the noncontrol chapters
were assigned to one of the noncontrol authors, they had discovered “strong
support for the Spalding-Rigdon theory of authorship.” However, they had
simply forced a model on the data and then circularly concluded that agreement
of the predictions with their model provided evidence for their model.
Just because a model can be mathematically fit to some set of data does not mean it is
the right model for the data. Neglecting to check whether the data were
actually consistent with the model applied to them is a serious mistake,
whether due to ignorance, inexperience, or willful blindness. Such verification
is not easy to do with highly multivariate data such as stylometric data, but
it is nonetheless necessary if one wishes a reliable analysis with
Disregarding Statistical Problems of
Homogeneity and Multiplicity
Criddle and associates disregarded two fundamental statistical
issues in their analysis: homogeneity of variance and multiplicity in
hypothesis testing. The NSC procedure employed by them assumes that the
variance 30 of the word frequencies in the text blocks is the same (homogenous) for all of
the text blocks. However, the text blocks in their study ranged in size from
about one hundred words to more than fifteen thousand words. The variances
calculated in text blocks spanning such a wide range will produce widely
differing variance estimates. Criddle and associates did not realize that the
NSC results will have questionable reliability when the homogeneity of variance
assumption is violated.
Further, the study
simultaneously classified 239 chapters from the Book of Mormon into seven authorship
categories in a single statistical procedure. In such situations of multiple
simultaneous classifications (multiplicity), some of the calculated
probabilities will appear to be unusually large even though they were simply
the result of chance. The probability that a text should be associated with a
certain candidate author versus another can be overstated. For example, a text
on the stylistic fringe of an author’s cluster of texts can stray into a nearby
author’s cluster and appear to be closer to that author rather than the true
author. Consequently, classification probability results must be interpreted
collectively rather than individually so as not to overinterpret the results.
Criddle and associates did not account for the multiplicity effect of
classifying a large set of test texts when interpreting their results.
Confounding the Primary Candidate
Author’s Differing Writing Styles
Finally, Criddle and associates assumed that an author’s writing
style is constant throughout his or her lifetime. They should have investigated
this assumption, particularly for their prime candidate. If they had, they
would have discovered that the Rigdon texts written prior to 1846 show evidence
of being systematically different from those written after 1863. Most notably,
while both sets of texts are distant from the Book of Mormon chapters, Rigdon’s
post-1863 writings are closer to the Book of Mormon than his pre-1846 writings.
This is illustrated in figure 7.
If Rigdon had been involved somehow in composing the Book of
Mormon, the “early Rigdon” rather than the “late Rigdon”
texts would be closer to the Book of Mormon chapters. The opposite is the case,
and this clearly contradicts the Spalding-Rigdon theory for Book of Mormon
authorship. The existence of two distinct Rigdon styles makes interpreting
Criddle and associates’ results highly problematic. Since they composited texts
containing Rigdon’s two differing writing styles, they blur what they call “Rigdon’s
signal.” Whatever “signal” might have been present in the texts
contained two styles rather than one distinguishable style. Because the Rigdon
writing style closest to the Book of Mormon was chronologically disjointed from
the book’s publication, whatever “signal” that Criddle and associates
thought they had found was a signal that came into existence over thirty years
too late to support their contention. (If anything, the results might suggest
that Book of Mormon language and style influenced Rigdon’s later style, and not
the other way around.)
Fifth Study: Extended Nearest Shrunken
Most recently, in a new study we developed a modification to the
closed-set Nearest Shrunken Centroid classification method (NSC) to enable it
to be applied to open-set classification problems—Extended Nearest
Shrunken Centroid (ENSC) classification.31 The open-set modification allows for the existence of an unknown candidate
author with a distribution of characteristic features nominally consistent with
the test text and incorporates this possibility into the calculation of the
probabilities that the writing styles are similar. Without including the
possibility of an unknown author, if the candidate set does not include the
true author (using a closed-set approach for an open-set situation), the
calculated probabilities can be grossly overstated and lead to entirely
erroneous interpretations. The ENSC technique also accounts for differences in
text sizes and for the effect of multiple simultaneous comparisons.
In applying this technique, we used the same 110 characteristic
words as in the Criddle and associates study, as well as their
chapter-by-chapter designation of text blocks from the Book of Mormon. However,
we used only the “early Rigdon” texts to represent Rigdon’s style and
included Joseph Smith as a candidate author along with the possibility of an
unknown author. The open-set ENSC method produced far different results than
those reported by Criddle and associates.
To illustrate the results, first we present in figure 8 plots
showing the Book of Mormon texts along with the texts from the candidate
authors. The 110 dimensions of the word frequencies have been projected into a
two-dimensional space defined by the first two principal components. The
sequence of plots shows the clusters of texts for each author individually in
relation to the Book of Mormon texts. Overall, it can be seen that in all cases
the candidate authors’ texts cluster separately from the Book of Mormon texts.
Further inspection shows that Solomon Spalding’s cluster is actually the
farthest away from the Book of Mormon, with Sidney Rigdon’s cluster almost as
far removed. Parley Pratt’s cluster is grouped with Spalding and Rigdon.
Next we can see in figure 8 that Oliver Cowdery’s and Joseph
Smith’s clusters are closer to the Book of Mormon cluster than Spalding’s or
Rigdon’s, thus confirming Criddle and associates’ error in not including Joseph
Smith in their set of candidate authors. Finally, we note that, as a group, the
nineteenth-century authors are far more like each other in writing style than
they are like the writers of the Book of Mormon.
Next we present in figure 9 a comparison of the probability
results of applying NSC and ENSC to the Book of Mormon texts. It is important
to remember that the mathematics in these analyses is not asking, “Who
wrote these texts?” The mathematics is asking, “Which texts have the
most similar patterns?” The naively applied NSC method estimates that 61
percent of the chapters in the Book of Mormon are most similar in style to
texts written by Spalding or Rigdon, without Joseph Smith included as a candidate
author. When Joseph Smith is included, the Spalding-Rigdon chapter-attribution
proportion drops to 55 percent. But properly addressing the analysis as an
open-set problem using the ENSC method, the Spalding-Rigdon proportion of the
chapters is a mere 8 percent, and we find that 73 percent of the chapters are
attributed to “Someone Else” other than the candidate set of authors.
However, for completeness we
need to take the analysis one step further. The control authors are only useful
to demonstrate the reliability of the analytical technique. So, after doing
preliminary tests, Criddle and associates should have removed the control
authors (Longfellow, Barlow, and Isaiah/Malachi) to make their final
probability computations. To complete the study properly, after showing the
reliability of ENSC, we excluded the control authors, included Joseph Smith as
a candidate author, and used only the “early” Sidney Rigdon texts as
Criddle and associates should have done. When we did so, we found that NSC
assigns only 12 percent to Spalding-Rigdon while assigning 61 perent of the
chapters of the Book of Mormon to Cowdery. These results are shown in figure
10. Therefore, even when naively using NSC, Criddle and associates should have
concluded that the evidence does not support the Spalding-Rigdon theory without
including Cowdery as the primary actor in their theory.
As also shown in figure 10, ENSC assigns 0 percent of the Book of
Mormon chapters to Spalding-Rigdon, 4 percent to Cowdery, 3 percent to Smith,
and 93 percent to “Someone Else,” indicating that the writing styles
of the candidate authors show very little resemblance to the writing styles in
the Book of Mormon. Finding that the writing styles of Joseph Smith and Oliver
Cowdery are perhaps slightly evident in the Book of Mormon texts is not
inconsistent with the claim that Joseph translated the entire book with Oliver
acting as his scribe, and that Oliver’s hand transcribed the final manuscript
for the printer in preparation for its first publication.
These results confirm that the Criddle and associates study was
fatally flawed in concept and execution. Contrary to their contention,
stylometric evidence does not provide credible support for the claim that the
writing styles exhibited in the Book of Mormon match their candidate
authors—Spalding, Rigdon, Pratt, or Cowdery. In fact, the evidence
supports the claim that someone other than their candidate authors wrote the book.
This is true even when Joseph Smith is considered as a candidate author.
Therefore, based on these findings, we conclude that Criddle and
associates have greatly exaggerated their claim to have calculated astronomical
odds in favor of Spalding-Rigdon authorship of the Book of Mormon. The results
of a properly conducted stylometric analysis are consistent with the Larsen et
al. and Hilton results: stylometric evidence does not support the
Spalding-Rigdon theory of Book of Mormon authorship.
Although conceptually attractive, NSC classification has limited
applicability in stylometric analysis. The ENSC method is far better suited for
the analytical challenges faced by researchers investigating open-set
In sum, an authorship attribution study requires the consistent,
coherent, and congruent conjunction of historical, biographical, and
stylometric evidence to support the conjecture of a writer as the author of a
text with disputed authorship. Such a combination of mutually supporting
evidence was not set forth by Criddle and associates.32 Even as a stylometric analysis the Criddle and associates study is invalid
since they made a fundamental error in their study design by considering Book
of Mormon authorship to be a closed-set problem and then making the logical error
of saying the results exclude any other possible authorship, when in fact the
researchers had not even allowed for the possibility of other authors in their
study design. The open-set possibility is sometimes called the “none of
the above” possibility, and in authorship attribution studies an open set
is more often the case than not.
The Criddle and associates study used the Nearest Shrunken
Centroid (NSC) classification method in an attempt to find evidence in support
of the Spalding-Rigdon theory. However, their study design was fundamentally
flawed. Although NSC is a sound classification technique, the Criddle and
associates study was an unsuitable and mistaken use of the technique. The
compounding effect of at least eight major errors rendered their results
The paper’s statistical methodology was innovative but misapplied
because they failed to realize the need to use an open-set procedure and they
did not account for the statistical complications of applying a genomics technique
to stylometric analysis.
We conclude that Criddle and associates’ research methodology
applied to the long-discredited Spalding-Rigdon theory is fatally flawed
and does not provide any new insights into Book of Mormon authorship. Sidney
Rigdon did not write the Book of Mormon.
Appendix: Exposing a
In an unpublished manuscript,33 Matthew Jockers attempts to justify the research decision not to include Joseph
Smith as a candidate author of the Book of Mormon in the study by Jockers,
Witten, and Criddle reported in Literary and Linguistic Computing in 2008.34 Jockers might well seek to justify this decision since this methodological
lapse alone is fatal to the credibility of the published paper. As we will see,
this effort at after-the-fact justification on the basis that Smith’s “personal
writings reveal a great deal of stylistic variation” is nothing but
self-serving special pleading.
First of all, “a great deal of stylistic variation” is
hardly a basis upon which to exclude an author as a candidate, especially when
that person is listed as the sole “author” on the first printed
edition of the book. Further, all historical accounts corroborate Smith’s claim
that he dictated the book to scribes word by word.35 No one else can reasonably be suggested as the prime candidate for authorship.
All other candidates must be considered as secondary candidates at best.
Jockers states that “the Smith material is too heterogeneous
[highly variable] to be considered a genuine sample of Smith’s style.”
High variability certainly makes stylometric analysis difficult, but that
cannot be used as a reason to exclude the most likely author as a candidate. If
the analytical method is not capable of handling high variability, then Jockers
should acknowledge his method’s weakness, abandon it, and find a more capable
method. This is how statistical analysis ought to proceed. Further, if he is
only interested in studying easy problems, then he should acknowledge his
preference, abandon the difficult problem, and find an easier problem to study.
In addition, Jockers offers no basis for what constitutes “too
heterogeneous.” The reader wants to ask, “Compared to whom? According
to what scale of measurement?” Jockers does not say. Nor does he say why
high variability in an author’s style indicates that a writing sample is not
genuine. Jockers has no grounds upon which to make such a comparative statement
without showing that the other candidate authors had a consistent style while
Smith did not, and that the other candidates’ texts are reliable indicators of
style while Smith’s are not. He fails to do so, and so this claim is nothing
but his own impression of the problem, with no mathematical basis behind it.
With the intent of filling in the gap caused by the exclusion of
Joseph Smith as a candidate author in the published study, Jockers compiled in
his unpublished study a set of twenty-four documents ranging in size from a
mere 112 words to 2,300 words in Smith’s handwriting. These he used to
characterize Smith’s style using his previous methodology. As the test set he
compiled ninety-six documents ranging from 105 to 10,927 words dictated by
Smith to twenty-three different scribes. He added to this set 219 texts by the
candidate authors in the published paper—Spalding, Rigdon, Cowdery,
Pratt, Isaiah/Malachi, Longfellow, and Barlow.
Using cluster analysis to group texts with similar style and to
separate texts with differing styles, Jockers found style variations—as
one would expect—among the texts dictated to different scribes. However,
he does not show that the relative differences in any way distort or mask
Joseph Smith’s style within the documents. So his analysis provides no useful
Next he applied NSC using the published study’s closed-set
candidate authors. He reports that with a set of 106 words he could distinguish
among these authors, but he was able to achieve an error rate no lower than 13
percent. This is an extraordinarily high error rate. For The Federalist
Papers he reported a 0 percent error rate. The Federalist
Papers is a closed-set problem, as are Joseph Smith’s writings, so
the error rates could reasonably be expected to be about the same. This
unacceptably and anomalously high error rate should have indicated either that
he did not have enough words in the stylistic feature set or that he did not
have a set of truly useful distinguishing words. Ignoring this obvious
weakness, he proceeded to apply his method anyway. The results are predictable
For reasons that he does not explain, Jockers considered
Spalding, who had died in 1816, when Joseph Smith was still a boy, to be a
viable author for Smith’s personal writings. In his results, some of the ninety-six
Smith documents were attributed to Spalding, who could not have had anything to
do with Joseph Smith’s writings since none of the Rigdon-Spalding theorists
have yet managed to bring Spalding back from the dead to compose the Prophet’s
diaries. Instead of seeing a big red flag telling him that his method was not
informative, Jockers asserts that Joseph Smith was so influenced by Spalding
that even his letters to his wife and his diary entries were modeled after
Spalding. This is clearly a flawed conclusion. In addition, there is no
historical evidence to support the claim that Cowdery, more than being just
Joseph’s scribe, was instead the author of Joseph’s writings. But Jockers’s
theory requires that Cowdery was more likely the author of Joseph Smith’s known
writings than Joseph Smith himself. Apparently any data point, no matter how
incongruous, can be marshaled to support a version of the Spalding-Rigdon
theory, requiring ad hoc fixes.
Jockers states that there is “a curious stylistic affinity
between the style of Spalding and the style of the personal writings” of
Joseph Smith. There is nothing curious about it at all. The style measurement
is not real. If a “Spalding signal” shows up so prominently (as
Jockers claims) in texts that Spalding could not possibly have written, then
any assertion that his style is contained in other texts of questioned
authorship is obviously invalid. Whatever Jockers measured must have been
nothing more than noise. His “Spalding signal” is just “Spalding
noise”—to which his biases tune his own ear.
Let’s look at more details in his results. His method says that
14 percent of the ninety-six Joseph Smith documents were written by Spalding,
Longfellow, Barlow, or Isaiah/Malachi. Going further, 10 percent were written
by dead people—Spalding and Isaiah/Malachi. A method that produces such
unreliable results is obviously useless.
Of the ninety-six documents in Jockers’s test set, only twelve
can be used to compare the possible effect of Joseph Smith’s scribes on the
documents attributed to his authorship. These twelve documents involve other
individuals acting as Joseph’s scribe: Cowdery (eight instances), Rigdon
(three), and Pratt (one) acted as Joseph’s scribes. For these documents,
Jockers’s results indicate the following:
1. None of the twelve
were attributed to Joseph Smith. So either Joseph Smith did not dictate any of
the documents attributed to his authorship (i.e., personal letters and personal
diary entries)—an unlikely scenario—or Jockers’s method is
2. In the case of
Rigdon as scribe, all of his written documents were attributed by Jockers to
Pratt. So either Rigdon inexplicably wrote in Pratt’s style when he was Joseph’s
scribe or Jockers’s method is not informative.
3. In the case of Pratt
as scribe, the only document tested by Jockers was attributed to Pratt. So
either a single text of only 123 words (a short paragraph) is too small to
reflect anything other than the style of the hand holding the pen or Jockers’s
method is not capable of identifying the true author.
4. In the case of
Cowdery as scribe, two documents were attributed to Cowdery, two to Rigdon, two
to Pratt, and two to other candidates. So either Cowdery was such a literary
genius that he could write in his own style and mimic with equal ease the style
of two of his friends, plus the style of a renowned poet and two Old Testament
prophets for no useful purpose, or Jockers’s method produces meaningless
Jockers’s conclusions are an attempt to justify his
methodological irregularities by claiming either that (a) Joseph Smith somehow
did not write any of the documents (even those written in his own hand) or that
(b) his writings are inadmissible as evidence of his personal writing style
because of “stylistic variation” and thus need not be considered. If
Jockers is admitting that his methodology is incapable of dealing with “stylistic
variation,” then he is admitting that his method is inadequate for
Paul J. Fields
(PhD, Pennsylvania State University) is a consultant specializing in research
methods and statistical analysis.
Schaalje (PhD, North Carolina State University) is a professor of statistics at
Brigham Young University.
(MA, Brigham Young University) is a research scholar for the Neal A. Maxwell
Institute for Religious Scholarship, Brigham Young University.
D. Howe, Mormonism
Unvailed . . . (Painesville, OH, 1834). For an overview of the
Spalding theory, see Matthew Roper, “The Mythical ‘Manuscript Found,'” FARMS
Review 17/2 (2005): 7–140; and Roper, “Myth, Memory, and ‘Manuscript
Found,'” FARMS Review 21/2 (2009): 179–223.
Latter-day Saint and non–Latter-day Saint students of the issue have
concluded that even if “Manuscript Story” was not the only version of
Spalding’s tale, the story would not have differed substantially in content and
style from the Oberlin document. Roper argues that “Manuscript Story”
can be seen as fictional apologetic for the theory that the Indians were the
lost ten tribes. See Roper, “Myth, Memory, and ‘Manuscript Found,'”
3. Roper argues that elements in
the 1833 testimony collected by Hurlbut and later testimony from other Conneaut
associates of Spalding, which some Spalding theorists take as evidence for
additional manuscript sources for the Book of Mormon, are more plausibly
accounted for as describing elements of the Oberlin story. See Roper, “Myth,
Memory, and ‘Manuscript Found,'” 179–223. Roper and Fields examine
the misuse of historical evidence by Criddle and associates in the essay that
immediately follows this one.
L. Jockers, Daniela M. Witten, and Craig S. Criddle, “Reassessing
authorship of the Book of Mormon using delta and nearest shrunken
centroid classification,” Literary and Linguistic Computing 23/4 (2008): 465–91.
is a branch of molecular biology concerned with researching the structure,
function, evolution, and mapping
of the entire DNA sequences of organisms.
H. Partridge, “Characteristics of Joseph Smith’s Style and Notes on the
Authorship of the Lectures on Faith,” Task Papers in LDS History 14
(December 1976), 20.
“Characteristics of Joseph Smith’s Style,” 5.
“Characteristics of Joseph Smith’s Style,” 23.
“Characteristics of Joseph Smith’s Style,” 6, 7.
10. Partridge, “Characteristics
of Joseph Smith’s Style,” 13.
“Characteristics of Joseph Smith’s Style,” 15.
12. Wayne A.
Larsen, Alvin C. Rencher, and Tim Layton, “Who Wrote the Book of Mormon?
An Analysis of Wordprints,” BYU Studies 20/3 (1980):
225–51; reprinted in Noel B. Reynolds, ed., Book of Mormon Authorship: New
Light on Ancient Origins (Provo, UT: Religious Studies Center,
Brigham Young University, 1982), 157–88.
Rencher, and Layton, “Analysis of Wordprints,” 240.
14. D. James
Croft, “Book of Mormon ‘Wordprints’ Reexamined,” Sunstone,
March–April 1981, 15–21.
15. Wayne A.
Larsen and Alvin C. Rencher, “Response to Book of Mormon ‘Wordprints’
Reexamined,” Sunstone, March–April 1981, 23–26.
16. David I.
Holmes, “A Stylometric Analysis of Mormon Scripture and Related Texts,” Journal
of the Royal Statistical Society, Series A 155 (1992): 91–120.
17. John L.
Hilton, “On Verifying Wordprint Studies: Book of Mormon Authorship,” BYU Studies 30/3 (Summer 1990): 89–108; reprinted in Noel B. Reynolds, Book of Mormon
Authorship Revisited: Evidence for Ancient Origins (Provo, UT:
FARMS, 1997), 225–53.
18. A. Q.
Detection: How to Prove Authorship and Fraud in Literature and Documents (New York: Charles Scribner’s Sons, 1978).
19. Hilton, “On
Verifying Wordprint Studies,” 101.
20. See note 16.
21. G. Bruce
Schaalje, John L. Hilton, and John B. Archer, “Comparative Power of Three
Author-Attribution Techniques for Differentiating Authors,” Journal of
Book of Mormon Studies 6/1 (1997): 47–63.
22. Jockers, Witten,
and Criddle, “Reassessing Authorship of the Book of Mormon,”
23. Dean C. Jessee, Personal
Writings of Joseph Smith, rev. ed. (Salt Lake City: Deseret Book,
24. Dean C. Jessee, “Priceless
Words and Fallible Memories: Joseph Smith as Seen in the Effort to Preserve His
Discourses,” BYU Studies 31/2 (Spring 1991): 33.
document is the earliest extant attempt by the Prophet to write a history of
his life, and his only autobiographical work containing his own handwriting.”
Dean C. Jessee, ed., The Papers of Joseph Smith, Volume 1: Autobiographical and
Historical Writings (Salt Lake City: Deseret Book, 1989), 1.
26. “Despite its
brevity, this first journal contains more of JS’s handwriting than do any of
his other journals. Almost half of the entries in the journal were written
either entirely or primarily by JS himself; some of the remainder were
apparently dictated. His openly expressed hopes and concerns, prayers and
blessings, and observations on his own state of mind are a rich source of
insight into spiritual and emotional dimensions of JS’s personality.” Dean
C. Jessee, Ronald K. Esplin, and Richard Lyman Bushman, eds., The Joseph
Smith Papers: Journals, Volume 1, 1832–1839 (Salt Lake City:
Church Historian’s Press, 2008), 4.
27. Jessee, Esplin,
and Bushman, Joseph
Smith Papers, 1:55.
28. Joseph Smith to
William W. Phelps, 27 November 1832 (700 words); Joseph Smith to Henry G.
Sherwood, 7 November 1839 (58 words); Joseph Smith to Emma Smith, 27 June 1844
(142 words). The numbers in parentheses indicate the number of words in these
letters written in Joseph Smith’s hand.
written entirely in Joseph Smith’s hand include the following: Joseph Smith to
Hyrum Smith, 3–4 March 1831 (915 words); Joseph Smith to Emma Smith, 6
June 1832 (950 words); Joseph Smith to Emma Smith, 13 October 1832 (983 words);
Joseph Smith to Newell K. Whitney, 1833/1834 (130 words); Joseph Smith to
William W. Phelps, 18 August 1833 (2,366 words); Joseph Smith to Emma Smith, 19
May 1834 (415 words); Joseph Smith to Almira Scobey, 2 June 1835 (134 words);
Joseph Smith to Sally Phelps, 20 July 1835 (284 words); Joseph Smith to Emma
Smith, 4 November 1838 (907 words); Joseph Smith to Emma Smith, 12 November
1838 (580 words); Joseph Smith to Emma Smith, 1 December 1838 (64 words);
Joseph Smith to Emma Smith, 21 March 1839 (676 words); Joseph Smith to Emma
Smith, 4 April 1839 (1,037 words); Joseph Smith to Emma Smith, 9 November 1839
(326 words); Joseph Smith to Emma Smith, 20 January 1840) (274 words); Joseph
Smith to the Wilkinson Family, February 1840 (103 words); Joseph Smith to Newel
K. Whitney, 12 December 1840 (65 words); Joseph Smith “Agreement,” 14
May 1841 (211 words); Joseph Smith to Ebenezer Robinson, 24 February 1842 (30
words); Joseph Smith “Resolution,” 5 March 1842 (53 words); Joseph
Smith to the Whitneys, 18 August 1842 (469 words); Joseph Smith to Lucien
Adams, 2 October 1843 (92 words); Joseph Smith to William Clayton, 9 December
1843 (48 words); Joseph Smith to Barbara Matilda Neff, May 1844 (927 words).
30. Variance is a
measure of the dispersion or inconsistency among multiple observations of the
31. G. Bruce
Schaalje, Paul J. Fields, Matthew Roper, and Gregory L. Snow, “Extended
nearest shrunken centroid classification: A new method for open-set authorship
attribution of texts of varying sizes,” Literary and Linguistic Computing 26/1 (2011): 71–88.
32. The contextual
evidence does not warrant considering Sidney Rigdon as a viable candidate as
the author of the Book of Mormon,
as Matthew Roper and Paul Fields discuss in the essay that follows.
33. Matthew L.
Jockers, “Testing Authorship in the Personal Writings of Joseph Smith
Using NSC Classification,” 2011, accessed 28 July 2011, http://www.stanford.edu/~mjockers/pubs/SmithNSCAnalysis.pdf.
34. Matthew L.
Jockers, Daniela M. Witten, and Craig S. Criddle, “Reassessing authorship
of the Book
of Mormon using delta and nearest shrunken centroid classification,” Literary
and Linguistic Computing 23/4 (2008): 465–91.
35. See the following
essay by Matthew Roper, which discusses this historical context.