Rebuttal to JockersBy: Mormon Heretic
Ever since the Book of Mormon was published in 1830, critics have tried to show that it came forth as the result of fraud. One of the earliest theories was the Spaulding Theory. As the theory goes, Solomon Spaulding wrote an unpublished novel about a group of Romans from the time of Constantine that were blown off course from Britain to the Americas (check out Part 1 and Part 2). Somehow (never adequately explained) Sidney Rigdon obtained the manuscript, and then transferred it surreptitiously to Joseph Smith who added religious information.
Fawn Brodie put together an appendix in her book No Man Knows My History outlining problems with the theory. (I wrote about this in a post called Debunking the Spaulding Theory.) Most people think the theory has been debunked, though the theory still has some adherents, such as Dale Broadhurst who maintains a website in favor of the theory.
Wordprint studies try to determine the true author of text. The idea of a wordprint is similar to a finger print. Each person uses a certain set of words such as “a, but, and, the, etc” in a way that is unique. By collecting information on word usage, a wordprint theoretically can identify an author.
In 2008, Mathew Jockers, Daniela Witten, and Craig Criddle of Stanford University created a stir when they produced a peer-reviewed article in Oxford’s journal titled Literary and Linguistic Computing. The authors concluded that major portions of the Book of Mormon exhibited Sidney Rigdon and Solomon Spaulding’s writing style, thus creating a resurgence of interest in the Spaulding Theory.
Traditionally, wordprint studies have used a statistical technique known as the Delta Method. Jockers, et al compared the Delta method to a new technique called Nearest Shrunken Centroid (NSC). NSC has been used cancer studies, but this was the first time it has been used in wordprint studies. The Jockers study found the NSC method to be much more reliable than the Delta method. Many New Order Mormons and anti-Mormons were pleased with the study. But there were some big questions about the method.
In January 2011 Bruce Schaalje, Paul Fields, and Matthew Roper of BYU, along with Gregory Snow of Intermountain Health Care released a study outlining problems with the Jockers study in the same Oxford journal of Literary and Lingustic Computing. While acknowledging that NSC is a good method for wordprint studies, they detailed several problems with the Jockers study, noting a “naive application of NSC methodology” led to “misleading results.” Jockers et al had used a closed set of 7 authors for their study. Schaalje’s study showed that an open set of candidate authors “produced dramatically different results from a closed-set NSC analysis.”
I reviewed the Schaalje study in depth on my blog. For those looking for a “Cliff’s Notes” version, let me summarize the strengths of the BYU study over the Stanford study:
- Jockers excluded Joseph Smith as a candidate. Now Jockers noted this weakness, and had valid reasons for excluding Joseph, but this is still a MAJOR problem. Joseph frequently used scribes, even for personal letters and journals. Jockers felt that none of the writing samples could positively be identified as authentically written by Joseph, so they excluded him. I understand the concern, and think Jockers did the right thing, but this is still a massive problem.
- Schaalje’s method gives a “none of the above” above option. Jocker’s method had to pick a winner, even if none of the authors were a good match.
- The “goodness of fit” test. Schaalje creates a method to show Jocker’s conclusions were much weaker than he implied.
- Schaalje’s method was reliability tested against a known author. He did a test on the Federalist Papers including and excluding Alexander Hamilton as a candidate author. Jocker’s methodology picked Rigdon when Hamilton was excluded. Using Schaalje’s open set method, Schaalje’s method picked “none of the above” when Hamilton was excluded. When Hamilton was included, both Jockers and Schaalje’s method correctly picked Hamilton.
- Jockers used too small of sample texts (114 words) for his training set. Jockers noted this as a possible weakness, but Schaalje showed that this was a significant problem.
I have to say that the BYU guys really thought through this problem well. Jockers has plans for an updated study to include Joseph Smith, and others. Judging from the BYU study, I think the Stanford folks have some serious problems. What are your thoughts?