Amazon’s readability statistics by example

I already mentioned Amazon’s text stats in a post where I tried to explain why they were far from being useful in every situation : A note on Amazon’s text readability stats, published last December.

I found an example which shows particularly well why you cannot rely on these statistics when it comes to get a precise picture of a text’s readability. Here are the screenshots of text statistics describing two different books (click on them to display a larger view) :

Comparison of two books on Amazon

The two books look quite similar, except for the length of the second one, which seems to contain significantly more words and sentences.

The first book (on the left) is Pippi Longstocking, by Astrid Lindgren, whereas the second is The Sound and The Fury, by William Faulkner… The writing style could not be more different, however, the text statistics make them appear quite close to each other.

The criteria used by Amazon are too simplistic, even if they usually perform acceptably on all kind of texts. The readability formulas that output the first series of results only take the length of words and sentences into account and their scale is designed for the US school system. In ...

more ...

Canadian research on readability in the ‘90s

I would like to write a word about the beginnings of computer-aided readability assessment research in Canada during the ‘90s, as they show interesting ways of thinking and measuring the complexity of texts.

Sato-Calibrage

Daoust, Laroche and Ouellet (1997) start from research on readability as it prevailed in the United States : they aim at finding a way to be able to assign a level to texts by linking them to a school level. They assume that the discourses of the school institutions are coherent and that they can be examined as a whole. Among other things, their criteria concern lexical, dictionary-based data and grammatical observations, such as the amount of proper nouns, of relative pronouns and of finite verb forms.

Several variables measure comparable aspects of text complexity and the authors wish to avoid being redundant, so they use factorial analysis and multiple regression to group the variables and try to explain why a text targeted a given school grade. They managed to narrow down the observations to thirty variables, whose impact on readability assessment is known. This is an interesting approach. The fact that they chose to keep about thirty variables in their study shows that readability formulas lack ...

more ...

A note on Amazon’s text readability stats

Recently, Jean-Philippe Magué advised me of the newly introduced text stats on Amazon. A good summary by Gabe Habash on the news blog of Publishers Weekly describes the perspectives and the potential interest of this new software : Book Lies: Readability is Impossible to Measure. The stats seem to have been available since last summer. I decided to contribute to the discussion on Amazon’s text readability statistics : to what extent are they reliable and useful ?

Discussion

Gabe Habash compares several well-known books and concludes that the sentence length is determining in the readability measures used by Amazon. In fact, the readability formulas (Fog Index, Flesch Index and Flesch-Kincaid Index, for an explanation see Amazon’s text readability help) are centered on word length and sentence length, which is convenient but by far not always adapted.

There is another metric named ‘word complexity’, which Amazon defines as follows : ‘A word is considered “complex” if it has three or more syllables’ (source : complexity help). I wonder what happens in the case of proper nouns like (again…) Schwarzenegger. There are cases where the syllable recognition is not that easy for an algorithm that was programmed and tested to perform well on English words ...

more ...

About Google Reading Level

Jean-Philippe Magué told me there was a Google advanced search filter that checked the result pages to give a readability estimate. In fact, it was introduced about seven months ago and works to my knowledge only for the English language (that’s also why I didn’t notice it).

Description

For more information, you can read the official help page. I also found two convincing blog posts showing how it works, one by the Unofficial Google System Blog and the other by Daniel M. Russell.

The most interesting bits of information I was able to find consist in a brief explanation by a product manager at Google who created the following topic on the help forum : New Feature: Filter your results by reading level.
Note that this does not seem to have ever been a hot topic !

Apparently, it was designed as an “annotation” based on a statistical model developed using real word data (i.e. pages that were “manually” classified by teachers). The engine works by performing a word comparison, using the model as well as articles found by Google Scholar.

In the original text :

The feature is based primarily on statistical models we built with the help of ...

more ...

Lord Kelvin, Bachelard and Dilbert on Measurement

Lord Kelvin

Here is what William Thompson, better known as Lord Kelvin, once said about measure :

« I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be. »
William Thompson, Lecture on “Electrical Units of Measurement” (3 May 1883)

Bachelard

I found this quote in an early essay of the French philosopher Gaston Bachelard on what he calls “approached knowledge” (Essai sur la connaissance approchée, 1927). For him, measures cannot be considered for themselves, and he does not agree with Thompson on this point. According to him, the fact that a measure is precise enough gives us the illusion that something exists or just became real.

I quote in French, as I could find a English edition nearby, the page numbers refer to the book published by Vrin.

« Et pourtant, que ce soit dans la mesure ou dans une comparaison qualitative, il ...

more ...