I would like to write a word about the beginnings of computer-aided readability assessment research in Canada during the ‘90s, as they show interesting ways of thinking and measuring the complexity of texts.
Sato-Calibrage
Daoust, Laroche and Ouellet (1997) start from research on readability as it prevailed in the United States : they aim at finding a way to be able to assign a level to texts by linking them to a school level. They assume that the discourses of the school institutions are coherent and that they can be examined as a whole. Among other things, their criteria concern lexical, dictionary-based data and grammatical observations, such as the amount of proper nouns, of relative pronouns and of finite verb forms.
Several variables measure comparable aspects of text complexity and the authors wish to avoid being redundant, so they use factorial analysis and multiple regression to group the variables and try to explain why a text targeted a given school grade. They managed to narrow down the observations to thirty variables, whose impact on readability assessment is known. This is an interesting approach. The fact that they chose to keep about thirty variables in their study shows that readability formulas lack depth and adaptability. This is quite straightforward.
The most important lesson they draw concerns the way future work should be done : the exact numerical result is not the most important, it is more useful to detect texts which exhibit special characteristics in order to design teaching or evaluation instruments (« repérer des textes possédant des caractéristiques spécifiques pour élaborer des instruments d’enseignement ou d’évaluation » p. 231). Their research was supported by users who gave feedback all along the process.
Intelligibility Assessment
Préfontaine and Lecavalier (1996) tackle the notion of intelligibility, which according to them encompasses the components of readability and goes beyond them (« dépasse et englobe les composantes de la lisibilité » p. 100). To measure it, they look at three different levels, i.e. micro-, macrostructural and conceptual levels.
They split the criteria to be met in four main indicator groups :
- A microstructural readability score is computed using a readability formula.
- The macrostructural one is obtained manually : the text is divided into sequences, then it is mainly about the number and the type of reading operations required by each sequence.
- The conceptual intelligibility score plays a major role. If the concepts are unknown to the reader, clear sentences will not help him cope with this problem (p. 105). Thus, they observe the familiarity of the concepts (with general terms considered as familiar, on a three-level scale) and their distribution (i.e. conceptual density, using this formula : number of concepts * 100). They also determine if the idea they express is a key to understand the text.
- The text difficulty according to the readers, using four-level Likert items.
It is an interesting way to combine several factors which account for text complexity. They look out for its conceptual dimension, which is of paramount importance. Nevertheless, the main drawback is that they have to assess the familiarity of the concepts by hand, which raises two problems : the reproducibility of the operation and its stability.
They conclude that intelligibility is hard to measure, mostly because the topics and the readers are varying. According to them, taking the age and the level of education of the readers as a point of reference is (once again) useful. They also mention that the interests of the readers could be an variable too. That leads to the notion of profile.
References
- F. Daoust, L. Laroche, and L. Ouellet, “Sato-Calibrage: Présentation d’un outil d’assistance au choix et à la rédaction de textes pour l’enseignement”, Revue québécoise de linguistique, vol. 25, iss. 1, pp. 205-234, 1996.
- C. Préfontaine and J. Lecavalier, “Analyse de l’intelligibilité de textes prescriptifs”, Revue québécoise de linguistique, vol. 25, iss. 1, pp. 99-144, 1996.