The Disfluency Effect in Reading Comprehension: Findings from Paper and Screen Experiments

The Disfluency Effect in Reading Comprehension: Findings from Paper and Screen Experiments Ai Hagihara, Yuhi Sezaki, Sayaka Takano, Satoshi Nakamura (Meiji University) Akiyuki Kake (Wacom)

Introduction: Current Situation in Education The spread of digital devices → Use of smartphones, tablets, and laptops is increasing in schools. In Japan − Digital textbook adoption: about 88% nationwide − Computers: about 1.1 devices per student Digitalization is also progressing in education. 1

Introduction: Current Situation in Education fonts textbooks s reference book digital materials handwriting d blackboar own notes friends notes Many different text appearances exist in today’s learning environment. 2

Related Work: Text Appearance and Memory Disfluency Effect: Presenting text in a hard-to-read style can improve memory. Ai Hagihara Hard-to-read text may help memory, − When text is inverted upside down, it becomes easier to but how does it affect comprehension? remember the content. [Sungkhasette+ 2011] − Handwritten text that is harder to read can be remembered better than easy-to-read fonts. [Ito+ 2020] • Sungkhasettee, V.W., Friedman, M.C. & Castel, A.D.. Memory and metamemory for inverted words: Illusions of competency and desirable difficulties. Psychon Bull Rev 18, 2011. • Risa Ito, Karin Hamano, Kosuke Nonaka, Ippei Sugano, Satoshi Nakamura, Akiyuki Kake, Kizuku Ishimaru. Comparison of the Remembering Ability by the Difference Between Handwriting and Typeface, International Conference on Human-Computer Interaction (HCII 2020), Vol.CCIS, volume 1224, pp.526-534, 2020. 3

Research Question Does the difference in text appearance affect comprehension? Tom put in some counters. Points Number of counters Score 10 2 A 8 0 0 4 10 40 0 4 B (1) What is the score for A ? Equation: 10 × = Answer : points handwriting font 4

Related Work: Text Appearance and Comprehension − Using two fonts and the author’s handwriting − Comprehension was measured by a 5-point questionnaire. No clear effect of text appearance. [Tanigami 2019] − Using four fonts − Participants answered fill-in-the-blank questions. Hard-to-read text resulted in higher scores. [Negishi+ 2018] • Tanigami, A., 2019. The effects of font styles or handwriting on the fluency of reading texts and memory. Hikone Ronso , 18–29(in Japanese). • Negishi, I., Kodama, M., 2018. Influence of japanese font type on learning effectiveness. Kogaku Kyoiku (Japanese Journal of EngineeringEducation) 66, 48–412 (in Japanese). 5

Purpose Investigating the effect of text appearance on reading comprehension accuracy using fonts and handwriting Hypothesis Presenting a reading passage in a text appearance that is difficult to read makes it easier to understand. 6

Experimental Design: Text Appearance fonts From previous studies handwriting The two most contrasting styles in past ratings MS Gothic: like sans-serif fonts in English Handwriting 1: easy to read, rounded, neat, feminine MS Mincho: like serif fonts in English Handwriting 2: hard to read, strong, free, masculine • Ito, R., Hamano, K., et al., 2020. Comparison of the remembering ability by the difference between handwriting and typeface, in: HCI International 2020-Posters • Takano, S., Yamazaki, I., Ito, R., Hamano, K., Sugano, I., Nakamura, S., Kake, A., Ishimaru, K., 2022. Examining the effect of similarity to one’s own handwriting on ease of memory. Technical Report 2. Kenkyu Hokoku Human Computer Interaction (HCI). (in Japanese). 7

Experimental Design: Reading Comprehension Test Reading Comprehension Task ─ SPI Test ✦ Common reading test in Japanese job hunting. ✦ Designed to measure logical reading skills. ✦ This test does not require any prior knowledge. ✦ Answers are not written directly in the text, so participants need real understanding to answer. 8

Paper-based Experiment ✦ Printed on A4 paper ✦ Participants answer by handwriting ✦ Answer with Correct (◯) or Incorrect (×) 9

Example of the passage (translated in English) Before release into the wild, the crested ibis underwent over a year of training. This illustrated the difficulty of reintroducing a species that had once disappeared from the natural world. In a large cage that recreated a natural environment, trees were planted to serve as roosting spots. For feeding practice, loaches and bitterlings were released into a pond. To help the birds recognize natural predators, models of crows were placed and their calls were played. The successful release of the Oriental white stork in 2005 at a prefectural park in Toyooka, Hyogo Prefecture, was made possible thanks to environmental improvements carried out with the cooperation of local residents. On Sado Island as well, ponds were constructed to ensure feeding grounds. Among local farmers, there is growing momentum to keep rice fields filled with water even in winter. Residents should not be overly burdened in their daily lives for the sake of protecting the crested ibis. However, helping with reintroduction efforts within reasonable limits is necessary to maintain the ecosystem. The Ministry of the Environment plans to have 60 ibises successfully established in the wild by around 2015. According to data from transmitters attached to the released birds, they currently appear to be acting alone. In the future, if they are able to soar through the sky in flocks, this may also contribute to regional revitalization as a new tourism resource. The Ministry of the Environment is also conducting artificial breeding projects aimed at reintroducing endangered species such as the Tsushima leopard cat and the Okinawa rail. Creating environments in which rare species can live also leads to the conservation of satoyama, traditional rural landscapes that bring blessings to human life. Source: Yomiuri Shimbun, October 13, 2008, “We Want to See Ibises Soaring in the Sky” 10

Example of the question (translated in English) Based on the passage, mark ○ if the statement is true, or × if it is false. 1. To restore endangered animals to the wild, we should leave everything to nature’s blessings without human intervention. 2. In order to reintroduce endangered animals into the wild, people's understanding of ecosystems is essential. 3. Creating an environment where rare species can live also contributes to the conservation of satoyama (traditional rural landscapes). 11

You can see the full English version and some example questions from this QR code. 12

Paper-based Experiment Participants: 36 University students Presentation of Text Styles: • All participants experienced all text styles Time Limit: 3–5 min. depending on passage length 13

Paper-based Experiment Procedure Practice Test Reading Comprehension Test Questionnaire on the Completed Test PostExperiment Questionnaire ×8 repetitions Questionnaire Items: • Perceived readability of the presented text style • Confidence in one’s answers • ...and other related measures 14

Paper-based Experiment: Result Average Readability Scores by Text Style ＊p<0.01 MS Mincho is the easiest text style to read 15

Paper-based Experiment: Result Accuracy of Reading Comprehension by Text Style No significant difference in accuracy 16

Paper-based Experiment: Result Relationship Between Readability Ratings and Accuracy • High-readability group: 89.0% • Low-readability group: 84.3% The difference was not statistically significant. More readable text was associated with better performance. 17

Paper-based Experiment: limitation ✦ No significant differences in accuracy by text style or readability ✦ However, there may be differences in response time. → We conducted a screen-based experiment to measure response time. ✦ Readability was rated right after each question. Confidence in the answers may have influenced the ratings. 18

Screen-based Experiment ✦ Shown on a laptop and answered by clicking ✦ Questions and text on separate pages(navigated by buttons) 19

Screen-based Experiment Procedure Practice Test Reading Comprehension Tests PostExperiment Questionnaire Readability rating Change: ✦ Readability ratings were collected after all tasks ✦ Response times were recorded 20

Screen-based Experiment: Result Average Readability Scores by Text Style *p<0.01 **p<0.05 MS Gothic is the easiest text style to read 21

Screen-based Experiment: Result Accuracy of Reading Comprehension by Text Style No significant difference in accuracy 22

Screen-based Experiment: Result Relationship Between Readability Ratings and Accuracy • High-readability group: 76.4% • Low-readability group: 68.6% The difference was not statistically significant. However, it showed a trend against the hypothesis. 23

Screen-based Experiment: Result Relationship Between Number of Correct Answers and Average Response Time per User per User (s) Average Response Time 250 Positive correlation (r=0.31) 200 150 Spending more time on the task was associated with slightly higher accuracy. 100 50 0 10 12 14 16 18 Number of Correct Answers 20 24

Screen-based Experiment: Result Average Answer Time (s) Relationship Between Readability Ratings and Average Response Time MS Mincho Handwriting 1 Handwriting 2 Readability MS Gothic Texts rated as more readable tended to have longer response times. Less readable texts may not have been read carefully. 25

Summary of Results paper A4 paper screen laptop MS Mincho was rated as the most readable. MS Gothic was rated as the most readable. → It is often used in newspapers and textbooks. →It is often used in websites. The hypothesis was rejected. The hypothesis was rejected. = More readable text styles would lead to higher performance More readable text styles took longer response time. 26

Comparison of memory and comprehension In our previous memory test • Handwriting 2 (hard-to-read) had the highest score. In this comprehension test • Easy-to-read text tended to have higher scores. The effect of text appearance may be the opposite for memory and comprehension. Depending on the purpose of learning, changing the text appearance may be effective. • Ito, R., Hamano, K., et al., 2020. Comparison of the remembering ability by the difference between handwriting and typeface, in: HCI International 2020Posters 27

General Discussion Reading comprehension scores did not improve when the text was presented in a harder-to-read style. • People may not want to read long passages carefully if they are written in messy handwriting. • The disfluency effect may need an optimal level of cognitive load. • Long, complex texts may create too much load, reducing the effect. • With shorter texts, the effect may still occur. 28

Future Work Conduct further experiments targeting various types of understanding ✦ Test whether the disfluency effect applies to different comprehension tasks Apply findings to educational contexts ✦ Personalizing digital textbooks based on individual readability preferences ✦ Improving learning efficiency by optimizing text presentation 29