Don't Break the Melody: Encouraging Accurate Handwriting Practice with Sound Feedback

ACM Multimedia Asia 2025 (MMAsia 2025) Kuala Lumpur, Malaysia December 11, 2025 Don’t Break the Melody: Encouraging Accurate Handwriting Practice with Sound Feedback Reo Hatogai, Sayuri Matsuda, Kento Watanbe, Satoshi Nakamura (Meiji University) Akiyuki Kake (Wacom)

Melody-Based Handwriting System A handwriting system that generates melodies from pen strokes and mutes sound when tracing deviates. QR code of our system 1

Core Ideas of Our Approach 1. Proposed a melody-based handwriting system • Encourages accurate tracing while maintaining enjoyment 2. Conducted a controlled experiment comparing 3 auditory feedback conditions • Our method led to slower and more precise handwriting 3. Implementation of a real-time multimodal prototype • Browser-native and low-latency system 2

Background In Japan, one of the most common strategies for memorizing characters is to write them repeatedly. Problem: Tends to be monotonous and tedious, reducing motivation. 3

Enhance Motivation for Handwriting Practice Amplifying the sound of writing supports sustained engagement in handwriting tasks [Kim+ 2012] However, Existing solutions typically focus on individual pen strokes but often miss the "fun" factor Purpose: Encourage careful handwriting while maintaining engagement [Kim+ 2012] Junghyun Kim, Tomoko Hashida, Tomoko Ohtani, and Takeshi Naemura. 2012. Effects of auditory feedback for augmenting the act of writing. AH’12:Proceedings of the 3rd Augmented Human International Conference 13 (2012), 1–4. 4

Proposed Method Handwriting practice system that generates melodies in real time based on pen trajectories. Spatial Gate: Sound plays only while the stroke stays within the threshold of the model stroke. Major Pentatonic Scale: Pleasant melodies regardless of stroke direction. Character practice screen Melody design in the canvas 5

Experiment 1: No sound vs Sound • To examine whether presenting sound during handwriting practice is associated with higher enjoyment and increase practice attempts. • N=26 (between subjects) ーMelody condition (unrestricted) ーNo melody condition 6

Experiment 1: Procedure • 16 Sanskrit characters unfamiliar to Japanese participants • Participants practiced all 16 characters for three days, completing at least three repetitions of each character per day, with additional practice allowed. • (Minimum: 16 𝑐ℎ𝑎𝑟𝑎𝑐𝑡𝑒𝑟𝑠 × 3 𝑟𝑒𝑝𝑒𝑡𝑖𝑡𝑖𝑜𝑛𝑠 × 3 𝑑𝑎𝑦𝑠) • Collected a single-item enjoyment rating on a 5-point Likert scale 16 Sanskrit Characters 7

Experiment 1: Results Higher enjoyment ratings were associated with more practice attempts in the melody condition. Relationship between total practice attempts and enjoyment 8

Experiment 1: Discussion Melody increased enjoyment and voluntary practice. However, sound played anywhere on the canvas, risking participants to play with the audio instead of practicing handwriting. We introduced a restricted playback design that links sound continuity to accurate tracing. We verify this restricted playback design in Experiment 2. 9

Experiment 2: No vs Sound vs Proposed • Same 16 Sanskrit characters with Experiment 1 • N=36 (between subjects) • Comparative Methods ーBaseline group (no sound feedback) ーSound group (continuous sound feedback) ーRestricted sound group (proposed method) 10

Experiment 2: Procedure • Participants practiced 16 Sanskrit characters 5 times each, using the assigned feedback condition. • After a 5-minute break, they performed a test without auditory feedback or tracing models. Test page • Wrote all characters 3 times each 11

Experiment 2: Results (Practice phase) Average tracing speed (ms/px) Examined audio feedback influence in tracing speed Slower in the restricted sound group Baseline Sound Restricted Tracing speed per pixel (ms) for each group 12

Experiment 2: Results (Practice phase) Pixels Evaluated deviation from the model No difference Baseline Sound Restricted Mean Euclidean distance by group 13

Experiment 2: Results (Test phase) Writing speed (ms/px) Examined whether practice feedback influenced silent writing speed No significant differences Baseline Sound Restricted Writing speed in the test phase 14

Experiment 2: Discussion • Restricted sound group traced slower • Participants attempted to avoid breaking the melody Spatially-contingent auditory feedback encourages more deliberate handwriting • No difference in writing speed in the test phase • May have restored participants’ habitual writing pace 15

General Discussion Melody-based feedback can enhance engagement and encourage more attentive handwriting. However, further refinement is needed for sustained accuracy gains. 16

Future Work • Similarity analysis was not possible • The size of handwritten characters varied among participants • Implement smoother audio transitions to improve feedback continuity • Deploy the system in classroom environments for non-native learners of Japanese 17

Summary Background: Repetitive character practice is tedious Purpose: Encourage careful handwriting and maintain engagement Method: Handwriting practice system that generates melodies in real time based on pen trajectories Results: Tracing speed was slower in the restricted sound group Future Work: Use normalized spatial metrics 18