ozchi2023_yamazaki_slide

The Effects of Order and Text Box Size of Open-ended Questions on Withdrawal Rate and Length of Response Ikumi Yamazaki Kenichi Hatanaka, Satoshi Nakamura, Takanori Komatsu （Meiji University）

Our previous work[1] Open-ended question Close-ended question ・・・ More respondents left the survey when the open-ended question was placed first. [1] Yamazaki, I., Hatanaka, K., Nakamura, S., and Komatsu, T.(2023): “A Basic Study to Prevent Non-Earnest Responses in Web Surveys by Arranging the Order of Open-ended Questions.” International Conference on Human-Computer Interaction (HCII 2023), vol. LNCS, volume. 14011, 314-326. 2

Background: Text box size in the survey Small text box size Large text box size 3

Order of questions × Text box size By combining the position of the open-ended question and the text box size, controlling the withdrawal rate and length of responses. ・・・ ✖ ・・・ 4

Hypothesize Presenting open-ended questions with a large text box first causes high withdrawal. ・・・ ✖ 5

FIRST (Small) FIRST (Large) LAST (Small) LAST (Large) 6

Results: the withdrawal rate Withdrawal rate（%） 70 60 50 40 • Group-FIRST (Large) is the highest withdrawal rate. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 12 13 14 • 30~45% respondents in Group-FIRST (Small) and Group-FIRST(Large) left before answering a question. 7

Results: the withdrawal rate Withdrawal rate（%） 70 60 Group-FIRST (Large) ↓ 50 40 • Group-FIRST (Large) is the highest withdrawal rate. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 12 13 14 • 30~45% respondents in Group-FIRST (Small) and Group-FIRST(Large) left before answering a question. 8

Results: the withdrawal rate Withdrawal rate（%） 70 60 50 40 • Group-FIRST (Large) is the highest withdrawal rate. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 12 13 14 • 30~45% respondents in Group-FIRST (Small) and Group-FIRST(Large) left before answering a question. 9

Results: the withdrawal rate Withdrawal rate（%） 70 60 50 40 About 20% of the respondents left before answering a question. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 10

Results: the length of response Total Group-FIRST (Small) 35.6 Group-FIRST (Large) 51.4 Group-LAST (Small) 36.5 Group-LAST (Large) 49.8 • Group-FIRST (Large) and Group-LAST (Large) have more letters. • We tested a two-factor analysis of variance for each question and found a significant difference in text box size for all questions. 11

Results: the length of response Total Group-FIRST (Small) 35.6 Group-FIRST (Large) 51.4 Group-LAST (Small) 36.5 Group-LAST (Large) 49.8 1.5 times different • Group-FIRST (Large) and Group-LAST (Large) have more letters. • We tested a two-factor analysis of variance for each question and found a significant difference in text box size for all questions. 12

Results: the length of response Total Group-FIRST (Small) 35.6 Group-FIRST (Large) 51.4 Group-LAST (Small) 36.5 Group-LAST (Large) 49.8 • Group-FIRST (Large) and Group-LAST (Large) have more letters. • We tested a two-factor analysis of variance for each question and found a significant difference in text box size for all questions. 13

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 14

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 15

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 16

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 17

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 18

Results: device comparison The withdrawal rate 70 • It was higher for smartphones than PCs. Withdrawal rate (%) 60 50 40 • About 60% in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey. 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • A certain number of PC respondents also left at the stage of the open-ended question. Question number Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） Group-FIRST（Small） Group-FIRST（Large） Group-LAST（Small） Group-LAST（Large） 19

Discussion • The Group-FIRST (Large) had the highest withdrawal rate and Group-FIRST (Small) was the second. → The order of the open-ended questions may have affected the withdrawal. • The length of responses was longer in large text box. → The respondents try to input many characters. • The withdrawal rate for smartphones was higher than that for PCs. → Possible influence of external factors such as meeting up and multitasking such as mailing. 20

Conclusion Background: Open-ended questions are important in web surveys. Purpose: Examining the effects of the order and size of the questions on the withdrawal rate and differences between smartphones and PCs. Hypothesis: People would most leave in large text box at first of open-ended questions. Results: The withdrawal rate ・was highest in large text box at first. ・for smartphones was higher than for PCs. 21