An AI-Interactive Shopping Navigation Assistance System for Visually Impaired People via Passive Kinesthetic Guidance (RO-MAN 2026)

An AI-Interactive Shopping Navigation Assistance System for Visually Impaired People via Passive Kinesthetic Guidance Zhan Shi, Larissa R. de S. Shibata, Dayuan Chen, Zhenyu Liao, Yusuke Tamura, Yasuhisa Hirata Department of Robotics, Tohoku University, Japan [email protected] The 35th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 2026), Kitakyushu, Japan, August 26

Motivation 43.3 M 295 M people worldwide were blind by 2020 with moderate to severe vision impairment Projected to rise significantly [1] Shopping is a challenge Informational challenge Identify and locate the desired item Physical challenge Move safely through narrow, crowded aisles Human assistance solves both, but is costly and often unavailable. [1] R. Bourne et al., “Trends in prevalence of blindness and vision impairment over 30 years,” The Lancet Global Health, vol. 9, no. 2, pp. e130–e143, 2021. 1

Related Work Informational challenge Physical challenge Physical challenge Informational AI assistants Active guidance robots Passive brake guidance Deep-learning shopping assistants [2] AI-based shopping support [3] AI suitcase [4] CaBot [5] RT-Walker [6] Shopping guidance [7] (ours) Recognizes products Understands what the user is asking for Provides safe mobility support Simple and low power The user stays in control Cannot ensure safe mobility in narrow aisles Expensive and complex The robot leads, the user follows Guidance is not smooth enough when turning What is needed: human-like interaction, safe guidance, user control [2] D. Pintado et al., “Deep Learning-Based Shopping Assistant for the Visually Impaired,” IEEE International Conference on Consumer Electronics (ICCE), pp. 1–6, 2019. [3] L. R. d. S. Shibata et al., “An AI-Based Shopping Assistant System to Support the Visually Impaired,” 2025 SICE Festival with Annual Conference (SICE FES), pp. 215–221, 2025. [4] S. Kayukawa et al., “How Users, Facility Managers, and Bystanders Perceive and Accept a Navigation Robot for Visually Impaired People in Public Buildings,” IEEE RO-MAN, pp. 546–553, 2022. [5] J. Guerreiro et al., “CaBot: Designing and Evaluating an Autonomous Navigation Robot for Blind People,” ACM SIGACCESS Conference on Computers and Accessibility, pp. 68–82, 2019. [6] Y. Hirata, A. Hara, and K. Kosuge, “Motion Control of Passive Intelligent Walker Using Servo Brakes,” IEEE Transactions on Robotics, vol. 23, no. 5, pp. 981 –990, 2007. [7] Z. Shi et al., “Robotic Shopping Guidance System for Visually Impaired Users Using Servo Brakes,” IEEE AIM, pp. 1416–1421, 2024. 2

Proposal Our goal: help visually impaired users shop on their own, from asking for a product to reaching the shelf safely. To do this, we integrate natural-language target selection with passive braking guidance. human-like interaction safe guidance, user control Contributions I want to buy some chips. Intuitive target selection Enabled by an LLM-based natural-language interface Smooth paths Enabled by Kinematic-aware A* planning with turn penalty Safe and stable guidance Enabled by servo-brake control with convergence-aware damping Voice Interaction Target Shelf Selection Passive Braking Guidance 3

System Overview Voice Interaction Module User Voice Input ASR Voice Feedback LLM-based Target Inference Kinesthetic Feedback • • Voice command Pushes platform Path-Planning Module Shelf ID Target Mapping Kinematic-aware Planning Target Yaw Generation Navigation Actuation Embedded System Localization Module Target Yaw Pose / Heading Camera Input Brake Controller Marker Recognition Servo Brakes Encoder Fusion Master controller Laptop running the four software modules Sensor RealSense D456 RGB-D camera and two wheel encoders Actuator Two Arduino boards driving a pair of POB-0.3 servo brakes(Gear ratio modified to reach 12 N·m) 4

Hardware Platform[7] No active motors Stop: Both brakes applied at the same time Turn: Differential braking between the two wheels Voice Interaction Button Servo Brake & Encoder Stop System Master Controller Camera Turn Battery Embedded System &Brake Controller [7] Z. Shi et al., “Robotic Shopping Guidance System for Visually Impaired Users Using Servo Brakes,” IEEE AIM, pp. 1416–1421, 2024. 5

Voice Interaction The LLM understands free speech, but the planner needs one clear answer. Constrained prompting (based on GPT-4o-mini) • • • Tell the model whom it helps: a visually impaired shopper Show example products for each of the 11 shelves (snacks, beverages, …) Allow only one answer: a shelf ID, or −1 if unclear I want something salty to Press the snack on. physical button Idle Listening Shelf ID 3. ASR LLM Transcribing Inferring Path planner Responding If unresolvable, ask again 6

Localization & Mapping Marker Marker-corrected pose Markers on the shelves tell the cart where it really is Shelf Odometry fusion Wheels track motion all the time and markers correct the drift Goal node Topometric map The store becomes a grid, designed from the aisle width and the cart width Target mapping The cart starts from the nearest grid node Shelf ID becomes the goal node 7

Overshoot Problem What is overshoot? The cart turns past the desired heading and has to come back. Why does it matter here? Passive guidance: the user pays for it. Narrow aisles: there is no room for it. Blind users: they cannot anticipate it. Overshoot after turning We attack this in two places: the path and the brake control 8

Path Planning Preventing overshoot before it happens A* finds the path — but the cost function decides its shape. How to design the cost? Our cart is passive, so every turn is physical work for the user. We penalize turns quadratically to keep them gentle. 𝑓 𝑛 =𝑔 𝑛 +ℎ 𝑛 𝑔 𝑛 = path cost + 𝑊𝑡𝑢𝑟𝑛 𝛥𝜃 2 𝑊𝑡𝑢𝑟𝑛 = 20.0 𝑒𝑚𝑝𝑖𝑟𝑖𝑐𝑎𝑙 ℎ 𝑛 : Euclidean distance to the goal 𝛥𝜃: Turn angle between consecutive segments 9

Control Stopping the overshoot that remains Inertia and the user's delayed reaction make the cart keep turning after the heading is corrected. Our addition: convergence-aware damping on the outer wheel 𝜏𝑏𝑎𝑠𝑒 𝜃𝑒𝑟𝑟 = 𝑠𝑎𝑡 0,𝜏𝑚𝑎𝑥 𝑘𝜏 𝜃𝑒𝑟𝑟 − 𝜃0 𝜃𝑒𝑟𝑟 Within the dead-zone 𝜃𝑒𝑟𝑟 ≤ 𝜃0 , both brakes release and the user pushes freely. 𝜏𝑎𝑢𝑥 = 𝑚𝑖𝑛 𝛼 · 𝜏𝑏𝑎𝑠𝑒 , 𝜏𝑑 𝑖𝑓 𝜃𝑒𝑟𝑟 𝜃ሶ𝑒𝑟𝑟 < −𝜂 τbase Otherwise zero. 𝜂 prevents noise from triggering the damping near 𝜃𝑒𝑟𝑟 ≈ 0. Push Force τaux An example of when to turn left. 10

Experiments Experiment 1: Path-Planning To compare the proposed kinematic-aware planner with the shortest-path baseline. Experiment 2: Full-System Navigation To verify that the complete pipeline works from voice input to reaching the shelf. 11

Experiment 1: Path-Planning Baseline: shortest-path A* vs Proposed: Kinematic-aware A* (turn penalty) Experimental setup • • • 9 blindfolded participants, unknown start positions Each participant tries both planners Only the planner differs between the two trials Evaluation metrics Mock supermarket environment 2.8 m 1.1 m main aisle secondary aisle 14 11 ArUco markers shelves Task completion rate Completion time Path length Smoothness cost* 2 ሷ 𝐽𝑠 = ∫ 𝜃𝑒𝑟𝑟 𝑑𝑡 * Higher value = more abrupt braking 12

Baseline Video 13

Proposed Video 14

Path-planning Results Task completion rate: 100% (18 trials) Completion time: +15.4% 22.79 ± 6.24 s → 26.30 ± 7.00 s (p = 0.0273) Path length: +5.5% 6.37 ± 0.36 m → 6.72 ± 0.41 m (p = 0.0195) −82.6% Smoothness cost: 1.08×10⁵ → 1.88×10⁴ (W = 0, p = 0.0039, r = 1.0) Time Path length Smoothness We used the two-sided Wilcoxon signed-rank test, with the matched-pairs rank-biserial correlation as the effect size. 15

Experiment 2: Full-System Navigation Full pipeline: voice → localization → planning → braking Setup: 10 blindfolded participants × 5 trials 16

Full-system Results Task performance Subjective ratings Task initiation rate: 100% Goal reaching: 88% (44/50) 6 failures: 4 from rotating toward targets behind the initial heading; 2 from localization drift Trajectory RMSE: 0.097 ± 0.042 m Questionnaire results for the seven evaluation items (n=10) Everyone rated the guidance clear and safe, but human-likeness divided opinion. 17

Conclusion Passive braking can guide the user safely while keeping them in control. • Penalizing turns made the guidance much smoother, at a small cost in time and distance. • The system took users to the right shelf in most trials, and they found the direction clear and the movement safe. Next steps • • • • Forward turning + short backward routing Robust to wheel slip and bad initialization More natural dialogue Trials with visually impaired users in a real store 18