---
title: An AI-Interactive Shopping Navigation Assistance System for Visually Impaired People via Passive Kinesthetic Guidance (RO-MAN 2026)
tags:  #ショッピングカート  
author: [Tamura Lab.](https://image.docswell.com/user/tamlab)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/8EDKP43K7G.jpg?width=480
description: Zhan Shi, Larissa R. de S. Shibata, Dayuan Chen, Zhenyu Liao, Yusuke Tamura, Yasuhisa Hirata, "An AI-Interactive Shopping Navigation Assistance System for Visually Impaired People via Passive Kinesthetic Guidance," Proceedings of the 35th IEEE International Conference on Robot and Human Interactive Communication, pp.1532-1537, 2026.
published: August 26, 26
canonical: https://image.docswell.com/s/tamlab/ZL38XL-ROMAN2026_shi
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/8EDKP43K7G.jpg)

An AI-Interactive Shopping Navigation
Assistance System for Visually Impaired
People via Passive Kinesthetic Guidance
Zhan Shi, Larissa R. de S. Shibata, Dayuan Chen,
Zhenyu Liao, Yusuke Tamura, Yasuhisa Hirata
Department of Robotics, Tohoku University, Japan
z.shi@srd.mech.tohoku.ac.jp
The 35th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 2026), Kitakyushu, Japan, August 26


# Page. 2

![Page Image](https://bcdn.docswell.com/page/V7PKNM43J8.jpg)

Motivation
43.3 M
295 M
people worldwide were blind
by 2020
with moderate to severe vision
impairment
Projected to rise significantly [1]
Shopping is a challenge
Informational challenge
Identify and locate the desired item
Physical challenge
Move safely through narrow, crowded aisles
Human assistance solves both, but is costly and often unavailable.
[1] R. Bourne et al., “Trends in prevalence of blindness and vision impairment over 30 years,” The Lancet Global Health, vol. 9, no. 2, pp. e130–e143, 2021.
1


# Page. 3

![Page Image](https://bcdn.docswell.com/page/2JVVK9XNJQ.jpg)

Related Work
Informational challenge
Physical challenge
Physical challenge
Informational AI assistants
Active guidance robots
Passive brake guidance
Deep-learning shopping assistants [2]
AI-based shopping support [3]
AI suitcase [4]
CaBot [5]
RT-Walker [6]
Shopping guidance [7] (ours)
Recognizes products
Understands what the user is asking for
Provides safe mobility support
Simple and low power
The user stays in control
Cannot ensure safe mobility
in narrow aisles
Expensive and complex
The robot leads, the user follows
Guidance is not smooth
enough when turning
What is needed: human-like interaction, safe guidance, user control
[2] D. Pintado et al., “Deep Learning-Based Shopping Assistant for the Visually Impaired,” IEEE International Conference on Consumer Electronics (ICCE), pp. 1–6, 2019.
[3] L. R. d. S. Shibata et al., “An AI-Based Shopping Assistant System to Support the Visually Impaired,” 2025 SICE Festival with Annual Conference (SICE FES), pp. 215–221, 2025.
[4] S. Kayukawa et al., “How Users, Facility Managers, and Bystanders Perceive and Accept a Navigation Robot for Visually Impaired People in Public Buildings,” IEEE RO-MAN, pp. 546–553, 2022.
[5] J. Guerreiro et al., “CaBot: Designing and Evaluating an Autonomous Navigation Robot for Blind People,” ACM SIGACCESS Conference on Computers and Accessibility, pp. 68–82, 2019.
[6] Y. Hirata, A. Hara, and K. Kosuge, “Motion Control of Passive Intelligent Walker Using Servo Brakes,” IEEE Transactions on Robotics, vol. 23, no. 5, pp. 981 –990, 2007.
[7] Z. Shi et al., “Robotic Shopping Guidance System for Visually Impaired Users Using Servo Brakes,” IEEE AIM, pp. 1416–1421, 2024.
2


# Page. 4

![Page Image](https://bcdn.docswell.com/page/5EGLQZV5JL.jpg)

Proposal
Our goal: help visually impaired users shop on their own, from
asking for a product to reaching the shelf safely.
To do this, we integrate natural-language target selection with passive braking guidance.
human-like interaction
safe guidance, user control
Contributions
I want to
buy some
chips.
Intuitive target selection
Enabled by an LLM-based natural-language interface
Smooth paths
Enabled by Kinematic-aware A* planning with turn penalty
Safe and stable guidance
Enabled by servo-brake control with convergence-aware damping
Voice
Interaction
Target Shelf
Selection
Passive Braking
Guidance
3


# Page. 5

![Page Image](https://bcdn.docswell.com/page/4JQYKL6L7P.jpg)

System Overview
Voice Interaction Module
User
Voice Input
ASR
Voice
Feedback
LLM-based Target Inference
Kinesthetic
Feedback
•
•
Voice command
Pushes platform
Path-Planning Module
Shelf
ID
Target Mapping
Kinematic-aware Planning
Target Yaw Generation
Navigation Actuation
Embedded System
Localization Module
Target Yaw
Pose /
Heading
Camera Input
Brake Controller
Marker Recognition
Servo Brakes
Encoder Fusion
Master controller
Laptop running the four software modules
Sensor
RealSense D456 RGB-D camera and two wheel encoders
Actuator
Two Arduino boards driving a pair of POB-0.3 servo brakes(Gear ratio modified to reach 12 N·m)
4


# Page. 6

![Page Image](https://bcdn.docswell.com/page/K74WPD45E1.jpg)

Hardware Platform[7]
No active motors
Stop: Both brakes applied at the same time
Turn: Differential braking between the two wheels
Voice
Interaction
Button
Servo Brake
&
Encoder
Stop
System Master
Controller
Camera
Turn
Battery
Embedded System
&Brake Controller
[7] Z. Shi et al., “Robotic Shopping Guidance System for Visually Impaired Users Using Servo Brakes,” IEEE AIM, pp. 1416–1421, 2024.
5


# Page. 7

![Page Image](https://bcdn.docswell.com/page/LJ1Y9Z42EG.jpg)

Voice Interaction
The LLM understands free speech, but the planner needs one clear answer.
Constrained prompting (based on GPT-4o-mini)
•
•
•
Tell the model whom it helps: a visually impaired shopper
Show example products for each of the 11 shelves (snacks, beverages, …)
Allow only one answer: a shelf ID, or −1 if unclear
I want
something
salty to
Press the
snack on. physical button
Idle
Listening
Shelf
ID 3.
ASR
LLM
Transcribing
Inferring
Path
planner
Responding
If unresolvable,
ask again
6


# Page. 8

![Page Image](https://bcdn.docswell.com/page/GJWGQ9X272.jpg)

Localization & Mapping
Marker
Marker-corrected pose
Markers on the shelves tell the cart where it really is
Shelf
Odometry fusion
Wheels track motion all the time
and markers correct the drift
Goal node
Topometric map
The store becomes a grid,
designed from the aisle width and the cart width
Target mapping
The cart starts from the nearest grid node
Shelf ID becomes the goal node
7


# Page. 9

![Page Image](https://bcdn.docswell.com/page/4EZLW96473.jpg)

Overshoot Problem
What is overshoot?
The cart turns past the desired heading and has to come back.
Why does it matter here?
Passive guidance: the user pays for it.
Narrow aisles: there is no room for it.
Blind users: they cannot anticipate it.
Overshoot after turning
We attack this in two places: the path and the brake control
8


# Page. 10

![Page Image](https://bcdn.docswell.com/page/Y76WGK2G7V.jpg)

Path Planning
Preventing overshoot before it happens
A* finds the path — but the cost function decides its shape.
How to design the cost?
Our cart is passive, so every turn is physical work for the user.
We penalize turns quadratically to keep them gentle.
𝑓 𝑛 =𝑔 𝑛 +ℎ 𝑛
𝑔 𝑛 = path cost + 𝑊𝑡𝑢𝑟𝑛 𝛥𝜃 2
𝑊𝑡𝑢𝑟𝑛 = 20.0 𝑒𝑚𝑝𝑖𝑟𝑖𝑐𝑎𝑙
ℎ 𝑛 : Euclidean distance to the goal
𝛥𝜃: Turn angle between consecutive segments
9


# Page. 11

![Page Image](https://bcdn.docswell.com/page/G75MVP2X74.jpg)

Control
Stopping the overshoot that remains
Inertia and the user's delayed reaction make the cart keep turning after the heading is
corrected.
Our addition: convergence-aware damping on the outer wheel
𝜏𝑏𝑎𝑠𝑒 𝜃𝑒𝑟𝑟
= 𝑠𝑎𝑡 0,𝜏𝑚𝑎𝑥 𝑘𝜏 𝜃𝑒𝑟𝑟 − 𝜃0
𝜃𝑒𝑟𝑟
Within the dead-zone 𝜃𝑒𝑟𝑟 ≤ 𝜃0 , both brakes release and the user pushes freely.
𝜏𝑎𝑢𝑥 = 𝑚𝑖𝑛 𝛼 · 𝜏𝑏𝑎𝑠𝑒 , 𝜏𝑑
𝑖𝑓 𝜃𝑒𝑟𝑟 𝜃ሶ𝑒𝑟𝑟 < −𝜂
τbase
Otherwise zero. 𝜂 prevents noise from triggering the damping near 𝜃𝑒𝑟𝑟 ≈ 0.
Push
Force
τaux
An example of when to turn left.
10


# Page. 12

![Page Image](https://bcdn.docswell.com/page/9J29364QER.jpg)

Experiments
Experiment 1: Path-Planning
To compare the proposed kinematic-aware planner with the shortest-path baseline.
Experiment 2: Full-System Navigation
To verify that the complete pipeline works from voice input to reaching the shelf.
11


# Page. 13

![Page Image](https://bcdn.docswell.com/page/DEY419MYJM.jpg)

Experiment 1: Path-Planning
Baseline: shortest-path A* vs Proposed: Kinematic-aware A* (turn penalty)
Experimental setup
•
•
•
9 blindfolded participants, unknown start positions
Each participant tries both planners
Only the planner differs between the two trials
Evaluation metrics
Mock supermarket environment
2.8 m
1.1 m
main aisle
secondary aisle
14
11
ArUco markers
shelves
Task completion rate
Completion time
Path length
Smoothness cost*
2
ሷ
𝐽𝑠 = ∫ 𝜃𝑒𝑟𝑟
𝑑𝑡
* Higher value = more abrupt braking
12


# Page. 14

![Page Image](https://bcdn.docswell.com/page/VJNY2LWR78.jpg)

Baseline Video
13


# Page. 15

![Page Image](https://bcdn.docswell.com/page/YE9P14XZJ3.jpg)

Proposed Video
14


# Page. 16

![Page Image](https://bcdn.docswell.com/page/GE8D4Q9YED.jpg)

Path-planning Results
Task completion rate: 100% (18 trials)
Completion time: +15.4%
22.79 ± 6.24 s → 26.30 ± 7.00 s (p = 0.0273)
Path length: +5.5%
6.37 ± 0.36 m → 6.72 ± 0.41 m (p = 0.0195)
−82.6%
Smoothness cost:
1.08×10⁵ → 1.88×10⁴ (W = 0, p = 0.0039, r = 1.0)
Time
Path length
Smoothness
We used the two-sided Wilcoxon signed-rank test, with the matched-pairs rank-biserial correlation as the effect size.
15


# Page. 17

![Page Image](https://bcdn.docswell.com/page/LELM6XW97R.jpg)

Experiment 2: Full-System Navigation
Full pipeline: voice → localization → planning → braking
Setup: 10 blindfolded participants × 5 trials
16


# Page. 18

![Page Image](https://bcdn.docswell.com/page/4JMY5L9VJW.jpg)

Full-system Results
Task performance
Subjective ratings
Task initiation rate: 100%
Goal reaching: 88% (44/50)
6 failures: 4 from rotating toward targets
behind the initial heading; 2 from localization drift
Trajectory RMSE: 0.097 ± 0.042 m
Questionnaire results for the seven evaluation items (n=10)
Everyone rated the guidance clear and safe, but
human-likeness divided opinion.
17


# Page. 19

![Page Image](https://bcdn.docswell.com/page/PJR93KGW79.jpg)

Conclusion
Passive braking can guide the user safely while keeping them in
control.
• Penalizing turns made the guidance much smoother, at a small cost in time and distance.
• The system took users to the right shelf in most trials, and they found the direction clear
and the movement safe.
Next steps
•
•
•
•
Forward turning + short backward routing
Robust to wheel slip and bad initialization
More natural dialogue
Trials with visually impaired users in a real store
18