Robotic Shopping Guidance System for the Visually Impaired Users using Servo Brakes (AIM2024)

Robotic Shopping Guidance System for the Visually Impaired Users using Servo Brakes Zhan SHI, Yusuke TAMURA, Zhenyu LIAO, Weizan HE, Yasuhisa HIRATA Dept. of Robotics, Tohoku University, Japan [email protected] [email protected] 2024 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2024), Boston, USA | July 18

Motivation 285 million people are estimated to be living with visual impairments. (39 million: blind, 246 million: low vision) Effects on QoL: - Reduced independence in daily activities - Challenges in navigation and mobility - Difficulties in accessing information Daily living challenges: personal care, household management, social interactions, shopping, … 2

Challenges in Supermarket Shopping Locating Items - Difficulty in finding specific products within large store layouts Navigation Barriers - Avoiding obstacles and safely navigating crowded aisles Product Identification - Challenges in recognizing and distinguishing between similar products - Reading price tags and ingredient labels is often impossible without assistance 3

Related Studies Voice navigation NavCog [Ahmetovic+ 2016] NaviLens Cons Pros - Detailed instructions - Environmental noise - Privacy - Directional cues - Cognitive load - Cannot ensure the safety Autonomous navigation Co-robotic cane [Ye+ 2016] AI suitcase (CaBot) [Guerreiro+ 2019] Pros - Safer path to avoid obstacles Cons - Risk of unexpected behavior - Lack of “sense of control” 4

Objective Development of a robotic assistant system that safely guides visually impaired individuals in supermarkets - Provide appropriate feedback to users - Ensure user safety - Maintain user’s sense of control - Seamlessly integrate into the shopping environment 5

Design Strategies Provide appropriate feedback to users Tactile/Force feedback are useful for intuitive understanding of surroundings. Auditory feedback is useful for detailed instructions. Ensure user safety It is necessary to recognize the surrounding environment and avoid collisions. Maintain user’s sense of control It is desirable for users to move independently. Seamlessly integrate into the shopping environment The appearance of the robotic system should be suitable for use in a supermarket. Shopping cart-type passive robot with servo brakes 6

Shopping Cart-type Passive Robot RGBD camera & IMU RGB-D camera Controller Speaker Speaker Left brake Right brake Servo brakes Controllers Battery 7

Brake Control Stop: Activate both brakes at the same time Turn Left: Activate the left side brake Turn Right: Activate the right side brake 8

Navigation Global path planning - Represent the environment as a graph with nodes - Use a graph-based algorithm to calculate the 15 12 9 6 3 0 16 13 10 7 4 1 17 14 11 8 5 2 optimal path At a node: The brakes are controlled to face the direction of the next node using IMU based on the relative position between the current node and next node. Between nodes: When a marker is detected The brakes are adjusted to move towards the marker. 9

Marker-based Navigation Torque (Nm) Turn Left Turn Right 12 8 4 0 -35 -30 -5 0 5 30 35 Target direction θ (deg) ArUco Marker ✓ <latexit sha1_base64="i88lUfDMtsFoMgl+H48iZ0wr3Ao=">AAACaXichVG7SgNBFD1Z3/GRqI1oE1wiVmEivrASbSx95QFJCLvraEY3u8vuJKDBH7CyE7VSEBE/w8YfsPATxDKCjYV3NwuiQb3DzJw5c8+dMzO6YwpPMvYcUTo6u7p7evui/QODQ7H48EjWs2uuwTOGbdpuXtc8bgqLZ6SQJs87Ltequslz+sGqv5+rc9cTtrUtDx1eqmp7ltgVhiaJyhZlhUutHFdZigWRaAfpEKgIY92O36KIHdgwUEMVHBYkYRMaPGoFpMHgEFdCgziXkAj2OY4RJW2NsjhlaMQe0LhHq0LIWrT2a3qB2qBTTOouKRNIsid2x5rskd2zF/bxa61GUMP3ckiz3tJypxw7Gdt6/1dVpVmi8qX607PELhYDr4K8OwHj38Jo6etHZ82tpc1kY4pds1fyf8We2QPdwKq/GTcbfPMSUfqA9M/nbgfZmVR6PjW3Masur4Rf0YsJTGKa3nsBy1jDOjJ07j5OcY6LyKsyrIwp461UJRJqRvEtFPUTrt2MLg==</latexit> 10

Experiments Environment: Simplified environment simulating supermarket’s unique right-angle turning paths ID: 0 ID: 1 ID: 3 Participants: 6 healthy young participants with blindfolds Procedure: Participants complete navigation tasks on two routes ID: 2 ID: 4 11

ID: 3 ID: 0 ID: 4 Path: 0 ▶︎ 3 ▶︎ 4 Auditory feedback: 12

Experimental Results Task completion rate: 12 times out of 12 (100%), participants successfully reached their destination. Task completion time: Proposed method (1st trial) Proposed method (2nd trial) Baseline (with caregiver) 0 5 10 15 20 25 30 35 Completion time (s) Subjective evaluation by participants (scale 1-10): Perceived Safety: 8.2 Ease of Use: 8.0 Movement Smoothness: 9.5 (linear motion), 7.5 (rotation) 13

Conclusions We developed a robotic assistant system that safely guides visually impaired individuals in supermarkets. Shopping cart-type passive robot with servo brakes Future works - Navigation in crowded environment - Integration with product identification - Field test Acknowledgment This work was partially supported by JST COI-NEXT Grant Number JPMJPF2201. 14