ABSTRACT:

Mechanically advanced bionic hands continue to underperform in everyday contexts because surface electromyography (muscle electrical activity, EMG)–based control obliges amputee users to generate each discrete movement command separately, leading to slow task execution and increased cognitive load. Therefore, the lack of intuitive, non invasive interfaces that enable more natural control remains a leading cause of device abandonment. Vision guided assistance, where a camera is added to a prosthesis so that it can “see” the object to be grasped and adjust automatically, offers a promising alternative. However, current camera based prosthesis systems underperform in realistic settings due to narrowly trained models that struggle with unpredictable scene variations. Recent progress in Vision–Language Models (VLMs) provides a scalable foundation for scene understanding, affordance extraction, and object‑level reasoning from a hand‑mounted camera, overcoming the limitations of narrow, task‑specific vision models. This project develops a closed‑loop cognitive control architecture that integrates VLM‑based perception with a lightweight Large Language Model (LLM) based reasoning module to support grasp selection, preshaping, and real‑time adjustment, complemented by a shared‑control interface that blends autonomous assistance with user‑initiated EMG corrections. The research advances three components: (1) a real‑time VLM perception module for extracting object identity, coarse pose, and grasp‑relevant affordances from continuous egocentric video; (2) a constrained LLM reasoning module that interprets perceptual cues and user initiated EMG input to generate structured, safe prosthetic commands; and (3) a shared‑control framework that enables the prosthesis to propose actions while allowing the user to refine or override them, reducing cognitive effort without compromising agency. The integrated system will be validated on a physical prosthetic hand that the users will employ to perform everyday manipulation tasks under realistic conditions such as clutter, occlusion, object motion, and shifting user goals. By unifying multimodal perception, reasoning, and shared autonomy in a single control architecture, the work establishes VLM‑driven semi‑autonomous prosthetic control as a new direction that enhances prosthesis capability, enables context‑aware manipulation, reduces user effort, and ultimately improves independence and quality of life for upper‑limb amputees.