2026 ASEE Annual Conference & Exposition

Robotic arm manipulators have traditionally been controlled using heavy and complicated teach pendants, scripted sequences, or pre-programmed routines. While these methods have been industry standards, they may often fail to provide intuitive or natural control mechanisms, despite the objective of replicating the dexterity and fluidity of the human arm. This work explored a novel control methodology that integrated computer vision-based gesture recognition with wearable inertial measurement units (IMUs) to enable a more natural, human-like motion of the robotic arm. By leveraging advances in machine learning and sensor fusion, the system translated user gestures and arm movements into precise commands for the manipulator.

This approach enhanced the dexterity of the arm, simplified the programming of repetitive tasks, and facilitated rapid adaptation to sudden environmental changes. The integration of real-time computer vision for dynamic adjustments, along with wearable sensors that track precise motion changes, ensured seamless interaction between the user and the robotic system. The results demonstrated an approach that augmented robotic manipulation by making it more adaptable, user-friendly, cost-effective, and efficient across various fields.

To evaluate the effectiveness of the work, a combination of quantitative and qualitative assessment methods were employed. Gesture recognition performance was tested by analyzing live data readouts and observing the placement of landmarks and links overlaid onto the camera feed. Hand motion demonstrations were conducted under varying lighting conditions and backgrounds to ensure robustness and reliability. The vision-based gesture recognition system achieved a high accuracy of over 95% in real-time classification and positioning of discrete hand gestures. IMU tracking accuracy was evaluated by comparing users’ intended motions with the system's corresponding outputs across each axis of rotation. When fused with visual input, IMU data contributed to smoother and more precise control, reducing latency to less than 600 milliseconds and minimizing errors in robotic arm positioning. User feedback indicated a significantly lower cognitive load and a more natural, engaging interaction experience. System responsiveness and task performance were measured using a standardized set of robotic manipulation tasks, including pick-and-place operations and trajectory following. The system remained responsive and adaptable under dynamic conditions, including sudden changes in background, lighting, or user position, thus demonstrating robustness in real-world scenarios. Beyond technical metrics, user feedback was assessed to evaluate usability, intuitiveness, and the overall learning curve.

The full paper will provide a comprehensive overview of the system’s design and implementation. It will cover key technical components, including the computer vision-based gesture recognition algorithm, the integration of wearable IMUs, the development of a custom data fusion algorithm for motion tracking, and the communication protocol linking the sensing units to the robotic arm. The paper will include the testing protocols, different use case scenarios, and reflections on the system's design process. It will conclude with a discussion of the broader impact of the project in both research and educational contexts. For example, several potential applications of this approach include kinesthetic programming and playback, safer and more accessible remote handling of hazardous materials, and assistance for individuals with arm weaknesses or similar impairments.

In addition to the technical details, the paper will emphasize the project’s interdisciplinary educational value. Developed as part of a research and design-intensive undergraduate engineering capstone, students were required to apply interdisciplinary concepts from embedded systems, machine learning, control systems, robotics, and human-robot interaction into real-world scenarios. Through iterative prototyping, testing, documentation, and presentations, students gained hands-on experience in the research process while contributing to a broader technological solution. Conclusively, the paper will showcase how undergraduate research not only contributed to the development of a functional prototype but also served as a meaningful platform for experiential learning.