2026 ASEE Annual Conference & Exposition

The objective of this research project is to develop a controller for a low-cost robot arm using a multimodal Large Language Model (LLM) for educational purposes. LLMs have had a profound impact in recent years in the areas of search, chatbots, document production and summarization, problem solving, art, multimedia generation, image analysis, code generation and beyond. Typically, in these applications the input and output of the LLMs comprise text, images, voice or videos. More recently researchers are exploring the integration of hardware, such as robotics, into an LLM solution. The benefit of integrating an LLM is that the LLM can handle the user interaction through natural language understanding and multimodal input (voice, image, video) very effectively. Multimodal models also possess the ability to perform object detection and understand and communicate the spatial relationships of objects in an image. These models can also automatically generate code for a robot system. The challenge then is to integrate these capabilities of the multimodal LLM into a robotics system, such as a robot arm. We have demonstrated a method of interfacing an LLM with a low-cost robot arm with the purpose of educating undergraduate engineering majors (EE, ME, CMPEN, CMPSC) in the use of multimodal LLMs in engineering systems. It is expected that multimodal LLMs and generative AI will have a major impact on the way students learn engineering systems as well as how students design robotics engineering systems in the future. The goal is to develop an educational resource with a hands-on component that will guide undergraduate students into understanding the role of LLMs and computer vision and specifically, how to integrate a LLM into a working system to control a robot arm using human voice, text and image inputs. This experience will provide students with valuable knowledge in preparation for the workforce and research activities in a rapidly developing field.
In our approach, a user may request, in voice or text, information about objects on a flat surface (such as colored blocks on a table). The user may ask questions in natural language such as “How many red blocks are in the image?”, and request actions such as “Pick up the red block that is next to the large blue block in the image”, or “align the red blocks in a horizontal line”, etc. In effect, the LLM interprets the natural language nuances in the user commands based on spatial understanding of the objects in the image, then generates the individual steps to achieve the task, and finally generates and executes Python code that controls the robot arm to complete the task in the real world. We also include a discussion of the limitations of spatial reasoning capabilities of current multimodal LLMs and future directions. The technology described has been developed with undergraduate student participation and will undergo additional testing and evaluation with students in a robotics course.
It is hoped that this resource will provide educators with a tool that can be adapted and used to introduce and educate engineering on the role of LLMs with natural language and vision understanding along with code generation in a robotics system.