ROSA: Robot Operating System Agent

In recent years, the field of robotics has witnessed significant advancements, with robots playing crucial roles in industries such as manufacturing, healthcare, and space exploration. As robotic systems become more complex and capable, the need for intuitive and efficient human-robot interaction has grown exponentially. However, traditional methods of controlling and programming robots often require specialized knowledge and technical expertise, creating barriers for operators and limiting the accessibility of robotic technologies. It is not uncommon for operators to require advanced degrees and meticulous training to effectively operate even modestly capable robotic systems. ROSA, the `"Robot Operating System Agent," addresses these challenges by introducing a first of its kind AI Agent that enables human-robot interaction (HRI) in the form of natural language. ROSA leverages large language models (LLMs) and open-source software like the Robot Operating System (ROS) and LangChain to implement a Reasoning and Acting (ReAct) agent capable of understanding and executing commands on its robotic host. By integrating with the ROS and ROS2 ecosystems, ROSA provides easy access to a wide range of tools and functionalities that allow users to perform tasks such as system diagnostics, monitoring, and invoking existing navigation and manipulation tasks, without the need for extensive technical training. ROSA aims to make robotic systems more accessible and user-friendly, empowering a broader range of operators to interact with robots in a more intuitive manner. ROSA is designed to benefit a wide range of users within the robotics community. Robotics researchers and developers can use it to streamline development and verification & validation (V&V). Field operators and technicians, tasked with operating robots in dynamic environments, can now perform routine operations without requiring specialized training and a deep understanding of ROS. Further, educators and students will find ROSA enhances their learning experience by allowing them to focus on fundamental robotics concepts, free from the complexities of command-line interfaces. Finally, hobbyists and makers, who may not possess extensive technical backgrounds, can engage with robotics in a more accessible manner, fostering greater experimentation and innovation. Further, ROSA offers substantial advantages in a variety of practical scenarios. In disaster response, it enables rapid deployment and control of robots through natural language commands, making it easier to manage operations in emergency situations. In space exploration, ROSA can assist astronauts in controlling robotic arms and assistive robots, where traditional interfaces may be impractical due to constraints of the working environment. In industrial automation, it can simplify the monitoring and control of robotic systems on manufacturing floors, empowering personnel without deep technical expertise to operate complex machinery. Finally, in education and training, ROSA can serve as a versatile platform for teaching robotics concepts, lowering barriers for academic institutions and learners alike. This presentation will highlight ROSA's capabilities using real-world robotic systems deployed at NASA Jet Propulsion Laboratory. We will illustrate how various open-source technologies coalesce to enable seamless and natural human-robot interaction. We will also discuss the use of "open-weight" LLMs for on-board inference, as well as some of the issues that come along with it. Additionally, the audience will see ROSA in action at the JPL Mars Yard on a NeBula-Spot robot. Finally, we will discuss some of the ethical implications surrounding the use of AI in robotics, and discuss some of the upcoming challenges we may face as AI systems become more prevalent in our daily lives.