AMD demonstrated a system that allows robotic arm manipulation using natural language commands through a local AI agent system built with the Lemonade framework. The demonstration runs entirely locally, showcasing the effectiveness of Lemonade for Physical AI. The setup combines RAI for robot control, O3DE for simulation, ROS 2 Jazzy for the robotics interface, and the Lemonade SDK for local AI model deployment on the GPU. This allows tasks such as picking, placing, and sorting objects to be executed in real time.
The system uses a general-purpose Vision-Language Model (VLM) as a reasoning agent, which receives input from the camera feed, reasons over available tools, and delegates motion planning to MoveIt. This approach preserves the reasoning capabilities of language models while maintaining functionality, without requiring task-specific policies. The trade-offs include per-step inference latency and a requirement for vision-capable models, but the system avoids the need for data collection, fine-tuning, or training infrastructure. Any model compatible with Lemonade can be used immediately, with decisions interpretable as explicit tool calls.
The integration of RAI with Lemonade enables local AI model deployment on AMD Ryzen AI platforms, leveraging the Radeon iGPU through ROCm for fast, private inference. The system ensures that sensor data remains on-device, preserving privacy and enabling real-time responsiveness even offline. The software stack includes O3DE for simulation, ROS 2 Jazzy for robotics interface, RAI for routing agent calls, and the Lemonade SDK for local model serving. The pipeline operates as a closed perception–decision–action loop, with decisions originating from the Lemon, where reasoning is done on-device.
Source: amd
