FLARE: Fine-tuned large LAnguage models for Resource-Efficient action generation in robotics

Despite recent progress in robotic manipulation, robots still face difficulties generating actions across new tasks, objects, and environments. While foundation models such as Large Language Models (LLMs) show potential in robotic learning, they have several limitations with complex manipulation tasks. In addition, LLMs often depend on pre-trained actions or require reinforcement learning, and end-to-end robotic models demand vast amounts of data and computational power. Furthermore, building extensive multimodal datasets for real-world robotic applications is time-consuming, and training large foundation models is resource-intensive. This paper presents a framework that overcomes these challenges by employing an LLM model fine-tuned with a Parameter-Efficient Fine-Tuning (PEFT) technique to tailor them for robotic tasks. During the fine-tuning, our approach does not need real-world data because it is generated synthetically, without relying on images or multimodal inputs. This allows LLMs to directly produce generalized action plans in real-world settings, enabling the robot to perform seven tasks - including pick-and-place, stacking, lifting, and directional movements - after just a few hours of training on simulated data. By integrating a YOLO-based vision module for perception, our modular architecture achieves task success rates comparable to state-of-the-art robotic learning models on specific tasks. The primary advantages of our method are that it is trained entirely on synthetic data, provides exceptionally fast inference, and operates efficiently on a single commercial GPU for both training and inference. These features make this framework highly practical and accessible for industry use, offering a cost-effective solution in terms of time and resources.