An Interactive Autonomous Forklift Robot Based on Large Language Models

Abstract

This paper introduces Hercules, an autonomous robot designed to simulate a specialized cellulose bale forklift, integrating cutting-edge technologies in natural language processing (NLP) and speech synthesis. Equipped with advanced sensors and communication capabilities — such as 2D cameras, LiDAR, and a microphone array — Hercules interacts with its environment and operators using its sensors, actuators, and natural language supported by text-to-speech and speech-to-text functionalities. The robot's architecture enables real-time interaction with non-expert users, allowing it to autonomously navigate, detect objects, and perform precise grasping tasks. This work outlines the development and integration of state-of-the-art large language models into a robotic system to enable two-way communication and execution of commands, demonstrating the robot's efficiency, accuracy, and versatility in a simulated warehouse environment. Our experiments and evaluations exhibit Hercules' capabilities to understand, respond to, and execute natural language commands effectively, presenting promising prospects for enhanced human-robot interaction in practical industrial settings.