Imagine just telling your vehicle “I’m in a hurry” and it will automatically take you the most efficient route to where you need to go.
Engineers at Purdue University have found that autonomous driving cars (AVs) can do this with the help of ChatGPT or other chatbots that are implemented through artificial intelligence algorithms called the Big Language Model.
The research report was published on the preprint server arXiv and will be presented at the 27th IEEE International Conference on Intelligent Transportation Systems on September 25. This may be one of the first experiments to test how a true self-driving car can use a large language model to interpret passenger commands and drive accordingly.
Ziran Wang, assistant professor at Purdue University’s Lyles School of Civil and Architectural Engineering, who led the research, believes that if vehicles are to one day be fully autonomous, they need to understand all orders issued by passengers, even implicit ones. For example, when you say you are in a hurry, a taxi driver will know what you need without having to specify which route the driver should take to avoid traffic jams.
Although today’s autonomous vehicles have features that allow you to communicate with them, they require you to be clearer than when talking to a person. In contrast, large language models can interpret and respond in a more humane way because they are trained to extract relationships from large amounts of text data and learn over time.
“Traditional systems in our vehicles have a user interface design where you have to press a button to convey the information you want, or an audio recognition system that requires you to speak very clearly so that your vehicle can understand you,” Wang said. “But the power of large language models is that they can more naturally understand the variety of things you say. I don’t think any other existing system can do this.”
Cui Can, a doctoral student at Purdue University, sits in a test of a self-driving car. The microphone in the console receives his commands, and the large language model in the cloud interprets them. The vehicle travels according to instructions generated by the large-language model. Photo source: Purdue University/John Underwood
Conduct a new study
In this study, large language models did not drive autonomous vehicles. Instead, they use existing features of self-driving cars to assist in driving. Wang and his students found that by integrating these models, self-driving cars can not only better understand passengers, but can also personalize driving to satisfy passenger satisfaction.
Before starting the experiment, the researchers trained ChatGPT using a variety of prompts, ranging from more direct commands (e.g.,”Please drive faster”) to more indirect commands (e.g.,”I feel a little carsick right now”). When ChatGPT learned how to respond to these commands, researchers provided it with large language model parameters to follow, asking it to consider traffic rules, road conditions, weather, and other information and ranging detected by vehicle sensors such as cameras and light detection.
The researchers then provided these large language models via the cloud to experimental vehicles with four-level autonomy defined by SAE International. Level 4 is one level away from what the industry considers fully autonomous vehicles.
When a vehicle’s speech recognition system detects a command issued by a passenger during an experiment, a large language model in the cloud will infer the command based on parameters defined by the researchers. These models then generate instructions for the vehicle’s drive-by-wire system (connected to the throttle, brakes, gears and steering gear) on how to drive according to the command.
As study participants sat in the driver’s seat testing self-driving cars and spoke commands, a Purdue University researcher sat in the back and monitored feedback from large language models and vehicle cameras. Photos of the vehicle from back to front: Zhou Yupeng, a master’s student, and a doctoral student at Purdue University. Student Cui Can. Photo source: Purdue University/John Underwood
In some experiments, Wang’s team also tested a memory module they installed into the system that allows large language models to store data about passenger historical preferences and learn how to incorporate them into responses to commands.
Researchers conducted most of their experiments at a test site in Columbus, Ind., that used to be an airport runway. This environment allows them to safely test the vehicle’s response to passenger commands while traveling at highway speeds on the runway and handling two-way intersections. They also tested vehicles parked at passenger instructions in the parking lot of Purdue University’s Rose Aide Stadium.
Study participants used large language models of the commands they learned and new commands when riding in vehicles. Based on their post-ride survey feedback, participants expressed a lower rate of discomfort with the decisions they made in the self-driving car compared to data on how people felt when riding a fourth-level self-driving car without the help of a large language model.
The team also compared the performance of self-driving cars to baseline values created based on ride data that people on average consider safe and comfortable, such as how long and how fast the vehicle will respond to avoid rear-end collisions. Vehicles accelerate and decelerate. The researchers found that the self-driving cars in this study performed better than all baseline values when driving using large language models, even when responding to commands that the model had not yet learned.
The trunk of the self-driving car tested contains a drive-by-wire system that allows a large language model in the cloud to assist the vehicle in responding to passenger commands. Picture from left to right: Ph.D. from Purdue University. Student Yang Zichong and assistant professor Wang Ziran at Purdue University. Photo source: Purdue University/John Underwood
future direction
Wang said the large language models in this study took an average of 1.6 seconds to process passenger commands, which is considered acceptable in non-time critical scenarios, but should be improved if autonomous vehicles need to respond faster. This is a problem that generally affects large language models and is being addressed by industry and university researchers.
Although not the focus of this research, it is well known that large language models like ChatGPT are prone to “hallucinations,” which means they may misunderstand what they have learned and react in the wrong way. Wang’s research was conducted in a setup with a fail-safe mechanism that allows participants to ride safely when large language models misunderstand commands. Participants ‘understanding of these models improved throughout the ride, but illusions remain an issue that automakers must address before they consider implementing large language models in autonomous vehicles.
In addition to research conducted by university researchers, automakers need to conduct more testing using large language models. Wang said integrating these models with the controls of autonomous vehicles also requires regulatory approval so that they can actually drive the vehicle.
In the meantime, Wang and his students are continuing experiments, which may help the industry explore adding large language models to autonomous vehicles.
The test self-driving car was part of a demonstration in the parking lot of Purdue University’s Rose Aide Stadium. Photo source: Purdue University/John Underwood
Since their research tested ChatGPT, researchers have evaluated other public and private chatbots based on large language models, such as Google’s Gemini and Meta’s Llama AI assistant series. So far, they have seen ChatGPT perform best in terms of safe and efficient riding in autonomous vehicles. The announced results will be released soon.
The next step is to see if the large language models of each self-driving car can talk to each other, such as helping the self-driving car determine which should go first when parking in four directions. Wang’s laboratory is also launching a project to study how to use large visual models to help self-driving cars drive in extreme winter weather common in the Midwest. These models are similar to large language models, but are trained based on images rather than text.
There are links in the video below this video. If you are interested, you can open it and have a look.
Thank you for watching this video. If you like it, please subscribe and like it. thank
Original text:https://techxplore.com/news/2024-09-autonomous-vehicles-passengers-chatgpt.html
More information: Can Cui et al., Personalized autonomous driving using large language models: Field experiments, arXiv (2023). DOI：10.48550/arxiv.2312.09397
Journal information: arXiv
Oil tubing:
