CVP Leverages Machine Learning, Experiments with Self-Driving Car
According to a study by the World Health Organization (WHO), over 1 million people die annually due to motor vehicle accidents, with almost 40,000 of those being in the United States. Up to 90% of these car accidents are due to human error. A form of machine learning called reinforcement learning (RL) promises to greatly reduce driving errors and potentially save a lot of lives. CVP is using reinforcement learning in a small-scale car and hopes to eventually use the knowledge gained to decrease deaths in regular passenger vehicles.
Amazon Web Services (AWS) publicly released their DeepRacer vehicle this summer. It is a 1/18 scale vehicle with video cameras and a computer-powered on-board system sufficient enough to drive itself. As a long-time partner of AWS, CVP received this vehicle in late 2018 and the recent public release of the platform allowed us to easily build a track for it in our Fairfax, VA, headquarters. To make the car truly self-driving without having to program thousands of lines of code, we started off by creating our own “reward function” that gives the artificial intelligence model broad parameters for how we want it to conduct itself on the track and not run off the “road.” Using several virtual servers, we then tried out various test drives in a virtual track simulator repeatedly until the environment arrived at a trained model that best satisfied the reward function and that also did a good job of driving around an oval track. After a review of the results, we downloaded the model from the cloud, put it onto a thumb drive, and loaded it into our autonomous vehicle.
Our next step was to make a track for the car to navigate. After a lot of duct tape and temporary foam flooring, we had a real-world (not virtual) version of a track that the real-world (albeit small) car could drive around with no human intervention. While passengers can’t ride in the car just yet, the technology we developed is similar to what is used in other autonomous vehicles, including:
- Teslas: except they have more cameras and higher-powered computers
- Open-source driving systems: from companies like comma.ai, (https://comma.ai/)
This concept of reinforcement learning is actually very generalizable beyond self-driving cars, and CVP is using it on a variety of projects, including competing in and placing on the leaderboard for the Office of the National Coordinator for Health Information Technology’s patient matching challenge in 2017. We trained a reinforcement learning model to know what a “good” match (versus a “bad” match) is between the two systems and it extrapolated from this game of twenty questions to a model that could accurately match hundreds of records per second. We are currently exploring the use of reinforcement learning for other similar data matching problems. Stay tuned for future blog posts highlighting those as well!