If you check the weather for San Francisco, there's a one third chance you’ll see clouds or fog in the forecast, according to the National Oceanic and Atmospheric Administration (NOAA). To ensure Waymo’s Trusted Testers can safely get where they are going, whether it’s to work in the misty morning or back home to a hazy Haight at night, we’re engineering the Waymo Driver to handle the challenges of fog.

Imagine you are driving through downtown San Francisco and a cyclist, traveling against the flow of traffic, cuts right in front of you. Or maybe you are navigating a narrow, two-way street with a car heading the opposite direction, and you nudge over for them to pass. Perhaps, you are driving late at night when an occluded worker pops out behind a truck, right into the middle of the street. These are just a few examples of the common yet complex scenarios the Waymo Driver encounters in cities like San Francisco.

Here’s our autonomous driving system—the Waymo Driver—in San Francisco earlier this year. It’s the kind of journey we’ve made tens of thousands of times since we first started driving autonomously in the city in 2009. As the Waymo Driver navigates dozens of vehicles and pedestrians, it’s met with a huge variety of other road users—from double-parked vehicles whose riders can hop out at any second, to scooters cutting across traffic even when they have a red light.

At Waymo, we use simulation to advance the Waymo Driver's abilities in several ways. We can amplify our number of real-world miles to accelerate our Driver’s learning, prepare for rare events, train new models, validate new software, and even evaluate how the Waymo Driver would have performed in actual fatal crashes that have already occurred. For simulation to be valuable as a learning tool, it has to closely match our target domain – performing rider-only trips for Waymo One and goods delivery trips for Waymo Via in the real world– in which case, there is only one question that matters. Did the Waymo Driver safely and efficiently complete the trip?

Waymo’s story starts in the San Francisco Bay Area. Since early 2009, when we completed our first 1,000 autonomous miles across California, we’ve driven the length and breadth of the region, becoming intimately familiar with the many unique challenges of driving in San Francisco and the surrounding area.

On October 8th, Waymo opened its fully autonomous ride-hailing service to the general public in Phoenix. Right now members of the public are hailing vehicles with no human driver controlling the car – either in the vehicle or remotely – to help them get to where they’re going as part of their everyday lives.

For any emerging technology to be trusted, it helps to first be understood. In the past, people could see how their cars worked, looking under the hood and tinkering with them with the help of a user manual. In 2020, vehicles have so much technology that they’ve become difficult for the general public to comprehend. We want to change that. With this blog series, we’ll unpack the different parts of our technology stack to explain the fundamentals of self-driving technology. How does the Waymo Driver perceive the world? How does it learn to understand its surroundings? How can it predict the intentions of other drivers and pedestrians? And how does it keep our riders safe? We’re starting with one of the foundational questions: how does a self-driving car know where it is? 

Imagine a tiny city where you control everything that happens on the streets. You manage how many cars zip down the roads and how fast they are going. You dictate how many cyclists are on a roundabout or whether they follow the road rules. The “weather” around the vehicle can change multiple times a day from blue skies and sunshine one minute to heavy rain showers the next, but only if you want it that way. One may say such a city doesn’t exist, but if you drive out to the middle of Merced County in California, you’ll find it at Castle, a former Air Force Base our team uses to help build the World’s Most Experienced Driver™. 

Last year around this time, I found myself on a racetrack in Hockenheim, Germany, at Formula Student Germany as a student on the MIT/Delft team. Formula Student is an international design competition, where students with various backgrounds, ranging from mechanical and software engineering to finance and marketing, join forces to design, build, and race a prototype self-driving racecar. This project was one of the most challenging and rewarding experiences of my life; seeing an idea transform from a concept pitch and a system architecture diagram to a car driving itself on a racetrack in Germany was extremely rewarding, but by no means easy.

In March 2020, we launched the Waymo Open Dataset Challenges, inviting researchers to build and test their machine learning models using Waymo’s diverse self-driving dataset. We received over 100 submissions from around the world and invited the winners to present their work at our virtual Workshop on Scalability in Autonomous Driving at CVPR 2020. Today, we are excited to introduce some of the winners and their learnings.

One of the key qualities of a good driver is being able to anticipate and predict what others on the road might do. For example, what is the probability of another car merging into our lane or the cyclist in front of us making a left turn? The ability to accurately predict the intentions of other road users allows the Waymo Driver to make the safest possible decisions.

COVID-19 has had a significant impact on the world, affecting people’s lives and forcing many businesses to suspend their operations. At Waymo, we're actively monitoring the situation, taking steps to support our local communities, and contributing to COVID-19 response efforts. While Waymo has temporarily suspended its on-the-road operations as we put the health and safety of our riders, partners, and employees first, we are still driving our technology forward with our work in simulation.

In early 2009, when Waymo was first founded as the “Google Self-Driving Car Project,” Google founders Sergey Brin and Larry Page challenged our first engineers to drive autonomously without human intervention or disengagements along ten challenging 100-mile routes in our home state of California. By December 2009, the team had completed their first route, and nine months later in mid 2010, we had wrapped up the last.

To help safely navigate the complexities of the road, our self-driving technology needs to see and identify what’s around it. To perceive its surroundings, the Waymo Driver relies on our powerful custom sensor suite of lidar, cameras, and radars, while neural nets empower the “brain” of our self-driving system to understand the sensor data and respond to a wide range of scenarios.

Last August, we invited the research community to join us in accelerating self-driving technology with the release of one of the largest multi-sensor self-driving datasets available today. Even as COVID-19 continues to develop, we are committed to fostering an environment of innovation and learning - one that can continue to grow and thrive in our temporarily virtual world. That is why today, we are launching the next phase of our program: expanding the Waymo Open Dataset by an additional 800 segments and inviting researchers to participate in Waymo’s Open Dataset Challenges.

At Waymo, we’re building the World's Most Experienced Driver - our combination of hardware, software, and compute that powers vehicles to safely get people and things where they’re going.

At Waymo, we use machine learning to detect and classify different types of objects and road features. The powerful neural nets that make up our perception system learn to recognize objects and their corresponding behaviors from labeled examples of everything our Waymo Driver encounters, from joggers and cyclists, to traffic light colors and temporary road signs, or even trees and shrubs. Over the past decade we have built up an enormous collection of objects captured by our powerful custom-designed hardware.