March 16, 2023
Driving Research Forward: The Waymo Open Dataset Updates and 2023 Challenges
Waymo’s recent technology advancements and rapid expansion across Phoenix, San Francisco, and Los Angeles wouldn’t be possible without the underlying innovative research that helps drive our progress forward. As a pioneer in the AV industry, we have continuously contributed to the research community through publishing and expanding the Waymo Open Dataset — one of the largest and most diverse autonomous driving datasets ever released. Today, we are excited to announce the latest expansion of the dataset with even more sensor data and labels. We are also launching our new and exciting 2023 Waymo Open Dataset Challenges -- this is our 4th annual Challenge edition, and we invite all researchers to participate!
Since launching the Waymo Open Dataset in 2019, over 36,000 researchers around the world have used it to explore new approaches and make important progress in computer vision, behavior prediction, and broader machine learning topics. They have published over 1,300 academic papers, including research on point-based 3D object detection, end-to-end modeling for trajectory prediction, and bird’s-eye-view representation from multiple camera images, as well as work in fields beyond autonomous driving. We have also utilized the dataset in collaboration with Google Brain to create a new benchmark for Causal Agents — agents whose presence influences human drivers’ behavior in any format — aiding the research community in building more reliable models for motion forecasting.
- 2D Video Panoptic Segmentation Challenge: Given a panoramic video of camera images, produce a set of panoptic segmentation labels for each pixel in each image, where each object in the scene is tracked across cameras and over time.
- Pose Estimation Challenge: Given one or more lidar range images and the associated camera images, predict 3D keypoints for pedestrians and cyclists in the scene.
- Motion Prediction Challenge: Predict the future positions of multiple agents in the scene given 1 second of past history. We have made lidar sensor data available as a model input.
- Sim Agents Challenge: Produce sim agent models that control agents in the scene, which will be evaluated against the goal of being human-like. This is the first competition on simulated agents, a fundamental but relatively underexplored topic in autonomous driving, now available to the research community.
In addition to the 2023 Challenges, we’re releasing newer versions of the Perception and Motion datasets, as well as introducing a new dataset structure, the Modular dataset.
Perception Dataset:
- We have added the road graph definition for all perception segments, enabling research on roadgraph reconstruction. Also, with this release, both the Perception and Motion datasets will have corresponding roadgraph and lidar available, which can enable research on end-to-end learning for prediction.
- Given the large number and size of features now included in the dataset, we’re adopting a new modular data format and an accompanying python library, which enables users to download and use different features individually. This new format will make it easier for users to efficiently access and explore our data.
- We have added a new feature for 2D video panoptic segmentation, which provides masks for each image designating how many cameras cover each pixel. This can be used to compute the weighted Segmentation and Tracking Quality metric.
- We have added lidar sensor data for all segments. Compared to our Perception dataset, this drastically increases the amount of lidar segments available, enabling further long-tail, semi-supervised and unsupervised perception research. Furthermore, lidar can provide rich signal about the behavior and intents of road users, opening opportunities for end-to-end perception and behavior modeling.
- We have also added driveways to the dataset.
