A new Robo-Car report card is not really what it seems

The last batch The California Department of Motor Vehicles has just released reports on the disengagement of autonomous vehicle manufacturers, the closest object to a robotic report card. The columns and columns of data it contains do not quite illuminate the secrets of the highly secretive autonomous vehicle industry. But his many pages clearly indicate that, even though the hype around Silicon Valley race cars may have cooled, progress continues towards the day when humans are not stuck in the wheel: December 2017 and November 2018, against 500 000 the previous year.

These reports indicate the number of times that each company's vehicle has "disengaged" from stand-alone mode and returned to the old-fashioned, manual, and human manual mode. Waymo, for example, said that a pilot had to take over once every 11,017 miles, 97% better than last year. GM Cruise announced a takeover every 5,205 kilometers, 320% better than a year ago.

But like many standardized tests, reports are missing a lot. These are bad tools to understand the functioning of this technology and compare the progress of each company to those of its competitors. Some in space fear that reports may encourage engineers to design their cars to limit disengagements, in order to extract statistics.

Nevertheless, in a new sector based on complex software and hardware, in which secrecy is paramount, any public information can be revealing. And there is a knowledge to be learned from these thousands and thousands of pages.

Before diving, a few words about the reports themselves. California is the only state to require something similar (developers must also publicly report all collisions). And to understand them, you have to accept some big warnings:

• The reports are not scientific because each company reports its data in a different way, offering different levels of detail and idiosyncratic explanations of the reasons that drove their vehicles to leave the autonomous driving mode.
• They are packed with vague language and lack of context. It was supposed to be the first year that all companies used a standard data entry format but, first, they did not all seem to have followed the rules. And secondly, their language is still very vague. Waymo, for example, cites a series of disengagements for "an unwanted movement of the vehicle that was undesirable under the circumstances". Uber calls a ton of disengagement "takeover by precaution or discretion of the operator". Are these situations similar? Who knows!
• The reports are of little use to anyone who wants to compare rival companies, because these companies do not do the same tests. Waymo performs most of his tests in simple suburbs; Zoox focuses on the complex city. It's better to track each team's progress, but it's not great because these companies change how and where they test over time. And comparing the number of kilometers traveled by each company does not tell you much about anyone's progress.
• Crucial point: these reports only cover driving on public roads California. So we do not know anything about Ford, which focuses its tests on Detroit and Pittsburgh. And we do not see data for Waymo's growing test program in Phoenix. (Dawn Is include data from its testing activities in Pittsburgh, but without detailing specific disengagements.)

If we put aside the mistaken idea of ​​using disengagements as a metric, we can still learn a lot from these reports. They also include information on the number of kilometers traveled by the companies, the size of their fleet and the type of roads they test.

Let's start our 21st century Kremlinology by noting the mix of players. Oldies like Waymo, General Motors' Cruise, Nissan and Uber are joined by little-known newcomers like Roadstar.Ai, aiPod, Nullmax and WeRide. The smallest names have become big names: Aurora has just raised $ 530 million of funds, Nuro, focused on delivery, reported nearly a billion dollars, and Apple has finally unveiled information on its driving project Autonomous secret (and perhaps in difficulty). All three filed their first disengagement report this year. So, yes, money does not circulate as freely in Autonoworld as it once did. But the growing number of competitors indicates that the dream of the autonomous car is still alive.

The reports also indicate that the testing operations of almost all businesses are growing. Cruise tested 109 vehicles in the state during the period covered by the report; now, there are 194. (Most of this news is Generation 3, the company's power generation model, AV). Waymo has nearly tripled the number of cars in the state. Uber is the exception: he closed his self-driving program in California almost a year ago, after one of his test vehicles hit and killed a woman in Arizona. In total, the DMV had approved the testing of 665 Robocars in California at the end of November 2018, up from 326 the previous year.

Some of these vehicles covered more ground than others – and show how much each company's self-driving strategy differs from that of its competitors. Aurora's five vehicles traveled only 32,858 miles in stand-alone mode, far less than Waymo or Cruise, with a higher disengagement rate than the two companies. This low number is intentional, says Aurora co-founder Sterling Anderson. Riding in public has a cost: in fuel, in salary for drivers sitting behind the wheel (and also receiving a participation at startup) and in case of risk of accident. Aurora realizes the essential of its development in simulation and on track of test. He reserves his tests on the road, in public, to see how his technology evolves. When he stops seeing high disengagement rates, he moves into a new geographical area, with new challenges.

"Road tests should not be a development," says Anderson. "Road tests, it's more checking."

Competitors might call this the feat of strength – Aurora's way of explaining low distances and large disengagements. What's interesting is that the company represents a second act for each of its three founders: Chris Urmson led the Google program (now Waymo), Drew Bagnell helped start the Uber and Anderson team to the Tesla autopilot system. The fact that they used their new start to install on this unusual test method shows that the methods of creating this technology are evolving and diversifying themselves, sign of a maturing sector. gradually.

More great cable stories