For the sake of transparency I should disclose that I don’t drive a Volkswagen Polo. I’ve never owned one either. So I don’t have any skin in this game.
South African Volkswagen Polo owners were called out in January 2022 by numerous media outlets, including Eyewitness News, the Citizen, Independent Online, BusinessDay and SowetanLive.
“Your suspicion was correct: VW Polo drivers are the worst,” read the headline of a TimesLive article.
The article quoted a study conducted by South Africa’s Road Traffic Management Corporation (RTMC). A 26 January press release from the corporation declared: “Volkswagen Polo Drivers are the Worst, a Study Confirms.”
There’s just one problem. The study they all reported on did not find that Polo drivers were “the worst” drivers in South Africa. The claim is a statistical blunder and should never have been reported as widely as it was.
More Polos on the road mean more Polos in accidents
The study, South African Fatal Crashes in Context, was released by the RTMC in December 2021. It found that between 1 October 2017 and 30 June 2021 the Volkswagen Polo accounted for the highest percentage of fatal crashes.
But that doesn’t reveal anything about the ability of Polo drivers. It reveals something more obvious, and a great deal less exciting: the Polo is a common model of car in South Africa.
Lightstone Auto is a market analytics company and records vehicle sales data and other information. A spokesperson from the company gave Africa Check its estimates of the most popular cars in South Africa. The Polo was the second and the Polo Vivo the fourth most common model in the country. (The RTMC did not distinguish between the two.)
Together, these two models were more popular than any other single model of car and made up around 6.36% of all “units in circulation”.
More Polos, more problems
Dr Leanne Scott, an associate professor of statistics at the University of Cape Town, told Africa Check that for a study to reliably determine which drivers are “the worst”, it would need to control for several factors.
To begin with, “the study would need to take into account the base rates of car ownership”. This refers to the proportion of drivers that drive each type of car. That the Polo is so popular could have a significant impact on the results.
The report acknowledged this, though it used the National Administration Traffic Information System (Natis), a national register which stores, among other information, the license and registration details of South African vehicles, to estimate the total number of vehicles on the roads and which were involved in crashes.
In its breakdown of the makes and models of cars involved in the highest number of fatal crashes, the RTMC estimated that Polos made up roughly 17.0% of all cars by model, and were involved in 16.7% of all fatal crashes.
This showed that the number of Polos in fatal accidents was not out of proportion with the number on the roads. The study actually noted that Polos were not overrepresented in fatal crash statistics, only in recorded speeding infringements.
But this wasn’t the only factor the study failed to control for.
Laundry list of issues with study
The RTMC could link only 66.9% of cars involved in fatal crashes to a vehicle in the Natis database. This places an obvious early limitation on the accuracy of the study and its conclusion.
It is possible to draw reasonable conclusions from an incomplete data set. However, the RTMC study has other issues which make it impossible to determine which South African drivers are “the worst”.
Dr Sebnem Er, also a researcher and lecturer in the statistics department at the University of Cape Town, told Africa Check that “lots of other factors need to be considered when dealing with this type of data”.
A particular type of car, for example, might be more likely to be involved in a fatal crash, regardless of driver ability. Er pointed to the Toyota Quantum as an example.
The Quantum was involved in the third highest proportion of fatal crashes of any model of car in South Africa, but Er explained that it shouldn’t be included in the same category as cars like the Vivo. Er said that “almost all Toyota Quantums in South Africa are minibus taxis”, a common form of public transportation in the country. According to Er, this – whether a vehicle is a taxi or not – “needs to be controlled [for] first”.
The study did note some of these factors, but did not control for them in its analysis, or compare only the same types of vehicle in its comparison of makes and models.
Why can’t some types of vehicle be compared to others? One South African media outlet, which reported accurately on the RTMC study, explained it well.
The Mail & Guardian’s head of digital Adam Oxford wrote: “A taxi that is on the road for 10 hours a day or more is obviously more likely to be involved in an incident than a Polo that does a 20-minute commute twice every 24 hours.”
Oxford found that the Toyota Quantum accounted for more fatal crashes per 1,000 vehicles than any other model. But this still tells us nothing about the ability of Quantum drivers. Controlling for other factors, like the amount of time spent driving each day, would almost certainly produce a different result.
Counting only fatal accidents introduces new issues. Scott told Africa Check that “being involved in a fatal accident does not imply that the driver was at fault, or driving badly”.
Even if they had the same likelihood of being involved in an accident, a car model with fewer safety features would be more likely to be involved in a fatal accident than a car with more or better safety features.
The RTMC did not control for any of these factors, and the study did not conclude that drivers of a particular model of car were “worse” than any other drivers.
The worst data crime of all? Confusion is the RTMC’s fault!
The Mail & Guardian rightly pointed to the importance of studies like this in helping us better understand road accidents in South Africa. The RTMC can be commended for this.
But the corporation is also responsible for the misleading press release which misrepresented its own study’s findings and called Polo drivers “the worst”.
Some blame also lies with the media’s reporting on the press release. It’s a clear example of why journalists should do more than read the press release about a study. Luckily we have a guide to help, on how to craft accurate science journalism out of press releases.
Add new comment