COMMENT: Farm murder rate calculations should be transparent - Politicsweb’s isn’t
Calling it a fact-check, Politicsweb editor James Myburgh earlier this year reviewed the work of Africa Check and the BBC’s Reality Check on South Africa’s farm murder rate.
“While these ‘fact-checkers’ have raised legitimate points around the methodology for calculating the murder rate for white farmers, it is unclear why instead of trying to come up with a better estimate they went with the assertion that this was, alone among all the many great mysteries of the universe, fundamentally unknowable,” Myburgh wrote.
He quoted a BBC Reality Check article that said “we don't know how many white people there are on South Africa's farms. And that means we can't work out how likely those white people are to be murdered.”
Myburgh then presented what appeared to be a convincing rebuttal: “In fact, contrary to these claims, Statistics South Africa’s 2016 Community Survey did produce estimates of the size of that section of the white population that is affected by farm attacks.”
He said the survey “found that there were 47,218 white-headed households engaged in some kind of agricultural activities on ‘farm land’ – including both commercial farm land and smallholdings”.
Myburgh used this figure and a number of 51 white farmers murdered in 2016/17 collected by the Transvaal Agricultural Union to calculate a “murder rate for white farmers” of 108 per 100,000. (Note: The number murdered white farmers was 55, according to the union.) This rate was 3.2 times higher than the national murder rate that year, he argued.
Africa Check welcomes anyone “fact-checking the fact-checkers” as the fact-checking process is not beyond criticism and mistakes can be made. But Myburgh’s analysis fell short of a number of basic fact-checking principles.
The first step fact-checkers take is to contact the person they’re fact-checking. Why? This allows the person or organisation to present their evidence, ensuring that the fact-checker doesn’t overlook crucial information or context.
Myburgh wanted to know why Africa Check - and others - haven’t attempted to come up with a “better estimate” for the “murder rate for white farmers” in South Africa. If he had asked me this before publishing his analysis, I would have said two things.
First, we haven’t fact-checked the “methodology for calculating the murder rate for white farmers”, as he claims. Our previous work examined claims made about the murder rate for all people living, working and visiting farms and smallholdings.
This was what we described as “near impossible” to calculate. At the time, there wasn’t a reliable estimate of the number of people - of all races - living, working and visiting farms and smallholdings. There still isn’t.
The second thing I would have told him was that I too had thought the 2016 Community Survey could provide an estimate of the number of agricultural households.
Back in November 2017, I contacted Stats SA to ask how many agricultural households were based on farmland and had listed their involvement in agriculture as their "main source of household income".
To my disappointment, Stats SA said they had not released absolute numbers, only proportions, due to a number of problems.
The question, then, is how did Myburgh get hold of numbers that Stats SA said they didn’t release?
When I contacted Myburgh in June 2018, he said his estimate of “47,218 white-headed households” was from “a table from Stats SA from the community survey that I requested and got”. He wouldn’t name the person who sent him the data or provide evidence of the correspondence.
The statistical agency said they had no knowledge of Myburgh requesting the numbers. The table he published is also not included in any of the community survey publications.
“The absolute numbers that are referred to were not published by Statistics South Africa and no absolute numbers were provided to the user,” Kevin Parry, media relations for economic statistics, told Africa Check.
Parry said they had low rates of response to a number of questions and, as a result, decided only to include proportions for the households that did answer.
A note below a table that shows 37.9% of white agricultural households were located on farmland suggests this, although it is not explicit. The note says the table’s figures “represent the proportions of all households who responded to the question”. (Note: Stats SA, unfortunately, didn’t include this proviso in all its reports on the community survey, including a presentation of the findings by statistician-general Pali Lehohla.)
Because of this, Stats SA says you mustn’t take that 37.9%, apply it to the 143,361 white agricultural households and say there are 54,334 of them are on farmland. (Note: This is what BBC Reality Check did and we’ve written to them about it.)
It’s also impossible to extract the numbers Myburgh published with Stats SA’s SuperWEB2 data tool. The applicable options to produce the table are simply unavailable.
The way households were sampled also hindered the calculation of absolute numbers for those involved in agriculture, according to the agency’s deputy director-general for marketing, communication and publishing, Ashwell Jenneker.
“The sampling in the community survey is designed to report service delivery and other socioeconomic indicators at municipal level. In other words, household involvement in agriculture was not the basis of the sampling specifications,” he told Africa Check.
“Therefore the number of households involved in agriculture in this sample may not be representative of the agriculture sector in the country in its entirety.”
The shortcomings in the community survey are an extreme disappointment. There are also concerns about the accuracy of the survey’s data on the number of immigrants in South Africa.
Stats SA appears to have squandered an opportunity to provide important information about communities who have become political footballs.
Africa Check is a signatory of the International Fact-Checking Network’s Code of Principles. This means, among other things, that we are committed to the transparency of our sources and methods of research.
The principles state that “signatories want their readers to be able to verify findings themselves. [They] provide all sources in enough detail that readers can replicate their work.”
We do this by linking to all the data we use and explaining how we accessed it. This means that a different person can reach the same conclusion, using the same data.
Myburgh’s calculations are not replicable, meaning you can’t check them and just have to trust his conclusions. This isn’t how fact-checking and verification works.
I can’t prove that Stats SA didn’t provide Myburgh with the absolute numbers. But if he comes clean on who gave him the data, we can further press the statistical agency on why they claim the numbers can’t be extracted.
I agree with Myburgh that the farmer murder rate should not be “fundamentally unknowable”. I want a reliable estimate to be calculated. That’s, in part, why this response has been delayed. I didn’t want to say “we don’t know” again.
But 2019 could be the year things change.
Stats SA is currently conducting a census of commercial agriculture, due to be released towards the end next year. It will give us a new estimate of the number of commercial farmers in South Africa, defined as those who are value-added tax registered. The last estimate we have is from 2007.
This gives organisations like the Transvaal Agricultural Union and AfriForum a year to collect and disaggregate their data on the victims of farm murders. If they do their part, we might finally be able to calculate the risk commercial farmers face in South Africa.
Further reading:
https://africacheck.org/2017/05/08/analysis-calculating-farm-murder-rate-sa-near-impossible/
https://africacheck.org/factsheets/factsheet-statistics-farm-attacks-murders-sa/
“While these ‘fact-checkers’ have raised legitimate points around the methodology for calculating the murder rate for white farmers, it is unclear why instead of trying to come up with a better estimate they went with the assertion that this was, alone among all the many great mysteries of the universe, fundamentally unknowable,” Myburgh wrote.
He quoted a BBC Reality Check article that said “we don't know how many white people there are on South Africa's farms. And that means we can't work out how likely those white people are to be murdered.”
Myburgh then presented what appeared to be a convincing rebuttal: “In fact, contrary to these claims, Statistics South Africa’s 2016 Community Survey did produce estimates of the size of that section of the white population that is affected by farm attacks.”
He said the survey “found that there were 47,218 white-headed households engaged in some kind of agricultural activities on ‘farm land’ – including both commercial farm land and smallholdings”.
Myburgh used this figure and a number of 51 white farmers murdered in 2016/17 collected by the Transvaal Agricultural Union to calculate a “murder rate for white farmers” of 108 per 100,000. (Note: The number murdered white farmers was 55, according to the union.) This rate was 3.2 times higher than the national murder rate that year, he argued.
Africa Check welcomes anyone “fact-checking the fact-checkers” as the fact-checking process is not beyond criticism and mistakes can be made. But Myburgh’s analysis fell short of a number of basic fact-checking principles.
Contact the person you’re checking
The first step fact-checkers take is to contact the person they’re fact-checking. Why? This allows the person or organisation to present their evidence, ensuring that the fact-checker doesn’t overlook crucial information or context.
Myburgh wanted to know why Africa Check - and others - haven’t attempted to come up with a “better estimate” for the “murder rate for white farmers” in South Africa. If he had asked me this before publishing his analysis, I would have said two things.
First, we haven’t fact-checked the “methodology for calculating the murder rate for white farmers”, as he claims. Our previous work examined claims made about the murder rate for all people living, working and visiting farms and smallholdings.
This was what we described as “near impossible” to calculate. At the time, there wasn’t a reliable estimate of the number of people - of all races - living, working and visiting farms and smallholdings. There still isn’t.
The second thing I would have told him was that I too had thought the 2016 Community Survey could provide an estimate of the number of agricultural households.
Back in November 2017, I contacted Stats SA to ask how many agricultural households were based on farmland and had listed their involvement in agriculture as their "main source of household income".
To my disappointment, Stats SA said they had not released absolute numbers, only proportions, due to a number of problems.
The question, then, is how did Myburgh get hold of numbers that Stats SA said they didn’t release?
‘No absolute numbers were provided to the user’
When I contacted Myburgh in June 2018, he said his estimate of “47,218 white-headed households” was from “a table from Stats SA from the community survey that I requested and got”. He wouldn’t name the person who sent him the data or provide evidence of the correspondence.
The statistical agency said they had no knowledge of Myburgh requesting the numbers. The table he published is also not included in any of the community survey publications.
“The absolute numbers that are referred to were not published by Statistics South Africa and no absolute numbers were provided to the user,” Kevin Parry, media relations for economic statistics, told Africa Check.
Parry said they had low rates of response to a number of questions and, as a result, decided only to include proportions for the households that did answer.
A note below a table that shows 37.9% of white agricultural households were located on farmland suggests this, although it is not explicit. The note says the table’s figures “represent the proportions of all households who responded to the question”. (Note: Stats SA, unfortunately, didn’t include this proviso in all its reports on the community survey, including a presentation of the findings by statistician-general Pali Lehohla.)
Because of this, Stats SA says you mustn’t take that 37.9%, apply it to the 143,361 white agricultural households and say there are 54,334 of them are on farmland. (Note: This is what BBC Reality Check did and we’ve written to them about it.)
It’s also impossible to extract the numbers Myburgh published with Stats SA’s SuperWEB2 data tool. The applicable options to produce the table are simply unavailable.
Survey shortcomings disappointing
The way households were sampled also hindered the calculation of absolute numbers for those involved in agriculture, according to the agency’s deputy director-general for marketing, communication and publishing, Ashwell Jenneker.
“The sampling in the community survey is designed to report service delivery and other socioeconomic indicators at municipal level. In other words, household involvement in agriculture was not the basis of the sampling specifications,” he told Africa Check.
“Therefore the number of households involved in agriculture in this sample may not be representative of the agriculture sector in the country in its entirety.”
The shortcomings in the community survey are an extreme disappointment. There are also concerns about the accuracy of the survey’s data on the number of immigrants in South Africa.
Stats SA appears to have squandered an opportunity to provide important information about communities who have become political footballs.
Transparency of sources and methodology
Africa Check is a signatory of the International Fact-Checking Network’s Code of Principles. This means, among other things, that we are committed to the transparency of our sources and methods of research.
The principles state that “signatories want their readers to be able to verify findings themselves. [They] provide all sources in enough detail that readers can replicate their work.”
We do this by linking to all the data we use and explaining how we accessed it. This means that a different person can reach the same conclusion, using the same data.
Myburgh’s calculations are not replicable, meaning you can’t check them and just have to trust his conclusions. This isn’t how fact-checking and verification works.
I can’t prove that Stats SA didn’t provide Myburgh with the absolute numbers. But if he comes clean on who gave him the data, we can further press the statistical agency on why they claim the numbers can’t be extracted.
New data could bring better estimates in 2019
I agree with Myburgh that the farmer murder rate should not be “fundamentally unknowable”. I want a reliable estimate to be calculated. That’s, in part, why this response has been delayed. I didn’t want to say “we don’t know” again.
But 2019 could be the year things change.
Stats SA is currently conducting a census of commercial agriculture, due to be released towards the end next year. It will give us a new estimate of the number of commercial farmers in South Africa, defined as those who are value-added tax registered. The last estimate we have is from 2007.
This gives organisations like the Transvaal Agricultural Union and AfriForum a year to collect and disaggregate their data on the victims of farm murders. If they do their part, we might finally be able to calculate the risk commercial farmers face in South Africa.
Further reading:
https://africacheck.org/2017/05/08/analysis-calculating-farm-murder-rate-sa-near-impossible/
https://africacheck.org/factsheets/factsheet-statistics-farm-attacks-murders-sa/
Add new comment