Why social network surveys don't necessarily reflect the views of SA youth

Do 38% of South Africa’s youth believe that the policies recently announced by Julius Malema’s Economic Freedom Fighters will win the party support? Would 26% of the country’s youth vote for the disgraced former ANC Youth League president’s new party?  Do 42% of young people believe that foreigners have a negative impact on South Africa? And do as many as 52% of young black South Africans truly believe the Democratic Alliance will bring back apartheid if they win an election?

These are some of the startling claims made by the “consumer insights company”, Pondering Panda, which boasts that its use of “mobile technology” to conduct youth surveys allows it to “deliver much larger samples, faster and more cost-effectively than traditional research suppliers”.

Courting controversy

Over the past six months, the company has peppered the South African media with headline-grabbing press releases announcing its latest findings. It has notched up an impressive hit rate. Its website lists close to 200 news articles published between late January and early August 2013 that quote from, or are based on, its surveys. A sizable number regurgitate the contents of the company’s press releases.

As a result, Pondering Panda has garnered hundreds of thousands of rands – possibly millions – in free publicity and has secured some of South Africa’s best-known brands as its clients. The company’s CEO, Butch Rice, has also made a name for himself with his highly critical and controversial comments about traditional research and polling methods and those who use them.

The research industry in South Africa is a "Jurassic Park", he was quoted as saying at a conference in May this year. "The traditional research industry still clings stubbornly to invalid analytical techniques, inefficient research designs, and cumbersome and expensive reporting, which takes an inordinately long time to deliver."

The tagline on Pondering Panda’s website promises: “A new wave of research. Finally.”  The company boasts that its surveys have “faster turnaround times” with “typically 2,000 responses in a 24-hour period”.

But do these surveys accurately reflect the views of South Africa’s youth? Are they reliable? And are the media being critical enough when electing to publish their findings?

Opting-in to social media

Pondering Panda’s press releases contain an important caveat. “All interviews were carried out on cellphones ... across South Africa,” they state, and the responses are “weighted” to be nationally representative of age, gender and race. Findings are generally attributed to “South African youth”.

What the press releases do not say is that all their research is done using an “opt-in system” that surveys users of the MXit social media platform. (It is an odd omission, given that the Pondering Panda website states that it has “exclusive access to MXit”, which it describes as “Africa’s biggest social network”.)

As an example, we decided to look a little more closely at the Pondering Panda survey which found that the “majority” of young blacks in South Africa believed the Democratic Alliance (DA) would bring back apartheid if they came to power.

The survey gave the company headlines, and lots of free publicity, after its release in April this year. Reporters who wrote about it made little or no attempt to seek further comment or to interrogate and contextualise the findings. Entire paragraphs were lifted from the Pondering Panda press release or attributed to its spokesperson.

Pondering Panda states that “full results are available on request”. When Africa Check requested the results of the DA survey, we were sent a spreadsheet document. It indicated that the survey was conducted over six days and that there were 3,009 respondents aged between 15 and 34. It did not give a detailed explanation of how the data was collected and weighted. The only mention of MXit in the document was the statement: “All data gathered over MXit social network”.

The devil in the detail

How are the surveys conducted? Rice told Africa Check: “Data is collected by an opt-in option on the MXit base, which has 6.5 million active users on a monthly basis. The sample sizes are significantly larger than those of traditional research suppliers, often running into tens of thousands. Data from the Census is used to weight or balance the sample, in line with standard research practice by all companies,” he said, adding that “the key to sample representivity is dispersion”.

Herein lies a problem: MXit, like many social media tools, does not require people to use their real names when signing up. Many don’t.

Rice claimed Pondering Panda had checks and balances in place to ensure the platform yielded “valid and reliable data”, including checking a respondent’s identity against details they supplied when signing up to MXit and cross-comparing it to information supplied when they complete surveys.

How effective these checks and balances are remains unclear. An Africa Check researcher signed up for MXit under a false name, provided a false date of birth, downloaded the Pondering Panda app and completed a number of surveys. One of them asked for details on race and geographic location. The Africa Check researcher claimed to be a black male, aged 18 to 24, from the Northern Cape. (The age correlated with the date of birth given on signing up.) The researcher is, in fact, a 38-year-old white male from Gauteng.

A further problem is that there is some confusion about the number of active MXit users in SA. Pondering Panda claims on its website that MXit has 6.5-million active users in SA. But stats published by MXit in October 2012 stated there were almost 50-million registered users and 9,346,806 active users. Yet six months later, in March 2013, Vincent Maher, vice president of growth and product strategy at MXit, said the “latest active users’ statistics” for South Africa was 6.3-million per month.

Questions and answers

Another key factor to consider is the way the survey question was formulated: “Some people think the DA will bring back apartheid if they won an election. Do you think this is true?”

The question is problematic because different people will interpret it differently, said Derek Luyt, a researcher, analyst and director of the Centre for Accountable Governance.

For instance, some respondents may think they are being asked whether it is true that “some people” think the DA will bring back apartheid, not whether they themselves hold that view. Respondents could also have very different ideas about what is meant by the word “apartheid”. Some of them were probably born after apartheid was scrapped, or were too young to remember exactly what it was like.

“It’s not quite as bad as the old, ‘Do you prefer apartheid or socialism?’ question, but it’s not a question that is particularly meaningful either,’ Luyt told Africa Check.

"Unquestioning" journalists

Lauren Shapiro, a director of Futurefact - one of South Africa’s longest-running research and survey companies  -  said social media research, like that conducted by Pondering Panda, certainly has a place. But she cautioned that it was always necessary to reveal the technical details of how such surveys were conducted.

Rice, a co-founder of Research Surveys, now TNS South Africa, lectured in statistics and marketing research at the University of Cape Town for 10 years. He disparages “just about all” other research methods. Despite questions about the accuracy of data collection on MXit, compared to more conventional door-to-door surveys where people are interviewed face-to-face, he told a marketing conference: “The future of market research is digital, and the digital future is mobile. It is an indictment of market research in South Africa that companies are still investing in door-to-door interviews and focus groups”.

Luyt said he felt that “a large part of the problem is the unquestioning attitude of the media”. “Instead of dissing others, Pondering Panda would probably do the media more of a favour by trying to enlighten journalists about research methods rather than trying to dazzle them with sexy headlines.

"There is a definitely a place for the type of surveys Pondering Panda does, but it is also important to accept the limitations and not to claim to do more than you actually do. You cannot compare it to the work of researchers who do deeper, longer-term research and try to understand things in greater depth and detail.”

Media consultant Gordon Muller believes that research using social media platforms is becoming increasingly important for the quick insights they give.

“To dismiss all other kinds of research as irrelevant and wasteful is wrong...They all have a place. From a macro point-of-view, using a dipstick approach is where social media comes in. But I would urge caution to any researcher who says they are talking on behalf of all youth in South Africa.

“My real issue is why trained journalists are just picking it up and running with it,  without examining it critically and asking questions?”

Conclusion - Surveys  do not represent views of SA youth

Pondering Panda cannot claim that their findings are representative of “South African youth” or - in the case of the DA survey - of the “majority of young blacks”. In the latter case, at most they can claim that 52% of the MXit users who installed their app, took the time to respond to the survey and who said they were young and black believed that either (1) the DA would bring back apartheid or (2) that "some people" believed the DA would bring back apartheid. It is not quite the same thing as the “majority of young black South Africans”.

Pondering Panda's suggestion that it conducts cellphone "interviews" is also misleading. There are no interviews in the true sense of the word. Respondents who have downloaded the Pondering Panda app choose which surveys to fill out and tick off the answers to questions.

The real blame, however, lies with the media outlets that have unquestioningly swallowed  Pondering Panda’s headline-grabbing press releases. There are probably a host of explanations: budget cutbacks, overworked and inexperienced reporters and an insatiable appetite for cheap, cut-and-paste web content.

The reporting on Pondering Panda, for the most part, has not been journalism. Sadly, far too many of the articles that Pondering Panda so proudly link to on their website are classic examples of “churnalism”. And the reporters have become little more than public relations tools, unquestioningly and unthinkingly churning out copy that has a gimmicky appeal but very little merit.

Edited by Julian Rademeyer