Two surveys are currently conducted in the automotive industry in South Africa. Both surveys track vehicle quality and customer service offered by the different automotive brands. Both aim to assist manufacturers in identifying quality problems as well as help consumers in making informed decisions when purchasing cars. Very importantly, both surveys are published.
There are, however, conflicting results from the two sets of research, which use different methodologies. To explain the different results and gain understanding of which results have higher credibility, Synovate analysed the different methodologies and presented the results at the SAMRA conference.
With such similarity in the surveys in terms of the market being researched, the topic being researched and the structure of the questionnaire, the difference in results can only be attributed to the difference in the method of data collection. Synovate contacts respondents telephonically while the other research agency uses a mail survey approach.
The question is not 'who is right?' but rather 'which is the most reliable way to collect data for this sort of survey - telephonic or mail?'
"While the use of mail surveys has its place in the South African market, this is restricted to a very specific kind of survey with specific requirements," says Richard Rice, speaker at the SAMRA convention and Client Services Director (Automotive) at Synovate.
Synovate believes that for a survey of this nature, in this industry in South Africa, mail should not even be considered as a collection option. One cannot squeeze international models and approaches into the dynamic South African market.
"With the South African mail service and infrastructure, and the relatively small South African market in comparison to European and American markets, one has to come up with a very good reason before using this technique over telephonic to collect data" added Rice.
Mail surveys: Catch as catch can
The pitfalls of using mail in this country are numerous, and Rice likened the process of using mail to the casting a very wide net and hoping to 'catch' whatever respondents one can. All or a large portion of the entire population are targeted, and those who want to respond will. This leads to unpredictable and varying results. Essentially, respondents select themselves and there is an immense lack of control over the sample.
Other pitfalls of using a mail survey for this project are:
• Slow turnaround
For a survey into product quality, the time between taking ownership of the vehicle and the interview should not be more than four months, after which wear and tear starts to take effect. This time runs out quickly when one depends on the postal system in South Africa.
• Sample frame control
Mail surveys require an explicit list of individuals that are eligible for participation in the survey. The mail survey uses a list from NATIS (National Traffic Information Traffic Systems). This list is of questionable quality and does not differentiate between private customers and dealer and manufacturer staff.
• Response rates are relatively low and inconsistent
• Self-selection: the group that chooses not to respond is NOT identical to that which does. Those with more vehicle problems will naturally respond. Those who have less interest or investment in the topic will not. The sample is therefore biased and unrepresentative, and more problems per vehicle will be reported.
• Data Quality
• Some questions will simply not be completed, or the wrong person will complete it. This skews results.
• In a complex questionnaire like the quality questionnaire, branching and routing means that it is very easy to answer the wrong questions. This leads to double counting of quality problems - something no manufacturer would want to see.
Telephonic surveys: Fish where the fish are
Using the telephonic method, the database is carefully scrutinised and cleaned. Then a truly random sample is selected from only those respondents who qualify according to strict criteria of the survey. The researcher has a high level of control over the sample and can ensure its accurate representation.
Other benefits :
• Telephonic surveys can accommodate complex questionnaires without causing respondents to answer the wrong question.
• Those customers that have not experienced many problems with their vehicle will only be subjected to a short questionnaire.
• Telephonic interviews can be conducted speedily - meaning that respondents will be contacted before wear and tear on their vehicle starts having an adverse impact on the results.
• The sample is tightly controlled. Lists are scrutinised. Quality gates ensure 100% submission of sales and customer data. Those who do not meet criteria are excluded (including fleet managers and employees).
• The researcher selects who participates.
• Response rates are high as it is more difficult for a respondent to opt out.
• The full questionnaire is completed and questions that are not understood by the respondent can be clarified by the interviewer
In conclusion, Rice stated that "in different environments, the mail methodology can be successful. In South Africa, tracking studies are best conducted using the telephone".
Rice added: "Data collection is a crucial step in the research process. Respondents are the raw material for us - and the method that proves best at obtaining the most reliable information from respondents should be preferred and used."
For more information or a copy of the paper presented at the SAMRA Conference, please contact Richard Rice at .
