Survey Quagmire in Macau

Opinion by Keith Morrison

Macau sinks in local surveys that have little or no credible meaning or impact on its development, and whose quality renders them immediately fit for the bin. In fact, does anybody take any notice of them? We are bombarded with telephone surveys on a huge range of topics, with questions that are jaw-droppingly naïve, low-level, biased, selective, simplistic, ill-thought-out, full of irrelevances and spurious proxies, of questionable validity, reliability, utility, and with unsafe extrapolations. They ask leading questions and violate elementary principals of survey design and conduct, with no consequential benefit or impact, and from a sample whose representativeness is questionable. What a waste of time and money.

We have surveys asking for opinions on quality of life, the Macau government, consumers’ and employees’ confidence and satisfaction with Macau life and government, graduates’ career choices, even drainage reconstruction. But opinion alone may be insufficient; knowing people’s opinions might be useful if it affects practices, planning, decision and policy making, but there is little or no evidence of this. Whilst closed questions (usually asked in such surveys), can yield simple frequencies, without more open-ended ‘why’ questions, surveys do little to help planners or developers. We have opinion surveys on Macau’s leaders, asking questions that have a spurious precision, e.g. awarding percentage points for how well they are performing. What, in reality, is the difference between, say 55 and 56 per cent? It’s meaningless.

In a recent telephone survey on the ‘Mobile Web Weather Service’ App, from Macau’s Meteorological and Geophysical Bureau, most of the questions were irrelevant, as the interviewee reported not having or using the App, yet the interviewer persisted in asking questions about it. Rule 1: inappropriate respondents; invalid responses.

The surveys that I have seen in Macau are hopelessly reductionist, asking monstrously over-simplistic, invalid questions about complex issues. I am reminded of a recent television interviewer who asked interviewees to reply ‘yes, no or maybe’ to what was a vast, complex matter. Nonsense.

Perception-based, opinion-based data are important, as people act on the basis of their opinions: if I think there is a mouse under the table I will act as if there really is a mouse under the table, whether there is or is not. ‘All well and good’, you might say, but have I seen any consequent action? Not in Macau. Nor have I seen sufficient rigour in terms of validity and reliability in opinion-based surveys in Macau, and the absence of corroborating, objective data undermines their value and reliability.

Further, we have to be wary of eye-catching newspaper headlines. A recent headline stated that in Macau that the ‘mainland becomes pupils’ 1st choice for higher education’ and that ‘about 40 per cent of local senior high school pupils chose to study at the mainland universities’. However, the data presented simply didn’t support this. The data came from Form 4 and Form 5 students, i.e. students who had not yet gone to university and who were still one or more years away from even applying for university, so it was simply an idea, nothing more, i.e. the word ‘chose’ was misleading. The data came from 1,128 students in 11 local schools. How representative was this? Was it a random sample, a random stratified sample, or simply an opportunity/volunteer sample, i.e. biased? A survey’s credibility, validity, reliability and inferential trustworthiness hinge in large part on the sampling strategy used, not only sample size.

In the same survey, of the 91.8 per cent who indicated that they planned to continue studying, 38.7 per cent, i.e. 401 students (rounded) indicated that they ‘chose’ to study at mainland universities. So, out of around some 9,000 students in those two age groups in Macau, less than 4.5 per cent here indicated that they wished to study in mainland universities. That hardly inspires confidence in the claims being made. Further, 25.2 per cent of the survey’s students ‘chose’ to enrol in local universities, 4.9 per cent in Taiwan, and 3 per cent in a ‘foreign country’, i.e. mainland universities were not their first choice. As the Nobel prize-winning physicist, Wolfgang Pauli wrote so tellingly: ‘This isn’t right. This isn’t even wrong’.

Maybe with a new Chief Executive we will have decent surveys which impact on subsequent decision making, instead of drowning in endless, meaningless survey slurry.