Approximately a week after Datanalisis took the wind out of the opposition’s sails, Consultores 21 filled them back up with a poll showing the race to be a statistical dead heat.
The easy explanation of such a difference is that one or both of the polling firms are being paid off in the much discussed “guerra de encuestas” (war of surveys). However, these are the two most reputable pollsters in Venezuela and both have enviable track records over the past ten years, coming within a couple of percentage points in most every election. They make most of their money from corporate clients who simply want accurate information with which to make decisions about investments and marketing strategies, and both firms would lose their shirts if they started cooking their data. Here I am going to try to look at legitimate reasons that these two reputable pollsters could come up with such different results.
Consultores 21’s June Perfil 21 has HCF at 47.9% and HCR at 44.5% The first thing to notice is that this poll is not showing a surge for HCR; indeed their polling has his support unchanged over the past six months (44.5% in Dec ‘11, 44.8% in March ‘12, and 44.5% in June ‘12).The second issue to note is the small number of undecided respondents, which has been in single digits for the past six months. This is most likely an artifact of how they ask the question. Here it is:
If the presidential election were next Sunday who would you prefer to vote for: Hugo Chávez or Henrique Capriles Radonski?
The results are then reported as HCF, HCR, and “does not choose.” Compare this to Datanalisis’ question:
If the presidential elections were next Sunday, which of the following options would you vote for:
Hugo Chávez
Henrique Capriles Radonski
Neither
Don’t know
Datanalisis reports these four options along with “doesn’t respond.” I will assume that C21’s “doesn’t choose” (7.6%) is the same as Datanalisis’ “doesn’t respond,” (11.6%) and means the respondent simply does not select any of the options. The big difference between the two polls, then, is that 17.1% of Datanalisis’ respondents choose options that are not available to C21’s respondents: “neither” (3.9%) and “don’t know” (13.2%). If all of these respondents (17.1%) were to go for HCR in a poll in which they had to choose (or not answer) then HCR would reach the 44.8% he gets in C21’s polling.
Indeed we could speculate that perhaps some respondents are afraid of manifesting their support for HCR because of precedents such as the Tascón list (which lists the names and ID numbers of all of those who signed the 2003 petition in favor of a recall referendum and which is still used to screen people for government positions) or may for some reason fear losing some benefits from the missions or other social policies. And perhaps only if they are obliged by the question will they really express their choice of candidates. Something like this is said to have happened in the 1990 Nicaraguan elections which the Sandinistas lost, despite polls consistently showing that they would win handily. Analysts suggested that a good percentage of respondents did not respond truthfully to pollsters because they feared repercussions if the Sandinistas won.
While the correspondence in numbers is striking, I consider this hypothesis unconvincing. Not only has it proved wrong time and again—having been trotted out during every election since the 2004 referendum—it doesn’t quite seem to make sense. If people were really guided by such a fear factor, one would think it would be resilient enough to resist the formulation of the question. They would presumably simply not answer, or answer in favor of Chávez. It is hard to imagine that the absence of “undecided” or “don’t know” options would make them give in and answer in favor of Capriles. We would have to know more about how C21 and Datanalisis actually do their fieldwork and how their interviewers present themselves to make this hypothesis plausible.
I think a more mundane explanation having to do with survey sampling is more likely. If the difference simply resulted from the formulation of this one question, or because of the sensitive nature of this one question, we would expect there to be more correspondence between the two polls in other questions. But this is not the case. We will see that in relatively innocuous questions that are similar in C21 and Datanalisis’s surveys, the percentages also tilt towards typically opposition responses. Let’s look at one question that is unlikely to be affected by a presumed fear factor, a question regarding personal situation. In their June Omnibus survey Datanalisis asked the following.
How would you evaluate your personal situation currently?
Good 81.9%
Bad 17.6%
As mentioned yesterday, responses to this question have a high correlation with positive evaluations of HCF. They have soared in recent months and this explains a lot of HCF’s 62% approval rating. Datanalisis also breaks these numbers down by political tendency. So 91.4% of government supporters evaluate their personal situation positively and 8.4% evaluate it negatively. In contrast 67.8% of opposition supporters evaluate their personal situation positively while 31.5% evaluate it negatively.
In their June Perfil 21, Consultores 21 asked the following question:
Speaking about your personal situation, how would you evaluate the economic situation of you and your family?
Good 49%
Bad 48%
The comparison is not easy because the questions are different. I think the difference between the two polls is exaggerated because an individual is more likely to be in a “good situation” than a family (which in Venezuela would be understood as extended family) since one single family member who is out of work, in jail or was recently the victim of a crime can affect an individual’s response even if everyone else in the family is fine. But these are the results that one would expect if C21’s sample simply has a larger representation of opposition supporters than Datanalisis. The same difference in direction (if not degree) occurs in any number of questions. The C21 poll, compared to Datanalisis has:
Situation of country -6.6%
Confidence in CNE -7.1%
Support for PSUV -11.7%
Approval of HCF -9%
Approval of HCR +4.4
Support for Opposition parties +1.1
The differences are somewhat clouded because in every case the questions are slightly different. But we can see that the overall tendency across the board is that, compared to Datanalisis, C21’s results tend towards the opposition.
So whose sample to believe? Modern polling firms do not actually do sampling the way it is taught in introductory survey methods classes, i.e., drawing a random sample and assuming that it will mirror the population. They don’t do that because true random samples are in practice almost impossible to achieve and trying to interview everyone within that sample can actually give you less accurate results since missing cases will have systematic biases. For example, people in rural areas are very difficult to reach; additionally, the number of people that live in households can vary by class, racial and ethnic minorities, which are usually less likely to be sampled. So what pollsters do instead is draw the best sample they can in accessible locations and then weight the results based on the “known” characteristics of the population. Of course, what is “known” about the actual characteristics of the population depends on the census, which has all of the same problems. Thus, pollsters are obliged to use their judgment and experience to know how to weight their samples and that is what can sometimes generate differences between serious pollsters (see for example the recent criticisms of Gallup’s underestimations of support for Obama).
My tendency is to trust Datanalisis’ numbers most at the moment, simply because C21’s results seem so far away from the rest of the polling companies. This would be consistent with the conclusions of Iñaki Sagarzazu —who has become the Nate Silver of Venezuelan polling with his Spanish-language blog YV Polis. In his analysis of January 2012 polls, he has shown that C21 has a pro-opposition bias, while Datanalisis currently does not have a bias.
But I should finish with two points. First, this is “bias” with respect to the average of all polling results, not with respect to the “real” views of the population (to which we have no independent access). There is no guarantee that the average of all polls is not itself biased. Second, in the last major national election (September 2010 legislative elections) C21 was the closest of all polling firms—calling the overall vote within a few tenths of a percent, while Datanalisis erred pro-government by several points. Much of this, of course, depends on turnout. When turnout favors the government, Datanalisis’ sample is truer, and when turnout favors the opposition Consultores 21’s sample performs better.
Datanalisis will be coming out with new numbers in about two weeks and it will be interesting to see if they pick up any movement.
