Every once in a while, we like to get in the weeds on the methodological and cultural issues of actually conducting research with Hispanics. In their latest research-nerd article, Pew discusses the difference between non-probability or opt-in/online panel surveys and traditional probability-based polls. One of the issues with online panels raised by Pew and other researchers is that they do not necessarily represent the universe of people in the target accurately. That’s because the respondents aren’t selected at random.  They self-select by opting into a panel to participate in surveys in exchange for incentives or prizes. So the sample represents only the type of people who like to participate in online panels, which according to one study make up only 6% of the population.  For Hispanic research, this is an even bigger problem that requires much more adjustment in the methodology and sample design to correct.

Online Panels are More Affordable

Online panels are very popular in market research because the cost per complete is much lower than traditional methods. It allows clients to reach a larger sample size for much cheaper than phone or in-person data collection. That’s because the respondents are already part of a larger database that panel providers manage and these providers, in essence, “rent” the respondents out to research companies seeking to get their opinion.  Respondents are recruited into these panels using a variety of methods and they earn points or incentives the more surveys they participate in.  The provider needs to do a good job retaining these panelists so they stay involved for as long as possible, which offsets the cost of recruiting them in the first place.  Turnover is a big issue. 

What about Hispanic panelists?

The largest panel providers have a harder time recruiting a representative mix of Hispanic respondents to begin with; that’s because you have to actually go out of your way to recruit the entire range of Hispanics across geographic, nation of origin, language usage and acculturation. And you need enough respondents to represent all the subgroups well in sufficient numbers across regions and acculturation levels.  Then, you need to keep them engaged and participating in surveys to avoid turnover of those respondents within the various subgroups. 

Needless to say, it is complicated and expensive to recruit Hispanics this way, so the biggest panel providers choose not to do it (or don’t have the cultural expertise to do it correctly). They typically recruit a mainstream audience and a subset of that ends up being Hispanic and it falls out however it may.  In order to correct for the ultimately non-representative sample of Hispanics they obtain, the clients rely on weighting and other tricks to try to balance the sample that was recruited, using techniques meant for the mainstream audience.  Weighting can help but if it is a poorly recruited sample, weighting will only magnify idiosyncrasies present in the sample that aren’t present in the general population. For example, Hispanic online panelists under-represent Mexicans, blue-collar workers, Spanish-dominant and foreign-born Hispanics.  They over-represent female, tech-savvy, East Coast, Caribbean, South American and US born, English-dominant Hispanics.  You can use weighting to try to correct for these issues, but if the sample is idiosyncratic to begin with, weighting could magnify quirks in the sample.

The culture of panelists

Delving deeper, panel providers use various methods to recruit respondents for their panels and these methods have an inherent cultural bias to start. Sometimes recruiters use ads placed in online media or they use referral programs to acquire new panelists.  Recruiters for online panels typically don’t recruit in Spanish.  Each one of these methods needs to be culturally-attuned to attract the broad range of Hispanic respondents, much in the way an advertiser has to create a culturally-relevant message to attract a broad range of Hispanics.

Unless panel providers are specifically using recruitment methods that target and appeal to Hispanics across acculturation ranges and the various subgroups, they are likely to get Hispanics that are more similar to general market targets than they are to other Hispanics (e.g. they are more acculturated, have similar attitudes, etc.). Different cultures respond differently to various rewards/incentive scenarios which impact if and how they respond to the various appeals to become panelists.

Maintaining a stable panel also requires cultural competence

To keep respondents from quitting the panel too soon, panel providers use various techniques to keep them engaged.  Needless to say, providers must pick engagement methods which are relevant to the culture.  A points-based system might be less appealing to Spanish-dominant Hispanics than say a raffle-based system; raffles, games and awards being cultural constructs with differing appeal based on where you grew up or patterns picked up from your parents. Engagement also impacts retention. If you aren’t able to keep the broad range of Hispanics engaged (because perhaps your panel management program is in English or is poorly constructed), then the higher turnover will eliminate respondents who might be less acculturated and the composition of the Hispanics in the panel shifts over time.

But some panels do have Spanish-dominant respondents, right?

The decidedly smaller subset of Spanish-speaking respondents that some panel providers tend to have are often atypical, non-representative Spanish speakers. And, in addition, there typically aren’t enough in the sample to reach “feasibility” which means online panels often do not have enough respondents at the incidence levels needed to achieve the number of completes desired across markets, acculturation levels and other breaks.  Even combining multiple sample providers, it is difficult to get feasibility because online panelists tend to be members of multiple panels (as many as 80% are participants in multiple panels). So, after de-duplicating, the number of available respondents is still often not enough.  There are some panels that do a better job of recruiting and retaining Hispanics, but even then, the numbers are so small that adding any additional targeting whittles down the universe of respondents where it is often impossible to get feasibility across the various variables. All this poses the biggest problem with the Spanish-dominant portion of the sample.

And, Spanish-dominant panelists are atypical to begin with.

Even the Spanish-dominant respondents that are available in the panels tend to be atypical in important ways to non-panelists. While these online panelists may speak Spanish more than English and might be considered “less acculturated” in some ways, they tend to be more tech-savvy (they are participating in online panels after all), more educated, more white collar and also exhibit regional and country-of-origin skews that make their responses different compared to the average Spanish-speaking, Spanish-dominant Hispanics. Some of those variables can be corrected for by implementing quotas (of say Mexican-origin Hispanics, etc.) but we’ve seen that attitudes and behaviors are still different from offline Spanish-dominant Hispanics, even with the smaller panels that specialize in Hispanics.  In the general market, the online panelists are close enough, from a cultural standpoint, to their non-panelist counterparts that their responses are generally representative of the universe overall.  But with Spanish-dominant Hispanics, the online panelists often look, act and behave quite differently from the universe of Hispanics overall which can give clients a false sense of security as to the findings as they pertain to Hispanics.

How do mainstream researchers typically use panels to poll Hispanics?

When it comes to Hispanic consumers, marketers typically look at them as part of a larger sample (a broader, mainstream target).  The most common method of collecting Hispanic data is from a mainstream (online) survey by setting a quota of Hispanics out of the main sample.  These researchers invite people of all ethnicities to the survey and they stop collecting surveys when they hit the requisite number of their various subgroups including Hispanics.  They typically do not impose sub-quotas on country-of-origin, language usage, geographic distribution, years in the US or self-identity, so the Hispanics who do end up taking the survey end up being more acculturated or exhibit skews that are not representative of Hispanics overall. Recruiting them from an online panel, compounds this dynamic even further.

Then, in the analysis phase, the researchers make broad conclusions about Hispanics as represented by the subset of that sample that happened to be Hispanic even though demographically, behaviorally and psychographically they may be very different from the overall universe of Hispanics. This is compounded by the fact that many mainstream researchers simple do not have a very deep grounding in this cohort and so they report more shallow findings and are unable to detect any idiosyncrasies that are due to the methodology versus true cultural insights.

So, what’s the best way to sample Hispanics?

Well, there is no perfect way of sampling consumers in general.  All methodologies represent trade-offs.  The key is to weigh the trade-offs accordingly. Obviously, it is important to find panels that do the best job of recruiting and retaining Hispanics panelists. Even if there is availability of sample with these panels, we like to obtain a portion of the less acculturated sample offline to ensure we’re not only getting tech-savvy, panel-oriented respondents.

Cultural Edge uses a hybrid technique with carefully-designed quotas across acculturation levels, country of origin, language usage and self-identity.  We call it WideNet™ sampling.  We combine the more affordable online panelists with a well-constructed mobile CAPI intercept methodology focusing on obtaining those Hispanics that aren’t well-represented in the panels.  Mobile CAPI involves a team of mobile interviewers intercepting respondents at various sampling points throughout a given market with a tablet or laptop to allow them to self-administer the online survey in person. If they need help, the interviewer can assist them.  This ensures that we’re not only capturing tech-savvy, well-educated, online panel-oriented Hispanics but the wider range of Hispanics across the socioeconomic spectrum.  This is not a convenience-based sample.  We specify in advance how many respondents will be collected via each data collection mode and what acculturation levels we want to obtain via mobile CAPI and via the online panel. We structure quotas based on available Census data on country of origin, language usage and other dynamics.  Other researchers who use mixed mode data collection tend to use a convenience-based sampling method: they collect as much sample as possible online and collect the remaining respondents via intercept which is less expensive but it puts too much weight on the online respondents.  It is also suboptimal because it is very hard to replicate the same mix accurately when doing multiple waves of research for comparability across waves.

What happens if you sample Hispanics incorrectly?

A good example of the flaws inherent in incorrectly sampling Hispanics, is the controversial exit poll conducted by Edison Research after the 2016 election that stated that 29% of Hispanics voted for Trump – higher than even McCain or Romney. The exit poll was conducted in toss-up districts that were not representative of Hispanics overall; they tended to be communities that did not have a representative mix of Hispanics, were more affluent and more English-dominant. This poll, under-sampled voters that were Spanish-speaking and over-sampled more assimilated / acculturated Hispanics who live in lower Hispanic density districts. Not only that, but the survey went beyond “self identification” and asked those who identified as non-Hispanics if they had Hispanic ancestry which creates a new definition of Hispanic that includes people who do not even consider themselves Hispanic. The true number of Trump support among Hispanics is difficult to obtain, but other probabilistic polls (via phone) using Census-based weighting suggest a much lower number.

In summary

So, if you see a study that makes extraordinary claims about Hispanics, make sure to look at the methodology. If the survey was conducted in English or only via online panels, that data could be questionable. Ask about the data collection method, whether online only or intercept, whether they used bilingual interviewers and culturally-correct Spanish-language adaptation of the research instrument.  Ask about weighting but also about what the acculturation, educational level, regional (East, West, South, Midwest), country of origin and language usage breakdowns were.  If these metrics don’t match Census data, then the data could be suspect. But most importantly, ask them if they have decades of experience gaining insights from Hispanic and cross-cultural audiences because regardless of the methodology used, if they don’t have a deeper understanding of the target, the insights will ring shallow and the findings will be superficial.