Using Facebook to generate sampling frames in the Philippines

The Philippines was hit hard by the COVID-19 pandemic. The government responded with one of the longest and strictest lockdowns in the world, placing an enormous economic burden on the millions of poor households in this densely-populated archipelago, on top of the public health crisis. Unfortunately, Filipino policymakers have so far been hindered by a conspicuous dearth of reliable data and evidence, preventing them from making rapid, informed decisions. The nation is now preparing to face many difficult decisions and tradeoffs over the coming year or more of mitigation and recovery. It is critical that the government has a quick, reliable, and efficient way of generating insights into the needs and experiences of its population.

One of the key obstacles for evidence generation has been the inaccessibility of reliable, representative, and up-to-date sampling frames (population lists used to choose people to survey) with phone numbers for surveying. In the past, surveying often involved face-to-face household visits, which is no longer an option in the current pandemic. Many government household lists are not comprehensive and may lack phone numbers even when they do exist. Many people change phone numbers frequently due to the deactivation of SIM cards, causing new sampling frames for phone surveys to expire quickly. Finally, many government contact lists have limited demographic information to allow for targeted sampling. These limitations make it difficult to generate reliable sampling frames especially with contact information for people in low-income communities — the most critical segment for COVID-19 response efforts.

These limitations make it difficult to generate reliable sampling frames especially with contact information for people in low-income communities — the most critical segment for COVID-19 response efforts.

Without effective sampling frames, efforts to generate data through rapid surveys are hampered. Surveys are either forgone completely or are limited to opt-in online surveys that require an internet connection. This can result in highly biased samples that don’t reach poorer population segments and therefore generate limited — or misleading — insights.

In response to this problem, IDinsight has begun to experiment with using targeted Facebook ads to generate sampling frames for phone surveys, with remarkable success. The Philippines is the ideal place to test Facebook to create sampling frames because nearly half of adult Filipinos (99 per cent of online Filipinos) have a Facebook account, due to the social media platform offering free access (without requiring a data plan). Many poor Filipinos who cannot otherwise afford to spend money on internet plans use Facebook as their sole access to the web. In urban areas — currently the most critical for COVID-19 response — internet signal is fairly reliable and Facebook use nearly ubiquitous. Furthermore, Filipino internet users are the most active internet and social media users in the world, spending nearly 10 hours each day on average online (45 per cent more than the global average), nearly 4 of which are on social media.

We piloted this methodology in Parañaque, a diverse city of more than 700,000 in Metro Manila. Using Facebook’s ad campaign platform, we were able to micro-target for location, age, and gender to balance against the 2019 Labor Force Survey (a representative sample of households in the Philippines). We were even able to target based on market segments as a proxy for socioeconomic class to try to gather respondents from poorer households. In our ad, we offered a chance to earn ~$1 of prepaid phone balance by taking a 20-minute phone survey, which likely further skewed our sample towards lower-income households for whom $1 was compelling. After about four days and roughly $240 spent, we had a sampling frame of 1,566 unique respondents, from which we were able to quickly reach a balanced sample of ~600 households. The process was incredibly fast, efficient and inexpensive. In fact, we collected over a thousand numbers in a day, and the majority of this time and money was spent collecting numbers for people ages 46 and older, who are likely less active on Facebook.

This sample certainly cannot be interpreted as statistically representative of Parañaque residents: our survey only captured the experience of respondents who were active on Facebook and responded to the ad, were reachable by phone, and were (at least in part) incentivized by $1 of prepaid phone credit. However, our households were remarkably similar to the population of Parañaque in several demographic characteristics we asked for: neighbourhood, age, and gender. Furthermore, the average household size matched with that of Metropolitan Manila, and the prevalence of pre-existing conditions matched closely with that of the Philippines population. The exceptions — education and income — were intentionally skewed towards lower socioeconomic classes. We feel confident that, although imperfect, this methodology allowed us to describe our sample, meaningfully compare them to the broader population, and generate rigorous evidence rapidly made available for decision-makers.

Since some of our key findings (which we’ll share in a follow-up post later this week) were about low levels of knowledge about COVID-19 prevention and treatment, we also believe that these findings are policy-relevant to the broader Parañaque population: we expect our sample to be better informed than others with comparable education and socioeconomic status since they’re online more. If their knowledge is low, we should expect others’ would be too.

We believe this methodology shows a lot of potential, particularly in the Philippines. However, we intend to apply a more rigorous methodology to compare and test its reliability. Though large government contact lists remain unreliable and inaccessible for now, we anticipate that through our local government partners we can obtain smaller-scale, more reliable sampling frames in one or two locations. We are currently seeking funding partners to validate survey results based on Facebook-generated sampling frames by potentially using local government contact lists to serve as alternative sampling frames. We would run the same survey based on both sampling frames and compare results to answer the question: Are survey results from Facebook sampling frames similar to those that we would get using traditional sampling frame methodologies?

While this validation could never manufacture representativeness in our non-representative Facebook sampling frames, it would lend greater confidence to the evidence generated through this novel methodology. If we saw that the results between a truly representative sample and this sampling approach were similar in general, or for particular subgroups (for example, young people or highly active internet users), then we would feel confident about using the Facebook sampling frame approach to draw broader conclusions about those groups in the future.

To see our survey’s findings and recommendations, you can read our policy brief here.

Please contact meg.battle@idinsight.org if you’re interested in partnering with us for future research with this methodology.