GiveDirectly staff waiting to assign participants to the next free lottery station. ©IDinsight/Rico Bergemann
Imagine you are living in a community and an outside organization is offering to give everyone a one-off cash transfer worth 1,000 USD. No strings attached. You would likely wonder: ‘What’s the catch?’. Is this organization trustworthy? How are they going to give out the cash fairly? Will some people get more cash than others?
Now imagine you want to implement a cash-based program providing large one-off cash transfers to an entire refugee settlement with about 10,000 households. In this context, there is a history of tribal tensions and protracted displacement, where people have escaped from war and now live in a rural refugee settlement with few economic opportunities. How will you disburse so much money without causing inflation? How can you do it fairly without fueling tensions or causing conflict?
GiveDirectly is implementing such a project in Uganda and is working with IDinsight to evaluate its impact. Working through the questions and challenges outlined above, GiveDirectly and IDinsight have decided to run a public lottery with 10,000 households to provide a fair and transparent process to randomize the order in which recipients get enrolled into GiveDirectly’s program.
Public lotteries in the social sector seem fairly uncommon based on the low amount of publicly available knowledge on whether, when, and how to do them. In this blog, we share our experience of implementing a large-scale public lottery. We want to provide insights into why we chose this approach and the key considerations that shaped the lottery, as well as challenges we encountered and how we addressed them. We hope that other implementers and researchers may benefit from our experiences and to help widen the knowledge base on public lotteries in the social sector.
Since GiveDirectly will provide cash transfers to all 10,000 households living in the Kiryandongo refugee settlement, it decided to implement its project over about 24 months and divide the beneficiary population into 24 groups of roughly the same size.
This sequential approach has a number of advantages. Operationally, it is simpler for GiveDirectly to enrol households into the program over time rather than attempting to enrol all households at once. It ensures the settlement is not flooded with cash, reducing the risk of inflation. Finally (and most exciting for IDinsight!) it provides the opportunity to generate rigorous evidence on the effects of the cash transfers by allowing a ‘phased in’ randomized control trial to be conducted1.
Kiryandongo refugee settlement hosts more than 15 different ethnic groups, primarily from South Sudan. Some of the tribal tensions originating from the civil war in South Sudan remain a concern. Against this backdrop, GiveDirectly provides a cash transfer roughly equivalent to over a year of monthly World Food Program (WFP) cash distributions for an average-sized household2. This is a significant amount of money that may contribute to excitement about the project, but could also cause suspicion and jealousy.
It would have been easy and cheap for IDinsight to use desk-based approaches to randomly assign households into groups (e.g., using software like Stata). However, given the concerns about causing conflict, GiveDirectly and IDinsight (in consultation with community leaders and the United Nations Refugee Agency UNHCR) decided to use a public lottery. Using a lottery is more transparent. Because the recipient picks his or her own number rather than being assigned, he or she will be more likely to accept that it was ‘luck’ (e.g., random) that determined the outcome of the lottery and that there was no foul play.
To divide all recipients into 24 groups, they had to draw a number between 1 and 24. The number picked by the recipient would determine the timing of when they receive their transfer. To ensure those 24 groups are of similar size, we had to consider whether we should have a lottery with a fixed amount of 24 numbers for each draw. This is also known as a lottery ‘with replacement’ as each number that has been picked is returned into the lottery after each draw so that all participants have the same chance of drawing any of the 24 numbers. However, this could result in an imbalance in group sizes if certain numbers get picked more often than others, which in turn could have implications on the specifications of our study. Alternatively, each of the numbers picked could be excluded from the lottery until all 24 numbers have been drawn — then the lottery would restart with 24 numbers. This is also known as a lottery ‘without replacement’ and would result in perfectly equal-sized groups as each number can only be drawn a fixed number of times. However, this means that the chance of picking a certain number is determined by the draws of previous participants. One main purpose of the lottery was to ensure that respondents feel they have a fair chance to pick any of the 24 numbers. Imagine a situation where there is only one ball left and the number on this ball is 22. For this particular participant, the chance of picking number 1 (or any other number but 22) is zero. We were concerned about how participants would react to such a situation and whether this would undermine the purpose of conducting a public lottery. Therefore, we decided to use the ‘with replacement’ approach and were confident that a large number of participants would result in similar-sized groups.
It is evident that one cannot just show up with a bucket full of lottery numbers and start conducting a lottery with about 10,000 households. The confusion about what this is all about would be immense. GiveDirectly decided to hold community sensitization events throughout the settlement in the weeks before the actual lottery. These events included carefully explaining the purpose of the lottery and role-playing the lottery itself. It also gave ample opportunities for participants to ask questions. Ultimately, these events helped increase the turnout rate at the actual lottery and likely contributed to the overall efficiency as more people were aware of the proceedings.
When planning to implement this lottery, our main concerns were to balance efficiency, transparency, and fairness. GiveDirectly and IDinsight decided to run the lottery in parallel to the WFP food and cash distribution, which takes place for about six to eight days each month.
How to ensure a high turnout rate? Our priority was to ensure that the majority of future recipients would participate in the lottery, otherwise, what would be the point of conducting a public lottery in the first place? This concern was the primary reason for running the lottery during the WFP food and cash distribution as most refugees would be present during these days to collect their food or cash.
Who picks the lottery? We considered either working with community leaders to choose the lottery number for each household or letting participants pick their numbers themselves. From a fairness and transparency perspective, both approaches had their advantages. While the community-leader approach would also have been faster, the leaders themselves advised against this. GiveDirectly implements the program over two years. No participant wants to pick one of the later numbers, and letting participants draw for themselves ensured a sense of control and responsibility for the outcome of the draw.
How to correctly identify households? If people drew for themselves, we needed to identify them accurately. Running the lottery during the WFP food and cash distribution helped in that process, as only registered and verified households get to enter the distribution compound in the first place. This contributed to the improved speed of the lottery because it reduced the number of ineligible participants who would have otherwise needed to be cross-checked against the database of recipient households.
How would participants pick their lottery numbers? Considering the options for how to practically implement the lottery, it became clear that we should not just use simple paper snippets with the lottery numbers. Paper snippets would not mix very well, and would thus affect the randomization. So we decided to use ping pong balls and write the lottery number on them with permanent markers. This choice brought a new challenge though, as participants were more likely to be able to peek into the bucket containing the 24 ping pong balls. The team came up with a construction that used local kitenge material to cover the bucket preventing participants from sneaking a peek.
How to pilot a lottery? Basically, we could not pilot the lottery itself. Imagine the confusion if you set up a different lottery and then need to tell people that the results are irrelevant and only the next lottery counts. You really only have one try. What we could pilot, however, was the general set-up and process of the lottery. And so we decided to collect relevant information for GiveDirectly’s program implementation3 during the WFP food and cash distribution a month prior to the actual lottery. In addition to the data collected, we were thus able to generate insights into factors like ‘How many enumerators do we need to process people fast enough without creating backlogs?’ or ‘How can we utilize the space most efficiently to manage the crowds and queues?’ or ‘How can we make sure that no one can accidentally skip the lottery?’. Furthermore, the WFP food and cash distribution is a very routinized process — everyone is used to a very specific set of steps they need to go through in order to receive their food or cash. Suddenly introducing an additional tent that people need to stop by may have created confusion. Therefore, the pilot allowed people to get familiar with GiveDirectly’s presence at the food and cash distribution.
How to process up to 2250 participants a day? Unfortunately, participants do not show up at food and cash distribution in equal-sized groups. This leads to considerable fluctuations in terms of the number of people that we needed to process at any given time. On food and cash days, this meant ‘lotterizing’ up to 2250 participants in a single day. To be able to facilitate such a process without causing long lines that interfered with the WFP food and cash distribution, we ran up to 14 lotteries in parallel. Furthermore, the process itself needed to run like clockwork. We used staff for queue control and to direct participants to the next free lottery station in an assembly-line process. The explanation of what participants were required to do was concise, and we used a streamlined survey with minimal data entry to record the participants’ details, resulting in an average time per lottery of about two and a half minutes.
The decision about whether or not to conduct a public lottery should be well considered and informed. For anyone considering a public lottery, we recommend close consideration of the following three areas:
1) You only have one try, and it needs to work: Once a participant picks number 1, there would likely be considerable protest and damage to trust and public perception if you have to tell them to re-run the lottery. This leaves little to no room for mistakes.
2) The lottery needs to be as random (re: fair) as possible: For us, this meant using ping pong balls to mix well (e.g., instead of paper snippets) and ensuring that the lottery buckets are shaken after each draw so that participants do not keep drawing the same numbers from the top.
3) Community and community leader buy-in is key: The contextual factors in Kiryandongo create a very sensitive environment to implement a public lottery (and the cash program overall). It meant that participants needed to understand the purpose and process of the lottery well and have a sense of control and responsibility for the outcome of their draw determining the timing of their transfer.
Overall, the lottery was a success. From a study perspective, we obtained a fairly balanced distribution of cohort numbers. From an implementation perspective, we managed to run a lottery with about 10,000 households in only eight days, which is much less than the four weeks we originally anticipated. And from a participant management perspective, fewer people were peeved about the number they picked than we expected.
During our baseline data collection, we asked respondents whether they thought the lottery provided for a fair process to determine the order in which households will be enrolled into GiveDirectly’s program and whether they think it was the fairest approach possible. About 90% of study participants said that it was a fair process and 85% said that it was the fairest approach possible. Expectedly, there is a difference between those that picked later numbers and those that picked earlier numbers.4 However, most participants perceived ‘their luck’ as a fair way to determine the order of program enrollment and even those with later numbers turned out to be much more accepting of their pick than we had guessed.
If you have any questions or need further details on how we implemented this lottery, please feel free to reach out to us.
We support NGO partners to improve program design, streamline implementation, evaluate impact, and accelerate scale-up.
4 June 2020
24 March 2020
1 April 2021
23 June 2021