Jennifer Unangst

The landscape of data generation is changing rapidly. New analytic possibilities have emerged with the increasing availability of “organic data”—data that have not been designed specifically for research purposes, such as administrative records and social media data. While these types of data have their strengths, surveys continue to be an important source of information, particularly about populations that are not likely to be represented in other data types.

Survey data are also used to support far-reaching decisions, such as where to direct aid or which public-serving programs to continue. To this end, researchers must be careful in considering how design choices throughout the survey lifecycle can influence real-world impact. Failure to think about this carefully can lead to undesirable outcomes, such as data that cannot be used to address the intended research goal or results that lead to inappropriate or potentially harmful conclusions. When researchers face competing priorities in their studies, statisticians can advise or help avoid common pitfalls to preserve the underlying objective of “doing good.”

Statistics Without Borders (SWB) provides services for a number of nonprofit organizations that are using surveys to better understand the communities they serve. SWB volunteers have supported two projects that used surveys to engage with vulnerable or remote populations. Each project addressed a different point in the survey life cycle: one focused on the design phase before data collection, the other on the analysis phase after data collection had ended. This article discusses the two studies to show how nonprofit organizations are using surveys to make a positive difference, and how statisticians can support their missions.

These case studies also illustrate the types of statistical and logistical challenges commonly faced in surveys. Others planning to conduct surveys may find these examples instructive for their own work. Permission to share learnings from these projects was granted by each project partner for educational/training purposes.

Designing a Sample of Restavek Children: Trade-offs between Bias, Variance, and Cost

The first survey project was set in the Caribbean country of Haiti. The SWB team was asked to consult on the sample design for a nonprofit organization called Haiti Now. Since 2010, Haiti Now has sought to improve the lives of impoverished children by reducing barriers to education and increasing access to emotional support. Assistance provided by Haiti Now primarily targets Restaveks: children engaged in a type of domestic servitude.

The families of Restavek children are typically from poorer rural populations, and send the children to live with hosts who agree to provide food, shelter, and education in exchange for domestic labor.

As of 2014, Fafo and the Haitian Institute of Childhood estimated that there were 335,000 to 494,000 child domestic workers in Haiti—about 10% of the child population.

The primary objective of the forthcoming Haiti Now survey is to better understand the living and working conditions of Restavek children, along with their physical, mental, and psychological health. Most importantly, the survey will facilitate comparisons between Restavek and non-Restavek children, to inform relative risks to educational attainment and well-being. If possible, results from the survey will also be nationally representative and used to estimate prevalence of the Restavek population.

Given the desired comparisons, the survey will cover both Restavek and non-Restavek children. Because children represent a particularly vulnerable population in research, the interviewers will be trained to identify and respond to distress or discomfort during the survey. Before implementation, the study will also undergo review by an independent review board (IRB) in Haiti. An ethical review is an important step, since the objective of the research project is to benefit and “do no harm.”

To protect the intended research subjects and the integrity of the study, the IRB may suggest changes to the data collection protocol, survey questionnaire, and/or sampling approach.

Findings from the survey will be used to inform future organizational initiatives and engage stakeholders, so support can be provided where it is needed most.

At the start of the project, the team agreed that the optimal sample design would meet multiple objectives: (i) achieve a sufficient number of Restavek respondents, despite Restaveks being a relatively rarer group; (ii) limit selection biases that could arise through the sampling method; (iii) keep cost and data collection effort at a reasonable level; and (iv) address as many of the research questions as possible. These objectives were often in conflict with one another, and each design option represented some trade-off among them.

The crux of the challenge in designing the sample for this study was finding an efficient way of reaching Restaveks. Because Restaveks reside with host families, it seemed natural to use the household as a sampling unit. However, there is no easy way to determine which households have Restavek children.

It was quickly apparent that a multi-stage cluster design would be needed to keep costs reasonable. This approach selects a sample of geographic areas at the first stage to limit the number of locations to which data collectors are required to travel. At the next stage, households are selected from each sampled area. Finally, one child may be selected per household.

Clustering can reduce statistical precision, because respondents from the same cluster tend to be more similar to one another; however, clustered designs are almost always implemented for face-to-face surveys due to timeline and budget constraints. The clustered design provided some guardrails for the remaining details of sample selection, but a key question remained: Within the sampled areas, how should households be selected? The team discussed three options for household selection that represent compromises between bias, variance, and cost.

Enumeration—The “Pipe Dream” Sample: The optimal solution with respect to bias and variance is to go door-to-door before data collection to determine which households in the sampled areas contain Restaveks. Enumeration is the procedure of building a list of all households, and is often used for large-scale national survey programs, such as the USAID’s Demographic and Health Surveys. For the Haiti Now survey, household enumeration would produce a sampling frame of eligible households—households with children—with an indicator for which of those households specifically have Restaveks.

The availability of such a frame has several statistical benefits. First, drawing a random sample from a near-compete list would maximize survey coverage with little risk of selection bias, assuming that households report the presence of Restaveks accurately. With a pure probability-based foundation, this approach can also support the analysis goals: nationally representative comparisons between Restavek and non-Restavek children, and nationally representative prevalence estimates for Restaveks.

This approach can also be optimal for variance, because it provides the greatest degree of control over the allocation of the sample and, specifically, the number of Restaveks selected for the survey. Because the sampling frame would have an indicator of which households have Restaveks, households can be stratified explicitly into two mutually exclusive groups: those with Restaveks and those without. Explicit stratification permits the target number of completed surveys to be allocated optimally among Restavek and non-Restavek households to meet desired precision goals. Not surprisingly, the greatest drawback to this approach is that it is an incredibly expensive endeavor, requiring a significant amount of time and labor to implement.

Random Walk—The “Pseudo Probability” Sample: Another solution for finding Restavek children is to select a random sample of households (or a pseudo-random sample, if a frame of households does not exist) and to screen those households for the presence of Restavek children as part of the survey. Pseudo-probability sampling approaches (such as random walk, which is the focus) can be implemented without the need for an existing sampling frame.

Random walk procedures are akin to systematic random sampling—with some caveats. Although variety of procedures can be used for this process, the basic approach is to choose a starting point (ideally at random) in each sampled geographic area. Then, data collectors walk through the area, following a set of directional rules, and sample a predetermined number of households at a fixed interval along the route.

Compared to the enumeration approach, the pseudo-probability sample has an increased risk of selection bias. Bias can be introduced through non-random selection of the starting point, as well as through the interviewers’ discretion when following the rules that define the walk. Even with the increased risk of bias, though, random walk approaches are regularly used for nationally representative surveys in low- and middle-income countries, where complete and updated sampling frames are difficult to come by.

The obvious benefit of this approach is that it significantly reduces the cost of implementation, because it doesn’t require that all households in the area be enumerated and screened. However, because the presence of Restaveks, and of children in general, is not known before data collection, there is little control over the number of Restaveks who will be observed in the final sample. While the random walk approach could support prevalence estimates of Restaveks, comparisons between Restavek and non-Restavek children would probably be hindered by the difficulty of finding them and having them represented in the sample.

Nonprobability—The “Fit for Purpose” Sample: The last option that the team discussed was a nonprobability form of matched case-control sampling. With this method, local leaders and gatekeepers from each sampled area would be asked to point out households that have Restavek children. Once a Restavek child from each of these households is interviewed, the team would visit neighboring households to recruit non-Restavek children with similar age, sex, and socioeconomic profiles.

This approach is expected to be the most cost-efficient and could produce a larger number of Restavek respondents compared to the random walk approach. However, because households are not chosen at random, it has the greatest risk of selection bias. Restavek households known to community leaders may represent systematically different living conditions, either better or worse, than the households that leaders are not aware of. Consequently, while this type of design would support analyses aimed at understanding the relative risks to Restaveks versus non-Restaveks, results would be difficult to generalize to the broader populations at large and would not support estimating the prevalence of Restaveks in Haiti.

Ultimately, the lead investigator determines the decision of the final design. When statisticians are involved at the design stage of a study, their role is to advise about the trade-offs of various design options and inform collaborators of the capabilities and limitations of the resulting data. If all objectives cannot be met within the limiting factors of the study (e.g., timeline, budget, staffing), it may be necessary to reassess the objectives and redefine the absolute priority. Thus, the design can be an iterative process—a feedback loop—where statisticians and nonprofit collaborators balance bias, variance, and practical considerations.

Using Weights to Make Inferences from a Sample of Coastal Residents in Madagascar

The next example focuses on the coast of Madagascar, where a conservation-focused organization (not named because the data are still being anonymized for publication) works with coastal communities to support sustainable natural resources management and fishing practices. In 2018, this organization conducted a near-complete census of residents in 14 coastal villages in the Menabe region of Madagascar, an area that includes approximately 3,395 adults.

To evaluate the social impact of their work in the area, the organization designed a survey of literacy, fishing practices, sanitation, and hygiene. Using the census list as a sampling frame, they drew a two-stage sample of residents, and interviewers went to the selected villages and administered the survey. Informed consent was obtained from each respondent before beginning the interview.

One of the analytic goals of the survey was to make inference to the population of residents in the area for the key survey topics. SWB was asked to support the analysis by producing survey weights. Producing weights is often considered a less-creative endeavor than sample design, but serves a key function by supporting inference from the sample to a broader population; specifically, by reducing discrepancies between the characteristics of the sample and the intended population.

Such discrepancies can arise at multiple points throughout the study: The list from which the sample was drawn may have errors; the sample may have been drawn in a way that over- or under-represents certain characteristics in the population; and some sample members may choose not to respond to the survey. Left uncorrected, these sources of deviation can lead to inappropriate conclusions about the population of interest.

The relationship between sample selection and weights: In general, the first step in calculating survey weights for a probability-based sample is to create a weight that reflects the influence of the sample design. This “design weight” accounts for the probability of each unit being selected into the sample.

The design weight is required for each member of the selected sample, including non-respondents, and is intended to correct for any intentional (or unintentional) ways that the sampling method may cause deviations from the population structure.

In a multi-stage design, the design weight is calculated as the product of the inverse selection probabilities across all stages of sampling. The Madagascar survey is an example of this.

The sample for the survey was drawn using two stages. The first stage drew a sample of villages with probability proportional to their population sizes, so larger villages had a greater chance of being included in the sample. The second stage drew a stratified simple random sample of residents from the selected villages. The number of sampled residents in each village was allocated proportionally to the population size across the strata, so residents from the same village had an equal chance of being selected.

In this sample design, the design weight would be formed by multiplying the inverse selection probability for the village by the inverse selection probability for the resident. In this example, the design weight for a sample member can be interpreted as the number of individuals in the population represented by that person. This design weight supports inference to the intended population for the survey.

From sample selection to the field. After a sample has been drawn, fieldwork constraints can sometimes introduce deviations from the original sample design, and thus affect inferential integrity of the original design weights.

The Madagascar survey saw two examples of this. First, substitution was used to replace nonresponding residents, so the target number of completed surveys could be maintained—in other words, when sampled residents opted not to complete the survey, other residents were interviewed instead.

Substitution is not an uncommon practice for mitigating nonresponse. When implemented, it is preferred that replacements be identified randomly and before data collection, because interviewers may choose replacements who represent groups with a higher propensity to respond. This is primarily of concern when the response propensity is correlated with the survey outcomes, because a nonrandom substitution could introduce bias.

The Madagascar survey was fortunate in that survey participation was unusually high due to the tight-knit nature of the local community; in addition, when substitution occurred, the substitute was identified as the next randomly ordered person in the list. Consequently, substitution of non-respondents was not expected to be a significant risk of bias.

The second example of a design deviation occurred with a different application of the substitution method: Substitutions were also made when two members of the same household happened to be selected for the sample, a consideration that arose because some survey questions were about the household (e.g., source of drinking water). In this case, one of the household members was replaced by another resident from the village, generally someone of the same age, gender, and vicinity of the original sample member, in attempt to minimize bias.

Adjusting design weights to reduce risks of bias. These examples raise an interesting question of inference: If the inference supported by the design weights is altered by non-response or field-based modifications to the sample, how can that risk of bias be reduced and the original inferential goals be met? This leads to the final step in producing weights for a probability sample: adjustments to the design weights.

While the design weights can correct for the influence of the sampling method on the representativeness of the data, adjustments can correct for issues of representativeness that occur after the initial sample has been drawn. In the land of probability surveys, these deviations are often attributed to naturally occurring processes such as non-response or errors in the sampling frame. In reality, they can also occur when practical considerations necessitate modifications to the original sample design.

Because the survey team in this example had conducted a recent census, reliable information was available on the population of residents in the region. This included the distribution of residents by age, gender, and educational attainment. Education was of particular interest because it is likely to correlate highly with many of the key survey outcomes, including literacy and fishing practices. This information could be used in attempting to mitigate biases in the sample by calibrating the design weights.

These adjustments were implemented by increasing or decreasing each sample member’s design weight in a way that guarantees representativeness with respect to the specified domains. For example, using age, gender, and education to adjust the design weights for the Madagascar survey ensures that the weighted distribution of the sample aligns with the population distribution with respect to these variables.

Weight adjustments are not a panacea for biases that can arise in survey data. However, weight adjustments offer some protection against bias and can enhance inferential value, especially when the variables used to inform the adjustments are highly correlated with the survey outcomes. (For a practical treatment of how to prepare survey weights, see Valliant and Dever’s Survey Weights: A Step-by-step Guide to Calculation.)

Final Comments

Despite the increasing availability of alternative data sources, surveys continue to be an important tool for influencing and measuring impact throughout the world. SWB members regularly interact with organizations that use surveys to make a positive difference. Although these groups face common constraints, such as a limited timeline, staffing, or budget, a sound statistical approach is crucial to maximizing the intended benefits to the population of interest.

The two case studies described here demonstrate inherent trade-offs with bias and variance that survey statisticians encounter in practice. Careful consideration of these trade-offs is paramount to ensure the data can be used for the public good.

It should be noted that while SWB volunteers and clients alike wish to produce the highest-quality data possible to support their philanthropic and research objectives, the reality is that all survey projects require some degree of compromise with respect to bias and variance.

The role of a statistician in supporting the mission of nonprofit collaborators evolves depending on the point in the survey life cycle where statisticians are asked to consult. When statisticians are involved at the design stage, they can advise on the trade-offs of various sample designs and inform clients of the resulting capabilities and limitations of their data. In contrast, when statisticians are involved after the data have already been collected, their focus shifts to identifying tools that can extend or strengthen the utility of the data.

In either case, the contributions of statisticians have a significant impact on the quality of the data, outcomes of the study, and communities that a study is intended to serve.

Further Reading

Bauer, J.J. 2014. Selection Errors of Random Route Samples. Sociological Methods & Research 43(3): 519–544.

Lohr, S.L. 1999. Sampling: Design and Analysis. Pacific Grove, CA: Duxbury Press.

Lunde, H., Liu, J., and Pederson, J. 2014. Child domestic workers in Haiti 2014: Tabulation Report. Fafo Research Foundation.

Valliant, R., and Dever, J.A. 2018. Survey Weights: A Step-by-step Guide to Calculation. College Station, TX: Stata Press.

About the Author

Jennifer Unangst is a research statistician at RTI International. In her eight years at RTI, she has contributed to more than 15 survey projects in the U.S. and abroad. She received her master’s degree in statistics from North Carolina State University and has been an active volunteer with the ASA’s outreach organization Statistics Without Borders for two years.

Back to Top