The Politics of Global Data Reporting: A Triangulated Solution for Estimating Modern Slavery

Over the past four years, the Walk Free Foundation has been developing a Global Slavery Index to estimate the level of modern slavery in different countries around the world. We define modern slavery as situations of exploitation that a person cannot refuse or leave because of threats, violence, coercion, abuse of power, or deception.

Any attempt to measure social statistics is fraught with issues of measurement bias and measurement error. When the indicator is as politically sensitive as the Global Slavery Index, it is even more important to address these challenges rigorously and transparently.

This article discusses the main challenges—often ignored or underplayed—in collecting and estimating social statistics and describes the approach taken by the Walk Free Foundation to minimize these problems.

Measurement Error and Measurement Bias in Social Statistics

All social statistics are prone to a degree of measurement error and bias, often to a far greater extent than is recognized. Take Gross Domestic Product (GDP), one of the most basic and central measures of national economic activity, used in countless statistical analyses as both a key dependent and independent variable, and handily compiled in a range of frequently used databases, including the World Bank’s World Development Indicators.

Rarely are these estimates of GDP questioned, yet we have very good reason to believe that they are subject to quite substantial error and bias. By definition in economic theory, GDP should be exactly the same as Gross Domestic Income (GDI): They are two different ways of measuring the same underlying level of economic activity. Yet, even in highly developed countries with sophisticated statistical agencies, there is still almost always a significant difference between the two estimates. One, or more likely both, must be wrong.

These differences matter. Andrew Chang and Philip Li (2015) conducted a meta-analysis that replicated academic studies that used published GDP data, replacing GDP with GDI. They found that simply switching these two different estimates of the same economic concept produced qualitatively different findings in about one in eight of the studies they replicated.

Likewise, GDP estimates typically suffer from measurement bias: aspects of economic activity that are routinely excluded from measurement. Illegal economic activity falls into this category, and it can be far from trivial. The UK Office for National Statistics estimates that including sex work and recreational drugs in GDP estimates would add 0.7% to GDP. In Italy, estimates of the Mafia’s economic activity can add as much as 7% to the country’s GDP.

As feminist economists have consistently pointed out, estimating the economic value of unpaid household work—still a highly gendered form of economic activity in virtually every country—can have an even more-drastic effect. In 1997, the Australian Bureau of Statistics estimated that unpaid household work had an economic value of over A$200 billion—equivalent to over 45 percent of measured GDP.

Other examples of measurement error and bias in social statistics abound. When working with household surveys in developing countries, for instance, economists typically prefer measures of household consumption to household income, since the latter is prone to under-reporting by households involved in the informal economy or who are nervous of the tax system.

This has clear implications for analysis. For instance, inequality estimates based on consumption instead of income data are likely to systematically underestimate the level of inequality, because the difference between income and consumption (savings) is greater the higher up the income distribution.

The Politics of Social Statistics

The preceding discussion of the shortcomings of existing economic and social statistics is important, because it is noteworthy that these shortcomings typically receive less attention in discussions of established measures than when new measures are promoted.

When the United Nations Development Programme proposed its new Human Development Index (HDI) in 1990, for instance, the development economist Mark McGillivray attacked it as “yet another redundant composite development indicator” that conveyed no more or better information than GDP per capita. Thirty years later, the World Bank economist Martin Ravallion echoed this, with a broader critique of “mashup indicators” of development.

What these criticisms miss is that all social statistics are—like it or not—political. That social statistics can be, and sometimes are, manipulated for political advantage is beyond doubt. China regularly stands accused of manipulating its national economic data. Statistical manipulation to improve the government’s apparent budget position was one of the main charges in the 2016 impeachment of Brazil’s president Dilma Rousseff. In Rwanda, Paul Kagame was accused of manipulating poverty statistics to win a third presidential term.

Even in the absence of such direct political manipulations, all social statistics are fundamentally political in their design, construction, and collection. What better affirmation could there be of the feminist mantra that the personal is political than evidence that up to one-third of economic activity in developed countries takes the form of unpaid household work, still predominantly undertaken by women, often on top of paid and counted economic activity?

Likewise, the fundamentally political nature of social statistics is why the Human Development Index has not only survived the kind of critique that McGillivray and Ravallion have levied; it has flourished. In his contribution to the intellectual history of the UN, Sir Richard Jolly argued that one of its biggest contributions had been the ideas it has promoted rather than any concrete programs it has undertaken itself.

The HDI is one such example. It was, and remains, highly correlated with GDP, but its use has contributed to a change in the focus of international development from wealth generation to poverty eradication.

Meta-Data to the Rescue? Reliability Indicators

Thus far, we have discussed two interlinked characteristics of social statistics: data reliability (measurement error and bias) and the political nature of social statistics. The link between these two issues is both micro- and macro-level.

At the micro-level, national governments and statistical agencies can manipulate individual data points to produce a more favourable outcome. At the macro-level, the design and collection of social statistics often has systemic political consequences.

Given these concerns, attention has increasingly turned to meta-data solutions that attempt to provide further indicators of the reliability of such social statistics. The Data for Sustainable Development Thematic Working Group of the Sustainable Development Solutions Network, along with the Global Partnership for Sustainable Development, is at the forefront of highlighting the need for better data in international development work. Some private data analysis companies are also advocating the value of mining Big Data for useful proxies to vulnerability conditions.

The World Bank produces an annual Statistical Capacity Indicator (SCI) that provides a composite score as a total evaluation of each country’s “ability to collect, analyze, and disseminate high-quality data about its population and economy.” The composite score is based on an assessment of the methodology, data sources, and periodicity of published data from each country based on 25 criteria. A weighted average across these three dimensions is calculated to place each score on a scale from 0 to 100.

Li Cai and Yangyong Zhu provide a blueprint for an universal, two-layer data quality standard for assessment, which includes many of the critical concerns of potential data sources (see Figure 1). Through this framework, the World Bank SCI goes a long way toward capturing data quality, but is by no means comprehensive. In particular, few of its indicators address the issues of credibility and integrity of data that are key for intensely political indicators such as the Global Slavery Index (GSI).

Figure 1. Cai and Zhu’s Data Quality Framework.

Country Experts to the Rescue? Expert Surveys

An alternative approach to providing indicators of intensely political issues that are likely to be subject to manipulation, precisely in those countries where they are prevalent, is to use country experts to provide systematized qualitative assessments that can be transformed into a quantitative index.

Probably the best-known and longest-established example of this approach is Transparency International’s Corruption Perceptions Index (CPI). The CPI ranks countries on their level of perceived public sector corruption with a scale from 0 to 100, based on a combination of business surveys and expert assessment. Using Cai and Zhu’s framework (2015), we could see this as an attempt to prioritize credibility and integrity over more-technical statistical issues, such as accuracy and completeness.

The CPI has been subject to a barrage of criticisms. Some of these relate not to the index itself, but to the way it is used. Both media outlet and scholarly analyses sometimes use the index uncritically as an objective measure of the actual level of corruption in both the public and private sector.

More fundamentally, however, the index has been criticized for its reliance on the perceptions of a relatively small number of elite observers. This has the potential to simply reinforce and replicate stereotypes and widespread (but potentially wrong) assumptions.

Another problem with the use of country experts, expressed in Cai and Zhu’s framework, is the issue of timeliness. The CPI again provides a useful example of this. In the 2015 CPI, Brazil dropped five points and seven places, largely because of the Petrobras scandal, which was another factor in Rousseff’s impeachment. Yet the alleged corruption in Petrobras took place long before 2015—it was the revelation of these accusations that led to the drop in Brazil’s score.

Likewise, we are willing to bet that Malaysia dropped six places in the 2016 rankings in the wake of the 1MDB scandal, even though much of the purported corruption happened five years previously.

Country experts, then, can provide credibility and integrity in social indicators, but at the cost of timeliness and accuracy.

A Triangulated Approach: Estimating the Prevalence of Modern Slavery

In constructing an index to measure the prevalence of modern slavery, the Walk Free Foundation has had to confront these issues head on. Any measurement of slavery is going to be intensely political and face the same kind of challenges and trade-offs between different aspects of data quality that confront the Corruption Perceptions Index.

In addressing these challenges, we have developed a triangulated approach that seeks to make use of the benefits of different types of data while minimizing their shortcomings (see Figure 2). The GSI draws on two main sources of data: tailor-made surveys that attempt to estimate levels of modern slavery directly, and national vulnerability indicators, drawn from a range of national sources, to supplement the survey data and provide the basis for reasonable extrapolations.

Figure 2. GSI Triangulated Framework.

Of course, no measure will be perfect. This is the first step in our approach: to recognize and be transparent about the shortcomings and challenges of our methodology.

The GSI employs multiple vulnerability factors assessed for their relevance to human security theory and vulnerability to trafficking and exploitation. These vulnerability factors are used to group countries together statistically with similar vulnerability profiles for extrapolation based on our known nationally representative survey data.

However, in an effort to minimize the same shortcomings faced by common measures such as the GDP and CPI, in 2018, we will also use the World Bank’s Statistical Capacity Indicators (SCI) values as an external validation tool for our vulnerability measures. SCI will allow us to assess the vulnerability rankings of each country and to determine whether we may want to further interrogate any outliers that also appear to have low confidence in terms of reliability from the SCI data.

Once these outliers have been determined systematically, we will turn to our expert consultations to help us to determine the most-accurate possible adjustments for each case.

This proposed method of data triangulation presents a transparent and critical approach to the aggregated data employed in our vulnerability modeling. It will also help to ensure that our data are as robust and reliable as possible as we create vulnerability country profiles and groupings for extrapolation.

The GSI is committed to transparency and critical reflection. This triangulated approach to assessing the prevalence of modern slavery around the world is a necessary adjustment to operationalize the available data with the most-robust methods. All data may be political, but all methods need not necessarily be so tainted.

Our triangulated data approach in the 2018 GSI aims to address existing challenges in our reliance on reported global data and to move toward a more empirically justified and transparent method for addressing outliers where the national data does not reflect the reality of modern slavery in these critical countries. In countries such as Uzbekistan, where state-imposed forced cotton picking evades measurement, and North Korea, where state-imposed forced labor is similarly difficult to measure, this triangulated approach will allow the GSI to approach resolution in the measurement of the vulnerability systematically in these cases.

Conclusion

While all human development and human rights data may be potentially flawed, simply ignoring its existing value is not an alternative when the lives and freedom of millions of vulnerable people are at stake. A recurring theme in this work is “do not let the perfect be the enemy of the good.” Ultimately, we cannot propose or inform effective policy, have an impact on programs, or add to the limited empirical knowledge base about human trafficking if we cannot count or estimate those who are most vulnerable to this crime.

The future of vulnerability modeling and statistical extrapolation in human trafficking will lead us to better understand where we may find victims and which conditions trigger the greatest increase or decrease in victimization. However imperfect global data on human rights and development may be, it remains our best shot at helping those who are enslaved today. Our efforts to triangulate and validate these data will only improve their value.

Further Reading

Allen, K. 2014. Accounting for drugs and prostitution to help push UK economy up by £65 bn. The Guardian.

Alvarez-Cuadrado, F., and Vilalta, M. I2012. Income Inequality and Saving. St. Louis, MO: Washington University.

Australian Bureau of Statistics. 1997. Unpaid Work and the Australian Economy. ABS Catalogue No. 5240.0 ISBN: 0 642 54298 8.

Cai, L., and Zhu, Y. 2015. The Challenges of Data Quality and Data Quality Assessment in the Big Data Era. Data Science Journal 14, p. 2.

Khan, M. 2016. The truth behind China’s manipulated economic numbers. Telegraph UK.

Li, P., and Chang, A.C. 2015. Measurement Error in Macroeconomic Data and Economics Research: Data Revisions, Gross Domestic Product, and Gross Domestic Income (2015-11-10). FEDS Working Paper No. 2015-102.

McGillivray, M. 1991. The Human Development Index: Yet Another Redundant Composite Development Indicator? World Development 19(10), pp. 1,461–1,468.

Ravallion, M. 2011. Mashup Indices of Development. Oxford University Press and International Bank for Reconstruction and Development/World Bank. doi: 10.1093/wbro/lkr009.

Squires, N. 2012. Mafia is Italy’s Biggest Business. Telegraph UK.

United Nations Development Programme (UNDP). 1990. Human Development Report 1990: Concept and Measurement of Human Development.

Von Schirach, P. 2016. The Impact of False and Misleading Economic Data. Global Policy Institute.

World Bank. Statistical Capacity Indicator.

About the Authors

Davina P. Durgana, PhD is senior statistician and report co-author of the Global Slavery Index of the Walk Free Foundation. She was recently named the 2016 Statistical Advocate of the Year by the American Statistical Association and as a Forbes Top 30 Under 30 in Science in 2017 for her work on statistical modeling, human security theory, and human trafficking. She applies analytical models to understanding vulnerability, risk, and prevalence of human trafficking domestically and internationally.

Graham K. Brown, PhD is a professor of international development and head of Social Sciences at the University of Western Australia. After completing his PhD at the University of Nottingham, UK, he worked in the University of Oxford Department of International Development for five years before moving to the University of Bath, UK. He has been a visiting fellow at Stanford University and the National University of Singapore, and holds research associate status with the University of Oxford, Leuven University, and the University of Auckland. Brown’s research is concerned with the relationship between inequality, identity, and political mobilization, with an empirical focus on Southeast Asia.

Back to Top

Tagged as: , , , , , ,

Leave a Response

Please note: comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.