Impact evaluation, for GAFAM or for citizens ?

Impact evaluation, for GAFAM or for citizens?

GAFAM, as is well known, massively invests in artificial intelligence and quantum computing. But GAFAM also invests in another technology: impact evaluation. Between two images, which one will maximize the number of clicks on an Amazon product page? Between two possible organizations of an Instagram feed, which one will maximize the time we spend on the app? These are all impact evaluation questions, at the heart of GAFAM's economic model. 

Impact evaluation is a research discipline at the intersection of statistics and econometrics, to which we contribute. It is generally used to evaluate medical treatments but also public policies. For example, what is the impact of halving class sizes on student learning? Or what is the impact of increasing tariffs on the economies of the countries concerned? While the basic idea of impact evaluation is simple - comparing a "test" group that receives the policy or medication to a control group that does not - things get complicated when moving beyond the ideal framework of randomized clinical trials, and new evaluation methods are constantly emerging. 

This is why GAFAM is gradually investing in this field by recruiting researchers, as they have done in AI and quantum computing. Amazon employs 400 PhD economists, as many as the US Federal Reserve, and eight times more than Harvard's economics department. Many leading econometricians, including Nobel laureates, also work part-time for this company. Evaluation tools created by our team have been used by Amazon researchers to optimize advertising frequency on Alexa. Amazon is not an exception; all GAFAM companies recruit economists to optimize their products using tools created by academics. 

So far, nothing surprising: many areas see private sector use of public research. One might regret that this research indirectly helps GAFAM optimize products with well-established undesirable effects (increased mental health issues in teenagers due to overexposure to social media, misinformation). However, impact evaluation can be a tool to measure these negative effects and help public authorities regulate these companies. For example, one article showing the negative effect of social media on mental health uses a method created by our team. 

What's more surprising is that these evaluation tools are underutilized by those who fund them, namely governments. Evaluating the effects of halving class sizes or increasing tariffs on the economy may seem more complex than evaluating the effect of changing an ad image on click-through rates. In fact, evaluating the impact of public policies is entirely feasible: for example, halving class size increases student levels, and the 2018 tariffs imposed by the Trump administration increased import prices and penalized American consumers. A few governments in fact conduct extensive evaluations, either internally or with academics. However, many others, despite controlling vast data resources, conduct very few evaluations themselves and prevent researchers from accessing their data. And policymakers often propose populist measures (increasing tariffs, reducing VAT to boost consumption, increasing grade repetition in schools, to cite recent examples) that research has already shown do not work.

Another issue concerns the ownership of evaluation methods. While GAFAM has so far mainly consumed these methods, given the resources invested, it is likely that these companies will soon produce them themselves, and innovation will shift from universities to GAFAM, as has happened in AI and quantum computing. Consequently, the best evaluation methods would no longer be a common good made available to all by researchers in the form of open-source software, but would become the property of private companies. 

To address this issue, one could hope that public decision-makers invest more in producing evaluation methods. One of us benefits from a European Union grant that allows us to pay a small team of developers, and these resources enable us to produce much more professional software than when we wrote our programs ourselves in the rare free moments of otherwise busy schedules. The questions and bugs that our users still report to us almost daily remind us that this money is crucial. 

A more realistic solution would be to reorient the work of statisticians and econometricians. Currently, academic journals often prioritize the sophistication of methods over their degree of applicability. Similarly, writing software associated with methods is not highly valued. Producing fundamental research with uncertain applications is important, but our discipline devotes, in our view, too much researcher time to these researches. To encourage econometricians to produce the tools needed by economists evaluating public policies, journals could give more weight to the applicability of methods in their publication decisions, require researchers to propose software implementing their method, and developers of these software should systematically be co-authors of articles.

Specialised in economics of education and health economics, and more generally in the field of public policy evaluation, Clément de Chaisemartin is an econometrician focused on developing new, more robust and transparent public policy evaluation tools. 

Laureate of the 2019 AFSE-Malinvaud Prize with his co-author Xavier D’Haultfoeuille (CREST-ENSAE) for their paper Fuzzy Differences-in-Differences, published in the Review of Economic Studies, Clément was awarded an important European Research Council (ERC) Consolidator Grant in 2022 for his project REALLYCREDIBLE (read project description).

Learn more about Clément de Chaisemartin's work (personal website)