{"id":14539,"date":"2023-05-30T15:42:53","date_gmt":"2023-05-30T13:42:53","guid":{"rendered":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/faut-il-croire-aux-evaluations-dimpact-des-politiques-economiques%e2%80%89\/"},"modified":"2023-09-08T11:00:25","modified_gmt":"2023-09-08T09:00:25","slug":"should-we-believe-in-economic-impact-assessments","status":"publish","type":"post","link":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/should-we-believe-in-economic-impact-assessments\/?lang=en","title":{"rendered":"Should We Believe in Economic Impact Assessments?"},"content":{"rendered":"<p>How can we ensure, as rigorously as possible, that a public policy has the desired economic effects? <a href=\"https:\/\/sites.google.com\/site\/clementdechaisemartin\/\">Cl\u00e9ment de\u00a0Chaisemartin<\/a>, a researcher in the Department of Economics, focuses on this fundamental question in his \u2018Really Credible\u2019 project<span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_1');\" onkeypress=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_1');\" ><sup id=\"footnote_plugin_tooltip_14539_1_1\" class=\"footnote_plugin_tooltip_text\">(1)<\/sup><\/a><span id=\"footnote_plugin_tooltip_text_14539_1_1\" class=\"footnote_tooltip\"><a href=\"https:\/\/www.sciencespo.fr\/department-economics\/en\/news\/c-de-chaisemartin-awarded-erc-cog-grant.html\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Completing the revolution: Enhancing the reality, the principles, and the impact of economics&#8217; credibility revolution<\/em><\/a><\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_14539_1_1').tooltip({ tip: '#footnote_plugin_tooltip_text_14539_1_1', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script>, which has received support from the European Research Council highly selective ERC Grants programme. Interview.<\/p>\n<h5>A common way of conducting economic research is to use observed data to study the causal links between economic variables. This is known as empirical economics. What is the advantage of this approach?<\/h5>\n<p>Cl\u00e9ment de Chaisemartin: Economics is a discipline that has for a long time primarily drawn on theoretical models. However, models based on different assumptions predict different, and even opposite cause-and-effect relationships between economic variables. For example, a neoclassical economist would argue that raising the minimum wage will lead companies to use robots rather than workers, thus reducing employment. A neo-Keynesian economist would posit the opposite: raising the minimum wage can increase employment by stimulating demand or by increasing worker motivation and hence productivity. Who\u2019s right? It\u2019s hard to know without data! Empirical economics allows us to determine which of these divergent mechanisms is accurate.<\/p>\n<h5>Despite its promise, this approach has been challenged, notably by the American economist Edward Leamer, who has made a name for himself by dismissing it as biased and therefore not very credible. Why this criticism?<\/h5>\n<p>Cl\u00e9ment de Chaisemartin: Empirical economics has been the subject of all kinds of criticism, including that of Leamer<span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_2');\" onkeypress=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_2');\" ><sup id=\"footnote_plugin_tooltip_14539_1_2\" class=\"footnote_plugin_tooltip_text\">(2)<\/sup><\/a><span id=\"footnote_plugin_tooltip_text_14539_1_2\" class=\"footnote_tooltip\">Edward L. Leamer,\u00a0 <a href=\"https:\/\/www.jstor.org\/stable\/1803924\" target=\"_blank\" rel=\"noopener noreferrer\">Let&#8217;s Take the Con out of Econometrics<\/a>,<i> The American Economic Review<\/i>, Vol. 73, No. 1 (Mar., 1983), pp. 31-43<\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_14539_1_2').tooltip({ tip: '#footnote_plugin_tooltip_text_14539_1_2', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script>. Unpacking these critiques requires understanding the fundamental problem in attempting to measure the effect of one variable on another. In my example, the effect of an increase in the minimum wage on employment at a given time and in a given place is the difference between the actual level of employment and the <i>counterfactual <\/i>level of employment that would have been observed in the same place and at the same time had the minimum wage not increased. By definition, this counterfactual level of employment is not directly observed: we do not directly observe the level of employment that would have prevailed in a region that chose to increase its minimum wage if it had made the opposite choice. Thus, the empirical economist evaluating a public policy will have to <i>reconstruct <\/i>the counterfactual situation that would have been observed without the policy. And unlike an architect, who can give free rein to subjectivity when developing a building, it is preferable for an applied economist not to give free rein to creativity when developing a counterfactual. Otherwise, the assessed effect of the economic policy will depend on methodological choices, and another economist making different choices could come to a different conclusion. Study results are sometimes sensitive to subjective researchers\u2019 choices. This has undermined the credibility of empirical economics.<\/p>\n<h5>In response to this criticism, economists have sought to strengthen their econometric models and develop new methods in experimental economics, enabling the creation of \u2018reproducible\u2019 models. Many claim this addresses the credibility issue, but you don\u2019t think it\u2019s enough. What do you think is needed to close the gap?<\/h5>\n<p>Cl\u00e9ment de Chaisemartin: Economists have been formally tackling the identification of causal links between economic variables since the 1980s. A toolbox of standardised techniques based on often plausible and testable hypotheses gradually took shape, notably under the aegis of Nobel Prize winners <a href=\"https:\/\/en.wikipedia.org\/wiki\/Joshua_Angrist\" target=\"_blank\" rel=\"noopener noreferrer\">Joshua Angrist<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Guido_Imbens\" target=\"_blank\" rel=\"noopener noreferrer\">Guido Imbens<\/a>, and came to be known as the \u2018credibility revolution\u2019. But there was still one technique that, until recently, escaped this critical scrutiny of the valuation methods used by economists: two-way linear fixed-effects regressions\u2009\u2013\u2009the awful name alone is suspect.<br \/>\nIn a paper with my co-author <a href=\"https:\/\/faculty.crest.fr\/xdhaultfoeuille\/\">Xavier D\u2019Haultfoeuille<\/a>, we showed that this technique, frequently used in studies based on natural experiments to evaluate the effect of policies, is only reliable if the effect of the evaluated policy is temporally and spatially identical. This assumption is often implausible. For example, the effect of a minimum wage on employment is unlikely to be the same in advanced economies, where few jobs can easily be automated, as in less advanced economies. Of concern is that a quarter of the most cited articles recently published in the <i>American Economic Review<\/i>, one of the most prestigious journals in economics, uses two-way fixed-effects regressions. In other words, much of the recent research in applied economics is still based on fragile assumptions, and therefore cannot be considered credible. The \u2018credibility revolution\u2019 is far from over!<br \/>\nThe goal of my \u2018Really Credible\u2019 project is to propose evaluation methods that remain valid even if the effect of the policy being evaluated varies over time and space, as is often the case. Next, I want to develop computer programmes that researchers can easily use to apply the methods I propose. Finally, I will use these programmes to revisit a large number of articles that have used two-way fixed-effects regressions and use my method instead. This will allow me to see whether the conclusions that researchers obtained with two-way fixed-effects regressions withstand the scrutiny of comparison with the results obtained using my method.<br \/>\nThis is important because the two-way fixed-effects regressions have been used to answer pressing questions, such as the effect of rising Chinese exports on employment in the USA and Europe, or the effect of rising temperatures on agricultural yields and population health. It is crucial to see whether the answers to these questions change with more robust methods.<\/p>\n<h5>Is your ultimate goal to ensure that decision-makers and the general public have a better understanding of economic research methods, and therefore more confidence in them?<\/h5>\n<p>Cl\u00e9ment de Chaisemartin : Joshua Angrist<span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_3');\" onkeypress=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_3');\" ><sup id=\"footnote_plugin_tooltip_14539_1_3\" class=\"footnote_plugin_tooltip_text\">(3)<\/sup><\/a><span id=\"footnote_plugin_tooltip_text_14539_1_3\" class=\"footnote_tooltip\">Joshua Angrist is Professor of Economics at the Massachusetts Institute of Technology (MIT) and co-winner of the 2021 Nobel Prize in Economics with Guido Imbens and David Card. He specialises in labor economics and the economics of education.<\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_14539_1_3').tooltip({ tip: '#footnote_plugin_tooltip_text_14539_1_3', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script> once told me that a good criterion for knowing whether my research was good was that I should be able to explain it to my grandparents. The credibility revolution embraces the idea that research should be accessible to all and transparent. So it\u2019s possible that one of the benefits of this revolution I\u2019m partaking in is to make economic research understandable and credible to as many people as possible, potentially enabling it to foster more consensus on the pros and cons of different policy options. The way impact assessments shape the beliefs of citizens and public decision-makers is an interesting empirical question that is understudied. For example, an article<span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_4');\" onkeypress=\"footnote_moveToReference_14539_1('footnote_plugin_reference_14539_1_4');\" ><sup id=\"footnote_plugin_tooltip_14539_1_4\" class=\"footnote_plugin_tooltip_text\">(4)<\/sup><\/a><span id=\"footnote_plugin_tooltip_text_14539_1_4\" class=\"footnote_tooltip\">Sultan Mehmood, Shaheen Naseer and Daniel L. Chen \u201c<a href=\"https:\/\/users.nber.org\/~dlchen\/papers\/Training_Policy_Makers_in_Econometrics.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Training Policymakers in Econometrics<\/a>\u201d, working paper.<\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_14539_1_4').tooltip({ tip: '#footnote_plugin_tooltip_text_14539_1_4', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script> showed that training Indian public decision-makers in modern public policy evaluation methods affected their beliefs and practices. It would be interesting to conduct such an experiment with Sciences Po students or graduates! I\u2019d also love to carry out an experiment of this type with audiences much further removed from institutions (anti-vaxxers, conspiracy theorists, etc.). It\u2019s paradoxical that, at a time when the social sciences are making great strides in producing robust research on public policy impacts, our societies are becoming increasingly polarised and clashing\u2009\u2013\u2009sometimes violently\u2009\u2013\u2009over choosing between different public policy options. Better dissemination of research findings and education in evaluation methods might help build consensus.<\/p>\n<p>Interview by H\u00e9l\u00e8ne Naudet (Communications Department) and Melissa Mundell (Economics Department)<\/p>\n<pre>University Professor <a href=\"https:\/\/sites.google.com\/site\/clementdechaisemartin\/\">Cl\u00e9ment de\u00a0Chaisemartin<\/a> joined the Department of Economics in 2021 after conducting research and teaching at the University of California, Santa Barbara. An econometrician, he specialises in public policy evaluation methods. He has also taken part in impact assessments of public education policies in France (Internats d\u2019excellence) and abroad. He is also a Research Affiliate of the Abdul Latif Jameel Poverty Action Lab (J-PAL), created by the Department of Economics at the Massachusetts Institute of Technology.<\/pre>\n<p>&nbsp;<\/p>\n<div class=\"alert alert_warning\"><div class=\"alert_icon\"><i class=\"icon-lamp\"><\/i><\/div><div class=\"alert_wrapper\">One of the fundamental missions of the European Research Council (ERC) is to fund innovative research in Europe. ERC grants are awarded on a competitive basis to projects led by junior and senior researchers working in Europe. The only selection criterion is scientific excellence. ERC Consolidator Grants are designed to help mid-career researchers consolidate their own teams and carry out innovative projects across all academic disciplines, like Cl\u00e9ment de Chaisemartin. Since its creation in 2009, the permanent teaching and research staff in Sciences Po&#8217;s Economics Department have distinguished themselves by winning no fewer than <a href=\"https:\/\/www.sciencespo.fr\/department-economics\/en\/content\/erc-projects.html\">nineteen ERC grants<\/a>.<\/div><a href=\"#\" class=\"close\"><i class=\"icon-cancel\"><\/i><\/a><\/div>\n\n<div class=\"speaker-mute footnotes_reference_container\"> <div class=\"footnote_container_prepare\"><p><span role=\"button\" tabindex=\"0\" class=\"footnote_reference_container_label pointer\" onclick=\"footnote_expand_collapse_reference_container_14539_1();\">Notes<\/span><span role=\"button\" tabindex=\"0\" class=\"footnote_reference_container_collapse_button\" style=\"display: none;\" onclick=\"footnote_expand_collapse_reference_container_14539_1();\">[<a id=\"footnote_reference_container_collapse_button_14539_1\">+<\/a>]<\/span><\/p><\/div> <div id=\"footnote_references_container_14539_1\" style=\"\"><table class=\"footnotes_table footnote-reference-container\"><caption class=\"accessibility\">Notes<\/caption> <tbody> \r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_14539_1('footnote_plugin_tooltip_14539_1_1');\"><a id=\"footnote_plugin_reference_14539_1_1\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8593;<\/span>1<\/a><\/th> <td class=\"footnote_plugin_text\"><a href=\"https:\/\/www.sciencespo.fr\/department-economics\/en\/news\/c-de-chaisemartin-awarded-erc-cog-grant.html\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Completing the revolution: Enhancing the reality, the principles, and the impact of economics&#8217; credibility revolution<\/em><\/a><\/td><\/tr>\r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_14539_1('footnote_plugin_tooltip_14539_1_2');\"><a id=\"footnote_plugin_reference_14539_1_2\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8593;<\/span>2<\/a><\/th> <td class=\"footnote_plugin_text\">Edward L. Leamer,\u00a0 <a href=\"https:\/\/www.jstor.org\/stable\/1803924\" target=\"_blank\" rel=\"noopener noreferrer\">Let&#8217;s Take the Con out of Econometrics<\/a>,<i> The American Economic Review<\/i>, Vol. 73, No. 1 (Mar., 1983), pp. 31-43<\/td><\/tr>\r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_14539_1('footnote_plugin_tooltip_14539_1_3');\"><a id=\"footnote_plugin_reference_14539_1_3\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8593;<\/span>3<\/a><\/th> <td class=\"footnote_plugin_text\">Joshua Angrist is Professor of Economics at the Massachusetts Institute of Technology (MIT) and co-winner of the 2021 Nobel Prize in Economics with Guido Imbens and David Card. He specialises in labor economics and the economics of education.<\/td><\/tr>\r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_14539_1('footnote_plugin_tooltip_14539_1_4');\"><a id=\"footnote_plugin_reference_14539_1_4\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8593;<\/span>4<\/a><\/th> <td class=\"footnote_plugin_text\">Sultan Mehmood, Shaheen Naseer and Daniel L. Chen \u201c<a href=\"https:\/\/users.nber.org\/~dlchen\/papers\/Training_Policy_Makers_in_Econometrics.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Training Policymakers in Econometrics<\/a>\u201d, working paper.<\/td><\/tr>\r\n\r\n <\/tbody> <\/table> <\/div><\/div><script type=\"text\/javascript\"> function footnote_expand_reference_container_14539_1() { jQuery('#footnote_references_container_14539_1').show(); jQuery('#footnote_reference_container_collapse_button_14539_1').text('\u2212'); } function footnote_collapse_reference_container_14539_1() { jQuery('#footnote_references_container_14539_1').hide(); jQuery('#footnote_reference_container_collapse_button_14539_1').text('+'); } function footnote_expand_collapse_reference_container_14539_1() { if (jQuery('#footnote_references_container_14539_1').is(':hidden')) { footnote_expand_reference_container_14539_1(); } else { footnote_collapse_reference_container_14539_1(); } } function footnote_moveToReference_14539_1(p_str_TargetID) { footnote_expand_reference_container_14539_1(); var l_obj_Target = jQuery('#' + p_str_TargetID); if (l_obj_Target.length) { jQuery( 'html, body' ).delay( 0 ); jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight * 0.2 }, 380); } } function footnote_moveToAnchor_14539_1(p_str_TargetID) { footnote_expand_reference_container_14539_1(); var l_obj_Target = jQuery('#' + p_str_TargetID); if (l_obj_Target.length) { jQuery( 'html, body' ).delay( 0 ); jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight * 0.2 }, 380); } }<\/script>","protected":false},"excerpt":{"rendered":"<p>How can we ensure, as rigorously as possible, that a public policy has the desired economic effects? Cl\u00e9ment de\u00a0Chaisemartin, a researcher in the Department of<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":14346,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[336,32,335],"tags":[78],"_links":{"self":[{"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/posts\/14539\/?lang=en"}],"collection":[{"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/posts\/?lang=en"}],"about":[{"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/types\/post\/?lang=en"}],"author":[{"embeddable":true,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/users\/3\/?lang=en"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/comments\/?lang=en&post=14539"}],"version-history":[{"count":3,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/posts\/14539\/revisions\/?lang=en"}],"predecessor-version":[{"id":14568,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/posts\/14539\/revisions\/14568\/?lang=en"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/media\/14346\/?lang=en"}],"wp:attachment":[{"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/media\/?lang=en&parent=14539"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/categories\/?lang=en&post=14539"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sciencespo.fr\/research\/cogito\/home\/wp-json\/wp\/v2\/tags\/?lang=en&post=14539"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}