A case for ‘counterfactual’ thinking in policymaking

"Counterfactual" is an arcane term in our vocabulary, sometimes even for someone with higher education. It is the language of science, understood and used by scientists. If we are not scientists, why do we even need to bother about counterfactuals?

In the simplest possible terms, counterfactuals tell us what could have happened but did not. More specifically, it is about what could have happened in the absence of a phenomenon. For example, a counterfactual question would be to ask what would have been the unemployment rate in Bangladesh in July 2020 had Covid-19 not happened. On the surface, it seems to be an inconsequential question, almost silly. Why should we care about something that could have happened when it did not happen in the first place?

In reality, counterfactual is one of the most fundamental concepts of science, used in determining cause and effect. Does a certain medicine cure a certain disease—in other words, does it cause the remission? What would happen to the disease without the medicine? Does a foreign degree cause your income to go up—what would happen if you do not get it? Does a school-feeding programme cause children's grade to improve—what happens to their grade without the school meal? Does a shock like the Covid-19 outbreak cause agricultural productivity to fall—what would happen to productivity in the absence of the pandemic?

Some of the answers may seem obvious. For example, we see foreign degree holders get highly paid jobs all the time, so we can safely bet on its efficacy. If we find that this year's agricultural productivity has increased from that of the last year, we will have a reason to believe that Covid-19 did not have any adverse effect on productivity.

But if we think counterfactually, we will realise that the answers are not always so simple. For example, what if those who study abroad are also more likely to be rich and have powerful friends and relatives? How would we conclude that their highly paid job is the result of their foreign degree, not the other factors? How can you decide on studying abroad, costing an arm and a leg, if you are not reasonably sure about the value of your degree? Can you know the value unless you do it yourself?

Similarly, the impact of Covid-19 on agricultural productivity may be confounded by many other factors. For example, if the weather this year is more favourable than the weather last year, agricultural productivity may increase. It is also possible that this weather-induced increase is so high that it offsets any loss of productivity that Covid-19 may cause because of labour shortage or inability to purchase inputs due to the financial crisis created by the pandemic. In this case, if we compare the productivity of this year with that of the last year, we may believe that Covid-19 did not have any negative impact on productivity.

Finding out the effect of something that did not happen seems impossible, right? But scientists have developed many clever and sophisticated techniques, such as Randomised Control Trial (RCT), to "simulate" the counterfactual scenarios and determine causal relationships.

In a recent survey conducted by BIGD, Brac University, on the impact of Covid-19 on the Boro farmers in Bangladesh, we have found that Boro productivity this year is indeed better than that of the last year. Boro rice production in 2018-19 was 1,653 kilograms per acre (Bangladesh Bureau of Statistics, 2020). From our national survey, we estimate the per acre production to be 1,738 kilograms this year—about five percent higher than the production last year. But as it is clear by now, higher productivity this year does not mean that Covid-19 did not negatively affect productivity. That is why, applying our counterfactual thinking, we asked the farmers how much production they expected if Covid-19 did not break out. Then we asked them how much production they have got, or going to get (in case they have not yet harvested) in reality. Comparing these two estimates, we found that the approximate loss of production per acre caused by Covid-19 has been about seven percent.

To understand how Covid-19 affected Boro production, we asked the farmers about how the pandemic affected their production. They talked about labour shortages, delays in buying inputs (possibly because of disruption in transportation) and inability to purchase adequate inputs (possibly because their household income sources collapsed during the pandemic). These are all plausible reasons why productivity might have gone down. But if we did not think counterfactually, we would have simply compared the productivity this year with that of the last year and concluded that Covid-19 did not have any impact on Boro productivity! Of course, our calculation of the loss of productivity is imprecise as we had to rely on the mere estimate of the farmers about the counterfactual productivity. Yet, this estimate is better than mistakenly concluding that Covid-19 had no impact at all on productivity.

It is important to think counterfactually for better decision-making even if we are not scientists or researchers. Though in many cases, our good sense is good enough to make a decision. For example, we cannot let our children go hungry, and so nutritious school meal, especially in a poor region, is generally a good idea. But counterfactual thinking is crucial in too many cases of our personal and collective lives. Particularly, for policymakers, the implications of counterfactual thinking are astronomical. Whether to spend billions of dollars on a bridge, whether to invest in early childhood education, whether to tighten the monetary control—these are all counterfactual questions. With limited resources and too many problems to solve, policymakers always have to critically think about the what-if question.

In most cases, we cannot run scientific experiments to learn the cause and effect. But thinking counterfactually gives us a critical perspective that helps us make better decisions.

Nusrat Jahan is Head of Business Development and Knowledge Management at the Brac Institute of Governance and Development (BIGD), Brac University.