​MODULE 4​

​Welcome back.​
​In this course, we have been talking about behavioral economics, and we​
​have been discussing numerous ideas related to psychology and economics​
​and the intersection of the two.​
​But one of the thing that has become very clear as we have proceeded​
​along the course is how important experiments are to the evolution of​
​the ideas of behavioral economics.​
​We have been looking at numerous lab experiments, field experiments,​
​randomized controlled trials, and so on.​
​In this module, we are going to look in depth at this experimental method.​
​What does an experiment do?​
​What is an appropriate experiment?​
​What are the ethics concerned with conducting an experiment?​
​And finally, what is the econometrics behind experimental data analysis?​
​So let's start and deep dive into experimental methods.​
​First, I will give you a brief overview of the history of randomized​
​experiments, not only in economics, but also in other related fields.​
​We will start with the history of experiments in​
​clinical trials in medicine.​
​Of course, that is where randomized control trials in social sciences and​
​in economics came into being, that was the source of the inspiration.​
​So clinical trials in medicine can be, can be traced back to​
​the, the days of James Lind.​
​James Lind in 1774 wanted to understand if lemons cure scurvy, the dreaded disease​
​that was very prevalent among sailors.​
​If you remember those days, sailors used to take long journeys across the seas.​
​They did not have enough nutrition in our vitamins and scurvy, the​
​dreaded disease where gums would bleed and so on would result.​
​So James Lind was very interested in understanding whether lemons​
​can be used to increase the vitamin C content and therefore​
​decrease the incidence of scurvy.​
​Now he recruited eight sailors in all, just eight imagine.​
​So eight sailors were divided into two groups, four who were administered lemons,​
​and four who were in the control group.​
​So that was one of the earliest use of randomized trials in medicine.​
​Then of course, it took a long time for the field to evolve.​
​Ronald Fisher, the statistician proposed the explicit use of randomization​
​in experimental design for causal inference through two major books.​
​One of the book is known as Statistical Method for Research Workers and other​
​one the famous Design of Experiments.​
​In these two books, he made a case that randomized experiments be used​
​to make clean causal inference.​
​It's only 1948 that Austin B. Hill and others while wanting to​
​understand if streptomycin the antibiotic cures tuberculosis ended​
​up using randomized control trial.​
​And why did they use the randomized control trial?​
​Because there was a limited supply of streptomycin and they thought it​
​would be unethical not to conduct a trial because they wanted to​
​understand the efficacy of this drug.​
​ his paper led to a widespread use of randomized control trial in the​
T
​medical community, and today RCT is a most important tool that medical​
​community uses to understand efficacy of various treatment methods.​
​Around that time in the year 1954, Jonas Salk developed the polio​
​vaccine through numerous field trials.​
​He was interested in understanding if the Salk vaccine prevented for polio, and in​
​the process he developed and designed a double blind randomized control trial.​
​The result was stark.​
​The result showed that polio vaccine caused a reduction in the risk of polio.​
​And what he did was he used the Fisher's exact test.​
​Yes, it's the same Fisher that we talked about to understand​
​what the treatment effects were.​
​[Music]​

​ e​
W
​will now move on to the use of randomized experiments in social experiments.​
​What is the history of that?​
​One of the first use of randomized experiment in the​
​context of social experiments was done by the RAND Corporation.​
​They ran a study between 1971 and 1982 where they wanted to​
​understand the causal effect of healthcare insurance on healthcare.​
​The issue was very specific.​
​There was widespread free healthcare for a large number of participants in the US.​
​The concern was that these participants would overuse​
​the health insurance program.​
​So Rand Corporation through 7,700 participants with a age younger than​
​65, wanted to understand what different kinds of copay environment in, in​
​insurance would do to the insurance usage and through the spending.​
​So what they did was they developed three different treatment conditions.​
​One is the status quo free care, the second referred to three​
​different plans with varying degrees of copay or cost sharing.​
​And the third is a health maintenance organization plan or an HMO plan where​
​a patient could only access doctors in and around within a certain network.​
​Participants with varying degrees of cost sharing were found to have fewer physician​
​visits and fewer dental visits, fewer hospitalizations than those with free​
​care and less spending on health overall.​
​This had a remarkable effect on the cost, efficacy of the insurance program.​
​Finally, the cost sharing treatments also led to the reduction in the​
​use of services as cost sharing treatment groups opted for not​
​initiating the care in the first place.​
​Another famous randomized experiment that has been done in the context of social​
​experiments is the one that was done by Burtless and Hausman, the negative​
​income tax experiment in the year 1977.​
​President Nixon had proposed a series of social safety measures or social security​
​measures, and people were very interested in understanding whether these social​
​security measures led to a reduction in, in the effort people were putting in.​
​More specifically, this research wanted to answer if providing cash transfers​
​to poor reduced willingness to work.​
​ S and Canada tested this idea of a negative income tax or a​
U
​subsidy transfer, if you will, in a series of five social experiments.​
​What was the big takeaway?​
​The big takeaway was there was no effect on individual labour​
​supply of providing cash transfers.​
​However, this resulted in an increase in divorce rates and unintended consequence,​
​if you will josh Angrist, the Nobel Prize winner in economics, who has made seminal​
​contribution in identification and other econometric techniques has mentioned​
​that there were too many treatment arms and possibly the experiment that was​
​designed by Burtless and Hausman was underpowered to detect credible effects.​
​That's a little bit of history about rules of randomized experiments in​
​the context of social experiments.​
​But as you know, by now randomized experiments has had a rich checkout​
​history in lab experiments as well.​
​A lot of initial lab experiments were used in the context of​
​individual decision making.​
​Thurnstone 1931 was the first to use lab experiments to understand​
​individual decision making and behavior.​
​He wanted to do something very interesting.​
​He wanted to invite people and understand what the nature of​
​indifference is between pairs of goods.​
​These pairs of goods could sometimes be hats and shoes.​
​It could be coats and shoes, and it could also be hats and coats.​
​So he used the choice made by people to deduce the indifference​
​curves corresponding to the choice that was Thurnstone, 1931.​
​Mosteller and Nogee in 1951, conducted a lab experiment that measured in​
​a restricted manner, the value to individuals of additional money income.​
​Von Neumann Morganstein in the famous book, theory of Games and Economic​
​Behavior, 1944 proposed four different reasons why lab experiments could be used.​
​He said that lab experiment could be used to have participants participate​
​in a game with opportunities to take or refuse certain gambles or risks​
​entailing the use of real money.​
​B, lab experiments could be used to construct utility curve​
​corresponding to the behavior.​
​C, it could be used to make predictions from utility curves about the​
​future, individual behavior towards other and more complicated tasks.​
​And D, lab experiments could be used to test the predictions by​
​examining subsequent behavior towards more complex risks.​
​Lab experiments have also been used quite extensively to​
​understand social interactions.​
​We have come across many of these interactions, social​
​interactions in our earlier modules.​
​One of the early work that understands social interactions using Lab​
​Experiment is a work by Melvin Dresher and Merrill Flood, both of​
​them located at the Rand Corporation.​
​In 1950, they conducted a lab experiment, which.​
​Implemented a version of Prisoners Dilemma, and you can see the Prisoner's​
​Dilemma game, the version of the game that they used in front of you.​
​They played this game.​
​They got participants to play this game hundred times with repetition.​
​ s you already know, that in a prisoner's dilemma, the Nash​
A
​equilibrium is row2 and column1.​
​But the socially efficient outcome is given by the strategy row1, column2.​
​Now, this means that the socially efficient outcome is seldom reached​
​by rational agents who always tend to play the Nash equilibrium.​
​The Nash equilibrium, which gives lower welfare to people is attained​
​but the socially efficient outcome, which gives lower over higher​
​overall welfare is not attained.​
​The empirical results show that the Nash equilibrium is not really​
​reached by human participants or lab participants, but they don't play the​
​socially efficient strategy either.​
​So this departure from the rational prediction made people think, what​
​is it that predicts how people behave or what strategies people choose in​
​prisoners' dilemma type environments.​
​Another famous study from a historical point of view that deployed the use of​
​lab experiments is the one by Guth and others published in Journal of Economic​
​Behavior and Organization in 1982.​
​It gave birth to what is now known as the ultimatum game.​
​A strategic form game that tries to understand bargaining between​
​a proposer and a receiver.​
​And it is probably the second most extensively used lab experimental game in​
​economics after only the dictator game.​
​So here is a little bit of history about the use of randomized experiment in​
​medical science in the context of social experiments and in the context of lab​
​experiments using which much of behavioral economics insights have been derived.​
​We looked at the history of use of randomized experiments, not only in​
​economics, but also in medical science.​
​What I'm now going to do is I'm going to now give you an overview of how​
​experiments have been used in modern day research and a brief summary​
​of the typology of experiments.​
​What are the different types of experiments that have been used and what​
​are the different types of experiments that you can possibly use to understand​
​the nature of the reality around you?​
​But first, let me make a distinction between behavioral economics​
​and experimental economics.​
​Behavioral economics is a sub-discipline of economics where​
​experiment is more like a method.​
​It's like a tool using which you can study not only behavioral economics,​
​but also other branch of economics, psychology, and other disciplines.​
​So what is an experiment?​
​An experiment is a way through which you develop identical environments,​
​but change one thing at a time to measure causal inference.​
​What are the advantages of using experimental methods?​
​Well, there are two main advantages.​
​Experiments often allow you to control the data collection process very precisely.​
​As a result, you're able to capture variables of interest that are​
​consistent with psychological factors.​
​Typically, these psychological factors or other variables, such as these are​
​not really observe, are not really available in observational data.​
​The second big advantage of experiment is it allows us to​
c​ leanly establish causality.​
​Now, we will understand in the next lecture how an experiment helps​
​us to arrive at causality, but that's in the econometrics part.​
​Let me give you a quick overview of the type of experiments that​
​you will come across in your engagement with the subject.​
​There is, of course, the lab experiment, the artefactual field experiment, also​
​called the lab in the field experiment, and then there is online experiment,​
​which is gaining a lot of popularity.​
​The field experiment and the randomized control trials.​
​Besides, of course, the usual natural field experiment.​
​So let's start with lab experiments.​
​Where have lab experiments been used and where is it now being used?​
​Of course a big use of lab experiment historically in the context of economics​
​has been the market experiments.​
​These experiments have engaged in price discovery testing.​
​These experiments measure how quickly and accurately market prices converge​
​to known fundamental asset values.​
​So here is an example by Plott and Pogorelskiy 2017, in case you're​
​interested, feel free to dig deeper.​
​What I'll do in this module is I'll give you a series of references​
​so that you know how these lab experiments have been used, are​
​being used in the literature today.​
​The lab experiments have also been used quite extensively to understand​
​different aspects of market mechanisms, in particular different​
​aspects of auction mechanisms.​
​These experiments have tested, for example, different trading structures​
​such as double auctions, call markets, or posted bid markets to examine​
​how the auction design affects price efficiency and market performance.​
​Kagel and Levin 1993 is a seminal paper that tries to understand precisely this.​
​How are prices formed in double auction markets?​
​Now, in order to answer that question, they introduced 3​
​different types of auctions.​
​One is, of course the first price sealed bid auction or FPA.​
​The second is the second price sealed bid auction or SPA.​
​And finally, the third price sealed bid auctions or the TPA.​
​In an FPA, the bidders submit sealed bids.​
​The highest bidder wins and pays their bid amount.​
​As you know in SPA or the second price sealed bid auction, the bidder submits​
​sealed bids, the highest bidder wins, but pays the second highest bid.​
​The true valuation of the highest bidder is not really known in a​
​second price sealed bid auction.​
​In the third price sealed bid auction, a novel experimental format was​
​introduced where the highest bidder wins, but pays the third highest bid.​
​The outcomes in all these different treatments was of​
​course the bid value and revenue.​
​I'm sure you know about the revenue equivalence theorem.​
​So that was lab experiments in the context of market design.​
​Lab experiments have been extensively used in behavioral experiments as​
​you have been seeing throughout.​
​We have looked at heuristics and biases.​
​ e have looked at risk preference, time preference, and social preference,​
W
​and lab experiments have been extensively used to develop the key​
​ideas within each of these topics.​
​If you remember in heuristics and biases, we talked about the Linda problem.​
​How is the Linda problem solved?​
​Well, people are invited to a laboratory environment.​
​They are shown the description of Linda and asked to make a choice.​
​In the series of experiments that Kahneman and Tversky did​
​to develop prospect theory.​
​Participants were invited to a lab and they were asked to make​
​choices across different gambles.​
​Likewise, in time preference as well to develop the idea of present bias​
​discounting experiments have been used.​
​In social preferences, the use of laboratory experiments​
​have been quite extensive.​
​They have been used to study social interactions, measuring​
​altruism, measuring reciprocity, trust, understanding, deception,​
​and so on and so forth.​
​Here is an example of use of a lab experiment in the context of social​
​preference that is inspired by my own work published in Journal of Public Economics.​
​In this paper, I try to unpack the relationship between corruption on​
​one hand and trust on the other.​
​So what you see on the left is a graphical representation or a​
​skater plot of corruption and trust across the countries of the world.​
​What is very clear is there is a negative relationship from a cross-sectional​
​standpoint between corruption and trust, there is also a negative movement of​
​countries from a high trust low corruption coordinate to a low trust, high corruption​
​coordinate as is indicated by the direction of most of the arrows that you​
​see which points towards the southeast.​
​Now this is of course correlation.​
​How do we understand whether trust affects corruption or corruption affects trust?​
​I designed a lab experiment where we had three treatments.​
​One, a baseline treatment where participants would​
​simply play the trust game.​
​Second, treatment 1 where the participants would play first, a bribery game aimed​
​at detecting corruption behavior, and then a trust game and a next treatment​
​where participants would play not a bribery game, but a game that is​
​strategically equivalent or an ultimatum game, and then play the trust game.​
​What we were interested in knowing is how does trust behavior vary across​
​the 3 conditions in the baseline game, in the game where the trust​
​game is preceded by act of corruption or unethical behavior and a trust​
​came where it is preceded by and strategically identical ultimatum game.​
​We conducted a subsequent experiment as well, which was aimed at capturing​
​social norm associated with different kinds of behavior in these games.​
​So the point of this exercise is to help you understand the kind of problems where​
​you can use lab experiments creatively.​
​And as our paper showed that corruption led to a decaying social capital or​
​corruption led to decrease in social trust as measured by the trust game outcomes.​
​Here is of course, the most famous lab experimental game as I had mentioned.​
​So this is an ultimatum game, and this has motivated a whole range of lab-based​
​ xperiments and now online experiments.​
e
​In an ultimatum game, a proposal splits a pie, as you probably know between​
​herself and the responder, the share takes a value between zero and 10.​
​So proposer, as you can see on the, in the graph given on the right, the​
​proposer decides how much to share s between zero and 10, and the responder​
​decides whether to accept it or reject it.​
​If the responder accepts it, then the responder gets 10 minus S. If​
​the responder accepts it, then the responder gets S shared and the proposal​
​gets 10 minus S. If the responder rejects it, then the responder gets​
​a zero and proposer gets a zero.​
​So the so game perfect, Nash equilibrium predicts that the​
​proposer would pay a very small amount and the responder would accept.​
​However, a series of lab experiments conducted across the world shows that​
​the average offer is not very small.​
​In fact, the average offer is around $4 dollars or 40%.​
​If you look at it from a percentage point of view.​
​This literature also found that 40 to 50% of the proposer end​
​up doing a 50-50 split, which is contrary to the nash prediction.​
​And the offers were less than $2 dollars are typically rejected.​
​This is equivalent to rejection of any offer of around 20% or lower.​
​Now the question is why do respondents reject?​
​And the answer that these lab experiments have come up with is the​
​fact that people dislike unfair options.​
​Now if you look at it, this kind of analysis is not possible​
​using observational data.​
​You have to construct very specific, well-designed experiments that​
​will allow you to make inference about psychological motivations​
​behind certain kinds of behavior.​
​Now, one of the advantages and disadvantages of lab experiments, if I may​
​add, lab experiments used to be much more important in the past than it is today​
​in generating evidence in social science.​
​So what are the advantages?​
​Well, one major advantage is that you, as a researcher, you have total control​
​over the data collection process.​
​The identification is clean, and that means that you will be able to​
​make very precise causal inference.​
​The experimental treatment at the end of the day is implemented or it's​
​introduced by a researcher's manipulation.​
​This advantages meant that these lab experiments could be very​
​creatively used with small changes by experimental economists.​
​However, there are some major disadvantages of using lab experiments.​
​The typical subject pool in a lab experiments are the students,​
​students from universities, students from colleges, and so on.​
​It is important to note that in many of these lab experiments and in fact​
​in all of these lab experiments used in economics, the participants are​
​incentivized meaning that they're actually paid $4 dollar if the proposer​
​decides to share $6 with the receiver and the receiver accepts it, the​
​researcher actually hand out the money.​
​However, who is likely to be incentivized by these small amounts of money?​
​The economists figured out that the student subject pool​
i​n universities and colleges considered this as a good incentive.​
​As a result, a large number of these experiments started being conducted with​
​student subjects as the participants.​
​This, of course, had a consequence in terms of generalizability​
​and external validity.​
​The results were questioned.​
​Are these results really generalizable to the outer world?​
​Are these externally valid?​
​If a result holds in IIM Bangalore, does it also hold in IIM Ahmedabad.​
​These were questions that started getting raised.​
​[Music] As we noted that the lab experiments attracted​
​participants who were mainly college students or university students.​
​Now, these students were said to be WEIRD participants.​
​WEIRD standing for Western, Educated, Industrialized, Rich, and Democratic.​
​Not everybody belongs to that category in the world.​
​So there was a felt need to move these experiments outside of the​
​lab of US and Northern European universities to the field.​
​What you see here is a lab in the field experiment, also called an​
​artefactual field experiment, according to certain specific terminologies​
​developed by John List and Harrison.​
​So this lab was constructed in a Panchayat Bhawan in Bihar, as​
​you can see participants from nearby villages have come in.​
​They have a computer in front of them.​
​The computer is connected to a server, and they make choices through interactions​
​using the laptop that is in front of them.​
​These are constructed by introducing a lab in the field, and the participants​
​usually are non-standard subject pool.​
​Here are a couple of examples of lab in the field randomized experiments.​
​With some co-authors, I have investigated the role of affirmative​
​action policy on self-confidence and competitive, competitive choice.​
​The treatments varied the affirmative action policies on the basis of cost,​
​and we wanted to invite people from different caste categories and see​
​how their approach to decision making changed under different treatment​
​conditions where certain treatments would have affirmative action​
​policy and certain others would not.​
​Other famous studies include those that compare farmers before and after harvest,​
​using lab constructed in the field.​
​Could we have studied these questions using student participants?​
​Of course we have, could have.​
​But how generalizable would it be compared to creating the,​
​constructing the lab in the villages?​
​It is, it stands to reason that a lab in the field is more​
​generalizable than a lab experiment.​
​So the advantages of lab in the field experiment is greater generalizability​
​and a greater sense of control.​
​The disadvantages relative to large scale field experiments is that it is less​
​generalizable and less generalizable compared to large scale field experiments.​
​We will see what these field experiments are.​
​But at the same time, it is also more expensive than lab experiments.​
​If you have to set up a lab in the villages, you need to develop​
t​he paraphernalia that will allow you to successfully run that lab.​
​Here is another example.​
​Suppose you are interested in understanding the question.​
​Do teachers expectations about a student's ability differ depending on who the​
​student is, depending on the gender, the caste, the income levels, and so on.​
​How do we answer such questions?​
​Well, the only way to answer the question is to go to such schools and​
​compare the expectation of teachers of different caste categories with their​
​actual performances that will require us to set up a lab in these schools,​
​several schools, and make sure that we observe the protocols while comparing,​
​while eliciting the expectations of the teachers, and then we can compare these​
​expectations with the students' outcomes.​
​So we have been talking about different kinds of experiments.​
​So we covered lab experiments and give you an overview of what the​
​advantages and disadvantages of lab in the field experiments are.​
​We'll now move on to the next, very important and an emerging format that​
​is increasingly being used in economics.​
​It's an online experiment.​
​And this online experiment is being is something that you can conduct on​
​Amazon Mturk, Prolific and other such platforms, Qualtrics and TGM, where​
​there are a large number of participants waiting to respond to researchers.​
​In these platforms, you can answer very interesting questions such​
​as, what are the expectations about the future of the economy?​
​What is the role of narratives?​
​What are the subjective models of the economy among​
​people and experts and so on?​
​Increasingly, a large number of experiments are being conducted online.​
​Because of the enormous advantages that it has.​
​So for example, you can conduct these online experiments at a fraction​
​of the cost that it would take for you to conduct a lab experiment.​
​Importantly, these online experiments are usually responded to by​
​participants who are located in different parts of the country.​
​These are large scale data collection exercises that often are​
​representative of some of the key characteristics of the population.​
​It is important to note that being representative of some of the key​
​characteristics is very different from what a representative sample is.​
​So for example, you can say that you want 50% men and 50% women in your sample.​
​Now, this is more or less representative of the true population of the country.​
​What you can do is within the data collection platform that you​
​have constructed, you can create quotas of 50% men and quotas of 50%​
​women, and then do this experiment.​
​This also means that you have greater generalizability because​
​your survey is being filled out by people from different parts of India.​
​Your results are more generalized.​
​They're certainly more generalizable than the results that you would obtain​
​if you had conducted the experiment within the pressings of your university.​
​However, despite the fact that these quota based representativeness and large​
​scale nature of online experiments have made it very popular, there are some​
​disadvantages that we should be aware of.​
​It is not always clear what the credibility of the data is.​
I​ncreasingly, we get to hear that large language models and other​
​AI tools are being used to fill in survey responses when no human​
​participant is actually present.​
​Now that's a problem.​
​The other big problem is that you usually are restricted in terms of the amount​
​of time that you can give to an online participant to fill out the survey.​
​It is not really possible for an online participant to respond to a​
​survey that lasts for over one hour.​
​A maximum time limit of 20-30 minute is probably ideal, and that leads to lack​
​of control of the experimental condition within which the data has being collected.​
​Now, these quota based online experiments are very popular.​
​A very close second is a survey experiments.​
​A survey experiment is a special type of online experiment where a treatment​
​condition is embedded within a survey.​
​It's a very famous economist, Stefanie Stantcheva, based in Harvard University​
​along with her collaborators have attempted to answer very interesting​
​questions using these kind of large scale survey experiments.​
​She has attempted to answer, for example, how do people understand,​
​perceive and form their attitudes towards economic issues and policies?​
​This branch of research has attempted to understand, has attempted to​
​ask important questions such as how do people understand, perceive,​
​and form their attitude towards economic issues and policies?​
​What kind of policies are we talking about?​
​Let's say gains of trade, something that is very topical at the current moment.​
​With the increasing number of tariffs that are being imposed on countries​
​trade across the world, how do people view these gains and losses from trade?​
​What are their perceptions?​
​To what extent do the support trade policy.​
​While answering these questions, they create experimental conditions where​
​they manipulate the salience of gains and losses and examine how people's​
​attitude towards such policies change in different treatment conditions.​
​These experiments, the survey experiments, I mean are unincentivized and hence they​
​can be conducted at a very large scale.​
​There's a recent paper on Zero-sum Thinking, which again, Stefanie​
​Stantcheva, Nathan Nunn and others have co-authored employs 20,000 residents from​
​different parts of the United States.​
​Finally, in this typology of experiments that we are discussing,​
​let's discuss perhaps the most significant of all experimental types.​
​They can be broadly classified as field experiment and randomized control trials.​
​Now field experiments and randomized controlled trials are those types​
​of experiment, that test hypothesis where the outcome is observed in​
​naturally occurring environments.​
​So for example, students in schools, for example, patients in​
​hospitals, for example, laborers in factory conditions, right.​
​So they're not really being invited to a lab or a lab in the field or in front of​
​their computers, as is typically in an online experiment, but they're operating​
​in their natural state of affairs.​
​They're operating in their natural habitat.​
​The outcome is observed without them knowing that they're part of a​
​systematic, randomized experiment.​
I​t is important to know at this stage that how are field experiments​
​different from our cities?​
​Well, it may be a little bit confusing, but as far as I can tell,​
​there is one crucial distinction.​
​While there is no consensus on what should be called a field experiment and what​
​should be called a randomized controlled trial, it is fair to say that field​
​experiments test very precise theoretical predictions by testing hypothesis using a​
​land randomized experiment in the field.​
​RCTs do the same thing, but they're more directly linked to a policy.​
​Here are a couple of examples from our field experiments and one example from​
​a randomized control trial that will probably make this distinction clear.​
​John List and others, they test altruism and the role of social​
​pressure in charitable giving.​
​They propose that there are 2 competing motivations for giving.​
​One is a pure altruism and the other, the reason why people give​
​is that they dislike to say no.​
​Now, John List and others, they conduct a field experiment where the​
​outcome is the amount of money that is braised in charitable giving, and​
​they design two treatment conditions.​
​One control condition where there is a door-to-door campaign to raise fund.​
​Another is a treatment condition where there is a pre-announcement​
​that there will be a door-to-door campaign to raise funds.​
​What is the main outcome that they were interested in?​
​They were mainly interested in the likelihood of donation in the control​
​condition and in the treatment condition where people know that somebody will​
​come at their doorstep at some point.​
​So John List and others through this experiment shows that when people know​
​that there will be a knock at their door at a future point in time for donations,​
​they usually don't stay at home.​
​In other words, they show that the dislike to say no to somebody asking​
​for donation is a very important motivation of why people end up​
​giving in charitable environments.​
​A second, and a very famous example is the study that was conducted by Marianne​
​Bertrand and Sendhil Mullainathan.​
​They published a paper where they ask whether Emily and Greg are more​
​employable than Lakisha and Jamal.​
​This is something that you probably have already seen in​
​the context of discrimination.​
​Now, this is a field experiment that gave us a very important insight, and the​
​insight was that people from different racial backgrounds are invited for​
​interviews with different likelihood.​
​Now these kind of experiments, as I was mentioning, are slightly different​
​from randomized controlled trials, which are randomized experiments​
​where the hypothesis being tested has a very direct policy relevance.​
​So here is an example.​
​Suppose you are interested in understanding the effect of treated​
​bed nets on malaria and income and child mortality, et cetera.​
​Then the experiment designed to test that would be called a randomized​
​controlled trial, or an RCT.​
​Guess whose paper studied the effect of treated magnets on malaria?​
​ es, it is Michael Kremer and Ted Miguel.​
Y
​Of course, many of you know that Michael Kremer was awarded the Nobel Prize in​
​2019 for his contribution to development economics using the​
​methods of randomized control trials.​
​He was awarded the Nobel Prize, along with Abhijit Banerjee and Esther Duflo for​
​having made very similar contributions.​
​Now, what are the advantages of field experiments and RCTs the big advantage​
​is that these are generalizable and there is considerable external validity​
​for the findings of the results that arrive through field experiments or RCTs.​
​These are large scale RCTs.​
​RCTs are usually large scale experiments, and the results​
​obtained are observed outcomes in naturally occurring environment.​
​People are not really aware that they are being observed and therefore their actions​
​are not subject to some of the standard criticisms of lab environments, that there​
​is social desirability bias, there is Hawthorne effect, et cetera, et cetera.​
​Now, what is a disadvantage of field experiments in RCTs?​
​The biggest disadvantage is it's very expensive and logistically challenging.​
​As a young student, it would be very hard for you to pull off a randomized​
​control trial or a field experiment.​
​A second big disadvantage is that oftentimes because an a researcher​
​doesn't have the level of control that a researcher is in a lab​
​experiment has often the precise mechanism is not very clear.​
​Additional treatments are hard to implement because it is so expensive,​
​so people have to rely on survey-based measures to understand why a treatment​
​effect has emerged in the first place.​
​The advantages of a lab experiment on the other hand is that you​
​can conduct multiple treatments and therefore arrive at the mechanism​
​that is driving the results.​
​Here is a well-known example where RCT was successfully used to inform policy.​
​A fundamental issue in education, economics, in low income countries​
​is, how do we increase the learning levels of the students.​
​Now in order to understand how to increase learning levels of​
​students, Banerjee, Cole, Duflo and Linden evaluated 2 programs.​
​One is a Balsakhi program, another is a computer assisted learning program.​
​The Balsakhi program is one where young women were employed to teach children who​
​lag in learning how to perform better.​
​The computer assisted learning program, on the other hand, made use of​
​computers to improve numeracy skills.​
​So the researchers in this case, they conducted randomized controlled trials​
​where the treatment condition, where Balsakhi program was implemented.​
​Would be compared to the control condition, the treatment condition,​
​where a computer assisted learning program would be implemented was​
​compared to the control condition.​
​So how do these programs increase the learning levels is​
​the key question of interest.​
​Now, you can see how this question is different from the field experiment​
​that was discussed earlier.​
​There is a direct policy relevance of a Balsakhi program or a​
​computer assisted learning program.​
​In the sense that if the effects were promising, then this could​
​ e scaled up and made an effective policy by the government.​
b
​Beyond all these typologies, there are a whole range of reasons why experiments​
​could be used in a non-randomized fashion.​
​So here I provide you with some examples of how non-randomized​
​experiments were used.​
​Blouin and Mukand has a very interesting paper in the Journal of Political Economy​
​that was published in 2019 where they make a very clever use of experiments,​
​though in a non-randomized way.​
​They were interested in understanding what the role of the Rwandan​
​government's nation building programs were in terms of interpersonal​
​trust, social harmony and so on.​
​As many of you probably know, Rwanda suffered a genocide in the 1990s where​
​two major tribes, Hutus and the Tutsis fought mercilessly against each other.​
​So the Rwandan government wanted to introduce the radio based nation​
​building programs so that there could be greater harmony among people.​
​Some villages were exposed to the radio programs while others were not, and this​
​was to an extent, randomly determined by nature of the terrain of Rwanda.​
​What Blouin and Mukand did was they went to two different villages.​
​A village which got exposed to the radio program and a village that did not get​
​exposed to the radio program, and they measured the social trust among people​
​between these two sets of villages.​
​Now, this is not a randomized experiment, but here the researchers are using​
​the trust game, which you have come across as a tool to understand the​
​nature of the social interactions.​
​Anandi Mani and others have very interesting paper where they show​
​if that financial scarcity decreases cognitive bandwidth among people.​
​And this is where they go to farmers before Harvest and​
​Farmers other after harvest.​
​Notice there is no use of randomization in this design.​
​Here are a couple more examples where experiments have been​
​used in a non-randomized way to make very interesting points.​
​Uri Gneezey, John List and others they have a very interesting paper​
​where they show the gender difference in competition is actually a result​
​of nurture rather than nature.​
​In order to make that point, they compare behavioral differences in competition,​
​behavioral differences in self-confidence and risk across different societies,​
​societies that are matrilineal in nature.​
​And societies that are patriarchal in nature.​
​Now, these societies are very similar in terms of their wealth,​
​but they're different in terms of the role of women in the family.​
​And they showed that many of these differences in gender differences​
​in risk and gender differences in competition actually were​
​reversed in a matrilineal society.​
​Is there a randomization here?​
​No.​
​Finally, Linda Babcock and others show in their paper published in 2017​
​that there are gender differences in accepting and receiving requests for​
​tasks that are low promotable in nature.​
​While doing that, they measured the likelihood of volunteering for such​
​low promotability tasks across gender.​
​ e have seen this paper and the research related to this paper​
W
​in the earlier lecture on gender.​
​Now, in this research, is there a randomized experiment?​
​No.​
​But they do a very careful measurement exercise by running carefully calibrated​
​experiments that measure the likelihood of men and women to volunteer in​
​low promotability environments.​
​To summarize, while many of the times experiments are conducted with​
​randomization built into it, which allows us to make causal inference.​
​Here are some examples to show that randomization may not​
​always accompany an experiment.​
​These are experiments that are done to carefully measure outcomes​
​that are otherwise not measurable.​
​These exercises help us understand the deeper mechanisms of the changes​
​that are happening in our society.​
​Last but not the least, I would like to highlight the role of experimental​
​data to understand mechanisms.​
​And what comes to my mind is a very well-known paper by Gautam Rao,​
​who study the role of intermixing in private schools of Delhi.​
​Because of a court mandate, the schools of Delhi, the private schools​
​of Delhi, had to admit students from lower socioeconomic status background.​
​What Gotham Brown did was he compared students who were exposed to students​
​from lower socioeconomic backgrounds with those who were not exposed to​
​students from social lower socioeconomic backgrounds and used very clever and​
​very cleverly used dictator games and different kinds of other field experiments​
​to understand what exposure to the social other, in this case, what exposure to​
​students from poor backgrounds do two whole range of discriminated preferences.​
​So these kind of experiments are very useful to understand what is​
​going on in the society to, they're typically used to complement other​
​kinds of data exercises, as you may have seen in the earlier cases as well.​
​So experiments are primarily, primarily used in a randomized sense in​
​economics, but it doesn't always have to be used in a randomized sense in​
​economics.​
​Welcome back.​
​We have been talking about different kinds of experiments that economists​
​have been using to investigate market and other institutions in economics.​
​I've given you numerous references throughout this module with the​
​expectation that you will invest, investigate independently the​
​content of these research articles and deepen your understanding of how​
​experiments may be used in economics.​
​We'll now move on to the next section, which has got to do with​
​ethics of running experiments.​
​When we conduct an experiment, we essentially use human​
​beings as our subjects.​
​In biological sciences, many of the initial development of science​
​happens through experimentation on all kinds of lab animals.​
​But we economists, we deal with human beings, so we should be extremely​
​careful about how ethical, our scientific conduct is, and the next section​
​is going to talk more about that.​
​Why is this important?​
​ ell, this is important because there are numerous historical​
W
​violations of ethics in research.​
​I will go through a few of them, but again, I encourage you to look deeper in,​
​into these and other related violations.​
​The first one that comes to our mind is the Nazi experiment on​
​Twins, which Dr. Joseph Mengele conducted between 1943 and 45.​
​These experiments were con conducted in the Auschwitz concentration​
​camp under the instance of the third Reich, in these experiments,​
​about 1500 sets of twins were used.​
​They were primarily children, and they were subjected to medical procedures​
​of all kinds of different things.​
​The aim was to change the color of the eyes sometimes, sometimes​
​Dr. Joseph Mengele engaged in unnecessary amputations, sometimes​
​deliberate infection of disease was introduced, and all of these​
​in the pretext of genetic research.​
​When World War II ended and these information got unravelled, there​
​was a sense of horror that such a thing could have happened.​
​More recently, the infamous Stanford Prison Experiment was​
​conducted by Zimbardo in 1971.​
​He recruited 24 male students and assigned them to either a​
​prisoners or a prison guard.​
​This was kind of a mock prison, and in this prison, the students​
​would play the role of a prisoner and the role of a prison guard.​
​But thankfully, this experiment was stopped after six days.​
​It was realized that the guards or the students who were assigned the​
​role of guards had become sadistic and they were taking pleasure from​
​giving pain to the prisoners, and the prisoners were severely stressed.​
​Now, these kind of experiments cannot be conducted anymore.​
​But you can imagine the kind of horror that unravelled in Stanford University​
​when faculty members learned that this kind of an experiment was being​
​done by their faculty colleague.​
​The third study that comes to our mind is, again, a very infamous one​
​it's the Tuskegee Syphilis study conducted between 1932 and 1972.​
​About 600 African American men were made part of this study, and the​
​study was conducted in rural Alabama, out of these 600 African American​
​men, about 399 men had syphilis.​
​They were deliberately kept away from medicine, which was penicillin,​
​which by then had been invented.​
​It was invented in 1947, around 1947.​
​But these men were prevented from having access to penicillin when the​
​treatment of syphilis was already there.​
​Now, this happened over a 40 year span during which the civil rights​
​movement had already taken place.​
​To have done this with only African American men was quite astounding.​
​Finally, let's come to the Milgram Obedience Experiment.​
​Once again, a very infamous experiment that was conducted in 1961 and 1962.​
​Milgram was a Yale psychologist and he wanted to understand the effect​
​of authority on people's behavior.​
​His research question was motivated by the fact that in the first half of the 20th​
​century horrific crimes were conducted by ordinary men and women, and he wanted to​
​ nderstand why they ended up conducting those crimes, and more importantly, he​
u
​wanted to understand whether obeying authority or a higher power had anything​
​to do with why such horrific crimes were conducted by ordinary men and women.​
​So in order to understand this question,​
​Milgram asked his study participants to administer powerful electric shocks to​
​learners if they ever gave a wrong answer.​
​And these shocks could be up to 440 volt.​
​Now the question that he wanted to understand was under the influence​
​of authority, would people be willing to administer something as horrible​
​as a powerful electric shock?​
​To his surprise and to the surprise of many others, he​
​found that participants obeyed.​
​Even after hearing the screams of pain, and he showed that people were​
​capable of harming others if they were asked to do so by a higher authority.​
​Now these are some examples that show the historical violations of basic human​
​rights and human ethics that resulted in development of a set of protocols​
​that we are expected to observe today.​
​So what are the modern requirements?​
​First and foremost, an experimental participant must give a formal​
​informed consent before participation.​
​Now, you can check online for what an informed consent template looks like.​
​But it should clearly state the risk of participating in the study.​
​We will go over the content of the informed consent in greater​
​detail in the subsequent slides.​
​Secondly, and ethics training is mandatory for all experimenters.​
​Anybody who uses experiment for research must mandatorily undergo ethics training.​
​There are several online platforms that train people for how, on​
​how to conduct ethical research.​
​City program is one of them.​
​IIM Bangalore subscribes to city program and if you ever need to go in that​
​direction, feel free to reach out to the IRB at IIMB, if you want City Program​
​certification through IIM Bangalore.​
​Thirdly, the entire experimental design must be submitted to an ethics committee​
​or the institutional review board.​
​The institutional review board will investigate and review the entire​
​experimental protocol suggest changes to figure out, suggest changes, and​
​will give advices and suggestions about how to further minimize risk​
​and discomfort to human participants.​
​The key takeaway here is that the past violations have led to strict ethical​
​guidelines in our current times.​
​So what are the core ethical principles that determine research protocols?​
​Well, these core ethical principles were laid down in​
​the Belmont Report of the 70s.​
​The Belmont Report stands on three main pillars.​
​The first one is the respect for persons.​
​The second one is the beneficence and the third one is justice.​
​Let's go through each of them individually.​
​The first pillar suggests that there should be respect for persons who​
​are participating in the experiment.​
​Primary in this aspect is the fact that individual participants should​
​ e treated as an autonomous agents.​
b
​They should have the autonomy to say yes or no and to the decisions that they're​
​making in the context of the experiment.​
​Second, there should be protection for those with diminished autonomy,​
​and we will see what that means.​
​Third, there should be a right to make informed decisions, and this informed​
​decision must be reflected in the consent form that must be provided​
​to an individual participant at the very beginning of the research study.​
​The second key pillar of the Belmont Report is beneficence.​
​Beneficence fundamentally says that do not do any harm to a participant.​
​Another way of thinking about beneficence is that you want to maximize benefits​
​while at the same time minimize risks.​
​It's very important to understand that it's primarily the researcher who​
​will benefit from a research study.​
​A researcher will publish a paper that will lead to career advancements for​
​the scientific community there is much benefit to be had for the knowledge that​
​is produced as part of the research.​
​However, the risk often is born by the participants despite the fact that there​
​are some other forms of compensations that are often, that are sometimes given out.​
​Now beneficence means that you want to maximize these benefits, but at the​
​same time, keep in mind that the risk to the participants must be minimized.​
​And this risk benefit assessment is what needs to be independently evaluated​
​by an institutional review board.​
​The third key pillar of experimental ethics is justice.​
​What it says is there should be a fair distributions of​
​the benefits and the burdens.​
​There should be equal treatment of all participants, and there should be no​
​exploitation of vulnerable populations.​
​The Belmont Report, as I mentioned earlier was published in 1979 and forms​
​the foundation of modern research ethics.​
​As we have been discussing, the participants must have full informed​
​consent before they go on to respond to the questions in a research study.​
​In this informed consent, the following are the required elements.​
​The informed consent text must clearly write down the purpose of the research,​
​how long the research will last, what are the procedures involved in the research​
​process, the risks and discomforts that may emerge from the research​
​process and the potential benefits that may result from the research.​
​It is also important to know that the required elements also contain​
​the confidentiality measures.​
​A lot of times the responses that are gathered from participants, they​
​contain non-anonymous information.​
​It is important for the respondents to know whether the non-anonymous information​
​will be anonymized before the research analysis or if they will be stored in​
​a manner that is safe and confidential.​
​If relevant required elements must also contain information about​
​compensation, and as many of you know by now all experiments in economics​
​necessarily has a compensation part.​
​This compensation part must be clearly laid down in the informed consent form.​
​It is also important to state that participation in the experiment​
​is entirely voluntary and that a participant has a right to withdraw at​
​ ny point in time during the experiment.​
a
​Second, there must be special considerations for human subject research​
​so far as researchers are concerned.​
​For example, researchers must make sure that miners are made to participate​
​in an experiment only after a thorough parental consent and a child ascent.​
​There are other non-minor vulnerable populations with certain kind of​
​disability populations that belong to marginalized communities.​
​Depending on how to what extent the IRB thinks the population in​
​vulnerable, the experimental protocol must clearly lay down extra layers​
​of protections for such population.​
​Third in economics, as we have been mentioning throughout​
​the course, deception is not allowed in experimental research.​
​By deception, one means giving an information that is​
​objectively false or untrue.​
​As economists, we cannot give information that is false or untrue.​
​A dominant norm in the discipline has evolved that prevents us from​
​deceiving our experimental participants.​
​But if under extreme condition you were to deploy deception as​
​part of an experimental protocol.​
​You must suitably debrief the experimental participants.​
​This is something that is done in other subjects such as psychology and​
​other applications in business science.​
​However, if you want to do this, you should definitely consider debriefing the​
​participants, and you should keep the IRB informed that deception is being used.​
​Once again, if you are interested in economics as a discipline or​
​to contribute to the knowledge of economics, then deception means​
​that your research will not receive the credibility that it should.​
​Fourth and finally, in online research, the digital consent form​
​replaces the physical consent form.​
​And given that many of you are probably going to conduct online​
​research or research with participants joining the experimental platform​
​online, you will probably make use of the digital consent forms.​
​It's important to remember at the same time that consent is ongoing.​
​It's not a one-time thing.​
​It is relevant as long as the experiment is under progress.​
​It is also useful for researchers to understand how the institutional review​
​boards are formed so that you as a researcher is better able to approach​
​the institutional review board with your research question and the ethics protocol.​
​First and foremost, what is the purpose and composition of the IRBs?​
​The IRBs are supposed to protect human subjects form harm.​
​It has a minimum of 5 members, and these members are mostly scientists​
​and non-scientists, along with other members of the community.​
​These are all reputed members of the community who have gone and engaged​
​with the ethics of experiments for a reasonably long time, and IRB feels​
​that these members will be able to evaluate the research protocol​
​independently and objectively.​
​The task of this board, which may contain in the minimum of 5 members,​
​will be to review all research that involves human subject.​
​Now when this committee has thoroughly reviewed all the research that​
​involves human subject, what are the review categories that the board can​
r​ ecommend for a particular research?​
​Well, the following are the three main review categories where​
​do your study may be relegated.​
​The IRB may end up saying that the study is completely exempt​
​from the IRB review process.​
​That happens for very few studies and only in studies where there is minimal​
​risk in educational settings, surveys, for example, they consider studies​
​conducted have very minimal risk.​
​Most of these studies are then given an exemption from a full review.​
​The second category, which the board can recommend for your study is no​
​more than a minimal risk, in which case there is an expedited review process.​
​This expedited review process means that the review board or a subset of​
​the review board, as determined by the chair of the board, will quickly give​
​you a yes or no and ask for minimum modification to your research protocol.​
​Finally, the review board can also give you a full review in which case they​
​assess that the risk associated with this research is greater than the minimal risk.​
​Most of the studies in economics fall under exempt or expedited review, but​
​you should be prepared to face the full board if the IRB requires or thinks​
​that it is important for your research to be evaluated by the full board.​
​Next, so what are the submission requirements to an IRB?​
​Well, first and foremost, as a researcher, you have to fill out a detailed IRB form.​
​This IRB form will note down the research protocol and clearly​
​lay down the methodology which you are using for your research.​
​This consent form and the recruitment materials should be made available as​
​part of the IRB application packet.​
​Nowadays, oftentimes no physical posters or flyers are used,​
​but there is extensive online canvassing for recruitment purposes.​
​Now, this risk benefit analysis should also be part of your initial​
​application for IRB, you should be able to independently tell the board​
​what in your assessment is the risk and benefit of the research study.​
​And finally, you should also give a clear data protection plan.​
​If you plan to keep the data in an online folder in a cloud service, then​
​who has access to that cloud service?​
​What are the preventive measures that are being taken by the cloud​
​service to make sure that the research data will not be leaked?​
​These are information that the IRB may seek from you.​
​A very important part of modern research practices is data management,​
​simply because the volume of data has increased by lips and bounds.​
​It is important to note down in the IRB application how you wish to anonymize or​
​de-identify data, if at all you intend to.​
​If you do not intend to anonymize or de-identify the data, there should be a​
​clear reason why the research question demands a protocol where you cannot​
​anonymize or de-identify the data.​
​The data management also requires you to securely store and password protect​
​the data, and there should be clear policies related to to what extent the​
​data will be retained and whether it'll be destroyed after the publication process.​
​As we have been talking about the vulnerable populations need special​
​attention and additional safeguards.​
​These vulnerable populations could be children, they could be prisoners,​
​they could be pregnant women.​
​ hey could also be cognitively impaired individuals, or​
T
​economically and educationally disadvantaged groups of population.​
​However, it's important to be cognizant of these vulnerabilities​
​and make sure that the experimental protocol takes into consideration the​
​vulnerabilities of such populations before rolling out the experiment.​
​It is important at the very end of the day to learn that ethics is an ongoing​
​responsibility for the researcher.​
​Any report that adversely affects any event that adversely affects​
​individuals should be immediately reported to the review board.​
​If a research continues for more than a year, there should be annual review​
​of the research in the board, by the board, and it is important that any​
​changes in longitudinal studies that may have been implemented by the​
​researcher is brought to the notice of the board, and a suitable corrigendum​
​or a correction form is deployed for the, and the board is kept informed.​
​The researchers must maintain accurate records for a minimum of​
​three years, and the publish the results of the research responsibly.​
​At the end of the day, the researcher cares most about the results​
​that emerge from their study.​
​But there is a whole range of things that a researcher must be careful about and​
​cognizant of, and the most important of of that is the ethical considerations for​
​participants of experimental research.​