TERUJI THOMAS: (00:01) Hi, I'm Teru Thomas. I'm a senior research fellow at the Global Priorities Institute and I'll be talking about in this paper, A Paradox for Tiny Probabilities and Enormous Values. This is a paper written with Nick Beckstead, which you can find on GPI’s website and in fact, the paper is based on a chapter from Nick's PhD thesis. So the main ideas in the paper are really due to him. 

(00:28) One of the main themes, I suppose, the main theme of our current research at GPI, is the hypothesis of longtermism. It's not entirely clear how best to formulate this hypothesis. But the rough idea is that what's best to do is typically determined by its possible effects on the long run future. Now, why might this be true? Here's the basic idea. The future could contain many, many people and other valuable things over a long, long time with the capacity to flourish. And whether there are such people over such a long time and whether they do flourish is something we can imprint in principle effect. These people are causally downstream from us at least. And because of their sheer numbers, any remotely systematic effects on such a large number of people, over such a large amount of time could have enormous value far larger than that of any effect we could plausibly have on the relatively small number of people who live in the short term and far larger than any sacrifice that we ourselves could make. So longtermism is plausibly true insofar as we can have such systematic effects. Now, it may seem that most of the things we can do today, especially as individuals, have at most a very small probability of having such a very large effect. And that may be okay for longtermism because even if the probability of success is small, the size of the effect may be so large that trying to achieve it remains overwhelmingly worthwhile. So that is one position you could take. On the other hand, it's very natural to be a bit skeptical. We all face big, concrete challenges every day and we may face many clearly foreseeable dangers over the next century. So it might seem to involve a troubling degree of fanaticism or recklessness to make any significant sacrifice based on sometimes rather speculative sounding stories about very low probability impacts on the long-run future, no matter how large the stakes in principle it might be. 

(02:39) Okay. There's a lot to unpack there, but the general question I'm raising is, how should we think about tiny probabilities of enormous payoffs? Today, I'm going to focus on some of the theoretical tradeoffs that arise in this area even when we're thinking about very clean and artificial cases. The general issue is relevant to many different types of goods. So we could ask, how should we think of a tiny chance of living for a very long time? Or how should we think about a tiny chance of benefiting many people in a certain way? Or how should we think about a tiny chance of life on Earth or life from Earth flourishing for billions of years. 

(03:21) For the purposes of illustration, I'm going to focus on the value of having more people, rather than fewer people, assuming that these people would all have good lives. Many philosophers,  though very far from all, think that having more people with good lives in the universe, wherever and whenever they may be, is better than having fewer people with good lives. It's just the more the merrier. And although this claim is rather controversial, there are also some good reasons to accept it. At any rate, it's an important example because one of the most obvious ways in which we could have systematic effects on the long-run future is just by affecting the number of people who will ever exist. So for example, (1) we can reduce extinction risk. That is, we can have some effect on the probability that humanity will go extinct in the next few centuries. And that some effect on the probability that there will be very many future people or almost none. And (2) even setting aside extinction, we may have some leverage on whether and under what circumstances, humanity settles other planets thus potentially opening up vastly more resources and vastly more room for a large human population. So those are just two examples. As you can imagine, there are many other ways, some more mundane and some more fantastical, in which this type of value could be important. 

(04:45) So that's the setup. The question is specifically, how to think about tiny probabilities of large numbers of people? My first task is going to be to explain a theoretical dilemma that arises from this question. 

(05:02) So here goes. God has given you a people creating machine with a big green button and pressing the button will lead to the creation of, let's say, 100 people, and all of these people, I'm assuming, will have throughout a very high quality of life. So that's what I’ll called Deals 0: Create 100 people for certain. Now you're about to press the button when the devil comes along. “Hang on,” he says, “I've got another machine, you might like better. It will create 1000 people, 10 times as many people and it works with probability 0.99999999999.” So just 1 billionth of 1% lower than before. “Sounds great!” And you trade machines. That's Deal 1. And you're about to press the big green button on the new machine when the devil interjects. “Just wait one nanosecond,” he says, “I've got another machine you might like better. It will create 10,000 people. 10,000 good lives, 10 times as many and the success rate is again, only zero point, only slightly lower, I should say, 0.999999999992. So just, again, 1 billionth of 1% lower than before. “Wow! Sounds great,” you think. “Thanks, Satan.” So it keeps going like this. And during an hour of extremely rapid negotiation, the devil offers you a sequence of deals, but each of the deals decreases the probability of success by 1 billionth of 1% and it increases the possible payoff by a factor of 10. So in the kth deal, the probability of success is 0.99999999999k power and the payoff is 100 x 10k. And at the end of the hour, you find yourself with this machine that could create some truly mind bogglingly enormous number of good lives but it only works with a truly mind bogglingly small probability less than 1/10100, let's say. You press the button and the machine falls to pieces in your hands. That may seem like the devil has played you for a fool. 

(07:34) All right. Here's the first few that you could take of this example. You might think, each deal is better than the one before, therefore, despite what many people are inclined to think the last deal is better than the one you started with. And more generally, since the specific numbers don't matter in the example, you would accept what we call Recklessness. Recklessness says that any finite payoff no matter how good it may be, is worse than a tiny probability of some other sufficiently large but finite payoff. And this is true no matter what tiny means. So in the example, the devil brought the probability down to 1 in 10100, but of course, he could have gone even lower. By the way, you can find somewhat more formal statements with the various conditions in the paper, but I think these informal statements are good enough for today. So that's recklessness.

(8:39) If you don't like recklessness, then the second view is what we call Timidity. So on this view, the last deal is not better than the first and therefore, despite what many people are inclined to believe, one of the devil's deals is no better than the one before. You should have rejected the devil's offer at some point. So therefore, sometimes a 1 billionth of 1% decrease in the probability of success cannot be outweighed by a tenfold increase in the payoff. And more generally, since the specific numbers in the example don’t matter, sometimes a tiny decrease in the probability of success cannot be outweighed by any increase in the payoff. And again, this is true, whatever tiny means. In the example, we took a tiny decrease to mean, 1 billionth of 1%, but we could have used any other standard. So that's the second one of the dilemma, Timidity. 

(09:40) There is a third view, which we can call Non-Transitivity. So I was thinking before, if each deal is better than the one before, then the last deal is better than the first. But maybe this inference is invalid. So you might think, “There are some cases where A is better than B and B is better than C, but A is not better than C. This is non-transitivity. And some people among philosophers, most famously Larry Temkin and Stuart Rachels, think that this is the correct view about other cases like this. But I won't consider non-transitivity in this talk. I'm just going to set it aside. 

(10:24) Okay. So it seems we have to choose between recklessness and timidity and the rest of this talk is about that choice. Both of these views seem strange to many people, but going beyond that immediate intuition, they both have fairly strange consequences and I'm going to draw some of those out. 

(10:49) First though, just to emphasize, in this talk I'm not assuming any particular theory of evaluation under uncertainty. However, there is a dominant theory of that kind, namely Expected Utility Theory, so maybe it's worth just mentioning how things look from that point of view. If we assume expected utility theory then it's not hard to see that, first of all, timidity is equivalent to having a bounded utility function. That is, it’s equivalent to the claim that the utility of finite payoffs is bounded above. And by the same token, recklessness is equivalent to having an unbounded utility function, one that's not bounded above. Both unbounded and bounded utility functions lead to well-known difficulties within the context of expected utility theory. So what are the difficulties I have in mind? In the case of unbounded utility functions, we have a variety of problems related to the St. Petersburg gamble and related to Pascal's Wager, and I'll talk about those a little later. In the case of bounded utility functions, we know that these lead to quite ridiculous degrees of risk aversion in certain cases and there are also some quite subtle problems related to the idea of separability. So what I'm effectively explaining here is that some of these difficulties are very general. They're almost totally general in the sense that they arise directly from recklessness or directly from timidity plus maybe some other weak background assumptions, but even without expected utility theory. 

(12:32) Okay, so let's do it. There are lots of problems considered in the paper and I'm just going to talk about one problem for timidity and one or two problems for recklessness.

(12:46) So here's the first problem. Remember, we're assuming for the sake of illustration, that the existence of people with good lives makes the world better. Okay? The existence of people with good lives makes the world better. So with that frame of mind, ask yourself, is it better to create some good lives with a tiny probability or a trillion times more good lives with a trillion times greater probability? Surely, we should think that the latter is better. But timidity implies that this is sometimes incorrect. And in a quite strange way.

(13:32) Let me be a bit more specific. Suppose the situation were like this. We can influence how many people there will be in our galaxy. But we have no influence over what happens in other galaxies. In fact, we have no real contact at any time with these other galaxies. Suppose it's like that. Then one might accept the following claim. Regardless of the situation in other galaxies, instead of creating some people with some probability in our galaxy, it would be better to create a trillion times more people with a trillion times greater probability. And timidity forces us to deny this claim. 

(14:23) Let me try to explain why. The case is slightly complicated and it's probably a bit hard to take in during a talk, but I'll try to explain it slowly. So we're comparing two different options, A and B. What happens when we choose one of these options? It depends on the outcome of a kind of cosmic coin toss at the beginning of the universe. So let P be the probability that the coin lands heads and q be the probability that the coin does not land heads or tails, it lands on its edge. Of course, it's extremely unlikely that the coin would land on its edge. Let's suppose that q is only 1 trillionth of P. Okay? So that's the general setup. Now consider option A first. If the coin landed heads or if it landed on its edge, then option A leads to the existence of little n people. Otherwise, on tails, to 0 people. It doesn't have to be literally 0, we can just think of 0 as some baseline number. Option B is different. It may lead to vastly more people in n + X , where let's say X is a trillion times bigger than n. But that only happens if the coin lands heads. So B leads to vastly more people than A does, but with slightly smaller probability. So that's the table. Now, timidity says that in some such case, B is not better than A. A tiny decrease in probability cannot in some such case be compensated by any increase in size. Okay. So according to timidity, in short, B is not better than A. But here's the thing. Suppose that the n people I've just highlighted, the n people who would exist on heads would exist in a galaxy far away. So as far as these two options go, we have no influence on whether those people exist or the probability that they exist or any influence on them at all. So setting aside the situation in that other galaxy, option A only creates n people with a tiny probability q and B creates a trillion times more people, X people, with a trillion times greater probability. So according to the claim on the previous slide, B, which creates a trillion times more people with a trillion times greater probability, is better than A. And that's what we just saw timidity denies. So that's one of the problems with timidity. 

(17:22) This seems to me, a counterintuitive and unhappy result for timidity. But recklessness has its own problems and I'll sketch two of them now. 

(17:34) In this discussion, I'm going to use the following Dominance Principle, which I call Outcome-Outcome Dominance, but it's the same thing that people usually call Statewise Dominance in the literature. It says, if no matter what, the outcome of A will be at least as good as the outcome of B, then A is at least as good as B. This is a very popular and natural principle. I think there might be some reasons to deny it in full generality, but I'm going to apply it in some fairly simple cases, where on its own at least, it looks impeccable. 

(18:11) The first problem I will mention is that recklessness essentially entails what we call Infinity Obsession. Infinity obsession is a claim that any finite payoff, no matter how good, is worse than any positive probability of an infinite positive payoff. So for example, an infinity-obsessed agent would give up an arbitrarily good but finite utopia for a mere 1-in-10100 chance of an infinite payoff. And another famous example is Pascal's Wager, which is roughly the thought that we should all have faith in the Christian God, just in case he exists, no matter how small that probability may be. Now, as far as that goes, infinity obsession is very similar to recklessness. Whether reckless or infinity-obsessed we will, under appropriate maybe somewhat artificial circumstances, pursue extreme long shot bets at great cost. So what's the difference really? It's not just that recklessness involves infinite payoffs and infinity obsession involves finite payoffs. There's another important difference. In the case of a merely reckless agent, there is in any given situation, typically, some possible evidence of failure short of complete proof that would cause him to give up on the long shot bet. In contrast, the infinity-obsessed agent will not give up on the long shot, unless he becomes completely certain of failure. That is, unless the probability of the infinite payoff goes all the way to zero. So the Infinity-obsessed agent really is fanatical in a way that the merely reckless agent is not. 

(20:03) Now, having said all that, the immediate practical implications of infinity obsession may be somewhat unclear for various reasons. But infinity obsession concerns arbitrarily small probabilities of infinite payoffs a2nd it's worth making the standard point that given an infinite universe, it's hard to entirely rule out infinite payoffs. For example, it's hard to entirely rule out the creation of infinitely many good lives. So infinity obsession and similar principles are practically relevant, at least to that extent. 

(20:38) Okay. So I've claimed that recklessness essentially entails infinity obsession. Perhaps that's not too surprising, but let me try to explain how it works. So consider any finite payoff x and positive probability p. First of all, recklessness says that no matter how tiny p may be, there's some finite y such that a p chance of y is better than x for sure. So we could trade our finite utopia. We should trade our finite utopia for a p chance of some amazing payoff y. That's the first point. However, if z is an infinite payoff, then a p chance of z is presumably at least as good as the same p chance of y. Formally, that's an application of outcome-outcome dominance, but you can take it in its own terms. Therefore, the third point, putting it all together, a p chance of z is better than x for sure. And this is true for any p and z. So we should give up our utopia x for any chance p of any finite payoff of z. That's the argument.

(22:11) I'll now describe a second problem for recklessness of an even more theoretical nature. So far, I've relied on a dominance principle, which I called outcome-outcome dominance. And I'll continue to take that for granted. But if we do so then recklessness leads to violations of another almost equally attractive principle, which we call Prospect-Outcome Dominance. So to make it especially nice, suppose that A and B are probabilistically independent prospects. If A is better than every possible outcome of B, then the principle is A is better than B. Okay? So if A and B are probabilistically independent and if A is better than every possible outcome of B, then A is better than B. So why is this attractive? Suppose I have a choice between A and B, prospect-outcome dominance then reflects the following kind of reasoning. Should I take A or B? If I found out the outcome of B, I would inevitably want to trade it for A, so I should just take A. You might ask why do I require that A and B are probabilistically independent? You might think that the principle is pretty plausible even without that, and I agree, but it's nice for this argument I just gave that when you find out the outcome of B, you don't learn anything about A. So you're really comparing some particular outcome of B with the whole prospect A. 

(23:47) Okay. So why is it that recklessness violates prospect-outcome dominance? In outline, the argument goes like this. So, first bullet point, given recklessness, there must be some prospect C that's better than every one of its own outcomes. Now that's already pretty weird, but we can turn it into a violation of prospect-outcome dominance. So the second point, if we take two independent copies of C, call them A and B. These are probabilistically independent and because C is better than every one of its own outcomes, it follows that A is better than every possible outcome of B and also B is better than every possible outcome of A. So prospect-outcome dominance would say that A is better than B and B is better than A. That's a contradiction. So given recklessness, prospect-outcome dominance must be false. Okay. So this argument relies on this claim that there is a prospect C that is better than everyone of its own outcomes. 

(25:06) Why is that true? Well, I'm not going to go into the details here, but this problem may be familiar to many of you in the context of Expected Utility Theory. There we have the famous example of the St. Petersburg gamble. So this gives a (1/2)n chance of 2n units of utility. And this St. Petersburg gamble has infinite expected utility even though each outcome only has finite utility. So it would seem this is a bit hand wavy, but you can turn it into a proper argument that the St. Petersburg gamble is better than every one of its own outcomes. And the general construction is really the same, but using recklessness to generate an analogous sequence of higher and higher value outcomes. 

(25:59) Okay. So the last couple of arguments have been pretty technical and it may have been difficult to follow some of the details. If you're watching on video, then I guess, you can go back and pause and try to sort through the details or just look in the paper. But I really just wanted to get the rough idea across. And here's the upshot. 

(26:19) Many of the philosophical and theoretical problems that are traditionally associated with an unbounded utility function in the context of expected utility theory arise directly from recklessness without needing expected utility theory. So first of all, we have infinity obsession, which you may recognize is related to the classical problem of Pascal's wager, and we have a certain kind of dominance failure that is closely related to the St. Petersburg gamble. 

(26:50) Let me wrap up. 

(26:52) So I presented a dilemma between recklessness and timidity and roughly speaking, recklessness encourages us to take extreme long shot bets, while timidity is something like an extreme form of risk aversion. Recklessness, as I pointed out at the beginning, is especially favorable to longtermism, although longtermism might well be true without it. And more than that, recklessness would tend to turn our attention towards certain speculative seeming causes within longtermism. For example, it might turn our attention to certain kinds of long shot bets on Infinite payoffs, as we saw in discussing infinity obsession. But be that as it may, both views, recklessness and timidity seem to face serious worries and objections, which I think deserve further consideration. 

(27:40) Thanks very much for listening.