Annie Duke is an author, speaker, and consultant in the decision-making space, as well as Special Partner focused on Decision Science at First Round Capital Partners. Annie is the bestselling author of multiple books, most notably Thinking In Bets and Quit. Annie was previously a championship winning poker player, having amassed over $4 million in tournament winnings, won a World Series of Poker Bracelet, and having won the World Series of Poker Tournament of Champions. As if that wasn’t enough, Annie also completed her PhD in Cognitive Psychology at UPenn in 2023.
Annie joined host Robert Glazer on the Elevate Podcast to discuss learnings from her poker career, how to think in bets, and the importance of knowing when to quit.
—
Listen to the episode here
Annie Duke On Thinking In Bets, Rethinking Quitting, And More
Annie’s Background And Accomplishments
Welcome to the show. Our quote is from Tony Robbins. “It’s in your moments of decision that your destiny is shaped.” My guest, Annie Duke, is a master decision-maker. She’s an author, speaker, and Consultant in the decision-making space and a special partner focused on decision science at First Round Capital Partners.
Annie is the best-selling author of multiple books, most notably Thinking in Bets and Quit. She was previously a championship-winning poker player, having amassed over $4 million in tournament winnings. She won a World Series of Poker Bracelet and won the World Series of Poker Tournament of Champions. If that wasn’t enough, Annie has also completed her PhD in Cognitive Psychology at the University of Pennsylvania in 2023. Annie, welcome to the show.
Thank you for having me. I’m excited to be here.
Annie’s Early Life And Introduction To Games
I always like to start with background, childhood, and the beginning. I’d imagine passion for poker games must have emerged at a relatively early age. When did you actually start playing poker or games like poker?
The main social time that my family had was cards or games, but mostly cards, and we didn’t play a lot of poker. We played a lot of hearts, gin, and a game called Ohel, which is a beginner version of bridge.
What about euchre?
No, we didn’t play Euchre, but Ohel is fun. Ohel has similarities. We played Scrabble, but mostly we played those games. We played cards. Very occasionally, my dad would take out one of those octagonal things with the circles in them that you stick the chips in. The chips are like the worst plastic. We were playing poker, but it was always like the silliest possible games, like a game called Pass the Trash, which some people know as Anaconda, where you get dealt, you pass three to your right, three to your left, or hold a baseball. there were wild cards and you were passing it. It wasn’t poker, but that was very rare. Not a lot of poker, but lots and lots and lots.
This was a game-playing family.
Yes, everything, like not that social time, was games, and it wasn’t like. I don’t remember ever playing Sorry. My brother and I would play Monopoly sometimes.
These were games of skill more than games of chance.
Yeah, and adult games. I wasn’t playing Chutes and Ladders. Monopoly was the most kid-friendly game I can remember playing, but it was an adult card game.
You must have been good, given where you ended up. As a kid and being thrown in this environment, to me, this is like when you start beating your dad about, when did you start beating the adults?
When I was young, you’d have to be good, like in comparison to what? Compared to 7 other 7-year-olds, I’m sure I was the best 7-year-old gym player, but I was playing against my dad, who was in his 30s, and my brother, who was 2 years older than me, which is a lot, obviously, at that age. I was not as good as they were, and I did a lot of losing. I was not the best loser on the planet. My brother and I both have stories about beating our father. When he was a teenager, my brother took up chess, and he got pretty good at it. He had big chess books that he would carry around and study.
This is some real DNA here you have.
My dad is the one who taught him to play chess. I think when my brother was 1fourteen4, he played a game of chess with my dad and won. My dad was like, “Rematch.” He played him again and won. My dad, that was it. That was the last time my dad and my brother ever played chess. When I was sixteen, I beat my dad at Scrabble. I think it was particularly bad because my dad is like the language guy. He was an English teacher and he’s written a bazillion books on linguistics like etymology, morphology, like English language, and stuff like that. When I was sixteen, I beat him in Scrabble and that was it. I never saw the Scrabble board again, not with him.
Was he always teaching you while you played, or were you forced to learn to keep up?
I don’t remember him explicitly teaching. I don’t remember a lot of it. It would have been better if you played this card to tell you the truth. Even when I was a teenager, I became his bridge partner. I don’t remember a lot of that. I think it was a lot of it was like, “Look, here’s the deal. I love my dad. He’s awesome. He likes to win.” He may not even give me instructions because he is boring.
I’m guessing the apple doesn’t fall far from the tree.
He was a big competitive tennis player. Now, he plays poker all the time. Since my brother and I were pros, he learned actual poker as opposed to the weird games we were playing when we were young. He will call up and say, “I won $200 and $30 in my poker game on Thursday night, and then he’ll tell me hands, and it’s very cute because I’m often horrified like, “What? What happened to my poker?”
Now, you didn’t start out intending to play poker. In fact, it was a story, and we’ll get into the book and quit there, but what did you think you were going to do?
I was going to be a college professor at university with my own lab in cognitive science.
What happened to throw you off that path?
I got sick. To set the stage, the way that a PhD program works is you do two. At least this PhD program worked. Most of them work this way. You do two years’ worth of classwork, two and a half. Your first two years, and then the first half of your third year, you’re doing classwork. You’re also TAing. You’re doing that. You’re starting to figure out what your research is going to be, and you’re working on that.
The second semester of your third year, at this time, this has changed since then, but at that time, you did something called your qualifying exams, which you still do now, or otherwise called your major area exams. At the time, what that meant was you got between 3 and 5, usually pretty broad questions. You did not get a reference list or anything like that.
They were like, be off with you, and I’ll see you in four months when you have written very detailed responses to these broad questions. In my case, I was actually studying first language acquisition. One of the questions was, what is the evidence for the language function being innate? That’s how broad it was. you had to put together a whole thing.
As I recall, it was 175 pages single-spaced, like it was a lot. I did those. I emerged pale and malnourished from that. After you finish that, then you’re allowed to teach your own classes because you are qualified. I was still teaching my own classes, and then I was deep in my research. Usually, by the end of five years, you’re expected to be completed and off to the races. That is actually the path that I was on.
I presented a lot of conference work that I did, got published, and then went out for my job talks at different universities to try to get hired as a professor for the next fall. That’s when I had a chronic illness that became super acute. I actually ended up in the hospital for a couple of weeks. The only reason I give that detail is not, “In my first year of graduate school, I quit.” It was like two weeks from defending my dissertation.
You couldn’t.
I couldn’t. It was two seconds. I was going to take some time off and I was going to come back the next fall, go back out on the job market, and finish the PhD work. In the meantime, that’s when I started playing poker to support myself. I loved it, and obviously, it took me a long time to go back to graduate school because I had finished.
Decision-Making And Uncertainty
These things all converge. I know you delivered your first speech on decision-making before the peak of your poker career and your first hit book, Thinking in Bets, talked about how to make decisions under uncertainty. I’m curious. We live in a pretty uncertain time. What are some of the key takeaways from that work and that book on how you make good decisions when you don’t have all the information?
First of all, I think it’s interesting because people say this a lot as if we live in very uncertain times.
Not that any time is certain, yeah.
When we think about the past, we already know what happened. It doesn’t feel as uncertain because it’s very hard for us to put a lot of moving pieces. It’s interesting, like it’s hard for us to go back to when we didn’t live and imagine what somebody feels like looking into the future, but it’s even hard for us to do that for ourselves.
There’s something called hindsight bias, which makes it feel like it was knowable already. You knew it all along. I find that with COVID, for example, people have a very hard time putting themselves in March of 2020. It wasn’t clear. Was it respiratory droplets? Did it kill children or not? Who was it? We didn’t know any of those things.
Now everybody wants to go back and say, “I told you so.” There were a lot of unknowns there. Let’s remember that. I think it’s a very hard thing for us to do. Pretty much all the decisions we make are made under conditions of uncertainty. There are two sources of uncertainty. The first is plain old luck. I’m sure you’ve heard people say, you may have said it yourself, I make my own luck. No, you don’t because you can’t.
Pretty much all the decisions we make are made under conditions of uncertainty.
I half believe in that, half, yes. I actually don’t think you make your own luck. I think you can put yourself in situations where you’re not wiped out by bad luck or helped by good luck.
Yes, you can do that. That is true, but that means you make your own decisions that change the probability of you having a good or bad outcome or your risk of ruin. When people say I make my own luck, it actually doesn’t make any sense because luck is exogenous.
I magnify or diminish my own luck. Is that more accurate?
No, actually, because you can increase or decrease the probability of a bad outcome or increase or decrease your expected value, but you don’t have control over what you’re going to observe on the next thing. Here’s an example. Let’s say that we decide we’re going to flip coins and we’re going to bet on it. You offered me a set of different bets, and I could choose any of them. In one case, I have to give you $10 for every dollar that you give me.
In another case, you’re going to give me $10 for every dollar that I give you. Now, depending on which one I choose, no matter what, the coin’s going to be 50/50. No matter what, whether I call it correctly, it’s going to be 50/50, but I’ve changed the distribution of outcomes for myself depending on what I choose, but luck is luck nonetheless. Got that. What I like to say is I try to make good decisions that reduce the impact of getting bad outcomes. That’s a good way, I think, to think about it or reduce the probability of getting bad outcomes or increase my expected value. That’s that.
That’s the first thing and that’s luck. The second way that uncertainty exerts its forces upon us is hidden information. In the description and the thing that I told you in this scenario that I told you, we know that we have a coin that’s 50/50. We’re not in a hidden information situation. I know what the coin is, I know what the bet is and now it’s up to luck. Do I hit five heads in a row? I’m going to do better if I’m calling heads than if I hit five tails in a row, but I don’t have control over that.
For most of the decisions that we make, we also have hidden information. There’s a whole bunch of stuff that we don’t know. That’s more like the COVID situation. With the COVID situation, there’s a lot of luck if we go back to March of 2020 or before that. We have no control over mutations of the viruses, for example. We can make decisions that reduce or increase the probability of getting the disease, but that’s all we can do. Maybe I’m doing things that are going to make it so that I’m only 10% to actually getting sick, but I’m going to observe that 10% and I can’t control that.
I also don’t have control in terms of my own body and how the disease is going to act upon me. For some people, they’re asymptomatic, and for other people end up on a respirator at that time. If we think about that time, we don’t have control over that part, so that’s the luck part. There’s also a lot of hidden information and we talked about that. Is it airborne? Is it respiratory droplets? Can I get it from touching a hand, a toilet seat, or a grocery store shelf or how is it transmitted? We don’t know. We could feel that because, at the beginning, and I was doing this too, people were quarantining their packages.
That’s like a lot of effort, but that’s because we didn’t know, there was information that we didn’t have that we actually couldn’t get it from touching a package, but at some point, we thought you could. You can see that in terms of COVID, but obviously, that applies to pretty much any decision that you make. You can see how that’s very poker-like because, in poker, I don’t have control over the deal of the cards. I’m trying to make a good bet, but maybe I’m in a situation where I’m going to lose 10% of the time. I don’t have control over when I’m going to observe that 10%. All I can do is make a bet that’s positive.
Just got to have enough money when you do.
Obviously, there’s hidden information. I don’t know what other people are holding. You can see those two forces coming together. When you say like, how do you make good decisions under those circumstances? We could talk for three hours about how you actually do that. We could talk for two days about how you do that, but if we get down to the core of it, it’s recognizing what we talked about. understanding that there is luck.
It’s actually being clear about what you don’t know and the impacts of that.
Being clear about what and don’t know and being clear about what the influence of luck is likely to be. In other words, what’s the distribution of possible outcomes?
This was my thing early on about masks and people were losing their minds. I was like, look if it turns out that the mask was not necessary, I didn’t waste much. It wasn’t a liberty thing. If it turns out, it seemed asymmetric to me, particularly when people were unsure and confused. As you said, it’s clear in retrospect. I’m not sure it’s clear in retrospect. I’m not sure if this happened tomorrow that we have a consensus, but there were some things where I guess you get, to make bad decisions when you get very tribal and myopic and not pragmatic. A lot of these things were like, it’s not a huge cost to me.
Cass Sunstein and I actually wrote an article about this, and this is a concept in how to decide, which was the book that I wrote after Thinking in Bets. You’re pointing out that some things are what we would call free rolls. Free roll is a gambling concept, and it comes from back in the day when you would go into a casino and they would hand you a free roll of nickels.
All the online casinos did this as they were starting.
They give you free rolls, but they actually come from a free roll of nickels. The idea was, if you go and you play those nickels in the slots and you lose it, then you’re not worse off than you were before you walked in. Now, you should pop the nickels, but we’re going to ignore that part. What you’re getting at is that one of the things that we always want to be thinking about with decision-making is what the downside is compared to the upside potential.
One of the things we always want to be thinking about with decision-making is what the downside is compared to the upside potential.
As you said, there’s a whole bunch of stuff we don’t know. We’re trying to make these forecasts of what do we know about the downside or upside potential. One of the mistakes we make as decision-makers is that we don’t embrace uncertainty and want to be certain before we decide. That can create a whole bunch of bad things. One thing is that it can slow us down.
When people talk about paralysis by analysis, you’re talking about trying to get to 90 or 100% certain about something that you don’t need to do. This is a good example of masks. We use it. Cass and I use the example in our paper. I think that what happened was that separate from the tribalness, people were like, I want to know what the upside of this is.
They wanted to know that for sure, but what you’re pointing out is that but you can think about it in a different way, which is what’s the downside. That’s where we want to start. What’s the downside to doing this? In that particular case, we knew that people had been wearing masks for many reasons for a long time, like construction and surgery, and so on, and people weren’t dying.
It wasn’t like your surgeon who had a mask on during surgery all of a sudden was dying or something. We knew there was no downside except that you were uncomfortable. If you have bad breath, it smells something. There’s basically no downside. What’s the upside? I have no idea, but it could be very large. This is the idea. This is the central concept of a free roller.
That’s asymmetric.
Totally asymmetric. That’s the idea.
Nassim Taleb writes a lot about these asymmetries and how we make poor decisions under him.
The thing is, you don’t actually need to know what the upside is. You need to know that there’s any symmetry and that the downside is limited. Now, there’s an interesting thing that it comes up in poker a lot, where you get what’s called a negative free roll, where the upside is very limited, but the downside is huge. That also comes up, where you can have very limited upside and a lot of downside, but I think what’s important is for people to recognize that you have to think about both sides of the equation.
When the very first study about hydroxychloroquine came out, it looked like there might be a lot of upside. It looked like there might be a lot of upside. It was unknown. Remember, you have to think about the other side too. Actually, the downsides of hydroxychloroquine were quite well known. That’s a situation where you would actually want to understand what the upside is better so you could actually do cost benefit.
Drinking bleach is pretty asymmetric.
That was a case where I think a lot of people took that because they thought there was an upside, but they actually forgot to think about the reverse. You always want to think about two sides of the equation. You don’t need to know both if one side is very big or limited rather. If there’s basically no upside, then it doesn’t matter what the downside is. It’s like, why would you do it? If there’s basically no downside, you don’t necessarily need to know what the upside is. You need to know that the potential is there. As you gather more information, then you can start to make more precise decisions.
I can see why you’re a good card player. It’s an academic approach to card playing. We can talk about that forever.
Three roles aside.
The Concept Of Quitting And Its Cultural Perception
We’re going to quit by switching to talking about Quit, one of my favorite books I’ve read in a while. I have stuff circled everywhere in it. Let’s start out. One of the concepts you discussed early in the book is we have a culture that celebrates perseverance and discourages people from quitting even when they should.
I think there’s one thing you point out, which I think a lot of people point out: good quitting and bad quitting. I remember my favorite quote was Elizabeth Edwards while she was dealing with her husband and the affair and cancer. She said something like, “Resilience is about deciding whether you want to be, you want to be miserable or don’t want to be miserable over something and making that choice.” How do we overcome this celebration of perseverance that we have?
How do we overcome it? That’s an interesting question.
How should we think about it maybe I think a lot of people don’t realize that’s such a force in our life, that’s become a cultural force.
Interestingly enough, I think there’s a little bit of a chicken and egg problem, which is that there’s definitely a bias towards sticking to things culturally. Winners never quit and quitters never win.
Is that a Western civilization? No, that’s a universal.
There’s definitely that, but then there’s also a whole host of cognitive biases that are in a big pile that make it very hard for you to walk away. The aphorisms and the Vince Lombardi thing against quitting may actually reflect our cognition. It may be reflecting that. Some of the cognitive biases that make it very hard to quit, I’ll give you a big one.
There are a whole bunch of other ones in the book, but the biggest one is probably the sunk cost fallacy or cost effect, which is that we take into account what we’ve already spent on something, time, effort, money, attention, so on and so forth, and deciding whether to continue on and spend more. I’m sure everybody has felt this. They’re in a relationship and thinking about breaking up and having trouble doing it because what about I put so much time into this already?
Construction projects come to mind.
You’re in Boston or the Big Dig. They did actually get there. The California bullet train is one right now where there are many billions of dollars into a train going nowhere because they can’t actually solve the huge engineering problem of blasting through mountains in a seismically active area.
Let’s double-click on that because there’s a principle that I thought was fascinating, which is the pedestal. There are some things where all of the difficulty is in a very small piece. In California, they did all the other stuff first and then got to the hardest. Now, that makes the Sun Cost even harder.
That’s the problem. A classic way to think about some cost is if let’s imagine that you have a band that you love and they’re going to play and it’s an outdoor concert that’s in July. A friend of yours says, “Do you want to go? You love the band, but the day they have the free ticket is like this freak weather event where there’s horrible rain, but it’s also freezing cold. There’s a freak July event where it’s 20 degrees out in the middle of July and it’s freezing cold, like rain pelting you.
The question is, if they had the free ticket, would you go? Most people will say no, obviously not. I wouldn’t go to the concert because I don’t want to be miserable out in the cold. The question you want to ask is, let’s imagine you bought a ticket a few weeks ago and paid $1,000 for the ticket. Would you go? Everybody’s like, “Yes, I don’t want to waste my money.”
Even though it’s going to be equally miserable.
It’s going to be equally miserable. The thing is, when you think about the free situation, what you realize is that you’re saying that going forward, it’s not worth my time to go to this because I’m going to be miserable and unhappy and maybe I’m getting pneumonia. It’s going to be horrible. It actually doesn’t matter that you have already spent $1,000 on it because that is already spent. It’s gone. You can’t get that money back.
All you can do is decide if you want to be miserable at that concert.
This comes up in the stock market a lot, where you’ll have somebody looking at whether they want to sell a stock already in their portfolio. What you’ll see is a disconnect between whether they would buy the stock and whether they hold the stock. If I come to it fresh and I’ve never held the stock before, I say this is not a buy. It shouldn’t matter.
It should be a sell, yeah.
What my history with the stock is, it’s not a buy, but what happens is that when we’re what’s called in the losses, meaning like we bought it at $50 and now we’re at $40, so we’re in the losses, then we’ll tend to hold the stock, trying to get our money back. You can see how that’s similar to the concert situation because we’re in the losses of $1,000. That’s a cognitive problem, but it doesn’t matter because if the stock’s not a buy, it’s not a buy.
When we’re in the losses, we resort to luck more. We’re in the win. We want to cash it in and don’t care about luck.
This is a very famous finding by Daniel Kahneman and Amos Tversky from 1979. It was a key piece of what won Kahneman the Nobel Prize, which was his prospect theory. What happens is that when we’re in the losses, we’re very risk-seeking. When we’re in the gains, we’re very risk averse. The classic way to think about it is, let’s imagine that you owe me $100. We’ve been gambling, we’ve been flipping coins, and you’ve lost $100. Now I say to you, “You can either pay me the $100 or the coin.”
When we’re in the losses, we’re very risk-seeking. When we’re in the gains, we’re very risk-averse.
Double or nothing. I watch my boys lose tons of money through double or nothing.
You can do this double or nothing. I want to be clear that automatically, those are the same in terms of what we call expected value, which is, over time, what would you expect to win or lose? Half the time, you’re going to lose $200 and half the time, you’re going to lose zero, which averages out to $100, which is what you could also give me.
The only difference between those two propositions is that one is not risky and the other is risky. I’m adding luck into the equation. When people owe me the hundred, they’ll be like, “Yes, let’s do double or nothing. I want to do that.” in other words, they’re seeking that risk. They’re trying to get luck to help them out. Now let’s imagine that I lost $100 to you and I say, “Do you want to flip a coin, double or nothing?” People are like, “What? No, give me my $100.”
Locking the gains.
Again, it’s identical. Those are mirrors of each other. You want to make $100 either way. The only difference is the risk, and then I can actually make it worse and say, “You owe me $100. We could flip a coin, if you lose, you’ll owe me $220. If you’ll win, you’ll get back to even. Now, this is actually worse because, on average, you’re going to lose $110 in the long run. You’ll go for that. You will do that. You’ll be like, “Okay,” because you so badly want to get out of the losses, so you will not quit in that situation. On the reverse, if I say, “I’m going to pay you $220 or zero.”
The extra upside isn’t as interesting.
You’re like, “No, give me the $100. I want the $100.”
After I read your book, I started putting stop losses on any stock trade.
This is the issue. I think this is something that is actually important to understand. Stop losses are actually a good example of this. I like how you talk about people moving them. When they’re losing money, they move them down. I didn’t mean when I lost $10. I meant when I lost $15. Here’s the thing that I think is important is the reason why I wrote a book called Quit is because I think that the stopping things part of the equation is very neglected. There’s a very famous book by Angela Duckworth called Grit. It’s a fantastic book.
She was on the show. We talked about it.
It’s amazing.
The Intersection Of Grit And Quit
Tell us about the quit and grit coin flip.
She’s writing a book about the value of sticking to things, but what I felt was neglected was this other side of the coin. You also have to quit a lot of things. Things that aren’t working out, you shouldn’t quit. We know this is totally obvious. We want people to quit drinking when they’re alcoholics. We want people who are going up Everest to not continue up Everest when there’s a snowstorm.
I was going to bring the Everest example because, in some things, you need to metaphorically live another day, poke or otherwise, and this is physically. If you talk about Everest, people don’t realize climbing Everest is a season commitment and people turn back months into it right before the last stage because if you make that decision incorrectly, you die. Some of the best climbers in the world, as well as the smartest climbers, have been wiped out by it. There are a lot of alive people who are upset that they didn’t make the summit.
Yes, that’s correct. This is a very classic sub-cost problem because you spend a lot of time. The thing is like this happened this year, there was another tragedy on Everest. Under conditions where you shouldn’t continue, people do. People can recognize that, but then when it comes to shutting down a startup or quitting a job or firing an employee, which is a form of quitting, you’re right, from the employer’s perspective, getting out of an unhealthy relationship, like all these things, we find it very hard to quit. We’re getting punished twice for not being able to make that decision.
One is that we’re doing something that is not helping us to achieve our goals. It might kill us as a matter of fact. That’s one way, but then the other thing is that we can’t actually turn that attention or time or money or effort to something that is great. That’s this opportunity cost problem. Is that we can’t go pursue other opportunities that would be more likely to get us to our goal. What I wanted to point out was Grit and Quit are the same decisions. We need to treat them like they’re the same decision.
It’s resource allocation.
I think what happens is that we think that coming down on the grit side of the equation is the better choice, but it’s not true. It’s if what you’re doing.
You don’t want to be gritty in an abusive relationship.
Exactly, but people are afraid. I had somebody write to me after reading Quit and saying, “Thank you so much for writing this book. I quit my job. I’d been living with a toxic boss, toxic and abusive boss for months. I refused to quit because I thought it was a sign of character that I was persevering through that and I read your book and I realized that wasn’t true.”
You had an insight that was so simple, but I was like, “Huh.” You never truly know the outcome of quitting. What society does is we tend to criticize the early quitters and then condemn the late, “Jesus was way too late and whatever.” If they quit too early, you have no idea they quit too early. As you understand, we’re judging the end of the movie without seeing it.
You never truly know the outcome of quitting. What society does is tend to criticize the early quitters and then condemn the late.
They couldn’t quit at the exact right time. The next time they showed up on the football field, they could have been crap.
It never doesn’t feel premature.
By the way, Tom Brady is a great example of this. When he retires, everybody’s like, “I’m not sure he should be retiring.” Then he’s, “I’m bored. I’m playing again.” He sucked and everybody’s like, “Why’d you come back?” That was a stupid decision.
I’ve thought about this with Lionel Messi. His whole career, he wanted to and he sucked if you think about expected value. I wrote an article about his whole career and his winning the World Cup. They have this magical thing. They win the World Cup. He’s near the end. You can go out on top. It seems like there’s so much more to lose than there is to gain continuing.
How mad are people still like Barry Sanders, who was like, “I’m great. I’m going to go now. Thanks a lot.”
He quit too early, but he’s still walking. He’s still cognitive ability.
That’s the thing. Did he quit too early?
No, I’m saying that was the criticism of Crit Too Early because people were deprived of watching him get destroyed and beaten up.
Then you have somebody like Peyton Manning, where people are like, “He’s won too many seasons.” Then you can’t win either way. I think this is a problem. You also have this huge problem of survivorship bias.
Since I started studying this, 90% of the articles on LinkedIn or the things around, this is why, Annie Duke, all fail the survivorship bias test.
They do. Survivorship bias is that you hear about the people who survive. That’s pretty simple. I saw actually a great example.
By the way, you don’t know whether the qualities that were ascribed to their success or survival are even correct because there could have been a hundred people who failed who did the same thing?
Of course, like this, I think this is the problem. I saw a great example of survivorship bias. During the down market, somebody posted something about how they had gone out for their series A and said, “We went out for a series A, it took almost two years and like 21 rejections, but we finally got it done.” For everybody who’s thinking about giving up because you can’t raise a series A, don’t give up. This might be the one.
What? I think here’s the problem with survivorship bias. I was actually talking to David Epstein who wrote Range and The Sports Gene. He is amazing. He’s got Range Widely, which is a great Substack people should totally subscribe to. We were chatting about survivorship bias and he made a good point, which is the problem with survivorship bias is it rewards risky strategies because, if you think about like you have two strategies that both have the same average outcome. Imagine that everybody lands at the average.
Nobody’s going, “Look at that.” You’re not going to notice the survivor there. If you’re thinking about a startup or even like climbing Everest in a blizzard, the person who survives that, who finally raised as a series A, even though it was a very risky strategy and you had way too high a risk of death and you only had whatever, then everybody’s going to notice that person and think that, “I should do that too.” It’s actually rewarding dumb behavior.
Or luck. It’s like Art will say that you’re brilliant because you had a lease that expired the day before COVID hit. I’m actually giving my first graduation speech.
Congratulations. That’s cool.
Learning Early And Earning Later
Thank you. I reflected on what are some things that would have been helpful for me to hear and one of them was to learn in your twenties and earn later because I think what I’ve seen is the people who got the best mentors, the best training, the best learning if you chase money in your twenties, if you actually look at all these successful people they tend to be mentored by other people and otherwise. the best way to make money is to not focus on it early on.
I realized that sounded crazy because one of you could start a billion-dollar company, but I also looked up the odds. You’re much more likely to win a lottery or get hit by lightning every single year than start a billion-dollar company. If I were to give some advice to a group of students, it would be better to give advice that had a higher probability of success for the most part of the population.
That’s right. I think that’s exactly right. I think that what we have to do is say, given whatever the goals are that we want to achieve, I have a variety of options for how I can achieve that. The fact that I’ve started that should only matter in terms of taking into account the cost of switching. That’s all that should matter.
Whatever the goals we want to achieve, we have a variety of options for how to achieve them. The fact that we’ve started something should only matter in terms of taking into account the cost of switching. That’s all that should matter.
Otherwise, it should be given what I think my expected value is for the thing that I’m doing. How does that compare to the expected value for the thing that I’m thinking of switching to? I tell a story in the book, which I think is very demonstrative of this, a doctor named Sarah Olson Martinez, who had been like an ER doctor and administrator for fifteen years and had been miserable for a few years.
She reached out to me, actually, like when I was in the middle of writing the book, which was fortuitous timing. She had heard me on a podcast where I was talking about quitting and she wanted some advice. She had another job offer, which was to go and work for an insurance company. She couldn’t decide whether to quit her job or go and take this new job. I said, what’s going on in your job? I’m miserable. I hate it.
I’ve hated it forever.
24/7 and it’s ruining my relationship with my two young daughters because they’re like, “Why are you always on your phone and answering emails?” I don’t like the work. It went on and on.
You’re like, “Why wouldn’t you quit this even without a new job? Why is that less risky? It seems every day you stay in that it’s risky.”
I said to her, “This is a new job, so why aren’t you switching to it? Why are you worried about taking it?” I thought this was so profound. She said, “What if I take that new job and I hate that one too?” This is like very classic risk aversion. We have the aphorism that you’d rather be the devil. It’s that ambiguity aversion and risk aversion. that, we don’t like stepping into the unknown.
I actually took it right back to expected value with her. I said, “Imagine it’s a year from now and you’ve stayed in the position that you’re in. What’s the probability that you’re happy?” She said, “Zero. I know, I’ve been unhappy for five years.” I was like, “You’re worried that maybe you’re going to hit this new job. Let’s imagine it’s a year from now and you’ve taken this new job. What’s the probability you’re happy?”
She said, “The problem I have is that I’m not sure because I’m not in the job yet.” I said, “I know, but take your best guess.” She goes, “I think I’m like half the time, like 50% I would be happy.” I was like, “It’s 50% greater than zero.” I think it’s me and this like, “Yes.” She actually quit the next day. That was it, she was done.
Have you done the one-year follow-up with her?
Yeah. She’s happy in her new job, but she doesn’t have to be. That’s the thing because that would result in
Knowing When To Quit
Can you give an example in the book? I actually think, interestingly, that it talks about good quitting, but it also talks about survivor bias. Tell us the story of Slack because I think it’s interesting that it’s a great story. I also think there are probably 1,000 companies that have tried the Slack pivot but haven’t made it work.
Let me be clear, though. I’m super clear in the book that it doesn’t matter what happened. It’s a demonstration.
It was independently.
Exactly. It’s a demonstration that there are opportunity costs.
I knew this story, but most people don’t. I think it’s a great story. Why don’t you share it?
Stewart Butterfield has always dreamed of creating a multiplayer online cooperative world-building game. He had actually made an attempt back in the late ‘90s with a game that was somewhat popular, but then the dot-com crash happened and he couldn’t get any funding. There was functionality in the game where you could picture share because your inventory was like you had pictures of your inventory and I could share that with people. He ended up spinning that off into Flickr, which he sold to Yahoo.
He’s a startup guy. As soon as he was out of his obligations, he left Yahoo and went to try to do this again. This was in a more favorable funding environment. He started a company called Tiny Speck and created a game called Glitch. He’s great backers, Excel and Andreessen Horowitz, got $6 million in the bank, so they’re in good shape. The game was like a critic’s darling. It was Dr. Seuss meets Monty Python. People loved it. The problem, though, was that there was a big customer acquisition problem. For every one person who used the game for over 20 hours a week, 95 to 99 people came and stayed for 7 minutes and then left.
That’s a tough equation.
At the time that we come into the story, he has 5,000 diehard users. There are people who do like the game.
A VC needs to be a billion-dollar bet. He realized it wasn’t going to be a billion-dollar bet.
It’s not be a billion-dollar bet because they did a big marketing push, a big paid marketing push, and they were growing users for the six-week period during the marketing push, like 5% or 6% week over week. At the end of those six weeks, he realized that if they continued to grow at that rate, it would still be 31 weeks before they were breakeven. It was a ridiculous assumption because, at some point you’re obviously saturated in the core gaming market. What he realized was that this was not a venture-scale business. He just knew it. This was a small business that was maybe going to make somebody a little bit.
He was actually honest with himself.
He was very honest with himself. Actually, funnily enough, he himself says that he quit too late. That he should have quit before they ever did the paid marketing push because he already knew. That idea of wanting the certainty, he wanted the certainty. He didn’t want to be judged by other people, and he said he thought that people would think he was capricious or that he didn’t have grit or all that stuff. He was actually upset because he felt like he quit too late. Meanwhile, his investors and co-founders were like, “What? Why do you want to quit? We quit.”
No one ever gives the money back.
Anyway, he did give the money back. I think in the end, I did ask him, “Did they understand why you were giving the money back?” He said, “I don’t know. I explained to them my logic, but they knew there was no business if I wasn’t into it.” Anyway, so then he’s a startup guy. Two days later, actually, he said, “I have an idea for another company.” During the time that we have been developing Glitch, we’ve had this internal communication tool, which is like the best of email and instant message, and you could attach things and create group chats and stuff like that.
He was like, “It’s a great way for teams to communicate with each other. Our employees love it. Maybe that could be the product.” He didn’t have a name for it. He was like, “I got to give it a name.” He called it Searchable Log of All Company Knowledge, to SLACK. The investors actually rolled their money back over into that product. Obviously, they did very well with that. He did well with that, but the point of this book is not that he ended up with a $20 billion exit.
If you could, you would have a $20 billion exit.
It’s that Slack was right under his nose. He was developing Glitch, but if he didn’t quit Glitch, he couldn’t see Slack as the opportunity. First of all, from an expected value standpoint, Glitch was not worth pursuing because he knew he wasn’t going to get to a billion-dollar exit. Even if he quit to go work at McKinsey, that’s fine. It doesn’t matter what the outcome of that is; he’s supposed to quit, but then there was also an opportunity right under his nose that he was myopic about.
He had blinders on. He couldn’t see it because the other problem is once you’re engaged with something, you don’t see the other opportunities that are available to you, and sometimes those opportunities are actually better than the thing that you’re doing. A very simple example of that is when I am at a job, I tend not to look for other jobs.
Objectively look at it. As I said, the super interesting thing about this Slack story is I bet you that there are thousands of other businesses who are not quitting because they think they can pull off the Slack pivot, which is a one-in-a-million odds type thing.
That’s exactly right. It’s the survivorship buy. That would be a survivorship buy. I don’t want people to get that impression. I think that what people need to realize is the thing they’re working on is worth it. That’s what the core question is. if you are going to say, “It’s not worth it. I want to shut down development of that, but then I want to develop something new.” You had to do what Stewart Butterfield did, which is treat it like a brand-new startup, which is what he did, and that was fine.
He evaluated it on its own merits, and so did the investors.
Exactly, because he returned the capital to the investors. The investors didn’t have to roll it back in. He was like, this isn’t worth it, but there’s this other thing. By the way, he let most of his employees go because he treated it like a startup and that’s fine. If you’re a startup person, you want to try another product, that’s fine, but you shouldn’t do it because you want to save things. That’s not why he was doing it. He wasn’t trying to save anything.
He wasn’t trying to save it. They were independent and happened to flow. I want to double-click on this because I thought there were so many business implications of this point. I’ll say it less tactfully. We suck at making decisions when we’re in the moment and embedded in a situation. One of the things you talked about is this is for sales teams or otherwise, this decides in advance, quit criteria, and then having someone that holds you accountable to that. Talk about that, and give us an example.
Here’s the thing. What we’re trying to do is make an expected value decision. That’s unemotionally. Hopefully, math isn’t a particularly emotional thing.
Most of our decisions are clouded.
Two plus two is still in seven.
Clouded by emotions, yes.
If we know that’s what we’re trying to do. Ultimately, the decision whether to stick or quit is the thing that I’m doing, and it has a higher expected value. That doesn’t mean money. It means, in general, whatever your goals and values are, is it actually helping you advance toward those things? It could be happiness. Is that going to make you happier? Is it a higher expected value than other things you could do with that time, with less switching costs? If we know that, what we want to do is quit at a time when the expected value goes sideways.
When we’re no longer in a positive situation, the problem is that when we’re in that moment, we’re bad at it. We’re very good at rationalizing and sticking to things. Good at it, but this account is right around the corner. I had another meeting and they seemed positive. I know tomorrow is the day that I’m going to turn it around or whatever. It’s hard for us not to rationalize away this quit decision. Now, we are very good at looking back at things and saying, “I should have quit earlier. We’re very good at that.”
We’re good at Monday morning quarterback.
We’re great Monday morning quarterbacks for every week. Less survivorship bias and hindsight bias but the other thing that we’re good at is casting ourselves into the future and saying, “Here are the circumstances under which I would quit.” If we take that person who wrote me, it would be like, “I’ve been in this job with a toxic boss, and I thought I was showing character by sticking to it.” I don’t think there’s any question that before taking to the job, if I had said, “What happens if you have a toxic boss?” they would say, “I’ll quit,” even though we have the intuition that because we know that in advance when we end up in that.
When the boss is toxic, you or she might get less toxic. It might be a temporary thing.
It might change or if I stick to it, it’s going to build my character or whatever, but we know that we’re pretty good at doing that in advance. We need to add a step in and make it. It’s not that we could imagine what those signals would be that would make us walk away, but we actually have to be explicit about it. We have to write them down.
We have to commit in advance to walking away when we see those signals. Hopefully, we have someone who’s going to help us do that because people from the outside looking in can generally see our situation more clearly than we can. We’ve all experienced that where we see someone where we’re like, “Why are you still in this job? You’ve been talking about quitting for six months.
It’s horrible. you should be quitting.” When we’re the person in the job, we can’t do it ourselves. A stop loss is a very good example of using this strategy and it’s called kill criteria. A stop loss says, “I’m going to buy this stock, but I know that when I own the stock, it’s going to distort my decision-making, particularly if I’m losing money in the stock, then I’m not going to be good at it.”
Right now, I don’t have any of that problem because I don’t own the stock yet. I’m going to buy it, but then I’m going to commit to it, and the simplest form is a stop loss. If I buy it at $50, if it gets to $40 or below, I’m automatically going to sell it. That’s very simple. You could make it more getting down to the fundamentals. Here would be an example.
Let’s imagine that I am buying Bitcoin, and it’s back in 2020 or something. I’m buying Bitcoin because I believe that it will be a very good hedge against inflation. That’s my thesis. Now, it’s not about whether Bitcoin goes up or down but about whether it’s a good hedge against inflation. What I can do is set parameters for if Bitcoin goes down, which are correlated to some strength.
If it turns out not to be a hedge, then my thesis for owning it is incorrect.
As inflation goes up, as Bitcoins goes down, and there’s some durability to that relationship, and I can set what those parameters are, then I would have to sell it. What you’ll see is that people who do believe that, who don’t commit to that in advance, when it turns out not to be a hedge against inflation, they’ll give you reasons like, philosophical or whatever. If you actually write those things down, you’ll be much better at walking away. I did this with a sales team.
I love the sales example because I think this is what every sales manager has to help the team figure out. Where do you spend the time, where do you be gritty and where do you be quitty?
The worst thing is, if you’re an amazing seller, I don’t want you working on accounts that aren’t going anywhere because you’re afraid that if you close the account yourself, somehow I’m going to think you’re like a wuss.
By the way, prospects or people tend to want what they can’t have. It’s always interesting because when the business actually can’t take on business and it’s not aligned, they’re like, “Sorry, we can’t take on clients right now.” Intuitively, they’re much more interested.
That is true, that’s very true. That is a good point. What I did was it was like 40 ICs, like 40 sellers and managers and I sent them out a prompt individually. Imagine that you got a lead through an RFP or RFI, and you were pursuing it for 6 months, it’s now 6 months later, and the deal is dead. You did not win the deal. Looking back, you realize there were early signals that you were not going to win the deal. What were those? That’s what I asked them.
They generated like a whole bunch of, it’s 40 people doing it independently, and they generated a whole bunch of ideas. Here’s an example of what they said. This was a software-as-a-service company. They said that the prospect didn’t want to demo or want to know anything about the product. They wanted the price. They wanted to price. I asked for a rationale and the rationale was, what means we’re a stocking horse? probably they’re trying to beat somebody else down on price.
RFP is that you have to go get three quotes for whatever it is.
That was one. I said to them, “How often are you still pursuing deals when that’s the case?” They go, “100% of the time.” That became a kill criteria, which was if they don’t want a demo and they’re asking about price, you walk away and say, “Thanks, no thanks.” Another one was that they’d had several meetings and they couldn’t get a decision-maker in the room. It was always junior people.
With that one, it wasn’t, “You’re going to kill the deal right now.” It was offered executive alignment for the next meeting. “We’ll bring an executive from our side, you bring an executive from your side, and let’s see how it goes.” If they say no to that, then you would kill. You would do that. Another one was that the RFP was clearly written with a competitor in mind. The thing was that people would be pursuing these for months and months.
They don’t see the cost.
No, they don’t understand the cost at all. Why is that still in your funnel? First of all, that’s bad for budgeting purposes because the probability of closing is zero. That’s not what it’s going to be called.
If you’re working on it, you’re not going to put the probability as zero, so you’re going to put the probability as at least 30%.
That’s right when it’s actually zero. Now, your pipeline is incorrect, and that’s time you can’t spend on something that actually might be valuable. Anyway, so we created a list of kill criteria, and then in terms of accountability, what happens is that those kill criteria now become part of deal review because part of the problem for sellers who are naturally gritty people is that they want to win. That’s very competitive.
They want to win. What this does is it allows them to win in two different ways. One way they can win is by closing the deal. The other way they can win is actually by following the kill criteria because now, if they go into deal review, it’s like, “What’s going on with this deal that you’ve been pursuing for three months?” They’re like, “I still can’t get a decision-maker in the room.” The sales managers are like, “I’m sorry. Should I review the kill criteria with you?”
Rethinking Goal Setting And The True Costs Of Achievement
Incentives and behavior. This ties to something else you talked a lot about. We tend to underestimate the value of making progress towards goals rather than this binary of complete versus incomplete framing. Losing a deal is a good thing in some contracts. The faster you get to this person’s never going to sign criteria. In this environment, both personally and professionally, when we look at goals as binary, what does that cost us?
Goals are great in the sense that having goals is good. It gives you a North star. It motivates you to move toward them. I think that what happens with goals is that what we forget is that when we set a goal, it’s actually a proxy for some cost-benefit analysis that we’ve done. There are things that we’d like to achieve. There are different ways that we can achieve them. In other words, I can set. There are lots of different goals that I could set in order to achieve the thing that I’m trying to actually achieve.
Any particular route that I choose to take, which is going to be whatever the specific goal that I set is going to have costs and benefits to it. What I’m trying to figure out is which one is going to have the highest benefit given the costs that I’m willing to bear for the things that I’m broadly trying to achieve among the goals that I can set. Remember that when we set those goals, we’re setting it given not knowing the influence of luck that there might be in the future or that I’m going to learn new stuff. That would be cool.
Things happen, events that would logically change your priority. Anyway, if you didn’t change your goals after COVID hit, you might not be alive.
You might not be alive. What I always say is that the problem we have as decision-makers is that we don’t have a time machine, and we’re not omniscient. It’s like super sad because we’d be way better at this if we were. When we set a goal, we’re making our best guess. Facts are going to change on the ground. If those facts change, we’d probably want to change our goals, but we don’t. I think one of the best examples of this is this woman, Siobhan O’Keefe, who was running the London Marathon in 2021.
When we set a goal, we’re making our best guess. Facts are going to change on the ground. If those facts change, we’d probably want to change our goals, but we don’t.
She broke her fibula on mile eight. She’s very painful. The medical staff told her to quit. She didn’t, she kept running. She actually finished the marathon. Now, why did she do that? That’s the question. Why did she do that? The reason is because there was a goal, because there was a finish line. If there was no finish line, she wouldn’t have kept running. She was in horrible pain, but there was a finish line. Now, why do I know the finish line was the problem? She didn’t stop running after 13.1 miles.
She didn’t keep running past 26.2 miles. She stopped running at exactly what the goal was, which was the finish line. If we think about not the problem of running in that type of excruciating pain but also the opportunity costs. She may be costing herself the ability to ever run another race again. It’s totally obvious if you asked her before the race or if you came to her in the future.
If you broke your leg, you’re going to continue.
She would say, “No. I’m from the future. You’re going to break your leg. Do you want to start the race? She was going to start the race. I’m going to break my leg.” It’s the fact that she was already in it. She had already started. She had a finish line. We think there’s so much value in getting to the finish line but what we have to realize is that it may not be the right finish line anymore. I think that kill criteria become so important because they basically create a set of unlesses for a goal, which is, “Look, I’m going to try to finish this marathon unless.”
I’m going to continue to develop this product unless. I’m going to stay in this job unless. That’s what those kill criteria are. I’m going to keep climbing up this mountain unless there’s a blizzard or it’s foggy. If my visibility gets below some distance in front of me, I will turn around. Those unlesses allow us to both get the benefit of having a goal that we’re mostly going to stick to, but when we see those unlesses, we can actually get ourselves to walk away from them.
I think if anyone, personally or any organization, thinks about that, it would help them to make better decisions. I know you mentioned earlier your relationship with Daniel Kahneman, who the world lost this year, and you wrote something paying tribute to him. What were some of the most important things that you learned from working with him over the years?
I don’t know that I’ve ever met someone more willing to say that he was wrong. He was truly, like in the real sense of the word, very humble. His ability to say, “Yeah, I was wrong about that.” This is work that he’s published.
True scientist.
It was funny. I was teaching a class and someone came into the class, obviously a little bit with a chip on their shoulder with a list of all the things that didn’t replicate in Kahneman’s book, Thinking Fast and Slow. The thing was, I already had talked to Danny about that. He was like, “If I had to write it over again, I wouldn’t have this in there. I wouldn’t have this in there.” I went in particular, and he was quite chagrined about the chapter on priming, which turns out not to replicate, but he was the first one to tell you that.
Here I am, like me, and he’s him, and he would like to ask for my opinion on things. want to know stuff from me. He was so open-minded, and he felt like he had something to learn from everybody he talked to. He was so willing to change his mind. We need people who are excited to change their minds these days. We do need more people like him. He was like such a lovely human.
Last question. What’s a personal or professional mistake that you’ve made that you’ve learned the most from? You can go poker or not poker on this one.
Many things. I think that as time has gone on, I, too, have become much more open-minded to the idea that I’m wrong about a lot of things. I had to learn that early on. You have to listen to other people. Even the worst poker player in the world has something to teach you. They probably do something better than you. When I was young, I spent a lot of time wanting to prove that I was right, and that was stupid because you’re right about so little in your life. I think that was a big one. I would say that’s both professional and personal. There you go. I’ve answered both with one thing because that’s true, whether it’s your professional life or personal life, that hurts you.
It’s interesting. It blends a little bit to something we talked about before and I’ve seen this and specific instances come to mind with people. When you get things right a couple of times in a row, and particularly if there’s some good luck or timing, you tend to make some huge mistakes after that.
That’s very true.
It’s like, again, you get the hot hand and you bet big and it works, and then it almost actually sets you up for some pretty bad.
That’s very true.
Annie, where can people learn more about you and your work in the books?
If you go to AnnieDuke.com, that’s a good resource for everything. I have a Substack called Thinking in Bets. I teach a class on Maven, which is fun. Great platform. My next cohort is launching in September of 2024. I think those are the main places where you can catch me.
This has been an amazing discussion. I’m going to quit while we’re ahead and hope that maybe we can get you to come back.
Sure.
Again, sometime in the future, but thank you very much for joining us. We could have gone on for at least another hour.
This was super fun. Thank you.
To our readers, thanks for tuning in to the show. Remember, if you enjoyed this episode, I’d appreciate it if you could leave us a review or a quick rating, as that’s what helps new users discover the show. Thanks again for your support. Until next time, keep elevating.
Important Links:
- Annie Duke
- First Round Capital Partners
- Thinking in Bets
- Quit
- Grit
- Range
- The Sports Gene
- Range Widely
- Angela Duckworth – Past Episode
- Thinking Fast and Slow
- Thinking in Bets
