In this edition of Author Talks, McKinsey Global Publishing’s Adam Volk chats with data scientist Seth Stephens-Davidowitz about his new book, Don’t Trust Your Gut: Using Data to Get What You Really Want in Life (Dey Street Books, May 2022). How can we make the best choices in life, love, and work? In this case, the obvious answer is the right one: research is more effective than intuition. Even still, most decision making stems from gut instinct—so Stephens-Davidowitz set out to change this. An edited version of the conversation follows.
What made you want to write this book?
There are a few motivations for writing the book. Number one: I love self-help books. In the intellectual circles I’m running in, it’s kind of uncool to love self-help. You’re supposed to like history, and philosophy, and theory—maybe novels. If you looked at my Amazon Kindle, it would just be filled with books on “how to be happier,” “how to be more successful,” or “how to date better” (the last when I was single). As much as I love self-help, as much as I am a common connoisseur of self-help, I’m kind of disappointed when I read a lot of these books, in that I don’t feel like the evidence reaches the standards that I, as a data scientist, expect they should reach.
They feel a little wishy-washy. They tell you something, and you feel good for a few seconds, but then I think, “Wait. What is that based on?” Even when they quote a study, it seems like they first have an idea, and then they just find a study to defend it. You can find a study to defend any idea you want. There are so many studies that you could cherry-pick a bad study, too. So I thought, “What happens if I spend a few years reading all kinds of studies on these big topics that I’m interested in, that everybody’s interested in, and say, ‘This is the best evidence on these topics’?” I call it “self-help for data geeks.”
I think a good principle for writing—if there are any potential writers following along—is to write the book you want to read. That definitely was how I approached this. The other motivation for this book is that since childhood I’ve been a huge baseball fan, a huge New York Mets fan. Any baseball fan knows that the game has been transformed by analytics, so it’s a totally different game than when I was a kid. Some people think it’s been ruined by analytics because the analytics are almost too good. They put the infielders in places where it’s really difficult for the hitter to get a hit. This analytics revolution was discussed in Michael Lewis’s book, Moneyball, which was then adapted into a movie with the same title.
But when you think of the big decisions in our lives—how we date, who we marry, how we parent, how we spend our time, how we pick a career—I would argue, and I think most people would agree, that the majority of people basically use their gut. You talk to a few people, you get some advice, maybe you read some of these self-help books that aren’t based on very much, and then you say, “Yeah. I’ll do this. It feels about right.”
Well, what happens if you instead took a Moneyball-for-your-life approach to these decisions? That was the motivation. Don’t Trust Your Gut is the result of this motivation to write the self-help book that I wish existed.
Well, what happens if you instead took a Moneyball-for-your-life approach to these decisions? That was the motivation. Don’t Trust Your Gut is the result of this motivation to write the self-help book that I wish existed.
What surprised you in your research and writing?
There are all kinds of things that surprised me, and I think the most surprising thing was just how much new research there is that I didn’t know about. This research that was wishy-washy is now so much less wishy-washy. In one chapter, “The AI Marriage,” I talk about this study where Samantha Joll and 85 other scientists collected data on 11,000 couples. There were hundreds of variables on everything about everybody, like what their hobbies are, what their political views are, and what their backgrounds are. They used the most advanced machine learning, these random-forest models, to try to answer questions like, “What predicts a happy marriage?” I think that is so cool. That’s arguably the most important question in life: What predicts that people are happy in relationships?
It had a little bit of a twist. There are some subtleties, but the main lesson is that happiness in relationships is incredibly hard to predict. The predictive power is way lower than you’d expect, which, when you actually think about it, really changes how you should approach looking for a mate in ways I discuss in the book.
The happiness equation
How can we use data to increase our happiness and productivity?
There are these projects that started only a few years ago. They only could be done because of recent technology: the iPhone. They ping people at different times during the day, and they say, “What are you doing? Who are you with? How happy are you?” My favorite app is Mappiness, a project by Susana Mourato and George MacKerron. They have three million data points!
They’ve done incredible stuff, and I think they’ve revolutionized our understanding of happiness, sometimes in quirky ways. One of my favorite studies of this project is that they follow sports fans’ happiness before, during, and after a sporting event of their team. Basically, if your team loses, you get eight points of pain, and if your team wins, you get four points of happiness—much less. So, on average, sports fans are getting a bad deal.
The major lesson I took from a lot of these studies came as I was describing them to my friends. I was obsessed with these studies. I was getting so excited. I would say, “You’ll never believe how much nature improves happiness,” or “You’ll never believe how much being in a beautiful environment improves happiness,” or “You’ll never believe the activities that make people happier, like how much friendship matters for happiness.”
You tell people these things, and it’s like, “No, duh,” which is sometimes how people respond to a lot of research. But then I thought about it, and I realized, “All my friends are saying these things are obvious, but if you look at how they spend their days or how they live their lives, they’re doing none of the obvious things that make people happy.” Most of them live in cities. They’re not spending time in nature. They don’t decorate their environments. They live in ugly surroundings, which makes people miserable. They work all the time, and work’s the second most miserable activity—the only thing more miserable is being sick in bed, according to another study by MacKerron, co-authored by Alex Bryson. Some people spend very little time with their friends because they’re so busy at work.
The obviousness of the happiness research is profound, and it says that these things you’re chasing don’t make people happy—the things that make people happy are incredibly simple. I end the book with the data-driven answer to life. To anybody who’s unhappy, I ask them, “What percent of your day are you doing things close to the data-driven answer to life, and what percent of the day are you not?” I actually did my own study.
Bryson and MacKerron ranked 40 activities based on how much happiness they elicit, and I worked with Spencer Greenberg, where we just asked people, “How happy do you think each of these activities makes people?” There are all these activities that people overrate the happiness of and underrate the happiness of. The other thing that we found is that over time, people, if anything, are spending less time doing the activities that make them happy. The country has gotten richer and richer, and people are spending less time doing happy activities. That’s one of the reasons that as the country has gotten richer, happiness has remained flat or even dropped.
Over time, people, if anything, are spending less time doing the activities that make them happy. The country has gotten richer and richer, and people are spending less time doing happy activities. That’s one of the reasons that as the country has gotten richer, happiness has remained flat or even dropped.
How much of a factor is luck in business and life?
Luck is a really important factor in life. I talk about art a lot because there are all these cool studies on art. With any successful artist or any successful piece of art, you see enormous amounts of luck. The Mona Lisa was just a normal painting in the Louvre, and then it was stolen one day, and it became this worldwide story of, “Who stole the Mona Lisa?” Overnight it was the most famous painting in the world, with everyone saying, “This painting’s amazing.” Everyone’s analyzing why it’s the best painting ever drawn, and it was just a total lucky thing.
So, there’s obviously a lot of luck, but then in aggregate, over a lifetime, the studies show there are definitely ways you can dramatically improve your luck. There’s this really cool study by Sam Fraiberger and others where they study successful painters over time, and they say there’s basically one thing that distinguishes people who make it in painting and people who don’t. It’s what they call “a relentless and exhaustive search for your break.”
Basically, some painters just present their art in the same place over and over again and hope that someone will find them. Other painters are traveling everywhere, and eventually, they stumble upon some gallery that they would’ve never expected to give them a huge break. After the fact, you say, “Oh. They got really lucky,” or “They’re no better than another painter who didn’t make it.” Well, actually, they did all these things that made them luckier. There are these really cool studies in many fields that show a relationship between quantity and outcome; the more you do something, the more chances you get to get lucky.
Putting more out there in the world is a way to get luckier. There are these secrets to getting luckier rather than just whining about luck. I kind of hate going against whining because whining is definitely my preferred life strategy, but I definitely do think the data show that there’s a reason that some people seem so much luckier than others.
Counterintuitive facts for successful business
What are some of the key traits that are shared by a lot of successful entrepreneurs?
There’s this study recently on the age of successful entrepreneurs. They found the median age of successful entrepreneurs is 42 years old, and that’s true even in areas such as tech, where people think that successful entrepreneurs are in college or are 20-year-olds or 25-year-olds. And this is pretty shocking: the age of success in business increases all the way up until the age of 60. I think 60-year-olds have about three times the chance of success as 30-year-olds.
All these facts go against a lot of conventional wisdom. I think that’s partially because young entrepreneurs get so much attention—in part because it’s initially counterintuitive. Mark Zuckerberg builds an empire at the age of 19 from his dorm room, and Hollywood makes a movie about it; the story gets told, and everyone’s like, “Oh. That’s really common.” Then you look at the data, and you realize, “No, no, no. Actually, that weird thing everyone now thinks is normal is actually totally abnormal.” These great stories capture our attention, and we overemphasize their prevalence.
When you look at the data, you frequently see that the story is unrepresentative—the anecdote is unrepresentative. How many 40-year-olds don’t start a business because they think, “My chance of entrepreneurship is over”? Probably tons. And the data actually says, “No. You’re 40, you’re 50, you’re 55, you’re 60—and you could still start a successful business. You have a better chance than if you were 25 or 30.” Some of these pernicious ideas come from unrepresentative stories that the data can help us overthrow.
Based on your research, what kinds of businesses tend to be most successful?
I was reading a paper by two friends of mine and a couple of other researchers for my economics PhD program. They studied anonymous data from tax records. They had data on everybody in the top 1 percent or top 0.1 percent of incomes in the United States. They asked, “Who are the rich people in the United States?” And they have a sentence that blew my mind open: “The typical rich American is the owner of a mid-sized regional business such as an auto dealership or a beverage distribution company.”
I go, “What the heck?” First of all, I don’t know about you, but I didn’t even know what a beverage distribution company was. Second of all, when I thought of a rich person, the first person that jumped into mind wasn’t an auto dealership owner. You think of a CEO, a rap star, an athlete—all these crazy ideas.
So, one of the things we learn is the value of owning versus wages. There’s a fun fact that points to this very thing: The richest NFL player in history is not the best player—not Jerry Rice, not Joe Montana, not even Tom Brady. It’s Jerry Richardson, this guy who caught 15 passes, had a very brief career, and then retired and started buying up Hardee’s franchises. He became a billionaire to the point that he eventually bought the Carolina Panthers. That shows the value of owning versus salaries. Even within owning, there are huge differences by field.
There’s a fun fact that points to this very thing: The richest NFL player in history is not the best player—not Jerry Rice, not Joe Montana, not even Tom Brady. It’s Jerry Richardson, this guy who caught 15 passes, had a very brief career, and then retired and started buying up Hardee’s franchises. He became a billionaire to the point that he eventually bought the Carolina Panthers.
Certain fields are horrible businesses. Any business that you think is sexy may very well be horrible. There’s this study of how quickly the average business in a field goes out of business, and for a dentist, it’s about 19 years. For record stores, the shortest single one lasted 2.5 years. Other businesses, like toy stores, beauty supply stores, clothing stores—they’re just awful. They’re out of business in two years, three years, four years. Everybody wants to do it, and the competition’s ferocious, so something that’s a little more boring—like a beverage distribution or auto dealership company—is probably a little better.
That’s a necessary but almost insufficient condition because there are all these boring businesses where you look at the data, and they’re not making many millionaires. There’s a whole chart of how many people in the top 0.1 percent or top 1 percent are from different fields. All these fields have tons of businesses, and nobody is in the top 1 percent or 0.1 percent—like pest cleaners, carpentry businesses, or a million other businesses. The problem is they’re getting killed by competition.
If you have a business, as soon as you have any profits, somebody else is going to come into the sector, start a business, and take away all your profits. You need some way to block that, and that’s where auto dealerships are so powerful: they’re basically legally protected local monopolies. There are all these laws about who can start an auto dealership, but once you have one, you basically have this protection against other people starting their auto dealerships. The other fields may not be legally protected, but they have something. In Don’t Trust Your Gut, I talk about other examples.
Basically, there’s a thought process you can go through that says, “What’s preventing you from someone else coming in and taking all your profits?” You want to own something where someone can’t take away your profits, which, again, is not rocket science. If you have those two questions—who are the rich people, and what’s preventing your profit from being taken— you’re way ahead of everybody else in getting rich because a lot of people don’t understand these two simple questions, and they will really help you on your path to being wealthy.
Should we ever trust our guts?
My book has the provocative title, Don’t Trust Your Gut. My last book was called Everybody Lies. One thing that is true—either in my gut or according to data—is that titles have to grab people’s attention, so that’s the rule I followed. The more accurate idea for my book may be: “Your gut can be reliable, but you should know data on different topics to help you in making decisions—don’t just wing it as most of us do.” But I think saying, “Here is some data that can help you in making decisions” isn’t quite as provocative as, “Don’t trust your gut.”
I don’t want people to read my book and say, “I can’t use my gut ever to make a decision,” or “I always need to know the data.” I’m just pointing out that there’s more data out there than has ever existed, and sometimes the data really does go against what you think. This approach of winging it in life isn’t the best one. A great example is how if you’re 40 and you think you’re too old to start a business, you can look at the data on who actually starts businesses and rethink that assumption. If you are a huge music fan and think now is time to quit your job and start a record business, at least know the data that record businesses are the quickest to go out of business.
I don’t want people to read my book and say, “I can’t use my gut ever to make a decision,” or “I always need to know the data.” I’m just pointing out that there’s more data out there than has ever existed, and sometimes the data really does go against what you think. This approach of winging it in life isn’t the best one.
Just know the data. I’m not saying you have to follow it a hundred percent, but know that data, even if it’s just Googling around for data yourself. It’s a different method of making decisions. An example is if you’re suffering from a health problem. When I was in my 20s, I’d just go to the doctor and do whatever they told me. Now, there are studies that algorithms are better than doctors on many big decisions, so before I go to the doctor, I read up and try to find out the most likely cause of my ailment—then I talk to my doctor.
It’s about being more informed about the power of data and using the data at your fingertips to help you make decisions—which is the point of my book. Another great example is the hiccups. For years, we’ve been relying on our guts and on old wives’ tales to get rid of the hiccups. I no longer support that approach. My guess is that if I do a deep dive into the literature, there will somewhere be better information on how to get rid of the hiccups.
Watch the full interview
