Survivor Bias: You're Looking at the Wrong Holes!
Studying the successes of others is not the most important data to consider as you try to grow your own company or ascend the career success ladder. Understand Survivor Bias and failure analysis, then use it to cultivate your own secret sauce.
In this podcast episode of My Job Here Is Done we talk about survivor bias where the tendency is to study successful businesses, leaders, or outcomes and ignore the accompanying failures. However, analyzing the failures is where you will find the data you need to consider as you try to grow your own company or ascend the career success ladder.
We had planned to do a podcast on failure analysis but we didn’t have a great hook or story to tie into it. Out of the blue, our good friend and listener David Yanoshik suggested that we look at Abraham Wald’s work during WWII and how he likely saved hundreds of pilots' lives and uncountable amounts of money for the US during the war.
Well, co-host Dave is a pilot, and sure enough, that caught his attention, especially after co-host Kelli wrote the episode notes and brought the story to life. So thanks David Yanoshik, we tell you what you won for helping us at the beginning of the episode. Spoiler Alert - don’t hold your breath.
Abraham Wald, a statistician who worked in New York city was helping with the war effort during World War II, and he was on assignment with the military.
Many of the United States military bomber planes were being shot down on missions over Germany. The naval researchers knew they needed data to reduce the number of planes and pilots lost so they began to analyze the damage done to planes that safely returned from missions.
They painstakingly reviewed and diagramed the bullet holes on these planes and began to see a pattern. You can see a picture of the data used for this, here.
Most of the damage on the planes that were safely returned was done to the wings and body so the naval researchers at the time concluded that reinforcing those areas with armor would improve the likelihood of a safe return, so they began putting reinforced steel armor on those areas.
Side note: You might wonder … why not just reinforce the entire airplane? The answer is you can’t - aircraft fly by a carefully balanced equation of weight and center of gravity. Add too much overall weight and it won’t fly, and add even a little extra weight, but placed in the wrong areas, and it won’t fly.
The naval researchers we’re thrilled with their scientific work, study, and solving of the problem. Until Wald stopped the process cold in its tracks after he was brought in to double-check the researcher’s theory.
He immediately pointed out the error in that thinking.
The researchers only looked at the planes that returned from missions, not the ones that were shot down. He looked at the data in a different way by analyzing both the planes that were shot down, and the ones that were not, and he came up with another solution - the successful solution.
Since the returning planes, the ones who survived, did not have damage to the places like the cockpit, the engines, and parts of the tail - he concluded those areas should be reinforced - not where the bullet holes were found on the planes that safely returned and were not shot down.
Brilliant - and it worked - and Survivorship Bias theory was born.
My Job Here Is Done takes a look at how survivor bias relates to business and why we don’t often look at and deeply analyze failure. We notice it’s too inconvenient to study because the data is often hidden and hard to find. But we also highlight how important it is to create the elusive and hard to define mixture of success we all call secret sauce.
Co-host Kelli offers some sobering advice in this episode: “If you’re not constantly troubled by the data you’re getting, you are looking at the wrong data.”
Most businesses and leaders that have failed, either suffered a single identifiable major mistake that is easy to find, or they succumbed to Death by a Thousand Paper Cuts - history shows failures are almost always at those two extremes.
To succeed, to grow, to maintain that competitive edge, you must always be reinventing. Think how important that thought is when you look at companies long loved and now forgotten. Toys R Us, Palm Pilot, Pan Am, Woolworth’s, Howard Johnsons, Radio Shack. All of these companies could have survived - actually should have survived - but they didn’t - ask why?
Here are some other links on the subject - Bullet Holes & Bias: The Story of Abraham Wald Survivorship Bias - The tale of Forgotten Failures. and for the more technical minded - The Mathematician Who Helped Win WWII
Here is a more in-depth article: Why do we misjudge groups by only looking at specific group members?
Visit the My Job Here Is Done website to listen to all of the episodes and learn how you can work with us if you like our thoughts and ideas.
"Survivor Bias: You're Looking at the Wrong Holes!"
My Job Here is Done™ Transcript (for general use only – machine-generated and it may not be accurate.)
NOTE: This transcript and the audio portion of this episode contain colorful language that may not be appropriate for all audiences.
Kelli (00:00) Hey, Dave, during a war, why do some airplanes come back from missions and land safely while others go missing?
Dave (00:06) Can I get you another Crème de Menthe Frappé Kelli? What are you talking about?
Kelli (00:14) I'm nearly completely sober. What do you think?
Dave (00:20) Well, I imagine some airplanes get shot down and others don't.
Kelli (00:24) Yes, and the ones that come back were shot up, too, right? Riddled with bullet holes, but those kept on flying. Why?
Dave (00:31) Ahhhh Kelli, we do a business growth and career success podcast. If we don't stay on topic, how are we going to survive?
Kelli (00:39) Exactly! To survive in business, we need to understand Survivor Bias.
Dave (00:45) I don't think I'm biased about survival. I'm pretty clear. I want that all the time.
Kelli (00:50) Well, that's good. But to increase your chances, let me tell you about Abraham Wald before you fly on your next business trip.
Intro (00:58) Hi, I'm Dave, and I'm Kelli, and this is my job here is done. If you really want that next promotion or you're a rising star entrepreneur, we have some stories to tell that will absolutely help you. I've been starting and running businesses all my life and I've worked for the man like a dog for decades. Together, we'll share stories, ideas and notions that will help you absolutely soar past that cruiser sitting next to you. And if you're grinding forward with your growing business, we know where the landmines are. Let's find them. Hey, it's only about 20 minutes. What do you have to lose? Nothing or everything.
Kelli (01:44) Thanks for joining us today. I'm Kelli.
Dave (01:46) And I'm Dave, the pilot of this bullet-ridden airplane Kelli is referring to. Welcome to the podcast. Before we get started, just a quick reminder that you can learn more about the podcast, listen to all of the previous episodes, read some of our bonus blog material, and interact with us at our website. www.myjobhereisdone.com. And on social media at myjobpodcast.
Kelli (02:10) This is the one about survivor or survivorship bias. I started off by referencing warplanes and of course, that confused my cohost a little bit because Dave is a pilot and I may add, easily distracted by anything with an engine. And I like screwing with his head occasionally.
Dave (02:30) Well, I'm glad that gives you a little bit of pleasure, Ms. Kelli. I had some time to process this now, and I get what you want to talk about today. Here's your other Crème de Menthe Frappé, sip on it a while and I will start us off.
Dave But first, thanks to our dear listener - okay, not like a deer. No, that would be stupid deer from the last episode, our dear friend and listener David Yanoshik, who pointed us to the story that we're going to tell today. For your suggestion, David, you will win the $1 million prize, the new Tesla Model Four-, and a-year supply of keto-friendly cauliflower rice, and that's just as soon as we get enough money to be able to afford that kind of thing for you. But it's yours. You're the winner.
Kelli (03:24) Don't hold your breath.
Dave (03:26) Okay, let's get right into this. You may have heard of the term Survivor Bias before, but just the words survivor and bias put together doesn't really help tell a story or allow for that A-HA moment we're always striving for. But once you get it, it makes a lot of sense. Consider a survivor, or a winner, any person, event, or endeavor that turns out well in some regard, and then contrast that with a succumber or a loser, as a person, event or endeavor that failed miserably.
Kelli (04:00) We naturally and incorrectly study the survivors or winners when we look to make our own improvements in outcomes. But forgetting the succumbers or losers as valid points of data.
Dave (04:14) Hence the bias towards survivors, right?
Kelli (04:16) Yes, exactly. Simply put, survivor bias is the tendency to study successful outcomes and ignore the accompanying failures.
Dave (04:25) Okay, let's look at some real-life examples of survivor bias and how we can naturally fall into the trap of embracing it.
Kelli (04:34) How many books about epic leadership failures have you read?
Dave (04:37) I don't think I've read any, they're not very popular.
Kelli (04:41) No, nobody writes about them.
Dave (04:43) Yeah. Let's think about this for a second. We're going to the other side of the coin. We're not looking at the successes. We're trying to look at the failures. So, let's ask this.
Kelli (04:53) Why did MySpace fail?
Dave (04:55) You had a MySpace account.
Kelli (04:57) I did not. My kid did.
Dave (04:59) No, you had a MySpace. Everybody had a MySpace account.
Kelli (05:03) Okay? Everybody but me.
Dave (05:05) How about Yahoo?
Kelli (05:06) It was the search engine and King. Why did it fail?
Dave (05:10) Kelli, don't say it's because Google came along. Because that's exactly survivor bias, and you likely just fell into it as we gave those two examples here's. Survivor bias at its best. This happens often. Company B is well behind company A in all metrics sales, reputation, growth, et cetera. Company B is tempted to copy the strategy of company A. Why?
Kelli (05:36) Because that shit is working for them.
Dave (05:38) It's working for company A. But is it going to work for company B? I don't know. You have to look at the publicly available positives, copy the look and feel of their website, the way they talk, the way they sell, the way they market their products. And you know what normally happens when you just look at the obvious and you try to implement it?
Kelli (05:59) You fail.
Dave (06:00) You do. You fail. What was not included in your simple copy and paste of the competition was the succumbed facts or the lessons learned from the invisible failures, the things that happen that aren't public-facing, that actually made a huge difference on the outcome.
Kelli (06:16) Think about how some of these things go against conventional wisdom. These wildly successful business founders like Steve Jobs, Bill Gates, Mark Zuckerberg. They dropped out of college and became millionaires.
Dave (06:28) Kelli, can't you just do that? Can anybody?
Kelli (06:30) Well, no.
Dave (06:34) Or we'd have more millionaires from high school and college drop out.
Kelli (06:37) Everybody would be a millionaire, right?
Dave (06:39) Well, that's not what we're taught. Remember, this breaks all the rules. Go to school, do good, complete your education, and you have a chance at making your fortune. These guys didn't.
Kelli (06:50) Was it all just luck?
Dave (06:52) I don't know. Let's name-drop a few businesses that had it all going for them but shouldn't have failed if you kind of think about it. Okay, you start off.
Kelli (07:01) All right. How about Blockbuster versus Netflix? Netflix didn't kill them. What really happened there?
Dave (07:08) If you dissect it just a little bit, one could come to the conclusion that Blockbuster did not continue to innovate.
Kelli (07:14) Right. But that's a common theme. How about BlackBerry versus Apple? Apple didn't take out BlackBerry.
Dave (07:20) No, BlackBerry took out themselves. This was a company who had the handle on corporate communications.
Kelli (07:27) They were the only game in town.
Dave (07:28) Even the President of the United States used Blackberries because the Secret Service could work with BlackBerry to secure it and make it very, very convenient for the President of the United States. Puff. Puff. Do you know what that sound is, Puff?
Kelli No,
Dave that's the sound of BlackBerry going away.
Kelli (07:44) Oh, yeah, Puff.
Dave (07:45) How about Atari versus Nintendo? If you're old enough to know video games and you don't know Atari, you need to study this one because there was nobody else out there.
Kelli (07:58) Again, the only game in town. They were the innovator.
Dave (08:01) They started the industry of video gaming and are no longer even a memory in people's minds.
Kelli (08:08) How about Borders Books versus Amazon?
Dave (08:11) Well, I can tell you this. Amazon did not put Borders Books out of business because Borders Books could have watched Amazon just a little bit and thought, hey, listen, let's start our own online site and let's do a test and see how that works. They didn't. They said this is never going to work. Let's just move on. Well, they've moved on.
Kelli (08:32) They moved on.
Dave (08:32) So Kelli has a great story to help sew this all together.
Kelli (08:36) During World War II, many US bomber planes were shot down on missions over Germany. The naval researchers knew they needed data to reduce the number of planes and pilots lost. So they began to analyze the damage done to the planes that safely returned from missions. They painstakingly reviewed and diagrammed the bullet holes on these planes and began to see a pattern.
Dave (08:59) I bet they were going to try to reinforce.
Kelli (09:02) Absolutely. Most of the damage on those planes that safely returned was done to the wings and body. So they concluded that reinforcing those areas with armor would improve the likelihood of a safe return. Problem solved.
Dave (09:15) Well, wait, as a pilot, I do know this. You just can't go putting armor across the entire plane. You just can't make it heavier.
Kelli (09:23) Right.
Dave (09:24) Or it won't get off the ground. Think of a paper airplane. Make a paper airplane, toss it in the air and the thing flies just fine, right?
Kelli (09:32) Yeah.
Dave (09:32) Now just tape a Penny on it. Just one little Penny, and then try that. You have to be careful where you put the armor.
Kelli (09:39) Exactly. There was a guy named Abraham Wald. He was a statistician. He worked in New York and he was helping with the war effort. He put the brakes on that theory.
Dave (09:49) That theory was reinforcing all of these airplanes that came back in the areas where they were shot.
Kelli (09:54) Right. He pointed out the error in that thinking. The researchers only looked at the planes that returned from missions, not the ones that were shot down. He looked at the data a different way. By analyzing both the planes that were shot down and the ones that were not and came up with another solution. Spoiler alert. Since the returning planes, the ones that survived, did not have damage to the cockpit, engine, and parts of the tail, he concluded those areas should be reinforced. Brilliant.
Dave (10:25) Now, he didn't have the actual data from the airplanes that got shot down because they're gone. What he did was he used reverse thinking, flipped it on its head, and said, but wait, let's not look at the survivors. Let's look at the ones that we can't find. There's some clue in there that's going to help us.
Kelli (10:42) This story is the perfect example of survivorship bias, how analyzing only the survivor data led to the incorrect decision.
Dave (10:52) What did we learn from the story? Well, one, to figure out a problem, you need to include those data that are not right in your face. And sometimes you just have to use your own common sense to start looking behind the curtain to see if you can find more information.
Kelli (11:07) It's an easy trap to fall into. It's not convenient to look for the non-obvious.
Dave (11:12) Yeah, it's not fun either. It's not an easy exercise to start looking for stuff that's not there, that could be important and should be included.
Kelli (11:20) And it's certainly a shift in thinking. But most people do not do this. They ignore the ones that succumbed, the losers.
Dave (11:27) Exactly.
Kelli (11:28) Here's you’re a-HA moment for the day.
Dave (11:31) Your ability to survive is directly dependent on having your secret sauce. Let me repeat that. Your ability to survive or that of your companies is directly dependent on having your own secret sauce.
Kelli (11:51) Finally, we define what the elusive secret sauce is exactly.
Dave (11:56) Survivors have the secret sauce and they know what it is. Fact. The succumbers lost or never had secret sauce.
Kelli (12:07) Another fact. Survivors have the secret sauce. And while you can copy the obvious, the website, the collateral, the way they advertise, you don't know the ingredients in their secret sauce. And not realizing that is where you make the fatal business or career-ending mistake.
Dave (12:26) Yeah, like secret sauce is the formula for Coke, maybe the eleven herbs and spices of KFC. What goes into the Google search algorithm?
Kelli (12:39) No, those are all trade secrets. And although they might contribute to the secret sauce, they're just one part of it.
Dave (12:44) Absolutely right. The secret sauce is your own proprietary recipe, and secret sauce that is done right makes it very hard for someone else to easily replicate it. And that's the key. If you've got something that you've determined is your secret sauce and you don't let people know about it, it's really hard for somebody else to copy that magical component of your ability to survive.
Kelli (13:12) So how do you make your own secret sauce?
Dave (13:15) Well, if you're asking me, I'd first start by analyzing your failures and the failures of others as it relates to your business and do it in the context of the problem you have at hand or the big goal you're after.
Kelli (13:27) For business, everything revolves around revenue and profit. Dissect each part of the business process and ask, Do I contribute or deduct from that metric? Keep score here. It's part of the recipe.
Dave (13:38) And this is going to sound a little harsh or shock and awe, but don't mind fuck yourself. And what I mean by that is don't let the things that feel good influence you into thinking that you're going in the right direction. The dopamine rush that you get in your brain will be obfuscating the hard or unpleasant data and you're just not going to go looking for it. Don't let yourself fall into that trap, because you are now falling into Survivor bias yourself.
Kelli (14:05) Here's a pro tip. If you're not constantly troubled by the data you're getting, you're looking at the wrong data.
Dave (14:12) Wow, I love that. If you're not constantly troubled by the data you're getting, you're looking at the wrong data. Just process that for a minute. It really starts to make sense. The other thing you need to do is bring in people from the outside. Don't just count on your group of insiders. You need to bring in people from the outside that have a fresh view so you can have them help you see the things you miss. Check your ego at the door. Somebody is going to call your baby ugly. You need to hear that. And it's tough.
Kelli (14:46) It's tough because nobody wants to hear that.
Dave (14:48) Nobody does. And it is so important that you look at it and say, yeah, the baby is being called ugly, but that doesn't mean that the baby is bad.
Kelli (14:56) But sometimes the baby is ugly and then they grow up and they start getting cute, right?
Dave (15:02) I think it's actually the opposite.
Kelli (15:04) If your business is customer-centric, stop the lunacy of asking what you're doing right for your existing customers.
Dave (15:11) Right? I mean, that is such an easy thing to do.
Kelli (15:14) You're getting meaningless ingredients for your secret sauce recipe. Go into closed/lost.
Dave (15:21) First of all, we should say that not everybody may understand the term closed/lost like you do so in the sales funnel at the very end, you have closed/won like, I won the deal, or you have closed/lost and you've lost the deal. The closed/lost data is so important.
Kelli (15:42) Yes. Contact the almost customers you lost during the selling cycle. Go back to the customers who canceled. Go to the ones you pissed off.
Dave (15:52) And you've got a few of them, I'm quite sure.
Kelli (15:54) Pay them, gift them, incentivize them, whatever it takes for them to be honest with you.
Dave (15:59) Yeah. I mean, literally tell them, hurt my feelings, please.
Kelli (16:03) Right. Just be honest. It's valuable to us.
Dave (16:05) It's going to be painful, but you are going to learn a ton that you would not have known otherwise. It will cause you to make changes.
Kelli (16:12) But isn't that the point?
Dave (16:14) That is the point. Over the years, we've learned that looking at and studying the successes of others is not the most important data to consider as you try to grow your own company or ascend the career success ladder.
Kelli (16:27) Right. Your chances of survival grow exponentially when you understand failure points, failure timing and fatal flaw analysis of those who came before you.
Dave (16:38) Yeah, listen to this. Most businesses and leaders that have failed either suffered a single identifiable, major mistake that's easy to find, you don't really have to study all that much, or they succumbed to death by 1000 paper cuts. History shows it's always at those two extremes. And you have to dig a whole lot deeper in death by 1000 paper cuts than you do for that one big hairy mistake that was made.
Kelli (17:08) Yeah. That's going to take a lot more time than that glaring screw-up.
Dave (17:12) You want a taste of success? Well, you need to have secret sauce on top of it. That's the only way you're going to actually have the sustainability to succeed long term. And secret sauce is something that you create. You don't steal it from somebody else.
Kelli (17:26) A lot of times it comes from within. You are born with a gift, right? Some people have the gift of gab. Some people are very charismatic. You can look at leaders, even political leaders, and you can see what their secret sauce is. It's the same in business.
Dave (17:41) To succeed, to grow, to maintain that competitive edge, you must always be reinventing. Think about how important that thought is when you look at companies long loved and now forgotten. We have a list here.
Kelli (17:55) How about Toys R US?
Dave (17:57) No more.
Kelli (17:57) No more.
Dave (17:58) Palm Pilot - had one. Loved it. Pan Am, the quintessential airline – gone. Woolworths - That's where you got SNH green Stamps, remember?
Kelli (18:10) Yeah.
Dave (18:13) SNH green stamps gone. The place with the best clam chowder in the world.
Kelli (18:20) New England clam chowder.
Dave (18:22) Howard Johnson's. And then there's RadioShack.
Kelli (18:27) Oh, and here's our shameless plug. For more about radio shack, check out our episode called Bernie’s Batteries.
Dave (18:34) All of these companies could have survived. Actually should have survived but didn't, ask why.
Kelli (18:41) Thanks so much for listening today. If you like our podcast, please tell a friend about us. Just one friend or colleague who you think would enjoy the content and stories we share.
Dave (18:50) And if you listen on an app where you can do a five-star review or add comments like on the Apple app or the Google podcast app or Spotify, that would really help us because that's where most of the listeners are. And if they see that you like it, maybe they'll like it and the audience grows.
Kelli (19:05) And check out our website. It's easy. Myjobhereisdone.com.
Chuck Fresh (19:11) I'm the announcer guy and I sound as good as the story you just listened to. My job here is done as a podcast production of 2PointOh LLC. Thank you and your awesome ears for listening. Want to get involved? Have your own special story to share? Tell us all about it and you might get some airtime just like me. Browse over to myjobhereisdone.com squish that all together into one word and look for the My Story link. Until next time ... My job here is done.
Listen On
