Do 12 million Americans really believe we’re ruled by shape-shifting alien lizards? Some headlines would have you believe it. Interestingly, the same number of people reportedly ingested bleach last year in hopes of preventing COVID-19, according to a survey from the CDC. That “fact” also jumped to the top of people’s newsfeeds. Surely there aren’t really so many crazies in our midst? Two recent articles in Harvard Business Review illustrate and explicate the problem with self-reported data.
Carey Smith | Founding Contrarian
Recently, my colleagues and I were discussing an article penned by some of the supposedly best and brightest business minds in academia. Harvard Business Review’s “How Venture Capitalists Make Decisions” updated a 2016 study that surveyed “the vast majority” of VC firms. The authors claimed the report “pulled back the curtain” on the mysterious levers and pulleys that help to keep the money flowing from those who have it to those who seek it.
We read it, discussed it, questioned some of the terminology and doubted many of the averrals. The VC world the authors described, based on solicited responses from select alumni and other databases, was not one we recognized, outside of those dreadful get-togethers that serve bad boxed wine and beaucoup de bluster. It certainly is not a world we want anything to do with.
The fact is, these “experts” presented what seemed to us to be a frequently inaccurate, certainly superficial rendering of VC firms’ inner workings, and the article reads as if it could have been generated by, well, shape-shifting alien lizards. That would at least explain why some of the VCs seemed to speak with forked tongues. And who could blame them for not wanting to reveal their special sauce? But perhaps the best part of all sounded as if it was lifted directly from a middle-school film made for the Future Venture Capitalists of America Club: “Once VCs have put money into a company, they roll up their sleeves and become active advisers.” Said one VC: “Our firm puts a huge amount of work into helping our companies — everything from assisting with hiring to acting as the founder’s psychologist.” Excuse me while I wipe away tears of laughter. I wonder just what they consider a huge amount of time? Pretty sure it doesn’t mean reading through a thousand résumés, just one of the many things we’ve done for our partner companies. (And that’s a thousand-plus résumés just for two job openings at just one partner company.)
Rather than revealing anything, the article’s only value seemed to be in reinforcing what we already knew: that VCs follow trends and one another, and that the business profs writing about their survey’s findings knew next to nothing about running a business. We were about to dismiss the article, with all its shortcomings, when something wonderful happened: We stumbled upon a second HBR article, this time penned by a graduate student, that addressed those same shortcomings head-on. Without making any grandiose claims, the writer proceeded to “pull back the curtain” on surveys, those crazy 12 million bleach drinkers, and “the limits of self-reported data” — probably the biggest flaw of the profs’ article.
As it turns out, the 12 million figure cited for both bleach drinkers and believers in alien-lizard overlords is not only misleading, it’s almost certainly wack. The author notes that the same specific number — 4% of respondents — has shown up in surveys so much that it has been dubbed the Lizardman’s Constant, the result of participants’ ignorance, inattentiveness or their desire to mock the proceedings. Self-reporting survey participants are no different than an unreliable narrator in a novel. You can’t trust what they’re telling you. The author’s point? Humans are unreliable, and surveys that rely on self-reported data — even the “most comprehensive survey to date” by some guys who occupy endowed chairs in academe — should always be taken with a grain of, well, definitely not bleach. And it’s up to every individual, and that includes entrepreneurs, to think critically about everything we read and hear.
Now that’s a conclusion I can definitely raise a glass to.