The Art of Skepticism in a Data-Driven World

A book to develop your critical-thinking skills

Published: Tuesday, January 11, 2022 - 13:03

The latest book I'm reviewing is about data and critical thinking, and it often makes you laugh. Alongside Moonwalking With Einstein (Penguin, 2011), it’s in the traditional Penguin paperback size and ideal to take with you anywhere. It’s so enjoyable you'll want to complete it ASAP. The writing style is warm, human, and packed with entertaining examples.

Apologies for my language, but this book focuses on a key challenge of our times: calling bullshit on lies. To put it more diplomatically, developing the critical thinking skills to spot and challenge misinformation.

The authors of Calling Bullshit: The art of skepticism in a data-driven world (Random House, 2020) are Carl Bergstrom and Kevin West from the University of Washington. Bergstrom is a theoretical and evolutionary biologist, and West is a data scientist. Their own critical-thinking skills shine through this fun book, along with plenty of honesty about their own mistakes. More important, it's packed with heuristics, lists, and tips to help us all better spot bullshit and even call it out.

Why this book matters and who should read it

Both in the preface and the first two chapters, the authors lay out a clear diagnosis of the problem. From historical quotes to modern examples of misinformation, they make clear how often humans are misled by apparently data-based claims. That's what makes this book so pertinent for data leaders.

As the authors highlight, many people are confident in challenging and debating rhetoric or ideas from the humanities. However, once data, tables of numbers, statistics, and graphs are introduced, many people are intimated. They assume the person talking knows best or that "the numbers don't lie." This malaise is only exacerbated by the rise of the internet, self-publishing, and social media. Now every crackpot idea and misuse of data or stats can have a megaphone.

For that reason, I recommend this as a book for both data leaders and all nontechnical leaders. It's a helpful protection for all leaders and explains the concepts in a language that enables leaders to better fulfill their roles. All leaders should be able to understand, critique, and challenge information presented to them. Data leaders should also consider this book as a clarion call to their role as gatekeepers: Those who first spot and, when needed, call out misrepresentation (as well as not producing it).

What is covered?

For a small book (under 300 pages) it's usefully comprehensive on the subject, and one that I suspect will become a classic. Its 11 chapters can be roughly divided into sections that define the issue, identify six different forms of bullshit, and explain what to do about it.

After careful definitional work that helps us recognize why we are where we are, the true gold in this book is the clarity of that middle section. It brings to light common mistakes (or deliberate misrepresentation of data) and how to spot them. Some of the content is reminiscent of the checklists I've shared previously from Tim Harford and David Speigelhalter. However, the former is more about our own mindset and biases and the latter mainly an introduction to statistics. This book goes deeper and better equips us to spot these cases.

Those forms of bullshit are:
• Causality—understanding that correlation is not causation and spotting when this is misleading
• Numbers and nonsense—the concept of "mathiness" and how even some leading academics publish nonsense equations
• Selection bias—the need to think critically about data sources, wider context, and selection process
• Data visualization—a great chapter that Edward Tufte would love, including spotting "ducks" and “glass slippers”

• Big data—actually, this is more about machine learning, a similar critique to what's found in Rebooting AI (Vinatage, 2019)
• Science journals—written as a critical friend who still believes in them but calls out pitfalls of incentives in the system

The final two chapters include vital checklists to arm yourself against such a tsunami of brown stuff. The authors list six pointers to spotting bullshit in general and a checklist of 10 points to help you do so online. The final chapter focuses on diplomacy, including some wise words about not alienating others or speaking up just to try and appear clever. The authors go on to lay out what is really a moral challenge for us all: When the time is right, how can you call out bullshit in a way that helps? It’s well worth reading and practicing those tactics.

My favorite tips for better critical thinking

There's so much in this book that I fear my review won’t do it justice. But I guess that’s OK because I'd prefer to whet your appetite to read the book yourself rather than fully summarize it. However, I will call out a couple of sections that I enjoyed most.

The Data Visualization chapter was a great example of covering a lot of material (and potential bullshit) in under 50 pages. This is a good complement to Alberto Cairo's How Charts Lie (W. W. Norton & Co., 2019). Their list both confirms and builds on the pitfalls that Cairo highlighted. They cover familiar problems like misleading axes, trends, 3D, and proportional ink. They also highlight cases of Tufte’s ducks (graphics where decoration and/or aesthetics overwhelm and obscure the data) as well as the great but gruesome definition of glass slippers (data shoehorned into inappropriate visual form, like periodic tables of everything). The authors build on this with the concept of "ugly sisters" (meaningless use of analogies in schematic diagrams, e.g., parts of a unicorn form different technologies). Well called out!

But my favorite tips were from the final chapter, which offers ideas on how to call out bullshit in a way that works. They include good advice on a psychological approach. But I really liked the tactics for exposing the error. These include:
• Use reductio ad absurdum (with an example of extrapolating to future sprinters running in negative time)
• Be memorable (with an example of MRI scans on a dead salmon)
• Provide analogies (with an example from baseball for understanding Seattle traffic flows)
• Redraw data visualizations (with an example showing more data in context)
• Deploy a null model (with an example of how to compare age bands better)

I hope you found this review helpful and that it has enticed you to read the book (or buy it as a present).

What has been your experience of both spotting bullshit and calling it out? Do you have any great stories of how you've managed to expose misinformation and so protect others from being mislead? If so, I'd love to hear your stories and perhaps publish a collection of the funniest or most touching examples.

As a final thought, let me leave you with a quote that's repeated in this book. It reminds us of the scale of this challenge, especially in the age of social media, and why we need to continue the good fight:

“Falsehood flies; the truth comes limping after.”
—Jonathan Swift (1710)

First published Dec. 1, 2021, on the Customer Insight Leader blog.

About The Author

Paul Laughlin

Paul Laughlin is a speaker, writer, blogger, CustomerInsightLeader.com enthusiast, and the founder and managing director of Laughlin Consultancy.