The Numbers Do Lie

This article was originally posted on MinistryIncubators.com.

When you hear a headline that says “studies show” or “research suggests,” the “proof” it’s referring to is normally statistical significance. Statistical significance is defined as a correlation between two sets of data that has a less than 5% chance of happening randomly. In research, statistical significance is enough to “prove” a hypothesis. This framework is logical, which means that the two phenomena are probably related. It has been used for around 100 years and is widely accepted. It has several flaws, though. ¹  

Do you know there’s a statistically significant correlation (greater than 99%) between the popularity of the first name Kelsey and NASA’s budget appropriation by year? What about the correlation between the number of movies Johnny Depp starred in and the number of season wins for the Los Angeles Chargers by year, also over 99%? Or the correlation between per capita consumption of margarine and the divorce rate in Maine? It’s over 99% as well.²

All of those things are true, and there are thousands of more examples. Why? Because when you have the amount of data that we have on people, products, trends, and finances, the emergence of random patterns becomes inevitable. If you compare every food consumption pattern with every state divorce rate, there’s bound to be at least one matching pair— not because those two things are connected, but because there’s only so many shapes that data can make on a line graph. Not all data is meaningful, and not all data means what we think it means.

For example, there was, at one point, a spike in one-star online reviews for Yankee Candle Company. Specifically, a large portion of those reviews mentioned that the candles had no smell. At first glance, we might assume that this was because Yankee Candle Company had some defect in a factory, or a significant decline in performance that meant their product was no longer of the same quality. However, there is another explanation, and it needs only one big hint. The spike in one-star reviews? It was in spring of 2020.

We treat data like it’s sacred, infallible, and irrefutable, but the reality is that the questions we ask, the hypotheses we pose, often matter as much as the data does (if not more). That question, how we interpret the information of the world around us, is where knowledge, science, and useful learning actually live, not in the data itself. We have to make a decision about what we think data means for it to mean anything. 

We in church world sometimes use data as a weapon—against pastors we don’t like or cultural shifts that feel uncomfortable to us, or as a shield—against critiques of our leadership, or of our relevance. 

But data is just an invitation. We should collect data on our community, our congregation, our culture, but the work is interpreting it. It’s art, not science. It’s paint, not a painting. I wonder if we could respond to data with more curiosity, with more willingness to treat it like an invitation to look deeper. I think of all the skills we have been taught for interpreting scripture – understanding things like context, translation, and audience, considering hidden meanings, counter-readings, and the relationship between metaphor, testimony, and truth. 

So this week, as you look at all the data that surrounds your work – your attendance, your giving, your expenditures, or your context – information about mental health, unemployment, demographics, can you treat it like an invitation into curiosity, and not an answer in and of itself? 

• If your attendance is down, could you ask what other needs your community is meeting with their Sunday morning, and how they are meeting them? 
• If your expenditures are up, could you ask what kingdom work is being done with that money, and how those expenses might be reflecting your values?
• If your community is largely disconnected from the church, could that be an invitation to empathetic engagement, and not an indictment of them (or of you)? 

Data is critically important, and the data in and of itself teaches us very little. But when we can utilize our knowledge of our community, our interpretive skills, and the guidance of the Holy Spirit, we can minister and lead in a way that is truly transformative.

1. This wasn’t the place to give this reality proper treatment, but this model of statistical significance was knowingly created as a methodology to prove scientists’ pre-existing beliefs about what large data sets around human intelligence, behavior, criminality, and ethnic background would say, in service of a larger ideological concept: eugenics. Statistics’ dark history is part of the reason its core methodology deserves additional critique. Read more here. ↩︎
2. All of these come from Tyler Vigen’s hilarious “Spurious Correlations” series, which creates hundreds of these graphs using mass data sets, and often includes AI-generated “explanations” for each correlation. ↩︎