Originally Published as: Surveys & Data Analysis: The Art and Science of Getting Reliable Data
Jacob Prater is a soil scientist and associate professor in Wisconsin. His passion is natural resource management along with the wise and effective use of those resources to improve human life.
I love analyzing data and finding out what it can tell me. I don’t think I’m alone in this, but maybe I’m a bit more enthusiastic about it than some. Before analyzing any data, it’s important to know what you’re looking at and what it can be used for. That means you need to have an idea of how the data was collected.
Survey data can be really useful—especially if it’s collected properly—and understanding what that looks like is a valuable skill. A well-executed survey will include several things: a clear purpose, a targeted population, a good response rate, clear questions, and reasonable analysis. Short is good, too.
To explain why methodology and approach matter (and to give you a chuckle), a buddy of mine once said, “If you wake up in the morning and scratch your ass and then scratch your nose and think the world stinks, maybe you should look at what you’re doing.”
Good results from a survey require that you ask the right people the right questions. There are some classic examples of incorrect predictions based on biased sampling—like when The Literary Digest incorrectly predicted that Roosevelt would lose the 1936 presidential election because their survey sample was biased toward affluent Americans who owned cars and telephones. That mistake illustrates how easy it is to get bad results. Needless to say, if you’re missing a major—or even minor—portion of the people you need to be asking, you can end up with erroneous results. This is the essence of sampling bias.
Dealing with sampling bias in survey responses is tricky. To avoid bias, some analysts apply weighting factors to adjust for over- or under-sampling specific groups (political pollsters do this all the time—it’s a bit of hand-waving). It’s not uncommon to apply weighting factors to make results appear more “realistic.” These factors may be informed by other data at best, or simply made up at worst. While common, this kind of data massaging should only be used when you have solid demographic data to inform the weighting.
For example, say you polled 100 people and 60 said they prefer tacos to hamburgers, with 40 not responding. If other survey data shows that the entire town loves Mexican food, it’s a safe bet that opening a taco truck would be successful.
Sampling bias can also appear for other reasons. Some people just don’t like to respond to surveys. I got a kick out of an editorial in one of Shield Wall Media’s publications where Gary Reichert said he won’t share his data with the government (and I agree—when you don’t have to, why would you?). Sometimes people are cautious, and often they should be. But what if the people who don’t respond are part of a group with a different outlook from everyone else? That’s tough to correct for. The only way to predict this kind of bias is to compare historical data with past survey results.
This comparison is also a great test of validity—whether the data actually measures what you think it measures—but it can only be done in hindsight.
Since sampling bias is a problem, best practice is to know your target population and get the highest response rate possible. That means using multiple methods—email, phone, mail—to reach people. Getting a representative sample while keeping costs manageable is the trick. Experts suggest that 5–30% response rates are good, and anything over 50% is excellent. A survey in this ballpark produces good data. (Any credible survey should disclose its response rate—that’s part of the “check the methods” step you should take when interpreting results.)
Beyond interpreting survey results through past data—like comparing marketing forecasts to previous forecasts and actual performance—you can also test your data for internal consistency. A colleague of mine uses a technique where he splits his data in half at random, builds a response model with one half, then tests random samples from the other half against it.
This may not apply in all cases, but it can help determine whether your sample is representative and what sample size you need. You might decide to invest in a larger sample once, then use that to determine what’s “enough” for future surveys. More is usually better—but efficiency matters too. The goal is to ensure the data is reliable, meaning it’s reproducible.
Now, let’s say population, sampling, and bias aren’t issues—then the next challenge is asking good questions. One thing we have to face is that what people say and what they do don’t always align. Public and private opinions differ, even in surveys. Ideally, we should all strive to be the same person in every setting—at work, at home, and anonymously online—but that’s rarely the case.
For illustration, imagine asking, “How many friends do you have?” Most acquaintances won’t say “no” when asked if they’re your friend, especially not in front of others. But if you ask who would loan you a tool, a car, or pick up the phone at 2 a.m. to help in a crisis, the list gets shorter. People have different definitions of friendship and often aren’t honest with themselves about it, especially under social pressure.
This is the problem with emotionally loaded questions—they invite idealized answers. Instead, questions should be specific and grounded in real behavior. For instance, to measure how many people would help a friend in a crisis, ask respondents to describe a time they did so. If they never have, that tells you something.
People often respond with idealized versions of themselves, especially about health, exercise, or recycling. They present a “best self” that doesn’t match reality.
Other pitfalls include leading and repetitive questions. Leading questions bias responses by pointing people toward a particular answer—basically creating a survey that confirms what you already believe (confirmation bias). I’ve even seen survey results dismissed outright because the organizers didn’t like the outcome. In one case, a group ignored numerous critical responses, calling them “haters” who didn’t understand the process. My jaw was on the floor. The response rate was high—well above 30%—the population was clearly defined, and the data was consistent.
Repetitive questions can also cause confusion. Asking the same thing in different ways may make respondents think the meaning has changed. While it might seem useful to test for consistency, it often produces confounding results and should be avoided.
Once you’ve gathered your data, you still need to handle it carefully. It’s like running with scissors—it can go well or it can go very badly. A classic example of poor analysis is “New Coke.” In the mid-1980s, Coca-Cola introduced a sweeter formula after internal and external taste tests (like the Pepsi Challenge) suggested people preferred sweeter drinks. The result was a marketing disaster—not because the taste testers were wrong, but because loyal customers didn’t want a new product. The survey should have first asked if customers wanted a change at all.
It’s also possible to detect when survey results are manipulated or taken out of context. Politics is full of examples. Consider a question like, “Does housing affordability and availability matter to you?” Most people will say yes. But when results like that are used to suggest respondents support specific government policies, the analysis has gone off the rails. Either the question was poorly written, or it was intentionally misleading. This kind of thing is why many people don’t respond to surveys—and why, when interpreting results, you should always look at both the questions and the conclusions.
Another way data can mislead is through the uncertainty of innovation. You never know exactly when innovation will happen or what its effects will be—but it’s fair to assume that necessity drives innovation. Pressure and hardship often push people to change. A friend once told me that “inspiration and desperation” drive innovation—he’s right. Seeing someone else succeed (inspiration) or being forced to adapt (desperation) both spark change.
This applies to technology adoption, like robotic dairies, where need drives progress. Even if you identify the forces driving change, they don’t always predict what kind of innovation will result.
These aren’t all the issues that can arise in data collection and analysis—but understanding them goes a long way toward separating bad data and weak conclusions from solid data and reliable insights.
