Fruit Salad Fallacies: Explaining “Control Groups”, “Random Sampling”, and “External Validity”
Research on homeschooling often uses very precise statistical concepts whose meanings aren’t always clear to all non-statistician readers. Today, we’re going to take some time and define some of these basic concepts. To keep you interested, we’re going to apply them using something completely different: fruit salad!
Fruit Salad Fallacies: Explaining “Control Groups”, “Random Sampling”, and “External Validity” written by Rachel Lazerus illustrated by Kierstyn King
1) Control Groups
Here is a peach.
And here are some blueberries.
There are a lot of things that both peaches and blueberries have in common. They’re both fruit, they’re both sweet, and they’re both delicious.
But could you describe a peach to a stranger, using only a blueberry as a reference point?
“Well, a peach is a lot bigger than a blueberry…and it’s not the same color…and it’s got fuzzy skin…”
There are so many differences that it might not come out too clearly.
…oops.
Trying to describe a peach using only a blueberry as a reference point might result in a very hairy pumpkin instead.
But, if I ask you to describe me a peach in terms of a nectarine…
“It’s about the same size and shape, and they have very similar coloring, and the main difference is that a nectarine’s skin is smooth, while a peach’s skin is fuzzy!’
…then you can describe, pretty accurately, what a peach looks like.
This is why control groups are so important in studies. When we have a control group, we have a group that looks very much like the experimental or treatment group that we want to study, but differs in just one area: the area we want to study. A nectarine and a peach are very similar—they just have one major difference. A blueberry and a peach are very different—different sizes, shapes, tastes, etc. For all that blueberries and peaches have in common, they’re not good comparisons for each other. For statistical studies to make sense, you want to make sure your groups are very similar to each other. This way, you won’t get confused by other factors.
Ideally, when we’re doing statistical studies that involve control groups, we would have two groups that are exactly the same except for one variable, so that we can measure exactly what the impact of this one variable is. Without the use of a control group, we can’t be sure that we’re actually studying the impact of that one variable.
If we’re studying the difference in test scores between homeschoolers and public school students, there are a lot of differences. Homeschoolers are more likely to be white than the average public school student. They’re more likely to be in two-parent families. They’re more likely to live in rural areas. So if we want to compare homeschoolers to public school students, instead of comparing the homeschoolers to the national average, we need to compare homeschoolers to public schoolers who share these same characteristics and can act as a control group—students who are white, in two-parent families, who live in rural areas. Comparing to the national average without using any control group is like trying to describe a peach only using a blueberry as a reference. You end up with a hairy pumpkin of statistical nonsense.
2. Random sampling
Suppose I have gone to the grocery store to get some fruit for a fruit salad. You look at my purchases and see that I bought peaches, bananas, strawberries, blueberries, and raspberries. And maybe your interest stops there, because you’re thinking about the delicious fruit salad I’m going to make now. But maybe you’re wondering why I bought these specific types of fruit, out of all of the fruits available in the store, and what (if anything) this says about my grocery store’s selection.
You could tell lots of stories about how I picked the fruit I bought. Maybe I only picked the cheapest fruits, or maybe I picked the ones closest to the front of the store. Maybe I picked my favorite fruits; maybe I picked your favorites. Maybe I deliberately selected the fruits for my salad such that there would be no green fruits.
But in all of these stories, there’s an element of choice. I selected the fruit that would go into our fruit salad. I didn’t tell you why I did it: all you know is that there must be some sort of decision-making process that went into picking the fruits at my grocery store.
Knowing this, what can you say about the fruit that are available at my local grocery store? Not much, besides the fact that these five types of fruit were there and present. Thus, the fruit that I bought from my grocery store is a non-random sample of the fruit available there.
If I were to take fruits at a completely random basis, then every single fruit in the store would have an equally likely chance of being purchased. This means I could have ended up with a fruit salad that you find disgusting—or that I find disgusting. By being selective about my fruit, I end up with what I think is a more delicious salad—but you also have much less information about what was available at the store than you would if the fruit were a random sample. And since I’m non-randomly picking only the fruits that I like, this means that my preferences are being expressed—and yours aren’t.
These kiwi were very sad that they weren’t included in the fruit salad.
Now when we’re talking about fruit, you probably already have a good idea of what’s available at a grocery store. But what if we’re talking about a population where the true parameters are more unknown, like the homeschooling community in the U.S.? If I select a sample based on only the people I already know or like, or find it easy to contact, I will not be able to use my sample to say much at all about the homeschooling community as a hole.
This is the problem with many homeschooling research studies: they use non-random samples, which means that they don’t reflect the whole of the many different facets and communities that homeschool. For a random sample to be truly random, every person in the study must have had an equal chance of being selected. When a researcher finds participants off of an organization’s email list, or by word of mouth, or through a blog, this is not the case—some individuals have a very high likelihood of being selected while others have absolutely no likelihood of being selected. When this happens, it’s called selection bias.
True randomness is hard to achieve in research studies. Getting a truly random sample takes a lot of time and effort. But without a randomly-selected sample, we don’t know what the population really looks like—we only know what the groups we’re already familiar with look like.
3. External validity.
My non-random fruit salad has now been made. Now it’s Kiery’s turn to make a fruit salad. Kiery is my friend and coworker: we are both short and are both fans of science-fiction and fantasy, so we are very much alike.
what CRHE staff do in our spare time.
Knowing what you do about me and my fruit salad, what can you say about Kiery’s fruit salad?
NOTHING!
Knowing what I like to put in fruit salad tells you nothing about what Kiery likes to put in fruit salad. This is because fruit salad has low external validity—you can’t generalize from one person’s preferences to another’s.
Having external validity means that the results can be generalized from one member of a group to all members of a group. There are some traits that do have external validity and can be generalized—knowing that Kiery and I have similar tastes in fantasy means that if one of us likes a movie, the other probably will too. Therefore, our taste in fantasy movies has external validity. But even though Kiery and I are similar in our heights and our workplace and our love of fantasy, none of that has anything to do with our preferences about fruit salad!
A major problem with a lot of studies about homeschooling is that they overstate their external validity: they think that the results have implications for people who weren’t part of the study. If a study has low external validity, it can’t be generalized: your study can only tell us about the people in your study, nothing more.
External validity is especially important when studying homeschoolers, whose individualism is well-known: each homeschooler has a story of why they’re being homeschooled, what works for them and what doesn’t, and so on. These stories can vary wildly even within a single homeschooling family. Because of this variety, it is important not to assume that what is true for one group of homeschoolers is true for another, or for all homeschoolers in general. But many studies of homeschooling outcomes which lack external validity claim to do exactly that—applying findings about, for example, an evangelical Arizona co-op or an association of hippie Vermont unschoolers to the entire homeschooling population. (At the same time, studies that do have external validity can capture really important aspects of the homeschooling community at large.) By looking at different kinds of homeschooling communities, not just self-selected homeschoolers, we can be sure that the homeschooling experience we’re describing is representing all homeschoolers, not just the groups that volunteered to be examples.
Two ways that people increase external validity in their studies are through using both random samples and control groups. This makes it more likely that the findings in the study represent the true population, and not just the sample. And to get that external validity for studies on the homeschooling population at large, it’s going to take time, energy, and access to a wide variety of homeschooling communities—even ones we’re not personally part of.
Now, just to be clear, even a study with great external validity isn’t going to say very much about the specific lived experience of your own homeschooled family. Instead, a study with external validity would tell us about the average homeschooled family. Science and statistics can only describe human behavior in probabilities, not predict it. It’s like how most of the strawberries you get at the supermarket will be average-sized, but there might be one or two really big ones or super-tiny ones in the carton you get.
Strawberries can come in all kinds of shapes and sizes — just like homeschooling families!
Once we have studies about homeschoolers with external validity, we still won’t be able to predict exactly how any one homeschooling family does, but we’ll know a lot more about what the average homeschooling family is like.
***
Research is important, and can tell us so much about the world around us. But when people use shortcuts in their research—when they don’t consider the important effects of control groups, random sampling, and external validity—the research they produce is inherently unreliable. Using a shortcut like this is like using rotten fruit in your fruit salad.
Major statistical fallacies can be found everywhere you look: in blog posts, politicans’ speeches, and even research. When put into terms of fruit salad, they seem obvious, right? But when they’re telling you information you’re already primed to believe, then they can even be seductive: you don’t want to examine that good-looking fruit too closely, in case it’s rotten inside.
When CRHE researches topics, we want you to be able to trust that we’re only using good fruit that we have carefully inspected. We know we’re doing research on topics that are very important to homeschooling parents and homeschooled students alike. We also care a lot about these topics, and we want to do them justice. When we research homeschooling studies, we take care to check the methodology of our sources and to think through the implications of their research. When we criticize a study done by someone else, even a very respected name in the field, we want you to understand what our problems are and why we’re doing this.
Fruit Salad Fallacies: Explaining “Control Groups”, “Random Sampling”, and “External Validity”
Research on homeschooling often uses very precise statistical concepts whose meanings aren’t always clear to all non-statistician readers. Today, we’re going to take some time and define some of these basic concepts. To keep you interested, we’re going to apply them using something completely different: fruit salad!
Fruit Salad Fallacies: Explaining “Control Groups”, “Random Sampling”, and “External Validity”
written by Rachel Lazerus
illustrated by Kierstyn King
1) Control Groups
Here is a peach.
And here are some blueberries.
There are a lot of things that both peaches and blueberries have in common. They’re both fruit, they’re both sweet, and they’re both delicious.
But could you describe a peach to a stranger, using only a blueberry as a reference point?
There are so many differences that it might not come out too clearly.
Trying to describe a peach using only a blueberry as a reference point might result in a very hairy pumpkin instead.
But, if I ask you to describe me a peach in terms of a nectarine…
…then you can describe, pretty accurately, what a peach looks like.
This is why control groups are so important in studies. When we have a control group, we have a group that looks very much like the experimental or treatment group that we want to study, but differs in just one area: the area we want to study. A nectarine and a peach are very similar—they just have one major difference. A blueberry and a peach are very different—different sizes, shapes, tastes, etc. For all that blueberries and peaches have in common, they’re not good comparisons for each other. For statistical studies to make sense, you want to make sure your groups are very similar to each other. This way, you won’t get confused by other factors.
Ideally, when we’re doing statistical studies that involve control groups, we would have two groups that are exactly the same except for one variable, so that we can measure exactly what the impact of this one variable is. Without the use of a control group, we can’t be sure that we’re actually studying the impact of that one variable.
If we’re studying the difference in test scores between homeschoolers and public school students, there are a lot of differences. Homeschoolers are more likely to be white than the average public school student. They’re more likely to be in two-parent families. They’re more likely to live in rural areas. So if we want to compare homeschoolers to public school students, instead of comparing the homeschoolers to the national average, we need to compare homeschoolers to public schoolers who share these same characteristics and can act as a control group—students who are white, in two-parent families, who live in rural areas. Comparing to the national average without using any control group is like trying to describe a peach only using a blueberry as a reference. You end up with a hairy pumpkin of statistical nonsense.
2. Random sampling
Suppose I have gone to the grocery store to get some fruit for a fruit salad. You look at my purchases and see that I bought peaches, bananas, strawberries, blueberries, and raspberries. And maybe your interest stops there, because you’re thinking about the delicious fruit salad I’m going to make now. But maybe you’re wondering why I bought these specific types of fruit, out of all of the fruits available in the store, and what (if anything) this says about my grocery store’s selection.
You could tell lots of stories about how I picked the fruit I bought. Maybe I only picked the cheapest fruits, or maybe I picked the ones closest to the front of the store. Maybe I picked my favorite fruits; maybe I picked your favorites. Maybe I deliberately selected the fruits for my salad such that there would be no green fruits.
But in all of these stories, there’s an element of choice. I selected the fruit that would go into our fruit salad. I didn’t tell you why I did it: all you know is that there must be some sort of decision-making process that went into picking the fruits at my grocery store.
Knowing this, what can you say about the fruit that are available at my local grocery store? Not much, besides the fact that these five types of fruit were there and present. Thus, the fruit that I bought from my grocery store is a non-random sample of the fruit available there.
If I were to take fruits at a completely random basis, then every single fruit in the store would have an equally likely chance of being purchased. This means I could have ended up with a fruit salad that you find disgusting—or that I find disgusting. By being selective about my fruit, I end up with what I think is a more delicious salad—but you also have much less information about what was available at the store than you would if the fruit were a random sample. And since I’m non-randomly picking only the fruits that I like, this means that my preferences are being expressed—and yours aren’t.
Now when we’re talking about fruit, you probably already have a good idea of what’s available at a grocery store. But what if we’re talking about a population where the true parameters are more unknown, like the homeschooling community in the U.S.? If I select a sample based on only the people I already know or like, or find it easy to contact, I will not be able to use my sample to say much at all about the homeschooling community as a hole.
This is the problem with many homeschooling research studies: they use non-random samples, which means that they don’t reflect the whole of the many different facets and communities that homeschool. For a random sample to be truly random, every person in the study must have had an equal chance of being selected. When a researcher finds participants off of an organization’s email list, or by word of mouth, or through a blog, this is not the case—some individuals have a very high likelihood of being selected while others have absolutely no likelihood of being selected. When this happens, it’s called selection bias.
True randomness is hard to achieve in research studies. Getting a truly random sample takes a lot of time and effort. But without a randomly-selected sample, we don’t know what the population really looks like—we only know what the groups we’re already familiar with look like.
3. External validity.
My non-random fruit salad has now been made. Now it’s Kiery’s turn to make a fruit salad. Kiery is my friend and coworker: we are both short and are both fans of science-fiction and fantasy, so we are very much alike.
Knowing what you do about me and my fruit salad, what can you say about Kiery’s fruit salad?
NOTHING!
Knowing what I like to put in fruit salad tells you nothing about what Kiery likes to put in fruit salad. This is because fruit salad has low external validity—you can’t generalize from one person’s preferences to another’s.
Having external validity means that the results can be generalized from one member of a group to all members of a group. There are some traits that do have external validity and can be generalized—knowing that Kiery and I have similar tastes in fantasy means that if one of us likes a movie, the other probably will too. Therefore, our taste in fantasy movies has external validity. But even though Kiery and I are similar in our heights and our workplace and our love of fantasy, none of that has anything to do with our preferences about fruit salad!
A major problem with a lot of studies about homeschooling is that they overstate their external validity: they think that the results have implications for people who weren’t part of the study. If a study has low external validity, it can’t be generalized: your study can only tell us about the people in your study, nothing more.
External validity is especially important when studying homeschoolers, whose individualism is well-known: each homeschooler has a story of why they’re being homeschooled, what works for them and what doesn’t, and so on. These stories can vary wildly even within a single homeschooling family. Because of this variety, it is important not to assume that what is true for one group of homeschoolers is true for another, or for all homeschoolers in general. But many studies of homeschooling outcomes which lack external validity claim to do exactly that—applying findings about, for example, an evangelical Arizona co-op or an association of hippie Vermont unschoolers to the entire homeschooling population. (At the same time, studies that do have external validity can capture really important aspects of the homeschooling community at large.) By looking at different kinds of homeschooling communities, not just self-selected homeschoolers, we can be sure that the homeschooling experience we’re describing is representing all homeschoolers, not just the groups that volunteered to be examples.
Two ways that people increase external validity in their studies are through using both random samples and control groups. This makes it more likely that the findings in the study represent the true population, and not just the sample. And to get that external validity for studies on the homeschooling population at large, it’s going to take time, energy, and access to a wide variety of homeschooling communities—even ones we’re not personally part of.
Now, just to be clear, even a study with great external validity isn’t going to say very much about the specific lived experience of your own homeschooled family. Instead, a study with external validity would tell us about the average homeschooled family. Science and statistics can only describe human behavior in probabilities, not predict it. It’s like how most of the strawberries you get at the supermarket will be average-sized, but there might be one or two really big ones or super-tiny ones in the carton you get.
Once we have studies about homeschoolers with external validity, we still won’t be able to predict exactly how any one homeschooling family does, but we’ll know a lot more about what the average homeschooling family is like.
***
Research is important, and can tell us so much about the world around us. But when people use shortcuts in their research—when they don’t consider the important effects of control groups, random sampling, and external validity—the research they produce is inherently unreliable. Using a shortcut like this is like using rotten fruit in your fruit salad.
Major statistical fallacies can be found everywhere you look: in blog posts, politicans’ speeches, and even research. When put into terms of fruit salad, they seem obvious, right? But when they’re telling you information you’re already primed to believe, then they can even be seductive: you don’t want to examine that good-looking fruit too closely, in case it’s rotten inside.
When CRHE researches topics, we want you to be able to trust that we’re only using good fruit that we have carefully inspected. We know we’re doing research on topics that are very important to homeschooling parents and homeschooled students alike. We also care a lot about these topics, and we want to do them justice. When we research homeschooling studies, we take care to check the methodology of our sources and to think through the implications of their research. When we criticize a study done by someone else, even a very respected name in the field, we want you to understand what our problems are and why we’re doing this.