This post summarizes our research review, which provides a critical analysis of Ray (2010). Click HERE to read a more in-depth version of the arguments presented.
In 2008, Dr. Brian Ray conducted a study on homeschoolers’ academic achievement using their scores on standardized tests. The results of the study were written up in two documents: a 2009 summary of the results, which was produced by HSLDA and is available on their website; and a 2010 academic article entitled “Academic achievement and demographic traits of homeschool students: A nationwide study,” which appeared in the online peer-reviewed journal Academic Leadership and may be viewed here.
These two publications are widely cited to support the claim that homeschoolers have higher levels of academic achievement than other children: Ray reports that his participants’ average standardized test score was 86% (as compared with the national average of 50%). Ray also found that his participants’ best subject was reading (where they scored in the 89th percentile) and their worst subjects were language, math, and social studies (where they scored in the 84th percentile). Students who were female; whose parents spent more than $600 per year on their educations; who had higher family incomes; whose parents had both graduated from college; who had fewer siblings; who had more structured educations; who spent more time in structured learning; and who started homeschooling younger all scored higher than other participants. Participants’ scores did not differ based on their curriculum or the amount of state regulation.
Unfortunately, Ray’s study has so many methodological problems that we can draw few conclusions from it. The most glaring methodological problems are:
1) Bias
Ray did not survey a random sample of homeschool graduates. The main method Ray used to collect students’ standardized test scores was to seek out homeschoolers who were already high achievers as participants—he gathered data from testing services where parents who were confident in their children’s ability to do well signed them up for tests; he recruited participants from state homeschool organizations filled with active and activist members for whom their children’s high academic achievement was an important part of their identity. In other words, the participants self-selected for the variable being studied. Ray’s scores are exactly what we would expect from a testing pool where only those parents who knew their children would do well volunteered to participate.
Furthermore, in many cases, the test administrators were the children’s parents. With no oversight to make sure that students were following time limits and not using outside resources, parents could influence children’s scores with impunity, or choose not to have the test graded if they thought their children did poorly. In addition, around one-third of Ray’s participants knew their children’s test scores before agreeing to participate. How many parents whose children scored poorly do you think opted out of the study?
2) Non-representative sample
Ray’s participants were a highly privileged group which excluded major segments of the population. Ray’s participants were overwhelmingly white and Christian, had married parents with college degrees, and came from families with high incomes and stay-at-home mothers. All of these factors have been shown to correlate with higher rates of academic success.
These participants were neither representative of all homeschoolers, nor of all school-aged children in the US. By failing to include less-privileged participants in his study—economically disadvantaged students, racial and ethnic minorities, immigrants, children with special needs, and those with disrupted families—Ray effectively disguised the fact that much of his participants’ academic success was likely due to background factors rather than to the educational method their parents used.
It is therefore impossible to use the test scores of this select group to claim that all homeschoolers do well academically, or that academic achievement is not a problem in the homeschooling community. We can only state that, in this limited and biased sample, most participants earned high scores on standardized tests.
3) Unanswered questions
Ray’s report of his findings provokes more questions than it answers. Ray lists four major testing services, several smaller testing services, and a number of statewide homeschooling organizations as sources for his data. He does not, however, report what percentage of his data came from each source. As such, it is impossible to tell whether his sample was biased in terms of affinity group or geography.
He also states that his demographic questionnaire was not tested for internal validity, nor did he provide a copy of the questions he posed to participants. We have no way of knowing if he introduced bias into the study by asking questions in a particular way.
Because of these problems, we can only conclude from this study that it is possible for homeschoolers to earn high scores on standardized tests. We cannot conclude that all homeschoolers do well on standardized tests—Ray’s study does not prove that homeschoolers have higher academic achievement than other children. It merely gives a description of the demographics of a particularly privileged subset of homeschoolers (composing approximately 2-3% of all homeschoolers) and an average of their standardized test scores. It cannot be used to make any claims about the relationship between homeschooling and public schooling, nor about homeschoolers as a population.
Instead, the study primarily shows Ray’s skill at manipulating data to serve his own agenda. Though the 2009 publication proudly proclaims that homeschoolers are “achieving well beyond their public school counterparts”, Ray’s peer-reviewed 2010 article acknowledges that his participants’ “test scores seem, in some ways, notably too high” to be plausible. He posits that “perhaps the present study captured a sample of the homeschool population at large that is over-represented by high achievers”, cautioning readers that his study was “not an experiment and readers should be careful about assigning causation to anything”.
What we can tell is that, even in Ray’s extremely homogeneous sample, some demographic variables still made a difference in homeschoolers’ scores, including gender, family income, parental education, and amount of educational structure. Ray’s participants also scored highest in reading and lowest in math. These findings suggest trends that might hold across broader segments of the population, though they have never been studied in a methodologically rigorous way.
Choosing the Data that Supports Your Agenda: A Look at Ray 2010
This post summarizes our research review, which provides a critical analysis of Ray (2010). Click HERE to read a more in-depth version of the arguments presented.
In 2008, Dr. Brian Ray conducted a study on homeschoolers’ academic achievement using their scores on standardized tests. The results of the study were written up in two documents: a 2009 summary of the results, which was produced by HSLDA and is available on their website; and a 2010 academic article entitled “Academic achievement and demographic traits of homeschool students: A nationwide study,” which appeared in the online peer-reviewed journal Academic Leadership and may be viewed here.
These two publications are widely cited to support the claim that homeschoolers have higher levels of academic achievement than other children: Ray reports that his participants’ average standardized test score was 86% (as compared with the national average of 50%). Ray also found that his participants’ best subject was reading (where they scored in the 89th percentile) and their worst subjects were language, math, and social studies (where they scored in the 84th percentile). Students who were female; whose parents spent more than $600 per year on their educations; who had higher family incomes; whose parents had both graduated from college; who had fewer siblings; who had more structured educations; who spent more time in structured learning; and who started homeschooling younger all scored higher than other participants. Participants’ scores did not differ based on their curriculum or the amount of state regulation.
Unfortunately, Ray’s study has so many methodological problems that we can draw few conclusions from it. The most glaring methodological problems are:
1) Bias
Ray did not survey a random sample of homeschool graduates. The main method Ray used to collect students’ standardized test scores was to seek out homeschoolers who were already high achievers as participants—he gathered data from testing services where parents who were confident in their children’s ability to do well signed them up for tests; he recruited participants from state homeschool organizations filled with active and activist members for whom their children’s high academic achievement was an important part of their identity. In other words, the participants self-selected for the variable being studied. Ray’s scores are exactly what we would expect from a testing pool where only those parents who knew their children would do well volunteered to participate.
Furthermore, in many cases, the test administrators were the children’s parents. With no oversight to make sure that students were following time limits and not using outside resources, parents could influence children’s scores with impunity, or choose not to have the test graded if they thought their children did poorly. In addition, around one-third of Ray’s participants knew their children’s test scores before agreeing to participate. How many parents whose children scored poorly do you think opted out of the study?
2) Non-representative sample
Ray’s participants were a highly privileged group which excluded major segments of the population. Ray’s participants were overwhelmingly white and Christian, had married parents with college degrees, and came from families with high incomes and stay-at-home mothers. All of these factors have been shown to correlate with higher rates of academic success.
These participants were neither representative of all homeschoolers, nor of all school-aged children in the US. By failing to include less-privileged participants in his study—economically disadvantaged students, racial and ethnic minorities, immigrants, children with special needs, and those with disrupted families—Ray effectively disguised the fact that much of his participants’ academic success was likely due to background factors rather than to the educational method their parents used.
It is therefore impossible to use the test scores of this select group to claim that all homeschoolers do well academically, or that academic achievement is not a problem in the homeschooling community. We can only state that, in this limited and biased sample, most participants earned high scores on standardized tests.
3) Unanswered questions
Ray’s report of his findings provokes more questions than it answers. Ray lists four major testing services, several smaller testing services, and a number of statewide homeschooling organizations as sources for his data. He does not, however, report what percentage of his data came from each source. As such, it is impossible to tell whether his sample was biased in terms of affinity group or geography.
He also states that his demographic questionnaire was not tested for internal validity, nor did he provide a copy of the questions he posed to participants. We have no way of knowing if he introduced bias into the study by asking questions in a particular way.
Because of these problems, we can only conclude from this study that it is possible for homeschoolers to earn high scores on standardized tests. We cannot conclude that all homeschoolers do well on standardized tests—Ray’s study does not prove that homeschoolers have higher academic achievement than other children. It merely gives a description of the demographics of a particularly privileged subset of homeschoolers (composing approximately 2-3% of all homeschoolers) and an average of their standardized test scores. It cannot be used to make any claims about the relationship between homeschooling and public schooling, nor about homeschoolers as a population.
Instead, the study primarily shows Ray’s skill at manipulating data to serve his own agenda. Though the 2009 publication proudly proclaims that homeschoolers are “achieving well beyond their public school counterparts”, Ray’s peer-reviewed 2010 article acknowledges that his participants’ “test scores seem, in some ways, notably too high” to be plausible. He posits that “perhaps the present study captured a sample of the homeschool population at large that is over-represented by high achievers”, cautioning readers that his study was “not an experiment and readers should be careful about assigning causation to anything”.
What we can tell is that, even in Ray’s extremely homogeneous sample, some demographic variables still made a difference in homeschoolers’ scores, including gender, family income, parental education, and amount of educational structure. Ray’s participants also scored highest in reading and lowest in math. These findings suggest trends that might hold across broader segments of the population, though they have never been studied in a methodologically rigorous way.