Last week, Scott Woodruff of the Home School Legal Defense Association (HSLDA) used homeschool testing data from Arkansas to argue that homeschool programs in that state are superior to those offered by public schools. We have been analysing Arkansas’ testing data for several years now, and were surprised to see Woodruff reference this data. Arkansas’ homeschool testing data runs in direct contradiction of both Woodruff’s specific claims about Arkansas’ students’ scores and HSLDA’s broader claims about homeschooled students’ performance nationwide.
Arkansas has long required homeschooling families to submit test scores for certain grades; this ended in 2015 under the HSLDA-supported House Bill 1381. The Arkansas Department of Education makes available homeschooling reports from 1997 to 2015 on their website. This data has limitations: The state does not collect demographic statistics on homeschoolers, and reports only the average of all students have taken the test for each grade, offering no way to know the distribution of homeschooled students’ scores scores. Furthermore, page 61 of the 2013-2014 Home School Report (the last year in which that test scores for homeschoolers were required) lists 13% of homeschoolers as “not known if they were tested or not.”
In his comments, Woodruff states that “[f]or many years, all Arkansas homeschool students were required to take standardized tests” and that these students “consistently out-scored their public school counterparts.” The actual records of the Arkansas department of education state otherwise. As seen on pages 35-54 of the 2006-2007 Home School Report (2006-2007 Home School Report), the last year a full battery of subject scores are reported, the average scores of homeschooled students and public school students varied. The mean score of the homeschooled students was often slightly better than the mean score of students attending the state’s public schools, but only slightly, and it is unclear whether this difference in scores is statistically significant or if it merely reflects underlying demographic patterns (students’ race, family income, and parental education levels were not reported).
Woodruff would have been only slightly inaccurate if he had stopped here, but he did not. Instead, he drew on two oft-misused national studies of homeschooled students’ test scores, those by Lawrence Rudner and Brian Ray, to claim that homeschooled students score 30 percentile points higher than public schooled students. At CRHE, we have frequently criticized these studies (see here and here), noting that both relied on volunteer samples and oversampled homeschooled students in two-parent, non-poor, college-educated families. But what makes Woodruff’s use of this data puzzling is that it stands in stark contradiction with the homeschool testing data Arkansas collects, which Woodruff already cited positively.
It is technically possible that homeschooled students in Arkansas score 20 to 30 percentile points lower than homeschooled students nationwide. It is more likely, however, that Arkansas’ testing data backs up what we have noted repeatedly—that Ray and Rudner’s national studies of homeschooled students’ performance dramatically oversample homeschooled students from families with background factors that correlate with high academic performance. As a result, their findings cannot be compared to the public school average (given the different demographic factors), and are not representative of the homeschool population as a whole, as the Arkansas data makes clear.
Woodruff also references homeschooled students’ higher than average SAT scores, failing to mention that the SAT is typically taken only by homeschooled students who plan to attend college, leaving out those for whom college has been rendered unobtainable. Further, CRHE recently published a study calling attention to the startlingly low number of homeschooled students who take the SAT, and highlighting concerns that homeschooled students may attend college at a lower rate than their public schooled peers.
We suspect that Woodruff felt he could play fast and loose with statistics in this piece because his readers may not know that Arkansas’ homeschool reports are public information. Unfortunately, Woodruff and the organization he represents, the Home School Legal Defense Association, have used this strategy to mislead lawmakers across the country for some time now. Only two states make available data from a wide cross-section of homeschooled students—Arkansas and Alaska—and neither state’s data backs up either Rudner’s or Ray’s findings. Until recently, data from Arkansas’ homeschool reports and Alaska’s popular homeschool charter programs have been little studied and little promoted, allowing Woodruff and others at HSLDA to make claims dramatically out of step with reality. In making these claims about Arkansas itself, Woodruff makes it clear that he cares more about claiming that homeschooling is academically superior to other methods of education than he does about the actual academic performance of homeschooled students.
Arkansas Data Contradicts HSLDA’s Claims
Last week, Scott Woodruff of the Home School Legal Defense Association (HSLDA) used homeschool testing data from Arkansas to argue that homeschool programs in that state are superior to those offered by public schools. We have been analysing Arkansas’ testing data for several years now, and were surprised to see Woodruff reference this data. Arkansas’ homeschool testing data runs in direct contradiction of both Woodruff’s specific claims about Arkansas’ students’ scores and HSLDA’s broader claims about homeschooled students’ performance nationwide.
Arkansas has long required homeschooling families to submit test scores for certain grades; this ended in 2015 under the HSLDA-supported House Bill 1381. The Arkansas Department of Education makes available homeschooling reports from 1997 to 2015 on their website. This data has limitations: The state does not collect demographic statistics on homeschoolers, and reports only the average of all students have taken the test for each grade, offering no way to know the distribution of homeschooled students’ scores scores. Furthermore, page 61 of the 2013-2014 Home School Report (the last year in which that test scores for homeschoolers were required) lists 13% of homeschoolers as “not known if they were tested or not.”
In his comments, Woodruff states that “[f]or many years, all Arkansas homeschool students were required to take standardized tests” and that these students “consistently out-scored their public school counterparts.” The actual records of the Arkansas department of education state otherwise. As seen on pages 35-54 of the 2006-2007 Home School Report (2006-2007 Home School Report), the last year a full battery of subject scores are reported, the average scores of homeschooled students and public school students varied. The mean score of the homeschooled students was often slightly better than the mean score of students attending the state’s public schools, but only slightly, and it is unclear whether this difference in scores is statistically significant or if it merely reflects underlying demographic patterns (students’ race, family income, and parental education levels were not reported).
Woodruff would have been only slightly inaccurate if he had stopped here, but he did not. Instead, he drew on two oft-misused national studies of homeschooled students’ test scores, those by Lawrence Rudner and Brian Ray, to claim that homeschooled students score 30 percentile points higher than public schooled students. At CRHE, we have frequently criticized these studies (see here and here), noting that both relied on volunteer samples and oversampled homeschooled students in two-parent, non-poor, college-educated families. But what makes Woodruff’s use of this data puzzling is that it stands in stark contradiction with the homeschool testing data Arkansas collects, which Woodruff already cited positively.
It is technically possible that homeschooled students in Arkansas score 20 to 30 percentile points lower than homeschooled students nationwide. It is more likely, however, that Arkansas’ testing data backs up what we have noted repeatedly—that Ray and Rudner’s national studies of homeschooled students’ performance dramatically oversample homeschooled students from families with background factors that correlate with high academic performance. As a result, their findings cannot be compared to the public school average (given the different demographic factors), and are not representative of the homeschool population as a whole, as the Arkansas data makes clear.
Woodruff also references homeschooled students’ higher than average SAT scores, failing to mention that the SAT is typically taken only by homeschooled students who plan to attend college, leaving out those for whom college has been rendered unobtainable. Further, CRHE recently published a study calling attention to the startlingly low number of homeschooled students who take the SAT, and highlighting concerns that homeschooled students may attend college at a lower rate than their public schooled peers.
We suspect that Woodruff felt he could play fast and loose with statistics in this piece because his readers may not know that Arkansas’ homeschool reports are public information. Unfortunately, Woodruff and the organization he represents, the Home School Legal Defense Association, have used this strategy to mislead lawmakers across the country for some time now. Only two states make available data from a wide cross-section of homeschooled students—Arkansas and Alaska—and neither state’s data backs up either Rudner’s or Ray’s findings. Until recently, data from Arkansas’ homeschool reports and Alaska’s popular homeschool charter programs have been little studied and little promoted, allowing Woodruff and others at HSLDA to make claims dramatically out of step with reality. In making these claims about Arkansas itself, Woodruff makes it clear that he cares more about claiming that homeschooling is academically superior to other methods of education than he does about the actual academic performance of homeschooled students.