One of the publications most widely cited to support the claim that homeschoolers have higher levels of academic achievement than other children is Dr. Brian D. Ray’s Progress Report pamphlet, produced by HSLDA in 2009 and hosted on their website. In this pamphlet, Ray discusses the findings of his 2008 study on homeschoolers’ academic achievement, where he combined homeschool students’ test scores on a variety of standardized tests with their parents’ responses to a web survey requesting demographic information.
In his 2009 pamphlet, Ray presents his findings in a somewhat deceptive way. He does not describe his methodology or where his data came from, and he only highlights the findings that align with his beliefs about homeschooling. More complete information on Ray’s study is available, however, in the actual published, peer-reviewed version of his article, “Academic achievement and demographic traits of homeschool students: A nationwide study,” which appeared in the journal Academic Leadership in 2010 and may be viewed here.
Demographic information about Ray’s participants reveals that they are overwhelmingly white and Christian, come from wealthy, intact, well-educated families, and are largely self-selected for their ability to do well at standardized tests. Somewhat unsurprisingly, participants in Ray’s study scored on average in the 86th percentile on standardized tests. Participants who were female; whose parents spent more than $600 per year on their educations; who had higher family incomes; whose parents had both graduated from college; who had fewer siblings; who had more structured educations; who spent more time in structured learning; and who started homeschooling younger all scored higher than other participants. Participants’ scores did not differ based on their curriculum or the amount of state regulation. 59% of participants had been homeschooled for their entire lives.
Ray’s study does not prove that homeschoolers have higher academic achievement than other children. It merely gives a description of the demographics of a particularly privileged subset of homeschoolers (composing approximately 2-3% of all homeschoolers) and an average of their standardized test scores. It cannot be used to make any claims about the relationship between homeschooling and public schooling, nor about homeschoolers as a population.
In the sections that follow, I will first give some background of the study, then outline the major points made by Ray’s Progress Report and those made in his published journal article. Next I will provide a critical analysis of Ray’s study, and finally I will summarize what his results actually mean.
[expand title=”Background of the study”]
Ray’s 11,739 participants were selected from a variety of sources, including four major testing services (BJU Press, Family Learning Organization, Piedmont Education Services, and Seton Testing Services); several smaller testing services (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, and Whatcom Home School Association); and a number of volunteers from statewide homeschooling organizations (Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network).
Parents contracted with these testing services to administer standardized tests to their children, including the Iowa Test of Basic Skills, the California Achievement Test, the SAT, etc. In March 2008, these parents were sent a letter about participating in Ray’s web survey, which they chose to do voluntarily. If they did not want to fill out the survey electronically, they could request a paper version by mail. Their surveys and their children’s test scores were sent to Ray.
The statewide homeschooling organizations contacted their members by mail to inform them of the study. The members could then complete the survey and send copies of their children’s test scores to the organization, which compiled them and sent them to Ray. Note that in many cases, test administrators were the children’s parents.
Ray averaged together students’ scores from all of the standardized tests combined and compared them based on a number of demographic factors.[/expand]
[expand title=”Progress Report (2009) major points”]
Ray’s 2009 pamphlet is unequivocally glowing in its endorsement of homeschooling. As he presents his data, he argues implicitly that it is purely homeschooling, rather than any other factor, which promotes the high academic achievement he observed in homeschooled children. The findings he discusses in the pamphlet are as follows:
His homeschooled participants scored approximately 30 percentage points higher than the national average on standardized tests.
Participants’ achievement was not affected by the amount of state regulation.
Participants whose parents were certified teachers did no better than those whose parents were not.
Amount of structure in the curriculum, amount of time spent learning, and number of years homeschooled made no difference in student achievement for his participants.
Parental education and income level had an effect on participants’ achievement, but less of an effect than for public schoolers.
Participants’ academic achievement was attained at a much lower monetary cost than public schoolers’ education on average.
Male and female participants scored equally highly, unlike in public schools.
Participants whose parents knew their test scores before agreeing to participate in the study scored the same as those whose parents did not know their test scores.
In the average participant’s family, the parents are married, Christian, and have 3.5 children and a computer. The mother stays at home or works part-time while the father works full-time. Their median income is $75k-80k, which Ray states is comparable to demographically matched couples nationwide.
In an effort to determine how different the study participants were from those homeschoolers whose parents declined to participate in the study, Ray compared the average score of his participants with the average scores of all test-takers at three of the major testing services that year (a total of 22,584 students who were mostly, but not all, homeschooled). He found that his participants did not score significantly differently from the test-takers whose parents declined to participate.[/expand]
[expand title=”Journal article (2010) major points”]
Ray’s 2010 journal article, entitled “Academic achievement and demographic traits of homeschool students: A nationwide study,” is a bit more measured in its conclusions. Buried in the second-to-last paragraph is the most important idea to keep in mind about this study: “this is a nationwide, cross-sectional, descriptive study…It is not an experiment and readers should be careful about assigning causation to anything” (Ray 2010: 36; emphasis added). Due to a number of methodological problems (discussed below), this study is not representative of any population larger than the people who actually participated.
Ray’s findings about the people he surveyed are as follows.
Participants were 50.3% male; 56% were ages 9-13, and the majority of them were in grades 3-8. Participants were 91.7% white; 97.9% had homes headed by a married couple. Families had an average of 3.5 children. The largest religious groups represented were Baptists (about 22%), Independent Fundamentalist/Evangelicals (about 16%), and Roman Catholics (about 12%). Participants’ fathers had a BA or higher at a rate of 66.3% and their mothers had a BA or higher at a rate of 62.5%. Median family income was $75,000 to $79,999; 98.3% of participants used a computer at home. Median expenditure per child was $400 to $599 per year. Most participants’ fathers worked full-time, while most participants’ mothers did not work for pay. 89.4% of participants’ parents had never been certified teachers. 10.2% of participants were enrolled in a full-service home education curriculum.
45.5% of participants took the Iowa Test of Basic Skills and 44.2% of participants took the California Achievement Test. Other tests used include the Comprehensive Test of Basic Skills, the Metropolitan Achievement Test, the Stanford Achievement Test, the Terra Nova, the Test of Achievement and Proficiency, the Woodcock Johnson, and 138 tests classified as “Other”.
Participants scored on average in the 86th percentile on reading, language, math, science, social studies, core, and composite scores. Their highest score was reading (where they scored in the 89th percentile); their lowest scores were language, math, and social studies (where they scored in the 84th percentile).
59% of participants had been homeschooled for their entire lives, and all participants had spent at least 51% of their academic lives being homeschooled. Amount of time spent being homeschooled did not statistically affect students’ scores. There was a significant interaction between the effectiveness of homeschooling and a participant’s grade[1]; however, the effect was small—the difference in homeschooling’s effectiveness between grades was not very large.
Participants’ scores did not differ significantly based on whether or not they were enrolled in a full-service curriculum.
Female participants significantly outperformed males in that there was a difference between females’ and males’ average scores that was not likely to be due to sampling fluctuations. However, the effect was small; that is, females did not outperform males by very much.
Participants whose parents spent more than $600 per year on them scored significantly better than those whose parents didn’t—there was a difference in student’s average scores that was not likely to be due to sampling fluctuations. However, the effect was small.
Participants with higher family incomes scored significantly better, and this varied with participants’ grade level; however, the effect was small.
Participants whose parents had never been certified teachers scored significantly better than those whose parents had; however, the effect was small.
Participants whose parents had both graduated from college scored significantly better than those whose parents hadn’t, and “the effect of parent education is more pronounced in some grades” (Ray 2010:25). However, the effect was small.
Participants’ scores did not differ significantly in states with more or less regulations on homeschooling.
Participants with more siblings scored significantly lower; however, the effect was small.
Participants with more structured educations scored significantly better; however, the effect was small.
Participants who spent more time in structured learning scored significantly better; however, the effect was small.
Participants who started homeschooling later scored significantly lower; however, the effect was small.
Participants’ scores did not differ significantly based on whether their parents knew their scores before agreeing to participate in the study.
Participants’ scores did not differ significantly from other students contracting with the three largest testing services.
Participants’ scores did not differ significantly based on whether they contracted with the larger testing services or the smaller ones.
Ray himself states that “the test scores seem, in some ways, notably too high” (Ray 2010: 35) to be plausible. His hypotheses as to why this might be include that standardized tests have become less difficult; that public schoolers’ academic abilities have declined due to No Child Left Behind; or that “perhaps the present study captured a sample of the homeschool population at large that is over-represented by high achievers even though it was designed to sample from a broad and robust sampling of the homeschool community that uses standardized tests” (Ray 2010:35).[/expand]
[expand title=”Critical Analysis”]
Methodological problems with Ray’s (2010) study include some obfuscation of important details; the fact that the participants are not representative of the national population of students; the fact that the participants self-selected for being high achievers; and the fact that his response rate was very low.
1) Lack of crucial details
Though it contained more methodological data than the 2009 pamphlet, Ray’s published 2010 study still lacked explanation of a few important details related to the survey questions and the sample population.
First, Ray did not include a copy of the questionnaire he administered to the research participants in the published version of his study. He states that the validity of the questions—that is, their ability to measure what he was trying to measure—was verified by other homeschooling researchers:
“The questionnaires used by Ray (1990, 1994, 1997, 2000) were designed by a cooperative effort of the researcher and others who had expertise in home education and survey research in the United States… [The 2009 survey] was reviewed and revised by persons who are familiar with home education (e.g., homeschool leaders and researchers) and consensus was reached on the validity of the items and their wording.” (Ray 2010:4)
It sounds like Ray did not receive input on his questionnaire from anyone outside homeschooling—a possible source of bias in the questions. Also, it doesn’t sound like he used any sort of pilot study or data aggregation to actually test his questions’ validity—he just relied on a few (unnamed) consultants telling him they approved. And judging from his most recent survey, Ray’s questionnaires tend to be poorly written. Nothing in this account reassures me that his questionnaire in this study measured what he was trying to measure.
Second, Ray describes the source of his data as four major testing services (BJU Press, Family Learning Organization, Piedmont Education Services, and Seton Testing Services); several smaller testing services (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, and Whatcom Home School Association); and a number of volunteers from statewide homeschooling organizations he contacted (Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network). Ray does not, however, report what percentage of his data came from each source. As such, it is impossible to tell whether his sample was biased in some way—varying amounts of data from each source may bias the results based on affinity group or geography. Though Ray claims that he included respondents from all 50 US states, this is not verifiable from his article—half his respondents could have come from Virginia; there is no way to tell.
2) Non-representative sample
In many ways, Ray’s sample of homeschooled children is not comparable to the total US population of school-age children. His sample is unrepresentative in terms of age, race/ethnicity, socio-economic class, and religion. Ray himself states this: “[H]omeschool families and their students do not appear to be a completely representative cross-section of all families in the United States. And it was not possible within the constraints of this study to confirm whether this sample is representative of the population of home-educated students” (Ray 2010: 35). The fact that the sample is not representative means that Ray’s findings cannot be generalized to anyone except the people who responded to his survey.
Age
In Ray’s study, elementary and middle school-aged children are overrepresented, while high schoolers and younger children are underrepresented. In the table below, the higher blue bars for grades 3-8 show that this population made up a larger percentage of Ray’s respondents than they do in the US as a whole. The higher red bars in 9-12 show that high schoolers made up a smaller percentage of Ray’s respondents than in the general population.
This skewing is problematic because some data indicates that homeschooling grows less common as kids age (Kunzman & Gaither 2013)—only 48% of religious and 15% of secular homeschoolers continue to homeschool for more than six years (Isenberg 2007), perhaps due to the increased difficulty of high school classes. If a large number of high schoolers are quitting homeschool and going to public or private school, that may have an effect on the average test scores for those who stay. It may also be the case that homeschooling children in high school is less effective, something that could be obscured in this study by the lower numbers of homeschoolers of high school age participating.
Race/ethnicity
In the general population in 2007, the US population was 57% white, 15% black, 19% Hispanic, 4% Asian, and 4% other (this is the census data Ray uses to compare with his results). In Ray’s study, 91% of respondents were white. His 2009 pamphlet does not refer to race/ethnicity at all; these disparities are merely noted in his 2010 article, and race/ethnicity was not used as a variable when measuring the academic achievement of homeschoolers (probably because there were not enough non-white respondents to make the measurements significant).
These graphs show that the approximately one-third of US residents who are black or Latino are essentially not represented in Ray’s study. This is relevant because, due to structural inequalities in education and other aspects of society, black and Latino children tend to score approximately 5% lower than white children on standardized tests.
Some data suggests that the percentage of homeschoolers who are minorities may not be nearly as low as is represented in Ray (2010),[2] in which case the artificially high scores for the homeschoolers in Ray’s study may be inflated partly because his sample was so white. However, even if the homeschool population is disproportionately white, it is deceptive to compare their scores to those of the general population.
Socioeconomic class
There are a number of indicators of socioeconomic class which Ray collected data on—marital and employment status of the parent(s), educational level of the parent(s), number of children, and family income. In all of these measures, the homeschooling families Ray surveyed are members of a higher socioeconomic class than the average across comparable families in the US. Children with more socioeconomic advantages tend to have higher academic achievement levels. The artificially high scores for the homeschoolers in this study are inflated partly because the sample was so wealthy.
In Ray’s study, 97.9% of respondents lived in a home with married parents, while in the general population, only 71.2% of families with children under 18 are headed by a married couple. Households headed by a married couple have a much higher median income than other households—for married couples the median income is around $71,000, while for single-parent households the median income is between $32,000 and $48,000 and for non-family households the median income is around $30,000. Studies have shown that children in two-parent households achieve better academic results than children in single-parent households.
In Ray’s sample, parents with higher educational attainment are greatly over-represented. Of the homeschooling parents in Ray’s sample, 8.9% had never attended college while 64.4% had at least a Bachelor’s degree. By contrast, in the US Census in 2010, 44.1% of the adult population had never attended college while 29.9% had a BA or higher. The graph below shows the highest academic achievement levels of parents in Ray’s study and US adults overall.
Ray’s homeschooling parents were twice as likely as the average US adult to have a Bachelor’s degree or higher, and only one-fifth as likely to have never attended college. Furthermore, Ray’s participants were not representative of all homeschoolers either. NCES data suggests that approximately 50% of homeschooled students do not have a parent with a Bachelor’s degree. Research has consistentlyshown that parents’ educational attainment affects their children’s academic success; the fact that Ray’s sample is not representative of the US population in educational attainment may explain some of the difference between their children’s test scores.
In Ray’s sample, there were on average 3.5 children per household; the national average is about 2. Ray’s homeschooling participants had much larger families than the average in the US—the chart below shows that while only 20.2% of US families had three or more children, 68.2% of Ray’s participants did.
The general consensus among researchers is that children in larger families have lower academic achievement—this is consistent with Ray’s finding that his participants with larger families had lower scores. Here again, Ray’s sample is unrepresentative of the general population in terms of family size. If Ray had made an effort to control for this background factor, it may actually have widened the gap between homeschoolers’ and public schoolers’ academic abilities and allowed his homeschool participants to score even higher, increasing the evidence for his thesis.
As previously mentioned, 97.9% of families in Ray’s sample were headed by a married couple with, on average, 3.5 children. 80.6% of the mothers Ray surveyed do not work for pay, while 95.9% of his participants’ fathers are employed full-time. In 2009, the poverty threshold for families with two parents and 3.5 minor children was $27,262.75 per year. Only 4% of the families Ray sampled fall into this category, but in the US as a whole about 20% of families fell below the poverty threshold in 2009. In 2009, the median income of families headed by a married couple with two or more children where the father works full time and the mother does not work was $55,666 per year, while in Ray’s sample the median income of homeschooling families was $75,000 to $79,000 per year. The graph below demonstrates how Ray’s sample seriously undercounts poorer families (making less than $50,000 per year) and overcounts middle class and rich families.
Religion
It is not surprising that approximately 96.7% of Ray’s respondents identified as some variety of Christian (this includes Seventh Day Adventist, Assembly of God, Baptist, Roman Catholic, Eastern Orthodox, Episcopal, Independent Charismatic, Independent Fundamentalist/Evangelical, Jehovah’s Witness, Lutheran, Mennonite, Methodist, Mormon, Nazarene, Other Christian, Other Protestant, Pentecostal, Presbyterian, and Reformed). His data were drawn from three sources: 68% from major testing services[3] (of which one was BJU Press, an explicitly Christian organization); 23% from eleven minor testing services, of which all but one are explicitly Christian (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, Whatcom Home School Association, Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network); and 8% from sources which he does not identify. Supposing that each organization contributed equally to the data (which we have no way of knowing, since Ray does not tell us), fully 38% of the respondents were recruited by explicitly Christian organizations.
Only 2.6% of Ray’s respondents identified as atheist/agnostic or Other, while only .7% followed non-Christian religions. This religious breakdown is substantially different from the national one. In 2008, only 76% of American adults identified as Christian. Of the remainder, 4% identified as members of non-Christian faiths, 15% identified as atheist/agnostic or having no religion, and 5% did not disclose their religious beliefs.
This is relevant because religious involvement has been shown to be correlated with better academic outcomes. Furthermore, there are two main schools of thought in homeschooling: the “ideologues” homeschool for religious or moral reasons, while the “pedagogues” homeschool because they believe that it is the most effective educational technique. Homeschoolers whose parents have such differing views of homeschooling might have different levels of academic achievement. The inclusion in Ray’s study of only the “ideologues”—the Christians—leaves out the many secular homeschoolers and unschoolers, who—if they were included—might influence Ray’s findings.
3) Self-selected sample
Perhaps the most damning methodological problem with this study is that the participants self-selected for the variable being tested. That is, students who were more likely to score well on standardized tests were more likely to participate in the study, which measured how well students do on standardized tests. It is likely that the parents having their children tested were either those required by state law to do so, or those parents most dedicated to their children’s success who wanted to determine where their children’s academic weaknesses lie.
Ray took data from testing services where parents who were confident in their children’s ability to do well signed them up for tests. He went to state organizations filled with active and activist members for whom their children’s high academic achievement in homeschooling was an important part of their identity. He averaged their scores and found that they were higher than the national average. Does this mean that homeschoolers have higher academic achievement than the national average? Not necessarily. It means that people agreed to participate in this study because they were already high achievers.
A good comparison would be, for instance, a hypothetical study on what percentage of the population has red hair. Suppose you start by asking all the red-haired people you know to participate, and ask them to promote the study to their red-haired friends. All these redheads are eager to participate in studying the group they belong to, while people with other hair colors have more interesting things to do than fill out a survey. Suppose then you ask all the people who have agreed to participate, and you find that 60% of them have red hair. Does this mean that 60% of the population has red hair? No (in fact, only about 4% of humans have red hair). It means that people agreed to participate in your study about red hair becausethey had red hair.
Let’s look at some statistics to put Ray’s study in perspective. We might expect that mostly those parents who think their children can score well on standardized tests would sign them up to take them. In Ray (2010), students who used the three largest testing services to take a standardized test made up around 51% of the study participants.[4] The total number of students who used these three testing services in 2009 was 22,584. If we assume that the population Ray was sampling was the set of all homeschool students who use a testing service to take standardized tests, then these 22,584 students probably compose about 51% of the population he was sampling. We can therefore hypothesize that around 44,000 homeschoolers per year use a testing service to take a standardized test. In 2009-2010, approximately 1.9 million students were homeschooled in the US. In other words, Ray’s findings are relevant to the 2% of homeschoolers whose parents had them take a standardized test using a testing service in 2009. See footnote 5 for an alternate analysis that posits that Ray’s findings may instead be relevant to 3% of US homeschoolers.
Meanwhile, about 4 million students in the US started high school in 2006-2007. Of those, about 3.1 million graduated from high school in 2009-2010. Approximately 2 million students took the SAT during the 2009-2010 school year. That means somewhere from 50-65% of high school seniors that year took the SAT. The SAT is a good comparison for homeschool standardized tests because both are voluntary and are sought by the most academically dedicated.
If we assume that it is mostly the students whose parents expect them to do well who sign them up to take standardized tests, we could hypothesize that the top scoring 50-65% of students in the US took a standardized test in 2009, while the top 2-3% of homeschoolers did. The national student average on standardized tests is the 50th percentile, while Ray’s (2010) participants scored around the 86th percentile. Ray’s scores are exactly what we would expect from a smaller, more selective testing pool where only those parents who knew their children would do well volunteered to participate.
Furthermore, in many cases, the test administrators were the children’s parents. With no oversight to make sure that students were following time limits and not using outside resources, parents could influence children’s scores with impunity, or choose not to have the test graded if they thought their children did poorly. In addition, a full 27.7% of Ray’s (2010) participants knew their children’s test scores before agreeing to participate. How many parents whose children scored poorly do you think opted out of the study?
4) Low response rate
For students in the four large testing services (either 68% or 71.5% of participants in the study), the response rate reported by Ray was 19-25%.[5] This means that, of all the people Ray contacted about participating in the study, only about one in four actually completed the survey and sent in their child’s scores. For students in the smaller testing services (either 31% or 29% of participants), the response rate was 11%; only about one in ten people that Ray contacted actually completed the survey.
In social science research of this type, “a response rate between 70 and 80% is the goal, with 70% being minimally acceptable and a response rate above 80% being the ideal, although highly unlikely. If the response rate falls below 60% it becomes very difficult to claim that the survey is representative of the entire sample because responses are missing from more than one third of surveyed individuals” (Mangione 1995 and Bachman & Schutt 2007; cited in Henninger & Sung 2012:306). Needless to say, Ray’s (2010) response rates are abysmal, making this study even less representative of the population Ray was attempting to sample.[/expand]
[expand title=”What the study actually says”]
The top-scoring 2-3% of homeschooled elementary- and middle-school-age children of wealthy white Christian married couples who volunteer them for standardized tests score approximately 36 percentage points higher than the national average of all children in public schools. This is not a very meaningful result.
Even in this highly homogeneous population, participants did better in reading than math. Indicators of higher scores included high parental expenditure per child, high family income, being a child of two college graduates, having fewer siblings, having a more structured education, spending more time in structured learning, and starting schooling earlier. Though these results are not transferable outside the study, they may be indicative of larger trends and merit further research.[/expand]
[expand title=”Conclusion”]
We cannot tell how well homeschoolers perform on standardized tests from the results of this study. A host of methodological problems make Ray’s findings unrepresentative of the homeschool population—meaning that they cannot be generalized to any population larger than 2-3% of all homeschoolers—and unrepresentative of the national population of school-age children, meaning his scores cannot be compared to the scores of public schoolers.
What we can tell is that, even in Ray’s extremely homogeneous sample, some demographic variables still made a difference in homeschoolers’ scores, including gender, family income, parental education, and amount of educational structure. Ray’s participants also scored highest in reading and lowest in math. These findings suggest trends that might hold across all homeschoolers if they were studied in a methodologically rigorous way.[/expand]
Bachman, Ronet and Russell K. Schutt. 2007. The practice of research in criminology and criminal justice (3rd ed.). Thousand Oaks, CA: Sage.
Bahr, Stephen J. 2001. “Social science research on family dissolution: What it shows and how it might be of interest to family law reformers.” Presentation at the conference, “The ALI Family Dissolution Principles: Blueprint to Strengthen or to Deconstruct American Families?”, J. Reuben Clark Law School, Brigham Young University, Provo, Utah, February 1, 2001. http://fact.on.ca/Info/divorce/bahr2001.pdf
Davis-Kean, Pamela E. 2005. “The influence of parent education and family income on child achievement: The indirect role of parental expectations and the home environment.” Journal of Family Psychology 19:2. 294-304. http://www.mikemcmahon.info/ParentEducationIncome.pdf
Dubow, Eric F., Paul Boxer, and L. Rowell Huesmann. 2009. “Long-term effects of parents’ education on children’s educational and occupational success: Mediation by family interactions, child aggression, and teenage aspirations.” Merrill Palmer Quarterly 55:3. 224–249. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2853053/
Grady, Sarah, Stacey Bielick, and Susan Aud. 2010. Trends in the use of school choice: 1993 to 2007 (NCES 2010-004). Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
Henninger, Alana and Hung-En Sung. 2012. “Mail survey in social research.” Handbook of Survey Methodology for the Social Sciences, ed. by Lior Gideon, 297-311. New York: Springer.
Isenberg, Eric J. 2007. “What Have We Learned About Homeschooling?” Peabody Journal of Education 82. 387-409.
Pong, Suet-ling, Jaap Dronkers, and Gillian Hampden-Thompson. 2003. “Family policies and children’s school achievement in single- versus two-parent families.” Journal of Marriage and Family 56:3. 681-699. http://www.jstor.org/stable/3600032
Rampey, Bobby D., Gloria S. Dion, and Patricia L. Donahue. 2009. NAEP 2008 Trends in Academic Progress (NCES 2009–479). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, Washington, D.C. http://nces.ed.gov/nationsreportcard/pdf/main2008/2009479.pdf
Ray, Brian D. 2009. Progress Report 2009: Homeschool Academic Achievement and Demographics. Home School Legal Defense Association.
Ray, Brian D. 2010. Academic achievement and demographic traits of homeschool students: A nationwide study. Academic Leadership, the Online Journal 8:1.
[1] That is, the effectiveness of homeschooling varied with a participant’s grade, and this observation was unlikely to be due to sampling fluctuations.
[2] Data from the National Center for Education Statistics suggests that the 2007 homeschooling population was 77% white, 4% black, 10% Hispanic, 2% Asian or Pacific Islander, and 7% Other.
[3] Note that at one point in his article, Ray states that 71.5% of his participants were drawn from the four large testing services (p. 8), while at another point his numbers indicate that the percentage was 68% (Table 24, p. 33). The source of this discrepancy is unknown.
[4] Assuming that 68% of the participants were drawn from a major testing service and that each of the four major testing services contributed equally (17%) to the data, 51% of the participants were drawn from three of the four testing services.
[5] It is unclear how Ray calculated these numbers. The total number of customers for the three largest testing services was 22,584, while the total number of study participants from the four largest testing services was 8,039. Supposing an equal number of study participants came from each of the four testing services, there were 6,029 students in the three largest testing services. This produces a 27% response rate. It is possible that the difference between this response rate and that reported by Ray can be accounted for by presuming that a large number of parents (perhaps more than 9,000) contracted with testing services (thus receiving an invitation to participate in the study) and then did not follow through and submit their children’s tests to be graded. Note that if this is the case, the total population of parents that contract with testing services (rather than that follow through and have their children tested) may be closer to 62,000, which is around 3% of all homeschoolers.[/expand]
Dr. Chelsea McCracken
Research Analyst
Coalition for Responsible Home Education
chelseamc@responsiblehomeschooling.org
661 Washington Street #563
Canton, MA 02021
Research with an Agenda (Ray 2010)
Click here to view this document as a pdf.
One of the publications most widely cited to support the claim that homeschoolers have higher levels of academic achievement than other children is Dr. Brian D. Ray’s Progress Report pamphlet, produced by HSLDA in 2009 and hosted on their website. In this pamphlet, Ray discusses the findings of his 2008 study on homeschoolers’ academic achievement, where he combined homeschool students’ test scores on a variety of standardized tests with their parents’ responses to a web survey requesting demographic information.
In his 2009 pamphlet, Ray presents his findings in a somewhat deceptive way. He does not describe his methodology or where his data came from, and he only highlights the findings that align with his beliefs about homeschooling. More complete information on Ray’s study is available, however, in the actual published, peer-reviewed version of his article, “Academic achievement and demographic traits of homeschool students: A nationwide study,” which appeared in the journal Academic Leadership in 2010 and may be viewed here.
Demographic information about Ray’s participants reveals that they are overwhelmingly white and Christian, come from wealthy, intact, well-educated families, and are largely self-selected for their ability to do well at standardized tests. Somewhat unsurprisingly, participants in Ray’s study scored on average in the 86th percentile on standardized tests. Participants who were female; whose parents spent more than $600 per year on their educations; who had higher family incomes; whose parents had both graduated from college; who had fewer siblings; who had more structured educations; who spent more time in structured learning; and who started homeschooling younger all scored higher than other participants. Participants’ scores did not differ based on their curriculum or the amount of state regulation. 59% of participants had been homeschooled for their entire lives.
Ray’s study does not prove that homeschoolers have higher academic achievement than other children. It merely gives a description of the demographics of a particularly privileged subset of homeschoolers (composing approximately 2-3% of all homeschoolers) and an average of their standardized test scores. It cannot be used to make any claims about the relationship between homeschooling and public schooling, nor about homeschoolers as a population.
In the sections that follow, I will first give some background of the study, then outline the major points made by Ray’s Progress Report and those made in his published journal article. Next I will provide a critical analysis of Ray’s study, and finally I will summarize what his results actually mean.
[expand title=”Background of the study”]
Ray’s 11,739 participants were selected from a variety of sources, including four major testing services (BJU Press, Family Learning Organization, Piedmont Education Services, and Seton Testing Services); several smaller testing services (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, and Whatcom Home School Association); and a number of volunteers from statewide homeschooling organizations (Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network).
Parents contracted with these testing services to administer standardized tests to their children, including the Iowa Test of Basic Skills, the California Achievement Test, the SAT, etc. In March 2008, these parents were sent a letter about participating in Ray’s web survey, which they chose to do voluntarily. If they did not want to fill out the survey electronically, they could request a paper version by mail. Their surveys and their children’s test scores were sent to Ray.
The statewide homeschooling organizations contacted their members by mail to inform them of the study. The members could then complete the survey and send copies of their children’s test scores to the organization, which compiled them and sent them to Ray. Note that in many cases, test administrators were the children’s parents.
Ray averaged together students’ scores from all of the standardized tests combined and compared them based on a number of demographic factors.[/expand]
[expand title=”Progress Report (2009) major points”]
Ray’s 2009 pamphlet is unequivocally glowing in its endorsement of homeschooling. As he presents his data, he argues implicitly that it is purely homeschooling, rather than any other factor, which promotes the high academic achievement he observed in homeschooled children. The findings he discusses in the pamphlet are as follows:
[expand title=”Journal article (2010) major points”]
Ray’s 2010 journal article, entitled “Academic achievement and demographic traits of homeschool students: A nationwide study,” is a bit more measured in its conclusions. Buried in the second-to-last paragraph is the most important idea to keep in mind about this study: “this is a nationwide, cross-sectional, descriptive study…It is not an experiment and readers should be careful about assigning causation to anything” (Ray 2010: 36; emphasis added). Due to a number of methodological problems (discussed below), this study is not representative of any population larger than the people who actually participated.
Ray’s findings about the people he surveyed are as follows.
Ray himself states that “the test scores seem, in some ways, notably too high” (Ray 2010: 35) to be plausible. His hypotheses as to why this might be include that standardized tests have become less difficult; that public schoolers’ academic abilities have declined due to No Child Left Behind; or that “perhaps the present study captured a sample of the homeschool population at large that is over-represented by high achievers even though it was designed to sample from a broad and robust sampling of the homeschool community that uses standardized tests” (Ray 2010:35).[/expand]
[expand title=”Critical Analysis”]
Methodological problems with Ray’s (2010) study include some obfuscation of important details; the fact that the participants are not representative of the national population of students; the fact that the participants self-selected for being high achievers; and the fact that his response rate was very low.
1) Lack of crucial details
Though it contained more methodological data than the 2009 pamphlet, Ray’s published 2010 study still lacked explanation of a few important details related to the survey questions and the sample population.
First, Ray did not include a copy of the questionnaire he administered to the research participants in the published version of his study. He states that the validity of the questions—that is, their ability to measure what he was trying to measure—was verified by other homeschooling researchers:
“The questionnaires used by Ray (1990, 1994, 1997, 2000) were designed by a cooperative effort of the researcher and others who had expertise in home education and survey research in the United States… [The 2009 survey] was reviewed and revised by persons who are familiar with home education (e.g., homeschool leaders and researchers) and consensus was reached on the validity of the items and their wording.” (Ray 2010:4)
It sounds like Ray did not receive input on his questionnaire from anyone outside homeschooling—a possible source of bias in the questions. Also, it doesn’t sound like he used any sort of pilot study or data aggregation to actually test his questions’ validity—he just relied on a few (unnamed) consultants telling him they approved. And judging from his most recent survey, Ray’s questionnaires tend to be poorly written. Nothing in this account reassures me that his questionnaire in this study measured what he was trying to measure.
Second, Ray describes the source of his data as four major testing services (BJU Press, Family Learning Organization, Piedmont Education Services, and Seton Testing Services); several smaller testing services (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, and Whatcom Home School Association); and a number of volunteers from statewide homeschooling organizations he contacted (Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network). Ray does not, however, report what percentage of his data came from each source. As such, it is impossible to tell whether his sample was biased in some way—varying amounts of data from each source may bias the results based on affinity group or geography. Though Ray claims that he included respondents from all 50 US states, this is not verifiable from his article—half his respondents could have come from Virginia; there is no way to tell.
2) Non-representative sample
In many ways, Ray’s sample of homeschooled children is not comparable to the total US population of school-age children. His sample is unrepresentative in terms of age, race/ethnicity, socio-economic class, and religion. Ray himself states this: “[H]omeschool families and their students do not appear to be a completely representative cross-section of all families in the United States. And it was not possible within the constraints of this study to confirm whether this sample is representative of the population of home-educated students” (Ray 2010: 35). The fact that the sample is not representative means that Ray’s findings cannot be generalized to anyone except the people who responded to his survey.
Age
In Ray’s study, elementary and middle school-aged children are overrepresented, while high schoolers and younger children are underrepresented. In the table below, the higher blue bars for grades 3-8 show that this population made up a larger percentage of Ray’s respondents than they do in the US as a whole. The higher red bars in 9-12 show that high schoolers made up a smaller percentage of Ray’s respondents than in the general population.
This skewing is problematic because some data indicates that homeschooling grows less common as kids age (Kunzman & Gaither 2013)—only 48% of religious and 15% of secular homeschoolers continue to homeschool for more than six years (Isenberg 2007), perhaps due to the increased difficulty of high school classes. If a large number of high schoolers are quitting homeschool and going to public or private school, that may have an effect on the average test scores for those who stay. It may also be the case that homeschooling children in high school is less effective, something that could be obscured in this study by the lower numbers of homeschoolers of high school age participating.
Race/ethnicity
In the general population in 2007, the US population was 57% white, 15% black, 19% Hispanic, 4% Asian, and 4% other (this is the census data Ray uses to compare with his results). In Ray’s study, 91% of respondents were white. His 2009 pamphlet does not refer to race/ethnicity at all; these disparities are merely noted in his 2010 article, and race/ethnicity was not used as a variable when measuring the academic achievement of homeschoolers (probably because there were not enough non-white respondents to make the measurements significant).
These graphs show that the approximately one-third of US residents who are black or Latino are essentially not represented in Ray’s study. This is relevant because, due to structural inequalities in education and other aspects of society, black and Latino children tend to score approximately 5% lower than white children on standardized tests.
Some data suggests that the percentage of homeschoolers who are minorities may not be nearly as low as is represented in Ray (2010),[2] in which case the artificially high scores for the homeschoolers in Ray’s study may be inflated partly because his sample was so white. However, even if the homeschool population is disproportionately white, it is deceptive to compare their scores to those of the general population.
Socioeconomic class
There are a number of indicators of socioeconomic class which Ray collected data on—marital and employment status of the parent(s), educational level of the parent(s), number of children, and family income. In all of these measures, the homeschooling families Ray surveyed are members of a higher socioeconomic class than the average across comparable families in the US. Children with more socioeconomic advantages tend to have higher academic achievement levels. The artificially high scores for the homeschoolers in this study are inflated partly because the sample was so wealthy.
In Ray’s study, 97.9% of respondents lived in a home with married parents, while in the general population, only 71.2% of families with children under 18 are headed by a married couple. Households headed by a married couple have a much higher median income than other households—for married couples the median income is around $71,000, while for single-parent households the median income is between $32,000 and $48,000 and for non-family households the median income is around $30,000. Studies have shown that children in two-parent households achieve better academic results than children in single-parent households.
In Ray’s sample, parents with higher educational attainment are greatly over-represented. Of the homeschooling parents in Ray’s sample, 8.9% had never attended college while 64.4% had at least a Bachelor’s degree. By contrast, in the US Census in 2010, 44.1% of the adult population had never attended college while 29.9% had a BA or higher. The graph below shows the highest academic achievement levels of parents in Ray’s study and US adults overall.
Ray’s homeschooling parents were twice as likely as the average US adult to have a Bachelor’s degree or higher, and only one-fifth as likely to have never attended college. Furthermore, Ray’s participants were not representative of all homeschoolers either. NCES data suggests that approximately 50% of homeschooled students do not have a parent with a Bachelor’s degree. Research has consistently shown that parents’ educational attainment affects their children’s academic success; the fact that Ray’s sample is not representative of the US population in educational attainment may explain some of the difference between their children’s test scores.
In Ray’s sample, there were on average 3.5 children per household; the national average is about 2. Ray’s homeschooling participants had much larger families than the average in the US—the chart below shows that while only 20.2% of US families had three or more children, 68.2% of Ray’s participants did.
The general consensus among researchers is that children in larger families have lower academic achievement—this is consistent with Ray’s finding that his participants with larger families had lower scores. Here again, Ray’s sample is unrepresentative of the general population in terms of family size. If Ray had made an effort to control for this background factor, it may actually have widened the gap between homeschoolers’ and public schoolers’ academic abilities and allowed his homeschool participants to score even higher, increasing the evidence for his thesis.
As previously mentioned, 97.9% of families in Ray’s sample were headed by a married couple with, on average, 3.5 children. 80.6% of the mothers Ray surveyed do not work for pay, while 95.9% of his participants’ fathers are employed full-time. In 2009, the poverty threshold for families with two parents and 3.5 minor children was $27,262.75 per year. Only 4% of the families Ray sampled fall into this category, but in the US as a whole about 20% of families fell below the poverty threshold in 2009. In 2009, the median income of families headed by a married couple with two or more children where the father works full time and the mother does not work was $55,666 per year, while in Ray’s sample the median income of homeschooling families was $75,000 to $79,000 per year. The graph below demonstrates how Ray’s sample seriously undercounts poorer families (making less than $50,000 per year) and overcounts middle class and rich families.
Religion
It is not surprising that approximately 96.7% of Ray’s respondents identified as some variety of Christian (this includes Seventh Day Adventist, Assembly of God, Baptist, Roman Catholic, Eastern Orthodox, Episcopal, Independent Charismatic, Independent Fundamentalist/Evangelical, Jehovah’s Witness, Lutheran, Mennonite, Methodist, Mormon, Nazarene, Other Christian, Other Protestant, Pentecostal, Presbyterian, and Reformed). His data were drawn from three sources: 68% from major testing services[3] (of which one was BJU Press, an explicitly Christian organization); 23% from eleven minor testing services, of which all but one are explicitly Christian (Basic Skills Assessment and Educational Services, Circle Christian School, Covenant Home School Resource Center, Idaho Coalition of Home Educators, Whatcom Home School Association, Education Network of Christian Homeschoolers of New Jersey, HSLDA, Homeschoolers of Maine, Massachusetts Homeschool Organization of Parent Educators, NYS Loving Education at Home, and the Oregon Christian Home Education Association Network); and 8% from sources which he does not identify. Supposing that each organization contributed equally to the data (which we have no way of knowing, since Ray does not tell us), fully 38% of the respondents were recruited by explicitly Christian organizations.
Only 2.6% of Ray’s respondents identified as atheist/agnostic or Other, while only .7% followed non-Christian religions. This religious breakdown is substantially different from the national one. In 2008, only 76% of American adults identified as Christian. Of the remainder, 4% identified as members of non-Christian faiths, 15% identified as atheist/agnostic or having no religion, and 5% did not disclose their religious beliefs.
This is relevant because religious involvement has been shown to be correlated with better academic outcomes. Furthermore, there are two main schools of thought in homeschooling: the “ideologues” homeschool for religious or moral reasons, while the “pedagogues” homeschool because they believe that it is the most effective educational technique. Homeschoolers whose parents have such differing views of homeschooling might have different levels of academic achievement. The inclusion in Ray’s study of only the “ideologues”—the Christians—leaves out the many secular homeschoolers and unschoolers, who—if they were included—might influence Ray’s findings.
3) Self-selected sample
Perhaps the most damning methodological problem with this study is that the participants self-selected for the variable being tested. That is, students who were more likely to score well on standardized tests were more likely to participate in the study, which measured how well students do on standardized tests. It is likely that the parents having their children tested were either those required by state law to do so, or those parents most dedicated to their children’s success who wanted to determine where their children’s academic weaknesses lie.
Ray took data from testing services where parents who were confident in their children’s ability to do well signed them up for tests. He went to state organizations filled with active and activist members for whom their children’s high academic achievement in homeschooling was an important part of their identity. He averaged their scores and found that they were higher than the national average. Does this mean that homeschoolers have higher academic achievement than the national average? Not necessarily. It means that people agreed to participate in this study because they were already high achievers.
A good comparison would be, for instance, a hypothetical study on what percentage of the population has red hair. Suppose you start by asking all the red-haired people you know to participate, and ask them to promote the study to their red-haired friends. All these redheads are eager to participate in studying the group they belong to, while people with other hair colors have more interesting things to do than fill out a survey. Suppose then you ask all the people who have agreed to participate, and you find that 60% of them have red hair. Does this mean that 60% of the population has red hair? No (in fact, only about 4% of humans have red hair). It means that people agreed to participate in your study about red hair because they had red hair.
Let’s look at some statistics to put Ray’s study in perspective. We might expect that mostly those parents who think their children can score well on standardized tests would sign them up to take them. In Ray (2010), students who used the three largest testing services to take a standardized test made up around 51% of the study participants.[4] The total number of students who used these three testing services in 2009 was 22,584. If we assume that the population Ray was sampling was the set of all homeschool students who use a testing service to take standardized tests, then these 22,584 students probably compose about 51% of the population he was sampling. We can therefore hypothesize that around 44,000 homeschoolers per year use a testing service to take a standardized test. In 2009-2010, approximately 1.9 million students were homeschooled in the US. In other words, Ray’s findings are relevant to the 2% of homeschoolers whose parents had them take a standardized test using a testing service in 2009. See footnote 5 for an alternate analysis that posits that Ray’s findings may instead be relevant to 3% of US homeschoolers.
Meanwhile, about 4 million students in the US started high school in 2006-2007. Of those, about 3.1 million graduated from high school in 2009-2010. Approximately 2 million students took the SAT during the 2009-2010 school year. That means somewhere from 50-65% of high school seniors that year took the SAT. The SAT is a good comparison for homeschool standardized tests because both are voluntary and are sought by the most academically dedicated.
If we assume that it is mostly the students whose parents expect them to do well who sign them up to take standardized tests, we could hypothesize that the top scoring 50-65% of students in the US took a standardized test in 2009, while the top 2-3% of homeschoolers did. The national student average on standardized tests is the 50th percentile, while Ray’s (2010) participants scored around the 86th percentile. Ray’s scores are exactly what we would expect from a smaller, more selective testing pool where only those parents who knew their children would do well volunteered to participate.
Furthermore, in many cases, the test administrators were the children’s parents. With no oversight to make sure that students were following time limits and not using outside resources, parents could influence children’s scores with impunity, or choose not to have the test graded if they thought their children did poorly. In addition, a full 27.7% of Ray’s (2010) participants knew their children’s test scores before agreeing to participate. How many parents whose children scored poorly do you think opted out of the study?
4) Low response rate
For students in the four large testing services (either 68% or 71.5% of participants in the study), the response rate reported by Ray was 19-25%.[5] This means that, of all the people Ray contacted about participating in the study, only about one in four actually completed the survey and sent in their child’s scores. For students in the smaller testing services (either 31% or 29% of participants), the response rate was 11%; only about one in ten people that Ray contacted actually completed the survey.
In social science research of this type, “a response rate between 70 and 80% is the goal, with 70% being minimally acceptable and a response rate above 80% being the ideal, although highly unlikely. If the response rate falls below 60% it becomes very difficult to claim that the survey is representative of the entire sample because responses are missing from more than one third of surveyed individuals” (Mangione 1995 and Bachman & Schutt 2007; cited in Henninger & Sung 2012:306). Needless to say, Ray’s (2010) response rates are abysmal, making this study even less representative of the population Ray was attempting to sample.[/expand]
[expand title=”What the study actually says”]
The top-scoring 2-3% of homeschooled elementary- and middle-school-age children of wealthy white Christian married couples who volunteer them for standardized tests score approximately 36 percentage points higher than the national average of all children in public schools. This is not a very meaningful result.
Even in this highly homogeneous population, participants did better in reading than math. Indicators of higher scores included high parental expenditure per child, high family income, being a child of two college graduates, having fewer siblings, having a more structured education, spending more time in structured learning, and starting schooling earlier. Though these results are not transferable outside the study, they may be indicative of larger trends and merit further research.[/expand]
[expand title=”Conclusion”]
We cannot tell how well homeschoolers perform on standardized tests from the results of this study. A host of methodological problems make Ray’s findings unrepresentative of the homeschool population—meaning that they cannot be generalized to any population larger than 2-3% of all homeschoolers—and unrepresentative of the national population of school-age children, meaning his scores cannot be compared to the scores of public schoolers.
What we can tell is that, even in Ray’s extremely homogeneous sample, some demographic variables still made a difference in homeschoolers’ scores, including gender, family income, parental education, and amount of educational structure. Ray’s participants also scored highest in reading and lowest in math. These findings suggest trends that might hold across all homeschoolers if they were studied in a methodologically rigorous way.[/expand]
[expand title=”References”]
Apodosis. “Guest Post: Methodological problems with Brian Ray’s study on youth and religion.” Love, Joy, Feminism. Patheos. http://www.patheos.com/blogs/lovejoyfeminism/2013/07/methodological-problems-with-brian-rays-study-on-youth-and-religion.html
Bachman, Ronet and Russell K. Schutt. 2007. The practice of research in criminology and criminal justice (3rd ed.). Thousand Oaks, CA: Sage.
Bahr, Stephen J. 2001. “Social science research on family dissolution: What it shows and how it might be of interest to family law reformers.” Presentation at the conference, “The ALI Family Dissolution Principles: Blueprint to Strengthen or to Deconstruct American Families?”, J. Reuben Clark Law School, Brigham Young University, Provo, Utah, February 1, 2001. http://fact.on.ca/Info/divorce/bahr2001.pdf
Davis-Kean, Pamela E. 2005. “The influence of parent education and family income on child achievement: The indirect role of parental expectations and the home environment.” Journal of Family Psychology 19:2. 294-304. http://www.mikemcmahon.info/ParentEducationIncome.pdf
Dubow, Eric F., Paul Boxer, and L. Rowell Huesmann. 2009. “Long-term effects of parents’ education on children’s educational and occupational success: Mediation by family interactions, child aggression, and teenage aspirations.” Merrill Palmer Quarterly 55:3. 224–249. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2853053/
“FAQs.” CollegeBoard. http://web.archive.org/web/20110416223006/http:/press.collegeboard.org/sat/faq
Glanville, Jennifer L., David Sikkink, and Edwin I. Hernández. 2008. “Religious involvement and educational outcomes: The role of social capital and extracurricular participation.” The Sociological Quarterly 49. 105-137. http://onlinelibrary.wiley.com.ezproxy.lib.indiana.edu/doi/10.1111/j.1533-8525.2007.00108.x/pdf
Grady, Sarah, Stacey Bielick, and Susan Aud. 2010. Trends in the use of school choice: 1993 to 2007 (NCES 2010-004). Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
Henninger, Alana and Hung-En Sung. 2012. “Mail survey in social research.” Handbook of Survey Methodology for the Social Sciences, ed. by Lior Gideon, 297-311. New York: Springer.
Isenberg, Eric J. 2007. “What Have We Learned About Homeschooling?” Peabody Journal of Education 82. 387-409.
Kosmin, Barry A. and Ariela Keysar. 2009. American Religious Identification Survey (ARIS 2008) Summary Report. Hartford, CT: Institute for the Study of Secularism in Society & Culture. http://commons.trincoll.edu/aris/files/2011/08/ARIS_Report_2008.pdf
Kunzman, Robert and Milton Gaither. 2013. “Homeschooling: A comprehensive survey of the research.” Other Education: The Journal of Educational Alternatives 2:1. 4-59. http://www.othereducation.stir.ac.uk/index.php/OE/article/view/10/55
Mangione, Thomas W. 1995. Mail surveys: Improving the quality. Newbury Park, CA: Sage.
Maralani, Vida. 2008. “The changing relationship between family size and educational attainment over the course of socioeconomic development: Evidence from Indonesia.” Demography 45:3. 693-717. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2831397/#b71-dem-45-0693
Marticke, Simone. 2005. “Hair color.” The Tech Museum of Innovation. http://genetics.thetech.org/ask/ask144
Pong, Suet-ling, Jaap Dronkers, and Gillian Hampden-Thompson. 2003. “Family policies and children’s school achievement in single- versus two-parent families.” Journal of Marriage and Family 56:3. 681-699. http://www.jstor.org/stable/3600032
Rampey, Bobby D., Gloria S. Dion, and Patricia L. Donahue. 2009. NAEP 2008 Trends in Academic Progress (NCES 2009–479). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, Washington, D.C. http://nces.ed.gov/nationsreportcard/pdf/main2008/2009479.pdf
Ratcliffe, Caroline and Signe-Mary McKernan. 2012. “Child poverty and its lasting consequence.” Low-Income Working Families, Paper 21. The Urban Institute. http://www.urban.org/UploadedPDF/412659-Child-Poverty-and-Its-Lasting-Consequence-Paper.pdf
Ray, Brian D. 2009. Progress Report 2009: Homeschool Academic Achievement and Demographics. Home School Legal Defense Association.
Ray, Brian D. 2010. Academic achievement and demographic traits of homeschool students: A nationwide study. Academic Leadership, the Online Journal 8:1.
Resmovits, Joy. 2013. “Graduation rate hits record high for high school students: Government report.” The Huffington Post. http://www.huffingtonpost.com/2013/01/22/graduation-rate-record-high-school-students_n_2522128.html
Rouse, Cecilia Elena and Lisa Barrow. 2006. “U.S. Elementary and Secondary Schools: Equalizing Opportunity or Replicating the Status Quo?” Opportunity in America 6:2. 99-123. http://futureofchildren.org/publications/journals/article/index.xml?journalid=35&articleid=89§ionid=541
Steelman, Lala C., Brian Powell, Regina Werum, and Scott Carter. 2002. “Reconsidering the Effects of Sibling Configuration: Recent Advances and Challenges.” Annual Review of Sociology 28. 243–69. http://www.annualreviews.org/doi/abs/10.1146/annurev.soc.28.111301.093304
US Census Bureau. 2009. Statistical Abstract of the United States (Table 8). Population 11. http://www.census.gov/prod/2008pubs/09statab/pop.pdf
US Census Bureau. 2010. “Income, Poverty and Health Insurance Coverage in the United States: 2009.” Current Population Reports, P60-238, and Historical Tables (Table F-23). http://www.census.gov/compendia/statab/2012/tables/12s0696.pdf
US Census Bureau. 2010. “Income, Poverty and Health Insurance Coverage in the United States: 2009.” Current Population Reports, P60-238, and Detailed Tables (Table HINC-01). http://www.census.gov/compendia/statab/2012/tables/12s0692.pdf
US Census Bureau. 2010. “Income, Poverty and Health Insurance Coverage in the United States: 2009.” Current Population Reports, P60-238, and Historical Tables (Table P-2). http://www.census.gov/compendia/statab/2012/tables/12s0700.pdf
US Census Bureau. 2012. “Poverty Thresholds 2009.” Social, Economic, and Housing Statistics Division: Poverty. http://www.census.gov/hhes/www/poverty/data/threshld/thresh09.html
US Census Bureau. 2012. Statistical Abstract of the United States (Tables 231 & 232). Education 152. http://www.census.gov/compendia/statab/2012/tables/12s0231.pdf[/expand]
[expand title=”Footnotes”]
[1] That is, the effectiveness of homeschooling varied with a participant’s grade, and this observation was unlikely to be due to sampling fluctuations.
[2] Data from the National Center for Education Statistics suggests that the 2007 homeschooling population was 77% white, 4% black, 10% Hispanic, 2% Asian or Pacific Islander, and 7% Other.
[3] Note that at one point in his article, Ray states that 71.5% of his participants were drawn from the four large testing services (p. 8), while at another point his numbers indicate that the percentage was 68% (Table 24, p. 33). The source of this discrepancy is unknown.
[4] Assuming that 68% of the participants were drawn from a major testing service and that each of the four major testing services contributed equally (17%) to the data, 51% of the participants were drawn from three of the four testing services.
[5] It is unclear how Ray calculated these numbers. The total number of customers for the three largest testing services was 22,584, while the total number of study participants from the four largest testing services was 8,039. Supposing an equal number of study participants came from each of the four testing services, there were 6,029 students in the three largest testing services. This produces a 27% response rate. It is possible that the difference between this response rate and that reported by Ray can be accounted for by presuming that a large number of parents (perhaps more than 9,000) contracted with testing services (thus receiving an invitation to participate in the study) and then did not follow through and submit their children’s tests to be graded. Note that if this is the case, the total population of parents that contract with testing services (rather than that follow through and have their children tested) may be closer to 62,000, which is around 3% of all homeschoolers.[/expand]
Dr. Chelsea McCracken
Research Analyst
Coalition for Responsible Home Education
chelseamc@responsiblehomeschooling.org
661 Washington Street #563
Canton, MA 02021