Virginia’s Religious Exemption (Ray, 1994)

Click here to view this document as a pdf.

Virginia is the only US state that allows parents to bypass compulsory education laws by claiming a religious exemption from school attendance, according to the University of Virginia School of Law’s Child Advocacy Clinic. One of the publications cited in support of the religious exemption is Brian Ray’s 1994 study entitled “A brief statistical analysis of academic achievement test data from home educated students operating under the Virginia religious exemption statute.” This study, which contains only one page of analysis accompanied by seven pages of tables, was not published in a peer-reviewed publication—it was released by Ray’s National Home Education Research Institute (NHERI), essentially a vanity press which operates in practice as the research arm of the Home School Legal Defense Association (HSLDA). It is highly unlikely the study would have been published as-is in a peer-reviewed journal, as it appears to be merely a stub of an article, lacking complete analysis and any description of methodology.

Ray’s study does not prove that religiously exempt homeschoolers in Virginia score 33% higher on standardized tests than public schoolers, although it has been claimed to do so. It does demonstrate that a small percentage (around 9%) of the state’s religiously exempt homeschoolers in 1994 were doing well academically; that they scored better on reading than math across the board; and that their math skills decreased with respect to public schoolers as they aged, while their language skills increased.

However, these homeschoolers were not randomly selected and most likely included the most academically successful religiously exempt students in the state. Ray did not collect information on or correct for demographic factors which have been shown to affect academic success, such as race, family income, family structure, parental education, etc. Furthermore, the participants in Ray’s study were overwhelmingly elementary schoolers—the number of older children sampled was so small as to be meaningless—and were probably drawn from suburban areas of the state, as these were the primary locations of religiously exempt homeschoolers in 1994. Today, the landscape has changed somewhat—the number of religious exemptions has more than tripled, and religious exemptions are now most common in extremely rural areas. Whatever little descriptive validity Ray’s study had in 1994, it is now no longer relevant.

In the sections that follow, I will first give some background of the study, then outline its major points. Next I will provide a critical analysis of the study, and finally I will summarize what his results actually mean.

[expand title=”Background of the study”]

Ray released the results of the study in January 1994, but the standardized tests he analyzed were conducted during one of three years (1991, 1992, or 1993); it is possible some participants were tested several times over multiple years, though the majority of tests analyzed (184) were administered in 1993.

Participants in the study took one of a number of standardized tests, which included the Stanford Achievement Test (SAT – 87 participants), the Iowa Test of Basic Skills (ITBS – 64 participants) or related Tests of Achievement and Proficiency (TAP – 6 participants), the California Test of Basic Skills (CTBS – 26 participants), the California Achievement Test (CAT – 17 participants), the Woodcock-Johnson Tests of Cognitive Abilities (8 participants), the Miller Analogies Test (MAT – 3 participants), the American College Testing test (ACT – 1 participant), and the Comprehensive Testing Program test (CTP – 1 participant). In a later study, Ray (2010) argued that combining scores from many standardized tests would not skew the results, though it would have been ideal to use only one standardized test in the study.

Ray states that he analyzed the standardized test scores of 213 Virginia students educated under the religious exemption law; however, this is misleading for several reasons. First, when he reports his data arranged by grade, the numbers he offers add up to 156, not 213. Second, while he may have had access to 213 test scores, he did not have demographic data for every participant. According to Ray, only 181 participants reported their grade, 152 reported their age, 184 reported their gender, and 197 reported the year they had taken the standardized test. It is unknown how many test scores were associated with a fully complete demographic survey.

Furthermore, Ray does not discuss how he recruited his participants or how he obtained their test scores and demographic data. Based on the methodology he used in later studies, it is likely he recruited his participants through state or local homeschooling organizations rather than by random sampling. In this case, his data would be subject to self-selection bias. However, we do not know because he does not tell us. Not all of the standardized tests used by participants in the study contained sections on science and social studies, so Ray’s sample size for those subjects was even smaller—102 for science and 78 for social studies.[/expand]

[expand title=”Study’s major points”]

Ray’s study is available for purchase on the NHERI website for $4. It is also available here. In one page of analysis, the findings Ray discusses are as follows:

• The test scores of 213 Virginia homeschoolers operating under the religious exemption were, on average, at or above the 83rd percentile. This places them approximately 33% higher than the national average score for public schoolers on standardized tests (where the national average is, by definition, the 50th percentile).
• These homeschoolers scored, on average, in the 90th percentile in reading, the 89th percentile in science, the 87th percentile in math, the 87th percentile in social studies, and the 83rd percentile in language.
• Demographically, the sample was comparatively young; 78% of the participants were in grades K-6. The largest plurality of participants (41%) submitted scores from the SAT, while the rest submitted scores from other standardized tests.
• Ray conducted a statistical analysis of student scores on the “basic battery” (a term he does not define, but which probably refers to an average of reading, math, and language scores) by grade level. He found that there were no significant differences; that is, there was no statistical difference between the average scores of students in different grades.
• Ray concludes that homeschoolers operating under the religious exemption perform very well on standardized tests.

As Ray does not conduct a full analysis of his data or provide a full set of tables, it is not possible to determine very much more. However, from the data he provides, it is also possible to deduce the following:

• Ray’s participants tended to score lower on math than on reading, a finding which is consistent with most existing research on homeschoolers.

• In general, the math scores of Ray’s participants tended to decrease in comparison with the national average as students aged, while Ray’s participants’ language scores tended to increase in comparison with the national average (represented here by the x-axis). (Note that for grades K-2, Ray’s sample size was smaller for language than for math.)

[/expand]

[expand title=”Critical analysis”]

Methodological problems with Ray’s (1994) study include failing to discuss crucial details, failing to correct for background factors, using a sample size which is too small, and drawing a sample that is not representative of the population.

1) Lack of crucial details

As mentioned above, Ray does not discuss how he recruited his participants, how he gathered their demographic information, how he dealt with the many incomplete demographic profiles he received, under what circumstances the participants took the standardized tests, and what statistical analyses Ray conducted. (It is standard practice to report the p-values for statistical tests, which he claims to have conducted, but he does not do so.)

Without this information, it is impossible to determine whether Ray’s sample was truly random or was exposed to selection bias; whether homeschooled students were administered the test under controlled conditions; whether Ray double-counted students who took tests in multiple years; and whether his statistical tests were valid.

This lack of crucial detail is a major reason why this study could not have been published as-is in a peer-reviewed journal.

2) No correction for background factors

Numerous studies have shown that race/ethnicity, socioeconomic class, parent education level, parental marital status, religion, and many other background factors influence children’s academic success. Ray does not collect, let alone analyze, data on his participants with respect to these demographic variables. It is thus impossible to know if his participants are comparable to the average public schooler. If Ray’s sample was more white, better educated, and of a higher socioeconomic class than the national average (as in much of his other research), it would be completely expected that they would score higher than the national average. It would not, however, indicate that homeschooling under Virginia’s religious exemption was the cause of those high scores.

3) Small sample size

In the 1994-1995 school year (the earliest year for which these statistics are available), there were 7,856 students in Virginia being homeschooled under the homeschool statute compared with 1,767 students being homeschooled under the religious exemption. The 156 students whose scores on the “basic battery” of tests (reading, math, and language) were analyzed comprised 9% of all homeschoolers operating under the religious exemption. If the study were based on a random sample which was demographically balanced, this would be an acceptable sample size. However, since homeschoolers who take standardized tests are only a small portion of the entire homeschooling population (perhaps only 2-3%), and because testing is entirely voluntary for those homeschooled under Virginia’s religious exemption, it is likely that this 9% of religiously exempt homeschoolers includes many of the highest academic achievers in the state, and unlikely that it includes students who are at the greatest risk for educational neglect. We thus know very little about the academic performance of the vast majority of religiously exempt students—91% of them—who did not participate in this study.

Furthermore, in the 2012-2013 school year (the most recent year for which statistics are available), there were a total of 29,886 homeschoolers operating under the homeschool statute in Virginia, compared with 6,429 operating under the religious exemption. If we assume the homeschooling population has stayed essentially the same in terms of demographics over the past 20 years (not necessarily a warranted assumption), Ray would need to sample 579 students today to get the same percentage of the state’s homeschoolers.

4) Non-representative sample

The children whose test scores were analyzed by Ray were not representative of the state’s religiously exempt homeschoolers, let alone the state’s public school children, in terms of age and geographic location.

Age   Ray’s sample was not representative of the Virginia school-age population in terms of age. For comparison, I have graphed Ray’s percentage of participants in each grade with the percentage of students enrolled in Virginia public schools in Fall 1993. Note the higher blue bars which indicate that Ray’s study contained a higher percentage of elementary schoolers than Virginia public schools did in 1993. The higher red bars indicate that Ray included a smaller percentage of middle and high schoolers than were attending public school in Virginia in 1993.

Particularly for the middle and high school grades, Ray’s sample sizes were so small as to be essentially meaningless—Ray only had the reading and math scores of 15 participants in grades 9-12 and 29 participants in grades 6-8. In his analysis of reading and math scores by grade level, Ray was only able to use the scores of two 11th graders and one 12th grader. He obtained the test scores of only six 15-year-olds, one 16-year-old, and one 17-year-old.

Location   Ray’s sample of religiously exempt homeschoolers is also not representative of Virginia’s population in terms of where they live, and the population of religiously exempt homeschoolers has undergone significant geographic changes in the past 20 years. In the 1994-1995 school year (the earliest year for which data are available), about 11% of religiously exempt homeschoolers lived in a city, while the majority of religiously exempt homeschoolers (about 68%) lived in suburbs: Fairfax County, a suburb of D.C. (205); Rockingham County, a suburb of Harrisonburg (145); and Chesterfield County (136), a suburb of Richmond. Finally, 22% of religiously exempt homeschoolers lived in rural areas.This skewing is problematic because some data indicates that homeschooling grows less common as kids age (Kunzman & Gaither 2013)—only 48% of religious and 15% of secular homeschoolers continue to homeschool for more than six years (Isenberg 2007), perhaps due to the increased difficulty of high school classes. If a large number of high schoolers are quitting homeschool and going to public or private school, that may have an effect on the average test scores for those who stay. It may also be the case that homeschooling children in high school is less effective, something that could be obscured in this study by the lower numbers of homeschoolers of high school age participating. Finally, it is impossible to claim that a sample size of one is representative.

In the 2012-2013 school year (the most recent year for which data are available), only about 6% of the 6,429 religiously exempt homeschoolers lived in a city, while about 68% lived in suburbs and 26% lived in rural areas. The number of religiously exempt homeschoolers per city district increased by an average of 5 homeschoolers, while the average increase in county districts was 48 homeschoolers.  The districts with the greatest increase in religiously exempt homeschoolers were Fairfax County (increasing to 600, a difference of 395); Stafford County (also a D.C. suburb, increasing from 0 to 248); Augusta County (a Staunton suburb, increasing from 23 to 250); and Rockingham County (increasing to 351, a difference of 206). On the other hand, the number of religiously exempt homeschoolers decreased or stayed the same in 29 districts, including Chesterfield County (decreasing to 55, a difference of -81).

For comparison, in the 2010-2011 school year (the most recent year for which data are available), about 28% of Virginia’s public schoolers lived in a city, while about 59% lived in suburbs and 13% lived in rural areas. The districts with the highest number of public schoolers were Fairfax County (174,000 students); Prince William County (also a D.C. suburb, 79,000 students); Virginia Beach City (71,000 students); Loudoun County (also a D.C. suburb, with 63,000 students), and Chesterfield County (59,000 students). A high district population seems to correspond slightly with a high population of religiously exempt homeschoolers (r = .5).

However, if we remove the outliers of Fairfax and Chesterfield Counties, the correlation essentially disappears (r = .16). This means that there is little relationship between the number of public schoolers and the number of religiously exempt homeschoolers in a district.

The districts with the highest ratio of religiously exempt homeschoolers to public schoolers are all rural counties which do not contain a major population center: Floyd County (7:100), Madison County (5:100), and Giles, Charlotte, Highland, and Warren Counties (4:100).

It seems from this data that, at the time of Ray’s 1994 study, most of Virginia’s religiously exempt homeschoolers were suburban families, congregating in counties rather than in cities. This trend has held steady in the years since Ray’s study—today, the majority of homeschoolers with religious exemptions continue to live in suburbs. However, religiously exempt homeschoolers are most common in extremely rural areas, where they comprise a higher percentage of the school-age population than in suburban areas. Religious exemptions for rural Virginia families have become more common since Ray’s study—rural residents composed 22% of religiously exempt homeschoolers in 1994 but 26% today. That the religiously exempt population has undergone this shift indicates that, however relevant Ray’s findings were in 1994, they are no longer applicable.[/expand]

[expand title=”What the study actually says”]

Ray’s 1994 study does not tell us very much. The 156 religiously exempt homeschool participants who took the “basic battery” of standardized tests scored, on average, around 33% higher than the national average of all public schoolers. However, these 156 students were not even representative of all Virginia religious exemptions—comprising only 9% of that population, they most likely were guilty of selection bias and contained far too few middle and high schoolers to achieve statistical reliability—let alone socioeconomically comparable to all public schoolers in the US, to whom Ray compared them.

What the study does tell us is that this tiny sample of academically successful religiously exempt homeschoolers scored better on reading than on math across the board, and that their math scores decreased with respect to the national average as they aged even as their language skills increased with respect to the national average.[/expand]

[expand title=”Conclusion”]

We cannot tell how well religiously exempt homeschoolers in Virginia are doing academically in the present day from Ray’s 1994 study. This study tells us only that a small, highly selective group of elementary schoolers were doing well in 1994. We know nothing about how well the majority of religiously exempt homeschoolers were doing in 1994, and the population has undergone so many changes since then that Ray’s findings are no longer relevant.[/expand]

[expand title=”References”]

Anonymous. 2014. “Academic Achievement.” Coalition for Responsible Home Education. https://responsiblehomeschooling.org/homeschooling-101/academic-achievement/

Bahr, Stephen J. 2001. “Social science research on family dissolution: What it shows and how it might be of interest to family law reformers.” Presentation at the conference, “The ALI Family Dissolution Principles: Blueprint to Strengthen or to Deconstruct American Families?”, J. Reuben Clark Law School, Brigham Young University, Provo, Utah, February 1, 2001. http://fact.on.ca/Info/divorce/bahr2001.pdf

Davis-Kean, Pamela E. 2005. “The influence of parent education and family income on child achievement: The indirect role of parental expectations and the home environment.” Journal of Family Psychology 19:2. 294-304. http://www.mikemcmahon.info/ParentEducationIncome.pdf

Gaither, Milton. 2008. “Brian D. Ray and NHERI, part 2.” Homeschooling Research Notes. http://gaither.wordpress.com/2008/10/07/brian-d-ray-and-nheri-part-2/

Glanville, Jennifer L., David Sikkink, and Edwin I. Hernández. 2008. “Religious involvement and educational outcomes: The role of social capital and extracurricular participation.” The Sociological Quarterly 49. 105-137. http://onlinelibrary.wiley.com.ezproxy.lib.indiana.edu/doi/10.1111/j.1533-8525.2007.00108.x/pdf

McCracken, Chelsea. 2014. “Homeschooling academics and demographics (Ray 2010).” Coalition for Responsible Home Education. https://responsiblehomeschooling.org/research-analysis/homeschooling-academics-and-demographics-ray-2010/

Rampey, Bobby D., Gloria S. Dion, and Patricia L. Donahue. 2009. NAEP 2008 Trends in Academic Progress (NCES 2009–479). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, Washington, D.C. http://nces.ed.gov/nationsreportcard/pdf/main2008/2009479.pdf

Ray, Brian D. 1994. “A brief statistical analysis of academic achievement test data from home educated students operating under the Virginia religious exemption statute.” Salem, OR: National Home Education Research Institute. http://www.hslda.org/hs/state/VA/Ray_Va_Study_94.pdf

Ray, Brian D. 2010. “Academic achievement and demographic traits of homeschool students: A nationwide study.” Academic Leadership, the Online Journal 8:1.

Rouse, Cecilia Elena and Lisa Barrow. 2006. “U.S. Elementary and Secondary Schools: Equalizing Opportunity or Replicating the Status Quo?” Opportunity in America 6:2. 99-123. http://futureofchildren.org/publications/journals/article/index.xml?journalid=35&articleid=89&sectionid=541

Tschiderer, Christine, Henry Sire, Allison Lansell, Stephanie Moore, and Andrew Block. 2012. “7,000 children and counting: An analysis of religious exemptions from compulsory school attendance in Virginia.” Child Advocacy Clinic. Charlottesville, VA: University of Virginia School of Law. http://www.law.virginia.edu/pdf/news/religious_exemption_report.pdf

Woodruff, Scott A. 2014. “Why the religious exemption from school attendance belongs and the study proposed in NJ 92 is unnecessary.” Purcellville, VA : Home School Legal Defense Association. http://www.hslda.org/hs/state/va/Talking_Points_HJR_92.pdf

Virginia Department of Education. 2010. “2010-11 Fall Membership by Division, by Grade.” Commonwealth of Virginia. http://www.pen.k12.va.us/statistics_reports/enrollment/fall_membership/2010-2011/division_totals_grade.xls

Virginia Department of Education. 2012. “1994-1995 to 2001-2002 Home-Schooled Students and Religious Exemptions.” Commonwealth of Virginia. http://www.doe.virginia.gov/statistics_reports/enrollment/home_school_religious_exempt/1994_2001.pdf

Virginia Department of Education. 2013. “Home-Schooled Students and Religious Exemptions, 2012-2013.” Commonwealth of Virginia. http://www.doe.virginia.gov/statistics_reports/enrollment/home_school_religious_exempt/2012_2013.xls

U.S. Department of Education. 1995. “Table 40: Enrollment in public elementary and secondary schools, by grade and state: Fall 1993.” Digest of Education Statistics. National Center for Education Statistics, Common Core of Data survey. http://nces.ed.gov/programs/digest/d95/dtab040.asp[/expand]

Dr. Chelsea McCracken
Research Analyst
Coalition for Responsible Home Education
chelseamc@responsiblehomeschooling.org
661 Washington Street #563
Canton, MA 02021
(617) 765-7096