Relationship between Reading Ability and Achievement
in a Graduate-Level Research Methodology Course
Kathleen M. T. Collins, PhD
Assistant Professor
Department of Curriculum and Instruction
University of Arkansas at Fayetteville
E-mail: kcollinsknob@cs.edu
Anthony J. Onwuegbuzie, PhD
Professor
Department of Educational Measurement and Research
College of Education
Sam Houston State University
E-mail: tonyonwuegbuzie@aol.com
Qun G. Jiao
The City University of New York
Baruch College, Newman Library
E-mail: gerry_jiao@baruch.cuny.edu
Although innumerable studies have been designed in the area of reading comprehension and reading ability from pre-kindergarten through undergraduate students, there have been limited investigations ascertaining the impact of graduates students’ reading ability on their academic performance. Given the notability of reading ability as a factor impacting the achievement levels of students in academic settings that require critical thinking skills (Brown, Fishco, & Hanna, 1993; Du Boulay, 1999; Onwuegbuze, Slate, & Schwartz, 2001; Pritchard, Romeo, & Muller, 1999; Zhu, 1999), this study sought to contribute to the extant research in this area by examining the relationship between reading ability of graduate students and their achievement in the context of a graduate level research methodology course.
Review of the Related Literature
Reading ability has been defined as the dynamic interplay of the reader’s individual characteristics, the text, and the reading context (e.g., academic vs. recreational reading) that facilitates comprehension of text information and the successful integration of newly acquired information into the reader’s pre-existing schema (Anderson & Pearson, 1984; Mason, 1984; Paris, 1987; Wixson & Peters, 1984). Moreover, students’ reading ability impacts their relative success or failure on the continuum of academic settings ranging from kindergarten to college because reading effects and outcomes are assessed continuously throughout college students’ lives, as well as across their varied professional experiences (Brown et al., 1993; Du Boulay, 1999; Pritchard et al., 1999; Zhu, 1999).
The degree that reading ability is a predictor of college student performance in the context of an organic and biochemistry nursing undergraduate course was documented in a study conducted by Van Lanen, Lockie, and McGannon (2000). The sample (n = 308) comprised traditional students (51%) and continuing education students (49%). The researchers were interested in identifying predictors of nursing students’ performance in a required chemistry course because their perusal of the literature indicated that undergraduates enrolled in college-level chemistry courses experienced a high failure rate. A regression analysis was conducted utilizing the final course grade as a dependent variable and students’ score on the Nelson-Denny Reading Test (NDRT) Form E (Brown et al., 1993) as one of the independent variables. Results indicated a statistically significant relationship between the NDRT scores and the final course grades for the entire sample (b =.19). Van Lanen et al. (2000) interpreted the results to indicate that the limited domain-specific (i.e., chemistry) language skills of the students mitigated their comprehension of the examination questions. Interestingly, results also indicated a significant and positive effect of reading scores on achievement for the continuing education group (b =.31) and not the traditional group This finding has implications for non-traditional graduate students.
The degree that graduate students’ reading skills impact their achievement level in courses requiring critical thinking skills has been documented by researchers in academic contexts such as hypermedia systems and computer applications (Zhu, 1999), business accounting (Pritchard et al., 1999), and, more recently, in a research methodology course (Onwuegbuzie et al., 2001). Zhu (1999) found that the utilization of link filtering systems, as a component of hypermedia system design, was impacted by the reading ability (i.e., reading comprehension) of the graduate students participating in the study. Zhu (1999) interpreted the results to suggest that cognitive overload associated with inefficient information process occurring in the reading process impacted graduate students’ performance as well as their attitudes toward hyperlink systems.
Pritchard et al. (1999) found that the cumulative grade point averages of accounting students (n = 235) were statistically significantly related to their reading comprehension and vocabulary scores on the NDRT. These results were interpreted to support the inclusion of reading comprehension strategies into the instruction of accounting.
Onwuegbuzie et al. (2001) identified several study skill weaknesses in the areas of note taking and reading skills among graduate students participating in the study. In particular, Onwuegbuzie et al. found that reading skill was a predictor of achievement in research methodology courses, and that approximately 87% of students (n = 122) demonstrated inadequate metacognitive awareness of content of the text by reporting that they could read several pages of a textbook without understanding its content. Students’ self-report data indicating that they experience difficulty understanding textbook content is supported by research revealing that graduate students, in contrast to undergraduates, are 3.5 times more likely to report that they procrastinate on their weekly reading assignments (Onwuegbuzie, 2004).
More recently, Onwuegbuzie and Collins (2002) assessed graduate students' reading comprehension and reading vocabulary by comparing their scores on the NDRT (Brown et al., 1993) to scores obtained by a large normative sample of undergraduates. These researchers found that the graduate students attained statistically significantly higher scores on the reading comprehension portion of the NDRT (Brown et al., 1993) than did a normative sample of 5,000 undergraduate students from 38 institutions studied by Brown et al. The effect size (i.e., Cohen’s [1988] d) associated with this difference was .71 (Hedges & Olkin’s [1985] z-based 95% Confidence Interval [CI} = .45, .97), which, using Cohen's (1988) criteria, was large. (According to Hess & Kromrey [2002], the Hedges & Olkin’s [1985] z-based confidence interval procedure compares favorably with all other methods of constructing confidence bands, including Steiger & Fouladi’s [1992, 1997] interval inversion approach.) Similarly, the graduate students achieved statistically significantly higher scores on the reading vocabulary portion of the NDRT than did the normative sample. The effect size associated with this difference was .45 (95% CI = .19, .71), which was deemed to be moderate (Cohen, 1988). Despite, the overall higher reading vocabulary and reading comprehension scores attained by the graduate students, Onwuegbuzie and Collins (2002) noted that some of the graduate students in the study received very low reading comprehension and vocabulary scores, with the lowest scores representing the 14th and 24th percentiles with respect to the normative undergraduate students' scores. This latter finding raises the following question:, to what degree are graduate students who score at the low end of the reading comprehension and/or reading vocabulary continuum more likely to struggle in courses that require the understanding of relatively complex material? Indeed, DuBoulay (1999) contends that students’ ineffectual use of strategies and, overall, their limited experiences identifying and responding to written arguments thwart college students’ comprehension of textbook material as well as curtail their abilities to decipher a written argument and to apply it to contemporary situations.
As a result, it is possible that students with the lowest levels of reading comprehension and/or vocabulary, as identified by measures such as the NDRT, are more likely to experience problems in research methodology courses. Thus, the purpose of the present investigation was to determine the relationship between their reading ability, specifically their reading comprehension and reading vocabulary, and their understanding of research concepts, methodologies, and applications. It was hypothesized that such an association exists among graduate students.
Method
Participants
Participants were 91 graduate students from various disciplines, enrolled in five sections of an introductory-level educational research course at a southeastern university. The vast majority of the sample was female (89.01%) and White (96.70%). In order to participate, students were required to complete an informed consent document.
Instruments and Procedure
Students were administered the NDRT Form G (Brown et al., 1993). The NDRT measures reading comprehension (38 items), reading vocabulary (80 items), and reading rate. However, only the reading comprehension and reading vocabulary scores were used in the present investigation. This test was utilized because of its widespread use over many years, its psychometric properties (i.e., score reliability and validity), and the fact that normative data are available on large samples of high school and students from two-year and four-year colleges. Interestingly, however, no normative information has been obtained for graduate students—thereby providing a further justification for the study. For the present study, score reliability, as measured by KR-20, was .91 (95% confidence interval [CI] = .88, .93) for the reading comprehension subtest and .82 (95% CI = ..76, .87) for the reading vocabulary subtest.
Conceptual knowledge, which involved students' knowledge of research concepts, methodologies, and applications, was measured individually in all sections via comprehensive written midterm and final examinations. The midterm examination form consisted of open-ended questions, involving items that required conceptual knowledge of the research process. All of the items pertained to content from the first half of the course were chosen from the instructor's item bank to ensure that the examination was typical of past examinations given by the instructor. The final examination also was constructed by the course instructor and paralleled the format of the midterm examination, yet covered the complete course content. Both the midterm and the final examination were administered under untimed conditions, and were scored on a 100-point scale by the instructor, using a key that specified the number of points awarded for both correct and partial-credit answers.
Results
An independent samples t-test revealed that the graduate students (M = 69.45, SD = 5.17) obtained statistically significantly higher (t = 6.26, p < .0001) scores on the reading comprehension portion of the NDRT than did a normative sample of 5,000 undergraduate students from 38 institutions studied by Brown et al. (1993) (M = 61.60, SD = 11.94). The effect size associated with this difference was .66 (95% CI = .45, .87), which, using Cohen’s (1988) criteria, was moderate to large. Similarly, the graduate students (M = 68.85, SD = 6.20) obtained statistically significantly higher (t = 3.59, p < .0001) scores on the reading vocabulary portion of the NDRT than did the normative sample (M = 64.52, SD = 11.46). The effect size associated with this difference was .38 (95% CI = .17, .59), which, using Cohen’s (1988) criteria, was moderate to large.
Interestingly, no statistically significant difference (t = 0.63, p > .05) in reading comprehension scores was found between the current sample of graduate students and the graduate students used in Onwuegbuzie and Collins’ (2002) investigation (M = 70.00, SD = 5.28). Similarly, no statistically significant difference (t = 0.75, p > .05) in reading vocabulary scores was found between the present sample and that of Onwuegbuzie and Collins' (2002) investigation (M = 69.63, SD = 6.09).
Table 1 (located at end of article) displays the means and standard deviations pertaining to the four variables of interest. Table 2 (located at end of article) presents the inter-correlational matrix involving the two independent variables (i.e., reading comprehension and reading vocabulary) and the two dependent variables (i.e., midterm and final examination). Of particular note is the statistically significant relationship between reading comprehension and reading vocabulary after applying the Bonferroni adjustment to control for Type I error. Using Cohen's criteria, this relationship indicated a moderate-to-large effect size.
The strength of the relationship between the two sets of variables was assessed by examining the magnitude of the canonical correlation coefficients. These coefficients represented the degree of relationship between the weighted reading ability variables and the weighted research achievement variables. In addition, the significance of the canonical roots was tested via the F-statistic based on Rao's approximation (Rao, 1952).
The canonical analysis revealed that both canonical correlations combined were statistically significant (F[4, 162] = 2.96, p < .05; Wilks’ Lambda = .86). However, when the first canonical root was excluded, the remaining canonical root was not statistically significant. Together, these results suggest that the first canonical function was statistically significant, but the second canonical root was not statistically significant. However, because the calculated probabilities are sensitive to sample size, particular attention should be paid to the educational (practical) significance of the obtained results (Thompson, 1980). The educational significance of canonical correlations typically is assessed by examining the proportion of variance shared (Thompson, 1980, 1984, 1988, 1990). Specifically, the canonical correlation indicates how much variance the sets of weighted original variables share with each other (Thompson, 1988). In the present study, the first canonical correlation (Rc1 = .33) appeared to be moderately educationally significant, contributing 10.9% (i.e., Rc12) to the shared variance. However, the second canonical correlation (Rc2 = .11) did not appear to be educationally significant. Consequently, only the first canonical correlation was interpreted.
Data pertaining to the canonical root are presented in Table 3 (located at end of article). This table provides both standardized function coefficients and structure coefficients. An examination of the standardized canonical function coefficients revealed that, using a cutoff correlation of 0.3 recommended by Lambert and Durand (1975) as an acceptable minimum loading value, both reading comprehension and reading vocabulary made an important contribution to the achievement composite--with reading comprehension clearly being the major contributor. With respect to the achievement set, only the midterm examination scores made an important contribution to the composite set.
The structure coefficients (Table III) indicated that both reading ability dimensions made important contributions to the first canonical variate. The square of the structure coefficient (Table III) indicated that reading comprehension and reading vocabulary made large contributions, explaining 85.0% and 53.0% of the variance, respectively. With respect to the research achievement cluster, both the midterm and final examination made noteworthy contributions, with the midterm scores making an extremely large contribution--explaining 98.0% of the variance.
Discussion
The finding that the graduate students in the present investigation attained higher scores on both the reading comprehension and reading vocabulary portions of the NRDT than did the normative sample of undergraduates provides incremental validity to the NDRT as a measure of reading ability. Further, the results from the current inquiry replicate the findings of Onwuegbuzie and Collins (2002) in as much as the reading comprehension and reading vocabulary scores for both samples were very similar. Therefore, it appears that, in general graduate students have higher levels of reading comprehension and reading vocabulary than do undergraduate students. To the extent that reading ability is central in graduate programs, this finding is particularly encouraging.
Notwithstanding, this conclusion regarding the reading ability of graduate students should be tempered by the fact that, as was the case in Onwuegbuzie and Collins’ (2002) study, several students obtained reading comprehension and/or reading vocabulary scores that were extremely low. Indeed, the lowest reading comprehension and reading vocabulary scores represented the 28th and 30th percentiles, respectively, with regard to the normative undergraduate students’ scores. In Onwuegbuzie and Collins’ (2002) study, the lowest-scoring students achieved at the 14th and 24th percentiles, respectively.
The present research was able to extend the work of Onwuegbuzie and Collins (2002) by determining whether these lowest-scoring students were more at-risk for failure in research classes. Specifically, the canonical correlation analysis revealed that both reading comprehension and reading vocabulary were moderately significant predictors of graduate students’ understanding of research concepts, methodologies, and applications. To the extent that this relationship is causal, the present finding suggests that students’ difficulties in research courses, to some degree, may be the result of inadequate reading ability, which would have implications for the instruction of these classes.
Bearing in mind the fact that graduate textbooks in general and graduate-level research textbooks in particular typically are more complex than are the books assigned to undergraduate students, having low levels of reading comprehension and reading vocabulary may be particularly debilitating for students, and may explain why as many as 60.0% of graduate students procrastinate on keeping up with the weekly reading assignments (Onwuegbuzie, 2004). As noted by Du Boulay (1999), college students are often overwhelmed by the quantity of reading required. Having inadequate levels of reading comprehension or reading vocabulary, as some students exhibited in the current study, may lead graduate to students being even more overwhelmed by reading assignments. These students may experience debilitating elevations in research-based anxiety (e.g., research anxiety, statistics anxiety) levels and other types of state-based anxiety (Ferrari, 1991; Milgram, 1991; Onwuegbuzie, 1997; Rothblum, Solomon, & Murakami, 1986; Solomon & Rothblum, 1984). Low levels of reading vocabulary also may help to explain the prevalence of writing anxiety among graduate students (Onwuegbuzie, 1999a; Onwuegbuzie & Collins, 2001).
Additionally, the relationship between reading comprehension and reading vocabulary scores may explain why graduate students are nearly 3.5 times more likely to report that they nearly always or always procrastinate on completing required weekly readings than are undergraduate students (Onwuegbuzie, 2004). Thus, future research should investigate whether graduate students with the lowest reading comprehension and reading vocabulary scores are more likely to procrastinate with respect to reading assignments than are their counterparts. Further, because a significant relationship between anxiety and procrastination prevails among graduate students, it is likely that reading ability, research-based anxiety, and academic procrastination are inextricably linked. This should be the subject of future investigations.
Onwuegbuzie et al. (2004) found that approximately 87% of graduate students report that they often read several pages of a textbook without understanding its content. It is likely that students with the lowest reading comprehension and reading vocabulary scores are more apt to continue reading research textbooks even when they do not understand the material. Indeed, this is consistent with DuBoulay’s (1999) conclusions that, overall, graduate students demonstrate a lack of adequate metacognitive awareness while reading and have a limited repertoire of useful strategies for comprehending written text and deciphering a written argument. Thus, future studies are needed in which the metacognitive awareness pertaining to reading of graduate students with low reading comprehension and reading vocabulary scores are investigated.
As noted by Brown and Day (1983), undergraduate students with low verbal ability also are unable to summarize, to select the topic sentence, to invert a topic sentence if it is implied, and to write a synopsis of a paragraph in the absence of an explicitly stated topic sentence. These attributes have been identified as representing important study skills among undergraduate students (Lammers, Onwuegbuzie, & Slate, 2001). It is possible that these links also generalize to graduate students. In any case, this warrants future inquiries.
The fact that participants in the study were predominantly female is an important limitation of the study. Thus, replications are needed using a large number of male graduate students. Additionally, bearing in mind that racial and ethnic differences in reading achievement have been found repeatedly at the public school level (e.g., Campbell, Voelkl, & Donahue, 1999; Coleman, 1999; Diamond & Onwuegbuzie, 2001; Donahue, Voelkl, Campbell, & Mazzeo, 1999; McKenna, Kear, & Ellsworth, 1995), the current study needs to be replicated using a much more diverse sample.
Nevertheless, an important practical implication can be derived from the results of the present study. Specifically, because of the fact that reading comprehension and reading vocabulary both predict performance in research methodology courses, and because approximately two-thirds of graduate students report that they want to reduce their tendencies to procrastinate when undertaking reading assignments, instructors of these courses should find ways to help them accomplish this goal. For example, statistics and research methodology instructors could ask or require students to undertake a written or oral summary of each assigned reading. Moreover, instructors can request that students create advance- or post-organizers (e.g., concept maps) of all material read in their research textbooks, because these techniques have been found to increase levels of performance (Onwuegbuzie, 1999b).
Another important implication of this finding is that the NDRT appears to be a useful tool for identifying graduate students who are at-risk for underachievement in research methodology courses. Because theses and dissertations tend to necessitate adequate research skills (Onwuegbuzie, 1997), and because a significant proportion of students do not complete their theses and dissertations, and hence their graduate degree programs (Bowen & Rudenstine, 1992; Cesari, 1990), it is possible that early identification of problems in reading comprehension and reading vocabulary may play an important role in helping to reverse this trend in non-completion rate.
References
Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading. In P. D. Pearson (Ed.), Handbook of reading research (pp. 255-293). New York: Longman.
Bowen, W. G., & Rudenstine, N. L. (1992). In pursuit of the Ph.D. Princeton, N.J.: Princeton University Press.
Brown, J. I., Fishco, V. V. & Hanna, G. (1993). Nelson-Denny Reading Test—Manual for scoring and interpretation, Forms G and H. Itasca, IL: Riverside Publishing.
Brown, A. L., & Day, J. J. (1983). Macrorules for summarizing texts: The development of expertise (Tech Rep No. 270). Urbana: University of Illinois, Center for the Study of Reading.
Campbell, J., Voekl, K., & Donahue, P. (1999, February 28). Report in brief: NAEP 1996 trends in academic progress. [WWW document] URL http://nces.ed.gov/nationsreportcard/96report/
Cesari, J. P. (1990). Thesis and dissertation support groups: A unique service for graduate students. Journal of College Student Development, 31, 375-376.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: John Wiley.
Coleman, B. (1999, February 28). The 1994 NAEP Trial Assessment [WWW document] URL http://nces.ed.gov/nationsreportcard/y25sa/tsa94.shtml
Diamond, P. J., & Onwuegbuzie, A. J. (2001). Factors associated with reading achievement and attitudes among elementary school-aged students. Research in the Schools, 8, 1-11.
Donahue, P., Voekl, K., Campbell, J., & Mazzeo, J. (1999). NAEP 1998 reading report card for the nation and the states. Washington, D. C.: U.S. Department of Education Office of Educational Research and Improvement.
DU Boulay, D. (1999). Argument in reading: What does it involve and how can students become better critical readers? Teaching in Higher Education, 4, 147-162.
Ferrari, J. R. (1991). Compulsive procrastination: Some self-reported characteristics. Psychological Reports, 68, 455-458.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
Hess, M. R., & Kromrey, J. D. (2002, April). Interval estimates of effect size. An empirical comparison of methods for constructing confidence bands around standardized mean differences. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Lambert, Z. V., & Durand, R. M. (1975). Some precautions in using canonical analysis. Journal of Market Research, XII, 468-475.
Lammers, W. J., Onwuegbuzie, A. J., & Slate, J. R. (2001). Academic success as a function of the sex, class, age, study habits, and employment of college students. Research in the Schools, 8(2), 71-81.
Mason, J. M. (1984). A schema-theoretic view of the reading process as a basis for comprehension instruction. In G. G. Duffy, L. R. Rochler, & J. Mason (Eds.), Comprehension instruction (pp. 26-38). New York: Longman.
McKenna, M., Kear, D., & Ellsworth, R. (1995). Children’s attitudes toward reading: A national survey. Reading Research Quarterly, 30, 934-956.
Milgram, N. A. (1991). Procrastination. In Encyclopedia of human biology, (Vol. 6, pp. 149-155). New York: Academic Press.
Onwuegbuzie, A. J. (1997). Writing a research proposal: The role of library anxiety, statistics anxiety, and composition anxiety. Library & Information Science Research, 19, 5-33.
Onwuegbuzie, A. J. (1999a). Writing apprehension among graduate students: Its relationship to self-perceptions. Psychological Reports, 84, 1034-1039.
Onwuegbuzie, A. J. (1999b). The effect of advance organizers in research methodology courses. National Forum of Applied Educational Research Journal-Electronic, 12E(3), 83-91.
Onwuegbuzie, A. J. (2004). Academic procrastination and statistics anxiety. Assessment & Evaluation in Higher Education, 29(1), 3-18.
Onwuegbuzie, A. J., & Collins, K. M. T. (2001). Writing apprehension and academic procrastination among graduate students. Perceptual and Motor Skills, 92, 560-562.
Onwuegbuzie, A. J., & Collins, K. M. T. (2002). Reading comprehension among graduate students. Psychological Reports, 90, 879-882.
Onwuegbuzie, A. J., Slate, J. R., & Schwartz, R. A. (2001). The role of study skills in graduate-level educational research courses. Journal of Educational Research, 94, 238-246.
Paris, S. (1987). Introduction to current issues in reading comprehension. Educational Psychologist, 22, 209-213.
Pritchard, R. E., Romeo, G. C., & Muller, S. A. B. (1999). Integrating reading strategies into the accounting curriculum. College Student Journal, 33, 77-81.
Rao, C. R. (1952). Advanced statistical methods in biometric research. New York: Wiley.
Rothblum, E. D., Solomon, L. J., & Murakami, J. (1986). Affective, cognitive, and behavioral differences between high and low procrastinators. Journal of Counseling Psychology, 33, 387-394.
Solomon, L. J., & Rothblum, E. D. (1984). Academic procrastination: Frequency and cognitive-behavioral correlates. Journal of Counseling Psychology, 31, 503-509.
Steiger, J. H., & Fouladi, R. T. (1992). R2: A computer program for interval estimation, power calculation, and hypothesis testing for the squared multiple correlation. Behavior Research, Methods, Instruments, and Computers, 4, 581-582.
Steiger, J. H., & Fouladi, R. T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 221-257). Mahwah, NJ: Erlbaum.
Thompson, B. (1980, April). Canonical correlation: Recent extensions for modeling educational processes. Paper presented at the annual meeting of the American Educational Research Association, Boston, MA.
Thompson, B. (1984). Canonical correlation analysis: Uses and interpretations. Newbury Park, CA: Sage Publications. (ERIC Document Reproduction Service No. ED 199 269).
Thompson, B. (1988, April). Canonical correlation analysis: An explanation with comments on correct practice. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED 295 957)
Thompson, B. (1990, April). Variable importance in multiple regression and canonical correlation. Paper presented at the annual meeting of the American Educational Research Association, Boston, MA. (ERIC Document Reproduction Service No. ED 317 615)
Van Lanen, R. J., Lockie, N. M., & McGannon, T. (2000). Predictors of nursing students’ performance in a one-semester organic and biochemistry course. Journal of Chemical Education, 77, 767-770.
Wixson, K. K., & Peters, C. W. (1984). Reading redefined: A Michigan Reading Association position paper. Michigan Reading Journal, 17, 4-7.
Zhu, E. (1999). Hypermedia interface design: The effects of number of links and granularity of nodes. Journal of Educational Multimedia and Hypermedia, 8, 331-358.
You are invited to join AE Extra staff!
Send your ideas and/or writing sample
to the Editor-in-chief:
Elizabeth Haller
Kent State University (e-mail: editoraee@hotmail.com)
Academic Exchange Extra invites reader response to any writings in this issue--especially articles advancing the scholarly debate of issues raised.
Web Master: Nicholas Eastham
