Evaluating Research Methods:
Assumptions, Strengths, and Weaknesses of Three Educational Research Paradigms
Angela M. Velez, PhD
Faculty Development Coordinator
In educational research, two distinct research paradigms are common—qualitative and quantitative. Scholars who utilize these paradigms have generally deemed themselves either qualitative or quantitative researchers, called “purists” (Onwuegbuzie & Leech, 2005). Rarely, though, does one see a qualitative researcher cross the border to quantitative territory, and vice versa, because the epistemological and axiological assumptions are distinct in these research paradigms. However, there has been a recent surge in the use of mixed methodology research in education because the data that emerge from these studies involve crossing borders and mixing methods to understand problematic situations; data are stronger, more valid, more generalizable, and tend to be much more extensive than data that emerge from single-method studies. This article seeks to explore the assumptions, strengths, and weaknesses of all three research paradigms—qualitative, quantitative, and mixed methods—to inform doctoral students or doctoral candidates about methodological issues that need consideration when they consider a research study.
Assumptions, Strengths, and Weaknesses of Paradigms
Assumptions: Quantitative Research
Quantitative research, like other research paradigms, has its own set of assumptions. For example, it seeks to understand the facts or causes of phenomena and does not regard the subjective states of a situation or of individuals (Reichardt & Cook, 1979); it also claims to be value-free (Stanfield, 2006). Further, quantitative research defends positivism and believes in the benefit of hypothetical-deductive procedures (Morales, 1995). Quantitative researchers claim that science is rooted in objective verification but do not account for the subjective nature of the researcher’s decisions made throughout the stages of the research process (Onwuegbuzie & Leech, 2005). In addition, often the epistemological assumptions of positivistic quantitative inquiry are that what happens in one social environment being studied in this method can be generalized to future social situations (Gall, Gall, & Borg, 2003). Lastly, quantitative inquiry assumes that science is the superior way of knowing, understanding and predicting human experiences and that the positivistic scientific method rules must be adhered to or the researchers and their findings are disregarded (Stanfield, 2006).
In sum, quantitative assumptions regard reality, experience, and situations as quantifiable, as measurable. What is not measurable is not worthy of being reported, and if something is
measured, validated and generalizable, then that something is generalized to all populations that are similar to what was studied and reported because reality is independent to personal experience (Gall, Gall & Borg, 2003).
Assumptions: Qualitative Research
Qualitative researchers are not immune to significant assumptions. First, qualitative inquiry believes that reality is subjective and that social environments are personal constructs created by individual interpretations that are not generalizable (Gall, Gall & Borg, 2003); these beliefs are rooted in constructivism rather than positivism. Therefore, the assumption is that there is not a generalizable reality that is quantifiable for a larger population than an individual case. Qualitative researchers assume that rich description and a deep understanding are indicative of their methodology, which insinuates that other research paradigms are not deep (Sechrest & Sidani, 1995). In terms of axiology, qualitative researchers believe that research is influenced by the values held by the researcher as well as by the theories, hypotheses or the framework that the researcher is using in his or her particular situation (Tashakkori & Teddlie, 1998).
One area of chief interest and concern in qualitative inquiry is context. Qualitative inquiry assumes that without a clear understanding of the contextual nature of a research study the reported data are not generalizable (Sechrest & Sidani, 1995). Overall, qualitative researchers not only believe in contextualized perspectives but also those which are pluralistic, interpretive, and open-ended (Cresswell & Miller, 2000). To qualitative researchers, the world is “shifting, changing, dynamic” (Filstead, 1979).
Assumptions: Mixed Methods Research
The last paradigm to explore in terms of assumptions is mixed methods. The largest assumption that the mixed method paradigm has is that most comprehensive research has a combination of both quantitative and qualitative methods in their studies. Researchers who employ mixed methods are not research “purists”, and most scholars who use strictly quantitative or qualitative designs do not welcome mixed methods with open arms (Tashakkori & Teddlie, 1998). Hence, they assume that to have a strong study with valid and generalizable results, one cannot use only one research method.
Mixed method research has its roots not in positivism or constructivism, but in pragmatism. To a mixed methods researcher, the method is not as important as the type of research question that is asked (Cresswell, 2003). Mixed methods researchers do not subscribe to any one philosophy; thus, they can use assumptions from both quantitative and qualitative paradigms as the research question deems appropriate (ibid.). In addition, while quantitative and qualitative approaches may have their own limitations separately, mixed methods researchers feel that those limitations can be lessened by choosing methods that compliment each other (Hammond, 2005). Scholars who believe strongly in mixed methodology research believe that mono-method research is a great threat to advancement in areas like education (Onwuegbuzie & Leech, 2005).
Strengths: Quantitative Research
In quantitative research, a noted strength is that what is being studied is not affected by the researcher as he or she is not usually involved with the subject or subjects being studied. What this does is decrease the chances of people’s responses or behaviors being affected or influenced by the outside researcher (Gall, Gall & Borg, 2003). This strength, however, has been questioned by others as it is improbable that no influence at all takes place in research, regardless of the method or paradigm used.
Another strength of quantitative study is that it has been a strong component of a great deal of published research data, and because of its ubiquity, rules, processes, templates, regulations and other guiding principles are available for researchers to use, copy, and clarify their research designs (ibid.) This means that there is consistency in process and procedure for studies that are labeled causal-comparative, for example, or quasi-experimental, to name just a few. Quantitative studies are said to be replicable and often the instruments created for quantitative studies are used in further research due to the rigors of creating an instrument that effectively measures a certain construct that can be used in different social or educational contexts while also being valid and reliable (Creswell, 2003).
Johnson and Onwuegbuzie (2004) suggest some strengths of quantitative research are that it tests and validates theories that are presently constructed about how and sometimes why phenomena happen; the data collection process is fairly quick; data is precise and numerical; it allows for generalizability when data are drawn from fairly large random samples (p. 19). Finally, quantitative studies do have strength in that they produce answers (if done properly) that are solid, unlike an opinion or common sense answer (Ratnesar & Mackenzie, 2006).
Strengths: Qualitative Research
Qualitative methods tend to be rich in narrative and description, and instead of providing an outcome they tend to discuss the process. Contrary to the goals of qualitative inquiry, quantitative inquiry does not attempt to first understand the contexts in which the humans being studied are interacting. While quantitative data can describe data numerically, it cannot and does not go one step further, as qualitative inquiry does, and first make sense of what is being observed, then understand it, and lastly discover the meaning through detailed explanations that do not exist in quantitative studies (Filstead, 1979).
Qualitative inquiry increases individual understanding of “otherness” through its in-depth studies of specific groups, for the goal is not to explain their reality but to understand it (Morales, 1995). The understanding of this “otherness” is rooted in the tradition’s anthropology, sociology, human geography, cultural studies, and social psychology (Atkinson & Delamont, 2006). While they are not traditional sciences, their histories are long and rich and provide a firm foundation for high-quality studies in the qualitative paradigm. For Johnson and Onwuegbuzie (2004), qualitative inquiry’s strengths include that: it is useful for describing complex phenomena; it is usually collected in naturalistic settings; it is responsive to local conditions and the needs of those studied and those who are informed by the study; the words of the participants lend to further studies into how and why phenomena occur; it allows for the study of dynamic processes (p. 20). If it does nothing except “bring about healing, reconciliation, and restoration between the researcher and the researched” (Stanfield, 2006, p. 725), it will have made a positive impact on the lives of at least the people involved in a qualitative study.
Strengths: Mixed Methods Research
Mixed methodology’s first strength is that it is a “workable solution” to the seemingly unending debates between qualitative and quantitative purists (Johnson & Onwuegbuzie, 2004). Another strength lies in its purpose, which is significant (Greene, Caracelli & Graham, 1989). Through the research of Greene, et. al. (1989), mixed methods researchers can ground their research in meaningful theory with a meaningful purpose. Their empirical study found five general purposes for mixed method paradigms: triangulation, complementarity, development, initiation, and expansion (p. 255).
Quantitative and qualitative methodologies have their own biases and limitations as separate entities, but when two methods are chosen distinctly because their biases and limitations cancel each other out, the result is triangulation as well as a much more solid study than if just one method was used (ibid.). The combining nature of this method is what makes it an intriguing alternative to a mono-method study. Unlike the dichotomies of qualitative and qualitative, mixed methods is both deductive and inductive; it is both objective and subjective; it is entirely practical and applicable to many researchers as it allows a researcher to study what is important to him or her, it allows the researcher to vary the methods used to study that interest, and it allows the researcher to use the results of the study to create positive movements in the researcher’s own specific area of interest (Tashakkori & Teddlie, 1998). It allows a researcher to begin the study with a positive attitude towards both quantitative and qualitative inquiry; it allows him or her to narrow or expand a focus as needed, where he or she can dive much further into the data to understand meaning than they could if a single method was used (Onwueguzie & Leech, 2005). Mixed methods research is flexible in that words can be used to add meaning to the quantitative data and numbers can be used to inform or supplement the words; the researcher can explore a broad range of questions because he or she is not restrained by their research methodology’s individual paradigm; finally, it provides stronger concluding evidence through convergence and corroboration of the research findings (Johnson & Onwuegbuzie, 2004, p. 21).
While all three methods do have similarities—they are all empirical, they all seek to describe data, they all configure arguments to explain that data, and they all theorize about why a phenomena occurred in the way that it did (Sechrest & Sidani, 1995)—their differences are seen as weaknesses by paradigm purists. For example, qualitative purists argue that quantitative inquiry does not offer any narrative explanations of the data, and sometimes those explanations are needed to understand the contexts in which the data were gathered. Further, quantitative inquiry claims to be value-free (free from researcher bias, unlike the value-laden qualitative research), but some scholars argue that this is not possible because researchers are humans and cannot be neutral or value free in any circumstance (Stanfield, 2006).
Weaknesses: Quantitative Research
One of the first weaknesses of quantitative inquiry that sets it apart from qualitative and mixed methods is that it is difficult to read and understand (Burns, 2000). The statistical aspects of a quantitative report can be technical and difficult to distinguish for average readers of educational journals.
Quantitative research has seen resurgence in the American edu-political system due to the 2002 No Child Left Behind Act and the 2001 Education Sciences Act, which base much of their claims on quantitative data, and it is the quantitative data that matters to policy makers (Wright, 2006). What is considered worthy of report and policy influence are quantitative data, which leaves many educators shaking their heads as these data are small representations of what is really going on in American schools and in American higher education. Some scholars and philosophers note that while quantifying information might be the “path of least resistance”, it might also be “the path of least significance” (Wartofsky, 1968 cited in Gall, Gall & Borg, 2003). This criticism is related to quantitative inquiry’s emphasis on statistical significance without regard for the significance of the effects involved (Tashakkori & Teddlie, 1998).
Weaknesses: Qualitative Research
Scholars outside of the United States have criticized American qualitative research and do not understand why it is that qualitative research is not worthy of the respect it has earned in other countries like the United Kingdom (Atkinson & Delamont, 2006). It has been suggested that American qualitative researchers do not look at their educational systems as problematic but rather as institutions that need exploring in order to define or describe the nature of the institutions versus institutions that have significant problems in need of resolution (Atkinson & Delamont); further, qualitative researchers need to avoid ethnocentricity and perform their research with a firm understanding of its very rich history in order “to produce thick descriptions in a reflexive manner” (p. 750). It could be possible that this is due to the trend of American institutions of higher education orienting their graduate students with competencies in mainly one (quantitative) research paradigm and not enough in any other (qualitative and mixed); what happens is that young scholars are not taught that research is a holistic process (Onwuegbuzie & Leech, 2005), not a dichotomous battle between competing methodologies whose sides they must choose between in order to fit in the inflexible walls of the American Academy.
A weakness of qualitative inquiry, from a quantitative perspective, is that its context-rich, value-laden, narrative-filled reports contain too much fluff and not enough hard evidence. Further, since the researcher is actively involved in the study and is likely passionate about either who is being studied or the context of the study that passion can be, and is, a form of bias and can perhaps lead to an inability to see other explanations for what is taking place (Krantz, 1995).
Because qualitative inquiry, by nature, is much more focused on a small population or case, it is criticized for its inability to include a broader range of information in its analyses; it is also negatively noted that qualitative researchers might express personal opinion more than report accurate findings from the study and choose to be selective in what they report as results (Tashakkori & Teddlie, 1998).
In terms of practicality, the more insightful the qualitative research is, the more time consuming it becomes (Atkinson & Delamont, 2006). Data gathering is not a simple process and involves many opportunities for human error, unlike some quantitative methods that involve inputting numerical data into a software program to get almost immediate results. Due to its narrative aspect, errors can and do occur in all stages of qualitative research. For example, language is the root of qualitative research; because of that, meaning is incredibly important. That importance is significant because a certain term or phrase might mean one thing to one participant in a study but mean another thing to another participant, and it also might mean something entirely different to the researcher (Gall, Gall & Borg, 2003).
A final caveat of qualitative inquiry is that it is more difficult to find scholarly resources that actively publish qualitative manuscripts than it is to find scholarly research for quantitative reports (Krantz, 1995). The reason might be that, unlike quantitative inquiry, qualitative methods do not yet have rigorous standards for reporting findings; another reason is that it is facing a great deal of backlash due to an increased focus on quantitative empirical data in education related to new government policies, which are fundamentalist and positivist in conception (Denzin & Lincoln, 2005).
Weaknesses: Mixed Methods Research
In terms of mixed method designs, it is important to understand that mixed method designs are not always the appropriate choice for all research (Onwuegbuzie & Leech, 2005). The question must decide which approach is best. Besides the appropriateness for the research question, mixed methods research is not easy and can be challenging for one researcher to carry out both a quantitative and qualitative study, either concurrently or in succession, as the researcher not only has to be familiar with both schools of research but he or she must also know how to combine them appropriately (Johnson & Onwuegbuzie, 2004).
Like qualitative inquiry, mixed methods research is time-intensive and can be quite costly (ibid.). If the study is done correctly, it must be planned carefully, and it must have a clear rationale that is defensible, which is part of the time-consuming aspect of this methodology (Greene, et.al., 1989). While it is possible to create a fantastic mixed method study, this approach is demanding in that it requires flexibility from the researcher to be adaptive to the needs of the problem being studied (Reichardt & Cook, 1979; Greene et. al., 1989).
It is clear by viewing the size of the paragraphs devoted to the weaknesses of the three paradigms that mixed methods seems to have the least amount of weaknesses; this is due to the fact that mixed methods research is not limited by epistemological, axiological and ontological assumptions that restrain mono-method research. Because of the freedom to choose paradigms that fit the research question, mixed methods essentially gives researchers the best of both worlds, but with that freedom comes a great deal of constraints for the researcher in terms of choosing methods to fit the question, as well as financial and time concerns.
While many scholars and researchers believe that the quantitative, qualitative, and mixed methods research paradigms are so divergent that they absolutely cannot be combined with any other (Smith, 1983; Lincoln & Guba, 1985), scholars like Onwuegbuzie and Leech (2005) offer that there are more similarities between the research paradigms than most people have considered, as each:
- involve the use of observation to answer research questions
- use safeguards to minimize bias and invalidity
- attempt to triangulate data
- attempt to provide explanations of findings
- interpret and create narrative conclusions about findings
- select and use analytical techniques to gain maximum meaning
- attempt to explain complex relationships
- utilize techniques to verify the data
- tend to use data reduction techniques
The strengths in these research methods lie in their methodologies—the veritable steps taken from creating a research question to answering it—which are surprisingly similar. The weaknesses, however, lie in the paradigmatic limitations created by epistemological, axiological, and ontological assumptions (mostly for the mono-method studies). While mixed methods research is not a methodological panacea, it is certainly a step towards greater methodological choice for educational researchers, both amateur and experienced, for generations to come.
Atkinson, P., & Delamont, S. (2006). In the roiling smoke: Qualitative inquiry and contested fields. International Journal of Qualitative Studies in Education, 19(6), 747-755. Retrieved February 2, 2008 from SAGE.
Burns, R.B. (2000). Introduction to Research Methods. London: SAGE.
Cresswell, J. (2003). Research design: qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: SAGE.
Cresswell, J., & Miller, D. (2000). Determining validity in qualitative inquiry. Theory Into Practice , 39 (3), 124-130. Retrieved August 28, 2006 from Academic Search Premier.
Denzin, N.K. & Lincoln, Y.S. (Eds.) (2005). Handbook of Qualitative Research. Thousand Oaks, CA: SAGE.
Filstead, W. J. (1979). Qualitative methods - a needed perspective in evaluation research. In Cook, T. D. and Reichardt, C. S. (eds). Qualitative and Quantitative Methods in Evaluation Research (pp. 33-48). London: Sage.
Gall, M., Gall, J., & Borg, W. (2003). Educational Research. Boston: Pearson Education, Inc.
Greene, J.C., Caracelli, V.J., & Graham, W.F. (1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11(3), 255-274. Retrieved on February 1, 2008 from SAGE.
Hammond, C. (2005). The wider benefits of adult learning: An illustration of the advantages of multi-method research. International Journal of Social Research Methodology , 8 (3), 239-255. Retrieved on January 29, 2008 from Academic Search Premier.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm hose time has come. Educational Researcher , 33 (7), 14-26. Retrieved on February 9, 2008 from SAGE.
Krantz, D.L. (1995). Sustaining vs. resolving the quantitative-qualitative debate. Evaluation and Program Planning, 18(1), 89-96. Retrieved on January 29, 2008 from Academic Search Premier.
Lincoln, Y.S. & Guba, E.G. (1985). Naturalistic Inquiry. Thousand Oaks, CA: Sage.
Morales, M. (1995). Uses of qualitative/quantitative terms in social and educational research. Quality and Quantity, 29(1), 39-53. Retrieved on February 5, 2008 from Ebscohost SocIndex.
Onwuegbuzie, A., & Leech, N. (2005). On becoming a pragmatic researcher: The importance of combining quantitative and qualitative research methodologies. International Journal of Social Research Methodology , 8 (5), 375-387. Retrieved on January 29, 2008 from Academic Search Premier.
Patton, M. Q. (1990). Qualitative evaluation and research methods, 2nd Ed. London: SAGE.
Ratnesar, N. & Mackenzie, J. (2006). The quantitative-qualitative distinction and the null hypothesis significance testing procedure. Journal of Philosophy of Education, 40(4), 501-509. Retrieved on February 9, 2008 from Academic Search Premier.
Reichardt, C.S. & Cook, T.D. (1979). Qualitative and Quantitative Methods in Evaluation Research. London: SAGE.
Sechrest, L., & Sidani, S. (1995). Quantitative and qualitative methods: Is there an alternative? Evaluation and Program Planning , 18 (1), 77-87. Retrieved February 2, 2008 from Academic Search Premier.
Smith, J. (1983). Quantitative versus qualitative research: An attempt to clarify the issue.
Educational Researcher, 12(3), 6-13. Retrieved on February 5, 2008 from JSTOR.
Stanfield II, J. (2006). The possible restorative justice functions of qualitative research. International Journal of Qualitative studies in Education , 19 (6), 723-727. Retrieved on February 1, 2008 from Academic Search Premier.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches (Vol. 46). (L. Bickman, & D. Rog, Eds.) Thousand Oaks, CA: SAGE.
Wright, H.K. (2006). Are we (t)here yet? Qualitative research in education’s profuse and
contested present. International Journal of Qualitative Studies in Education, 19(6), 793-802. Retrieved on February 1, 2008 from Academic Search Premier.
You are invited to join AE Extra staff!
Send your ideas and/or writing sample
to the Editor-in-chief:
Kent State University (e-mail: email@example.com)
Return to AE Home
Academic Exchange Extra invites reader response to any
writings in this issue--especially articles advancing the scholarly debate
of issues raised.