27 Reading and appraising research articles: Quantitative research

As you start to source and review the research on your topic, you need to start making sense of what you are reading and how it can be applied to your own project. Whilst you are not expected to an expert in understanding all the facet of research, you do need to develop the foundational skills to critical apprise components of the literature so that you can be a knowledgeable consumer of research.

Content includes:

• Sections of a research journal article
• Critical aspects of appraisal.

A review of the peer-review process

Submitted papers or articles are often subjected to a peer review by the journal. Peer review is designed to assess the validity, quality and often the originality of articles for publication. Its ultimate purpose is to maintain the integrity of science by filtering out invalid or poor-quality articles. When the article is submitted, it will be reviewed by two or more peer reviewers who are experts in the same field. Most of the time these reviews are blinded, meaning the author of the article does not know who the reviewers are, and the reviewers do not know who the author is. As a consumer of research, you have some assurance that the journal articles are of quality and have been vetted. However, to note, is that this process is not the end-all-be-all. Meaning, just because it has been peer reviewed and published, it does not remove the need to critically appraise a publication.

Sections of published research journal articles

Journal articles follow a similar format regardless of which journal they are published in. There may be some small changes, but for the most part they follow the IMRAD format:

• Title and abstract
• Introduction
• Methods
• Results
• Discussion
• References.

Title and abstract

The title presents a starting point in determining whether or not the article has potential to be included in an evidence based practice review. Ideally, the title should be informative. It should also help the reader to understand what type of study is being reported.

Word of caution – do not rely solely on a title. They can be misleading. Be careful that it is not just an informative article or opinion piece.

In qualitative articles, the title will normally include the central phenomenon and group under investigation. In quantitative articles, the title communicates the key variables and the population (PIO components). Again, these are the usual title components, but not always.

The abstract will be at the start of the paper. It contains a brief description of the study. Abstract components include a brief introduction, aims/goals, methods/methodology, and the results and conclusions. Some abstracts also include the research question, hypothesis, and implications to practice.

This is located after the title it is usually set apart using a box, shading, or italics. The abstract is a great starting point in quickly filtering articles that may be of use. It’s a nice thumbnail sketch. But caution! Do not forgo reading the article. The abstract should serve as a screening device only.

Visit Nursing students’ attitudes toward e-learning: A quantitative approach for an example of a title and abstract.

Introduction

The introduction is the start of the article after the abstract. It normally does not contain a heading.

If the abstract appears relevant, the move onto reading the introduction. The introduction contains the background as well as a problem statement that tells the reader why the study was conducted. It is presented within the context of a current literature review, and it should contain the knowledge gap between what is known and what the study seeks to find. It usually contains a statement of expected results or hypothesis as well.

The introduction will usually include a description of the central concepts or variables (sometimes these are in the Methods section instead); the study purpose, research questions, or hypotheses; a review of literature; theoretical/conceptual framework, and the study significance/need for the study.

Visit Nursing students’ attitudes toward e-learning: A quantitative approach for an example of a introduction.

Methods

This section describes how a study is conducted. In quantitative studies, the methods section contains the research design, sampling plan, methods for measuring variables and collecting data, the study procedures, participant protection, and analytic methods and procedures. In qualitative studies, the methods section discusses many of the same issues as quantitative researchers do but with different emphases. More information is provided about the research setting in a more organically focused fashion, in the context of the study.

Visit Nursing student’s attitudes toward e-learning: A quantitative approach for an example of a the Method.

Results

The results section contains the findings of the study. This section is very objective without any researcher interpretation. These are the raw findings from data. The statistical tests will be named, the value of the statistics will be listed, statistical significance will be listed, and the level of significance is listed. The level of significance, which we will come back to later in the modules, is an index of how probable it is that the findings were not just simply due to chance and is listed as the p value. In qualitative studies, the results are a bit different. They are organized according to major themes (thematic results) and most often include raw data as quotes from the study participants.

Figures and tables are usually presented – some inferential and some descriptive. Also look to see whether the results report statistical versus clinical significance. You might have to look up unfamiliarly terminology regarding statistical tests.

Visit Nursing students’ attitudes toward e-learning: A quantitative approach for an example of a results.

Discussion

This section should be tied back to the introduction. The findings should be discussed and meaning given to the results. The weaknesses/limitations and bias should be reports.

Visit Nursing students’ attitudes toward e-learning: A quantitative approach for an example of a discussion.

Critical  aspects of appraisal

Critical appraising evidence is an essential step in evidence-based practice in response to a clinical question (Polit & Beck, 2021). The process of systematically evaluating research for its reliability, importance, and significance is critical appraisal (Hall & Roussel, 2014). The process is necessary in considering whether the research was conducted appropriately, because not all research is awesome.

Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value, and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability, and relevance.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognize any potential for bias.

The beginning guidelines for critical appraisal include asking:

1. How relevant is the research problem to the actual practice of nursing?
2. Was the study quantitative or qualitative? How did you make that determination?
3. What was the underlying purpose (or purposes) of the study? (Therapy/Intervention, Diagnosis/Assessment, Prognosis, Etiology/Harm, Description, or Meaning?)
4. What might be some clinical implications of this research? To what type of people and settings is the research most relevant? If the things were accurate, how might you use the results of this study?

An inference is a conclusion drawn. It is a challenge to design studies to support inferences that are reliable (accurate and consistent data; concerns the truthfulness in the data obtained and the degree to which any measuring tool controls random error, and if it can be reproduced under the same conditions) and valid (soundness of the evidence; validity is about what an instrument measures and how well it does so, and whether the results really do represent what they were supposed to measure) in quantitative studies, and that are trustworthy in qualitative studies. Thus, every article, even if peer-reviewed, needs to be critically appraised.

Bias

Ever ask someone’s opinion about the “best” brand of shoes or “best” local coffee? Their answer is based on their own experience, thus biased to their opinion. Bias is a distortion or an influence that results in an error in inference.

When reading articles, we need to be very aware of the potential for bias. Some examples of factors creating bias include:

• Lack of participants’ candor
• Faulty methods of data collection
• Researcher’s preconceptions
• Participants’ awareness of being in a special study
• Faulty study design
• Survey question design

Survey questions are very commonly biased.  With regard to bias in surveys, it is always wise to utilize participant anonymity as this would make it more likely to obtain truthful answers especially when questions are controversially-based.

Surveys are used frequently in research, so let’s dive into how surveys can thwart research results. Depending on how they are asked, questions can lead the respondent intentionally down a path to a certain answer or are phrased in a manner that can be confusing to them, leading to unclear responses. The following are some examples:

Leading questions

These are easy to spot, as it is clear there is a “correct” answer that the question is leading you toward. This will always result in false information as there was never an option for an honest response.

Example: “What problems did you have with the product?”

You can see that this question was set up in a way to assume there was something wrong in the first place.

Alternative: “On a scale of 1-10, how satisfied were you with the product?”

Vague or ambiguous questions

On the surface, these can look honest and harmless, but their vague nature can confuse the participant into a poor or inaccurate response.

Example: “How did this hospital experience compare with other hospital experiences?”

Have they had other hospital experiences? If not, then what? Is there a benchmark to compare against? What “experience”?… price? Friendly nurses? Good medical care? So many terrible about this question format.

Alternative: “On a scale of 1-10, with 10 being highly likely, how likely would you be to recommend this hospital to others?”

Double-barreled questions

These types of questions ask the participant to provide an opinion on two topics but only provides an opportunity for one response.

Example: “How satisfied are you with your hospital stay and discharge instructions?”

If they are satisfied with the hospital stay but not the discharge instructions, how do they answer this?

Alternative: Split the question into two questions.

Absolute questions

These questions can bias the participants’ choices by forcing them into an absolutely categorical response when they might not have one. These use words such as “never”, “always”, “will”, etc.

Example: “Do you always have eggs for breakfast?”

The problem with the above example is that the answer will be no. The chances of someone having eggs for breakfast every single day is very slim. Obviously, this skews the data because there are no alterative choices.

Alternative: Give options! “Do you eat of the following for breakfast? Select as many as applicable. Add any foods.”

__ Eggs
__ Bacon
__ Protein Shake
__ Pancakes
__ Fruit
__ Cottage Cheese
__ Other (specify)
__ Other (specify)

Acquiescence bias questions

These questions are usually a binary yes/no or agree/disagree choice. Guess what? People are often more likely to respond positively when only two options are presented.

Example: “Is our level of customer service satisfactory?” Yes/No”

We will learn nothing of value form these types of questions.

Alternative: Either offer multiple options or offer an open form text box, such as: “Please describe your experience with customer service.”

Well? Did you find this section on biased survey questions helpful?

Lastly, motivated respondent bias occurs when people are highly motivated to complete a survey in hopes of changing public opinion or influencing the outcome of the research. Avoiding acquiescence bias questions absolutely helps to minimize this risk.

Research control

Research control is one way that researchers can address bias. In quantitative studies, randomness is often utilized. This allows certain aspects of the study to be left to chance rather than to researcher or participant choice. Randomization is the process of assigning participants to treatment and control groups, assuming that each participant has an equal chance of being assigned to any group. The most common and basic method of simple randomization is flipping a coin. For example, with two treatment groups (control versus treatment), the side of the coin (i.e., heads – control, tails – treatment) determines the assignment of each subject.

However, subjects in various groups should not differ in any systematic way. If treatment groups are systematically different, research results will be biased. Suppose that subjects are assigned to control and treatment groups in a study examining the efficacy of a surgical intervention. If a greater proportion of older subjects are assigned to the treatment group, then the outcome of the surgical intervention may be influenced by this imbalance. The effects of the treatment would be indistinguishable from the influence of the imbalance of variables, thereby requiring the researcher to control for the variables in the analysis to obtain an unbiased result.

Sometimes, for example collecting a large aggregate dataset, the researcher will choose to randomly select a particular number of data to utilize such as every third data point. Remember our Level of Evidence Hierarchy? The levels are ranked according to study types that tend to feature more control over bias.

Blinding is used in some studies to prevent biases stemming from people’s awareness. This involves concealing information from participants, data collectors, or care providers to enhance the objectivity.