Power point presentation

JUNE 1997. VOL 65, NO 6

R E S E A R C H C O R N E R

An overview of statistical
significance in nursing

Editof’s note: This column in /he Journal
highlights reseorch issues related to peri-
operative nursing practice. The outbors,
AORNs codirectors of perioperative
reseorch, provide practical advice for
reading, conducting, and using reseorch
in perioperative nursing pmctice.

esearchers talk about sturisri-
cul signrficuizce, and clini- R cians talk about clirzical sig-

rz$cutzce. Both terms often are
used in research reports, and it is
important for perioperative nurses
to understand how these terms are
similar and different. Although
these terms sound alike, they are
not interchangeable concepts.

QUANTITATIVE RESEARCH DATA

researchers collect data that are
numerical (eg, physiologic mea-
sures, test scores, ratings). The
people who provide this informa-
tion are the subjects, and the
group of study subjects makes up
the study sample. The sample is a
subset of the population of inter-
est (ie, all possible subjects who
meet the study criteria). After the
researchers collect the data, they
analyze them using a variety of
statistical procedures (eg, t tests,
analysis of variance, multiple
regression). The researchers use
the data analysis results to make
inferences (ie, decisions) about
the population of interest based
on the results obtained from the
study sample.

The basic premise of statisti-
cal testing is that the study sam-
ple is representative (ie, is a theo-
retical distribution) of the popula-

In a quantitative study.

tion of interest. The researchers
may make decision errors if the
sample does not reflect the popu-
lation of interest. Statistical tests
are only as good as the data that
are analyzed. If the data are
flawed, the statistical test results
also may be suspect.

During the planning phase of a
study, the researchers must make
two important decisions. The first
decision is to formulate the null
and research hypotheses. and the
second is to establish the level of
statistical significance (ie, the
alpha [a] level). Both of these
decisions are important for the
process known as hypothesis test-
ing, which is the basis of quanti-
tative statistical analysis.

Hypothesis formulation. In
research that is based on hypothe-
sis testing, nothing is ever
proven. This often is a point of
confusion for novice researchers
or new readers of research. In a
quantitative study, the researchers
formulate a null hypothesis and a
research (ie, alternative) hypothe-
sis. The null hypothesis states
that there is no relationship
between the variables of interest
in the study, whereas the research
hypothesis states that there is a
relationship between these vari-
ables. The research hypothesis is

SUZANNE C. BEYEA, RN, CS, PHD. is
AORN codirec tor ofperioperutiv
reseorrli.

LESLIE H. NICOLL, RN, MBA, PHD, is
AORN codirector of’perinperutii7e
research.

and clinical
research
based on previous research, sci-
entific principles, and the
researchers’ knowledge and
expertise. Based on the statistical
test results, the researchers either
accept or reject the null hypothe-
sis (ie, if the results demonstrate
that there is no difference
between variables, the null
hypothesis is accepted). If the
null hypothesis is rejected, the
researchers surmise that some-
thing else is true (eg, the relation-
ship between variables that is
stated in the research hypothesis
actually exists). Just because the
null hypothesis is rejected, how-
ever, does not mean that the
research hypothesis is proven.

level of significance. The
level of significance establishes
the risk that the researchers are
willing to take when testing the
null hypothesis (ie, the probabili-
ty of rejecting a null hypothesis
that really is true). Researchers
traditionally set the a level at .05,
although some researchers use
more stringent (ie, .Ol ) or more
relaxed (ie, .lo) a levels. When
setting the level of significance,
the researchers are determining
how crucial it is to have accurate
results. When researchers set the
a level at .05, it means that they
are willing to be wrong five times
out of 100 when rejecting the null
hypothesis. If researchers set the
a level at .01, it means that they
are willing to be wrong only one
time out of 100.

It is important to remember
that all statistical testing is based
on the concept of probability.

1128
AORY JOL’RhUAL

JUNE 1997, VOL 65, NO 6

When researchers set an a level at
.05 and the study results indicate
that the null hypothesis should be
rejected, the researchers make the
following conclusion: There is a
95% chance that these results do
in fact reflect what is happening
in the population of interest, and
there is a 5% chance that these
results occurred by chance alone
and the null hypothesis actually is
true.

Types of errors. In hypothesis
testing, researchers can make two
types of errors (Table 1). A type I
error occurs when researchers
reject a null hypothesis that actual-
ly is true. A type II error is just the
opposite (ie, researchers accept a
null hypothesis that actually is
false). Researchers are most con-
cerned with making type I errors
(ie, concluding that a difference
exists when there is none).

STATISTICAL SIGNIFICANCE
During the process of hypoth-

esis testing, researchers perform
the statistical test and calculate a
probability ( P value), and com-
pare that value to the a level. If
the P value is less than the a
level, the researchers reject the
null hypothesis. If the P value is
greater than the a level, the
researchers accept the null
hypothesis. Researchers often
write, “The probability was less
than .05 (P < .05),” to describe a

Table 1

result in which the null hypothe-
sis was rejected.

Researchers also may calcu-
late an actual probability, which
can be reported; however, they
still compare this figure to the a
for decision-making purposes. In
the research article “Effect of sur-
gical hand scrub time on subse-
quent bacterial growth” that
appears in this issue of the Jour-
nal, the researchers calculated
that the actual probability was
.02, which is less than the level of
.05 that they chose; thus, they
rejected the null hypothesis.’

CLINICAL SIGNIFICANCE
When researchers conclude

that statistical significance exists,
this means that there is a very low
probability that the findings
occurred by chance alone. In sig-
nificance testing, the word signif-
cant does not mean that the
results are important or that they
have clinical or practical use. A
study with nonsignificant statisti-
cal results still may be helpful in
understanding the lack of a rela-
tionship between a group of vari-
ables. Statistically nonsignificant
results also may help researchers
discover study design or measure-
ment flaws. Regardless of what
researchers conclude about the
statistical tests used in a study,
readers and consumers of research
always must consider the question

POTENTIAL OUTCOMES OF STATISTICAL DECISION MAKING

“What is the practical or clinical
Significance of this research?”

There are no statistical tests to
help readers and consumers of
research determine the clinical
significance of study results. We
must use our critical thinking
skills and clinical expertise to
study and interpret research
results, and then we need to
decide whether the statistical
results have any relevance to our
clinical practice settings.

Whereas statistical signifi-
cance relates to group differences
or effects, clinical significance is
more important at the individual
patient level. For example,
researchers might design a study
to evaluate a weight-loss inter-
vention. If subjects in the inter-
vention group lost an average of
20 lb (9 kg) and subjects in the
nonintervention group lost an
average of 15 lb (6.8 kg), the 5-lb
(2.27-kg) difference between the
two groups in this hypothetical
study would not be statistically
significant. The intervention,
however, might be clinically sig-
nificant to individual patients if
the subjects in the intervention
group reported an improved
sense of well-being and quality
of life.

In contrast, other researchers
might study oxygen saturation
differences related to patients’
body positions (ie, supine, sit-

ting). Mean
oxygen satura-
tion in the
supine posi-
tion might be

The actual situation in the population is that the null hypothesis is: g6%, com-
true false pared to 99%

in the sitting
position.
When tested
statistically,

The researcher concludes
that the null hypothesis is:

true (accepted) correct decision lype I 1 error

false (rejected) lype I error correct decision this difference

1129
AORN JOURNAL

JUNE 1997. VOL 65. NO 6

might be statistically significant
( P < .05), but the difference
might not be clinically significant
for individual patients.

INTERPRETING RESEARCH
When reading and interpreting

research reports, it is important to
assess the relevance of the find-
ings, regardless of whether they
are statistically significant. Nurs-
es can decide what constitutes
clinically important research
results by considering what dif-
ference the findings make to indi-
vidual patients and also the costs
involved in achieving the out-
comes. If the costs are high and
the benefits are minimal, statisti-
cally significant interventions
may not be cost-effective or clini-
cally significant.

The research article in this
issue of the Jouriial illustrates
some of the challenges in inter-
preting statistical and clinical sig-
nificance. The authors state,

Although the mean hacteri-
al count differed signifi-
cantly (P = .02) between
the two-minute and three-
minute surgical hand scrub
times, it fell below 0.5 log,
which is the threshold for
practical and cliriicol
significance.?

Some practitioners may be
tempted to adopt this study’s
findings in clinical practice. The
authors, however, recommend

N O T E S

that future studies be conducted
to test the clinical importance of
bacterial reductions achieved
with various surgical hand scrub
times, using the 0.5 log reduction
benchmark.’ Changes in surgical
hand scrub times should not be

When reading and
interpreting

research, it is
important to assess
the relevance of the

findings.

made based on one study.
Although this study’s results

are statistically significant, their
clinical significance is a complex
issue. The study was conducted
in one institution with a relative-
ly small sample. The authors
include evidence (ie, power
analysis) that the sample is repre-
sentative of the population of
interest; however, the population
in this case was the perioperative
RNs and surgical technologists
(STs) at the authors’ institution,
not the entire US population of
perioperative RNs and STs. The
results, therefore, cannot be gen-
eralized to another institution
unless the study sample is similar

2. Ibid.

to the population of RNs and STs
in the other institution. The
authors suggest two practice
changes, but they note that these
changes should occur only if
additional studies using other
surgical hand scrub agents sup-
port their finding^.^ These
authors clearly are aware of the
importance of differentiating
between statistical and clinical
significance.

This study presents new find-
ings and raises interesting ques-
tions. Statistical significance
should never be the sole reason
to change clinical practice; clini-
cal significance also must be
considered. Readers of research
always should ask the question
“So what?” For example, if there
is a statistically significant differ-
ence between patients’ oxygen
saturations in the supine and sit-
ting positions, so what? If this
difference is only 3%, is it clini-
cally significant? The same ques-
tion applies to the surgical hand
scrub time study results de-
scribed previously. The mean
bacterial count differed signifi-
cantly between the two scrub-
time groups, but the clinical sig-
nificance of this difference is not
clear. Such issues require further
study to clarify the clinical sig-
nificance of the findings. A
Readers who have questions and ideas
about perioperative nursing research are
encouraged to caN AORN’s codirectors of
perioperative research ot (800) 755-
2676 x 8277

1, S M Wheelock. S Lookinland, “Effect of surgical 3 . Ihid.
4. h i d . hand scrub time on subsequent bacterial growth,” AORN

Journal 65 (June 1997) 1087-1098.

1130
AORN JOURNAL