Methods for Hospital

Epidemiology and Quality

Improvement

Chapter 2

Data Collection and Management.

1
 Chapter 2
Data Collection and Management
Introduction

In this chapter we describe measurement and data collection

for infection management (IM) and quality improvement (QI).
First, we define standards, criteria, and indicators. We then go
on to describe critical incident reporting, audit, process and
outcome surveillance, risk adjustment, occurrence screening,
denominators, patient satisfaction and recovery surveys,
qualitative methods, and comparative and observational studies.

We next compare data collection for QI and for research, and

we conclude with a brief discussion on data recording. Subsequent
chapters are devoted to the statistical analysis of these data.

Standards are definitions of what constitutes good quality

care and standards statements describe professionally agreed
levels of performance (Sale 1996). In most cases the professional
group involved is an accreditation body. Standards are usually
broadly based so as to be applicable across institutions.

Criteria define specific components of a standard that are

measurable. An example of a standard might be “The medical staff
shall provide a continuing program of professional education”.
Associated criteria might include ”There will be weekly medical
and surgical Grand Rounds”.

The term criteria is also used in clinical audit where it

refers to ”Systematically developed statements that can be used
to assess the appropriateness of specific health care decisions,
services, and outcomes” (Baker and Fraser 1995). In this context,
the term standard refers to ”The percentage of events that should
comply with the criterion”.

Indicators.

Indicators are ”Quantitative measures that can be used as

guides to monitor and evaluate the quality of important patient
care and support service activities” (Decker 1991, Bernstein and
Hilborne 1993). An indicator is a measure that denotes the
possible presence of a quality problem if it should deviate from
accepted values. Hofer, Bernstein, Hayward, and DeMonner (1997)
have suggested that four questions should be asked before using
an indicator -

2
1. When cases identified by the indicator are examined, can a
set of definable and preventable processes of care known to lead
to a bad outcome be found?
2. Can a review instrument be created that will allow providers
to identify which process problems are present?
3. Are there substantially more process problems in those cases
identified by the indicator than in those cases not identified by
the indicator, and can the sensitivity and specificity of the
indicator be defined?
4. Is the indicator primarily useful for QI efforts by the
provider, or is it also useful as an external measure of quality
across providers?

As a general rule quality is improved when there are good

outcomes and staff strive continuously to improve them further
(Deming 1982). However, in the hospital setting, most indicators
to date have been used to attempt to identify substandard care.
Indicators can refer to one of Donabedian’s (1988) three hospital
characteristics - structure, process, or outcome (Bernstein and
Hilborne 1993).

Structure indicators include the organisation, equipping, and

staffing of such areas as accident and emergency departments,
intensive care units, and anaesthetic services (Decker 1991).
Structure indicators are not in general suitable for
epidemiological surveillance. However, institutions with
inadequate resources are unlikely to be able to optimize their
systems of patient care.

Nolan (1998) has described the advantages of adopting a

systems approach to patient care and he has referred to the
structure of systems. It is important to distinguish between
structure indicators as defined by Donabedian and the systems
structures referred to by Nolan, as the latter are frequently
similar to processes in the Donabedian classification. Although
we agree wholeheartedly with Nolan’s approach that we have
outlined in Chapter 1, we avoid his terminology to lessen
confusion and use instead the term system design.

Process indicators can be used to study a wide range of processes

of patient care. The best process indicators are those that -

1. Have undergone objective validation,

2. Are known to be reliable,
3. Are described in the Evidence Based Medicine literature
(Sackett, Richardson, Rosenberg, and Haynes 1997) and have been
the subject of correctly performed Systematic Reviews (Chalmers
and Altman 1995), and

3
4. Have been incorporated in Clinical Practice Guidelines (Woolf
1990-1993) or Care Maps (Sale 1996).

Examples of satisfactory process indicators include

perioperative prophylaxis to prevent deep venous thrombosis and
pulmonary embolism (Fauci, Braunwald, Isselbacher, Wilson,
Martin, Kasper, Hauser, and Longo 1998), the correct use of
preoperative prophylactic antibiotics to reduce the incidence of
postoperative surgical site infections (SSI’s) (Classen, Evans,
Pestronik, Horn, Menlove, and Burke 1992), and the early use of
angioplasty (Smith 2004) or thrombolytic therapy following
myocardial infarction (Fauci, Braunwald, Isselbacher, Wilson,
Martin, Kasper, Hauser, and Longo 1998). Other potentially useful
process indicators are inadequate handwashing fostering
nosocomial infection (Handwashing Liasion Group 1999), antibiotic
misuse causing antibiotic resistant infections (Fauci, Braunwald,
Isselbacher, Wilson, Martin, Kasper, Hauser, and Longo 1998),
discharge planning and readmissions (Naylor, Brooten, Jones,
Lavizzo-Mourey, Mezey, and Pauly 1994), bedding quality and
pressure ulcers, syringe design and needlestick injuries,
pharmacy computerisation and medication errors, and maintainence
of normothermia and tissue oxygenation and postoperative
complications (Treasure and Bennett 1999).

Process indicators are especially useful when the outcomes

of substandard care are uncommon, or when the difference in the
proportion of unsatisfactory outcomes in those who receive good
and substandard care is not large relative to random variation
and patient differences, for example death following myocardial
infarction. Because many individual deaths in this group may be
unrelated to the quality of care, very large samples and complex
adjustment for differing patient susceptibility (risk adjustment)
may be required to find differences that could be attributed to
substandard care (Iezzoni 1994); even then it may be easy to
misclassify hospitals providing good care as inadequate because
of random variation (Hofer, Bernstein, Hayward, and DeMonner
1997, Thomas and Hofer 1999).

In this and similar situations, surveillance of processes of

patient care (Baker 1997) would be much more likely to identify
substandard care correctly than outcome surveillance. In
addition, this would be accomplished using samples of manageable
size (Mant and Hicks 1995, Davies and Crombie 1995) with
reasonable speed thus enabling timely intervention.

As described in Chapter 1, a recent advance has been the

development of Care Maps. These standardised processes of patient
care have now been developed for a wide range of conditions. For

4
example, patients undergoing surgery, for example biliary tract
surgery or joint surgery, can be managed using standardised
processes of care that have at least consensual validity.
Consensual validity is obtained when groups of experts can agree
using a formal technique such as the Delphi method (Jones and
Hunter 1995).

The use of Care Maps with surgical patients ensures that

important steps in preoperative preparation and postoperative
care such as the appropriate administration of prophylactic
antibiotics to minimise the risk of postoperative SSI are not
forgotten, and that unnecessary investigations, treatments, and
delays are avoided. Surveillance can be accomplished by
monitoring important variations from the agreed protocol, and
these variations thus become process indicators. These variations
are sometimes referred to as variances but this terminology
should be avoided as variance has a special meaning in
statistics.

Care Maps are also called critical paths, clinical paths,

integrated care pathways, and clinical protocols (Sale 1996,
Pearson, Goulart-Fisher, and Lee 1995). We avoid the use of
terminology like critical paths because these terms have a
specific meaning in Operational Research (Sprent 1991) that is
not strictly applicable to the idea of Care Maps. Care maps are
in fact process maps and not critical paths.

Outcome indicators are most frequently used for surveillance. The

best known in this group are the nosocomial infection indicators
(Quality Indicator Study Group 1995), for example SSI’s,
infections due to antibiotic resistant organisms, and nosocomial
bacteraemia, urosepsis, diarrhea and pneumonia. Indicators such
as SSIs have undergone extensive validation, for example by the
National Nosocomial Infection Surveillance System (NNIS) of the
United States Centers for Disease Control (CDC)(Quality Indicator
Study Group 1995).

Examples of non-infectious indicators include needlestick

injuries, readmissions, pressure ulcers, falls, perioperative
myocardial infarctions, strokes, episodes of renal or respiratory
failure, bleeding disorders and surgical deaths, serious errors
in drug prescribing and adverse drug reactions, excessive lengths
and costs of stay, prolonged waiting times in accident and
emergency departments, and unplanned admissions to special care
units or returns to operating theaters (Nadzam and Nelson 1997).

Outcome Indicator types. There are two types of outcome

indicators, Rate based and Sentinel event indicators.

5
 Sentinel events are usually serious isolated incidents such
as anaphylactic reactions to prescribed drugs that the patient
should have been known to be allergic to, injuries due to
mismatched blood transfusions, unexpected deaths in hospital such
as maternal deaths following uncomplicated childbirth, rare
serious complications such as endophthalmitis or operations on
the wrong limb. These events should not normally be subject to
random variation and frequently lack denominator data. They may
indicate the existence of a quality problem requiring immediate
attention (Journal on Quality Improvement 1998). Since these
events require individual attention and since they should not
normally be subject to random variation, suitable denominator
data will usually be unavailable. However, as de Laval (1997) has
described so graphically, these sentinel events must be studied
carefully.

While human error is an important cause in the majority of

cases, important background factors may exist which need to be
understood. These include complexity, exhaustion, distraction and
inadequate supervision by senior staff. As we have described in
Chapter 1, blaming someone for a serious sentinel event is
unlikely to prevent its recurrence

Rate based indicators have a numerator, such as the number

of SSI’s after vascular operations, and a denominator, such as
the number of patients undergoing vascular surgery. There is
always a random component with rate based indicators. These
indicators frequently require risk adjustment or stratification
for differing patient susceptibilities. Risk adjustment is
described in Appendix 2 of this chapter.

Characteristics of indicators. (Quality Indicator Study Group

1995). Indicators are similar to diagnostic tests, and a good
indicator requires similar attributes to a good diagnostic test.
Hofer, Bernstein, Hayward, and DeMonner (1997) have suggested
that a good indicator of quality should have the following
characteristics -

1. It should be identifiable using currently available data or

data that could be collected easily with a minimum of expense and
personnel time.
2. It should identify cases with a high likelihood of having
received substandard care.
3. It should identify problems that are recurrent and that have a
predictable, manageable list of preventable causes.

Definitions. The first consideration, having selected an

indicator, is its definition. Definitions of indicators should be

6
succinct and unambiguous, and they should not vary over time. For
rate based indicators there must be a clear definition of the
numerator, for example the number of SSI’s, device related
bacteraemias, pressure ulcers, or the number of patients with
ventilator associated pneumonia in the intensive care unit in a
month. In addition, the denominator must be clearly defined, for
example the number of patients undergoing the operation, or the
number of patient-days or device-days of exposure during that
month.

It will usually be preferable to use published definitions

whenever possible. For example, one can use the definitions
described in the CONQUEST database (Palmer and Lawthers 1996,
NQMC [Link]/), or the definitions provided
by the relevant national accrediting agency such as the United
States Joint Commission’s National Library of Healthcare
Indicators (Nadzam and Nelson 1997) or the Australian Council on
Healthcare Standards National Quality Indicators (Australian
Council on Healthcare Standards 1995).

Validity. The best outcome indicators are those that have

undergone scientific validation, for example by undertaking
studies to assess sensitivity, specificity, and especially
positive predictive value (Hofer, Bernstein, Hayward, and
DeMonner 1997). An indicator is sensitive if it gives few false
negative results, that is it infrequently misclassifies
substandard care as satisfactory. The false negative rate is 1-
sensitivity. An indicator is specific if there are few false
positive results, that is there are few cases of satisfactory
care when the indicator is positive. The false positive rate is
1-specificity.

It is necessary to establish the strength of the

relationship between the outcome and the receipt of substandard
care. The positive predictive value of the indicator is the
proportion of positive results for which there is evidence of
substandard care, and it is influenced by the proportion of
episodes of substandard care that exist in the relevant hospital
population. Hofer, Bernstein, Hayward, and DeMonner (1997) give
examples of the use of these methods for the assessment of an
indicator.

An outcome indicator will be of little use if it can be

shown that substandard care can occur with nearly the same
frequency in the group with a normal outcome as in the one with a
positive indicator outcome, that is the positive predictive value
of the indicator is very low. Here, process indicators are likely
to be much more informative.

7
 Many indicators have not yet undergone scientific
validation. In this situation there should at least be consensual
validation, that is there should be agreement among experts using
a standardised technique such as the Delphi method (Jones and
Hunter 1995) that the indicator is a valid reflection of an
underlying process problem. However, scientific validation is
clearly superior to consensual validation and indicators that
have been selected using the latter method should be viewed with
reserve until scientific validation has been performed.

Unfortunately the process of scientific validation can be

complex and costly, and probably beyond the resources of
individual institutions. For this reason, workers selecting
indicators should, whenever possible, choose published
scientifically validated indicators for use in their hospitals.

In Chapter 1 we repeatedly emphasised the need first to

analyse and optimize systems so that they are functioning in a
predictable, consistent manner. When this process has been
undertaken carefully, it will often be obvious what needs to be
monitored.

In addition, when rate based indicators are to be studied,

the correct denominators must be used and risk adjustment is
frequently required. Denominators and risk-adjustment are
described in Appendix 1 and Appendix 2 of this chapter.

Reliability. A good indicator will be stable when repeated

assessments are made at different times, by different raters
(surveillance staff), and at different institutions. Rater
reliability is described in Chapter 3 Assessing Agreement.

Importance. The indicator should measure outcomes that are of

practical importance and a substantial proportion of variation
should be due to differences in the quality of care. It is
necessary that underlying quality problems be amenable to
improvement by management correcting substandard systems and
processes.

Recordability. Data required for analysis should be readily

available in the patients’ medical records, or they should
otherwise be accessible using available staff, databases, and
financial resources.

Selection of indicators for surveillance. Massanari, Wilkerson,

and Swartzendruber (1993) suggest that indicators could be
selected from the following areas -

8
1. Nosocomial infections.
2. Events related to administration of medications, for example
adverse drug reactions and injuries due to administration errors.
3. Accidents such as falls.
4. Conditions not present on admission, for example myocardial
infarctions and other organ injuries such as strokes, bleeding
conditions, pressure ulcers and pulmonary emboli and deep venous
thromboses.
5. Events related to treatment such as anastomotic breakdowns
following colonic surgery, unplanned readmissions, unplanned
admissions to special care units, and unplanned returns to
operating theaters.
6. Events related to equipment such as breakdowns and user
errors.
7. Excessive waiting times, lengths of stay, and costs of stay.
8. In addition, episodes of inappropriate or ineffective care are
likely to become important clinical indicators. Examples of
inappropriate care are the misuse of antibiotics that may
predispose to the emergence of antibiotic resistant organisms and
unnecessary patient admission or delayed discharge.

It is clear that the development of indicators is a complex

process that is unlikely to be undertaken by a single hospital in
isolation. Most hospital QI and IM staff will select indicators
that are appropriate for their institution from those that are
already available, for example in the CONQUEST database (Palmer
and Lawthers 1996, NQMC [Link]/).

Methods of Data Collection.

These include critical incident reporting, occurrence

screening, audit, epidemiological surveillance, outbreak
investigations, and patient satisfaction and recovery surveys.

1. Critical incident reporting. A critical incident is defined as

any occurrence that could affect the safety of a patient
(Buckley, Short, Rowbottom, and Oh 1997). Incident reporting is
essentially qualitative, as there are no denominators available
to calculate rates. The method is thus most suited to detecting
sentinel events or their precursor near misses (Leape 1994,
Journal on Quality Improvement 1998).

Many hospitals have, in the past, relied on senior nursing

staff on duty to identify critical incidents and report them to
the hospital administration. However, conventional hospital
incident reporting systems have rarely been highly successful
(Wolff 1994). Although serious patient injuries are usually

9
identified, many errors go unreported because staff are reluctant
to identify colleagues who have made mistakes, and they may be
keen to avoid the extra work a report will generate.

Leape (1994) has pointed out that accident prevention has

not been a primary focus of the practice of hospital medicine.
The practices of mortality and morbidity conferences, incident
reports, risk management activities, and quality assurance
committees usually focus on the individual’s role in incidents.
The immediate cause of the individual error is found, but the
root causes are rarely sought.

This approach assumes that individuals can be relied upon

not to make errors. Systems analysis methods, described in
Chapter 1, recognise that the capacity for error is universal and
that it is important to provide an environment in which the
potential for error is minimised (Leape 1994, Vincent 1997, Wolff
and Bourke 2000). In addition, critical processes need error-
proofing so that, if an error occurs, it can be prevented from
causing serious harm.

Following the practice in the airline industry which has for

long recognised that serious accidents and incidents are
frequently preceded by errors and “near misses” (Green 1990, de
Laval, François, Bull, Brawn, and Spiegelhalter 1994, de Laval
1997), staff in intensive care units and other hospital
departments are now using reporting and analysis of errors and
incidents, and subsequently instituting appropriate system and
process changes to prevent them or prevent them causing harm.

Since staff are reluctant to report people who may be

associated with problems, the hospital administration must foster
an environment in which problems are regarded as "treasures" from
which to learn how to improve rather than opportunities to punish
(James 1997). When a problem is brought to the attention of the
hospital authorities, the systems and processes that accompany it
require examination.

An open report format can be used and the reporting process

can be anonymous and it can be voluntary (Buckley, Short,
Rowbottom, and Oh 1997). Information is sought about location,
time, numbers and type of personnel present, and the state of the
ward or unit. In addition, information about possible causes,
consequences, and preventive measures is requested. The object is
to determine what has occurred, to examine why and how it
occurred, to institute measures (system and process improvements)
aimed at preventing it happening again, and when necessary, to
monitor these changes to ensure that they are effective.

10
 There is recognition that errors have complex causes (de
Laval, François, Bull, Brawn, and Spiegelhalter 1994, de Laval
1997, Andrews, Stocking, Krizek, Gottlieb, Krizek, Vargish, and
Siegler 1997) and that reducing them requires careful analysis of
underlying psychological, social, and system factors (Leape 1994,
Blumenthal 1994, James 1997). Apportioning blame does not
confront these complex causes and is unlikely to produce
improvement or to guard against recurrence of the problem.

Bates, Makary, Teich, Pedraza, Ma’luf, Burstin, and Brennan

(1998) have shown that hospital resident staff can provide useful
information about the occurrence of adverse events by using a
suitable computer program when writing patients’ discharge
reports. This method should have substantial potential, as all
patients require discharge documentation by resident medical
staff. Provided the information is used to learn rather than
blame, and junior medical staff understand that the information
will be employed to improve the processes of patient care and to
make them safer, there should not be difficulty in involving them
in this method of incident reporting.

2. Occurrence Screening. This is a further method for identifying

instances of poor quality care which merit subsequent review. It
begins with a list of negative indices reflecting adverse patient
events or outcomes. A number of such lists of negative criteria
or “occurrence screens” are now available for a range of medical
conditions and care settings (Wolff 1995, 1996, Wolff and Bourke
2000).

Occurrence screening involves the structured examination of

patient records either on a day by day basis or, more commonly,
following the patient’s discharge, the objective being to
discover evidence of adverse outcomes. The definition of an
adverse outcome (Wilson, Runciman, Gibberd, Harrison, Newby, and
Hamilton 1995) is “An unintended injury or complication which
results in disability, death, or prolongation of hospital length
of stay (LOS) and is caused by patient management rather than
patient disease”.

The last part of this definition of an adverse outcome

presents a problem in practice as expert reviewers using a
structured implicit review technique are often unable to agree
about the role of substandard care in the causation of particular
adverse outcomes (Localio, Weaver, Landis, Lawthers, Brennan,
Hebert, and Sharp 1996).

11
 Medical record administrators or specially trained nurses
can perform screening for adverse outcomes when coding patient's
discharge notes using a structured explicit review process
(Localio, Weaver, Landis, Lawthers, Brennan, Hebert, and Sharp
1996). Sensitivity can be high but specificity is unlikely to be
adequate because screening by medical record administrators or
nurses is likely to be less successful at determining which
events are due to substandard care.

A more labour intensive and expensive method is to employ

double sampling (Reinke 1991). This has a long history of use in
quality control in manufacturing industry. First, screening is
performed using explicit screening criteria as described above;
this identifies patient records where there are obvious adverse
events associated with patient care and records where there are
clearly no adverse events. In addition, a group in between is
identified in which adverse events may exist.

This screening is thus very sensitive but not very specific;

a substantial number of false positive results may exist in the
intermediate group. This group is then examined by experts using
a structured implicit screening process that enables them to
eliminate the false positive results, thereby achieving high
specificity. However, as already described, experts often fail to
agree about the association of an adverse event and the
occurrence of substandard care. Sanazaro and Mills (1991) have
pointed out that occurrence screening is limited by the
relatively low specificity of the method and the need for two-
stage review that can be inefficient and costly.

Study of the relationship of the method to the frequency and

nature of underlying substandard processes of care in the patient
population, including those whose screens are negative or are not
performed, using the validation methods described by Hofer,
Bernstein, Hayward, and DeMonner (1997) appears to be needed so
that its positive predictive value can be determined.

Occurrence screening is dependent on the quality of the

medical record and improvement in the quality of medical records
is likely to increase its utility. James (1997) has shown that a
computerised medical record linked to computer programs designed
to search for adverse outcomes and their precursor errors and
“near misses” greatly increases the likelihood of their
detection.

The occurrence screening method was pioneered by Craddick

(1979) and the California Medical Association and later developed
for use in the Harvard Medical Practice Study (Brennan, Localio,

12
Leape, Laird, Peterson, Hiatt, and Barnes 1990) and the Quality
in Australian Health Care Study (Wilson, Runciman, Gibberd,
Harrison, Newby, and Hamilton 1995). The early studies were
motivated by the need to understand and reduce the problem of
medical litigation and the Australian study was performed to
determine the frequency and nature of adverse events in
Australian hospitals.

The Harvard and Australian Studies were performed as

research projects. However, many hospitals use limited occurrence
screening as an ongoing quality assurance and improvement tool
(Wolff 1995, Wolff and Bourke 2000, James 1997). Wolff (1996) has
recommended screening of patient records after discharge by
medical records administrators. The following eight indicators or
criteria are used -

1. Death.
2. Return to operating theatre within 7 days.
3. Transfer to special care from a general ward.
4. Unplanned readmission within 28 days of discharge.
5. Cardiac arrest.
6. Transfer to another acute care facility.
7. Length of stay greater than 21 days.
8. Cancellation of theater booking.

Patient records that screen positive for one or more of

these criteria are then reviewed by a senior member of the
hospital medical staff who determines whether a genuine adverse
outcome associated with patient management has occurred. If this
is the case, the matter is referred to a committee that
recommends appropriate action to correct the underlying quality
problem. Since the monitoring process is continuous, there is
opportunity to determine whether the remedial action for previous
occurrences has been effective.

Wolff (1995) has shown that 49% of adverse occurrences are

detected by this method. For more serious adverse occurrences the
detection rate is 64%. During a three-year period it was possible
to reduce the number of detected adverse occurrences by over 50%
using this method. He has shown the method to be cost effective.

Adverse events and clinical indicators have a great deal in

common and many adverse events are also clinical indicators.
However, the distinction between them is important. Some clinical
indicators represent good practice, for example the short time
taken to assess incoming accident victims in an efficient
accident and emergency department. Furthermore, there are process
as well as outcome clinical indicators.

13
 Although most adverse events are likely to be indicators of
sub-optimal quality, this is not necessarily the case. For
example, many unplanned re-admissions are due to patient
characteristics that have little to do with the quality of
discharge planning (Miles and Lowe 1999), and falls in a
rehabilitation unit are often inevitable. However, changes in the
rates of these events when data are analysed sequentially in
control charts may indicate a quality problem.

Surveillance of nosocomial infection indicator outcomes is

now well developed but surveillance of non-infectious adverse
outcomes is less advanced. It would seem likely that surveillance
using occurrence screening by medical records administrators, in
addition to identifying sentinel events, could also be employed
to determine baseline rates of many occurrences that are not
sentinel events, for example patient falls and unplanned
readmissions. Staff could then plot monthly numbers of these
events on control charts as described in Chapter 4. However, it
is first necessary to undertake analysis and optimization of
hospital systems so that they are functioning in a stable,
reproducible manner. Often, when surveillance is commenced
without this essential first step, there is a marked increase in
the number of adverse events due to more careful searching. This
does not indicate worsening of patient care.

Because control charts employ sequential analysis and they

adjust for random variation, they can be useful even when high
rates due to substandard care are difficult to detect because of
concomitant random variation, as is the case with unplanned
readmissions. Excessive mean rates could be revealed by using
control charts and would indicate the likely presence of a common
cause problem, for example, the absence of an effective discharge
planning process. Rates exceeding control limits would suggest a
special cause problem such as a medical unit that discharged
unstable patients (Brook, Kahn, and Kosecoff 1992).

Occurrence screening is likely to be very susceptible to

gaming if used in a judgmental way. However, if it is employed as
a genuine learning tool and the data are examined sequentially in
control charts when this is appropriate, it would seem that the
method has a great deal of potential.

3. Audit. The term audit has relatively wide meaning but it is

used here to refer to the collection and analysis of data
pertaining to the processes and outcomes of patient care and the
economical use of hospital resources. However, audit must go
further than data collection and analysis; audit results must be

14
employed in a feedback loop to improve the quality of patient
care and the hospital’s efficiency. As described in Chapter 1, an
efficient feedback loop such as the Deming cycle is essential for
its success.

Audit was defined in the 1989 British National Health

Service Working Paper as “the systematic and critical analysis of
the quality of care, including procedures for diagnosis and
treatment, the use of resources and the resulting outcome and
quality of care for the patient” (Williams 1996). Hospitals in
the United Kingdom are required to implement a systematic audit
program.

There are two main types of audit (Wilson 1992). The first
is investigative audit. Here a process of patient care or
resource utilisation is analysed critically to find ways in which
it may be improved. The second is criteria audit which involves
comparing outcomes or elements of a process with previously
agreed criteria of satisfactory performance and usually
incorporating a standard which describes the proportion of cases
which should comply with the criteria (Baker and Fraser 1995).
Where performance falls short of satisfactory practice, system
and process changes may be made for improvement and the audit
repeated at a later date to ensure that the changes have produced
the desired result.

An example of investigative audit would be a study of the

contents of ward storage cupboards where items such as
intravenous sets and urinary catheters are stored. Some ward
staff may hoard these items so that excess expensive inventory is
kept. In some cases, when an item becomes obsolete, considerable
financial loss can occur. In other cases a ward may run short of
an important item at a critical time and valuable staff time is
wasted borrowing from other wards. The system of restocking may
be inefficient and waste the time of hospital store staff and
clerical staff. Examination of all aspects of the maintenance of
ward storage cupboards may reveal ways in which efficiency can be
increased and resource utilisation improved.

There is an account of an investigative audit of blood

product usage in McDermott and Scott (1988). Tuckfield, Haeusler,
Grigg, and Metz (1997) have described an audit to investigate and
reduce the inappropriate use of blood products; Zafar, Butler,
Podgorny, Mennonna, Gaydos, and Sandiford (1997) needlestick
injuries; and Walker, Williams, and Tawn (1994) the use of
preoperative chest radiography.

An example of a criteria audit would be the analysis of a

15
series of cases of postoperative deep venous thrombosis and
pulmonary embolism. Criteria for prevention, diagnosis and
treatment would be selected from current best practice using
evidence based medicine guidelines whenever available (Sackett,
Straus, Richardson, Rosenberg, and Haynes 2000). Records of
patients with the diagnosis of postoperative deep venous
thrombosis or pulmonary embolism would then be examined to
compare actual practice with accepted practice. Finally, if
serious discrepancies were found as indicated by deviations from
accepted standards, changes in the underlying processes of
thrombembolism prophylaxis and treatment would be required and
the audit would need to be repeated at a later date to see that
the changes had been effective.

Alternatively if current practice were found to be

satisfactory a new audit topic would be sought, although further
examination of thromboembolism prophylaxis and treatment might be
required at a later date if there were evidence of increasing
numbers of thromboembolic complications, or suspicion that
thromboembolism prophylaxis was being under-employed.

McDermott and Scott (1988) have described a criteria audit

of cholecystectomy practice and Dawson and Capaldi (1993) an
audit of the treatment of croup.

A difficulty with a subject like pulmonary embolus is that

many hospitals would have relatively few cases in a suitable time
frame for audit. In addition, most patients who fail to receive
appropriate preventive measures such as subcutaneous heparin and
anti-embolism stockings will not get the complication. For these
reasons the audit described above may fail to provide sufficient
data to be useful in all but the largest hospitals. In this
situation, it would be preferable to audit compliance with
accepted methods of deep venous prophylaxis in a series of
patients undergoing surgical operations for which thromboembolic
disease can be a relatively common complication, for example
major pelvic and lower limb procedures (Williams and Macdonald
1997).

The records to be audited could be for a consecutive sample

of patients undergoing a particular operation such as hip or knee
surgery, or a random sample of patients undergoing a common type
of operation such as lower abdominal procedures. The sample
should be of sufficient size for there to be reasonably narrow
confidence limits for rate-based criteria, such as the proportion
of patients receiving appropriate anti-embolism prophylaxis. A
sample of 100 records is frequently used.

16
 Audit must proceed as a formal planned activity (Williams
1996). It should be managed by the group concerned, for example
surgeons who perform abdominal procedures. It should be possible
for those involved to set time aside for the activity that is
separate from time for other commitments.

There should be a team to plan the audit and it should

preferably contain a person with training as a facilitator. This
is important not only for planning the study but also for
devising and implementing process changes if the audit result
shows this to be necessary. In addition a person from the
administrative staff should be involved, as change will not be
possible without the support of senior members of the
administration. Finally, clerical assistance is required.

When selecting a topic for audit the team must ensure that
it is a known or suspected problem area and that efforts to
produce improvement have the potential to be productive. For
example, there would be little point in auditing unplanned
readmissions if it were known that the great majority of them
were due to patient characteristics rather than substandard care.
In addition, the audit subject should be important, for example a
common disease or operation or one that is costly or has serious
complications.

When the topic has been selected the criteria must be

decided. These should be important to the management of the
patient's illness and should be valid indicators of the quality
of care, preferably incorporated in clinical guidelines and care
maps that have resulted from systematic reviews and evidence
based medicine (Dickinson 1998). They should be agreed to by the
members of the group. For example, for abdominal surgery the
group should come from the surgeons and other staff who perform
the operations and care for the patients in the wards and
theater.

The information needed to compare the criteria must be

available, for example in the patient's ward file and it must be
capable of extraction using available resources. The extraction
process must have acceptable sensitivity and specificity, and
predictive value (validity) (Hofer, Bernstein, Hayward, and
DeMonner 1997) and multiple extractors must be able to agree as
to whether a standard is or is not met (reliability). There must
be clear definitions of variations from optimum practice and
agreed upon standards.

Analysis of the data must be systematic and substandard

practice clearly identified in a report. Clear written

17
recommendations must be made for improvement. Practice changes
must be clearly documented and written plans for re-audit made
and implemented.

Audit is essentially an intermittent and recurrent activity.

The planning and implementation of an audit require a great deal
of thought and this can be one of its most valuable attributes
(Crombie and Davies 1992).

Berger (1998) has noted that the audit process advocated by

the British National Health Service has often produced
disappointing results. Among the reasons identified are
unreasonable expectations of audit and that audit often tends to
be managerially commissioned so as to be seen to be doing
something about quality. Management often adopts a narrow focus
on utilisation issues such as bed occupancy that may be in
conflict with clinicians desires to audit the management of
selected conditions such as myocardial infarction. Adopting a
narrow management focus on utilisation issues has been identified
as a limitation of utilisation review audits (Wolff 1994).

4. Surveillance. In contrast to audit, surveillance is usually an

ongoing activity, frequently performed by staff with
epidemiological training. Most surveillance is outcomes based
with audit frequently being performed to assess processes,
although process surveillance has also been described (Baker
1997). In a sense the two methods are often complementary with
surveillance used for monitoring outcomes and audit being
performed when a problem is observed or suspected.

Historically, surveillance grew out of the need to

understand and control hospital acquired (nosocomial) infections
(Pottinger, Herwaldt, and Perl 1997), but it is now being applied
in a limited way to non-infectious adverse outcomes of care
(Massanari, Wilkerson, and Swartzendruber 1995). In the section
on occurrence screening, we referred to its potential for
surveillance of rate based non-infectious adverse outcomes that
have potential value as clinical indicators.

The National Nosocomial Infections Study (NNIS) of the

United States Centers for Disease Control (CDC) began in 1970
when selected hospitals were invited to report their nosocomial
infection data for inclusion in a national database. This has now
become the National Nosocomial Infections Surveillance System.

Surveillance system. Each hospital should develop a surveillance

program that suits the needs of its patient population and works
within available resources (Pottinger, Herwaldt, and Perl 1997,

18
Massanari, Wilkerson, and Swartzendruber 1995). It is desirable
to concentrate on things that are practically important, that are
associated with substandard care, and that can be improved by
identifying and correcting underlying system and process
problems. The following are some examples -

A. SSI’s. A good method is to select operations within

specialties that are performed frequently and that are relatively
homogeneous, for example -

a. Vascular surgery - femoro-popliteal procedures,

intra-abdominal procedures.
b. Orthopaedic surgery - major lower limb joint procedures, back
procedures.
c. Abdominal surgery - colonic surgery, biliary tract surgery.
D. Urology - nephrectomy and other retroperitoneal procedures.
e. Gynaecology and obstetrics - abdominal hysterectomy and other
intraperitoneal procedures, caesarian section.
f. Breast surgery - mastectomy.
g. Cardiothoracic surgery - pulmonary resections, coronary artery
surgery.
h. Neurology - craniotomy, spinal procedures.

B. Organisms of special importance such as antibiotic resistant

bacteria, for example MRSA, ESBL-Klebsiella pneumoniae,
Vancomycin Resistant Enterococcus (VRE), Acinetobacter species,
and Clostridium difficile.

C. Nosocomial bacteraemias, pneumonias, and urinary tract

infections. Bacteraemia is often associated with central venous
lines so surveillance in departments which use these devices
frequently such as intensive care units, haematology-oncology
departments, IM departments, renal unit and intravenous nutrition
services is a good policy. Similarly, pneumonias are frequently
associated with mechanical ventilation and urinary tract
infections with indwelling urinary catheters, so that it is good
practice to perform surveillance within areas of the hospital
with high usage of these devices. However, the diagnosis of
ventilator-related pneumonia can sometimes be difficult.

D. Carefully selected non-infectious adverse outcomes that are

valid indicators of underlying substandard care processes.
Examples include needlestick injuries (D’Arco and Hargreaves
1995), pressure ulcers, unplanned returns to operating theater or
admissions to the intensive care unit, serious drug reactions and
toxicities, nosocomial renal and respiratory failure, stroke, and
myocardial infarction (Nadzam and Nelson 1997, Palmer and
Lawthers 1996), and important patterns of excessive waiting and

19
excessive length and cost of stay (Smith, Pryce, Carlisle, Jones,
Scarpello, and Pantin 1997).

Definitions. To allow for uniformity so that data may be compared

between institutions and also in a single institution over time,
it is important that the definitions of the surveillance
indicators be carefully standardised. For infectious disease
indicators the NNIS definitions (APIC Infection Control and
Applied Epidemiology 1996) are most useful and for non-infectious
indicators the CONQUEST database (Palmer and Lawthers 1966, NQMC
[Link]/) should provide useful guidance.

Surveillance methods. Pottinger, Herwaldt, and Perl (1997) have

described hospitalwide traditional surveillance, periodic
surveillance, prevalence surveys, selective surveillance, and
outbreak thresholds. Modern selective surveillance is the
preferred method for most acute care hospitals. (The terms
“targeted” and “sentinel” surveillance are often used for
selective surveillance. We use “selective” to avoid confusion as
“targeted” and “sentinel” can have other meanings, for example
sentinel event indicators described above).

Selective surveillance involves concentrating on selected

services such as intensive care, specific devices such as
mechanical ventilators and central venous lines, specific
infectious agents such as MRSA, and specific procedures such as
selected surgical procedures for the development of SSI’s. By
limiting the scope of surveillance in this way, data can be
collected about a wide range of indicators and susceptible
patient groups.

In addition, surveillance staff need to know when unusual

numbers or types of infections or other indicator outcomes might
be occurring in areas not currently undergoing surveillance.
Careful liaison with the microbiology laboratory and other
pathology services, regular ward rounds by the surveillance
staff, and regular monitoring of drug use by the pharmacy staff
should enable this to be accomplished.

When a problem occurs or is suspected, the surveillance

program will need modifying so that the possible trouble spot can
be included in the program. This is often referred to as
“targeting”. As with all QI work, the goal is to prevent problems
and, if they occur, to determine their causes promptly so that
systems and processes can be modified to control them.

Data sources. The surveillance staff must decide on appropriate

data sources. These will not necessarily be the same in all

20
institutions or for all selected indicators. Regular ward rounds
by the IM and QI staff are most important as a learning
environment fostering co-operation and consultation must be
generated. In addition, many nosocomial infections and other
indicator outcomes will be brought to the attention of the
surveillance staff during these rounds.

Patient records and pharmacy notes are valuable sources of

information. For example, temperature charts and antibiotic
orders may indicate the presence of a nosocomial infection, and
prescription orders for adrenain, antihistamines, and
hydrocortisone a major drug reaction. Microbiology reports are
most useful for bacteraemias, urinary infections, and infections
and colonisations by MRSA and other multiple antibiotic resistant
organisms (MRO’s). X-ray results are helpful for diagnosing and
following nosocomial pneumonias, although the definition of
nosocomial pneumonia and its diagnosis can present problems
because pulmonary infiltrates can have other causes (Craven and
Steger 1997).

Biochemistry reports are required to monitor the occurrence

of nosocomial renal failure (Page and Washburn 1977), for example
in high risk populations such as elderly patients undergoing
major surgery. In addition, the serum albumin may furnish
valuable information about the patient’s protein nutrition.
Haematology results provide information about anaemia, bleeding
disorders, infections, resistance to infection and blood
transfusions. Reports of ECG tracings and enzyme levels identify
cardiac episodes. Databases maintained by the hospital or other
health care organisations are essential sources of information.

Numerator data for indicator occurrences are obtained by

using the above methods. In addition, the surveillance staff must
also obtain appropriate denominator data so that rates for rate-
based indicators can be determined and, where necessary risk-
adjustment data. Denominators and risk-adjustment are discussed
in Appendix 1 and Appendix 2.

Staff training and program validity and reliability. It is

important that surveillance staff undergo training to ensure that
their data collection is accurate and reliable. Senior staff
should monitor the surveillance program to ensure the accuracy
and uniformity of the data collected. In addition, at appropriate
intervals such as yearly, the data collection system should be
checked by having IM specialists and QI supervisors undertake a
formal study of its validity and reliability (Cardo, Falk and
Mayhall 1993).

21
Data management, analysis, and communication of results. Data
must be collected, entered into an appropriate database,
tabulated, and analysed (Pottinger, Herwaldt, and Perl 1997).
Some important aspects of data collection are discussed later in
this chapter, and their analysis is the subject of Chapters 3, 4
and 5. It is very important that the results of the surveillance
program are communicated to relevant clinicians and hospital
management via an efficient feedback loop such as the Deming
Cycle. Specific problems should be reported as soon as possible
after their occurrence so that appropriate system and process
changes can be implemented. In addition, there should be a
regular report of the activities of the surveillance program, for
example every three months.

5. Outbreak investigation. A nosocomial infection epidemic is

defined as an increase in incidence over expected rates
(Beck-Sague, Jarvis, and Martone 1997, Checko 1996, Zaza and
Jarvis 1996, Wendt and Herwaldt 1997). If a surveillance program
is in progress, the increase may be readily apparent; otherwise
alert hospital staff may detect its presence. For infections
included in a surveillance program, statistical methods to be
described in later chapters may aid in its prompt identification.
A single case of an unusual organism or an unusual infection by a
common organism may constitute an epidemic. Systematic
investigation of the outbreak is required and one of the first
things is to look to previously identified causes. For example,
blood stream infections caused by Candida species are likely to
be due to contaminated total parenteral nutrition solutions,
blood stream infections in intensive care units to contamination
of intravascular devices, and gram negative pneumonia may be due
to contaminated respiratory therapy equipment. A case definition
is required. This states:

1. Who had the findings,

2. Describes the time period during which they occurred, and
3. Specifies the location where they occurred.

At the same time there should be a careful search for all

cases. This should be followed by the drawing of an epidemic
curve (Morton, Hebel, and McCarter 1990) and, from the patients’
hospital documents a line listing is created of the
characteristics of the affected patients that may be important.
Interviews with relevant staff and patients follow and a careful
review of microbiology isolates and other possible sources of
information is required.

At this stage a working hypothesis is formed and preventive

and control measures instituted. If required, the outbreak should

22
be reported to the relevant health authorities. A case-control
study, as described below and in Chapter 5, may be needed to
throw further light on the causes of the outbreak (Beck-Sague,
Jarvis, and Martone 1997).

6. Patient satisfaction and recovery. Donabedian (1980) has

pointed out that people are seldom asked to say what they think
the quality of medical care means. The respondent is often given
a list of attributes and asked to rank them or to select some of
them. The reviewer’s view of the boundaries and content of the
concept of quality tend then to be imposed upon the respondent.
Patients generally expect clinical quality to be inherent in the
provision of care, and feel more qualified to comment on the
attention, communication and concern shown towards them while
receiving the clinical care.

Cunningham (1991) has identified six categories that should

be considered when assessing patient satisfaction with hospital
care. These are -

1. Accessibility and responsiveness.

2. Delays.
3. Realistic expectations.
4. Communication.
5. Professionalism.
6. Continuity of care.

It is possible to use two broad approaches to the assessment

of patient satisfaction. Either qualitative or quantitative
methods may be used.

6(a). Qualitative methods. These are being used increasingly in

health care research and QI (Williams 1996, Shmerling, Schattner,
and Piterman 1993, Pope and Mays 1995). The most common and best-
known qualitative method is the Focus Group (Kitzinger 1995,
Schattner, Shmerling, and Murphy 1993, Dawson, Manderson, and
Tallo 1993). The best results are often achieved when the
patients in the group represent a homogeneous sample, for example
former emergency patients in one group, former total hip
replacement patients in another.

It is vital that a trained facilitator be used to lead

discussion about the hospital experiences of the patients. We
have referred to the role of a facilitator in the section on
teams and teamwork in Chapter 1. There may also be an opportunity
to follow up some of the patients’ observations outside the peer-
group pressure situation.

23
 The disadvantage of the focus group technique is that
results may be difficult to extrapolate to the wider patient
population. However, when collecting data for QI in an individual
institution this may not be a serious disadvantage.

Individual interviews may also be used for patient

satisfaction research. This approach is expensive and time
consuming so it is seldom a preferred option. However, it does
offer an opportunity for following up a specific comment or
complaint.

6(b). Quantitative methods. These include patient satisfaction

surveys and health survey questionnaires.

Patient satisfaction surveys are used frequently to comply

with accreditation requirements and it is important that they are
performed in an epidemiologically sound manner (Nelson, Larson,
Davies, Gustafson, Ferreira, and Ware 1991, Westbrook 1993,
Fitzpatrick 1991). The survey form should be one that has been
formally validated. Random sampling should occur and the
recommended number of patients should be included so that the
sample size is adequate to obtain a usable result.

A major problem is non-response. Patients who are less than

satisfied with their care are often too polite to say so and
prefer to avoid filling in the form. Patients and staff must
understand that the exercise is a genuine learning one, that true
responses are needed, and every effort should be made to ensure
that response rates are as near 100% as possible.

The data should be analysed using appropriate statistical

methods and the result formally reported. When problems are
identified their causes should be determined and corrective
action taken by improving relevant processes.

Consideration should be given to assessment of the success

of patients' recovery and health survey questionnaires are being
used increasingly to assess the outcomes for patients following
hospitalisation (Ware 1993, Krishnan and Chipchase 1997). Quite
elderly people are now being discharged from hospital soon after
major surgery and patients in their seventies and eighties are
frequently discharged from public hospitals as soon as three to
four days after operations such as total hysterectomy. While this
practice is not necessarily bad, it is important to ensure that
the patient's recovery is not jeapordised by early discharge, and
that financial savings within the hospital system are not being
made at the expense of outpatient services, which often have
limited resources.

24
 There are available well-validated survey questionnaires
such as the SF36, SF12 and MYMOP (Paterson 1996, Ware, Kosinski,
and Keller 1996, Garratt, Ruta, Abdalla, Buckingham, and Russell
1993) that can be used to assess this important aspect of the
quality of care. If significant preventable poor health is
identified by such a study, measures should be taken to improve
outpatient support for these patients or, as is happening with
some postoperative recovery programs, patients may be discharged
to convalescent care at places that are much less expensive to
maintain than acute care hospitals.

7. Other methods. Research methods are beyond the scope of these

notes (Altman 1991, Woodward 2005, Rothman and Greenland 1998,
Kerr, Taylor and Heard 1998). However, case-control and before
and after studies are performed by surveillance staff. We
recommend strongly that, whenever a research project is to be
preformed by IM or QI staff, a trained statistician should be
involved in the study from the beginning. In addition, ethical
approval must be obtained. It is important not to embark on a
research project when QI activities involving changes in systems
and processes are in progress as these changes will confound the
results of the research.

Examples of research studies include identification of risk

factors for nosocomial infections (Fernández-Crehuet, Díaz-
Molina, de Irala, Martínez-Concha, Salcedo-Leal, and Masa-Calles
1997, Joshi, Localio, and Hamory 1992, Salemi, Morgan, Padilla,
and Morrissey 1995, Britt, Schleupner, and Matsumiya 1978,
Delgardo-Rodríguez, Sillero-Arenas, Medina-Cuardos, and Martínez-
Gallego 1997, Mayotte, Pisano, Ram, Nakasato, and Rotella 1995,
and Moro, Viganò,and Lepri 1994); handwashing methods (Thompson,
Dwyer, Ussery, Denman, Vacek, and Schwartz 1997 and Ansari,
Springthorpe, Sattar, Tostowaryk, and Wells 1991); clinical
measurements in asthma (Gibson, Wlodarczyk, Hensley, Murree-
Allen, Olson, and Saltos 1995); and risk factors for
postoperative complications (Hall and Hall 1996) and other risk-
adjustment methods.

Further examples include comparing processes for delivering

postoperative chest physical therapy to reduce the incidence of
postoperative pneumonia (Hall, Tarala, Tapper, and Hall 1996),
comparing different methods of handwashing to reduce nosocomial
infections (Doebbeling, Stanley, Sheetz, Pfaller, Houston, Annis,
Li, and Wenzel 1992) or comparing processes for obtaining,
analysing and reporting emergency blood tests (Bailey, Topham,
Wantz, Grant, Cox, Jones, Zerbe, and Spears 1997, Kendall,
Reeves, and Clancy 1998. Additional examples are the

25
investigation of epidemic outbreaks and for the discovery of the
causes of and risk factors in the development of indicator
adverse outcomes such as nosocomial pneumonia (Joshi, Localio,
and Hamory 1992), nosocomial diarrhoea (Watanakunakorn,
Watanakunakorn, and Hazy 1996), and catheter-related infections
(Ena, Cercenado, Martinez, and Bouza 1992).

Some examples of randomised controlled trials include

studies of methods to improve antibiotic usage (Ehrenkranz,
Nerenberg, Shultz, and Slater 1992), methods to reduce
needlestick injuries (L’Ecuyer, Schwab, Iademarco, Barr, Aton,
and Fraser 1996), methods to reduce smoking in hospitalised
patients (Miller, Smith, DeBusk, Sobel, and Taylor 1997), methods
to improve discharge planning (Naylor, Brooten, Jones, Lavizzo-
Mourey, Mezey, and Pauly 1994) and nebuliser therapy methods
(Pierce, McDonald, Landau, LeSouef, Armstrong, Mitchell, Francis,
Martin, Musk, Antic, Clark, and Ryder (1992).

Case-control studies. In some instances when there is an epidemic

outbreak of a nosocomial infection, the cause may be difficult to
determine and a case-control study may aid in its identification
(Beck-Sague, Jarvis, and Martone 1977). In addition, the risk
factors for an important complication need to be understood so
that interventions can be instituted to reduce the incidence of
the problem among those patients (Trilla, Gatell, Mensa, Latorre,
Almela, Soriano, de Anta, and Miguel 1991, Herwaldt,
Swartzendruber, Edmond, Embrey, Wilkerson, Wenzel, and Perl 1998,
Myers, Baker, Van Datta, Abbey, and Robinson 1991, Shwartz, Ash
and Iezzoni 2001). This would be accomplished by tackling those
causes that are amenable to change such as improving preoperative
nutrition, cessation of smoking, and institution of chest
physical therapy to reduce the incidence of postoperative
pneumonia. Case-control studies are described in Chapter 5.

Before and after studies. Usually when a new method for patient
care is introduced, data are collected before and after the
change in order to gauge its effectiveness. There is considerable
potential for bias to limit the usefulness of this type of study
(Wen, Hernandez, and Naylor 1995), as the subjects for the older
study group have been selected by the process of time, and
special efforts must be made to strengthen the method. However,
as the time frame for QI studies is usually short, these
differences may be of less importance than when longer term
outcomes are being investigated.

Before and after measurements are often correlated; this

phenomenon is known as regression to the mean (Altman 1991).
Other problems include the differing preferences of the staff and

26
possibly the patients so that measurements of outcomes may be
made more diligently in one of the groups, or one of the methods
may be pursued with more vigour.

There are several ways in which these difficulties may be

mitigated. First, the result of any one study should be regarded
with caution and further monitoring should occur. If it can be
demonstrated that a change following the adoption of a new
process is sustained, or the change can be demonstrated at a
further time or in a different ward or hospital, belief in the
outcome will be strengthened. If the result of the study is
equivocal but the new method is in fact superior, a return to the
old process is likely to be accompanied by worsening outcomes.
These methods are sometimes called quasi-experimental designs
(Pellegrin, Carek and Edwards 1995).

A further useful method is sequential sampling (Reinke

1991), which has been used for many years in industrial quality
control. If the new method is superior, improvement following its
introduction will be sustained over time and sequential sampling
will demonstrate this. The data can be displayed in control
charts that are described in Chapter 4.

Special issues with QI studies. All QI studies require accurate

data for decision making. However, there is frequently neither
the time nor the resources available for instituting formal
research methods in QI studies; neither are they necessary to
answer many QI questions. It is one of the aims of medical
research to obtain results that are generalisable; QI studies are
often relevant only to the institution concerned. Unlike many
research projects, surveillance and audit are usually continuous
or recurrent activities.

Alemi, Moore, Headrick, Neuhauser, Hekelman, and Kizys

(1998) have described rapid data collection methods. They have
found the following factors to be important -

1. Planning for rapid data collection.

2. Collecting only data items that are needed.
3. Sampling patients using small representative samples.
4. Relying on numerical estimates made by process owners.

Berwick (1998) has described how making changes in the

processes of care is necessary for improving daily medical
practice. This can often be accomplished by conducting small
scale local tests using the Deming Cycle to learn from taking
action. Nelson, Splaine, Batalden, and Plume (1998) have
described eight principles for measurement in QI studies -

27
1. Seek usefulness, not perfection, in measurement.
2. Use a balanced set of process, outcome, and cost measures.
3. Keep measurement simple; think big but start small.
4. Use qualitative and quantitative data.
5. Write down the operational definition of the measures.
6. Measure small, representative samples.
7. Build measurement into daily work.
8. Develop a measurement team.

Berwick (1998) has emphasised the importance of using

measurement to learn rather than to judge. Much current
measurement seeks to compare and to judge. When there is an over-
emphasis on this aspect, managements and staff direct their
energies to “proving” that they are satisfactory, instead of
using data for learning and improvement. Useful data may be
suppressed and “gaming” and costly shifting of responsibility may
occur.

Complex and only partially effective methods of risk

adjustment are employed when making judgments so that comparisons
can be made fair; yet in spite of this activity, satisfactory
performance can still attract blame when differences are in fact
due to random variation. When measurement is used to judge, high
specificity is essential so that the risk of false positive
results is minimised. However, this can result in false negative
states; this is, as we have described in Chapter 1, often
potentially dangerous as it is the undetected problem that
frequently causes most difficulty and damage.

It is clear that formal research methods are frequently

neither necessary nor feasible in QI work. Indeed tackling QI as
a research activity will often guarantee failure.

When deciding which of two treatments is superior, it is

very important to avoid false positive results and a formal
research method is required; a new treatment that is expensive
and potentially toxic should not be found superior unless that is
definitely so. As we have described above, when improving quality
it can often be more important to avoid false negative outcomes.
To learn and improve, greater sensitivity is required so that
this is achieved.

Nevertheless, the principles from formal research for

avoiding bias are equally important in QI work. The ideas
described by Berwick, Batalden, Nelson, and others form the basis
for data collection and measurement methods in hospital
epidemiology.

28
Data collection and recording. It is important to consider
carefully what data to collect (Pottinger, Herwaldt, and Perl
1997). If information on an important variable is not collected,
it may be difficult or impossible to return later to do so. There
is often a temptation to collect much more data than are needed.
This too should be avoided; if data are collected and not used,
resources are wasted and their quality soon deteriorates. In
addition, surveillance resources can easily be overwhelmed so
that essential data may go uncollected.

It is best to determine as precisely as possible exactly

what data are needed. Discussion with colleagues is valuable and
a small pilot study (Altman 1991) should be considered to
determine whether the data collected are appropriate. In many
instances, similar studies will already have been performed and
these will give useful guidance. In particular, for nosocomial
infections, the data recommended for collection by NNIS will be
an excellent guideline (Pottinger, Herwaldt, and Perl 1997). We
have emphasised the need to analyse and optimise systems as an
essential first step in QI. When this has been performed, a
clearer picture will emerge of what data must be collected.

Data collection should be by a standard form (Boynton and

Greenhalgh 2004). The format suggested for the Enter program in
EpiInfo (Dean, Dean, Burton, and Dicker 1996) is very useful;
alternatively a commercial program such as eICAT (eICAT 2000) can
be used for nosocomial infection data. Often a paper form is
required initially with subsequent transcription to computer. In
some cases a hand-held computing device incorporating touch
screen technology can be used for data entry with direct transfer
to a larger computer by serial cable or infra-red device; this is
the preferred method in eICAT.

In general the less manual data entry the better. The

database program should have data checking capabilities so that
obviously wrong data are rejected; “Entry” in EpiInfo is very
good for this purpose and eICAT has routines for assessing data
quality. Thus, unless the data to be analysed are already
collected, for example in a large hospital or health care
database, EpiInfo or, for nosocomial infection data, eICAT is a
good choice for data entry.

When data are entered into the computer, care must be taken
to minimise errors. Although the computer program can reject
impossible data values, it cannot detect errors due to recording
a "Yes” answer when it should be "No” or to a misplaced decimal
point and so, following entry, data should be checked using

29
logical and statistical checks to detect and correct such errors.
Recent developments in the use of scanning equipment may make it
possible to scan data from a data collection form directly into a
database, thus reducing the likelihood of error. The use of hand
held devices with automatic transfer to a central database by
infra-red or similar method eliminates errors due to
transcribing.

Two other issues that frequently cause difficulty are

missing data values and date formats. When transferring data from
EpiInfo to a statistical analysis program, difficulties can occur
with each of these. Statistical programs represent missing values
with different symbols and there are a number of different date
formats used throughout the world.

It is important to study the manuals of the programs used

and to ensure that the data obtained from one program are in a
suitable format for the second program. Fortunately new variables
can easily be constructed in most modern programs such as EpiInfo
so that the necessary missing value codes can be generated.
EpiInfo automatically recognises European and U.S. date formats.

This concludes Chapter 2. In it we have -

1. Defined standards and criteria.

2. Discussed indicators including the difference between rate
based and sentinel indicators, the role of risk adjustment, and
the need for rigor in selecting indicators.
3. Described critical incident reporting, occurrence screening,
audit, epidemiological surveillance, patient satisfaction and
recovery, and comparative and observational studies.
4. Discussed data collection and recording with special emphasis
on the differences between QI studies and research.

In Chapter 3 we describe conventional statistical methods

for analysing QI data. Control chart methods, which are so
valuable for the analysis of these data, are dealt with in
Chapter 4.

Appendix 1.

Denominators.

It is important that, when comparisons are made, comparable

data are available, for example in the form of rates with
appropriate denominators such as bed-days, central line days,
urinary catheter days, or ventilator days. For SSI’s and
postoperative adverse events this is straightforward, the rate

30
being a proportion comprising the number of infections in the
numerator and the number of patients undergoing the relevant
operation in the denominator. However, for count data adverse
events such as new MRO colonisations and infections, deciding on
the best denominator may not be so straightforward. The available
denominators, such as occupied bed-days, may not be ideal since
infection and colonisation rates are often determined, in
addition to antibiotic usage and hygiene measures such as hand
washing, by the number of susceptible patients and the number of
people or objects shedding organisms into the environment,
neither of which is easily determined.

In practice, the number of count data nosocomial infections

is often related to the number of admissions to or discharges
from the hospital or its intensive care unit (separations) during
the period in question. In general, use of separations during the
time period in question as a denominator, so that the rate would
be a proportion, is incorrect. Patients could have been in
hospital for varying periods and the number in the hospital at
any time will vary. In addition, length of stay in hospital is
often an independent predictor of infection. Usually, in spite of
its limitations, the number of occupied bed-days will be a more
satisfactory denominator. For example, if there were 10 MRSA
infections and 25,000 occupied bed-days in an observation period
the rate would be 10/25000 or 0.4 per 1000 occupied bed-days for
that period.

Another possibility is to regard the hospital as a single

unit; this might be appropriate for monthly or yearly comparisons
for a single hospital for some adverse outcomes provided the
number of beds and their occupancy remain relatively constant.
However, it might not be satisfactory for such comparisons within
individual departments in a hospital if their bed status is very
variable, and it would be unsuitable for comparisons among
hospitals and departments.

We suggest that, when a single hospital or department is

studying its outcome data sequentially, it can be legitimate to
use observed monthly counts provided that bed-days or other
appropriate denominators are known to remain reasonably constant
during the period of observation. This is especially so with
outcomes that may not be independent, for example new
colonisations with multiple antibiotic resistant organisms
(MRO’s) such as methicillin resistant Staphylococcus aureus
(MRSA). Then, if there is a marked change in the rate of the
adverse event of interest and no cause can be found, an
inspection of the bed-days denominators may throw light on the
problem.

31
 Many hospital count data adverse events like MRO
colonisations and infections tend to cluster and may have too
high variability to be analysed by the conventional methods for
count data that assume independence and an approximate Poisson
distribution. In this situation, because it is the number of
susceptible patients and not the total number of patients that is
important, it is not unusual to see monthly counts of new MRO
isolates remain stable or even increase during periods of reduced
bed occupancy. It is unlikely to be easy to measure bed-days
taking patient susceptibility into account. However, areas like
ICU have more uniformly seriously ill and therefore more
susceptible patients. Thus denominators like bed-days may be more
satisfactory for their patient populations.

A fundamental difficulty is that the suggested denominator

may also be a risk factor. This can lead to severe bias. For
example, bacteraemias/central line days can give misleading
results when central lines are left in place for very varying
periods (McLaws and Taylor 2002). Ten central line days can be
from 10 patients each having one central line day or one patent
requiring 10 central line days. The risk of infection may be very
low in the former case but substantial in the latter, especially
if that patient is undergoing treatment for a septic condition.

When, for example, nosocomial pneumonias are ventilator

related it would be best to count the number of patients on
ventilators and the amount of time each required artificial
ventilation until pneumonia occurred or patients no longer
required ventilation. This would give as the denominator the
number of ventilator days. There may be difficulties obtaining
these data in some units. For example, some patients may have
more than one nosocomial pneumonia. It is usually important to
count the number of patients who have infections rather than the
number of infections. Thus, patients who already have nosocomial
pneumonia should strictly no longer be counted.

In some institutions it has been found that collecting this

information is unduly time consuming and a simple alternative is
to have a census each day or each few days of the numbers of
people on ventilators and the total numbers of patients
undergoing treatment. These daily counts are then added at the
end of each month and their ratio determined. This ratio is then
multiplied by the occupied bed-days for the month to obtain
ventilator days. Similar methods can be used for bacteraemias
associated with central intravenous lines (central line days) and
urinary infections in patients with indwelling urinary catheters
(catheter days).

32
 Many acute care public hospitals have such high uniform
occupancy that it is reasonable to consider the hospital as a
single unit for some indicator adverse occurrences when they are
being monitored on a regular basis. In this case the rate
denominator would be one and the count of positive outcomes would
be analysed. However, this may be unsatisfactory for individual
units and departments within a hospital if their bed occupancy
fluctuates markedly. For example an important though small area
such as the intensive care unit may go through a quiet or busy
period when the activities of other areas of the hospital are
unchanged.

If comparisons between hospitals are required, average bed

occupancy for the time interval being studied could be used as
the denominator. The Australian Council on Health Care Standards
(ACHS) compares similar types of hospitals. In this case,
comparisons could be made using average bed occupancy as a
denominator.

However, we question the use of such comparisons as a tool

for improving quality. It is usually much better for an
institution to analyse and optimise its system for infection
management and then to monitor and analyse its own process and
outcome data sequentially. This will enable it to determine
whether infection rates have become too great for random
variation to be a reasonable explanation for their occurrence.

When a hospital is being observed for some adverse outcome

and the data are being analysed with the assumption that the
denominator remains constant, it is important to ensure that
there are no important changes in bed occupancy during the period
of observation. If there are sustained changes in bed occupancy
due to bed additions or closures, the denominator should be
adjusted accordingly, or the analysis re-commenced after the
change. The analysis will often involve the use of control charts
and the control chart can be adjusted or re-commenced following
the change. Control charts are described in Chapter 4.

It is important to decide in which time period an event

should be included. For example, length of stay (LOS) is better
related to the time of the patient’s admission because LOS is
very dependent on decisions made at that time. Many hospitals
relate LOS to discharge date and this is less useful for quality
improvement work. Unplanned readmissions are probably better
related to the time of the original discharge rather than the
time of subsequent readmission because system and process
problems at the time of the discharge may account for excessive

33
numbers of readmissions.

It is likely that normal monthly fluctuations in the

denominator due to random variation, for example in the number of
occupied bed-days, are of lesser importance than changes due to
bed additions or closures. The number of very ill patients who
are likely to be susceptible to an adverse event such as a
nosocomial infection will usually fluctuate much less than the
overall bed-days. It may be useful to employ a data smoothing
technique such as the exponentially weighted moving average,
spline or lowess smoothing when examining denominators such as
occupied bed-days.

For hospital wide indicators, it is important that the type of

hospital be considered. When the adverse outcome is a device
related nosocomial infection occurring in an intensive care unit,
comparisons should only be made with similar units. However, we
believe that it is much more rewarding for individual hospitals
or units to compare their own work sequentially using control
chart methods and the Deming Cycle than it is to make comparisons
between institutions.

More work is needed to improve denominator usage (Sax and

Pittet 2002). In the meantime, occupied bed-days denominators
need to be used with care.

Appendix 2.

Risk Stratification and Adjustment.

In describing risk stratification and risk adjustment, we

mean by the former term that patients having similar expected
infection rates are placed in homogeneous groups for analysis
using direct standardisation. With risk adjustment, observed
infections and expected infection probabilities are added and
analysis is by indirect standardisation. These methods are
described in Chapter 3 and Chapter 4.

Risk adjustment is employed when a validated estimate of the

probability of an outcome occurring is available, for example an
APACHE score for the probability of death in patients in
Intensive Care Units (ICU) or the NNIS score for the probability
of a surgical site infection. In this case, the expected number
of adverse outcomes can be compared with the observed number in a
sample of patients or the data can be analysed sequentially in
control charts.

34
 For example, with ventilator related nosocomial pneumonia
there will be widely differing susceptibility among the patients
requiring artificial ventilation. Some patients will require only
a brief period of ventilation after a major operation or
following a drug overdose. Others will require ventilation,
usually for a longer period, as part of the treatment of a life-
threatening systemic illness, sometimes due to or accompanied by
a septic condition such as peritonitis. The former are unlikely
to suffer from nosocomial pneumonia while the latter may be
fortunate to avoid the complication. This problem may also apply
to other adverse outcomes such as nosocomial bacteraemias. Thus
stratification of patients into comparable susceptibility
groupings is important. Alternatively, risk adjustment can be
performed if, for each patient, it is possible to estimate the
probability of the adverse event occurring, for example using
logistic regression. The observed and expected number of the
events can then be added and compared.

Sophisticated statistical expertise is necessary to devise,

test and validate complex adverse outcome risk adjustment
procedures (Hofer, Bernstein, Hayward, and DeMonner 1997). In
addition, considerable cost may be involved. This is likely to be
beyond the resources available to many institutions.

Methods for stratifying for SSIs and risk adjusting for the
following outcomes are now well developed -

1. Intensive care unit mortality

2. Cardiac surgical mortality
3. Myocardial infarction mortality
4. Mortality following other major surgical procedures
5. Pneumonia severity
6. Abdominal aortic aneurysm mortality.

For surgical site infections the National Nosocomial

Infections Surveillance System (NNIS Risk Index) is widely
employed (Culver, Horan, Gaynes, Martone, Jarvis, Emori,
Banerjee, Edwards, Tolson, Henderson, Hughes and the National
Nosocomial Infections Surveillance System 1991). Surgical wounds
are classified by class (class 1 for clean, class 2 for
contaminated, and class 3 for infected wounds), length of
operation (less than or equal to the 75th percentile, or greater
than the 75th percentile for the operation concerned), and
American Society of Anaesthetists (ASA) Classification (classes 1
and 2 for not seriously ill patients, or classes 3 to 5 for
seriously to critically ill patients).

Ordinarily there will be a low rate of these infections in

35
patients in the lowest risk class (class 1 wound, normal
operation time, ASA class 1 or 2), such as 2% or 3%, even when
surgery is of high standard, and the rate will vary over time due
to random processes. However, with substandard care the rate will
rise, possibly to 6% or even higher. Surveillance is aimed at
early detection of such rate changes.

Unfortunately, the NNIS risk index is not helpful with some

groups of patients. These include caesarian sections,
craniotomies, and cardiothoracic procedures (Roy and Perl 1998).
However, using surveillance to monitor surgical site infections
in homogeneous groups of common major operations is a useful
alternative to risk stratification or risk adjustment.

It is important to distinguish between the risk adjustment

requirement for comparisons between institutions and that
required for sequential analysis of data when a single
institution is examining its own data, for example by using
control charts. As we have indicated, we believe that comparisons
between institutions have very limited potential to improve
quality. It is usually much better for the staff of an
institution to analyse and optimise their system and then to
perform sequential analysis of relevant process and outcome data.
This enables variation that is in excess of random fluctuation to
be detected in a timely manner.

In the case of comparisons between institutions, where

judgments are being made, it is important to avoid false positive
signals and high specificity together with complex risk
adjustment methods is required. Unfortunately, this approach is
likely to be self defeating as, when specificity is kept high,
sensitivity is lowered and false negative states become more
likely. Important variations may then be missed or only
identified following unacceptable delay. This is especially
likely with uncommon outcomes such as unexpected deaths and
uncommon hospital acquired infections. False negative results are
important for hospitals - if a quality problem is developing but
surveillance does not detect it, a serious complication could
occur before it is recognised.

When a single institution is examining its own processes and

outcomes sequentially, staff are usually analysing their data to
learn how to do better. It is then possible to increase
sensitivity so as to avoid false negative states. In a learning
environment, occasional false positive signals can generally be
tolerated. Surveillance is then being used more as a screening
test. Although prompt investigation is still required if a
possible problem is to be detected quickly, it may occasionally

36
prove to be a false alarm. Fortunately these events are usually
not difficult for an experienced hospital epidemiologist or
infectious diseases physician to detect.

However, with this approach, false negative results are less

likely and unrecognised quality problems would be less prone to
occur. Used in this way, the process becomes one of learning
rather than judgment. We believe that genuine progressive QI can
occur only in a learning environment.

As indicated above, when data are being generated from a

number of hospitals or departments and used for making
comparisons, complex risk adjustment methods will be needed and a
specific analysis required. In addition, Bayesian shrinkage
estimators should be used. When this is done, there is
substantial risk of missing a real problem or only detecting it
after unacceptable delay. A further difficulty in comparing
institutions is that demographic differences may exist for which
it is difficult of impossible to risk adjust.

Random variation, patient characteristics, and possible bias

due to error and gaming can all cause variation in addition to
substandard performance. Because risk adjustment methods are not
perfect and random variation can be relatively large compared to
variation due to substandard performance, very large samples may
be needed to detect the latter reliably for some outcomes. Since
few hospitals may have sufficient data within a realistic time
frame to attain the needed sample size, comparisons should be
made with great caution. For making comparisons and judgments, it
is better to rely on systems analysis and process surveillance;
these methods usually require intimate involvement by properly
trained management rather than statistical analysis. In
particular, the currently favoured method of employing league
tables is deeply flawed. It is more likely to increase adverse
outcomes than prevent them.

Consider the situation where one death is expected. It takes

at least four to occur for statistical significance to be
attained. Reliance on statistical analysis will result in
unacceptable delay if it is possible, employing systems analysis,
to determine that an environment exists in which excess deaths
may be expected to occur. Conversely, when an institution
analyses its systems and institutes measures to improve,
subsequent monitoring and analysis of relevant data using
statistical methods can be an invaluable adjunct to systems
analysis and improvement.

The frequency of positive indicator outcomes is dependent on

37
the susceptibility of patients to developing an adverse outcome
and random variation as well as the quality of patient care. This
susceptibility will be influenced by factors such as hospital
size and type (Australian Council on Healthcare Standards 1995).
In addition, the following are likely to be of importance –

1. Age
2. The prognosis of the patient's presenting acute illness
3. Presenting acute physiology features such as blood
pressure, pulse rate, level of consciousness, urine output,
respiration and oxygenation and the presence or absence of
jaundice and acute infection and,
4. Any associated major chronic illnesses that may be present.

Most work on risk adjustment has focused on adjusting the

probabilities of death during hospitalisation (Iezzoni 1994)
although there is considerable doubt that they are able to
achieve this objective reliably in many cases (Thomas and Hofer
1999). The methods are numerous and complex and often unsuitable
for use by quality assurance staff of individual hospitals
wishing to adjust their own surveillance data. In addition,
probabilities of death and susceptibilities to developing
infections and other indicator adverse events are not necessarily
the same.

We believe that, when an individual hospital is examining

its own performance sequentially, for example by employing
control charts, much less complex risk adjustment will suffice.
Graham and Cook (2004) have shown this to be the case with adult
ICU mortality.

Although better methods need to be developed, the simple

McCabe-Jackson score described in Chapter 1 has been shown to be
useful for risk stratification of nosocomial infections in
intensive care units (Gross, Stein, Antwerpen, DeMauro, Boscamp,
Hess, and Wallenstein 1991). The patient’s illness is classified
as rapidly fatal (the patient has a high likelihood of death
during the present admission), ultimately fatal (the patient is
expected to die within the next 5 years), and non-fatal. An
alternative is the Severity of Illness Classification (SIC)
proposed by Salemi, Morgan, Kelleghan, and Hiebert-Crape (1993)
or one more recently proposed by Charlson, Hollenberg, Hou,
Cooper, Pochapin and Pecker (2000). This is a simple general
stratification method similar to the ASA score for surgical
patients (Salemi, Morgan, Kelleghan, and Hiebert-Crape 1993). It
may prove useful when a particular method is not available and a
single institution is examining its own data sequentially for the
purpose of improving quality.

38
 Weightman, Gibbs, Sheminant, Thackray, and Newman (1997)
have shown that the very simple Tremblay risk score is useful for
stratifying cardiac surgical patients for the indicators death
and prolonged hospital stay. Hall and Hall (1996) have shown that
age and ASA status predict adverse events after abdominal
surgery. The NNIS risk stratification described above for
surgical site infections uses a simple 4 groupings (Roy and Perl
1998). Lawrance, Dorsch, Sapsford, Mackintosh, Greenwood,
Jackson, Morrell, Robinson and Hall (2001) have used easily
obtained data to risk adjust myocardial infarction mortality and
Sutton, Bann, Brooks and Sarin (2002) have proposed a surgical
risk scale to use with surgical audit. Thus, although better
methods for risk adjustment and stratification should be
developed for specific indicators, there are available now simple
techniques that can be employed by hospital QI and IM staff to
risk stratify or risk adjust their data.

There are further important issues with risk-adjustment when

it is used for comparisons between institutions. We have
mentioned the difficulty or impossibility of including all
important demographic factors in arriving at a suitable risk-
adjustment formula. In addition, calibration is very important as
a formula that predicts less adverse outcomes than it should for
a risk group will disadvantage institutions with large numbers of
patients in that risk group. Also, there are potential problems
when the risk-adjusted data are analysed in control charts. These
problems are dealt with in succeeding chapters. As we have
indicated, we believe that comparisons between institutions have
very limited value for QI. These difficulties with risk-
adjustment are further reasons for this belief. Institutions
should analyse and optimize their systems and then analyse their
own data sequentially, for example in control charts. If
comparisons have to be made, system and process indicators should
be used.

References.

Boynton P and Greenhalgh T “Selecting, Designing and Developing

Your Questionnaire” British Medical Journal 2004;328:1312.

Sale D "Quality Assurance for Nurses and Other Members of the

Health Care Team" London MacMillan 2nd edition 1996.

Decker M “The Development of Indicators” Infection Control and

Hospital Epidemiology 1991;12:490.

39
Bernstein S and Hilborne L “Clinical Indicators: The Road to
Quality Care?” The Joint Commission Journal on Quality
Improvement” 1993;11:501.

Hofer T, Bernstein S, Hayward R, and DeMonner S “Validating

Quality Indicators for Hospital Care” The Joint Commission
Journal on Quality Improvement 1997;23:455.

Donabedian A “The Quality of Care How Can it be Assessed?”

Journal of the American Medical Association 1988;260:1743.

Deming W “Out of the Crisis” Cambridge Cambridge University Press

Cambridge 1982.

Nolan T “Understanding Medical Systems” Annals of Internal

Medicine 1998;128:293.

Woolf S “Practice Guidelines: A New reality in Medicine” Parts 1-

3. Archives of Internal Medicine 1990;150:1811, 1992;152:946, and
1993;153:2646.

Chalmers I and Altman D “Systematic Reviews” BMJ Publishing 1995.

Sackett D, Straus S, Richardson W, Rosenberg W, and Haynes R

"Evidence-based Medicine How to Practice and Teach EBM" New York
Churchill Livingstone 2nd edition 2000.

Fauci A, Braunwald E, Isselbacher K, Wilson J, Martin J, Kasper

D, Hauser S, and Longo D Harrison’s “Principles of Internal
Medicine Companion Handbook New York McGraw-Hill 14th edition
1998.

Classen D, Evans S, Pestronik S, Horn S, Menlove R, and Burke J

“The Timing of Prophylactic Administration of Antibiotics and the
Risk of Surgical Wound Infection” The New England Journal of
Medicine 1992;326:281.

Handwashing Liaison Group “Hand Washing” British Medical Journal

1999;318:686.

Naylor M, Brooten D, Jones R, Lavizzo-Mourey R, Mezey M, and

Pauly M "Comprehensive Discharge Planning for the Hospitalised
Elderly" Annals of Internal Medicine 1994;120:999.

Treasure T and Bennett D “Reducing the Risk of Major Elective

Surgery” British Medical Journal 1999;318:1087.

40
Thomas W and Hofer T “Accuracy of Risk-Adjusted Mortality Rate as
a Measure of Hospital Quality of Care” Medical Care 1999;37:83.

Iezzoni L “Risk Adjustment for Measuring Health Care Outcomes”

Michigan Health Administration Press 1994.

Jones J and Hunter D “Consensus Methods for Medical and Health

Services Research” British Medical Journal 1995;311;376.

Baker O “Process Surveillance: An Epidemiologic Challenge for all

Health Care Organisations” AJIC American Journal of Infection
Control 1997;25:96.

Davies H and Crombie I "Assessing the Quality of Care" British

Medical Journal 1995;311:766.

Mant J and Hicks N “Detecting Differences in Quality of Care: The

Sensitivity of Measures of Process and Outcome in Treating Acute
Myocardial Infarction” British Medical Journal 1995;311:793.

Pearson S, Goulart-Fisher D, and Lee T “Clinical Pathways as a

Strategy for Improving Care” Annals of Internal Medicine
1995;123:941.

Sprent P “Management Mathematics” Penguin Books London 1991.

Quality Indicator Study Group “An Approach to the Evaluation of

Quality Indicators of the Outcome of Care in Hospitalised
Patients, with a focus on Nosocomial Infection Indicators”
American Journal of Infection Control 1995;23:215.

Nadzam D and Nelson M “The Benefits of Continuous Performance

Measurement” Nursing Clinics of North America 1997;32:543.

Culver D, Horan T, Gaynes R, Martone W, Jarvis W, Emori G,

Banerjee S, Edwards J, Tolson J, Henderson T, Hughes J and the
National Nosocomial Infections Surveillance System “Surgical
wound Infection Rates by Wound Class, Operative Procedure, and
Patient Risk Index” The American Journal of Medicine 1991;91
(supplement 3B):152.

Roy M and Perl T “Basics of Surgical Site Infection Surveillance”

in A Practical Handbook for Hospital Epidemiologists edited by
Herwaldt L and Decker M 1998 Thorofare Slack Incorporated 99.

Journal on Quality Improvement “Sentinel Events: Approaches to

Error Reduction and Prevention” book excerpt 1998;24:175.

41
de Leval M “Human Factors and Surgical Outcomes: A Cartesian
Dream” The Lancet 1997;349:723.

Nadzam D and Nelson M “The Benefits of Continuous Performance

Measurement” Nursing Clinics of North America 1997;32:543.

Australian Council on Healthcare Standards “Clinical Indicators -

A Users’ Manual” Version 4 1995.

Jones J and Hunter D “Consensus Methods for Medical and Health

Services Research” British Medical Journal 1995;311;376.

Gross P, Stein M, Antwerpen C, DeMauro P, Boscamp J, Hess W, and

Wallenstein S “Comparison of Severity of Illness Indicators in an
Intensive Care Unit” Archives of Internal Medicine 1991;151:2201.

Salemi C, Morgan J, Kelleghan S, and Hiebert-Crape B “Severity of

Illness Classification for Infection Control Departments: A Study
in Nosocomial Pneumonia” American Journal of Infection Control
1993;21:117.

Charlson M, Hollenberg J, Hou J, Cooper M, Pochapin M, and Pecker

M “Realising the Potential of Clinical Judgment: A Real-Time
Strategy for Predicting Outcomes and Cost for Medical Inpatients”
American Journal of Medicine 2000;109:189.

Weightman W, Gibbs N, Sheminant M, Thackray M, and Newman M “Risk

Prediction in Coronary Artery Surgery” Medical Journal of
Australia 1997;166:408.

Hall J and Hall J “ASA Status and Age Predict Adverse Events
after Abdominal Surgery” Journal of Quality in Clinical Practice
1996;16:103.

Lawrance R, Dorsch M, Sapsford R, Mackintosh A, Greenwood D,

Jackson B, Morrell C, Robinson M and Hall A “Use of Cumulative
Mortality Data in Patients with Acute Myocardial Infarction for
Early Detection of Variation in Clinical Practice: Observational
Study” British Medical Journal 2001;323:324.

Sutton R, Bann S, Brooks M and Sarin S “The Surgical Risk Scale

as an Improved Tool for Risk-adjusted Analysis in Comparative
Audit” British Journal of Surgery 2002;89:763.

Massanari M, Wilkerson K, and Swartzendruber S “Designing

Surveillance for Noninfectious Outcomes of Medical Care”
Infection Control and Hospital Epidemiology 1995;16:419.

42
Palmer H and Lawthers A “Computerised Needs-Oriented Quality
Measurement Evaluation System (CONQUEST) Washington, DC: Agency
for Health Care Policy and Research, President and Fellows of
Harvard College, 1996.

Vincent C “Risk, Safety, and the Dark Side of Quality” British

Medical Journal 1997;314:1775.

Wolff A "A Review of Methods Used for Medical Quality Assurance

in Hospitals: Advantages and Disadvantages" Journal of Quality in
Clinical Practice 1994;14:85.

Wolff A “Limited Adverse Occurrence Screening: An Effective and

Efficient Method of Medical Quality Control” Journal of Quality
in Clinical Practice 1995;15:221.

Wolff A “Limited Adverse Occurrence Screening: Using Medical

Record Review to Reduce Hospital Adverse Patient Events” Medical
Journal of Australia 1996;164:458.

Wolff A and Bourke J “Reducing Medical Errors: A Practical Guide”

Medical Journal of Australia 2000;173:247.

Leape L “Error in Medicine” Journal of the American Medical

Association 1994;272:1851.

Buckley T, Short T, Rowbottom Y, and Oh T “Critical Incident

Reporting in the Intensive Care Unit” Anaesthesia 1997;52:403.

Bates D, Makary M, Teich J, Pedraza L, Ma’luf N, Burstin H, and

Brennan T “Asking Residents About Adverse Events in a Computer
Dialogue: How Accurate Are They?” Journal on Quality Improvement
1998;24:197.

Andrews L, Stocking C, Krizek T, Gottlieb L, Krizek C, Vargish T,

and Siegler M “An Alternative Strategy for Studying Adverse
Events in Medical Care” The Lancet 1997;349:309.

Green R “Human Error on the Flight Deck” Philosophical

Transactions of the Royal Society of London 1990;B-327:503.

de Leval M, François K, Bull C, Brawn W, and Spiegelhalter D

“Analysis of a Cluster of Surgical Failures” Journal of Thoracic
and Cardiovascular Surgery 1994;107:914.

Blumenthal D “Making Medical Errors into Medical Treasures”

Journal of the American Medical Association 1994;272:1867.

43
James B “Every Defect a Treasure: Learning from Adverse Events in
Hospitals” Medical Journal of Australia 1997;166:484.

Reinke W “Applicability of Industrial Sampling Techniques to

Epidemiologic Investigations: Examination of an Underutilised
Resource” American Journal of Epidemiology 1991;134:1222.

Wilson R, Runciman W, Gibberd R, Harrison B, Newby L, and

Hamilton J “The Quality in Australian Health Care Study” Medical
Journal of Australia 1995;163:458.

Localio R, Weaver S, Landis R, Lawthers A, Brennan T, Hebert L,

and Sharp T “Identifying Adverse Events Caused by Medical Care:
Degree of Physician Agreement in a Retrospective Chart Review”
Annals of Internal Medicine 1996;125:457.

Sanazaro P and Mills D “A Critique of the Use of Generic

Screening in Quality Assessment” Journal of the American Medical
Association 1991;265:1977.

Miles T and Lowe J “Are Unplanned Readmissions to Hospital Really

Preventable” Journal of Quality in Clinical Practice 1999;19:211.

Craddick J “The Medical Management Analysis System - A

Professional Liability Warning System” Quality Review Bulletin
1979;5:2.

Brennan T, Localio R, Leape L, Laird N, Peterson L, Hiatt H, and

Barnes B “Identification of Adverse Events Occurring during
Hospitalisation” Annals of Internal Medicine 1990;112:221.

Brook R, Kahn K, and Kosecoff J “Assessing Clinical Instability

at Discharge” Journal of the American Medical Association
1992;268:1321.

Sackett D, Straus S, Richardson W, Rosenberg W, and Haynes R

"Evidence-based Medicine How to Practice and Teach EBM" New York
Churchill Livingstone 2nd edition 2000.

Tuckfield A, Haeusler M, Grigg A, and Metz J “Reduction of

Inappropriate Use of Blood Products by Prospective Monitoring of
Transfusion Request Forms” Medical Journal of Australia
1997;167:473.

Baker R and Fraser R “Development of Review Criteria: Linking

Guidelines and Assessment of Quality” British Medical Journal
1995;311:370.

44
Williams O “What is Clinical Audit?” Annals of the Royal College
of Surgeons of England 1996;78:406.

Wilson C "Strategies in Health Care Quality" Toronto [Link]

1992.

McDermott F and Scott D “Surgical Epidemiology, Surveys and

Auditing” in “Clinical Science for Surgeons” edited by Marshall V
and Ludbrook J Sydney Butterworths 2nd edition 1988.

Zafar A, Butler C, Podgorny J, Mennonna P, Gaydos L, and

Sandiford J “Effect of a Comprehensive Program to Reduce
Needlestick Injuries” Infection

Walker D, Williams P, and Tawn J “Audit of Requests for

Preoperative Chest Radiography” British Medical Journal
1994;309:772.

Berger A “Why Doesn’t Audit Work?” British Medical Journal

1998;316:875.

Crombie I and Davies H “Towards Good Audit” British Journal of

Hospital Medicine 1992;48:182.

Dawson K and Capaldi N “Acute Laryngo-tracheo-bronchitis (croup):

An Audit of Hospital Practice” Australian Clinical Review
1993;13:63.

Williams H and Macdonald D “Audit of Thromboembolic Prophylaxis

in Hip and Knee Surgery” Annals of the Royal College of Surgeons
of England 1997;79:55.

Dickinson E “Clinical Effectiveness for Health Care Quality

Improvement” Journal of Quality in Clinical Practice 1998;18:37.

Sax H and Pittet D “Interhospital Differences in Nosocomial

Infection Rates” Archives of Internal Medicine 2002;162:2437.

McLaws M-L and Taylor P “The Hospital Infection Surveillance

Programme” Journal of Hospital Infection 2003;53:259.

Pottinger J, Herwaldt L, and Perl T “Basics of Surveillance-An

Overview” Infection Control and Hospital Epidemiology
1997;18:513.

Massanari M, Wilkerson K, and Swartzendruber S “Designing

Surveillance for Noninfectious Outcomes of Medical Care”
Infection Control and Hospital Epidemiology 1995;16:419.

45
D’Arco S and Hargreaves M “Needlestick Injuries” Nursing Clinics
of North America 1995;30:61.

Smith H, Pryce A, Carlisle L, Jones M, Scarpello J, and Pantin C

“Appropriateness of Acute Medical Admissions and Length of Stay”
Journal of the Royal College of Physicians of London 1997;31:527.

Page S and Washburn T “Using Tracking Data to Find Complications

that Physicians Miss” Journal on Quality Improvement 1997;23:511.

Cookson S, Ihrig M, O,Mara E, Hartstein A, and Jarvis W “Use of

an Estimation Method to Derive an Appropriate Denominator to
Calculate Central Venous Catheter-Associated Bloodstream
Infection Rates” Infection Control and Hospital Epidemiology
1998;19:28.

Lawrance R, Dorsch M, Sapsford R, Mackintosh A, Greenwood D,

Jackson B, Morrell C, Robinson M and Hall A “Use of Cumulative
Mortality Data in Patients with Acute Myocardial Infarction for
Early Detection of Variation in Clinical Practice: Observational
Study” British Medical Journal 2001;323:324.

Zaza S and Jarvis W “Investigation of Outbreaks” in Hospital

Epidemiology and Infection Control edited by Mayhall G Baltimore
Williams and Wilkins 1996.

Cardo D, Falk P, and Mayhall G “Validation of Surgical Wound

Serveillance” Infection Control and Hospital Epidemiology
1993;14:211.

Beck-Sague C, Jarvis W, and Martone W “Outbreak Investigations”

Infection Control and Hospital Epidemiology 1997;18:138.

Checko P “Use of Statistics for Epidemiology” in APIC Infection

Control and Applied Epidemiology edited by Olmsted R St Louis
Mosby-Year Book Inc. 1996.

Morton R, Hebel R, and McCarter R “A Study Guide to Epidemiology

and Biostatistics” Rockville Aspen Publishers 3rd edition 1990.

Wendt C and Herwaldt L “Epidemics: Identification and Management”

in Prevention and Control of Nosocomial Infections edited by
Wenzel R Baltimore Williams and Wilkins 3rd edition 1997.

Donabedian A “The Definition of Quality and Approaches to Its

Assessment” Michigan Health Administration Press 1980.

46
Cunningham L “The Quality Connection in Health Care: Integrating
Patient Satisfaction and Risk Management” San Francisco
Jossey-Bass Incorporated 1991.

Williams C “Qualitative Research” in APIC Infection Control and

Applied Epidemiology St Louis [Link] 1996.

Pope C and Mays N “Reaching the Parts Other Methods Cannot Reach:
An Introduction to Qualitative Methods in Health and Health
Services Research” British Medical Journal 1995;311:42. (This is
the first paper in a series of seven devoted to qualitative
methods).

Kitzinger J “Introducing Focus Groups” British Medical Journal

1995;311:299.

Schattner P, Shmerling A, and Murphy B “Focus Groups: A Useful

Research Method in General Practice” Medical Journal of Australia
1993;158:622.

Dawson S, Manderson L, and Tallo V “A Manual for the Use of Focus

Groups” Boston INFDC 1993.

Paterson C “Measuring Outcomes in Primary Care: A Patient

Generated Measure, MYMOP, Compared with the SF-36 Health Survey”
British Medical Journal 1996;312:1016.

Ware J “Measuring Patients’ Views: the Optimum Outcome Measure”

British Medical Journal 1993;306:1429.

Ware J, Kosinski M, and Keller S “A 12-Item Short-Form Health

Survey” Medical Care 1996;34:220.

Garratt A, Ruta D, Abdalla M, Buckingham K, and Russell I “The SF

36 Health Survey Questionnaire: An Outcome Measure Suitable for
Routine Use Within the NHS?” British Medical Journal
1993;306:1440.

Nelson E, Larson C, Davies A, Gustafson D, Ferreira P, and Ware J

“The Patient Comment Card: A System to Gather Customer Feedback”
Quality Review Bulletin 1991;9:278.

Westbrook J “Patient Satisfaction: Methodological Issues and

Research Findings” Australian Health Review 1993;16:75.

Fitzpatrick R “Surveys of Patient Satisfaction” British Medical

Journal 1991;302:887 and 1129.

47
Krishnan J and Chipchase L “Orthopaedic Surgery Outcomes
Assessment Model” Journal of Quality in Clinical Practice
1997;17:109.

Hall J, Tarala R, Tapper J, and Hall J “Prevention of Respiratory

Complications After Abdominal Surgery: A Randomised Clinical
Trial” British Medical Journal 1996;312:148.

Altman D “Practical Statistics for Medical Research” London

Chapman and Hall 1991.

Woodward M “Epidemiology” London Chapman and Hall/CRC 2nd edition

2005.

Rothman K and Greenland S “Modern Epidemiology” 2nd edition

Philadelphia Lippincott-Raven 1998.

Joshi N, Localio R, and Hamory B “A Predictive Risk Index of

Nosocomial Pneumonia in the Intensive Care Unit” The American
Journal of Medicine 1992;93:135.

Kerr C, Taylor R, and Heard G “Handbook of Public Health Methods”

Sydney McGraw-Hill 1998.

Watanakunakorn P, Watanakunakorn C, and Hazy J “Risk Factors

Associated with Clostridium Difficile Diarrhea in Hospitalised
Adult Patients” Infection Control and Hospital Epidemiology
1996;17:232.

Ena J, Cercenado E, Martinez D, and Bouza E “Cross-Sectional

Epidemiology of Phlebitis and Catheter-Related Infections”
Infection Control and Hospital Epidemiology 1992;13:15.

Doebbeling B, Stanley G, Sheetz C, Pfaller M, Houston A, Annis L,

Li N, and Wenzel R “Comparative Efficacy of Alternative Hand-
washing Agents in Reducing Nosocomial Infections in Intensive
Care Units” The New England Journal of Medicine 1992;327:88.

Bailey T, Topham T, Wantz S, Grant M, Cox C, Jones D, Zerbe T,

and Spears T “Laboratory Process Improvement Through Point-of-
Care Testing” The Joint Commission Journal on Quality Improvement
1997;23:362.

Kendall J, Reeves B, and Clancy M “Point of Care Testing:

Randomised Controlled Trial of Clinical Outcome” British Medical
Journal 1998;316:1052.

Sibbald B and Roland M “Why are Randomised Controlled Trials

48
Important” British Medical Journal 1998;316:201.

Warlow C “The Design of Controlled Clinical Trials” Oxford

Textbook of Surgery edited by Morris P and Malt R Oxford Oxford
University press 1994.

Torgerson D and Sibbald B “What is a Patient Preference Trial?”

British Medical Journal 1998;316:360.

Treasure T and MacRae K “Minimisation: The Platinum Standard for

Trials” British Medical Journal 1998;317:362.

Roland M and Torgerson D “What are Pragmatic Trials?” British

Medical Journal 1998;316:285.

Schwartz D and Lellouch J “Explanatory and Pragmatic Attitudes in

Therapeutic Trials” Journal of Chronic Diseases” 1967;20:637.

Dean A, Dean A, Burton A, and Dicker D “Epi Info version 6: A

Wordprocessing, Database, and Statistics Program for Epidemiology
on Microcomputers” USD, Incorporated, Stone Mountain, Georgia,
1996.

Pocock S “Clinical Trials: A Practical Approach”

Chichester John Wiley and Sons 1983.

Ehrenkranz J, Nerenberg D, Shultz J, and Slater K “Intervention

to Discontinue Parenteral Antimicrobial Therapy in Patients
Hospitalised with Pulmonary Infections: Effect on Shortening
Patient Stay” Infection Control and Hospital Epidemiology”
1992;13:21.

L’Ecuyer P, Schwab E, Iademarco E, Barr N, Aton E, and Fraser V

“Randomised prospective Study of the Impact of Three Needleless
Intravenous Systems on Needlestick Injuries” Infection Control
and Hospital Epidemiology 1996;17:803.

Miller N, Smith P, DeBusk R, Sobel D, and Taylor B “Smoking

Cessation in Hospitalised Patients” Archives of Internal Medicine
1997;157:409.

Naylor M, Brooten D, Jones R, Lavizzo-Mourey R, Mezey M, and

Pauly M "Comprehensive Discharge Planning for the Hospitalised
Elderly" Annals of Internal Medicine 1994;120:999.

Pierce R, McDonald C, Landau L, LeSouef P, Armstrong J, Mitchell

C, Francis P, Martin J, Musk W, Antic R, Clark E, and Ryder E
“Nebuhaler Versus Wet Aerosol for Domiciliary Bronchodilator

49
Therapy” The Medical Journal of Australia 1992;156:771.

Smith G and Phillips A “Confounding in Epidemiological Studies:

Why Indepencent Effects May Not Be All They Seem” British Medical
Journal 1992;305:757.

Wen S, Hernandez R, and Naylor D “Pitfalls in Nonrandomised

Outcomes Studies” Journal of the American Medical Association
1995;274:1687.

Pellegrin K, Carek D, and Edwards J “Use of Experimental and

Quasi-Experimental Methods for Data-Based Decisions in QI”
Journal on Quality Improvement 1995;21:683.

Britt M, Schleupner C, and Matsumiya S “Severity of Underlying

Disease as a Predictor of Nosocomial Infection” Journal of the
American Medical Association 1978;239:1047.

Freeman J “Quantitative Epidemiology” in A Practical Handbook for

Hospital Epidemiologists edited by Herwaldt L and Decker M
Thorofare Slack 1996.

Black N “Why We Need Observational Studies to Evaluate the

Effectiveness of Health Care” British Medical Journal
1996;312:1215.

Fernández-Crehuet R, Díaz-Molina C, de Irala J, Martínez-Concha

D, Salcedo-Leal I, and Masa-Calles J “Nosocomial Infection in an
Intensive-Care Unit: Identification of Risk Factors” Infection
Control and Hospital Epidemiology 1997;18:825.

Shwartz M, Ash A and Iezzoni L “Use of Nonexperimantal Studies to

Evaluate Surgical Procedures and Other Interventions: The
Challenge of Risk Adjustment” in Surgical Research, Souba W and
Wilmore D editors San Diego Academic Press 2001.

Delgardo-Rodríguez M, Sillero-Arenas M , Medina-Cuardos M, and

Martínez-Gallego G “Nosocomial Infections in Surgical Patients:
Comparison of Two Measures of Intrinsic Patient Risk” Infection
Control and Hospital Epidemiology 1997;18:19.

Mayotte J, Pisano M, Ram s, Nakasato S, and Rotella D “Validation

of a Bacteraemia Prediction Model” Infection Control and Hospital
Epidemiology 1995;16:203.

Moro M, Viganò E, and Lepri A “Risk Factors for Central Venous

Catheter-Related Infections in Surgical and Intensive Care Units”
Infection Control and Hospital Epidemiology 1994;15:253.

50
Thompson B, Dwyer D, Ussery X, Denman S, Vacek P, and Schwartz B
“Handwashing and Glove Use in a Long-Term-Care Facility”
Infection Control and Hospital Epidemiology 1997;18:97.

Ansari S, Springthorpe S, Sattar S, Tostowaryk W, and Wells G

“Comparison of Cloth, Paper, and Warm Air Drying in Eliminating
Viruses and Bacteria from Washed Hands” American Journal of
Infection Control 1991;19:243.

APIC Infection Control and Applied Epidemiology CDC Definitions

of Nosocomial Infections Appendix A St Louis [Link] 1996.

Gibson P, Wlodarczyk J, Hensley M, Murree-Allen K, Olson L, and

Saltos N “Using Quality-Control Analysis of Peak Expiratory Flow
Recordings to Guide Therapy for Asthma” Annals of Internal
Medicine 1995;123:488.

Clayton D and Hills M “Statistical Models in Epidemiology” Oxford

Science Publications Oxford 1993.

Trilla A, Gatell J, Mensa J, Latorre X, Almela M, Soriano E, de

Anta M, and Miguel J “Risk Factors for Nosocomial Bacteraemia in
a Large Spanish Teaching Hospital: A Case-Control Study”
Infection Control and Hospital Epidemiology 1991;12:150.

Herwaldt L, Swartzendruber S, Edmond M, Embrey R, Wilkerson K,

Wenzel R, and Perl T “The Epidemiology of Haemorrhage Related to
Cardiothoracic Operations” Infection Control and Hospital
Epidemiology 1998;19:9.

Myers A, Baker S, Van Datta M, Abbey H, and Robinson E “Risk

Factors Associated with Falls and Injuries among Elderly
Institutionalised Persons” American Journal of Epidemiology
1991;133:1179.

Alemi F, Moore S, Headrick L, Neuhauser D, Hekelman F, and Kizys

N "Rapid Improvement Teams" Journal on Quality Improvement
1998;24:119.

Berwick D “Developing and Testing Changes in Delivery of Care”

Annals of Internal Medicine 1998;128:651.

Nelson E, Splaine M, Batalden P, and Plume S “Building

Measurement and Data Collection into Medical Practice” Annals of
Internal Medicine 1998;128:460.

eICAT Infection Management Services Princess Alexandra Hospital

51
Brisbane [Link].

Graham P and Cook D “Prediction of Risk of Death Using Thirty-day

Outcome: A Practical Endpoint for Quality Monitoring” Chest
2004;125:1458.

Craven D and Steger K “Hospital-Acquired Pneumonia: Perspectives

for the Healthcare Epidemiologist” Infection Control and Hospital
Epidemiology 1997;18:783.

Shmerling A, Schattner P and Piterman L “Qualitative Research in

Medical Practice” Medical Journal of Australia 1993;158:609.

Salemi C, Morgan J, Padilla S, and Morrissey R “Association

Between Severity of Illness and Mortality from Nosocomial
Infection” American Journal of Infection Control 1995;23:188.

Smith D “Primary Angioplasty Should be First Line treatment for

Acute Myocardial Infarction” British Medical Journal
2004;328:1254.

52