Systems Thinking Scale

STS User Manual

Page
Table of Contents
I. Introduction
A. Systems Thinking Scale 2
B. Project summary 3

II. Development of Systems Thinking Scale

A. Overview 4
B. Item generation 5
1) Definition of systems thinking
2) Theoretical dimensions
C. History of development
1) Initial items 6
2) Field testing 7
D. Initial psychometric analyses (30 items)
1) Exploratory descriptive analysis 9
2) Principal Axis Factor Analysis 11

III. Systems Thinking Scale (STS) Psychometric Analysis (20 items)

A. Validity
1) Discriminate 12
2) Concurrent Criterion-Related 14
B. Reliability 15

IV. Scoring 16

V. Future work
A. Development of items for 17
2 additional subscales
B. Status of future subscales

VI. Contact Information and Permission for Use 19

VII. Acknowledgement 19
A. Suggested Citation for the STS 19

VIII. References 20

VIX. Appendices
A. The Systems Thinking Scale 25
B. Results of Factor Loadings 27

STS Manual Version 3‐3‐2011 Page 1

I. Introduction

Knowledge in an emerging field is advanced more rapidly when valid and reliable
measures of the concepts central to that field are available. Systems Thinking is viewed
as a key concept in the success of continuous quality improvement (CQI) initiatives, yet
no valid and reliable instrument exists to measure Systems Thinking. This project was
funded by the Robert Wood Johnson Foundation from 2008 – 2010 as part of their
initiative to Advance the Science of Quality Improvement Research and Evaluation
(ASQUIRE).
The goal of this project was to develop and conduct psychometric testing of the
Systems Thinking Scale (STS), a measure of systems thinking in quality improvement
work.++++++++++++++++++++++++++++
The development of a valid and reliable measure of Systems Thinking has the potential
to contribute to the science of CQI in several ways:
1) Increase our understanding of one of the mechanisms by which CQI processes
achieve their results;
2) Assist CQI scientists and practitioners to build models of predictors of successful
CQI projects;
3) Increase our understanding of the human social capital available on CQI teams; and
4) Improve our education efforts of future CQI team members.

SYSTEMS THINKING SCALE (See Appendix A)

The Systems Thinking Scale is a 20-item instrument that uses a 5 point likert-type
scale:

0= Never
1= Seldom
2= Some of the time
3= Often
4= Most of the Time

A total score is computed by summing up the responses for each item. Scores
can range from 0 to 80.

STS Manual Version 3‐3‐2011 Page 2

Project Summary

Purpose of the Study: The goal of this Robert Wood Johnson Foundation-funded project was to develop
and conduct psychometric testing of a measure of systems thinking - the Systems Thinking Scale (STS).

Methods:

First, using a series of electronic interactions, 10 international experts in systems thinking and process
improvement identified the conceptual domains of systems thinking and generated a provisional set of 30
instrument items. These items were then assessed for content validity by academic and practicing
healthcare professionals (n=18) and revisions to wording and response categories were made. Principal
axis factor analysis with oblique rotation was used to assess construct validity; this resulted in dropping 4
items, thus providing a final measure of 26 items for psychometric analyses. Next, a series of
psychometric analyses was conducted with healthcare professionals (N=342) and health professions
students engaged in quality improvement initiatives and education (N=224). Reliability and concurrent,
discriminate, and predictive validity of the STS were evaluated.

Results:

Factor analyses indicated three factors of the STS: 20 items loaded high on Factor 1 (System
Interdependencies); 2 items loaded on Factor 2 (Personal Effort), and 4 items loaded on Factor 3
(Reliance on Authority). Because of the low number of items on Factors 2 and 3 and negative correlations
among them, these items were omitted from further psychometric testing. Thus, reliability and validity
were assessed using only the 20-item one-factor instrument. Test-retest reliability assessment (n=36)
showed a correlation of .74; internal consistency testing (n=342) using Cronbach’s Alpha had a coefficient
of.89. Discriminate validity was tested with 3 groups of healthcare professions students (n= 102) who
received high, low or no dose levels of systems thinking education related to process improvement. There
were no differences in STS mean scores at pretest. At post test, the high dose systems thinking
education group scored significantly higher on the STS than both the low and no dose groups (p=.05 and
.01, respectively). Concurrent and predictive validity assessments were attempted, but the lack of any
current “gold standard” measures of systems thinking and CQI project success limited our ability to make
judgments about these types of validity.

Conclusions and Implications

The 20-item Systems Thinking Scale has good reliability and construct and discriminate validity. The STS
is now publically available for use and a website has been established to provide information on its
usefulness and encourage further refinement. The Systems Thinking Scale has the potential to contribute
to the science of CQI in several ways: (1) increase our understanding of one of the mechanisms (systems
thinking) by which CQI processes achieve their results; (2) increase our understanding of the human
social capital available on CQI teams; and (3) improve our education efforts of future CQI team members.

STS Manual Version 3‐3‐2011 Page 3

II. Development of the Systems Thinking Scale

A. Overview

The development of the Systems Thinking Scale was generated over the course of
three phases.

Phase I: Item generation and confirmation by expert panels

The conceptual domains of Systems Thinking were identified (see Figure 1) using a
grounded theory approach in a series of electronic mail focus groups of CQI system
experts. This method efficiently obtained the opinions from experts across the globe.
The discussion of the panel of experts was analyzed for themes representing Systems
Thinking domains and concept attributes. Questionnaire items were then developed
within each domain. The experts were asked to review potential items and validate that
they were appropriate indicators of the domains.

Phase II: Initial field testing for clarity and feasibility

Preliminary field testing for item clarity and feasibility of the instrument was conducted
by administering the instrument to health care professionals. A project team member
administered the instrument to the professional and recorded questions and comments
about clarity of items, grammar, syntax, organization, appropriateness, and logical flow.
The project team also recorded the time to complete the instrument, its acceptability to
participants, and ease of use.

Phase III: Psychometric testing of final 20 items

The third phase consisted of extensive psychometric testing and additional feasibility of
the Systems Thinking Scale. The instrument was administered to over 550 healthcare
professionals and healthcare profession students over a variety of disciplines and a
variety of delivery settings.

Following are details of each of these development phases.

STS Manual Version 3‐3‐2011 Page 4

B. Item Generation and Item Confirmation by Expert Panel

Our study team conducted an extensive literature search and completed 6 electronic
mail discussions with 11 content experts and received feedback on definition, domains
and test item bank (see Figures 1, 2 and 3) Experts reviewed the preliminary test item
bank to assure congruence of the items with the working definition and identified
dimensions of systems thinking.

Figure 1. Working definition of systems thinking and dimensions derived by expert panel

Working definition:

Systems thinking: The ability to recognize, understand, and synthesize the interactions, and
interdependencies in a set of components designed for a specific purpose. This includes the ability to
recognize patterns and repetitions in the interactions and an understanding of how actions and
components can reinforce or counteract each other. These relationships and patterns occur at different
dimensions: temporal, spatial, social, technical or cultural. It is fundamental to undertaking specific
methodology or strategies to explore and redesign a set of components comprising a whole.

Figure 2. Theoretical Dimensions of Systems Thinking

1) Sequence of events

2) Causal sequence

3) Multiple causations possible

4) Variation of different types (random/special)

5) Feedback

6) Interrelations of factors, patterns of relationships

STS Manual Version 3‐3‐2011 Page 5

Figure 3. Systems Thinking Scale initial item bank

1. I think the harder people work the better the outcomes will be.
2. I seek everyone’s view of the situation.
3. I look beyond a specific event to determine the cause of the problem.
4. I think understanding how the chain of events occur is crucial.
5. I include people in my work unit to find a solution.
6. I think outcomes are random.
7. I think that lasting change relies on personal effort and motivation.
8. I think recurring patterns are more important than any one specific event.
9. I think of the problem at hand as a series of connected issues.
10. I consider the cause and effect that is occurring in a situation.
11. I consider the relationships among co‐workers in the work unit.
12. I think that systems are constantly changing.
13. I propose solutions that affect the work environment, not specific individuals.
14. I focus on my first idea because it is often the best.
15. I keep in mind that proposed changes can affect the whole system.
16. The main reason for success is to get the person in charge to change.
17. I think my first impressions turn out to be very useful.
18. I think more than one or two people are needed to have success.
19. I keep the mission and purpose of the organization in mind.
20. I think small changes can produce important results.
21. I consider how multiple changes affect each other.
22. I focus primarily on the opinions of a champion in the system.
23. I think about how different employees might be affected by the improvement.
24. I think personal commitment is important in creating lasting change.
25. I try strategies that do not rely on people’s memory.
26. I recognize system problems are influenced by past events.
27. I think that system‐wide change is easy to accomplish.
28. I consider the past history and culture of the work unit.

29. I consider that the same action can have different effects over time, depending on the state of the
system.

30. I think uncertainty and surprise are involved.

STS Manual Version 3‐3‐2011 Page 6

C. Initial Field Testing

We field tested the STS with healthcare professionals (see Table 1 for sample
demographics). Participants commented on the logical flow, clarity of items,
organization, appropriateness and grammar. Minor changes were recommended (font
size, minor rewording, and the replacement of “believe” to “think”).The revised 30-item
scale then underwent initial psychometric testing.

Table 1. Demographic for healthcare professionals (N=10) who field tested the STS.

Type of participant N(%)

Administrator` 4(40)
Pharmacist 1(10)
Physician 2(20)
Nurses 3(30)

X(SD) Range
Years working in healthcare 16.9(10.2) 5‐34

Years working on QI project 6.0(6.4) 1‐23

STS Manual Version 3‐3‐2011 Page 7

D. Initial Psychometric Testing
The 30-item STS was initially tested using exploratory descriptive analysis and principal
axis factor analysis.

1) Exploratory Descriptive Analysis

The STS (30 items) was administered to 342 healthcare professionals (see Table 2 for
demographics of the sample). The mean item scores ranged from .76 (for item 7) to
3.46 (for item 4) and several items had means greater than 3.0. There were several
items that were problematic: 1, 6, 7, 16, 17, 22, 27, and 30. Four items (6, 24, 27, and
30) were excluded from all further analyses based on poor item-total correlations or
poor item wording. The remaining 26 items were included in a series of exploratory
factor analyses.

Table 2 Descriptive statistics of health care professionals N=342

N (%)
Occupation Administration/Management 58(17)
Dietician 3(.9)
Staff Nurse 113(33)
Advanced Practice Nurse 20(5.8)
Pharmacist 4(1.1)
Therapist 17(4.9)
Physician 89(26.0)
Research/ Analyst 5(1.5)
Technician 12(3.5)
Missing 21(6.1)

Department Bone Marrow Transplant 10(3.1)

Cardiology 16(4.9)
Critical Care 59(18.2)
General Medical/Surgical Unit 98(30.2)
Emergency Room 2(.6)
Quality Department 15(4.6)
Family Medicine 13(4.0)
Pediatrics 18(5.5)
Pharmacy 1(.3)
Rehabilitation 21(6.6)
Surgery 9(2.8)
Other 55(16.9)
Missing 8(2.5)

STS Manual Version 3‐3‐2011 Page 8

Table 2 (Continued)
Descriptive statistics of health care professionals N=342
N(%) X (SD)
Years working in Healthcare 15.6 (11.3)
0‐5 years 90(26.3)
6‐15 years 99(28.9)
16 or more years 142(41.5)

Number of years worked on 6.6(8.5)

quality improvement
Never 17(5.0)
1 year or less 119(34.8)
2 to 13 years 132(38.6)
14 or more years 59(17.3)

STS Manual Version 3‐3‐2011 Page 9

 2) Principal Axis Factor Analysis

Construct validity was performed on the 26-item scale (See Table 1 above for sample
description (Healthcare Professionals, N=342). Construct validity is the degree to which
a measure performs in accordance with theoretical expectations and is the best way to
assess validity for abstract constructs (Carmines & Zeller, 1979). Construct validity was
assessed using Principal Axis (PA)Factor Analysis. PA Factor Analysis was used to
identify domains represented in this measure and further refine the set of items that
comprise the STS.

Principal axis factor analysis with oblique rotation was conducted for differing numbers
of extracted factors, with a 3-factor solution appearing most useful. Twenty items
loaded high and cleanly on factor 1 (ranging from .35 to .69). This factor is interpreted
as Interdependencies of system components. Two items (1 and 7) loaded cleanly on a
separate factor (.67 and .44) interpreted as Personal Effort Four items (14, 16, 17, and
22) loaded on a distinct 3rd factor, Reliance on authority though the loadings were not as
high as for other items (.53, .40, .44, .30). Factors were distinct in that they did not
correlate strongly with each other. Factors 1 and 2, however, correlated negatively (-
.21) (see Appendix B).

We ran preliminary reliability analyses and the coefficient alpha for the 20-item subscale
corresponding to factor 1 was .89. Coefficient alpha for the 2-item subscale
corresponding to factor 2 was .55: this was reasonable given that it is a 2-item scale.
Coefficient alpha for the 4-item subscale corresponding to factor 3 was .54, relatively
low for a 4-item scale.

Based on these reliability analyses and factor analyses, the 20 items of what is referred
to as factor 1 (system interdependencies) appeared to be the strongest performing
items of the intended construct of systems thinking. The theory behind the items that
loaded on what are referred to as factors 2 and 3 may be important, but these items, as
worded and based on item responses from this particular sample, do not function well
as part of a single overall measure of systems thinking. Experts on systems thinking
seem to agree that the concepts represented by the content in the items of these other
two factors are important. Thus, we have recently generated additional items for further
evaluation and they are undergoing analyses.

NOTE:
The remaining analyses were performed using the 20-item STS(Appendix A).

STS Manual Version 3‐3‐2011 Page 10

III. Systems Thinking Scale (STS) 20 items

A. Validity
1) Discriminate Validity

Discriminate validity using the 20-item STS was tested. Discriminate validity or known-
group difference validity, is confirmed when scores on an instrument are able to
discriminate among groups (Carmines & Zeller, 1979). The known groups in this
situation are a group of students who received systems thinking education during a 2-
hour course in the medical school (Low dose; N=78) and a group of students in a15-
week quality improvement course that emphasized systems thinking (High dose; N=11)
and a general health professionals course (nursing pharmacology N=32) who received
no education specific to systems thinking (See Table 3 for description of the sample).
The STS was administered in a pretest-posttest format and validity was assessed by
calculating the difference in change between the two groups. Discriminate validity was
supported as the post test significantly increased in the high dose group only (see table
4 and 5). RESULTS: Discriminate validity was supported as the post test significantly
increased in the high dose group only and the ANOVA was significant for the
comparison between the high and no dose groups (see table 4 and 5).
Table 3. Demographics of student participants.

N(%)
Type of student
Health Professions 1(.4)
Medical 184(82.1)
Nursing 27(12.2)
Public Health 11(4.9)
Missing 1(.4)
Level of education
Undergraduate 3(1.3)
Graduate 214(95.5)
Post‐Graduate 2(.9)
Missing 5(2.2)
Student healthcare experience
Yes 123(54.9)
no 99(44.2)
Number of years in healthcare
1 to 2 years 64(28.6)
3 or more years 54(24.1)
Prior involvement working on QI
Yes 59(26.3)
No 158(70.5)

STS Manual Version 3‐3‐2011 Page 11

Table 4. Results for Discriminate Validity. System Thinking Scale results for students receiving 3
levels of systems thinking education.

Sample Intervention Dose Pre intervention Post‐intervention

M (SD) M(SD)
Graduate entry pre‐ No intervention
licensure nursing 61.0 (7.1) 57.3 (5.5)
N=32
1st year Medical Students Low dose intervention 2
N=78 hour class on an error case 56.1 (7.4) 54.9 (8.6)
with 2 hour case study
work
Graduate Participated in a semester
interprofessional students course on QI that 60.1(8.3) 62.0(7.4)
N= 11 emphasized systems
thinking
15 3‐hour classes

Table 5. ANOVA results for differences among high, low and no dose of systems thinking
education.
Ttype of Mean Std. Error Sig. a 95% Confidence Interval
survey Difference for Difference
Lower Upper
bound Bound
Low dose High dose ‐2.865 1.486 .05 ‐5.814 .084

No dose 2.077 1.390 .14 ‐.681 4.835

High dose Low dose 2.865 1.486 .07 ‐.084 5.814

No dose 4.942* 1.856 .01 1.259 8.624

No dose Low dose ‐2.077 1.390 .14 ‐4.835 .681

High dose ‐4.942* 1.856 .01 ‐8.624 ‐1.259

Based on estimated marginal means

a. Adjustment for multiple comparisons: Least significant difference (equivalent to no
adjustments).
*. The mean difference is significant at the .05 level.

STS Manual Version 3‐3‐2011 Page 12

 2) Concurrent Criterion‐Related Validity

Concurrent criterion-related validity is confirmed when scores on an instrument are

correlated to a related criterion at the same point in time (Carmines & Zeller, 1979).

We sought out to find an instrument that measures continuous quality improvement

(CQI) proficiency or systems thinking. We could not find an instrument that measured
systems thinking so we choose the only measure identified that measures CQI
proficiency- the QI Knowledge Application Tool (QIKAT). The QIKAT is a standardized
written instrument to assess learner application of CQI knowledge and the integration of
CQI concepts. The QIKAT consists of 3 scenarios that describe quality improvement
opportunities. Participants are asked to answer a set of questions in response to the
scenarios, some of which assess system/process thinking ability. The QIKAT has
demonstrate inter-rater reliability (r=.55)and construct validity (Morrison et al., 2003).
For our analysis, during the first week of an interdisciplinary CQI course students
(N=11) were administered the STS and the QIKAT (see Table 6).

As a second concurrent-criterion-related validity test, we administered the STS to 1st

year medical students (N=78)who received four hours of systems thinking education
related to an error case during their block one studies. Tests of correlations of the
scores of the STS and the summative evaluation questionnaire (SEQ) (item # 3 that
dealt with systems thinking integration) scores were done.

RESULTS: Low correlations were found for both the QiKAT and SEQ. These low
correlations could be due to a lack of a good validated measure of CQI proficiency and
the subjectivity in the measurement of both the QiKAT and SEQ.

Table 6. Correlations between scores on Systems Thinking Scale (STS) and other indirect
measures of systems thinking.
Other measure of systems Pearson Correlation
thinking
Graduate interprofessional QI Knowledge Application tool .46
students (QiKAT)*
N= 11

1st year Medical Students SEQ Test item .28

N=78

*QIKAT assesses learner application of QI knowledge and the integration of QI concepts. The
QIKAT consists of three scenarios that describe QI opportunities. Participants are asked to
answer a set of questions in response to the scenarios, some of which assess system/process
thinking ability.

STS Manual Version 3‐3‐2011 Page 13

B. Reliability

Reliability Testing is done to ascertain if an instrument consistently measures an

attribute (Nunnally & Bernstein, 1994). Two forms of reliability, internal consistency and
test-retest, were obtained.

1) Internal Consistency Reliability

Internal consistency reliability was assessed using the test statistic Cronbach’s alpha
coefficient. This coefficient measures the internal consistency of a set of items on a
measure. For these analyses, the sample of health professionals and health professions
students was used (N=342). Refer to Tables 2 and 3 for demographics of this sample.
RESULT: The internal consistency of the STS (20-item) on this sample was found to be
.82.

2) Test‐Retest Reliability

A second type of reliability, test-retest, was assessed in which 36 participants were

administered the Systems Thinking Scale at a two-week interval. See Table 7 for the
demographics of the sample for this assessment.
RESULT: The STS had adequate test-retest reliability (Carmines & Zeller, 1979); the
correlation between the pre and post test was .74. Mean scores are listed in Table 8.

Table 7 Sample demographics for Test‐Retest N=36

N
Type of Healthcare Staff nurses 6
professional Advanced practice 4
Administrators 3
Physicians 2
Quality improvement professionals 2
Health professions students 19
Years in health care Range X= 24.6
Healthcare
professional
Years in Quality Range X=10.6
Healthcare
professional
Experience working in Range 1‐4 years 50%
healthcare
Students
Experience working on 20%
QI projects

STS Manual Version 3‐3‐2011 Page 14

Table 8 Mean score for test/retest sample
n Mean Standard Deviation

STS Pretest 26 61.43 6.51

STS Posttest 26 59.08 5.70

P < .001

STS Manual Version 3‐3‐2011 Page 15

IV. Scoring

Systems Thinking Scale (20 Items; See Appendix A)

Scoring
Responses to the items on the Systems Thinking Scale are provided by users on a
likert-type scale:
0=Never, 1=Seldom, 2=Some of the time, 3=Often, and 4=Most of the time.

Item scores are summed to provide a STS score. Scores can range from 0 – 80. There
are no reverse coded items. Below are the mean scores and ranges obtained on the
STS for varies categories of healthcare professionals (see Table 9).

Table 9. Mean Scores for different healthcare professionals.

Type of population Sample size Mean(SD) Range

(N)

Health care Providers 225 61.1(7.9) 40‐79

Health care Providers with 175 61.8(8.0) 40‐79

>5 years experience

Health care Providers with 121 62.8(7.6) 45‐79

>=5 years experience in QI

Quality Improvement 19 68.7(6.9) 54‐79

specialists

All students 169 56.4(7.7) 30‐76

Medical Students 82 56.1(7.5) 42‐72

Nursing Students 15 61.0(7.4) 48‐76

Public Health Students 8 63.1(6.3) 56‐72

STS Manual Version 3‐3‐2011 Page 16

V. Future Work

We have recently developed new items for the two subscales of the STS that did not
perform well on our initial psychometric assessment: Personal effort and Reliance on
Authority. Tables 10 and 11display the new items for these two subscales. We are
currently testing the STS with these new items included.

Table 10. Preliminary new items for the subscales Personal effort

New items: Personal Effort

1. I concentrate on the effort people put into their work.

2. I think that lasting change relies on personal effort and motivation.

3. I think people who do not get their desired results/outcomes did not work hard enough.

4. I think hard work and motivation are secondary to work process.

5. I think it is not how hard you work, but how smart you work.

6. I think how the system is organized is important to influence change than personal efforts by
individuals.

7. I think that success depends on making changes beyond those of one person.

STS Manual Version 3‐3‐2011 Page 17

Table 11. Preliminary new items for the subscale Reliance on Authority

New items: Reliance on Authority

1.. I focus on my first idea because it is often the best.

2. The main reason for success is changing the person in charge.

3. I think my first impressions turn out to be very useful.

4. I focus primarily on the opinions of a champion in the system.

5. I believe initial reactions or solutions to the problem are the best.

6. I focus on the opinion of stakeholders first.

7. I think the leaders of the organization have the best ideas.

8. I think a bottom‐up strategy is usually best.

9. I think people closest to the everyday work of the job should be heavily involved.

10. It is more important to think of system changes rather than what the leader wants.

STS Manual Version 3‐3‐2011 Page 18

VI. Contact Information Permission to use

For further information and permission to use the Systems Thinking Scale, you can
contact Drs. Mary Dolansky ([Link]@[Link]) or Shirley Moore
([Link]@[Link])

Or

Visit our website at [Link]/SystemsThinking

VII. Acknowledgement

Suggested Citation for the STS

Moore, S. M., Dolansky, M. A., Singh, M., Palmieri, P., Alemi, F. (2010). The Systems
Thinking Scale. Unpublished manuscript.

STS Manual Version 3‐3‐2011 Page 19

VIII. References

A Framework for a Systems Approach to health Care Delivery (2005). In [Link], W. D. Compton, J. H.
Grossman, & G. Fanjiang (Eds.), Building A Better Delivery System (pp. 19‐25). Washington D.C.:
The National Academies Press.

Tools for Complex Reasoning (2008). website [On‐line]. Available:

[Link]

Ackoff, R. (1974). In General. In Redesigning the Future (pp. 3‐53). John Wiley & Sons Inc.

Ackoff, R. L. (1974). Redesigning the Future. John Wiley & Sons.

Alemi, F., Safaie, F. K., & Neuhauser, D. (2001). A survey of 92 quality improvement projects.
[Link] [Link]., 27, 619‐632.

Alemi, F., Pawloski, L., & Fallon, W. F., Jr. (2003). System thinking in a personal context to
improve eating behaviors. [Link] Qual., 25, 20‐25.

Anderson, R., Crabtree, B. F., Steele, D. J., & McDaniel., R. R. (2005). Case Study Research: The View from
Complexity Science. Quality Health Research, 15, 669‐685.

Anderson, R. A., Issel, L. M., & niel Jr, R. R. (2003). Nursing homes as complex adaptive systems:
relationship between management practice and resident outcomes. [Link]., 52, 12‐21.

Bartels, J. E. & Bednash, G. (2005). Answering the call for quality nursing care and patient safety:
a new model for nursing education. [Link] Q., 29, 5‐13.

Begun, J. W., Zimmerman, B., & Dooley, K. (2003). Health Care Organizations as Complex Adaptive
Systems. In S. M. Mick & M. Wyttenbach (Eds.), Advances in Health Care Organization Theory
(pp. 253‐288). San Francisco: Jossey‐Bass.

Best, A., Moor, G., Holmes, B., Clark, P. I., Bruce, T., Leischow, S. et al. (2003). Health promotion
dissemination and systems thinking: towards an integrative model. [Link] Behav., 27
Suppl 3, S206‐S216.

Boonyasai, R. T., Windish, D. M., Chakraborti, C., Feldman, L. S., Rubin, H. R., & Bass, E. B. (2007).
Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA, 298,
1023‐1037.

Brown, C. A. (2006). The application of complex adaptive systems theory to clinical practice in
rehabilitation. [Link]., 28, 587‐593.

STS Manual Version 3‐3‐2011 Page 20

Capra, F. (1982). The Systems View of Life. In Simon & Schuster (Eds.), Turning Point ( Bantam
Paperback.

Casey, S. (1998). Set Phasers on Stun and other true tales of design, technology, and Human Error. (2
ed.).

Davidz, H. & Nightingale, D. (7 A.D.). Enabling systems thinking to accelerate the development of
senior systems engineers. Systems Engineering, 11, 1‐14.

DeVon, H. A., Block, M. E., Moyle‐Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J. et al. (2007). A
psychometric toolbox for testing validity and reliability. [Link]., 39, 155‐164.

Frank, M. (2002). What is "engineering systems thinking"? Kybernetes, 31, 1350‐1360.

Fraser, S. (2007). Understanding the Elephant; Why good practice doesn't spread in healthcare. Lulu.

Gardner, R. A. (2004). In The Process‐Focused Organization: A Transition Strategy for Success (pp. 28‐
31). ASQ Quality Press.

Gluck, P. A. (2008). Medical error theory. [Link] Am., 35, 11‐7, vii.

Gustafson, D. H., Sainfort, F., Eichler, M., Adams, L., Bisognano, M., & Steudel, H. (2003).
Developing and testing a model to predict outcomes of organizational change. Health
[Link]., 38, 751‐776.

Holden, L. M. (2005). Complex adaptive systems: concept analysis. [Link]., 52, 651‐657.

Kalim, K., Carson, E., & Cramp, D. (2006). An Illustration of Whole Systems Thinking. Health Services
Management Research, 19, 174‐185.

Ladden, M. D., Bednash, G., Stevens, D. P., & Moore, G. T. (2006). Educating interprofessional
learners for quality, safety and systems improvement. [Link], 20, 497‐505.

Leischow, S. J. & Milstein, B. (2006). Systems thinking and modeling for public health practice.
[Link] Health, 96, 403‐405.

Leischow, S. J., Best, A., Trochim, W. M., Clark, P. I., Gallagher, R. S., Marcus, S. E. et al. (2008).
Systems thinking to improve the public's health. [Link]., 35, S196‐S203.

Lepley, C. J. (1999). Affinity maps and relationship diagrams: two tools to enhance performance
improvement. [Link] Care Qual., 13, 75‐83.

Leykum, L. K., Pugh, J., Lawrence, V., Parchman, M., Noel, P. H., Cornell, J. et al. (2007). Organizational
interventions employing principles of complexity science have improved outcomes for patients
with Type II diabetes. [Link]., 2, 28.

STS Manual Version 3‐3‐2011 Page 21

Lipsitz, L. A. (2008). Dynamic models for the study of frailty. [Link] Dev., 129, 675‐676.

Madigosky, W. S., Headrick, L. A., Nelson, K., Cox, K. R., & Anderson, T. (2006). Changing and sustaining
medical students' knowledge, skills, and attitudes about patient safety and medical fallibility.
[Link]., 81, 94‐101.

March, J., Schon, D., Bennis, W., Staw, B., Vaill, P., & Sergiovanni, T. (2009). Reconceptions of
Leadership Theory and Practice. In [Link] & J. Corbally (Eds.), Leadership and
Organizational Culture: New Perspectives on Administrative Theory and Practice (pp. 18‐14).
Urbana and Chicago: University of Illinois Press.

Marck, P. (2005). Theorizing about systems: an ecological task for patient safety research.
[Link] Res., 14, 103‐108.

Matthews, J. I. & Thomas, P. T. (2007). Managing clinical failure: a complex adaptive system
perspective. [Link] Care [Link]., 20, 184‐194.

McNamara, C. (2009). website [On‐line]. Available:

[Link]

Mulej, M., Bastic, M., Belak, J., Knez‐Riedl, J., Pivka, M., Potocan, V. et al. (2003). Informal
Systems Thinking or Systems Theory. Cybernetics and Systems, 34, 71‐92.

Nagykaldi, Z., Mold, J. W., Robinson, A., Niebauer, L., & Ford, A. (2006). Practice facilitators and
practice‐based research networks. [Link] [Link]., 19, 506‐510.

Newhouse, R. P., Pettit, J. C., Poe, S., & Rocco, L. (2006). The slippery slope: differentiating
between quality improvement and research. [Link], 36, 211‐219.

Nieva, V. F. & Sorra, J. (2003). Safety culture assessment: a tool for improving patient safety in
healthcare organizations. [Link] Health Care, 12 Suppl 2, ii17‐ii23.

Peters, A. S., Kimura, J., Ladden, M. D., March, E., & Moore, G. T. (2008). A self‐instructional model
to teach systems‐based practice and practice‐based learning and improvement.
[Link]., 23, 931‐936.

Rubenstein, L. V., Mittman, B. S., Yano, E. M., & Mulrow, C. D. (2000). From understanding health
care provider behavior to improving health care: the QUERI framework for quality
improvement. Quality Enhancement Research Initiative. [Link], 38, I129‐I141.

Schnall, R., Stone, P., Currie, L., Desjardins, K., John, R. M., & Bakken, S. (2008). Development of a self‐
report instrument to measure patient safety attitudes, skills, and knowledge.
[Link]., 40, 391‐394.

Schyve, P. M. (2009). Systems Thinking and Patient Safety. Advance in Patient Safety [On‐line].
Available: [Link]

STS Manual Version 3‐3‐2011 Page 22

Senge, P. '. R. R., Smith, B., Roberts, C., & Kleiner, A. (1994). The Fifth Discipline‐ Fieldbook.
Doubleday.

Senge, P. M. (2006). The Fifth Discipline ‐ The Art & Practice of the Learning Organizations.
Doubleday.

Senge, P. (1990). The Fifth Discipline. Double Day.

Stroebel, C. K., McDaniel, R. R., Jr., Crabtree, B. F., Miller, W. L., Nutting, P. A., & Stange, K. C. (2005).
How complexity science can inform a reflective process for improvement in primary care
practices. [Link] [Link] Saf, 31, 438‐446.

Terpstra, J. & Leykum, L. K. (7‐7‐2009). Systems Thinking for Implementation Research &
Practice. VQ QUERI Series. Ref Type: Generic

Tesch, T. & Levy, A. (2008). Measuring service line success: the new model for benchmarking.
[Link]., 62, 68‐74.

Trochim, W. & Kane, M. (2005). Concept mapping: an introduction to structured

conceptualization in health care. [Link] Care, 17, 187‐191.

Trochim, W. M., Cabrera, D. A., Milstein, B., Gallagher, R. S., & Leischow, S. J. (2006). Practical challenges
of systems thinking and modeling in public health. [Link] Health, 96, 538‐546.

Vaill, P. (1989). The Grand Paradox of Management. In Managing as a Performing Art: New Ideas
for a World of Chaotic Change (pp. 77‐86). Jossey‐Bass.

Vaill, P. (1996). An Ordinary Day on the River. In Learning as a Way of Being (pp. 1‐21). San
Francisco, CA: Jossey‐Bass.

Vaill, P. (1996). Systems Learning. In Learning as a Way of Being (pp. 103‐119). San Francisco, CA:
Jossey‐Bass.

Vaill, P. (2009). The Peak Performance Cult. In Managing as a Performing Art: New Ideas for a World
of Chaotic Change (pp. 63‐76). Jossey‐Bass.

Vaill, P. (1989). The Peak Performance Cult. In Managing as a Performing Art (pp. 63‐76). Jossey‐
Bass.

Vaill, P. (1996). Learning as a way of being: strategies for survival in a world of permanent white water.
Jossey‐Bass.

Vaill, P. B. (1986). The Purpose of high‐performing Systems. In [Link] & J. E. Corbally (Eds.),
Leadership and organizational Culture; New Perspectives on Administrative Theory and
Practice (pp. 85‐104). University of Illinois Press.

Vojtko, V. (2009). website [On‐line]. Available: [Link]

STS Manual Version 3‐3‐2011 Page 23
Wagner, T., Kegan, R., Lahey, L., Lemons, R., Garnier, J., Helsing, D. et al. (2005). Thinking Systemically. In
Change Leadership: A Practicle Guide to Transforming our Schools (pp. 97‐130). Jossey‐Bass.

Wagner, T., Kegan, R., Lahey, L., Lemons, R. W., Garnier, J., Helsing, D. et al. (2005). Relating the parts to
the whole. In Change Leadership: A practical Guide to Transforming Our Schools (pp. 95‐130).
Jossey‐Bass.

Wolstenholme, E. (1993). A Case Study in Community Care Using Systems Thinking. Journal of
Operational Research Society, 44, 925‐934.

Wolstenholme, E. (1995). Organizational re‐engineering using systems modelling: rediscovering the

physics of the health service. IMA [Link]., 12, 283‐296.

Wong, M. C., Turner, P., & Yee, K. C. (2008). Involving clinicians in the development of an
electronic clinical handover system ‐ thinking systems not just technology. [Link]
[Link]., 136, 490‐495.

Ziegelstein, R. C. & Fiebach, N. H. (2004). "The mirror" and "the village": a new method for teaching
practice‐based learning and improvement and systems‐based practice. [Link]., 79, 83‐88.

STS Manual Version 3‐3‐2011 Page 24

 Appendix A
20 item Systems Thinking Scale

Making Improvement

Instructions:

Please read each of the statements and place an “x” in the answer box that indicates frequency
of agreement with the statement:

When I want to make an improvement. . . Never Seldom Some of Often Most of the
the time time

1. I seek everyone’s view of the situation.

2. I look beyond a specific event to
determine the cause of the problem.

3. I think understanding how the chain of

events occur is crucial.

4. I include people in my work unit to find a

solution.

5. I think recurring patterns are more

important than any one specific event.

6. I think of the problem at hand as a series

of connected issues.

7. I consider the cause and effect that is

occurring in a situation.

8. I consider the relationships among co-

workers in the work unit.

9. I think that systems are constantly

changing.
10. I propose solutions that affect the work
environment, not specific individuals.

STS Manual Version 3‐3‐2011 Page 25

 When I want to make an Never Seldom Some of Often Most of the
improvement. . . the time time

11. I keep in mind that proposed

changes can affect the whole system.

12. I think more than one or two

people are needed to have success.

13. I keep the mission and purpose of

the organization in mind.

14. I think small changes can produce

important results.

15. I consider how multiple changes

affect each other.

16. I think about how different

employees might be affected by the
improvement.

17. I try strategies that do not rely on

people’s memory.

18. I recognize system problems are

influenced by past events.

19. I consider the past history and

culture of the work unit.

20. I consider that the same action can

have different effects over time,
depending on the state of the system.

STS Manual Version 3‐3‐2011 Page 26

 Appendix B
Table Principal Axis Factor Analysis: Factors, Eigen values, and Percentages of Variance
Factor Loading
Item 1 2 3
I consider the past history and culture of the work unit .69 ‐.03 .14
I consider how multiple changes affect each other .66 ‐.06 ‐.05
I think about how different employees might be affected by the .65 ‐.01 .06
improvement
I consider that the same action can have different effects over time, .64 ‐.14 .08
depending on the state of the system
I keep in mind that proposed changes can affect the whole system .63 ‐.06 ‐.02
I consider the cause and effect that is occurring in a situation .62 .12 ‐.22
I keep the mission and purpose of the organization in mind .61 .02 .02
I propose solutions that affect the work environment, not specific .57 .10 .15
individuals
I look beyond a specific event to determine the cause of the problem .54 .15 ‐.24
I recognize system problems are influenced by past events .53 ‐.09 .01
I think small changes can produce important results .52 ‐.20 ‐.06
I consider the relationships among coworkers in the work unit .50 .02 ‐.09
I include people in my work unit to find a solution .50 .31 ‐.07
I try strategies that do not rely on people’s memory .48 ‐.02 .18
I think of the problem at hand as a series of connected issues .45 .10 ‐.19
I think understanding how the chain of events occur is crucial .45 .18 ‐.21
I seek everyone’s view of the situation .42 .13 ‐.22
I think more than one or two people are needed to have success .39 .02 .01
I think that systems are constantly changing .36 ‐.07 .03
I think recurring patterns are more important than any one specific event .35 .05 ‐.16
I focus on my first idea because it is often the best .13 .53 .08
I think my first impressions turn out to be very useful ‐.01 .44 .12
The main reason for success is to get the person in charge to change .01 .40 .06
I focus primarily on the opinions of a champion in the system ‐.15 .30 ‐.12
I think the harder people work the better the outcomes will be .04 .15 .67
I think that lasting change relies on personal effort and motivation ‐.26 .20 .44
Eigenvvalues 6.8 1.8 1.5

% of Variance 26.2 7.1 5.9

STS Manual Version 3‐3‐2011 Page 27