International Conference on Naturalistic Decision Making 2015, McLean, VA

How Military Intelligence Personnel Collaborate on a

Sense-Making Exercise
Chris BABERa1, Gareth CONWAYb, Simon ATTFIELDc, Chris ROONEYc, Neesha
KODAGODAc and Rick WALKERc
a
School of Electronic, Electrical and Systems Engineering, University of Birmingham
b
Human Systems Group, DSTL
c
School of Science and Technology, Middlesex University

ABSTRACT
Sense-making plays an important role in Intelligence Analysis, but can be difficult to study in situ.
Thus, it is common to use training exercises to study this phenomenon. In this paper, an exercise
was undertaken by Military Intelligence personnel. The behaviour of groups of analysts is
considered in terms the Data / Frame Model of Sense-making. The paper illustrates how
Intelligence Analysis need not follow a linear process but often involves parallel and overlapping
explorations of data, with multiple frames that might be minimal and sketchy. The use of
representations, such as link diagrams, provides a means of externalizing frames and it is
suggested that this shifts reasoning from deductive reasoning to abductive reasoning as the
exercise progresses.

KEYWORDS
Sense-making; Intelligence Analysis; Representations.

INTRODUCTION
While it is unlikely that there is a single, definitive way of ‘doing’ Intelligence Analysis, there are generic
descriptions of how Intelligence Analysis could be performed. For instance, NATO (2008) describes the
Intelligence (or Analysis) Cycle in terms of four phases:

 Direction: define objectives for Intelligence Requirements and Requests for Information;
 Collection: gather information by agents or assets;
 Processing: compile and interpret information to produce intelligence product;
 Dissemination: distribute appropriate parts of the intelligence to relevant parties.

Although this implies a flow from collection to dissemination, alternative descriptions emphasize the recursive
nature of the analysis process. For example, Elm et al. (2005) define this process in terms of ‘down-collect’
(sample from the available data for material deemed to be ‘on analysis’), ‘conflict and corroboration’ (ensure
accurate and robust interpretation of findings, and modify the ‘down-collect’ accordingly), and ‘hypothesis
exploration’ (construct coherent narrative to explain the findings, and reflect this narrative back to the ‘conflict
and corroboration’ activity). This recursion means that Intelligence Analysis is not linear (Heuer, 1999; Heuer
and Pherson, 2010; Roth et al., 2010; Kang and Stasko, 2011). Such recursion is neatly captured by the Data /
Frame model of sense-making model (Klein, Moon and Hoffman., 2006a, b).

Data / Frame Model of Sense-making

Central to sense-making in the Data / Frame model is the relationship between the data to which the analyst has
access and the different ‘frames’ that can be used to interpret, make sense of, or explain, these data. Klein et al
(2006a) point out that, “When people try to make sense of events, they begin with some perspective, viewpoint,
or framework – however minimal. For now, let’s use a metaphor and call this a frame.” (p. 88, emphasis
added).

1Corresponding Author: [Link]@[Link]; 0044 121 414 3965 ; School of Electronic, Electrical and
Systems Engineering, University of Birmingham, Birmingham. UK B15 2TT
 Baber, C et al. Intelligence Analysis and Sense-making

Most crucial of all to the Data / Frame model is the suggestion that the relationship between data and frame is
both reciprocal and parallel. In other words, a frame could be applied to a set of the data, or a set of the data
could suggest a frame. This reciprocity points to the continuous interweaving of activities of exploring data and
generating interpretations. What is particularly useful about the notion of a frame is that it need not imply a
‘solution’ or final ‘product’ but can serve as a temporary explanatory model of aspects of the data. This accords
with the suggestion from Kang and Stasko (2014) that analysis involves the interleaving of processes directed at
constructing conceptual models, collecting data, analysing data and producing key findings to share with other
people.

While the Intelligence Cycle might begin with ‘Direction’, this only gives a high-level sense of what the analyst
might be looking for. As ‘Collection’ and ‘Processing’ progresses, new problem opportunities arise through
‘discovery-led refinement’ (Attfield and Blandford, 2010). Having said this, at a tactical level, these are usually
considered separate activities, e.g., collection via an ISTAR asset overseas could be completed before the analyst
can see the results. Indeed, processing at the collection stage (in terms of ‘cleaning’ of data or in terms of
checking for potentially useful data could occur at a low level and then result in material which is passed on to
the analyst. Thus, one could read the Data / Frame model in terms of a ‘Direction’ providing a tightly specified
frame (so that the analyst will only collect and process data which are directly relevant to this frame), or in terms
of a familiar problem (so the frame could be based on previous experience of similar cases), or in terms of a
problem opportunity (so combinations of data would suggest particular frames which could be expanded and
explained). The point at issue is not how people use frames but how they define them in the first place (Roth et
al., 2010).

METHOD
The study reported in this paper used an exercise developed for a Visual Analytics Summer School 2012 and
reported at a previous NDM conference (Baber et al., 2013). Initial analysis was derived from ad hoc observation
of group performance and it was felt that a more controlled approach to data collection would be beneficial. This
paper presents the approach to data collection and analysis that was developed to study this Exercise.

Objective
The Exercise was designed with the assumption that the correct solution (as detailed in Appendix One) could be
arrived at by defining an accurate modus operandi (M.O.) of how a gang in the scenario operated. The M.O. was
as follows: a gang uses a yacht to transport drugs from Roskoff (France) to a marina in Exmouth (UK)2. The
yacht also carries a passenger who puts the drugs into a van hired by the marina’s management and drives to a
warehouse in Leeds (UK). The drugs are then distributed to drivers in a mini-cab company in Leeds and sold. In
order to make the exercise challenging, the data also relate to three other ‘stories’.

Procedure
The University of Birmingham ethics protocol was followed (i.e., participants were free to withdraw at any time
and all data collected (including images and video) would be anonymized before reporting). Participants were
given a briefing, which was intended to simulate the ‘Direction’ phase of the Intelligence Cycle, and the
Exercise concluded with a presentation by each group, which was intended to simulate the ‘Dissemination’
phase of the Intelligence Cycle. The briefing is given in Appendix Two.

Following the briefing, participants were allocated to groups of 4-6 members 3 and then taken to their own
incident rooms to complete their investigation. These rooms were equipped with whiteboards, large notepads,
pens, post-it notes and paper. Each group was provided with a pack of 49 slides. The pack included nine suspect
cards (with picture of the suspect and their correct addresses), together with a combination of telephone logs,
harbourmaster logs, maps, business accounts, witness and arrest statements, newspaper articles etc. Figure 1
provides an illustration of the types of information supplied4. Each sheet contained several topics, e.g., dates,
phone numbers, names, locations etc.

2 We should make very clear that the place names Exmouth, Leeds and Roskoff were included in an entirely fictional

capacity and that there is no implication that any of these towns, or indeed Exmouth Marina, have been involved or
implicated in any of the events in the Exercise.
3 Contemporary approaches to Intelligence Analysis often rely on groups of people working together in ‘Fusion’ centers

(Roberts, 2011; Treverton and Gabbard, 2008). For example, the US Army All-Source Analysis System (ASAS) involves
four analysts working together to provide data for a senior analyst. We took this as a template for our study and had people
working in teams of 4-6 people.
4 A complete pack of materials can be obtained from the lead author on request.

2
 Baber, C et al. Intelligence Analysis and Sense-making

Figure 1 : Information used in the Exercise

Participants
The study involved a workshop with serving British Army Intelligence Analysts, as part of a weekly Intelligence
Analysis programme. Sixteen officers and soldiers agreed to be observed during the investigative study. The
participants were divided into three groups, with two groups of five and one group of six. Five participants were
female and the remainder were male.

Data Collection and Analysis

The analysis involved three forms of data. Each group had a dedicated observer who recorded the activity on the
group on an activity sampling sheet every 10 minutes. Each observer, when they were not completing the
sampling sheet, took photographs of the diagrams that the groups were making, or of group activity, and made
contemporaneous notes of the group discussions. At the end of the exercise, each group presented its findings.

RESULTS AND ANALYSIS

In these data, counts of activity (related to number of topics discussed, number of actions performed etc. during
the sampling window) are presented in the form of graphs to provide a convenient means of comparing groups.
Deeper analysis is provided in the form of extracts from qualitative analysis of specific events in the exercise.

Activity Sampling Results

The information sheets contained information which can be classified in terms of Suspect, Date / Time,
Locations (Exeter, Roskoff, Leeds, Exmouth), Vehicle (yacht, van) Action (payment, social activity, business,
crime). This classification defines the set of topics that groups discussed. In the activity sampling, each mention
of a topic was counted in the sampling period. Thus, if the group said ‘Condiere owns the yacht called Sunny
Jim’, we would count 1 for ‘suspect – Condiere’ and 1 for ‘vehicle – yacht’.
3
 Baber, C et al. Intelligence Analysis and Sense-making

12

Number of Topics being discussed

10

6 Group A

4 Group B
Group C
2

0
10 20 30 40 50 60 70 80 90 100 120 130
Time into Exercise (mins)

Figrue 2 : Number of topics discussed over time

Figure 2 plots the number of topics mentioned at each sampling period. Overall, the average number of topics is
consistent across the groups, i.e., A = 6 (± 3), B = 5 (± 4) and C = 5 (± 3), which are similar to the results of
three of the groups in version one.

Processes of Sense-making
From the activity sampling data, it is possible that the groups tended to alternate between broad (several topics)
and narrow (few topics), which suggests movement from linking to the development of rules. To explore these
forms of interpretation, the following section presents extracts of discussion between participants. The
discussions are verbatim records of participants’ statements. Each extract is identified by group {A,B,C} and
speaker {a,b,c,d,e}.

SEEKING A FRAME
Group B (5 participants) began by discussing the Angel warehouse and then split into two sub-groups. One
subgroup searched for more information about the warehouse, while the other subgroup (consisting of
Participants Bb and Be) developed a social network diagram on the whiteboard. Thus, Group B framed the task
as a social network problem. They identified a possible American connection (presumably in terms of the
purchase of the marina by a US company in a deal brokered by Grosby).

DEVELOPING A FRAME
Group C argued between arresting Sarti and arresting Pasquidini. In particular, group C discuss the role of
Pasquidini and present inferences that feature him as a shadowy figure (Cb “the anonymous Frenchman”) who
Ca notes is “connected…He has links”, and Cd observes “He’s been calling all over the joint”. Against this
proposal Cb is concerned “All we have is the phone records” which raises the issue of what would constitute
evidence to support the arrest of an individual in this exercise. This discussion leads to the development of a
rule proposed by Ca “We need to take out the source in France to bring down the whole network.”

COMPARING, ELABORATING AND QUESTIONING FRAMES

In this exercise, the groups would regularly (every 20 minutes or so) collect around the representation that they
were creating and run through their analysis. At one level, this could be seen as rehearsal of their final
presentation. At a deeper level, this provides an opportunity to question the story. In other words, the groups
were elaborating and questioning the frames they were using (i.e., ‘conflict and corroboration’ and ‘hypothesis
exploration’ (Elm et al., 2005), and the rehearsals could be seen as ways of testing the narrative of their analysis.

Comparison of Solutions
All three groups identified Sarti (3/3) as a prime suspect (because she was central to so much of the exercise) and
all three teams named Pasquidini (3/3). Two of the groups proposed that Pasquidini should be arrested in France,
suspecting his involvement in supplying the drugs and loading them on to the yacht. Groups also discussed Pico
/ Cobo (2/3), as a driver of the van, and mentioned Calabrese (2/3) in support of this proposal; Calabrese was
Ricord’s chauffeur and had been arrested driving the van, and now Pico / Cobo was Ricord’s chauffeur so
looked suspicious. All three groups used the Harbour Logs (3/3) and Accounts (3/3) to provide evidence of who
was linked to whom and when events occurred, with the phone records (2/3) supplementing the links.

4
 Baber, C et al. Intelligence Analysis and Sense-making

DISCUSSION
The exercise was designed on the an assumption that the ‘best’ way to reach the correct assessment was to
identify the modus operandi (M.O.) of the gang and then look for information as to when this M.O. was likely to
be applied. However, while there are instances where the groups described the M.O., this did not seem to be the
primary focus on their analysis; rather, data were combined into sets, or frames, and group discussion seemed to
focus on these frames.

Using Frames
Frames begin in a sketchy (minimal) manner, either through the linking of data in representations or through the
linking of concepts in response to questions. This might be a function of the nature of the information provided
for the exercise, with all groups beginning their processing with the sorting of sheets into piles. An interesting
point to note here is when, or if, the piles of information sheets became ‘frames’.

The groups not only sought links between sampled information, but also developed their hypotheses through
testing them. Thus, group C not only raised hypotheses about Pasquidini, Sarti and Calabrese but also
challenged these hypotheses in their discussions. This suggests that these groups were not only engaging in the
‘down-collecting of material but also in ‘conflict and corroboration’. Further, the observation that all the groups
would rehearse their presentation at intervals during the Exercise suggests that they recognised the value of
‘hypothesis exploration’ as a core part of their analytical work.

In terms of the Data/ Frame model of sense-making, this paper offers some insight into the dynamics of the
process of sense-making in teams. The observational data suggest that teams prefer to work with a small number
of pieces of information (i.e., a mean of 5 pieces). Further, the groups tend to move between broad and shallow
and deep and narrow search (focusing on specific frames, but, generally working with more than one frame at
any one time). Previous work had shown that the less experienced analysts might have been either unable to
generate an appropriate frame (being swamped by data) or who might have stuck with a particular frame even
when it is not appropriate to the current set of data (Baber et al., 2013). This implies that differences in sense-
making of experienced and inexperienced analysts are not solely a matter of knowledge of the domain but relate
to the manner in which evidence is selected and processed, and hypotheses and frames employed.

The extracts of team discussions suggest that, even when teams focus on a frame, their attention is drawn to
other data and the analysis moves between several frames in short succession. This suggests that traversal of the
Data / Frame model is faster than one might expect. In other words, in this Exercise, teams seem to move
through the Data / Frame cycle quickly, with consideration of several frames, rather than taking a single frame
and processing this. This implies that the Exercise resulted in an abductive approach to reasoning, in which the
data were explored and different explanations generated, rather than a hypothetico-deductive approach. While
we would not claim that this represents all forms of Intelligence Analysis, it is interesting that this cyclical
approach is very different from the linear approaches implied by Pirolli and Card (2006) or Heuer (1999);
though it is difficult to assess if these were proposed as normative models, or as realistic models of how people
actually work. What we observed in this study was that, while people operate using ‘competing hypotheses’,
these tended to be articulated as loose, imprecise statements rather than as objectively grounded comparisons.

Creating and Using Artefacts as Representations

Artefacts are a way of externalizing a frame in which sets of data can be combined. The representations either
focused on the grouping of people (through link or social network diagrams) or events (through timelines), or a
combination of these. Initially these were a means of organising data. However, as the Exercise developed, the
representations became the focus of the final presentation. This meant that, rather than creating representations
to serve as aide memoire for their own discussions (as inexperienced groups did in Baber et al., 2013), the
groups were creating representations for an audience, i.e., their Commanding Officer to whom they would give
their presentation. This suggests that the activity was primarily a hypothesis creation activity in which relations
between topics and resulting inferences could be used to create hypotheses for further investigation.

As de Vries and Masclet (2013) point out, collaboration is often based on minimal representations. In this
Exercise, representations created in collaborative activity are not merely diagrams showing data; rather, they are
records of the discussion and thought-processes of the groups. This means that, in order to understand the
content of the representations, it is often necessary for someone from outside the group to have an explanation of
the assumptions, ideas and background knowledge that inform these representations. In other words, the role of
representations is often to capture ‘local’ discussion rather than to create a more ‘global’ view. In order to
develop from this local-to-global view of the information, it is important to have some notion of audience.
Intelligence Analysts often talk of ‘product’ as the output of their activity. What we note here is that ‘product’
can look much the same whether it is produced by experienced or inexperienced analysts, and the primary
difference is not the ‘product’ per se so much as the understanding of who will use that product, how they will

5
 Baber, C et al. Intelligence Analysis and Sense-making

interpret the product, and what aspects of the product they will find convincing. This distinction supports the
advice offered by Heuer (1999).

Conclusions
The study in this paper supports the observation that Intelligence Analysis is not a linear, orderly process (see
also Elm et al., 2005; Kang and Stasko, 2011; Roth et al., 2010). Even with so simple a set of evidence, we
could observe behaviour that was parallel (with several group members working on different lines of enquiry),
disjointed (with group members pursuing contradictory ‘frames’, e.g., arrest Sarti or Pico, or Sarti or Pasquidini),
and recursive (with groups dismissing a frame and then reintroducing it, e.g., dismissing the abandoned car and
then considering that it was used as the drug transport vehicle). This suggests that such behaviour is likely to be
a characteristic of this type of activity. From this, it is apparent that the activity is primarily one in which small
sets of data are combined and explained.

From the use of representations, it is apparent that experience dictates the manner in which people construct, use
and share representations. This suggests that the design of “sense-making support systems” (Weick and
Meader, 1993) should not focus simply on ways to support the construction of diagrams and other forms of
representation, but also needs to consider the manner in which these representations are to be used. For
example, tools which support the collation of information to help identify links between pieces of information
might help with ‘down-collection’ of data but does not provide support for ‘conflict and corroboration’ or for ‘
hypothesis exploration’.

REFERENCES
Attfield, S. and Blandford, A. (2010) Discovery-led refinement in e-discovery investigations:
sensemaking, cognitive ergonomics and system design, Artificial Intelligence and Law, 18,
387-412.
Baber, C., Attfield, S., Wong, W. and Rooney, C. (2013) Exploring sensemaking through an
Intelligence Analysis exercise, NDM: 11th International Conference on Naturalistic
Decision Making, Marseilles, France.
de Vries, E. and Masclet, C. (2013) A framework for the study of external representations in
collaborative design settings, International Journal of Human-Computer Studies, 71, 46-
58
Elm, W., Potter, S., Tittle, J., Woods, D., Grossman, J. and Patterson, E. (2005) Finding
decision support requirements for effective intelligence analysis tools, Proceedings of the
49th Annual Meeting of the Human Factors and Ergonomics Society, Santa Monica, CA:
HFES, 297-301.
Heuer, R. (1999) Psychology of Intelligence Analysis, Washington, DC: Central Intelligence
Agency, Center for the Study of Intelligence.
Heuer, R. and Pherson, R.H. (2010) Structured Analytic Techniques for Intelligence Analysis,
CQ Press.
Kang, Y.A. and Stasko, J. (2014) Characterizing the intelligence analysis process through a
longitudinal field study: implications for visual analytics, Information Visualization, 13,
134-158.
Klein, G., Moon, B.M. and Hoffman, R.R. (2006a) Making Sense of Sense-making 1:
Alternative Perspectives, IEEE Intelligent Systems, 21, 70-73.
Klein, G., Moon, B.M. and Hoffman, R.R. (2006b) Making sense of sense 2: A
macrocognitive model, IEEE Intelligent Systems, 21, 88-92.
NATO (2008) AAP 6 Terms and Definitions
Pirolli, P. and Card, S. (2005) The sensemaking process and leverage points for analyst
technology as identified through cognitive task analysis, Proceedings of the International
Conference on Intelligence Analysis, McLean, VA: Office of the Assistant Director of
Central Intelligence for Analysis and Production.
Roth, E.M., Pfautz, J.D., Mahoney, S.M., Powell, G.M., Carlson, E.C., Guarino, S.L., Fichtl,
T.C. and Potter, S.S. (2010) Framing and contextualizing information requests: Problem
formulation as part of the intelligence analysis process, Journal of Cognitive Engineering
and Decision Making, 4, 210-230.

6
 Baber, C et al. Intelligence Analysis and Sense-making

Weick, K.E. and Meader, D.K. (1993) Sense-making and group support systems. In: Jessup,
L., Valacich, J. (Eds.), Group Support Systems: New Perspectives, Macmillan Publishing
Company, New York, NY, USA, pp. 230–252.
Appendix One : Solution to Exercise
1. The Solution is to arrest Pasquidini (the passenger on the yacht, as outlined in the M.O. above). The
puzzle is to place Pasquidini in Exmouth and to see him as the van driver. There is no direct evidence
to this effect (which is why the Exercise is challenging). However, the combination of M.O. and
evidence from today and yesterday should help the groups narrow down their set of suspects and realise
that the yacht’s passenger is the van driver and that Pasquidini possibly travelled from France.
2. There are three people who are dubious but who have insufficient information to justify arrests:
a. Muriel Grosby has been involved in the deal to buy the marina and has a wide range of highly
suspicious transactions in her business accounts. She also owns the mini-cab firm which is
dealing the drugs. On the other hand, she is involved in charitable events with the Marina and
with making donations to it. The accounts and client list of Ricord Accountancy Services link
many of the characters together suspiciously – but not in sufficient detail to clearly indicate
nefarious activity.
b. Martina Sarti hires the vans which are used for transporting the drugs – but it is likely that she
hires vans on a regular basis for people coming into the marina and not specifically for the
smuggling operation. She has received money from Grosby but it is not obvious why this is
suspicious, given their relationship with the marina.
c. A petty criminal (Cobo a.k.a Pico) who comes from Leeds, is living next to Martina Sarti
(although it is likely that she is spending most of her time in Exmouth with David, the marina
manager), and is being paid as a chauffeur by the accountant Ricord.
In order to arrive at the solution, one approach would begin with the arrest of Calabrese, who (as pointed out in
the briefing was sentenced on 14th June 2012). The newspaper article detailing Calabrese’s sentencing notes that
he was arrested in November 2011. Two statements dating from November 2011 (one from Calabrese and one
from Bocognani, the former manager of the Marina) suggest that the gang’s M.O. is to ship drugs from Roskoff
on a yacht skippered by Perrin, to arrive at Exmouth in the early hours and for the drugs to be moved by van to
the Angel Warehouse in Leeds. A record of van hire shows that Sarti hired a van in early November 2011. A
review of other van hire logs shows that Sarti hired a van in August 2012 and hired a van yesterday (9th
September 2012). The Marina log shows that the only yacht due in today is the ‘Sunny Jim’, owned by
Condiere. The other evidence that corresponds to ‘today’ is the phone logs of Pasquidini, who calls Condiere,
Perrin, Angelleti, Munoz and Sarti. Pasquidini’s ‘suspect card’ shows that he lives in Roskoff.

Appendix Two: Briefing given to all participants

“Muriel Grosby is a businesswoman who lives in Leeds and runs a road haulage and mini-cab firm. While she
has no criminal convictions, local police have long been suspicious of her acquaintances and believe that she
has links with criminal activity, particularly relating to drug smuggling and people trafficking. A known contact
of Grosby, called Calabrese, was sentenced, on 14th June, to 9 years for drug smuggling.
Intelligence suggests that there is likely to be a shipment of class A drugs being delivered to a port in the South-
West of England in the next few weeks. Given resource and personnel constraints, it is not possible to follow
every suspect so you need to determine who should be arrested and where the best place might be to make such
arrests.
Following your investigation, you will give a presentation on your findings. The presentation will include:
1. Name of individual, or individuals, to target as Suspects.
2. The FIVE pieces of evidence that best support your proposal to 1.
3. Location of the arrest or arrests.
In order to make this exercise easier, you will select suspects from a set of nine people:
 Muriel Grosby, who I have already described;
 Jennifer Garlica who is Grosby’s sister-in-law and whose husband was killed last year in what looks
like a gangland hit;
 Vanessa Munoz who is the assistant manager of Exmouth Marina;
 Martina Sarti who works at the marina and is the girlfriend of the marina manager (Xavier David);
 Pierre Pasquidini who lives in Roskoff and travels regularly to the UK;
 Kenny Chiappe who drives a mini-cab in Leeds;
 Jake Ajachinsky who is a petty criminal;
 David Pico who is Jake’s best friend, is also a petty criminal and has a tempestuous on-off relationship
with Jake’s twin sister, Denise Ajachinsky, who is also a suspect.
For this Exercise, ‘today’s’ date is September 10th 2012 (this will help you make sense of the dates and times on
the documents you have).”