Chapter 3 – THEORETICAL FRAMEWORKS

Chapter Lead: P. Farrell

Contributors: P. Farrell, J. Gersh, E. Hollnagel,
I. MacLeod, C. Miller, A. Schulte, P. Stensson

3.1 INTRODUCTION

3.1.1 Background
The NATO RTO Human Factors and Medicine Panel Task Group 017 entitled, “Uninhabited Military
Vehicles (UMVs): The Human Factors of Augmenting the Force” involves studying ways to enhance military
Forces by leveraging the potential advantages of UMVs to act as force multipliers. This report identifies,
prioritizes, and addresses the Human Factors (HF) issues associated with integrating UMVs into the Force.
Simply put, reducing operator-vehicle ratio and improving interoperability will augment the Force – both of
these tenets require HF consideration.

Currently, up to six positions are involved in High Altitude Long Endurance (HALE) Uninhabited Aerial
Vehicle (UAV) missions such as the Global Hawk. Medium Altitude Long Endurance (MALE) and Tactical
UAVs have three positions operating them, such as the Predator B and SPERWER. Uninhabited Ground
Vehicles (UGV) and other small UAVs have a one to one operator to vehicle ratio. It is anticipated that future
uninhabited vehicles will operate as a team (or more commonly referred to as a swarm) with only a single
human as part of the team [1]. One vehicle could detect targets, while another could act as a communications
relay, and yet a third could target and track the target, and so on. Thus, the team of vehicles would augment
the human’s own ability, and in turn augment the Force.

[Link] Reducing Operator to Vehicle Ratio

One HF consideration is the number of vehicles that a single human (or a crew) can operate. This issue,
in part, is under investigation in Canada under the Intelligent Adaptive Interface programme [2,3,4]. It is
related to human information processing considerations and involves human limits to perceiving information,
working memory, goal setting, decision-making, and acting on those decisions.

[Link].1 Perception
For example, a UMV has the potential to provide persistent twenty-four hour surveillance of a stationary or
moving target. A human operator may have access to hundreds of gigabytes of data, but is likely to ignore,
shed, or chunk a large part of it. Automatic target recognition, directed attention algorithms, fusion
algorithms, large database visualisation and intelligent search algorithms may help the operator in managing
the data. Never-the-less, there is the potential to increase the data exponentially as more vehicles are added to
the team.

[Link].2 Memory
Simplistically, a human limit to working memory is seven plus or minus two items [5]. For dynamic working
memory, the number drops to two or possibly three items [6]. Thus, one might postulate that operators can

RTO-TR-HFM-078 3-1
THEORETICAL FRAMEWORKS

actively control a maximum of two uninhabited vehicles. In a recent experiment, a crew of three was able to
manage up to six UAVs towards completing the mission. We suspect active control of each vehicle is not
possible, but other system architectures can be employed so that a single operator can manage multiple
vehicles. For example, a single operator might control a single vehicle, which then controls all other vehicles
on the team. Or a single operator might supervise and direct all the vehicles (similar to an air traffic
controller). Or a single operator may act as a team player (maybe the team captain) in a collaborative team
where most team members are machines. Regardless of the strategy employed, as one adds more UMVs to
augment the Force, there is the potential to overload working memory.

[Link].3 Goal Setting

Goal setting becomes important for human-machine interaction, and even more critical for teamwork.
Individual goals need not be the same, and in most cases must be managed by training or by a hierarchical
structure of roles and responsibilities. It is important, however, that there is common intent amongst team
members [7] – that is a common understanding of the goals, information needs, and actions to be taken in
order to achieve the mission objective. Uninhabited vehicles that have the ability to dynamically set goals
based on mission objectives, environmental conditions, or human psychological and physiological conditions
(i.e., levels of automation/autonomy) could be treated as team members. Conceptually, all the team attributes
would still apply in this human-machine interaction case, including goal setting.

[Link].4 Decision Making

Decision-making involves understanding the mission objectives and desired states, interpreting the current
state of the world, developing possible actions/communications strategies, and choosing one of those
strategies that will impact the world and move the current states closer to the desired states in an effective
manner. At low levels of abstraction, cruise control and autopilots already sense the world, compare their
sensory information to goal states, and make appropriate decisions and actions based on relatively simplistic
rules and heuristics. At a certain level of abstraction, future intelligent and adaptive vehicles might make high-
level (human-like) decisions. Vehicles may have the ability to make decisions about collision avoidance,
target prosecution, and self-preservation. This raises many issues from goal de-confliction, to defining areas of
responsibility and authority amongst the team members, through to legal, moral, and ethical considerations of
machines making high-level decisions.

[Link].5 Behaviour
A human is limited to how many vehicles he or she can physically control simultaneously. There is an implicit
assumption that the human would not physically manoeuvre multiple vehicles simultaneously, but that the
operator would perform some form of waypoint navigation. Investigation is under way with two modes of
operation. In the first case, the vehicles are treated as separate entities, and in the second case the vehicles are
treated as a single entity (i.e., a swarm or team of vehicles). Automatic Pilots are sophisticated and mature
enough in sea and air environments to manoeuvre in controlled air and sea spaces with minimal human
assistance. However, in ground environments, artificial intelligent navigation still requires further
development for the vehicles to transverse the complex terrain – indoors as well as outdoors [8].

Although augmenting an operator with multiple vehicles is the first objective, the system designer must be
cognisant of human limits with respect to perception, cognition (goal setting, working memory, and decision-
making), and action. Thus, this report looks at how to augment the Force by optimizing the operator/vehicle
ratio while staying within human limits – this is indeed the Human Factors of Augmenting the Force.

3-2 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

[Link] Interoperability
The second objective is to augment the Force by improving interoperability. The concept is that multiple
operators from different forces (e.g., multiple military services, multiple countries, other government
agencies, and non-government agencies) can operate a single UMV. In some ways, this objective seems to
conflict with the first objective since there is the potential for multiple humans controlling one UMV instead
of one human controlling multiple UMVs.

Sharing assets can augment the Force. In a recent experiment off the east coast of Canada, the Air Force
operated a MALE UAV while the Navy commanded the UAV and the sensor information was shared to all
military services as well as other government agencies. A significant amount of coordination was required,
but the asset and particularly its product were shared successfully. This concept requires technical standards at
the level of machine design, and the coordination of competencies, authorities, and responsibilities at the level
of command and control of all parties involved.

[Link] Research and Development Areas

The Task Group identified that augmenting the Force will require research and development in the following
areas:

[Link].1 Collaborative Work – Optimal Task Distribution

• Virtual team performance
• Manned/Unmanned collaboration
• Interoperability
• Flexible level of automation
• Optimization of operator/vehicle ratio

[Link].2 Control Stations – Intelligent Operator Support

• Operator functional state assessment
• Intelligent adaptive interfaces
• Cognitive cooperation
• Knowledge management systems

[Link].3 R&D Areas Grouped into Five Thematic Areas

• Theoretical Frameworks
• Supervisory Control
• Advanced Interfaces
• Levels of Automation
• System of Systems

This chapter focuses on Theoretical Frameworks.

RTO-TR-HFM-078 3-3
THEORETICAL FRAMEWORKS

3.1.2 Why Theoretical Frameworks

Theoretical frameworks have been used to guide the design of technology, procedures, systems, and systems
of systems. UMV systems will also require theoretical frameworks to inform the design process. Most of the
frameworks used in traditional manned systems can be applied to uninhabited systems. However, revisiting
the theoretical frameworks discussion allows us to highlight aspects of the frameworks that are directly
applicable optimizing operator/vehicle ratios and interoperability of uninhabited systems. In the investigation
we may also find an emerging theory or framework that is unique to UMV systems.

A framework encapsulates the design process that typically is built on theory or a model as well as practice.
Three types of design processes (or frameworks) are systems engineering approaches such as described in [9]
“build a little and test a little”, and arbitrary/creative design. The systems engineering approaches may
produce optimal solutions, but are expensive both in time, energy, and money. “Build a little, test a little”
incremental design approach may be cost effective, but it requires several design cycles to optimize.
Arbitrary/creative design is often performed, but often at the expense of optimal system effectiveness.

The place for theory in design is as follows:

• Theory can be the starting point for design;
• Theory may identify the critical design decisions;
• Theory allows for a common taxonomy within and across systems;
• Theory helps track and maintain the aim throughout the system life cycle;
• Theory helps design system verification and validation; and
• Theory helps generate measures of effectiveness.

There are a number of theoretical frameworks that address operator/vehicle optimization and interoperability.
Theoretical frameworks developed for operator-manned vehicle interaction can be applied to uninhabited
systems when it comes to basic ergonomics, workstation design, task analysis, workload and situational
awareness. In most cases, human-machine interaction theories apply regardless if humans are inside or outside
of the vehicle, although ego- versus exo-centric frames of reference may become an issue specific to UMVs.
Human-human interaction theories (i.e., social behaviour) might better describe operators who interact with
vehicles as a team. Thus human-machine and human-human interaction theories are reasonable starting points
for exploring operator/vehicle and interoperability optimization.

The choice of framework for analysing and designing UMV systems may depend on the proposed solution.
For example, if reducing the operator/vehicle means going from three operators operating one vehicle to one
operator operating three vehicles, then one can imagine the requirement for intelligent help and levels of
automation. The theoretical framework will need to address the following aspects:
• Level of automation and time and cultural dependencies.
• Goal/constraint level interactions instead of action level interactions.
• Self-generating future plans.
• Environment and system unpredictability.
• Trust and system acceptability and predictability.
• Implications of truly autonomous (free will) systems.

3-4 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

• Animation and personification of machines.

• Self-awareness, environment awareness, and awareness of itself within its environment.

While the theory may be the starting point of the design, aspects of the design define the theoretical
framework to be applied. There is some initial iteration and recursion in determining the theoretical
framework, however this recursion should quickly converge so that the design can move forward.

3.1.3 Scope of Theoretical Frameworks Theme

At the present, no single theoretical model can be argued to encompass all aspects of human and systems work
and performance. However, theoretical frameworks help us focus on what we do know about systems and
illuminate what we yet need to discover. These frameworks should lead to an assessment of the effectiveness
of UMV systems in augmenting the force. As a start, workshop participants proposed three framework
categories.

[Link] Human Performance Models

• Human in control (hands-on, supervisory, collaborative)
• Human problem solving
• Human information processing
• Time and motion
• Human attention demand

[Link] Humans as Part of a System Models

• System Engineering
• Process/Task
• Team working
• Organisational Behaviour
• Joint Cognitive System

[Link] Human Cognition Models

• Mental processes
• Memory
• Personality and Motivation
• Interoperability (team dynamics, communications, politics, culture, and organisational issues within a
coalition of diverse forces, and the human within system of systems)

3.1.4 Identifying Relevant Theoretical Frameworks

A literature review was conducted as a first attempt to capture relevant theoretical frameworks. Ten out of
127 papers were found that were deemed highly relevant to this topic – particularly the human-machine

RTO-TR-HFM-078 3-5
THEORETICAL FRAMEWORKS

interface. Complete descriptions are found in Appendix 3-1. Synopses of seven of these papers are found in
the next section. Naturally, these papers did not address directly the objectives of reducing the operator-
vehicle ratio and optimising interoperability. Thus, we asked selected authors to comment directly on the
relationship between their theoretical framework and the objectives.

Two options were discussed at the Leiden Workshop that could help streamline and focus the investigation of
theoretical frameworks. The first option was to evaluate frameworks against a common military scenario
(see the scenario section, Chapter 2, para 2.1.6). However, if there is a significant mismatch between theory
and operation realities then the comparison may be invalid. That is, elements of theoretical framework must
be related to elements of military operations. Also, deciding on the rules for adopting frameworks would be a
formidable task.

The second option (the option we adopted) was to take a closer look at six frameworks and ask how would
they address operator-vehicle ratio and interoperability directly. The frameworks could be compared to
standard systems design approaches and to each other. In order to converge onto a core set of theoretical
frameworks, we solicited comments from selected authors and asked them to discuss their theory or model
specifically in terms of optimising the operator-vehicle ratio and interoperability. This option is limited by this
Task Group’s limited knowledge and reach in the member nations, although an effort was made to be as
thorough as possible.

At the Leiden Workshop, key programs and individuals who contribute to UMV human factors from NATO
countries were identified. Table 3-1 provides the survey questions, which were developed directly from the
Terms of Reference for this Task Group. The survey was sent to 10 individuals, and 6 surveys were returned.
Their responses are found in Appendices 2 through 7.

Table 3-1: Survey Questions

Force Multiplication
• Does the framework/model address operator to UMV ratio issues?
• If so, how could the framework/model help reduce the ratio?
• Does the framework/model address interoperability issues?
• If so, how could the framework/model help improve interoperability?

UMV Scenarios/Use-Cases
• Is the theory applicable to UMV situations (i.e., underwater, sea, land, air, space)?

Theory Evaluation
• Has the framework/model been evaluated, tested, and applied to commercial or military
operations?
• Do you have an example, closely related to the UMV situations, where the theory was
implemented?

3-6 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

3.2 FRAMEWORK DESCRIPTIONS

3.2.1 Literature Review

A literature survey was commissioned to assess the scope (breath and depth) of the literature related to
operator-agent interaction for intelligent adaptive interface design. The actual searching fields were beyond
suggested areas such as tele-robotics and human computer interaction, and included supervisory control,
information management and decision support, automatic manufacturing, medical diagnosis and consultation,
and other social behaviour areas. More than 500 abstracts were retrieved, 127 papers and reports were found
likely relevant to the research domain, and 82 were reviewed of which 27 dealt with theoretical models and
empirical evaluations. A detailed review was made for seven of those papers and these brief summaries are
listed below.

[Link] Models for the Design of Human Interaction with Complex Dynamic Systems
(Systems Engineering & Management Handbook, 1999)
by Mitchell, C.M. (1996)
This paper provides an overview of the evolution of human-machine systems models over the past forty years.
From perspectives of cognitive engineering, ecological psychology, and naturalistic decision-making,
the similarities between human-machine systems models and a variety of other recent approaches to
understanding and aiding human interaction in real-world systems were described. The paper also proposed a
set of tenets that characterizes models and human interfaces whose design was based upon the models.

The paper described the trend of developing robust models with the same levels of fidelity as system models.
The vision was to predict both system and human behaviour and provide quantitative assessments of the
proposed system design. In 60’s and 70’s, the crossover control model and the optimal control model
(Wickens, 1984) focused on continuously tracking behaviour for fully manual tasks. However, with Digital
Computer introduced, tasks are supervisory controlling of multiple subsystems. Since then, modelling objectives
changed from the pursuit of a global and analytic/computational operator simulation (i.e., a quantitative,
predictive model) to a more focused development of system and task representations that could be used for the
design of operator interfaces to complex dynamic systems, including displays, aids, and training systems.
Therefore, the objective was no longer to produce a black-box human operator simulator that functions as
robustly as traditional engineering models of system hardware and software, but rather the development of a
useful description/prescription of the system-task-operator interactions. Understanding and modeling human
cognition, problem solving, and decision-making became more and more popular and practical for the system
design.

The essence of this paper is the emphasis of the importance of context as the central tenet of Human-Machine
Systems Engineering. As Baron (1984) summarized this requirement succinctly: “[A human-machine systems
model] … embodies the idea that to model human performance, one must model the system in which that
performance is embedded.” Human behaviour, either cognitive or psychomotor, is too diverse to model unless
it is sufficiently constrained by the situation or environment; however, when these environmental constraints
exist, to model behaviour adequately, one must include a model for that environment (the perspective of
ecological interface design). Therefore, the system design should be context-oriented, descriptive, and
prescriptive.

RTO-TR-HFM-078 3-7
THEORETICAL FRAMEWORKS

[Link] A Theoretical Analysis and Preliminary Investigation of Dynamically Adaptive Interfaces

(Journal of Aviation Psychology, 11(2), 169-195, Copyright © 2001)
by Bennett, K.B., et al. (2001)
This paper described a dynamically adaptive interface (DAI), which changes display or control characteristics
of a system (or both) in real time. The goal of this DAI is to anticipate informational needs for desires of the
users and provide that information without the requirement of an explicit control input by the user.
This research found that DAIs have the potential to improve overall human-machine system performance if
they are properly designed. However, DAIs also have a very real potential to degrade performance if they not
properly designed. This study explores both theoretical and practical issues in the design of DAIs. The relation
of the DAI concept to decision aiding and automation was discussed, and a theoretical framework for design
was also outlined. This paper shows that one can apply theory to design!

In this paper, a preliminary investigation of the DAI design concept was conducted in the domain of aviation
(precision, low-level navigation). Three interfaces were evaluated including: non-traditional controls (a force
reflecting stick) and displays (a configurable flight director) were developed to support performance at the
task; a standard interfaces (conventional controls and displays), a candidate interface (alternative controls and
displays); and an adaptive interface (dynamically between the standard and candidate displays). The results
indicated that significant performance advantages in the quality of route navigation were obtained with the
candidate and adaptive interfaces relative to the standard interface; no significant differences between the
candidate and adaptive interfaces were obtained. The implication of these results was discussed, with special
emphasis on their relation to fundamental challenges that must be met for the DAI concept to be a viable
design alternative.

This paper is a good example for human-machine interaction from theoretical development to empirical
investigation to maximize overall performance. The design and comparison of three interfaces is a typical
method for such kind of research. The results are also very interesting as they raised the issue of the dilemma
for automation and adaptation. When, where, what, why, how to adapt is really a question theoretically and
practically for better operator-machine, operator-agent interaction. The theoretical framework and research
methodology are very useful for other similar research.

[Link] Integrating Perceptual and Cognitive Modeling for Adaptive and Intelligent
Human-Computer Interaction
(Proceedings of the IEEE Volume 90, Issue 7, July 2002, Page(s):1272-1289)
by Duric, Z., et al. (2002)
Through both theoretical analysis and empirical investigations, this paper used human cognitive, perceptual,
motor, and affective factors to adapt the interface design for intelligent human-computer interaction.
The essence of the paper was to monitor affective behaviour or emotional behaviour or non-verbal
information to answer what why, where, when, and how, and then adapt the display according to this
information. The method for interface design is more human-like in which the interface/machine or the agent/
automation embedded is regarded as another human assistant who can monitor perceptual and cognitive levels
and understand the “partner” or “teammate”, thus react for better collaboration with better overall results.
The idea was to emphasise the team collaboration that benefits human-human interaction. “It is not only
computer technology that needs to change to make such novel interfaces a reality. People have to change as
well and adapt to the interface that the computer presents them with. In the end, both people and the computer
have to understand each other’s intentions and/or motivations, provide feedback to each other as necessary,
and eventually adapt to each other.”

3-8 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

[Link] Adaptive Interfaces for Human-Computer Interaction: A Colourful Spectrum of

Present and Future Options
(Proceedings of the IEEE 1995, 292-297)
by Lajos, B. (1995)
This paper discussed adaptations that could be built into interfaces for human-computer interaction based on
different aspects of human behaviour such as physiological attributes (eye, ears, fingers, etc.), intellectual
characteristics (capacity, recognition, learning, decision, etc.), knowledge basis (knowing the environment,
the system, him/herself, etc.) and psychological states (concentration, vigilance, fatigue, patience, etc.).
As pointed out in the paper, that adaptive interfaces should be capable of adjusting to different forms of
information transfer, transforming the information contents, altering/merging modes of information flow, and
exchanging/combining communication media.

The paper also discussed the future of adaptive interfaces, the role of formal interaction modeling,
the importance of abstract/structural interface hierarchy, the integration of interaction modes and media,
the sophistication of interface modularity and the exploitation of the advantages in combining/integrating
conceptual-functional-physical design aspects. It is a good reference for the taxonomy of adaptive interfaces.

[Link] The Future of Watchstation Design: Evolution from Single Purpose to Intelligent
Watchstations
(2002 Command and Control Research and Technology Symposium,
Naval Postgraduate School, Monterey, California, 11-13 June 2002)
by O’Donnell, L. (2002)
The focus of this paper is to address the issues in interface design and the changes in the console design to
support distributed mission task activities for joint operations of global command and control systems.
Increased mission demands combined with smith weapons, automated functions and increased collaborative
warfighter functions have increased the multi-tasking requirements to be accomplished. Humans in a
warfighter role have shifted from a narrow task focus within a narrow job focus of a single purpose
watchstation and a high human-in-the loop interface workload, to becoming controllers of these distributed
systems and collaborative activities. In the paper, current watchstation design was described that requires the
human to perform manual system operations in combination with numerous independent synchronous
activities such as communications and adjacent equipment operations. Future watchstation will need to be
designed to support the work environment with: increased multi-tasking capabilities, dynamic monitoring of
task process, integrated system designs, and improved distributed team collaboration task capabilities.
Advances in technology have enabled the design of an effective watchstation design that will allow for multi-
modal user interfaces best suited to the task. Future watchstation designs should also utilize self-adaptive
interfaces, increased visual workspaces, agent technologies, integrated speech, and visual and direct touch
methods to reduce the human-interface workload and streamline the tasks. All these features are required for
future UAV/UCAV control interfaces.

The paper not only analyzed the current trends and advantages of intelligent interfaces (watchstations),
but also brought up a smart agent taxonomy to construct the flexible, dynamic, scalable, and robust distributed
system capabilities over system networks as multiple agent systems. Although the context discussed in the
paper was not focused on airborne multiple UAV control, but the discussions of several key technologies to
enabling an intelligent system and future research recommendations on intelligent user interface design can be
generally applied to any design of the framework for optimal operator – agent interaction.

RTO-TR-HFM-078 3-9
THEORETICAL FRAMEWORKS

[Link] An Architecture for Intelligent Interfaces: Outline of an Approach to Supporting

Operators of Complex Systems
(Human Computer Interactions, 3(2), 87-122)
by Rouse, W.B., Geddes, N.D. and Curry, R.E. (1997-1998)
This paper described a concept of a comprehensive support system design for operators of complex systems.
A variety of difficult design issues were addressed as well as ongoing efforts aimed at resolving these issues.

The main focus of the paper was to address design methodology and automation philosophies. Although the
suggested design methodology follows the traditional human factors engineering principles, automation
philosophy emphasises on maximizing overall performance by overcoming human limitations and enhancing
human abilities. The philosophy is that automation should be used as a backup – the default modes are usually
manual with automation invoked only when either anticipated operator performance is unacceptable or the
operator chooses to relinquish control. With the adoption of this operator-centred automation philosophy,
an architecture including intelligent management of information and tasks was proposed. Within it,
the concept of operator state is central to the functioning of the components of the intelligent interface.
The relevant elements include: activities, awareness, intentions, resources, performance. Another component
is the interface manager, which is similar to an executive’s assistant who zealously guards the superiors’ time
and resources. Although important questions were raised about what, when, and how to automate, there was
no clear answer in the paper. The paper presented a layout of the architecture for complex system design,
but it did not cover enough cognitive and perceptual aspects of a dynamic, complex, and interactive system,
especially for multiple UMV control.

[Link] A Model for Types and Levels of Human Interaction with Automation
(IEEE Transactions on Systems, Man and Cybernetics, 30 (3), 286-297)
by Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000)
A framework/model was proposed for types and levels of human interaction with automation (Sheridan’s
10 points scale). They also proposed four broad classes of functions which automation could be applied from
information processing point of view: a) information acquisition, b) information analysis, c) decision and
action selection, and d) action implementation. Within each of these types, automation can be applied across a
continuum of levels from low to high, from fully manual to fully automatic. A particular system can involve
automation of all four types at different levels. Since automation does not merely supplant but changes human
activity and can impose new coordination demands on the operator, appropriate selection is important based
on the primary and secondary evaluative criteria. The primary criteria look at human performance
consequences: mental workload, situation awareness, complacency, and skill degradation. The second criteria
include the automation reliability and the costs of consequences.

Although the paper considered human machine interaction mainly from information process point of view, the
proposed model could be a good starting point for considering what types and levels of automation should be
implemented in a particular system. The paper is concerned with human performance in automated systems
and emphasizes human-machine comparison. Automation is defined as a device or system that accomplishes
(partially or fully) a function that was previously, or conceivably could be, carried out (partially or fully) by a
human operator. The paper also touched a little bit on action automation – agents which track user interaction
with a computer and execute certain sub-tasks automatically in a contextually-appropriate manner. However,
this area should be elaborated more associated the theory developments and empirical investigations (which is
the future work as mentioned in the paper). Even in the proposed model, the issue in whether automation
unreliability has similar negative effects for all four stages of automation needs further examination.
Regarding costs of decision/action outcomes, individual differences between users in the same interface

3 - 10 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

should be addressed more, especially on user profile building on the interface (user modelling embedded in
interface). It is good to point out that empirical work needs to be done to explicitly compare the effects on human
performance of different levels of automation for information acquisition and analysis, in other words, the levels
of interface intelligence. Overall, this paper emphasised more function allocation between user and machine
regarding automation, but less was discussed on operator interface interaction. How the automation should
perceive, analyze, understand, react, and collaborate with user as an agent or assistant still remains untouched.

3.2.2 Frameworks from Survey Returns

The following framework descriptions were those of the six authors who responded to the survey. They serve
as the main frameworks from which we can find any common or unique theoretical elements required to
analyse UMV systems.

[Link] Cognitive Automation (CA)

Cognitive Automation makes a distinction between conventional automation and cognitive automation [10].
Also see chapter 5 for a full description of the theory. Conventional automation is predominantly focused on
subgoals and subtasks of the work process. High-level goals and their associated tasks are not known by the
automation system. Consequently, these systems observe only a small portion of what is relevant in a given
situation. For example, if the “altitude hold” autopilot is activated, it is doing its best to comply with this
assigned reference or goal state, even if a high mountain is in its way. The autopilot does not know, nor care
about the top-level safety goal of avoiding a crash into the mountain – this goal is exclusively in the realm of
the high-level operating element responsibilities (usually human supplemented by a machine). Never the less,
conventional automation is very convenient for simple automated functions where the supervisor has direct
visibility and influence on the states that the automation has control over.

Cognitive automation takes into consideration these high-level goals. The authors present the idea of an
Artificial Cognitive Unit (ACU) that implements a model of the Cognitive Process, and the ACU has been coded
in software. The ACU includes processing steps of interpretation ending up with beliefs, goal determination,
planning, and plan realisation to generate the instructions for action. These processing steps are based upon a
priori knowledge.

ACUs have the potential to achieve high-level goals compliant with those of the human operator, and therefore
may act as an operator assistant system as well as act autonomously. Then, a “cognitive” autopilot will take into
consideration the mountain in front, will know that to proceed stubbornly with altitude hold will end in disaster
and it will look for a way around all based upon the high-level goal for safe flight.

[Link] Extended Control Model (ECOM)

The Extended Control Model (ECOM) has been described in a conference paper [11] and a book chapter in a
textbook [12]. The pedigree of the model includes the description of the principle of contextual control [13],
the initial contextual control model [14], and the fully developed contextual control model [15]. Also see
chapter 7 for a full description of the theory.

Briefly explained, the model provides a framework for describing how a joint cognitive system can maintain
control of a situation or a process. A cognitive system is defined as “a system that can modify its behaviour on
the basis of past experience so as to achieve specific antientropic ends.” A joint cognitive system can be any
combination of humans and machines (technological artifacts) or humans and humans (social groups/

RTO-TR-HFM-078 3 - 11
THEORETICAL FRAMEWORKS

organisations). The model invokes the principle of multiple, simultaneous layers of control. The layers are
hierarchically organised with one or more instances at each layer. Control layers differ with respect to the time
window or time horizon they cover, as well as in the balance between feedback and feedforward control.
The model currently describes four layers of control called (from the top down) targeting, monitoring,
regulating, and tracking. It is applicable both to single systems (e.g., a driver and a vehicle) and to larger
entities and organizations.

[Link] Multiple Agent Interaction (MAI) Model

A framework is proposed based on Perceptual Control Theory (PCT) [16] to model multiple non-interacting
and interacting agents [17]. Agents, in this case, may be human or software agents that mimic human
cognition and behaviour. This modelling activity was performed in order to understand the system dynamics
of human-intelligent agent interaction, which would lead to design implications. PCT describes human
cognition as a means-end hierarchical network of control units. Each control unit involves the control of a
perception. At the lowest levels, control is subconscious and might be described by classical linear control
theory. At the highest levels, control is conscious and deliberate requiring rule-based thinking, logic,
and reasoning. Regardless of the hierarchical level, the control law remains the same – the output of each
control loop attempts to drive the perception closer to its internal goal. The key advantage of modelling agents
using PCT is that one can apply all the mathematical power of control theory including stability and
optimization analyses.

From a control theory perspective, human-machine interaction is often analysed by treating the machine as a
simple input-output transfer function with some known disturbances, and the human is part of the machine’s
controller algorithm [18]. On the other hand, multiple agent interaction should be treated as separate control
loops that have independent reference values, and that interact only through the influencing and sensing of
common states of the physical world. Conceptually, this interaction model would produce a complex set of
dynamics that are not always stable and not easily predictable. Thus, as the operator-vehicle ratio is reduced
by adding more intelligent agents, there is a risk of producing an unstable system if not carefully designed.

The following principles were highlighted during the analysis of multiple agent interaction using control
theory techniques:
• Closed-loop feedback modelling techniques can be used in the design of multiple agent systems.
• Designers should consider goals, sensing and decision-making strategies, and world states as part of
their system design.
• Agents should act on separate states, while gathering data from all sources.

These design principles have been applied to UMV research studies on the design of intelligent adaptive
interfaces, and selecting crews for UAV operations. The third design philosophy was applied to information
management business rules for the Multi-National Experiment 4 on Effects Based Operations with US Joint
Forces Command as the experiment lead. This experiment involved over one hundred players over a
distributed network from countries around the world. The interface design and business rules were critical in
order to successfully collaborate and conduct the experimental operation.

[Link] Military Relevance Philosophy (MRP)

See Chapter 2 for a full description of the theory. The human axiom is to facilitate the development of
technology and the use of it to actually serve the human in the best possible way. With respect to UMVs,
the human axiom is to put the technologies of automation and uninhabited systems into a necessary context in

3 - 12 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

order to reduce the risk of having these capabilities become counterproductive. This is especially important
when it comes to highly automated and uninhabited systems. Military effect is nothing but humans
influencing other humans using military powers, commonly facilitated by technological systems. The axiom is
to state that if there is no military human completely in charge of the systems’ effect, then the effect isn’t
actually military and thus has no military relevance.

By nature, it is impossible to exactly predict the future since every situation contains a certain amount of
uncertainty, which here is defined as the concept of Situation Uncertainty (SU). Furthermore, every situation
changes continuously, and increases with amount of time before an event. Having control or being in control
of situations with very high uncertainty is difficult. It is a significant difference between to have control and to
perform control. The state of having control is a possible result of the work done by performing control, but it
is not a necessary consequence of control (that is, even though control has been applied a system may diverge
because of high uncertainty). To be able to have control there must be awareness about the situation
(i.e., reduced uncertainty), the entity’s ability to control its environment, and the control of the entity itself.

When it comes to “controlling humans”, the concept of trust may replace the concept of control, because
interacting entities are autonomous, self-aware agents. Humans deal with uncertain and complex situations
trusting that the other human team member will act according to common values. One can imagine that trust
will play a significant role amongst human operators and autonomous UMV systems. Humans may grow to
trust the machine, but trust must be mutual under uncertain situations. The technological system must not only
trust the human, but also itself particularly when the system functions outside its design envelope (i.e., SU).
This is especially true for uncertainties in the higher levels of control, uncertainties about what to do and why
it has to be done, and less true when it comes to uncertainties about how to do things. Current technologies do
not have the capability to trust as defined herein.

Which results in the human axiom:

“Every reduction of human control over technology is negative. If control, in defiance of that, is
to be reduced it must be completely justified. Reduction of control may be justified if and only if
the worst possible consequences of the reduced control are exceeded by the benefits of reducing
the control (Patrik Stensson).”

Relative strengths (i.e., safety and force multiplication) follow directly from separation of human and
platform. Relative weaknesses are mostly consequences of reduced control because of the same separation as
well as the lack of mutual trust. Relevance of technological solutions, especially military ones that might
deliberately cause harm, needs to be judged thoroughly.

[Link] Playbook (PB)

As Unmanned Military Vehicles become more intelligent and capable, and as there is an attempt to control
more of them with fewer humans in the loop, there is a need to move toward a model of delegation of control
rather than the direct control that characterizes current practice [19]. Also see Chapter 7 for a full description
of the theory. Five delegation methods are identified and described, and can serve as building blocks from
which to compose complex and sensitive delegation systems: delegation through:
1) Providing goals;
2) Providing full or partial plans;
3) Providing negative constraints;

RTO-TR-HFM-078 3 - 13
THEORETICAL FRAMEWORKS

4) Providing positive constraints or stipulations; and

5) Providing priorities or value statements in the form of a policy.

The Playbook architecture supports delegation action types 1 – 4 in principle and has been implemented in
prior prototypes to include action types 2 and 4. While the work described above represents a general
framework for delegation interactions suitable for human interaction with smart automation of various kinds
and, perhaps uniquely, suitable for the tasking of multiple UMVs, the work has thus far progressed only to the
proof of concept stage. The Playbook ‘proper’ consists of a User Interface (UI) and a constraint propagation
planner known as the Mission Analysis Component (MAC) that communicate with each other and with the
operator via a Shared Task Model. The operator communicates instructions in the form of desired goals, tasks,
partial plans or constraints, via the UI, using the task structures of the shared task model. The MAC is an
automated planning system that understands these instructions and (a) evaluates them for feasibility and/or (b)
expands them to produce fully executable plans. The MAC may draw on special purpose planning tools
(e.g., an optimizing path planner) to perform these functions, wrapping them in its task-sensitive environment.
Outside of the tasking interface, but essential to its use, are two additional components. An Event Handling
component, itself a reactive planning system capable of making momentary adjustments during execution,
takes plans from the Playbook. These instructions are sent to control algorithms that actually effect behaviors.

The final type of delegation interaction offers the ability to provide priorities between alternate goals and states
and to do so more abstractly than the above methods. These abstract value statements that a supervisor might
provide are referred to as his or her “policy” for performance in the domain. A policy statement is an abstract,
general, a priori statement of the relative importance or value of a goal state in the domain. In its simplest form,
policy provides a method for human operators to mathematically define what constitutes “goodness.”

The work described above represents a general framework for delegation interactions suitable for human
interaction with smart automation of various kinds. This work is in the proof of concept stage although
exploration of Playbook-like interfaces is being conducted (under a DARPA-IXO SBIR grant). One of the
goals of this work will be to develop task libraries and task construction tools and interface concepts to move
the delegation interface work along toward implementation and utility.

[Link] System Process/Task Organisational Model for HF V&V (SPTO)

If “The Human Factors of Augmenting the Force” is indeed the objective then the outcomes produced should
have enough practicality to consider their applicability in augmenting force effectiveness in the ‘real’ world.
Amongst other considerations, it is important to consider the verification and validation (V&V) of a
theoretical framework for augmenting the force. V&V is made both statically (verification) and dynamically
(validation) – statically to ensure that it is basically logical and fit for intended purpose, dynamically to
examine its fit to reality [20].

The System Process/Task Organisational Model for HF V&V presented at Leiden was shown using an
embedded Theoretical Framework of Task Organisation. In the author’s view that framework always implies
levels of consideration on Task Context, Situation, Mission, Goal(s) [possibly both Strategic and Tactical],
Force Structure, Organisations, Cultural influences, the role(s) of technology, required system functions and
performance, roles of personnel, jobs, teamwork, individual tasks, and individual actor / entity properties and
activities.

If a system is considered dynamically it is with a task related perspective; if statically it is more at a function,
constraint, architecture, capability, or generic process perspective of consideration. However, the model is

3 - 14 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

seldom expressed explicitly. The model relies on checks on the effectiveness of the progress of a system
through its set process phases (i.e., What has occurred and When by forms of Measures of Effectiveness
[MOEs]). To provide explanation of the quality of work performance throughout the phases of the process an
associated Task Organisational Model provides evidence of the quality of work performance to satisfy
questions on How, Why, Where, and by Whom task effort has been applied in support of the satisfaction of
the goals of the system process (here possibly by the use of forms of Measures of Performance as associated
with MOEs).

If proof is required that the application of HF to UMV life cycle issues will augment the force then it is
necessary to provide a theoretical model with associated measures that if applied in practice can produce
evidence of the degree that the application of HF has succeeded in its aim. One such model has been
proposed. It is suggested that regardless of the form or efficacy of HF practices applied to the life cycle issues
of employment of UMVs in a force, that the adoption of some form of a System Process/Task Organisational
Model is necessary to fully evaluate a system, its capability, and its quality of fitness for purpose.

3.3 SURVEY RESULTS

The following table summarizes the survey results. For brevity, key sentences and/or phrases are taken from
the responses in Appendices 1 through 7. For more detail please refer to the Appendices.

RTO-TR-HFM-078 3 - 15
 THEORETICAL FRAMEWORKS

Table 3-2: Survey Results

Question CA ECOM MAI MRP PB SPTO

Does the framework/model address Yes. Yes. Yes. No. Yes. Yes.
operator to UMV ratio issues?
If so, how could the framework/ Will supplement Enables a Has the potential to Will help in Permits a single Considers
model help reduce the ratio? human teams and constructive show optimal increasing the operator to do automation,
thus affect operator discussion of trajectories through operator to everything from autonomy,
to UMV ratio. allocations. the state space as the UMV ratio joystick control to manning, and
ratio is reduced. instead of swarm control of any personnel.
reducing it. number of UAVs.
Does the framework/model address Yes. Yes, Indirectly. Yes. Yes. Yes. Yes.
interoperability issues?
If so, how could the Will handle Give guidelines Standards might Appropriate PB achieves Task
framework/model help improve problems covering about which ensure proper sensing human control interoperability by organizational
interoperability? the full scope of information is of information, and is necessary to building the knowledge model can be
interoperability needed. action must be achieve about how to utilize developed to
levels – both high coordinated to ensure interoperability. different resources consider
and low. de-confliction. into PB’s Planner. interoperability.
Is the theory applicable to UMV Yes. Yes (implied). Yes. Yes. Yes (implied). Yes, Indirectly.
situations (i.e., underwater, sea,
land, air, space)?
Has the framework/model been Yes. Yes. Yes. Not applicable. Yes. Yes, Implicitly.
evaluated, tested, and applied to
commercial or military operations?
Do you have an example, closely COSY flight. Automobiles. Multiple UAV Not Applicable. DARPA UAV project, No, only
related to the UMV situations, Nuclear Power interface design. PVACS. coalition
COSA.
where the theory was implemented? plant. operations.
Mini-UAV field
demonstration. Closely related
Manned-unmanned UMV example.
teaming.

3 - 16 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

3.4 DISCUSSION

3.4.1 Operator to UMV Ratio

Five out of six authors indicated that their framework would contribute to reducing the operator-vehicle ratio.
This was to be expected since the authors were invited to comment. All but MRP indicated how this would be
accomplished, or at the very least, how various operator-vehicle configurations could be analysed.

CA suggested that the Artificial Cognitive Unit could perform some high-level cognitive tasks that would be
otherwise done by the human, thus freeing the operator to manage multiple UMVs. ECOM provides a
theoretical basis to discuss levels of control and control allocation. MAI would caution that the real issue is
not the ratio of operators to vehicles, but that the addition of (human or machine) intelligent agents increases
the chances for instabilities. MRP would argue that if humans are to remain in control, then the ratio should be
increased rather than reduced. Given that the operator is the “coach” and the UMVs are “players” then PB
would claim that there could be any number of UMVs that adhere to the playbook. SPTO says that automation
and manning should be considered in parallel.

In general, frameworks provide a common lexicon and/or analogy around which any specific configuration may
be described and compared to other configurations. Some type of framework or model is required to predict
performance when the operator-vehicle ratio is reduced. Without this modeling investigation, it becomes
increasingly difficult to have confidence that the system will work.

3.4.2 Interoperability
Contributors inferred that their framework addresses interoperability, although there is some suggestion that
interoperability was interpreted differently amongst the theories. In all cases, human interoperability at the
highest level of abstraction was included in the interpretation. CA claims to cover the full scope of
interoperability. ECOM would provide guidelines for the information needs. MAI would produce standards
for information exchange between agents. MRP focused on human control as the means of achieving
interoperability. PB has a Planner that includes the use of resources. SPTO may require some modifications so
that the model addresses interoperability.

3.4.3 Applications and UMV Situations

The survey provided three questions on theory applicability, theory application, and specific UMV situations.
The intent was not to exclude a theory because it had not been applied to a UMV situation, yet weight those
theories that have been applied to UMV situations highly. All theories are applicable to UMV situations
according to the respondents. Five out of six frameworks have been tested and applied to commercial or
military operations. MRP is a philosophy and is strictly applicable to these questions. From the list of UMV
situations, CA, ECOM, and PB seemed to be the most mature followed by MAI. SPTO has been applied to
coalition operations, and MRP, again, is not applicable.

Like any application of theory there are compromises usually between cost, effort, and time. The final design
is usually a “healthy” compromise between the theory and practical considerations. The key point here is that
frameworks have gone from theory to implementation with positive and useful results.

RTO-TR-HFM-078 3 - 17
THEORETICAL FRAMEWORKS

3.4.4 Common and Unique Framework Elements

This section provides a first look at some of the elements that all the frameworks have in common, and then
highlights some of the unique framework elements. The first common element is the notion of control theory.
That is, most theories have the notion of desired states and actual states in conjunction with sensing the world
and influencing world states. This seems to be true for animate or inanimate actors within the environment.

Another common element is the idea of a hierarchy. There is an admission that the framework should be able
to cope with multiple levels of abstraction – from data sensing and perception through to higher goal setting,
reasoning, and decision-making. If the model is a single level abstraction, then it is might not be able to
address operator-UMV interaction.

A third common element is that most frameworks advocate the need for the actor to sense the world
(environment), sense its own state, sense the state of other actors within its immediate environment (including
the team lead), and understand the mission objectives. Effectively, the algorithm(s) must at least mimic to
some extent human cognition what humans seem to naturally do when working in a team of humans
(see Chapter 7).

Descriptive analyses are another common element amongst most frameworks. That is, investigation of
operator-vehicle ratio or interoperability is done primarily by logically reasoning through the topics using the
theory as a frame of reference. Some of the more mature theories have shown the instantiation of their
framework in a product or by modelling and simulation to show that the concepts indeed work.

A final common element is that the theory often leads to design philosophy and guidelines. In the case of
MRP and SPTO, the design considerations begin with philosophical statements. It becomes difficult to
separate the model from the model’s underpinning philosophy and perhaps it is not necessary to make this
distinction. Only that a practical output of theory and philosophy are design guidelines.

One of the unique framework elements includes PB with the notion of a playbook. This is not precisely rule-
based decision-making, nor is it free play. It does pre-suppose cooperative actors working together to achieve
a common objective. Not every actor knows precisely what the other actor is doing, but at least their actions
are consistent and they are moving toward a common goal. This uniqueness has tremendous potential for an
operator managing multiple UMVs.

Another unique framework element is the mathematical analysis of MAI. That is, symbolic mathematics was
used to draw conclusions from the framework. On the other hand, the assumptions that were applied to make
the mathematics tractable could be easily argued that the resultant model does not reflect the true situation.
Never the less, experimental results have shown that the theory has some validity and that the mathematics is
only an alternative to descriptive analyses to come to similar conclusions about multiple agent interaction.

A final unique element is the notion that a reduction in the operator-vehicle ratio is likely to lead to an
increase in situation uncertainty as stated by MPR. This seems like a logical statement, but one that designers
have a tendency to forget. Control theory has the notion of controllability and observability. That is, a stable
system has both observable states (states that can be sensed directly) that are controllable (can be influenced
directly). However, states that can neither be influenced nor sensed directly have a greater chance of being
unstable (i.e., increased uncertainty). The same principle may happen with a human operator with limited
capacity to sense and influence all of the UMVs simultaneously. There are just too many variables to track.

3 - 18 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

3.5 SUMMARY
The purpose of this chapter is to inform the reader for the need to consider a theoretical framework throughout
the life cycle of UMVs. The theoretical framework helps bound the problem and provides guidance on how to
operate the new system. A large number of theoretical frameworks were reviewed for their relevance to
UMVs and augmenting the Force by reducing the operator-vehicle ratio and/or optimising interoperability.
Six frameworks were examined more closely, and it was shown that they do address the two objectives.
Furthermore, the discussion led to the discovery of common and unique elements of the frameworks.
The common elements were:
• Control Theory;
• Hierarchy;
• Sensing the world, own state, other actors, and understanding the mission objectives;
• Descriptive analyses; and
• Design philosophy and guidelines.

The unique elements were:

• Playbook;
• Mathematical analyses; and
• Reduced ratio means increased uncertainty.

3.5.1 Recommendations
The recommendations focus on the common elements while the unique elements act as a reminder to the
designer as they consider their UMV systems.

The recommendation is that a UMV system designer should consider having a framework guide the design
process. The framework should have elements of control theory and hierarchy. The framework should address
sensing and understanding as much about the world, the actors in that environment, and the mission
objectives. The framework should lead to descriptive analyses, although mathematical analyses would be
helpful as well. The framework should yield design philosophies and design guidelines such as the situation
uncertainty principle.

3.6 CHAPTER 3 REFERENCES

[1] Forbes, K., Baker, K. and Youngson, G. (2005). Human-computer interaction between the operator and
the semi-autonomous UAV team. Contract Report, DRDC Ottawa, Ottawa, Canada.

[2] Kobierski, B. and Yves-Lamarre, J. (2004). Hierarchical Goal Analysis and Performance Modelling for
the Control of Multiple UAVs/UCAVs from an Airborne Platform, Contract Report, DRDC Toronto,
Toronto, Canada.

[3] Kobierski, B., Coates, C. and Torenvliet, G. (2005). PCT-Based Analysis of ALIX Scenario for Crew
Selection Research. Contract Report, DRDC Ottawa, Ottawa, Canada.

RTO-TR-HFM-078 3 - 19
THEORETICAL FRAMEWORKS

[4] Zhang, R. (2005). Prototype IAI for UAV Operators. Contract Report, DRDC Toronto, Ottawa, Canada.

[5] Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for
processing information. Psychological Review, 63, 81-97.

[6] Boff, K.R., Kaufman, L. and Thomas, J.P. (Eds.) (1988). Handbook of Perception and Human
Performance, John Wiley & Sons, NY, 1988.

[7] Pigeau, R. and McCann, C. (2000). The Human in Command: Exploring the Modern Military
Experience, edited by McCann and Pigeau, Kluwer Academic/Plenum Publishers, New York, 2000.
Chapter 12: Redefining Command and Control, pp. 165-184.

[8] Digney, B.L., Hubbard, P., Gagnon, E., Lauzon, M., Rabbath, C., Beckman, B., Collier, J.A.,
Penzes, S.G., Broten, G.S., Monckton, S.P., Trentini, M., Kim, B., Farell, P. and Hopkin, D. (2004).
Defence R&D Canada’s autonomous intelligent systems program. In: Proceedings of SPIE International
Society of Optical Engineering Volume 5422 – Unmanned Ground Vehicle Technology VI,
G.R. Gerhart, C.M. Shoemaker, D.W. Gage, Editors, September 2004, Orlando, Florida, USA, April 2004,
p. 13.

[9] Department of Defense (2004). MIL-HDBK-46855A, Department of Defense Handbook: Human

Engineering Program Process and Procedures. Record No. DDSM0123, August 2004.

[10] Onken, R. (2002). Cognitive Cooperation for the Sake of the Human-Machine Team Effectiveness.
In: RTO-HFM Symposium on The Role of Humans in Intelligent and Automated Systems. Warsaw,
Poland. 7-9 October 2002.

[11] Hollnagel, E., Nåbo, A. and Lau, I. (2003). A systemic model for Driver-in-Control. 2nd International
Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design, July 21-24,
Park City, Utah.

[12] Hollnagel, E. and Woods, D.D. (2005). Joint cognitive systems: Foundations of cognitive systems
engineering. Boca Raton, FL: CRC Press.

[13] Hollnagel, E. (1993a). Models of cognition: Procedural prototypes and contextual control. Le Travail
humain, 56(1), 27-51.

[14] Hollnagel, E. (1993b). Human reliability analysis: Context and control. London: Academic Press.

[15] Hollnagel, E. (2002). Time and time again. Theoretical issues in Ergonomics Science, 3(2), 143-158.

[16] Powers, W.T. (1973). Behavior: the control of perception. Chicago: Aldine.

[17] Farrell, P.S.E. (2003). Feedback in Error-correcting and Error-limiting Systems. In: Proceedings of the
50th AGM & Conference of Canadian Aeronautics and Space Institute, Montreal, Quebec, April 2003.

[18] Craik, K.J.W. (1947). Theory of the Human Operator in Control Systems: The Operator as an Engineering
System. British Journal of Psychology (38), 1947, pp. 56-61.

3 - 20 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

[19] Miller, C. (2003). Delegation Architectures: Playbooks and Policy for keeping the operator in charge.
A paper given at the Leiden Workshop for TG-078 Workshop. Smart Information Flow Technologies,
1272, Raymond Ave, St. Paul, MN 55108, U.S.A.

[20] Macleod, I.S. (2003). System Processing and Task Organizational Model for Human Factors
Verification and Validation. A paper given at the Leiden Workshop for TG-078 Workshop. Atkins
Aviation and Defence Systems, 650 Aztec West, Almondsbury Bristol, BS32 4SD, UK.

RTO-TR-HFM-078 3 - 21
THEORETICAL FRAMEWORKS

Appendix 3-1: PHASE I – LITERATURE REVIEW

A3-1.1 LITERATURE SEARCH AND PRELIMINARY REVIEW

The project Statement of Work called for a Phase I survey to assess the scope (breath and depth) of the
immediately available literature related to operator-agent interaction for intelligent adaptive interface design.
The actual searching fields were beyond suggested areas such as tele-robotics and human computer
interaction, they included supervisory control, information management and decision support, automatic
manufacturing, medical diagnosis and consultation, and other social behaviour areas, etc.

The keywords were used in searching are: adaptive automation, adaptive interface, intelligent interface,
and intelligent user interface. A number of databases were searched and included in Table A3-1.1.

Table A3-1.1: List of Databases Searched

Databases Starting Year Number of Hit

NTIS 1964 12

INSPECT 1969 4

Ei Compendex 1970 16

Biosis Previews 1969 5

EMBASE 1974 73

Pascal 1973 144

Transport Research 1970 63

Inside Conferences 1993 65

Mathscience 1940 239

SciSearch 1974 434

MEDLINE 1966 155

Information Science Abs. 1966 202

PsycINFO 1887 11

More than 500 abstracts were retrieved from all databases above, and there are 127 papers and reports were
found likely relevant to the research domain and were requested. Totally, 82 papers and reports were reviewed
as they have different focuses related to the topic of this research, as included in Table A3-1.2.

3 - 22 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Table A3-1.2: Numbers of Papers in Different Research Domain

Domain Number

Generic Review and Discussions 25

Architecture/Modelling, Technologies, and Implementations 30

Theoretical Models and Empirical Evaluations 27

Total 82

A3-1.2 PAPERS REVIEWED IN DETAIL

Of the literature identified in the search, and summarized in above section, a number of papers were selected
for more detailed review. This list was developed based on a review of the abstracts of the literature list in
Appendix 3-1, which are directly relevant to the topic of the project. While writing this report, there are two
theses and three important report coming in as ordered. However, due to the time constraints, 10 most relevant
papers (as list in Table A3-1.3) were reviewed in detail and summarized as below. The new literature will be
reviewed against the current analysis in this report in the next phase that is to revise the draft framework of
operator-agent interaction. The next section gives a short summary of the paper and brief comment on its
applicability to our themes.

Table A3-1.3: Categorization of Literature Reviewed in Detail

Paper Number
Domain Relevance

Intelligent Interface Introduction and Principles 1, 2

Theoretical Framework an/or Empirical Evaluation 3, 4, 5, 6, 8, 9, 10

Operator-Interface Interaction Technology and Tools 4, 5, 6, 7, 8, 9, 10

Agent Theory and Technology 1, 7, 8

Adaptation, Automation Philosophy 5, 6, 7, 8, 9, 10

A3-1.2.1 Special Issue on Intelligent Interface Technology: Editor’s Introduction

(Interacting with Computers, 12, pp. 315-322)
by Benyon, D.R. and Murry, D.M. (2000)
This paper discussed the reality of intelligent interface technology (IIT): “indirect management” of
information against “direct manipulation”. Through explaining the purpose of the reference model as a useful
way of thinking about IIT, the paper addressed the reasons why it is so difficult to represent fundamental
psychological data about users. One of the reasons for focusing on psychological models is that these are
characteristics which are most resistant to change in people and which can vary considerable between

RTO-TR-HFM-078 3 - 23
THEORETICAL FRAMEWORKS

individuals (as high as 20:1 – i.e., one person may take twenty times as long as another to complete the same
task). People can learn domain knowledge, but are less likely to be able to change fundamental psychological
characteristics such as spatial ability. However, one of the difficulties with capturing psychological data is that
the only signals that a computer can get are the sequences of tokens passed across the interface and attributes
of that sequence such as timing information. Although this bandwidth will increase, it still remains very
narrow compared to the wealth of information that we as human can perceive.

The definitions of the models used widely in IIT community were also given in the paper. It is a good
introduction paper although there are more models in describing human-machine interaction.
• Domain models are abstract representations of the domain, so will not include the details.
• User models describe what the system “knows” about the user. Some systems concentrate on
developing models of user habits, inferred by monitoring user-task interactions over time (i.e., by
keeping a dialogue record). Other user profile data can often be most easily obtained by asking the
user. Other systems try to infer user goals, although it is very difficult to infer what a user is trying to
do from the data typically available to a computer system (mouse clicks and a sequence of
commands). The user’s knowledge of the domain is represented in the student model.
• Interaction model: An abstract of the interaction (the dialogue record) along with mechanisms
(such as a rule-based, a statistical model, a genetic algorithm, etc.) for making inferences from the
other models, for specifying adaptations and, possibly, evaluating the effectiveness of the system’s
performance.

A3-1.2.2 Steps to Take before Intelligent User Interfaces Become Real

(Journal of Interacting with Computers, Vol. 12, No. 4, pp. 409-426, February 2000)
by Hook, K. (2000)
This paper focused on four challenges for the intelligent user interface (IUI): usability demands, creating
development methods, finding useful adaptations, and ensuring maintainability. The concept of an IUI as a
means is to overcome problems that direct manipulation interface cannot handle: information overflow,
cognitive overload. It demands better usability principles, better ways to improve the interaction, and better
tools to survive the full life cycle of a system. The paper indicated that very few IUIs that have succeeded
commercially have done their very simple adaptations based on simple knowledge of the user, or created their
adaptations based on what other users do rather than some kind any complex inferred model of the user’s
characteristics. That’s why there is a fear that IUI will violate usability principle and obscure the issue of
responsibility. The main problem is not whether or not intelligence at the interface is possible or desirable –
this depends on a lot on the task to be solved and the design of the total solution (with both adaptive and non-
adaptive parts). Instead, we can see a number of problems not yet solved that prevent us from creating good
applications. Therefore, there is a need to develop:
• Usability principles for intelligent interfaces (rather than direct-manipulation systems) that do not lead
users’ expectations astray: control, transparency, and predictability; privacy and trust.
• Reliable and cost-efficient IUI development methods including functional, data, task knowledge, user,
and environment analysis first.
• A better understanding of how and when intelligent can substantially improve the interaction
(i.e., design practice). Proper evaluations of whether the system supports users’ real tasks must
include an analysis of the organizational setting, users’ activities and cooperation with each other,
usage of other tools, etc.

3 - 24 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

• Authoring tools that enable easy development and maintenance of the intelligent parts of the system
(Scalability).

A3-1.2.3 Models for the Design of Human Interaction with Complex Dynamic Systems
(Systems Engineering & Management Handbook, 1999)
by Mitchell, C.M. (1996)
This paper overviewed the evolution of models of human-machine systems over the past forty years.
From perspectives of cognitive engineering, ecological psychology, and naturalistic decision-making,
the similarities between human-machine systems models and a variety of other recent approaches to
understanding and aiding human interaction in real-world systems were described. The paper also proposed a
set of tenets that characterizes models and human interfaces whose design is based upon them.

The paper described the trend of developing robust models with the same levels of fidelity as system models.
The vision was to predict system and human behaviour and provide quantitative assessments of proposed
system design. In 60’ and 70’, the crossover control model and the optimal control model (Wickens, 1984)
focused on continuously tracking behaviour for fully manual tasks. However, with Digital Computer
introduced, tasks are supervisory controlling of multiple subsystems. Since then, modeling goal changed from
pursing the development of a global and analytic/computational operator simulation (i.e., a quantitative,
predictive model) to more focused: the development of system and task representations that could be used for
the design of operator interfaces to complex dynamic systems, including displays, aids, and training systems.
Therefore, the goal is no longer to produce a black-box human operator simulator that functions as robustly as
traditional engineering models of system hardware and software, but rather the development of a useful
description/prescription of the system-task-operator interactions. Understanding and modeling human
cognition, problem solving, and decision-making became more and more popular and practical for the system
design.

The essence of this paper is the emphasis of the importance of context as the central tenet of Human-Machine
Systems Engineering. As Baron (1984) summarized this requirement succinctly: (A human-machine systems
model) … embodies the idea that to model human performance, one must model the system in which that
performance is embedded. Human behaviour, either cognitive or psychomotor, is too diverse to model unless it
is sufficiently constrained by the situation or environment; however, when these environmental constraints exist,
to model behaviour adequately, one must include a model for that environment (the perspective of ecological
interface design). Therefore, the system design should be context-oriented, descriptive, and prescriptive.

A3-1.2.4 A Theoretical Analysis and Preliminary Investigation of Dynamically Adaptive

Interfaces
(International Journal of Aviation Psychology, 2001, Vol. 11, No. 2, pp. 169-195)
by Kevin B. Bennett, et al. (2001)
This paper described a dynamically adaptive interface (DAI), which changes display or control characteristics
of a system (or both) in real time. The goal of this DAI is to anticipate informational needs for desires of the
users and provide that information without the requirement of an explicit control input by the user.
This research found that DAIs have the potential to improve overall human-machine system performance if
they are properly designed. However, DAIs also have a very real potential to degrade performance if they not
properly designed. This study explores both theoretical and practical issues in the design of DAIs. The relation
of the DAI concept to decision aiding and automation was discussed, and a theoretical framework for design
was also outlined.

RTO-TR-HFM-078 3 - 25
THEORETICAL FRAMEWORKS

In this paper, a preliminary investigation of the DAI design concept was conducted in the domain of aviation
(precision, low-level navigation). Three interfaces were evaluated including: non-traditional controls (a force
reflecting stick) and displays (a configurable flight director) were developed to support performance at the
task; a standard interfaces (conventional controls and displays), a candidate interface (alternative controls and
displays); and an adaptive interface (dynamically between the standard and candidate displays). The results
indicated that significant performance advantages in the quality of route navigation were obtained with the
candidate and adaptive interfaces relative to the standard interface; no significant differences between the
candidate and adaptive interfaces were obtained; no significant differences between the candidate and
adaptive interfaces were obtained. The implication of these results were discussed, with special emphasis on
their relation to fundamental challenges that must be met for the DAI concept to be a viable design alternative.

This paper is a good example for human-machine interaction from theoretical development to empirical
investigation to maximize overall performance. The design and comparison of three interfaces is a typical
method for such kind of research. The results are also very interesting as they raised the issue of the dilemma
for automation and adaptation. When, where, what, why, how to adapt is really a question theoretically and
practically for better operator-machine, operator-agent interaction. The theoretical framework and research
methodology are very useful for other similar research.

A3-1.2.5 Integrating Perceptual and Cognitive Modeling for Adaptive and Intelligent
Human-Computer Interaction
(Proceedings of the IEEE Volume 90, Issue 7, July 2002, pp. 1272-1289)
by Duric Z., et al. (2002)
Through both theoretical analysis and empirical investigations, this paper is trying to advocates the technology
and tools into interface design for intelligent human-computer interaction where human cognitive, perceptual,
motor, and affective factors are modeled and used to adapt the human computer interface. The essence of the
paper is to monitor affective behaviour or emotional behaviour or non-verbal information to answer W5
(i.e., what why, where, when, and how) and adapt the display according to this behaviour. The method for
interface design is more human-like in which the interface/machine or the agent/automation embedded is
regarded as another human assistant who can monitor perceptual and cognitive levels and understand the
“partner” or “teammate”, thus react for better collaboration with better overall results. The idea is to emphasis
the team collaboration, which is true and good in human-human interaction.
“It is not only computer technology that needs to change to make such novel interfaces a reality.
People have to change as well and adapt to the interface that the computer presents them with. In
the end, both people and the computer have to understand each other’s intentions and/or
motivations, provide feedback to each other as necessary, and eventually adapt to each other.”

A3-1.2.6 Adaptive Interfaces for Human-Computer Interaction: A Colourful Spectrum

of Present and Future Options
(IEEE International Conference on Systems, Man and Cybernetics, 1995, Volume 1, 292-297)
by Balint, L. (1995)
This paper discussed what kind of adaptations could be built into the interfaces allowing human-computer
interaction from different aspects of human behaviour: by its nature, physiological attributes (eye, ears,
fingers, etc.), intellectual characteristics (capacity, recognition, learning, decision, etc.), knowledge basis
(knowing the environment, the system, him/herself, etc.) and psychological states (concentration, vigilance,
fatigue, patience, etc.). As pointed out in the paper, that adaptive interfaces should be capable of:

3 - 26 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

• Adjusting the forms of information transfer;

• Transforming the information contents;
• Altering/merging modes of information flow; and
• Exchanging/combing communication media.

The paper also discussed the future of adaptive interfaces, the role of formal interaction modeling,
the importance of abstract/structural interface hierarchy, the integration of interaction modes and media,
the sophistication of interface modularity and the exploitation of the advantages in combining/integrating
conceptual-functional-physical design aspects. It is a good reference of Taxonomy of Adaptive Interfaces.

A3-1.2.7 Intelligent User Interfaces: An Introduction

(International Conference on Intelligent User Interfaces 1999 ACM Press)
by Maybury, M. (1999)
This paper introduced the concept of intelligent user interfaces (IUI) which aims to improve the efficiency,
effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models
of the user, domain, task, discourse, and media (e.g., graphic, natural language, gesture). As indicated in the
paper, IUIs are multi-faceted, in purpose and nature, and include capabilities for multi-media input analysis,
multi-media presentation generation, and the use of user, discourse and task models to personalize and
enhance interaction.

Two important areas addressed in the paper are agent based interaction and evaluation. Research on agent
technology has increased in prominence in applications, which includes the use of agents to express system
and discourse status via facial displays, multi-modal communication between animated computer agents, and
standards and open architectures for building agent based multi-modal interfaces – but the key questions are:
what can and should an agent do? How they should do it? How, when and why should they interact with
the user when doing it? In terms of evaluation, it can be glass-box (internal) and black-box evaluation
(end-to-end). Criteria for evaluation might include quantitative ones (e.g., time to perform tasks, accuracy of
tasks, percent of inter-assessor agreement) as well as qualitative ones (e.g., user indication of utility, ease of
use, naturalness). This is a good reference for the process of developing and testing agent technology based
interfaces.

A3-1.2.8 The Future of Watchstation Design: Evolution from Single Purpose to Intelligent
Watchstations
(2002 Command and Control Research and Technology Symposium, Naval Postgraduate
School, Monterey, California, 11-13 June 2002)
by O’Donnell, L. (2002)
The focus of this paper is to address the issues in interface design and the changes in the console design to
support distributed mission task activities for joint operations of global command and control systems.
Increased mission demands combined with smart weapons, automated functions and increased collaborative
warfighter functions have increased the multi-tasking requirements to be accomplished. Humans in a
warfighter role have shifted from a narrow task focus within a narrow job focus of a single purpose
watchstation and a high human-in-the loop interface workload, to becoming controllers of these distributed
systems and collaborative activities. In the paper, current watchstation design was described that requires the
human to perform manual system operations in combination with numerous independent synchronous
activities such as communications and adjacent equipment operations. Future watchstation will need to be

RTO-TR-HFM-078 3 - 27
THEORETICAL FRAMEWORKS

designed to support the work environment with: increased multi-tasking capabilities, dynamic monitoring of
task process, integrated system designs, and improved distributed team collaboration task capabilities.
Advances in technology have enabled the design of an effective watchstation design that will allow for multi-
modal user interfaces best suited to the task. Future watchstation designs should also utilize self-adaptive
interfaces, increased visual workspaces, agent technologies, integrated speech, and visual and direct touch
methods to reduce the human-interface workload and streamline the tasks. All these features are required for
future UAV/UCAV control interfaces.

The paper not only analyzed the current trends and advantages of intelligent interfaces (watchstations),
but also brought up a smart agent taxonomy to construct the flexible, dynamic, scalable, and robust distributed
system capabilities over system networks as multiple agent systems. Although the context discussed in the
paper was not focused on airborne multiple UAV control, but the discussions of several key technologies to
enabling an intelligent system and future research recommendations on intelligent user interface design can be
generally applied to any design of the framework for optimal operator – agent interaction.

A3-1.2.9 An Architecture for Intelligent Interfaces: Outline of an Approach to Supporting

Operators of Complex Systems
(Human Computer Interactions, 3(2), pp. 87-122)
by Rouse, W.B., Geddes, N.D. and Curry, R.E. (1997-1998)
This paper described a concept of a comprehensive support system design for operators of complex systems.
A variety of difficult design issues as well as ongoing efforts aimed at resolving theses issues were also
addressed.

The main focus of the paper is addressing two methodology issues to the design: design methodology and
automation philosophy. Although the suggested design methodology follows the traditional human factors
engineering principles, automation philosophy emphasises on maximizing overall performance by overcoming
human limitations and enhancing human abilities. The focus of the paper (which is different from many other
literatures) is the emphasis of automation as a backup – the default modes are usually manual with automation
invoked only when either anticipated operator performance is unacceptable or the operator chooses to
relinquish control. With the adoption of this operator-centred automation philosophy, an architecture
including intelligent management of information and tasks was proposed. Within it, the concept of operator
state is central to the functioning of the components of the intelligent interface. The relevant elements include:
activities, awareness, intentions, resources, performance. Another component is the interface manager which
is similar to an executive’s assistant who zealously guards the superiors’ time and resources. Although two
important questions were raised: when and how to automate, there was still no answers in the paper. Another
important question of what to automate was not addressed in the paper either. The paper discussed a nice
layout of the architecture for complex system design, but it did not cover enough cognitive and perceptual
aspects of a dynamic, complex, and interactive system, especially for multiple UAV/UCAV control.
The concept of automation control in the paper may not apply to the supervisory control mode in the UAV
case, where, it is still operator centred design, but automatic agents/ assistants will play significant roles.

A3-1.2.10 A Model for Types and Levels of Human Interaction with Automation
(IEEE Transactions on Systems, Man and Cybernetics, 30 (3), pp. 286-297)
by Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000)
A framework/model was proposed for types and levels of human interaction with automation (Sheridan’s 10
points scale). They also proposed four broad classes of functions which automation could be applied from
information processing point of view:

3 - 28 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

a) Information acquisition;
b) Information analysis;
c) Decision and action selection; and
d) Action implementation.

Within each of these types, automation can be applied across a continuum of levels from low to high,
from fully manual to fully automatic. A particular system can involve automation of all four types at different
levels. Since automation does not merely supplant but changes human activity and can impose new
coordination demands on the operator, appropriate selection is important based on the primary and secondary
evaluative criteria. The primary criteria look at human performance consequences: mental workload, situation
awareness, complacency, and skill degradation. The second criteria include the automation reliability and the
costs of consequences.

Although the paper considered human machine interaction mainly from information process point of view, the
proposed model can be taken as a good starting point for considering what types and levels of automation
should be implemented in a particular system. The paper is concerned with human performance in automated
systems and emphasizes human-machine comparison. Automation is defined as a device or system that
accomplishes (partially or fully) a function that was previously, or conceivably could be, carried out (partially
or fully) by a human operator. The paper also touched a litter bit on action automation – agents which track
user interaction with a computer and execute certain sub-tasks automatically in a contextually-appropriate
manner. However, this area should be elaborated more associated the theory developments and empirical
investigations (which is the future work as mentioned in the paper). Even in the proposed model, the issue in
whether automation unreliability has similar negative effects for all four stages of automation needs further
examination. Regarding costs of decision/action outcomes, individual differences between users in the same
interface should be addressed more, especially on user profile building on the interface (user modelling
embedded in interface). It is good to point out that empirical work needs to be done to explicitly compare the
effects on human performance of different levels of automation for information acquisition and analysis,
in other words, the levels of interface intelligence. Overall, this paper emphasised more function allocation
between user and machine regarding automation, but less was discussed on operator interface interaction.
How the automation should perceive, analyze, understand, react, and collaborate with user as an agent or
assistant still remains untouched.

A3-1.3 COMMENTS ON THE LITERATURE REVIEWED IN DETAIL

Despite a fairly significant investment in research over the past decade, there is still no generic framework or
architecture covering all aspects of human-machine interaction and the relevant technologies. In particular
from a Human Factors Engineering perspective, the lack of empirical investigations on validating proposed
frameworks makes many designs costly and ineffective. Many existing models either focus on user’s models
or task/domain models, rather than interaction models.

A3-1.3.1 What Does Interaction Model Do?

The supervisory control often only implicitly implies models of representing monitoring/situation assessment
system variables, states, or aggregate measures, especially when supervisory control systems become more
sophisticated. However, the human operator models explicitly represent the domain of application,
task constraints, and the flexibility inherent in human interaction with a complex system. Thus, the interaction
models need to reflect the work environment and its dynamic nature, as perceived by the operator given the

RTO-TR-HFM-078 3 - 29
THEORETICAL FRAMEWORKS

current system state and current system goals. The model of control activities must represent at least three
properties of both the control and controlled systems and the operator supervising them:
1) What changes to the system the operator wants to make;
2) Why the changes should be made, with respect to system goals and current state; and finally
3) How the needed changes to the system can be made, i.e., the operator activities undertaken to bring
about the desired state.

These models should represent the concurrent nature of the control activity and the choices available to the
operator given current system state. To be useful to the design, an effective model must be both descriptive
and prescriptive to describe what operator actually do and specifies what an operator should do.

Figure A3-1.1 illustrates the relations of user, system, and the interaction between them. In which, user can act
as a board of directors of a corporation, and the system can be regarded as many agents assisting the CEO who
is the intelligent interface. In order to help the board members to make decision, through CEO, agents will
provide necessary information to keep the board members informed what is currently being done and what is
going to happen at the next stage (descriptive and prescriptive). At the same time, agents will monitor the
board members’ cognitive workload and performance, and guard the resources and time. Agents can also learn
from past experience and change how they behave in given situations (adaptive). Agents will communicate
and cooperate with other agents and act according to the results of that communication (cooperative).
Therefore, user-system interaction basically is user-agent interaction for general cases. Here, agent technology
is taken as a means to facilitate the interaction between user and agents, which is the topic in next section.

Figure A3-1.1: Relation of User System Interaction.

A3-1.3.2 What Key Functions should an Interface Have?

Key functions within the user-system architecture include information management, error monitoring, and
adaptive aiding. One of the central knowledge sources underlying this functionality is an operator model that

3 - 30 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

involves a combination of algorithmic and symbolic models for assessing and predicting an operator’s
activities, awareness, intention, resources, and performance.

The intelligent interface should also have many features to support:

• More efficient interaction – enable more rapid task completion with less work.
• More effective interaction – doing the right thing at the right time, tailoring the content and form of
the interaction to the context of the user, task, dialogue.
• More natural interaction – supporting spoken, written, and gestural interaction, ideally as if
interacting with a human interlocutor.

A3-1.3.3 What Interaction Level should an Interface Have?

As pointed out by Lajos (1995), more complex adaptivity schemes should require and/or allow:
• More levels of interaction;
• More media and modes (content and form) of interaction;
• Involvement of more personal shaping factors into adaptation;
• Wide choice of skill-based, rule-based and knowledge-based interaction elements; and
• More modularity/hierarchical levels in the interface structure and functions.

The moral behind this is that people see not only with the eyes, but with the brain as well. In other words,
perception involves a whole and purposive cognitive process. From another perspective, people are selective
on their attention [8]. In other words, attention is not decided by what being presented (perception process),
but what being decided by the brain (cognitive process). People tend to forget what has been seen and what
has been heard, especially in a complex, dynamic, and challenging environment. The proactive feedback
system can better serve the purpose of communicate between user and interface for adaptation. There are
certain technologies and models can be used for such kind of intelligent system.

Other communication channels between user and interface should be considered as well such as verbal and
aural inputs. Multi-modal inputs are good for monitoring and communicating between user and interface,
but the more information being provided and processed, the more stress and workload for the interaction.
The appropriate level and channel of input and interaction should depend on the task itself. The reliability and
accuracy for those user models and algorithms are also critical for the whole system. It should also involve
trust and transparency issues: Trust, when the agent does things automatically; Transparency, when the
interface/agent effectively disappears, thus, enabling the user to interact directly with the objects of interest in
the domain and to achieve effective interaction with a minimum of cognitive effort. A dynamic adaptive
interface/agent should automatically provide information without the requirement for control input by the user
with the help of cognitive inference aid.

Figure A3-1.2 illustrates the three variables of human-machine interfaces: level of (task) complexity, level of
(interface) intelligence, and level of (user-interface) interaction. Task complexity is decided by the nature of
task (e.g., the number of vehicles to control, the level of stress, etc.). Interface intelligence is decided by agent
technology, and it should cover all aspect of human perception, behaviour, and cognition. It is also related
directly to automation level – how much work being done by the agents automatically. User’s interaction
depends on both the levels of task complex and interface intelligence. Obviously, if the level of task

RTO-TR-HFM-078 3 - 31
THEORETICAL FRAMEWORKS

complexity is high and automation level is low, then the user would probably interact more – but how much
user interaction really depends on the nature of the task and the interface.

Perception: visual verbal, aural, tactile, force

Level of Intelligence
Cognition: workload, situational awareness,
(Interface Domain)
complacency, skill degradation, fatigue,
frustration

Behaviour: manual control of input devices

Level of Interaction
Level of Complexity
(User Domain)
(Task Domain)

Figure A3-1.2: Taxonomy of Human-Machine Interaction.

3 - 32 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Appendix 3-2: COMMENTS ON THEORETICAL FRAMEWORK

FOR UNINHABITED MILITARY VEHICLES

Erik Hollnagel

In the following I shall attempt to comment on the points raised in the letter from Phil Farrell, in the ongoing
work on NATO RTO HFM TG-017. The comments refer to the theoretical framework that I presented at the
meeting in Leiden, 2003. This framework is referred to as the Extended Control Model (ECOM). The model
was outlined in the presentation given at the meeting in Leiden, and has also been described in a conference
paper [1] and a book chapter in [2]. The pedigree of the model includes the description of the principle of
contextual control [3], the initial contextual control model [4], and the fully developed contextual control
model [5].

These comments do not include a detailed description of the model. For details, see Chapter 7 (Section 7.7).
Briefly explained, the model provides a framework for describing how a joint cognitive system can maintain
control of a situation or a process. A cognitive system is defined as “a system that can modify its behaviour on
the basis of past experience so as to achieve specific antientropic ends.” A joint cognitive system can be any
combination of humans and machines (technological artefacts) or humans and humans (social groups/
organisations). The model invokes the principle of multiple, simultaneous layers of control. The layers are
hierarchically organised with one or more instances at each layer. Control layers differ with respect to the time
window or time horizon they cover, as well as in the balance between feedback and feedforward control. The
model currently describes four layers of control called (from the top down) targeting, monitoring, regulating,
and tracking. It is applicable both to single systems (e.g., a driver and a vehicle) and to larger entities and
organisations.

I should make clear that my experience is primarily from applications outside the military, typically process
industries and transportation (surface, air, space), where I have been working with issues of automation,
human-machine interaction, and risk and reliability.

A3-2.1 FORCE MULTIPLICATION

• Does the framework/model address operator to UMV ratio issues?
• Does the framework/model address interoperability issues?
ECOM will enable a modelling of the joint system (i.e., multiple operators and multiple UMVs), thereby
making possible an evaluation of specific allocations. The question of a ratio as such cannot be answered,
since it will depend on the type of scenario, on situational demands (and resources), time constraints, urgency,
etc. ECOM will enable a constructive discussion of allocations for different types of scenarios, but not provide
any automatic answers – as there are none!

I am not certain what exactly is meant by interoperability issues. In computer science it refers to the ability to
exchange and use information at different locations or points in a system, such as a network. ECOM will not
address interoperability directly, but will give guidelines about which information is needed – at different
points – to maintain control of the joint system. In that sense interoperability is indirectly addressed. In cases
where maintaining control is critical, this may lead to considerations of redundancy (of control), hence to
interoperability issues.

RTO-TR-HFM-078 3 - 33
THEORETICAL FRAMEWORKS

A3-2.2 UMV SCENARIO / USE CASES

• Is the theory applicable to UMV situations (i.e., underwater, sea, land, air, space)?
ECOM is not domain specific, and can therefore be applied to different types of UMVs. As noted above,
it can also be applied to different levels of system aggregation, since the unit of analysis is the joint system.
To do so requires a clear definition of the system boundaries, since these are functional rather than structural
(i.e., not necessarily defined in relation to physical entities.)

A3-2.3 THEORY EVALUATION

• Has the framework/model been evaluated, tested, and applied to commercial or military operations?
• Do you have an example, closely related to UMV situations, where the theory was implemented?
An early version of ECOM has been used in a military context [6] The model has also been used in research
on joint driver-vehicle systems (automobiles) (Hollnagel, Nåbo & Lau (2003), and to model planning and
maintenance at a nuclear power plant during a short outage [7] An example closely related to UMV will be
developed during the fall, in collaboration with Robert Taylor and sponsored by FMV, Sweden.

A3-2.4 REFERENCES
[1] Gauthereau, V. and Hollnagel, E. (2004). Stepping away from the centralization/decentralization
struggle: planning, control, and adaptation at a Swedish nuclear power plant. European Management
Journal (in print).

[2] Hollnagel, E. (1993a). Models of cognition: Procedural prototypes and contextual control. Le Travail
humain, 56(1), 27-51.

[3] Hollnagel, E. (1993b). Human reliability analysis: Context and control. London: Academic Press.

[4] Hollnagel, E. (2002). Time and time again. Theoretical issues in Ergonomics Science, 3(2), 143-158.

[5] Hollnagel, E., Nåbo, A. and Lau, I. (2003). A systemic model for Driver-in-Control. 2nd International
Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design, July 21-24,
Park City, Utah.

[6] Hollnagel, E. and Woods, D.D. (2005). Joint cognitive systems: Foundations of cognitive systems
engineering. Boca Raton: CRC Press.

[7] Worm, A. (2001). Breaking the barriers: Facilitating efficient command and control in multi-service
emergency management. 8th World Conference on Emergency Management (TIEMS), June 19-22,
Oslo, Norway.

[8] Bonneh, Y.S., Cooperman, A. and Sagi, D. Motion-induced blindness in. normal observers. Nature 411,
798-801 (2001).

3 - 34 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Appendix 3-3: REQUEST FOR COMMENTS ON THEORETICAL

FRAMEWORKS FOR UNINHABITED MILITARY VEHICLES

Chris Miller, Ph. D.

Smart Information Flow Technologies
2119 Oliver Avenue South
Minneapolis, MN 55405-2440
USA
(612) 578-SIFT (7438)
(612) 374-4668

Reference: Action Items from Sweden Meeting, March 2004

The NATO RTO Human Factors and Medicine Panel Task Group 017 entitled, “Uninhabited Military
Vehicles (UMVs): The Human Factors of Augmenting the Force” is studying ways to augment military
Forces by leveraging the potential advantages of UMVs to act as force multipliers. A specific goal of the Task
Group is to produce a NATO report that would identify, prioritize, and address the Human Factors issues
associated with integrating UMVs into the Force. The Task Group (TG) believes that “Augmenting the Force”
will require research and development in the following areas:

Collaborative Work – Optimal Task Distribution

• Virtual team performance
• Manned/Unmanned collaboration
• Interoperability
• Flexible level of automation
• Optimization of operator/vehicle ratio

Control Stations – Intelligent Operator Support

• Operator functional state assessment
• Intelligent adaptive interfaces
• Cognitive cooperation
• Knowledge management systems

Simply put, the Force will be augmented by:

a) Reducing the operator/vehicle ratio; and
b) Improving interoperability, and both require Human Factors input.

The Task Group has organized the Human Factors of augmenting the Force into five primary theme areas
(Enclosed is a short synopsis of each):
• Theoretical Frameworks
• System of Systems

RTO-TR-HFM-078 3 - 35
THEORETICAL FRAMEWORKS

• Supervisory Control
• Levels of Automation
• Advanced Interfaces

Theoretical frameworks/models were identified from the literature as well as the Leiden Workshop held in
June, 2003 that may help in the development of the four other themes. However, we would like to solicit
comments directly from the authors on their framework’s relevance with respect to reducing operator/vehicle
ratio and improving interoperability.

Therefore, we invite you to comment on your theory with respect to the following:

A3-3.1 FORCE MULTIPLICATION

• Does the framework/model address operator to UMV ratio issues?
• If so, how could the framework/model help reduce the ratio?
PB allows flexible levels of control to be used by the operator depending on need, trust, workload, etc.
As such, in principle, it permits a single operator to do everything from joystick control to swarm control of
any number of UAVs. Of course, the level of aggregation of the behaviour defined by the “plays” which the
operator can plausibly use to control must be more aggregated the more vehicles controlled – you can
plausibly tell 300 UAVs to execute a “secure area” play, tell a team of 4 UAVs a specific flight route to fly to
maintain surveillance of a city block, or provide dynamic joystick inputs (which could be thought of as micro-
plays – e.g., “Actuate Flaps: 30 degrees”) for a single UAV, but not provide joystick inputs to 300 UAVs
simultaneously.

• Does the framework/model address interoperability issues?

• If so, how could the framework/model help improve interoperability?
Plays, especially higher level plays, can be defined as abstract functions to be fulfilled in alternate ways by
various combinations of variously capable UAVs – with the details associated with controlling the specific
UAVs buried in finer-grained, lower level plays. SIFT has, in fact, recently demonstrated the ability for a user
to command a comparatively high level play (“Overwatch” = sustained surveillance of an area) without
reference to the specific vehicle(s) which will provide it. When operating at the higher levels of the play
hierarchy, the user simply commands/requests a “service” and leaves it the PB to determine how to best
provide that service given the available UAV resources – in our demonstration, potentially multiple instances
of two types of heterogeneous UAVs (a rotary winged GT-Max and a fixed wing Dakota) which might have
different current locations, fuel resources, onboard sensors, etc. The user can also provide constraints that
limit the range of lower level plays that are acceptable – e.g., requiring that an infrared sensor be used for the
Overwatch and, thereby, eliminating from consideration any plans that would involve a UAV that doesn’t
have that sensor type. Thus, PB achieves interoperability by building the knowledge about how to utilize
different resources into PB’s Planner and leaving the user free to simply define the kind of service s/he
requires and leaving it to the PB to decide how best to provide it.

A3-3.2 UMV SCENARIOS / USE-CASES

• Is the theory applicable to UMV situations (i.e., underwater, sea, land, air, space)?

3 - 36 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Anywhere we can define useful aggregations/patterns of repeatable behaviours (and codify the knowledge
about how to achieve them) into a hierarchical play representation, PB is useful.

A3-3.3 THEORY EVALUATION

• Has the framework/model been evaluated, tested, and applied to commercial or military operations?
See Parasuraman, [1] for empirical evaluations of portions of delegation-style interactions. PB has been
applied in simulation to multiple UAV tasking scenarios and platforms, a UGV application and we are
currently working on developing an application using PB to allow a designer to “task” a human performance
simulation with play-like mission scenarios in order to do, for example, workload analyses during platform
design.

• Do you have an example, closely related to the UMV situations, where the theory was implemented?
Yes, several. Our current DARPA UAV project, PVACS is our richest implementation thus far.

In answering these questions, you may wish to compare and contrast your theory with traditional or other
approaches. Also, please include references to the theory if available (electronic copies preferable).
We anticipate that your response will be 1 to 3 pages, however there is no set limit.

A3-3.4 REFERENCES
[1] Miller, C. and Parasuraman, R. (in revision). “Designing for Flexible Human-Automation Interaction:
Playbooks for Supervisory Control.” Submitted to Human Factors.

[2] Parasuraman, R., Galster, S., Squire, P., Furukawa, H. and Miller, C. (submitted). A Flexible Delegation-
Type Interface Enhances System Performance in Human Supervision of Multiple Robots: Empirical
Studies with RoboFlag. Submitted for inclusion in IEEE Systems, Man and Cybernetics – Part A,
Special Issue on Human-Robot Interactions, Julie Adams (Guest Ed.).

[3] Squire, P., Furukawa, H., Galster, S., Miller, C. and Parasuraman, R. (Accepted) Adaptability and
flexibility are key! Benefits of the “Playbook” interface for human supervision of multiple unmanned
vehicles. To be included in Proceedings of the 2004 Meeting of the Human Factors and Ergonomics
Society, September 20-24, New Orleans, LA.

[4] Miller, C., Goldman, R., Funk, H., Wu, P. and Pate, B. (2004). A Playbook Approach to Variable
Autonomy Control: Application for Control of Multiple, Heterogeneous Unmanned Air Vehicles.
In: Proceedings of FORUM 60, the Annual Meeting of the American Helicopter Society, Baltimore,
MD, June 7-10.

[5] Miller, C. and Parasuraman, R. (2003). Who’s in Charge?; Intermediate Levels of Control for Robots
We Can Live With. In: Proceedings of the 2003 Meeting of the IEEE Systems, Man and Cybernetics
society, October 5-8, Washington, DC.

[6] Parasuraman, R., Galster, S. and Miller, C. (2003). Human Control of Multiple Robots in the RoboFlag
Simulation Environment. In: Proceedings of the 2003 Meeting of the IEEE Systems, Man and
Cybernetics Society, October 5-8, Washington, DC.

RTO-TR-HFM-078 3 - 37
THEORETICAL FRAMEWORKS

[7] Miller, C., Parasuraman, R., Lee, J., Corker, K., Johnson, L. and Schreckenghost, D. (2003). The Etiquette
Perspective for Human-Automation Relationships: Applications, Models and Results. In: Proceedings of
the 47th Annual Meeting of the Human Factors and Ergonomics Society, October 13-17, Denver, CO.

[8] Lintern, G. and Miller, C. (2003). Identification of Cognitive Requirements for New Systems.
In: Proceedings of the 47th Annual Meeting of the Human Factors and Ergonomics Society, October
13-17, Denver, CO.

[9] Miller, C. and Parasuraman, R. (2003). Beyond Levels of Automation: An Architecture for More
Flexible Human-Automation Collaboration. In: Proceedings of the 47th Annual Meeting of the Human
Factors and Ergonomics Society, October 13-17, Denver, CO.

[10] Miller, C., Goldman, R., Funk, H. and Parasuraman, R. (2003). Delegation Systems: Staying in Charge
of Highly Flexible Automation. In: Proceedings of the 10th International Conference on Human-
Computer Interaction, June 22-27, Crete, Greece.

[11] Miller, C., Goldman, R., Funk, H. and Parasuraman, R. (2002). Delegation as a Model for Human-
Automation Interaction. In: Proceedings of the 3rd International NASA Workshop on Planning and
Scheduling for Space, October 27-29, 2002, Houston, Texas.

[12] Miller, C., Pelican, M. and Goldman, R. (1999). High Level ‘Tasking Interfaces’ for Uninhabited
Combat Air Vehicles. In: Proceedings of the International Conference on Intelligent User Interfaces,
January 5-8, Redondo Beach, CA.

[13] Miller, C. and Goldman, R. (1997). “Tasking Interfaces; Associates that know who’s the boss.”
In: Proceedings of the 4th USAF/RAF/GAF Conference on Human/Electronic Crewmembers, September
22-26, Kreuth, Germany.

3 - 38 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Appendix 3-4: REQUEST FOR COMMENTS ON THEORETICAL

FRAMEWORKS FOR UNINHABITED MILITARY VEHICLES

Patrik Stensson
Swedish Air Force

The theory “The Human Axiom” or

“The Philosophical Framework for Military Relevance”

A3-4.1 FORCE MULTIPLICATION

• Can the framework/model help in reducing the operator to UMV ratio?
Yes, but only indirectly through the ability of assessing the requirements for having appropriate human
control, the requirements for making the human be appropriately in charge. Not unlikely, it will help in
increasing the operator to UMV ratio instead of reducing it! And, the main issue is that the theory argues
precisely that the most valuable military effect is related to appropriate human control, and on the contrary,
inappropriate human control leads to a reduced military effect. That is, the theory states that gain of military
effect, force multiplication, is achieved by having appropriate human control. This is the concept of designed
and applied effect.

• How could the framework/model help reduce the operator to UMV ratio?
It may help in establishing the prerequisites for human control from an operational perspective, in order to
make technological solutions military relevant.

• Can the framework/model help with interoperability issues?

Yes, since interoperability consists of a significant amount of human interoperability, appropriate human
control is necessary to achieve interoperability.

• How could the framework/model help with interoperability issues?

See above…

A3-4.2 UMV SCENARIOS / USE-CASES

• Does the theory cut across UMV situations?
Yes, it’s completely general.

A3-4.3 THEORY EVALUATION

• Is the framework/model implementable and testable?
I don’t know

• Do you have an example, closely related to the UMV situations, where the theory was implemented?
No!!!!!!!!!!!!!!!!!!!!!

RTO-TR-HFM-078 3 - 39
THEORETICAL FRAMEWORKS

Appendix 3-5: REQUEST FOR COMMENTS ON THEORETICAL

FRAMEWORKS FOR UNINHABITED MILITARY VEHICLES

Phillip Farrell, Ph. D.

A3-5.1 DESIGN IMPLICATIONS FOR MULTIPLE AGENT SYSTEMS

As machines become more “intelligent”, humans will insist on interacting with them as they do with other
humans, otherwise humans may reject the intelligent agent technology that is expected to be part of future
UMV systems. In this context agents may be human as well as software.

A framework is proposed based on Perceptual Control Theory [1] to model multiple non-interacting and
interacting agents in order to understand the system dynamics that would lead to design implications for user-
intelligent agent interaction [2]. PCT describes human cognition as a means-end hierarchy of control units.
Each control unit involves the control of a perception. At the lowest levels, control is subconscious and can
might be described by classical linear control theory. At the highest levels, control is conscious, and deliberate
requiring rule-based thinking, logic, and reasoning. Regardless of the hierarchical level, the control law
remains the same – the output of each control loop attempts to drive the perception closer to its internal goal.
The key advantage of modelling agents using PCT is that one can apply all the mathematical power of Control
Theory, which includes stability and optimization analyses.

It is assumed that multiple agent interaction is more like human-human interaction than it is like human-
machine interaction. Table A3-5.1 lists some key differences:

Table A3-5.1: Critical Differences between Human-Machine and Multiple Agent Interaction

Human – Machine Interaction (HMI) Multiple Agent Interaction (MAI)

Plans, actions and system states are known or Plans and actions are not necessarily known a priori, and
knowable within limits. may produce unexpected system states.

HMI is specific, systematic, and often MAI is fuzzy and there may be many means to achieve
associated with Standard Operating the same end.
Procedures.

Human has beliefs (assumptions) about the One agent has beliefs (assumptions) about other agents
machine and the task. The machine design and the task, and those beliefs may be hierarchical and
takes into consideration certain assumptions dynamic.
about the human and the task.

Typically trust is binary – the machine works Trust must be built over time because it becomes more a
or the machine does not work. function of mission completion and success rather than
based on an individual agent’s work.

Typically there are two levels of automation – Potentially there could be a gradient of automation.
fully manual or fully automatic.

3 - 40 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Human – Machine Interaction (HMI) Multiple Agent Interaction (MAI)

It is possible to know how the system It may be difficult, if not impossible to trace information
processes information (i.e., outputs can be flows in and amongst multiple agents.
reconstructed from the inputs).

The context is typically static or well-defined. The context is dynamic and sometimes unknown a priori.

Only the human is truly autonomous. Potentially there could be multiple truly autonomous
agents.

From a control theory perspective, human-machine interaction may be analysed by treating the machine as a
simple input-output transfer function with some known disturbances – this problem is relatively easy to solve.
On the other hand, multiple agent interaction must be treated as independent (truly autonomous) control loops
that interact through a common portion of the world (often called the interface). Conceptually, this model
would produce a different set of dynamics that are unstable and not easily predictable. The clear advantage is
that multiple agents have the potential to multiply the force.

Two interacting agents as well as a generic multiple agent model was analysed using mathematical control
theory techniques and the following are a list of design recommendations that come from this modelling
exercise:
• Closed-loop feedback modelling techniques can be used in the design of multiple agent systems.
• Designers should consider goals, sensing and decision-making strategies, and world states as part of
their system design.
• Agents should act on separate states, while gathering data from all sources.

These design principles have been applied to UMV research studies on the design of intelligent adaptive
interfaces, and selecting crews for UAV operations. The third design philosophy was applied to information
management business rules for the Multi-National Experiment 4 on Effects Based Operations with US Joint
Forces Command as the experiment lead. This experiment involved over one hundred players over a
distributed network from countries around the world. The interface design and business rules were critical in
order to successfully collaborate and conduct the experimental operation.

This paper models intelligent agents using Perceptual Control Theory. Using standard mathematical control
theory techniques, conditions for stability are derived for two-agent and multiple agent interactions.

A3-5.2 FORCE MULTIPLICATION

• Does the framework/model address operator to UMV ratio issues?
Yes. The theory has the potential to determine, a priori, conditions for stability for any operator/vehicle
configuration.

RTO-TR-HFM-078 3 - 41
THEORETICAL FRAMEWORKS

• If so, how could the framework/model help reduce the ratio?

The theory can show that as one increases the number of intelligent agents, there will be as many regions of
local instability as there is stability. Thus, the design would need to be constrained more and more in order to
maintain a stable trajectory through the state space. The theory has the potential to show optimal trajectories
through the state space as the ratio is reduced.

• Does the framework/model address interoperability issues?

Yes.

• If so, how could the framework/model help improve interoperability?

One of the design implications of this stability analysis is that multiple agents can gather information from
each other and the world, but must act on separate parts of the world. This principle is best illustrated with a
wheel barrel example. Two people pushing one wheel barrel may have similar goals and perceptions, but if
their actions are not precisely coordinated then the system will quickly become unstable. Similarly, two forces
may want the information from sensors onboard a UMV, but if they attempt to control the vehicle and/or
sensors simultaneously (or even time diviplexed) there is great potential for conflicts. What this means for
interoperability is that standards might be required to ensure proper sensing of information, and action must be
coordinated (perhaps proceduralized) to ensure de-confliction.

A3-5.3 UMV SCENARIOS / USE-CASES

• Is the theory applicable to UMV situations (i.e., underwater, sea, land, air, space)?
Yes. The paper describes generic multiple agent interactions. It does not distinguish between animate and
inanimate, human or machine. In order to do the mathematics, however, it does make a fundamental
assumption of linearity – and we know that these UMV situations are not linear. Control theorists have
methods to deal with non-linear systems including piece-wise linearization. Thus the linear stability
conditions only approach the real stability conditions as the linear pieces reduce in size. Lyapunov approaches
to deriving stability conditions provide a global solution to nonlinear dynamical systems, however, the
challenge is to derive (often stumble onto) a Lyapunov function that satisfy certain energy criteria. I think the
best the Theory can do is provide heuristics, but it can provide it to any UMV situation.

A3-5.4 THEORY EVALUATION

• Has the framework/model been evaluated, tested, and applied to commercial or military operations?
Yes. This first application of these design considerations hope to be in the Adaptive Intelligent Agent project
conducted by DRDC Toronto. The theory will also be applied to a UAV crew selection methodology.
Also, the theory has been applied to the business rules for Multi-National Experiment 4. That is, all can view
information, but only specific people have write permissions to the database. A multiple UAV interface has
been designed [3] based on this framework, and will be experimented on in 2006.

• Do you have an example, closely related to the UMV situations, where the theory was implemented?
See above.

3 - 42 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

A3-5.5 REFERENCES
[1] Powers, W.T. (1973). Behavior: the control of perception. Chicago: Aldine.

[2] Farrell, P.S.E. (2003). Feedback in Error-correcting and Error-limiting Systems. In: Proceedings of the
50th AGM & Conference of Canadian Aeronautics and Space Institute, Montreal, Quebec, April 2003.

[3] Forbes, K., Baker, K. and Youngson, G. (2005). Human-computer interaction between the operator and
the semi-autonomous UAV team. Contract Report, DRDC Ottawa, Ottawa, Canada.

RTO-TR-HFM-078 3 - 43
THEORETICAL FRAMEWORKS

Appendix 3-6: REQUEST FOR COMMENTS ON THEORETICAL

FRAMEWORKS FOR UNINHABITED MILITARY VEHICLES

Dr. Axel Schulte

A3-6.1 FORCE MULTIPLICATION

• Does the framework/model address operator to UMV ratio issues?
The framework advocated by the Munich University of the German Armed Forces, represented by
Prof. Dr. Axel Schulte shall be denoted as Cognitive Automation (CA). CA as itself does not address operator
to UMV ratio issues explicitly since it is a general approach towards improving system performance, to begin
with. The basic idea behind CA is to mimic human performance including knowledge-based behaviour on the
machine side by a knowledge-rich system incorporating explicit goals for acting, a comprehensive situation
understanding and planning/problem-solving, decision-making and acting on behalf of the machine. In turn,
such a cognitive system being built following an according specification, will certainly affect the operator to
UMV ratio. There are several ways to introduce CA in to a work system (i.e., operator(s) controlling
UMV(s)). This can be done either as cognitive assistant system, supporting the operator’s work tasks by
taking workload off the operator, or as an autonomously acting unit taking over certain comprehensive tasks
and teaming up with the human. So, properly designed CA will supplement human teams and, thereby, bear
the potential to affect operator to UMV ratio.

• If so, how could the framework/model help reduce the ratio?

If this is a goal of automation, which certainly is true in many cases, the answer is yes.

• Does the framework/model address interoperability issues?

One characteristic of CA as defined above is the explicit representation and processing of domain relevant
knowledge including models for situation understanding and goals for acting. With respect to the
interoperability issue, there might be distinguished between several levels of interoperability, i.e., on the low
level of common protocols (e.g., Link 16) as well as on the high level of a common understanding of
situations, of tasks, and of rules of engagement within situational contexts. CA, enriched by respective domain
knowledge, will in principle be able to handle problems covering the full scope of these levels and, in turn,
will be enabled to take over tasks from humans in a supportive or even replacing manner. Thereby, the effect
of CA upon interoperability issues can be a tremendous one. Hence, the problem of knowledge elicitation and
knowledge representation is still an active research issue.

• If so, how could the framework/model help improve interoperability?

To answer this question, it should be made clear, that CA is a systems engineering approach to intelligent
systems design. The effectiveness of such a system is solely determined by the right ontology represented by
the knowledge put in during the design process. Properly designed, we can expect a positive effect upon
interoperability.

A3-6.2 UMV SCENARIOS / USE-CASES

• Is the theory applicable to UMV situations (i.e., underwater, sea, land, air, space)?

3 - 44 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

Simple answer, yes, since CA is a general framework. Of course, each of the mentioned domains has got it’s
own peculiarities. But, they all have in common, that there will be a complex dynamic situation, tactical or
whatsoever, needed to be interpreted and understood, that there are goals for acting to be considered, and that
there is some demand upon problem-solving and decision-making. Again, it is a matter of the kind of
knowledge you put into the system, whatsoever functionality will be provided. Currently the concept has been
proven in the aeronautical domain.

A3-6.3 THEORY EVALUATION

• Has the framework/model been evaluated, tested, and applied to commercial or military operations?
Yes, here is a very brief summary of activities:
• CASSY – Cockpit Assistant System: World’s first comprehensive knowledge-based pilot assistant
system for civil air transportation. Functional prototype successfully flight tested in 1994 with
operational airline captains.
• CAMA – Crew Assistant Military Aircraft. Follow up activity of CASSY designed for military
tactical air transport missions, including tactical situation assessment and low-level flight guidance
functions. Simulator tested in 1998. Flight tested in 2000 with operational German Air Force pilots.
Some minor product spin-off for the A 400 M program.
• TIMMS – Tactical Information and Mission Management System. Carries many of the CAMA ideas
into the Air-to-Ground attack domain, covering operator support in an networked air warfare
scenario, addressing some interoperability issues. TIMMS was simulator tested in 2000 as a
functional prototype by air force pilots. Later on it was adapted to a Eurofighter cockpit avionics
environment and some certification issues were addressed.
• PILAS – Assistant system for helicopter emergency medical service mission. CA serves a general
approach to the system design.
• MiRA – Forthcoming project on Military Rotorcraft Assistant.

• Do you have an example, closely related to the UMV situations, where the theory was implemented?
Yes, currently we are working on some UAV activities, in brief:
• COSY flight – Cognitive System for the flight domain. Autonomous guidance system for a single-
ship URAV mission. Project was set-up in order to prove concept of the implementation framework
COSA (Cognitive System Architecture). Finished by now.
• Currently COSA is being used for the implementation of an autonomous guidance system for a multi-
ship SEAD UCAV mission. Main focus is the implementation of machine-machine co-operation
capabilities according to the approach of CA. Will be tested in simulation.
• Development of a mini-UAV field demonstration of cognitive and co-operative capabilities on the
machine side as well as intelligent operator assistance in a ground control station. The vehicles are
one model-scale rotorcraft UAV and one fixed-wing.
• Manned-unmanned teaming – forthcoming project on integration of UAVs into military helicopter
missions in a network centric operations context.

References to particular publications can be provided on demand.

Prof. Dr. Axel Schulte. Bath, UK. May 2005.

RTO-TR-HFM-078 3 - 45
THEORETICAL FRAMEWORKS

Appendix 3-7: REQUEST FOR COMMENTS ON THEORETICAL

FRAMEWORKS FOR UNINHABITED MILITARY VEHICLES

Iain S. MacLeod
Defence College of Management and Technology
UK Defence Academy
Shrivenham

System Process/Task Organisational Model for HF V&V

A3-7.1 INTRODUCTION
The author believes that if the Task Group 017 remit is truly to be “The Human Factors of Augmenting the
Force” we should produce outcomes that have enough practicality to allow their consideration applicability in
augmenting force effectiveness in the ‘real’ world. For that reason I believe that the following should apply:
• Theoretical Frameworks should be formed from Theoretical Models and should have enough
substance that they can be tested in reality [believe that Leiden workshop had a strong emphasis on
Models].
• Following from the above, any Theoretical Model should be based on generic truth(s) underlying
human systems performance in the real world [both inhabited and uninhabited] in that it can be seen
to encompass any associated theoretical framework. Whilst, the use of UMVs within a network
enabled community will still involve the issues of interoperability that are currently recognised [1,2]
(for example, shared situational awareness, trust, the syntactic, semantic, and pragmatic), for UMVs
there will be some important nuances and differences to the issues as associated with inhabited
systems.
• It is likely that if a Theoretical Models are ‘true’ there will be few, maybe only one!
• A Theoretical Model architecture must address its capability to support for a Theoretical Framework
architecture or architectures.
• Theoretical Frameworks are only theoretical until tested when they then become a manifestation/
representation of some multi-dimensional aspect of reality.
• If a framework is to be related to reality it should consider selective combinations of Process,
Context, Situation, Mission, Goal(s) [possibly both Strategic and Tactical], Force Structure,
Organisations, Cultural influences, the role(s) of technology, required system functions and
performance, considered level of abstraction, temporal issues, roles of personnel, jobs, teamwork,
tasks, and individual actor / entity properties and activities. One example considering generic mission
process associated target recognition tasks is introduced in Reference 3.
• For the considered application of any framework the prime underlying tenet(s) to be examined should
be made specific, i.e., Command, control, workload, co-ordination, teamwork quality to name but a few.
• It is important that the consideration [possibly a verification and validation process] of a framework is
made both statically and dynamically – statically to ensure that it is basically logical and fit for
intended purpose, dynamically to examine its fit to reality. Here the role of forms of Measures of
Effectiveness and Performance are important.

3 - 46 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

A required quality of human work is always associated with an understanding and acceptance of work goals,
team and/or individual capabilities, appropriate skills, and a relevant knowledge plus the means to apply that
knowledge within the time constraints and environments influencing the work.

A3-7.2 THE SYSTEM PROCESS / TASK ORGANISATIONAL MODEL FOR

HF V&V
The System Process/Task Organisational Model for HF V&V presented at Leiden was shown using an
embedded Theoretical Framework of Task Organisation. In the author’s view that framework always implies
levels of consideration on selected combination as discussed previously.

One of the problems in the understanding of human work is the meaningful association of theoretical
interpretations [or hypothetical constructs] with a needed understanding for the application of human work
practices. It is argued that the association of generic process with active practises is one avenue towards that
understanding. Furthermore, derived system functions, constraints, and architecture support the quality of
satisfaction of the process(es) existing to allow system capability and not necessarily a system goal.
In contrast, consideration at the task level (whether that is at a higher level consideration of force/system tasks
or at the level of individual human tasks) always implies a goal driven application of appropriated effort to a
selection of available functions. Thus, if a system is considered dynamically it is with a task related
perspective; if statically it is more at a function/constraint/architecture/required capability system perspective
of consideration or the consideration of an underlying generic process. For a high level discussion on this area
see Reference 3.

A3-7.3 QUESTION / RESPONSE

Considering the issues that are to be addressed with relation to the proposed theory or Theoretical Model,
Table A3-7.1 gives some high level address.

RTO-TR-HFM-078 3 - 47
THEORETICAL FRAMEWORKS

Table A3-7.1: Response of ‘System Process / Task Organisational Model for HF V&V’ to Questions

Question Response on MODEL

Force Multiplication
Does the framework/model address operator to UMV Yes.
ratio issues?
By allowing consideration as required through
If so, how could the framework/model help reduce the frameworks considering/evaluating issues such as
ratio? Automation, Autonomy, Manning, and Personnel
within the force to examine questions of force
effectiveness against such as operator/UMV ratios.
Does the framework/model address interoperability Frameworks would need to be evolved to address
issues? specifics of interoperability and other issues.
However, the model does allow for that in that the
If so, how could the framework/model help improve
‘Task Organisational Model’ example of a
interoperability?
Framework within the overall model can be
developed to consider the particular tenets to be
examined. Interoperability is a many layered issue
depending on force structure, the organisational
level of the force being considered, the
standardisation of procedures, and communication
protocols [human and machine] to name but a few
issues [1,2].
UMV Scenarios/Use-Cases
Is the theory applicable to UMV situations (i.e., The model is generic enough to encompass any
underwater, sea, land, air, space)? framework considered scenario or its address
through particular Use Cases.
Theory Evaluation
Has the framework/model been evaluated, tested, and Yes – against many, but applications mainly
applied to commercial or military operations? implicit rather than explicit.
Do you have an example, closely related to the UMV No – examples only with relation to manned
situations, where the theory was implemented? systems, but within coalition operations it is
possible to consider individual members of the
coalition as semi autonomous and therefore to have
some properties similar to UMVs.

Some forms of Framework Models (as apposed to Local Models!) already suggested can be substituted/
incorporated into the Theoretical Framework level onto the Process of the ‘System Process/Task
Organisational Model for HF V&V’ Theoretical Model – examples are:
• (Perceptual) Control Theory approaches;
• Delegation using Task Hierarchy;
• Responsibility – Authority – Competency;

3 - 48 RTO-TR-HFM-078
 THEORETICAL FRAMEWORKS

• Layers of Control;
• Logical/Intuitive channel model for inducing trust;
• Systems Engineering Approaches;
• Theories of agency;
• Mission Data System (software framework, goals, states and constraints); and
• Rule-based automation (recognition-primed decision-making).

It is the generating of agreed rules for applying particular Framework Models to the generic Process/Task
Organisational Model that have to be formulated.

A3-7.4 AUGMENTING REALITY

Back again to real world reality – resulting from the form of Theoretical Models and Frameworks that are
adopted we should decide on the form/type of scenarios, functions (plus associated performance), constraints,
and capabilities of our UMV from the HF perspective. We now have to bring these into the reality of an
engineering life cycle for the UMV system.

Here we should be examining the System Engineering (SE) Models and their suitability to adopt the UMV HF
properties we have arrived at. All SE models are poor at considering HF at the early phase(s) of the Life Cycle
(i.e., variously termed Concept/ user requirements definition/advanced studies and preliminary analysis, etc.).
If HF is to become a mainstream discipline in its effects on the quality of systems design it must become part
of the logical specification of system functional requirements (currently HF mainly dwells on system
Constraints) and have influence on the system architecture. At the moment HF mainly dwells on attending
retrospectively to the PRODUCTS of others work. Thus it is almost impossible to make an HF CASE (like a
Safety Case or R&M Case) as there are no requirements to which HF can trace and show its progressive
address up to system acceptance and beyond.

Some SE Standards worth a look (in the order of their greatest attention to HF) are:
• INCOSE Systems Engineering Handbook, V2, July 2000;
• IEEE 1220-1988, Standard for Application and Management of the Systems Engineering process,
Full Standard version 1.0, July 27 1998; and
• ISO 15288, WD4, Life Cycle management – System life Cycle Processes, 16 February 1999.

A3-7.5 REFERENCES
[1] Whitworth, I.R. (2005). The Systems Design Challenge of NEC. In: Proceedings of People and Systems
Symposium, Institute of Electrical Engineers, London, November.

[2] Schade, U. (2005). Towards the Edge and Beyond: The Role of Interoperability. In: Proceedings of 10th
International Command and Control Research and Technology Symposium (ICCRTS), MacLean, Virginia,
June.

[3] MacLeod, I.S., Hone, G, and Smith, S. (2005). Capturing Cognitive Task Activities for Decision Making
and Analysis. In: Proceedings of 10th International Command and Control Research and Technology
Symposium (ICCRTS), MacLean, Virginia, June.

RTO-TR-HFM-078 3 - 49
THEORETICAL FRAMEWORKS

3 - 50 RTO-TR-HFM-078