INTERNATIONAL JOURNAL OF DECISION SUPPORT SYSTEM TECHNOLOGY

Volume 10, Issue 1, January-March 2018

([Link]/[Link]?id=190826)

Checking the consistency of solutions in decision-making problems with

multiple weighted agents
Domenico Maisano1 and Luca Mastrogiacomo2
1
[Link] @[Link] 2 [Link]@[Link]
Politecnico di Torino, DIGEP (Department of Management and Production Engineering),
Corso Duca degli Abruzzi 24, 10129, Torino (Italy)

Abstract
A decision-making problem diffused in various practical contexts is that of aggregating multi-agent
judgements into a consensus ordering, in the case the agents’ importance is expressed through a set
of weights. A crucial point in this aggregation is that the consensus ordering well reflects the input
data, i.e., agents’ judgements and importance. The scientific literature encompasses several
aggregation techniques, even if it does not include a versatile tool for a quantitative assessment and
comparison of their performance.
The aim of this paper is introducing a new indicator (p), which allows to verify the degree of
consistency between consensus ordering and input data. This indicator is simple, intuitive and
independent from the aggregation technique in use; for this reason, it can be applied to a variety of
practical contexts and used to compare the results obtained through different aggregation
techniques, when applied to a specific problem. The description is supported by various application
examples.

Keywords: Multiple agents, Consensus ordering, Aggregation, Paired comparison, Consistency, Weight.

1
1. Introduction
A very general decision-making problem is that of aggregating multi-agent judgments, concerning
different alternatives, into a consensus ordering. The adjective “consensus” indicates that this
ordering should reflect agents’ judgements as much as possible, even in the presence of
divergences. Summarizing, the problem is characterized by the following elements (see the scheme
in Fig. 1):
 A set of alternatives to be prioritized (a, b, c, d, e, ...);
 A set of decision-making agents1 (D1, D2, …, DM) expressing their opinion on the alternatives,
through various possible forms (e.g., paired-comparison judgements, evaluations/measurements
on ordinal/interval/ratio scales, linear/partial preference orderings, etc.);

INPUT D A T A: OUTPUT D A T A:
- A set of alternatives (a, b, c, …);
- A set of agents (D1, D2, …, DM) formulating AGGREGATION Consensus ordering
judgements on the alternatives; TECHNIQUE of the alternatives
- A set of importance weights of the agents.

Fig. 1. Input/output data concerning the general problem of the aggregation of multi-agent judgements on a set
of alternatives, into a consensus ordering.

 An importance hierarchy of agents, which is usually expressed through a set of weights or

importances (r1, r2, …, rM), defined on a ratio scale by a team of experts (Vora et al., 2014);
 A consensus ordering of the alternatives, which represents the output of the problem.
The problem of interest is quite old and has been studied in various fields, stimulating the
development of a variety of aggregation techniques. For example, in the field of social choice and
voting theory, we recall the method by Condorcet and that by Borda (Borda, 1781; Franceschini et
al., 2007); in the field of multicriteria decision making, the Electre (Figueira et al., 2005),
Promethee (Brans and Mareschal, 2005) or AHP (Saaty, 1980) methods, in the field of the Internet
intelligent agents, that by Yager (2001 ), etc..
Each of these techniques has its pro and contra; based on this consideration, an interesting question
may arise: For a generic decision-making problem, how could we identify the best aggregation
technique(s)? It is probably impossible to answer this question, since the “true” solution for a
generic problem is not known a priori (Figueira et al., 2005; Cook, 2006). Nevertheless, the fact
remains that one technique may be more or less appropriate than one other depending on: (i) the
practical purpose of the aggregation (for example, isolating the best alternative or a limited number
of excellent alternatives, excluding the worst alternatives, defining a complete ranking, etc.), (ii) the

1
By a decision-making agent, we will consider any of a wide variety of different types of entities; examples could be
human beings, individual criteria in a multicriteria decision-making process, intelligent entities in the field of artificial
intelligence, etc..

2
form in which the agents’ judgments and/or the importance hierarchy are expressed, and (iii) the
ability to encourage the involvement and participation of decision-makers in constructing a shared
solution (Zopounidis and Pardalos, 2010).
Also, techniques can be differentiated on the basis of various functional aspects, such as: degree of
simplicity of evaluations/elaborations required by the agents, efficiency in the use of the
information available, computational complexity, etc.. For a detailed description of these aspects,
we refer the reader to the relevant scientific literature and extensive reviews (Brans and Mareschal,
2005; Herrera-Viedma et al., 2014).
In the rest of this paper, we will not discuss the different aggregation techniques, which are seen as
“black boxes” able to transform the agents’ judgments (input) into a consensus ordering (output).
The goal of this paper is introducing a new indicator, denominated p, which allows a rough
verification of the consistency of the solution provided by a certain aggregation technique, in a
simple and intuitive way. This type of verification is important for at least two reasons:
1. Despite its seemingly simplicity, the aggregation problem of interest is surprisingly complicated,
since it has been demonstrated that finding the “optimal” consensus ordering (assuming that it is
possible) is NP-hard under certain conditions (Dwork et al., 2001). In order not to make the
problem computationally burdensome, it can be convenient to use relatively simple and user-
friendly techniques, as long as they are accompanied by practical tools for verifying the
effectiveness of the solution (Nuray and Can, 2006; Akriditis et al. , 2011);
2. A verification tool may assist in selecting, the best suited aggregation techniques to a specific
decision problem.
In the scientific literature various verification tools have been proposed. A common feature is that
they use some measures of correlation/similarity to compare the fused ordering with agents’
judgements (Ng and Kantor, 2000; Wu and McClean, 2006). For example, popular statistics are the
Spearman’s rank, the Kendall’s tau, and measures of likelihood/distance. Unfortunately, these tools
are generally designed for verifying the solution of specific aggregation techniques and, for this
reason, their range of application is limited by several aspects, such as:
 the form in which agent’s judgements are expressed;
 the degree of “completeness” of judgments; for example, many techniques are not applicable
when some alternatives are omitted or incomparable between each other;
 the form in which the importance hierarchy of the agents is expressed.
The proposed indicator is relatively versatile and practical, as we will show later in the paper. A
not-so-dissimilar tool was proposed for a more specific decision-making problem, in which the
agents’ importance is expressed in the form of a rank-ordering (Franceschini and Maisano, 2015a).
The remainder of the paper is organized in three sections. Sect. 2 introduces the indicator p,

3
focusing on its construction and practical use. Sect. 3 exemplifies the application of p in three
different types of decision-making problems. Finally, Sect. 4 presents a detailed discussion of the
new indicator, focussing on its strengths and limitations.

2. Description of p
Before getting into the discussion of p, we anticipate that it is virtually adaptable to a generic
aggregation technique, since it is mainly based on the comparison between (i) the paired-
comparison relationships derived from one agent’s judgements and (ii) those derived from the
consensus ordering. The decision of using paired-comparison relationships is motivated by several
reasons:
1. They allow to express the preference between two alternatives in a natural and intuitive way;
2. They can be derived from most of the forms in which agent’s judgments are typically expressed
– provided that they admit relationships of ordering among the alternatives; Fig. 2 shows several
examples of paired-comparison relationships resulting from different types of judgments. This
feature makes p potentially adaptable to a large amount of practical contexts;
3. They may also be obtained in the case some judgments are incomplete, i.e., they do not include
all the alternatives (e.g., see case (ii) in Fig. 2) or admit incomparabilities between some
alternatives (e.g., see case (v) in Fig. 2).

Form of the judgements Paired-comparison relationships

(i) Evaluations on an ordinal scale

a = Excellent b = Good a > b, a > c, a > d, b > c, b > d, c ~ d
c = Poor d = Poor

(ii) Evaluations on an interval/ratio scale

a = 1.52 b = Omitted judgement a || b, c > a, d > a, c || b, d || b, c > d
c = 5.18 d = 3.12
(iii) Paired-comparison on a ratio scale
a = 2∙b a = 4∙c a > b, a > c, d > a, b > c, d > b, d > c
a = 2/3∙d b = 2∙c
b = 1/3∙d c = 1/2∙b

(iv) Linear ordering a > b, a > c, a > d, b ~ c, b > d, c > d

a > (b ~ c) > d

(v) Partial ordering

a > [(b ~ c) || d] a > b, a > c, a > d, b ~ c, b || d, c || d

Fig. 2. Transformation of judgments expressed in various forms, into paired-comparison relationships.

Alternatives of interest are a, b, c and d; symbols “>”, “~” and “||” respectively mean “strictly preferred to”,
“indifferent to” and “incomparable to”.

We are aware that transforming judgments into paired-comparisons can cause a loss of information,
especially when these judgments are expressed on cardinal (i.e., interval or ratio) scales. However,
this transformation makes the proposed indicator applicable to a wide variety of practical contexts.

4
A preliminary operation for determining p is the construction of a table, which contains the paired-
comparisons obtained from the agents’ judgments and the consensus ordering. For the purpose of
example, let us consider a decision-making problem in which m = 4 agents formulate their
judgments concerning n = 5 alternatives (a, b, c, d and e). No matter in what form judgements are
expressed, as long as they can be turned into paired-comparison relationships like a>b, b>a, a~b or
a||b, where symbols “>”, “~” and “||” respectively mean “strictly preferred to”, “indifferent to” and
“incomparable to”. Through some aggregation technique (no matter what), we assume that agents’
judgements are aggregated into a consensus ordering d>a>b>c>e.
Agents’ judgements are then transformed into the sets of paired-comparison relationships reported
in Tab. 1(a). Likewise agents’ judgements, the consensus ordering is also transformed into a set of
paired-comparison relationships (see the last column of Tab. 1(a)).
Each j-th agent is associated with an indicator of importance (rj) and an indicator (cj) corresponding
to the number of paired-comparisons usable for evaluating the compatibility between agents’
judgments and consensus ordering. Conventionally, we classify as “usable” a paired comparison not
producing any relationship of incomparability (“||”), neither in the agent’s judgements, nor in the
consensus ordering, but only relationships of strict preference (“>”) or indifference (“~”).
Obviously, c j  C2n  j-th agent, being C2n the total number of paired-comparisons for n generic

alternatives (e.g., 10 in this specific example, since n = 5).

Tab. 1. (a) Table of paired-comparison data related to agents’ judgments and consensus ordering (i.e.,
d>a>b>c>e), in a fictitious decision problem; agents are sorted decreasingly with respect to their importance (rj).
(b) Corresponding consistency table.
(a) (b)
Paired From agents’ judgements From consensus Paired D1 D2 D3 D4
comparison D1 D2 D3 D4 ordering comparison
a, b a>b a || b a>b b>a a>b a, b 1 N/A 1 0
a, c a>c a>c a>c c>a a>c a, c 1 1 1 0
a, d a~d d>a a>d a || d d>a a, d 0.5 1 0 N/A
a, e a>e a>e a>e a || e a>e a, e 1 1 1 N/A
b, c b>c b>c b>c b>c b>c b, c 1 1 1 1
b, d d>b d>b b~d b || d d>b b, d 1 1 0.5 N/A
b, e b>e b>e b>e b>e b>e b, e 1 1 1 1
c, d d>c d>c d>c c || d d>c c, d 1 1 1 N/A
c, e c>e e>c c~e c>e c>e c, e 1 0 0.5 1
d, e d>e d>e d>e d || e d>e d, e 1 1 1 N/A
rj 0.40 0.30 0.15 0.15 cj 10 9 10 5
xj 9.5 8 8 3
pj 95% 88.9% 80% 60%
wj 4 2.7 1.5 0.75
Note: rj is the agent’s importance; cj is the number of “usable” paired-comparisons; xj is the agent’s total score; pj = xj/cj is the portion of
consistent paired-comparison relationships; wj = rj∙cj is the agent’s weight.

Subsequently, we construct a “consistency table” (in Tab. 1(b)), which turns the paired-comparison
relationships of each agent into scores, according to the convention in Tab. 2. The conventional
choice of assigning 0.5 points in the case of weak consistency is justified by the fact that this is the
intermediate case between that one of full consistency (with score 1) and that of inconsistency (with

5
score 0). Through a sensitivity analysis, we found that small variations in this score (e.g., using 0.3
or 0.7 instead of 0.5) have little impact on the resulting consistency indicators.
The consistency table also reports the sum of the scores (xj) relating to each j-th agent.

Tab. 2. Scores related to paired-comparison relationships of one agent, in the construction of the consistency
table.
Case Score
1. Full consistency, i.e., identical relationship of strict preference (“>”) or indifference (“~”); e.g., when 1
comparing the relationship a > b with itself.
2. Weak consistency, i.e., consistency with respect to a weak preference relationship only (“> or ~”, i.e., 0.5
strict preference or indifference); e.g., when comparing the relationship a > b with a ~ b.
3. Inconsistency (with respect to both strict and weak preference relationships); e.g., when comparing the 0
relationship a > b with b > a.
4. Incomparability between the two alternatives in the agent judgments and/or in the fused ordering. N/A

Next, the portion of consistent paired-comparisons can be calculated for each j-th agent as:
xj
pj  , (1)
cj

where: xj is the total score related to the j-th agent;

cj is the number of usable paired comparisons related to the j-th agent.
It can be noticed that the non-usable paired comparisons do not influence the evaluation of
consistency, since they have no contribution neither in the xj nor in the cj terms.
Next, the pj values of the different agents can be aggregated into p, by means of a weighted average
based on a set of weights (wj), defined as:
wj = rj ∙ cj . (2)
The proportionality with respect to the cj values is justified by the fact that it seems reasonable that
the most consolidated pj values, i.e., those calculated using a larger number of paired comparisons,
contribute more to p. The proportionality with respect to the rj values is justified by the fact that it is
expected that the higher the importance of agents, the higher the consistency between consensus
ordering and judgements. This condition does not entail that the judgements by the less important
agents should not to be consistent with the consensus ordering, but just means that the consistency
of the more important agents counts more than that of the other ones.
Based on the above considerations, the synthetic indicator p is defined as:
   
  m  
p    p j  m
wj    p  r j  c j  ,
m

   j m  p  0 ,1 . (3)
j 1
  wj  j 1
  rj  c j  
 j 1   j 1 
By applying Eq. 2 to the data in Tab. 1(b), we obtain p = 87.7%. This relatively high value indicates
that the aggregation produces a consensus ordering reflecting the agents’ judgements quite well.
This impression is confirmed by the pj-chart in Fig. 3, which represents the profile of the agents’ pj

6
values, sorted in terms of importance (rj): the consensus ordering reflects the hierarchy of
importance of the agents relatively well, since pj values tend to decrease as the rj values decrease;
this means that the more important the agents, the higher the consistency between consensus
ordering and judgements.
pj 100%

90%
p = 87.7%

80%

70%

60%

50%
D1
D1 D2
D2 D3
D3 D44
D Agent
40% 30% 15% 15% rj

Fig. 3. pj-chart relating to the decision-making problem in Tab. 1.

The judgments by D4 are not very consistent with the consensus ordering (see the relatively low
value of p4 = 60%); however this inconsistency does not affect p significantly, since D4 (i) is the
least important agent (along with D3) and (ii) it provides a limited number of usable paired
comparisons (i.e., c4 = 4) for the consistency assessment.
We remark that, in the case of full democracy, i.e., when all agents are equi-important (rj = r  j-th
agent), p corresponds to the overall portion of consistent paired comparisons (among the usable
ones), related to the totality of the agents.

3. Application examples
This section illustrates the potential of p when (i) verifying the consistency of the solution provided
by a generic aggregation technique and (ii) comparing the solutions obtained from different
techniques, which are applied to the same decision-making problem. As an example, let us consider
a specific multicriteria decision-making problem aimed at identifying the best among six different
locations (i.e., the alternatives of the problem a, b, c, d, e and f), in which to install a new
manufacturing plant. The six alternatives are evaluated on the basis of 5 criteria (D1 to D5), i.e., the
agents of the problem. Each j-th agent is associated with an indicator (rj) reflecting its relative
importance. Tab. 3 describes the various agents and their typology of judgements. Two of the five
agents (i.e., D1 and D5) have a negative preference sense, since the degree of preference
increases/decreases as the judgement value decreases/increases. The agents’ judgments are reported
in Tab. 4.
Having defined the problem, the goal is to aggregate agents’ judgements into a single consensus
7
ordering. This aggregation will be performed through three techniques based on very different
principles and structures: (i) the Electre-II method (Figueira et al., 2005), (ii) the Borda’s count
(Borda, 1781), and (iii) the algorithm by Yager (2001). The authors are aware that much more
advanced techniques can be found in the literature (Herrera-Viedma et al., 2014); however, we
chose these ones not to complicate the discussion.

Tab. 3. Description of the agents/criteria of a fictitious decision-making problem concerning the selection of the
best location where to install a new manufacturing plant.
Agent description Form of the judgement Unit Preference rj
sense
D1 – Mean distance from the suppliers Measurement on a ratio scale [km] Negative 35%
of raw materials
D2 – Average technical skills of labour Evaluation on a 3-level ordinal scale N/A Positive 20%
(Low, Medium, High)
D3 – No. of typologies of available Count on an absolute scale [-] Positive 20%
energy sources (gas, oil,
renewables, electricity, etc.)
D4 – No. of available transport Count on an absolute scale [-] Positive 15%
infrastructures (roads, highways,
railways, ports, airports, etc.)
D5 – Average hourly cost of labour Measurement on a ratio scale [€] Negative 10%

Tab. 4. Judgments on the alternatives of interest (a, b, c, d, e, f), by five agents (D1 to D5). In the last row, the
judgements related to each agent are turned into the corresponding (linear) preference orderings.
Agent D1 D2 D3 D4 D5
rj 35% 20% 20% 15% 10%
Unit [km] N/A [-] [-] [€]
Preference sense Negative Positive Positive Positive Negative
a 20 H 4 3 24
Alternatives

b 140 L 3 2 20
c 80 H 3 4 24
d 200 M 2 3 14
e 120 M 4 3 16
f 140 M 1 2 20
Preference ordering a>c>e>(b~f)>d (a~c)>(d~e~f)>b (a~e)>(b~c)>d>f c>(a~d~e)>(b~f) d>e>(b~f)>(a~c)

These techniques generate the three consensus orderings reported in Tab. 5. Sects. A.1, A.2 and A.3
(in the appendix) contain a simplified illustration of each of the three techniques and the way the
relevant fused orderings are constructed.

Tab. 5. Consensus orderings resulting from the application of three aggregation techniques to the decision-
making problem summarized in Tab. 3.
Technique Consensus ordering
Electre-II method a > c > e > [(b ~ f) || d]
Borda’s count a>c>e>d>b>f
Yager’s algorithm e > (a ~ c) > (b ~ f) > d

The consistency of the fused orderings will be assessed using p. To facilitate the construction of p,
it is convenient to turn the input judgements into corresponding preference orderings (see the last
row of Tab. 4) and then into paired-comparison data (see Tab. 6). In this specific case, preference
orderings do not include omissions or incomparabilities between the alternatives, therefore they do
not result into any paired-comparison relationship of incomparability (“||”) .
8
Tab. 6. Paired-comparison data related to the judgments by the agents of interest (Tab. 4) and three consensus
orderings (Tab. 5).
Paired From agents’ judgements From consensus orderings
comparison D1 D2 D3 D4 D5 Electre-II Borda’s count Yager’s algorithm
a, b a>b a>b a>b a>b b>a a>b a>b a>b
a, c a>c a~c a>c c>a a~c a>c a>c a~c
a, d a>d a>d a>d a~d d>a a>d a>d a>d
a, e a>e a>e a~e a~e e>a a>e a>e e>a
a, f a>f a>f a>f a>f f>a a>f a>f a>f
b, c c>b c>b b~c c>b b>c c>b c>b c>b
b, d b>d d>b b>d d>b d>b b || d d>b b>d
b, e e>b e>b e>b e>b e>b e>b e>b e>b
b, f b~f f>b b>f b~f b~f b~f b>f b~f
c, d c>d c>d c>d c>d d>c c>d c>d c>d
c, e c>e c>e e>c c>e e>c c>e c>e e>c
c, f c>f c>f c>f c>f f>c c>f c>f c>f
d, e e>d d~e e>d d~e d>e e>d e>d e>d
d, f f>d d~f d>f d>f d>f d || f d>f f>d
e, f e>f e~f e>f e>f e>f e>f e>f e>f

Likewise preference orderings, the three consensus orderings can be turned into paired-comparison
data (see the last three columns in Tab. 6). It is worthwhile remarking that the ordering resulting
from the application of Electre-II (i.e., a>c>e>[(b~f)||d]) is partial (not linear), as some of the
alternatives are incomparable between each other (i.e., b||d and d||f) (Nederpelt and Kamareddine,
2004).
In Sects. 2.1 to 2.3, we present the construction of the indicator p, related to each of the three
techniques proposed; in Sect. 2.4, we present a comparison between the solutions generated by the
three techniques, by means of the pj-chart.

2.1 Case of the Electre-II method

The construction of the consensus ordering through the Electre-II method is illustrated in Sect. A.1
(in the appendix). Tab. 7 contains the corresponding consistency table (compare it with the data in
Tab. 6). The scores related to (b, d) and (d, f) are replaced with “N/A”, because of the
incomparabilities in the corresponding paired-comparison relationships from the consensus
ordering. The total number of usable paired comparisons is therefore c j  C2n6  2  13  j-th

agent.
The pj values related to the five agents are reported at the bottom of Tab. 7. The relatively high
values of these indicators denote a certain degree of consistency between consensus ordering and
agents’ judgements. The pj values relating to the first four agents in terms of importance are
relatively high (all larger than 80%). Instead, the pj value relating to D5 is very low (p5 = 26.9%);
this aspect is due to the negative correlation between this criterion (i.e., average hourly cost of
labour) and the other ones (see Tab. 7).
Consistency between consensus ordering and agents’ judgements looks good, as denoted by the
relatively high value of p = 82.9% (obtained by applying Eq. 3). Furthermore, the agents’

9
importance hierarchy seems to be reflected quite well, since the pj values tend to decrease as the rj
value decrease.

Tab. 7. Consistency table related to the solution provided by the Electre-II method (i.e., a>c>e>[(b~f)||d]) and the
judgements in Tab. 6.
Paired D1 D2 D3 D4 D5
comparison
a, b 1 1 1 1 0
a, c 1 0.5 1 0 0.5
a, d 1 1 1 0.5 0
a, e 1 1 0.5 0.5 0
a, f 1 1 1 1 0
b, c 1 1 0.5 1 0
b, d N/A N/A N/A N/A N/A
b, e 1 1 1 1 1
b, f 1 0.5 0.5 1 1
c, d 1 1 1 1 0
c, e 1 1 0 1 0
c, f 1 1 1 1 0
d, e 1 0.5 1 0.5 0
d, f N/A N/A N/A N/A N/A
e, f 1 0.5 1 1 1
rj 0.35 0.20 0.20 0.15 0.10
cj 13 13 13 13 13
xj 13 11 10.5 10.5 3.5
pj 100.0% 84.6% 80.8% 80.8% 26.9%
wj 4.6 2.6 2.6 2.0 1.3
pj∙(wj /wj) 0.350 0.169 0.162 0.121 0.027 p = 82.9%

2.2 Case of the Borda’s count

Sect. A.2 (in the appendix) illustrates the construction of the consensus ordering when applying the
Borda’s count. The three top-positions of the resulting ordering (i.e., a>c>e>d>b>f) coincide with
those in the solution by the Electre-II method (see Tab. 5), while there are some differences in the
lower positions. The corresponding consistency table, shown in Tab. 8, is slightly different from
that in Tab. 7, due to some variations in the rows corresponding to (d, b), (b, f) and (d, f). The p
value (77.2%) is slightly lower than in the case of Electre-II, denoting a certain deterioration in
terms of consistency.

10
Tab. 8. Consistency table related to the solution provided by the Borda’s count (i.e., a>c>e>d>(b~f) and the
judgements in Tab. 6.
Paired D1 D2 D3 D4 D5
comparison
a, b 1 1 1 1 0
a, c 1 0.5 1 0 0.5
a, d 1 1 1 0.5 0
a, e 1 1 0.5 0.5 0
a, f 1 1 1 1 0
b, c 1 1 0.5 1 0
b, d 0 1 0 1 1
b, e 1 1 1 1 1
b, f 0.5 0 1 0.5 0.5
c, d 1 1 1 1 0
c, e 1 1 0 1 0
c, f 1 1 1 1 0
d, e 1 0.5 1 0.5 0
d, f 0 0.5 1 1 1
e, f 1 0.5 1 1 1
rj 0.35 0.20 0.20 0.15 0.10
cj 15 15 15 15 15
xj 12.5 12 12 12 5
pj 83.3% 80.0% 80.0% 80.0% 33.3%
wj 5.3 3.0 3.0 2.3 1.5
pj∙(wj /wj) 0.292 0.160 0.160 0.120 0.033 p = 76.5%

2.3 Case of the Yager’s algorithm

This technique can be applied when (i) the agents’ judgements are expressed through linear
preference orderings (Nederpelt and Kamareddine, 2004) and (ii) the agents’ importance ranking is
expressed through a further linear ordering (Yager, 2001; Franceschini et al., 2016). In this specific
case, we can use the agents’ preference orderings reported at the bottom of Tab. 4 and the
importance rank-ordering D1>(D2~D3)>D4>D5, which was obtained from the agents’ rj values (in
Tab. 3).
Applying the Yager’s algorithm, we obtain the consensus ordering e>(a~c)>(b~f)>d, the
construction of which is described in Sect. A.3 (in the appendix). Tab. 9 contains the corresponding
consistency table. The Yager’s algorithm does not produce a perfect solution in terms of
consistency with agents’ judgements, as proven by the relatively low p value (72.5%). This result is
not surprising, as this technique was criticized by Wang (2007) and Franceschini et al. (2014) for
generating a consensus orderings, which are often inconsistent with agents’ judgments.

11
Tab. 9. Consistency table related to the solution provided by the Yager’s algorithm (i.e., e>(a~c)>(b~f)>d)) and
the judgements in Tab. 6.
Paired D1 D2 D3 D4 D5
comparison
a, b 1 1 1 1 0
a, c 0.5 1 0.5 0.5 1
a, d 1 1 1 0.5 0
a, e 0 0 0.5 0.5 1
a, f 1 1 1 1 0
b, c 1 1 0.5 1 0
b, d 1 0 1 0 0
b, e 1 1 1 1 1
b, f 1 0.5 0.5 1 1
c, d 1 1 1 1 0
c, e 0 0 1 0 1
c, f 1 1 1 1 0
d, e 1 0.5 1 0.5 0
d, f 1 0.5 0 0 0
e, f 1 0.5 1 1 1
rj 0.35 0.20 0.20 0.15 0.10
cj 15 15 15 15 15
xj 12.5 10 12 10 6
pj 83.3% 66.7% 80.0% 66.7% 40.0%
wj 5.3 3.0 3.0 2.3 1.5
pj∙(wj /wj) 0.292 0.133 0.160 0.100 0.040 p = 72.5%

2.4 Comparison between the three solutions

A comparison between the solutions generated by the three aggregation techniques can be
visualized by means of the pj-chart in Fig. 4, which contains three corresponding profiles.
Key: Electre-II (p=82.9%)
pj 100% Borda's (p=76.5%)
Yager's (p=72.5%)

80%

60%

40%

20%
D1
D1 D2
D2 D3
D3 D4
D4 D5
D5 Agent
35% 20% 20% 15% 10% rj

Fig. 4. pj-chart concerning the three aggregation techniques in use, when applied to the decision-making problem
in Tab. 4.

Despite the three profiles denote that the agents’ importance hierarchy is generally respected (pj
values tend to decrease as rj values decrease), it can be noticed that the solution by the Yager’s
algorithm is significantly poorer than that ones by the other two techniques. A comparison using the
p values leads to the same conclusion.

12
3. Discussion
The indicator p is a simple and intuitive tool for assessing the consistency between consensus
ordering and agents’ judgements, in a variety of multi-agent decision-making problems; the pj-chart
enriches the synthetic information given by p, showing the level of consistency between the
solution and the judgements by each j-th agent.
Also, p is very versatile since it can be applied in the presence of incomparabilities between
alternatives, both at the level of agents’ judgments and consensus ordering. It could be applied even
in the case in which the solution of the problem is expressed in forms that are different from a
consensus ordering – such as measurements/assessments on ordinal/interval/ratio scales – as long as
they can be transformed into paired-comparison relationships; possible examples are the solutions
generated by the Thurstone’s Law of Comparative Judgements (Thurstone, 1927; Franceschini and
Maisano, 2015b) or the AHP method (Saaty, 1980), which are evaluations expressed on an interval
and ratio scale respectively.
The indicator p was presented as a passive tool for checking the consistency of the solution of one
or more aggregation techniques, in a specific decision-making problem. Reversing this perspective,
p could be used actively, for identifying the “optimal2” consensus ordering(s) for a specific
problem, i.e., that one(s) for which p assumes the maximum possible value (i.e., pMAX). The optimal
consensus ordering(s) may be determined by: (i) automatically generating all the possible solutions
to a specific problem – i.e., the possible partial orderings, without loops – through ad hoc software,
(ii) determining the corresponding p for each of them, and (iii) selecting the solution(s) for which
p = pMAX. Knowing the optimal consensus ordering(s) and the corresponding pMAX value may be
useful for better assessing the performance of a certain technique, in a specific problem. For
example, a technique that provides a solution with a p = 79%, in a problem where pMAX = 80%,
performs certainly better than a technique that provides a solution with p = 80%, in a specific
problem where pMAX = 98%. In fact, in problems where the agents’ preference orderings have a high
degree of similarity, the pMAX value will tend to increase (since it will be easier to find a consensus
ordering that “satisfies” all of the agents and their importance ranking), while in others in which the
differences are greater, it will tend to decrease (since it will be more complicated to find a
consensus ordering that “satisfies” all of the agents and their importance ranking). In this sense,
pMAX could be also seen as a measure of the degree of “dissimilarity” between the agents’ preference
orderings.
A limitation of p is that it requires the importance hierarchy of agents to be expressed by means of
a set of importance indicators (rj), defined on a ratio scale. When it is expressed in other forms, it
would be necessary to introduce some (potentially questionable) transformations for determining

2
The concept of “optimal” is relative to the conventions adopted when defining p.

13
the rj values. For example, assuming that the agents’ importance hierarchy is expressed by an
importance rank-ordering, rj values can be obtained using the agents’ inverse rank positions. One
way to avoid introducing questionable transformations is to define a variant of p, which can be
adapted to specific problems in which rj values are not available; we plan to do this as part of our
future research.
Another limitation of p is inherent in the calculation and aggregation of the pj values. The proposed
operational procedure (Eq. 3) is based on potentially questionable conventions: e.g., (i) the scores
for determining the xj values reported in Tab. 2, (ii) the (weighted) additive model for synthesizing
the pj values, (iii) the direct proportionality of weights with respect to rj and cj values.
Future research will aim at complementing p with other practical tools for enriching the verification
of the solutions generated by different aggregation techniques.

References
Akritidis, L., Katsaros, D., Bozanis, P. (2011). Effective rank aggregation for metasearching. Journal of
Systems and Software, 84(1), 130-143.
Borda, J.C. (1781). Mémoire sur les élections au scrutin, Comptes Rendus de l’Académie des Sciences.
Translated by Alfred de Grazia as Mathematical derivation of an election system, Isis, 44: 42-51.
Brans, J.P., Mareschal, B. (2005). Multiple criteria decision analysis: state of the art surveys. Springer –
International series in operations research and management science. Figueira, J., Greco, S. and Ehrgott,
M. eds., Ch. Promethee Methods, pp. 163–195.
Cook, W.D. (2006). Distance-based and ad hoc consensus models in ordinal preference ranking. European
Journal of Operational Research, 172(2): 369-385.
Dwork, C., Kumar, R., Naor, M., Sivakumar, D. (2001). Rank aggregation methods for the web. In
Proceedings of the 10th international conference on World Wide Web (WWW ’01), 613-622, May 01-05,
Hong Kong.
Figueira, J., Greco, S., Ehrgott, M. (2005). Multiple criteria decision analysis: state of the art surveys.
Springer, New York.
Franceschini, F., Maisano, D., Mastrogiacomo, L. (2016). A new proposal for fusing individual preference
orderings by rank-ordered agents: a generalization of the Yager’s algorithm. European Journal of
Operational Research, 249(1): 209-223..
Franceschini, F., Maisano, D. (2015a) Checking the consistency of the solution in ordinal semi-democratic
decision-making problems. Omega, 57(B): 188-195.
Franceschini, Maisano, D. (2015b). Prioritization of QFD Customer Requirements based on the Law of
Comparative Judgments. Quality Engineering, 27(4): 437-449.
Godsil, C.D., Royle, G. (2001). Algebraic graph theory, vol. 207. New York: Springer.
Herrera-Viedma, E., Cabrerizo, F.J., Kacprzyk, J., Pedrycz, W. (2014). A review of soft consensus models in
a fuzzy environment. Information Fusion, 17, 4-13.
Nederpelt, R., Kamareddine, F. (2004). Logical Reasoning: A First Course. King’s College Publications,
London.
Ng, K.B., Kantor, P.B. (2000). Predicting the effectiveness of naive data fusion on the basis of system
characteristics. Journal of the American Society for Information Science, 51(13), 1177-1189.
Nuray, R., Can, F. (2006). Automatic ranking of information retrieval systems using data fusion. Information
Processing & Management, 42(3), 595-614.
Saaty, T.L. (1980). The Analytic Hierarchy Process: Planning, Priority and Allocation, New York: McGraw-
Hill.
Thurstone, L.L. (1927). A law of comparative judgments. Psychological Review, 34: 273-286.
Vora, A., Paunwala, C.N., Paunwala, M. (2014). Improved weight assignment approach for multimodal
fusion. Proceedings of the 2014 International Conference on Circuits, Systems, Communication and
Information Technology Applications (CSCITA 2014), 70-74.
Wang, J. (2007). Fusion of multiagent preference orderings with information on agent's importance being

14
 incomplete certain. Journal of Systems Engineering and Electronics, 18(4): 801-805.
Wu, S., McClean, S. (2006). Performance prediction of data fusion for information retrieval. Information
processing & management, 42(4), 899-915.
Yager, R.R. (2001) Fusion of multi-agent preference orderings. Fuzzy Sets and Systems, 117(1): 1-12.
Zopounidis, C., Pardalos, P. (2010). Handbook of multicriteria analysis (vol. 103 in Applied Optimization),
Berlin: Springer.

Appendix
A.1 Description and application of the Electre-II method
This section provides a simplified description of the Electre-II method and its application to the
agents’ judgements in Tab. 4; for more information, see (Figueira et al., 2005).
In the initial phase of Electre-II, agents’ judgements are turned into sets of paired-comparison
relationships and then aggregated into a single set of outranking relationships (see Tab. A.1). The
third-to-fifth columns of Tab. A.1 report the indicators W+, W= and W– (also denominated as
“consistency scores”), i.e., the sums of the rj values of the agents for which a generic paired
comparison (a, b) results in the relationship a > b, a = b and b > a respectively. In detail, having
defined J+, J= and J– as the sets of agents for which a > b, a = b and b > a respectively, the
indicators W+, W= and W– are formally defined as:

W  r j where J   D j : a  b

jJ

W  r j where J   D j : a ~ b, (A.1)

jJ 

W  r j where J   D j : b  a
jJ 

These indicators are then used to perform the so-called concordance test, which is based on the
verification of both the conditions:
W  W 
 c1
W
, (A.2)
W
 c2
W
being W = W+ + W= + W-, and c1 and c2 two thresholds (generally) set to 0.7 and 1 respectively
(Figueira et al., 2005). When, for a generic paired comparison (a, b) the concordance test is passed,
it can be stated that aOb, where the symbol “O” denotes the outranking relationship. Results can be
visualised in a graph (see Fig. A.1, step 1) in which vertices represent alternatives and edges joining
two vertices represent the corresponding outranking relationship.

15
Tab. A.1. Determination of outranking relationships (“O”) between pairs of alternatives, in the initial phase of
the Electre-II method.

Paired Consistency scores Concordance test

comparison W+ W= W– (W++W=)/W (c1) W+/W- (c2) Outranking relationship
1 a, b 90 0 10 0.90  9.00  aOb
2 a, c 55 30 15 0.85  3.67  aOc
3 a, d 75 15 10 0.90  7.50  aOd
4 a, e 55 35 10 0.90  5.50  aOe
5 a, f 90 0 10 0.90  9.00  aOf
6 b, a 10 0 90 0.10  0.11  -
7 b, c 10 20 70 0.30  0.14  -
8 b, d 55 0 45 0.55  1.22  -
9 b, e 0 0 100 0.00  0.00  -
10 b, f 20 60 20 0.80  1.00  bOf
11 c, a 15 30 55 0.45  0.27  -
12 c, b 70 20 10 0.90  7.00  cOb
13 c, d 90 0 10 0.90  9.00  cOd
14 c, e 70 0 30 0.70  2.33  cOe
15 c, f 90 0 10 0.90  9.00  cOf
16 d, a 10 15 75 0.25  0.13  -
17 d, b 45 0 55 0.45  0.82  -
18 d, c 10 0 90 0.10  0.11  -
19 d, e 10 35 55 0.45  0.18  -
20 d, f 45 20 35 0.65  1.29  -
21 e, a 10 35 55 0.45  0.18  -
22 e, b 100 0 0 1.00  > c2  eOb
23 e, c 30 0 70 0.30  0.43  -
24 e, d 55 35 10 0.90  5.50  eOd
25 e, f 80 20 0 1.00  > c2  eOf
26 f, a 10 0 90 0.10  0.11  -
27 f, b 20 60 20 0.80  1.00  fOb
28 f, c 10 0 90 0.10  0.11  -
29 f, d 35 20 45 0.55  0.78  -
30 f, e 0 20 80 0.20  0.00  -
c1 and c2 have been set to 0.7 and 1 respectively.

The second phase of the Electre-II method is aimed at deriving a consensus ordering from the
outranking relationships. To this purpose, the outranking relationships are turned into so-called
preorders, either using a top-down or bottom-up procedure. In the top-down procedure, the
preliminary step is to identify and eliminate possible circuits, i.e., those combinations of outranking
relationships in which the transitivity property is violated – e.g., aOb, bOc and cOa. For each of the
circuits identified, the corresponding outranking relationships are deleted and the alternatives are
grouped in the same class and considered as indifferent (e.g., a~b~c). Next, the alternatives that are
not outranked by any other alternative are determined; this defines the first class of the top-down
procedure. The alternatives in that class are then deleted and the exploitation procedure is repeated
until all alternatives have been classified to obtain a (top-down) preorder. The bottom-up procedure
is analogous but it starts selecting the class of worst alternatives (alternatives outranking no other
alternatives) and reiterating the procedure.

16
 step 1 step 2 step 3 step 4 step 5
b b b b

a c c
Top-down
procedure
f d f d f d f d

e e e
Top-down preorder:
a>… a>c… a > c > e >… a > c > e > [(b ~ f) || d] Consensus ordering:
a > c > e > [(b ~ f) || d]
b

a c a c a c a
Bottom-up
procedure
f d

e e
Bottom-up preorder:
… > [(b ~ f) || d] … > e > [(b ~ f) || d] … > c > e > [(b ~ f) || d] a > c > e > [(b ~ f) || d]

Fig. A.1. Basic steps for transforming the outranking graph (step 1) in a consensus ordering (step 5), in the
second phase of the Electre-II methods; the alternatives selected in each step of the top-down and bottom-up
procedure are circled. For further information, see (Figueroa et al., 2005).

The two procedures generate two preorders, which do not necessarily coincide; for example, an
alternative which is not outranked and does not outrank any other alternative will appear as first on
the list of the top-down preorder and it will appear as last in the bottom-up one. To handle this
problem, it is necessary to build a single ordering with the combined result of the top-down and
bottom-up preorders. Several ways have been suggested to perform this synthesis (Godsil and
Royle, 2001). In this particular case, the top-down and bottom-up preorders are coincident, so there
is no need for further synthesis (Figueroa et al., 2005).

A.2 Description and application of the Borda’s count

The Borda’s count consists of two basic steps: (i) turning the judgments of each (j-th) agent into a
preference ordering, (ii) associating a score (kij) to the alternatives, corresponding to their
rank-position in the preference ordering (see Tab. A.2), and (iii) synthesizing the scores obtained by
each alternative, in an overall score – also known as Borda’s score (Bi) – which corresponds to a
weighted average based on the agents’ rj values. In formal terms, Bi is defined as:

Bi   rj  kij  ,
m
(A.3)
j 1

being:
rj the relative importance of the j-th agent;
kij the rank-position of the i-th alternative in the preference ordering by the j-th agent;
m the total number of agents.

17
Tab. A.2 Application of the Borda’s count to the decision-making problem in Tab. 3. The resulting consensus
ordering (i.e., a>c>e>d>b>f) is obtained by ordering the alternatives decreasingly with respect to their Bi values.
Alternative Rank-position in the preference orderings Bi Rank position in the
D1 D2 D3 D4 D5 consensus ordering
35% 20% 20% 15% 10%
a 1 1 1 2 5 155 1st
b 4 6 3 5 3 425 5th
c 2 1 3 1 5 215 2nd
d 6 3 5 2 1 410 4th
e 3 3 1 2 2 235 3rd
f 4 3 6 5 3 625 6th

The resulting consensus ordering is then constructed by ordering the alternatives in ascending order
with respect to their Bi values. By applying the Borda’s count to the input judgements in Tab. 3, the
results shown in Tab. A.2 are obtained.

A.3 Description and application of the Yager’s algorithm

The algorithm by Yager (2001) can be applied to specific decision-making problems in which:
 the agents’ judgements are expressed in the form of linear orderings (Nederpelt and
Kamareddine, 2004);
 the agents’ preference orderings do not include any omitted or incomparable alternative;
 the agents’ importance hierarchy is expressed in the form of a rank-ordering and not through the
typical set of importance values.
Summarizing, the algorithm is based on the following four steps:
1. Transformation of the agents’ linear orderings into preference vectors, according to the following
convention: we place the alternatives as they appear in the ordering, with the most preferred
one(s) in the top positions; if at any point t > 1 alternatives are tied (i.e., indifferent), we place
them in the same element and then place the null set (“Null”) in the next t – 1 lower positions. In
this way, the total number of elements of a vector will coincide with the number alternatives.
Tab. A.3 exemplifies the construction of the preference vectors from the orderings the last row of
Tab. 4.
2. Transformation of the preference vectors into “reorganized” vectors. This transformation
consists in (i) sorting the Di vectors decreasingly with respect to the agents’ importance and (ii)
aggregating those with indifferent importance (e.g., D2 and D3 in the example) into a single
vector. This aggregation is performed through a level-by-level union of the vector elements,
where alternatives in elements with the same position are considered as indifferent. The resulting
reorganized vectors are reported in Tab. A.4.
3. Definition of a sequence for the element-by-element reading of the reorganized vectors. The
sequence is determined applying a lexicographical order based on two dimensions: (i) level of
vector elements (from the bottom to the top) and (ii) importance of agents (in decreasing order).

18
 Tab. A.4 reports the full sequence numbers (S) associated with each element of the reorganized
vectors.
4. Construction of the fused ordering: alternatives are progressively included into a gradual
ordering, which is initially Null. A k-th alternative is included at the top of the gradual ordering,
at the first occurrence in the element-by-element reading sequence. In the previous example, the
resulting consensus ordering is: e>(a~c)>(b~f)>d. Tab. A.5 shows the step-by-step procedure for
constructing the consensus ordering.

Tab. A.3. Construction of preference vectors related to the orderings by four fictitious agents (D1 to D5).
Agents D1 D2 D3 D4 D5
Orderings a>c>e>(b~f)>d (a~c)>(d~e~f)>b (a~e)>(b~c)>d>f c>(a~d~e)>(b~f) d>e>(b~f)>(a~c)
j Elem. Elem. Elem. Elem. Elem.
6 {a} {a, c} {a, e} {c} {d}
5 {c} Null Null {a, d, e} {e}
Preference
4 {e} {d, e, f} {b, c} Null {b, f}
vectors
3 {b, f} Null Null Null Null
2 Null Null {d} {b, f} {a, c}
1 {d} {b} {f} Null Null
n=6 total alternatives are considered: a, b, c, d, e and f.
The agents’ importance ordering is D1>(D2~D3)>D4>D5.
j is the level of a certain element in the corresponding preference vector.

Tab. A.4. Reorganized vectors obtained by merging the preference vectors in Tab. A.3 and corresponding
sequence numbers (S) of their elements.

Agents (D1) (D2 ~ D3) D4 D5

j S Elem. S Elem. S Elem. S Elem.
6 21 {a} 22 {2a, c, e} 23 {c} 24 {d}
5 17 {c} 18 Null 19 {a, d, e} 20 {e}
Reorganized
4 13 {e} 14 {b, c, d, e, f} 15 Null 16 {b, f}
vectors
3 9 {b, f} 10 Null 11 Null 12 Null
2 5 Null 6 {d} 7 {b, f} 8 {a, c}
1 1 {d} 2 {b, f} 3 Null 4 Null

Tab. A.5. Step-by-step construction of the fused ordering when applying the YA to the preference vectors in Tab.
A.4.
Step (S) Element Residual alternatives Gradual ordering
0 - {a, b, c, d, e, f} Null
1 {d} {a, b, c, e, f} d
2 {b, f} {a, c, e} (b ~ f) > d
3 Null {a, c, e} (b ~ f) > d
4 Null {a, c, e} (b ~ f) > d
5 Null {a, c, e} (b ~ f) > d
6 {d} {a, c, e} (b ~ f) > d
7 {b, f} {a, c, e} (b ~ f) > d
8 {a, c} {e} (a ~ c) > (b ~ f) > d
9 {b, f} {e} (a ~ c) > (b ~ f) > d
10 Null {e} (a ~ c) > (b ~ f) > d
11 Null {e} (a ~ c) > (b ~ f) > d
12 Null {e} (a ~ c) > (b ~ f) > d
13 {e} Null e > (a ~ c) > (b ~ f) > d
End - - -

19