7 Conclusions and Future Research
The ability of ANNs to learn and generalize from examples, and to generate
robust solutions, makes them very suitable in a diversity of applications where
algorithmic approaches are either unknown or difficult to implement. A major
drawback, however, is that the knowledge learned by the network is represented
in an exceedingly opaque form, namely, as a list of numerical coefficients. This
black-box character of ANNs hinders the possibility of their more wide-spread
acceptance. The problem of extracting the knowledge embedded in the ANN in
a comprehensible form has been intensively addressed in the literature.
Another drawback of ANNs is that standard training algorithms do not guar-
antee convergence, and are highly dependent on the initial values of the networks
parameters. Efficient methods for determining the initial architecture and param-
eter values of ANNs are quite important, as they may improve the trained ANN
generalization capability, and reduce training times. In many problems some ini-
tial information is known, and an important problem is how this knowledge can
be used in order to design an initial ANN.
In this work, we introduced a novel fuzzy rule-base, the FARB, and showed
that its IO mapping is mathematically equivalent to that of an ANN. We used
this equivalence to develop new approaches for: (1) extracting knowledge from
trained ANNs, and representing it in a comprehensible form; and (2) knowledge-
based design of ANNs based on prior knowledge. These applications were illus-
trated for both feedforward ANNs and first- and second-order recurrent ANNs.
For large-scale networks, the corresponding FARB may include either a large
number of rules or complicated rules, and thus hamper the FARB comprehen-
sibility. In order to minimize the effect of this manifestation of the curse of
dimensionality [11], we also presented a systematic procedure for rule reduction
and simplification.
7.1 Future Research
The FARB is based on standard tools from the field of fuzzy logic, and the
FARB–ANN equivalence holds for a large variety of standard ANNs. This opens
E. Kolman, M. Margaliot: Knowledge-Based Neurocomputing, STUDFUZZ 234, pp. 77–81.
[Link] c Springer-Verlag Berlin Heidelberg 2009
78 Conclusions and Future Research
the door to many more potential applications which are based on transferring
concepts and ideas from fuzzy logic theory to ANNs, and vice versa.
7.1.1 Regularization of Network Training
The ANN–FARB equivalence provides a convenient tool for extracting symbolic
knowledge from trained ANNs. It is highly desirable that the extracted rule-base
include a small number of rules that are as simple as possible. Two potential
approaches for achieving this goal are: (1) first extract the FARB and then
apply some simplification procedure (as described in Chapter 4); and (2) during
training, add regularization terms [150] that will force the ANN to develop a
skeletal form, and then extract the equivalent FARB.
When using the latter approach, it is of course possible
2 to use regularization
terms that are standard in the ANN literature, e.g., wij [69]. However, if we
i,j
know beforehand that the KE approach is based on the ANN–FARB equivalence,
then it is natural to try and develop specialized regularization terms that are
particularly suitable for this approach, that is, terms that will potentially yield
a simple as possible FARB.
To make this more concrete, consider a FARB with the fuzzy terms larger
than 0 and smaller than 0 (we assume that the bias terms are included in the
xi s). A typical rule is then:
Rk : If x1 is larger than 0 and x2 is larger than 0 and . . . and xm is larger than
0, Then f = . . .
Suppose that there exist two indexes i = j and a number α > 0 such that
xi ∼
= αxj , (7.1)
for all the training samples.
Let Rk denote the rule that is obtained from Rk by deleting the the jth atom.
Condition (7.1) then implies that Rk and Rk will have similar DOFs, since a
high (low) truth value for the atom ‘xi is larger than 0’ will correspond to a high
(low) truth value for the atom ‘xi is smaller than 0’. Thus, the jth atom can
be deleted from the FARB, yielding a simpler FARB, with little effect on the
output.
We would like to add a regularization term that will force the ANN to sat-
isfy (7.1). For two vectors g, h ∈ Rn \ {0}, let
gT h
b(g, h) := ,
||g|| ||h||
that is, b = cos(θ), where θ is the angle between g and h. Recalling that
xi = (wi )T z, where wi = (wi1 , . . . , win )T , and z is the ANN input, we see
that requiring (7.1) is identical to requiring that b(wi , wj ) is maximized.
This suggests that a suitable cost criterion might be
E−λ b(wi , wj ),
i,j